What process should a company go through to determine whether big data is a fit for their goals?

They should figure out if they really have a big data problem first. This issue often starts with a misunderstanding about what big data is and isn't. In many cases, small data technology like a relational database will do the job. Typically, big data problems can be identified because the source of the problem is is being specifically held back by the limitations of small data technology. When organizations hit a known technical limitation, the way to solve this problem is likely with big data technologies.

Next, companies need to deeply understand the use case for big data. Companies and individuals who don't do this will often fail in their projects. The truth is that there isn't a specific step-by-step guide for developing a use case. This is where data engineers can help get the the information from the use case to create a data pipeline.

Finally, companies can start looking at big data technologies. Skipping these steps leads companies to use big data without a need for big data or to choosing the wrong technologies for the job.