What process should a company go through to determine whether big data is a fit for their goals?
They should figure out if they really have a big data problem first. This issue often starts with a misunderstanding about what big data is and isn't. In many cases, small data technology like a relational database will do the job. Typically, big data problems can be identified because the source of the problem is is being specifically held back by the limitations of small data technology. When organizations hit a known technical limitation, the way to solve this problem is likely with big data technologies.
Next, companies need to deeply understand the use case for big data. Companies and individuals who don't do this will often fail in their projects. The truth is that there isn't a specific step-by-step guide for developing a use case. This is where data engineers can help get the the information from the use case to create a data pipeline.
Finally, companies can start looking at big data technologies. Skipping these steps leads companies to use big data without a need for big data or to choosing the wrong technologies for the job.
Tags
Written by Jesse Anderson | Data Engineer, Creative Engineer and Managing Director at Big Data Institute

Jesse Anderson is a Data Engineer, Creative Engineer and Managing Director of Big Data Institute.
He trains and mentors companies ranging from startups to Fortune 100 companies on Big Data. This includes training on current edge technologies and Big Data management techniques.
He’s mentored hundreds of companies on their Big Data journeys. He has taught thousands of students the skills to become Data Engineers. He is widely regarded as an expert in the field and his novel teaching practices.
Jesse is published on O’Reilly and Pragmatic Programmers. He has been covered in prestigious publications such as The Wall Street Journal, CNN, BBC, NPR, Engadget and Wired.
More Q&As from our experts
- What are some of they key things that cause big data projects to fail?
- What's the difference between model-driven AI and data-driven AI?
- How is AI technology going to affect the workplace in the near future?
Related Terms
- Stereoscopic Imaging
- Cloud Provider
- Subscription-Based Pricing
- Data Center
- New Media
- Client-side
- Deterministic System
- Gesture Recognition
- Paperless Office
- Quantum Computing
Related Articles

Big Data and 5G: Where Does This Intersection Lead?

Multimodal Learning: A New Frontier in Artificial Intelligence

Uncovering Security Breaches

10 Big Data Do's and Don'ts
Tech moves fast! Stay ahead of the curve with Techopedia!
Join nearly 200,000 subscribers who receive actionable tech insights from Techopedia.
- Data Governance Is Everyone's Business
- Key Applications for AI in the Supply Chain
- Service Mesh for Mere Mortals - Free 100+ page eBook
- Do You Need a Head of Remote?
- Web Data Collection in 2022 - Everything you need to know
- How to Protect Microsoft 365 from NOBELIUM Hackers
- 5 Steps to Streamline Security for your Hybrid Network