Wait - what? Isn't big data the data everyone is talking about? It is, but maybe small data deserves a bigger piece of the conversation. Here we'll take a look at what small data is and how it can often pack a bigger punch than big data.
What Is Small Data?Small data is captured data that is discrete and precise enough to be comprehended by the human brain. Typically, it is collected for a specific purpose for a singular unit of an organization, such as recording how much actual effort is being spent on different activities by individuals in a team. The reason for collecting small data is established at the outset. In this case, it would be collected with the goal of optimizing how a team delivers its value.
By comparison, big data’s focus is collecting as much related information across the organization as possible, and then analyzing it to determine how it can help answer questions. What do our sales statistics tell us about market trends and further sales opportunities? How good is our support team at handling customer queries? Where do we need to improve our project delivery process to reduce overshoot against estimated budget?
It may seem obvious, but big data needs data as input, and lots of it. Very often, additional small data is required to support big data as answers to initial questions raise further ones. Plus, in order to perform analysis of this information there are a multitude of enterprise-level tools offered by vendors, tools that require significant investment and time to bring in-house, set up and configure to start giving results. It’s a systems integration project from the outset to connect with all the sources of data, and one that can take a number of months before business benefit is delivered.
Conversely, small data requires little analysis, can be captured in many ad hoc ways - such as in spreadsheets, task- and time-tracking tools, and even manual log books - and can be analyzed quickly and easily. I’ve seen benefits be realized from small data within a week or two of the outset of a productivity engagement. And that’s only because it takes a little time to capture the raw information. Typically, changes and benefits become evident quickly because of the focus of the data collected.
Small Data's Big BenefitsFrom my experience in coaching and managing teams, the following benefits result from small data for individuals and teams:
Small data can provide awareness of where individuals are actually focusing their time and energy versus what would give even greater value. Often when individuals start to capture small data, they quickly realize the significance of what they discover.
Through small data, individuals may identify changes they can put into action and be supported in doing by other members of the team. The team members become responsible for and drive their own change.
Measuring and being recognized for the positive changes achieved can create a greater sense of mutual understanding, worth and connection.
How Small Data Is CapturedAcross a software development department, big data can analyze the project plan information, making it possible to analyze the number of people, duration and effort required to deliver different types of projects. What’s missing is how each individual actually performs their project tasks on a day-to-day basis. By capturing this small data, we can start to learn how best to structure the project, its teams and their working day. What types of tasks does each person enjoy and do well? What would they like to delegate or drop? What types of communication work best with whom? What level of direction and mentoring do individuals need?
By changing the how, we derive benefits that are visible at the big data level, but not the changes that led to this. Analysis of big data can often result in a generalized model, for example, assuming that each person has a similar skill and experience level. Only by looking at the small data specifics of how each person works and contributes to the project (in their unique way), can these types of benefits be achieved.
Where Small Data Is UsedThere is definitely value to be gained from using big data, but recent reviews of the marketplace and product offerings are finding confusion around best practice and how to derive the best value from an implementation. A recent review by Gartner found that only 8% of companies surveyed have implemented big data analytics and 57% are still in research and planning stages.
For any data analysis, the key is not to pull in all the data you have and then try and look for value, it's to use data that can help in answering particular questions. And this is where small data wins out for two key reasons:
- The desired value and reason for collecting the data has to be understood up front.
- Small data gives both qualitative and quantitative answers, enabling precise changes to be made. In other words, there are fewer generalist assumptions made in small data.
Ultimately, small data will not replace big data, but there is a lot that a small data engagement can teach big data in how to get the best from both approaches. In considering any big data implementation, ask yourself what small data questions would help you gain value. It may help pack that greater punch into your resulting strategy. (Read another perspective on the value of big data in business in Can Big Data Analytics Close the Business Intelligence Gap?)