When you buy through affiliate links in our content, we may earn a commission at no extra cost to you. Learn 
As it is, says Cloudera CEO Mike Olson, the Hadoop analytics platform has some significant limitations involving speed and complexity. However, industry leaders – including those at his own firm – are working to change that, confident that this research tool will evolve in fundamental ways in the coming years.
At last year’s Strata Conference, Olson asked an audience to imagine how, if it were created today, Hadoop would be designed differently, positing that some specific changes could open new gateways to research in fields like education, energy and agriculture. Citing cosmologist and scientific author Carl Sagan, Olson stressed the importance of breaking down high-level data for a greater audience.
"I think we are poised on the edge of major discoveries that matter for business and for society," Olson said.
One example, he noted, is the scientific discovery made possible by the work of scientific professionals at the Large Hadron Collider in Switzerland. According to Olson, it’s the daily evaluation of 37 terabytes of data that have led to key advancements like the watershed Higgs-Boson discovery, which revealed key aspects of matter and energy. Olson added that Hadoop helps to assist in these fundamental research projects by capturing an enormous amount of data in a single repository, but that it is hard to interact with that data in real time.
Expounding on the ways that Hadoop and similar technologies can help scientists in key areas, such as health care, Olson also announced the Impala project, something that his company has been working on for two years, with two quarters of beta testing in coordination with major clients. The Impala project, said Olson, is a "100% open source" technology with Apache licensing that works with Hadoop as a "real-time query engine." This, Olson said, allows what he calls "speed of thought queries," where users can ask a question, get a response and formulate a new question to effectively use major data clusters.
"It gives you a new way to get at your data," Olson said, outlining how this kind of advancement could break open additional opportunities to solve major problems in the U.S. and abroad.
