January 1, 2015
By Maria Minsker, contributing writer

Is Hadoop Worth the Hype?

future behavior," Applebaum says.

Yet even Hadoop has its limitations.

Obstacles to Adoption

In the scheme of other database technology, Hadoop is still fairly immature. There's a steep learning curve associated with it, and the resulting talent gap has made it difficult for organizations to keep Hadoop experts on staff. "The skills are in high demand right now," Stirrup says. "Companies finally hire that one Hadoop person, and he immediately gets a higher pay offer and leaves. It's tough to learn, and there are not that many out there that have learned it," he adds.

There are other challenges too. Hadoop's ability to process massive amounts of data, for example, is both a blessing and a curse. Because it's designed to handle large data loads relatively quickly, the system runs in batch mode, meaning it processes massive amounts of data at once, rather than looking at smaller segments in real time. As a result, the system often forces users to choose between quantity and quality. At this point in Hadoop's life cycle, the focus is more on enormous data size than high-performance analytics.

And, because of the large size of the data sets fed into Hadoop, the number-crunching doesn't take place in real time. (Though once data is processed, it can be queried in real time.) This is problematic because "as the time between when you input the data and the time at which you have to make a decision based on that data grows, the effectiveness of that data decreases," says Arsalan Tavakoli, director of customer engagement at Databricks, a company that builds big data software.

The biggest problem of all, experts agree, is that Hadoop's seeming boundlessness instills a proclivity for data exploration in those who use it. Relying on Hadoop to deliver all the answers without "asking the right questions" will prove to be "terribly inefficient," Applebaum says. Extracting value and meaning from the data stored and processed in Hadoop has been a pain point for many early adopters, and is likely one of the main reasons why widespread adoption has been slow.

"It's powerful, but it's daunting for many reasons, so companies have been reluctant to get on board," Tavakoli says. But that's about to change.

As companies begin to recognize Hadoop's potential, demand is increasing, and vendors are actively developing solutions that promise to painlessly transfer data onto Hadoop, improve its processing performance, and operationalize data to make it more business-ready.

The Growing Market

Big data integration vendor Talend, for example, offers solutions that help organizations transition their data onto Hadoop in high volume. The company works with more than 800 connectors that link up to other data systems and warehouses to "pull data out, put it into Hadoop, and transform it into a shape that you can run analytics on," Stirrup explains.

Talend customer Lenovo replaced its custom data integration solution with Talend Enterprise Big Data, and was able to save significantly

Previous Page Next Page