Analytics Avalanche

A version of this article first appeared in eB-21, published 10 times a year in Europe by TBC Research (www.eb-21.com). Based in London and San Francisco, TBC Research helps senior business professionals make more informed technology decisions through its magazine, research, and events portfolio.

An inrushing tide of customer information has created a new challenge for data warehouse vendors. Now this process is not just about gathering data from various systems, it's also about doing it in a reasonable timeframe.

"When a company lays out a business problem, they have a business cycle," says Gaurav Dhillon, CEO of analytics vendor Informatica. "They might be figuring out an optimal pricing plan, or building a loyalty programme. But if you cannot solve the problem in the business cycle, you have no place competing for the project. The five year ERP projects won't be repeated." Research from IDC purports to demonstrate that the emphasis in data warehousing is also shifting on to information access tools such as OLAP, query and reporting, data mining and EIS systems. Its latest five-year forecast shows this sector overtaking data management and generation to take more than half of the market by 2004. In 1999 it had just 36 per cent. Such predictions have forced platform vendors to rush out analytic applications that sit on top of their backbone products.

But if this prediction holds true it's not because users have finished building their data warehouses, it's because the backbone is dominated by a few players: Oracle, IBM, Microsoft, Informix and NCR. In addition, the applications that feed them are better structured. "Companies have put in place ERP systems," says IDC analyst Dan Vesset, "and they're now in the middle of installing transactional CRM systems. Data is therefore in a more standard format than in legacy systems. The opportunity to build a data warehouse is greater, although it's still not easy because of the integration of different geographies and systems."

In fact, despite the shift, vendors warn you should ignore data generation and management at your peril. "Data is very much the poor relation in CRM and e-commerce projects. The amount of cases where companies have spent millions on projects but swept data issues under the carpet is frightening," says Simon Jennings, VP of Northern Europe at ETI. ETI provides a tool, ETI Extract, which automatically generates code to extract and load a data warehouse.

It's finding over half its business has switched to customer-oriented data and it has scored success with companies such as Anglian Water and Lloyds TSB. Jennings stresses the need for good meta data, the so-called "data about data", which applies context to raw data. "Vendors have huge issues around semantic translation, where different meanings of identical data can be reconciled," he says.

He gives the example of policy maturity data in insurance where the same figure could mean different things in different policies, so the net worth of a customer will not be the same. With good meta data, such issues can be overcome. "When you are consolidating customer information from multiple sources, such as after a merger, the semantic issues are huge. Putting in CRM is like becoming extremely missionary; it's not just about going to church on Sunday," says Jennings.