In Search Of...
Powerful, fast, and accurate search of the Web has been the norm for years now, to the point where tediously scrolling through irrelevant search results is almost a hazy memory. But while Google and its competitors have brought a semblance of order to the wild world of the Internet, frustrating and often futile searches still happen every day: whenever you're looking for information from documents stored on your company's own computers. The ever-growing stash of computer files, emails, voice recordings, multimedia products, and other important data generated every day can stymie even the most dedicated attempts to keep it all straight.
That's where enterprise search comes in. A broad category of solutions involving both hardware and software, and capable of searches ranging from a few hundred documents in a small company to billions in a Fortune 500 corporation, enterprise search is rapidly growing in stature as the number of documents generated--and the importance of being able to find them, and find them quickly--increases exponentially. Enterprise search represents the promise of being able to find that purchase order from two years ago buried in a company server with the same speed and accuracy as a Web search for Babe Ruth's lifetime statistics.
Companies are realizing that they may never be able to get all their information into a single database--a commonly stated goal a decade ago--so they need search to make sense of data while it remains spread throughout the enterprise. "One of the drivers for [enterprise search] is that when paper was king and not a whole lot was digital, you could find things online fairly easily because there wasn't much there," says Guy Creese, research director for the Burton Group. "Now we've moved from information scarcity to information overload.... It's a huge issue when the document you're looking for is one of tens of thousands."
"Search is no longer a nice-to-have. It's an absolute have-to-have," says James Waters, vice president of global marketing for enterprise search vendor Coveo. "There's a lot more to the data scene than there was even two years ago.... Think of the content that's coming: Forget PDFs and Excel spreadsheets--think of email, voicemails that go to email, wikis, blogs, audio and video, and all the Web 2.0 applications that are not currently searchable."
And yet, as the importance of search within the enterprise grows, many companies, especially small and midsize businesses (SMBs), have been slow to adopt the technology, the specifics of which are often vague to outsiders. "It's not a market that people are very well-educated about," says Matthew Brown, principal analyst for Forrester Research. "I would guess that if you went to SMBs and asked 100 of them if they knew what enterprise search was, and if they'd bought a product, maybe five to 10 would know what it is--and [only] two or three would have bought a product. So there's a huge opportunity there."
UNDERSTANDING
ENTERPRISE SEARCH
There are several important differences between Web search and enterprise search, not least of which is the fact that enterprise search is inherently far more difficult. Web search is based on algorithms that analyze naturally occurring metadata, such as the title of a site and how many other pages link to a given page. With no organic metadata to work with, enterprise search is a different and more complex animal, particularly since users want information that's of the best quality, not just the most popular. "If we talk about search in a generic fashion, one of the difficulties is that humans are involved--and, especially in tech search, language is involved," Creese notes. "ATM" means something different to a banker than it does to a programmer, and search needs to take that into account in order to provide relevant results.
A search application borders on useless if it requires users to invest time to tag their own documents for later retrieval, so enterprise search offers speedy, automatic metadata retrieval from multiple disparate sources, with no manual input from users. Several search solutions rely on technology developed by Teragram, a maker of language-processing products that automatically generate metadata for enterprise search. Using software to scan documents and tag them based on the information they contain, the technology makes it easier to organize those documents--and to search for them later using an enterprise search product.
Even so, search engines have trouble grasping natural language, so a big part of implementing enterprise search involves cleaning up content, says Theresa Regli, principal at the analyst firm CMS Watch, which produces The Enterprise Search Report. That involves coming up with descriptive names for data and making sure files of the same type are structured the same way. Regli says she advises companies considering an enterprise search solution to focus on cleaning up their data first--before going shopping. Otherwise, they risk reaching a point where the search solution gets stuck and the data will require cleanup anyway. "Enterprise search has been trying really hard to overcome the messiness problem, but I haven't seen anyone who's really been able to it," says Regli.
Security is another issue that becomes highly important in enterprise search. In a typical Web search, all the information one is searching for is generally assumed to not only be present online, but generally available to everyone. Enterprise search, by contrast, must include the ability to search for documents that only certain people should have access to, such as trade secrets and salary information. A successful enterprise search tool will make those documents available to the people who need them, but will not even let them appear as a result in a search by unauthorized personnel. Sorting and categorizing information in this way presents an added challenge on top of com-
piling documents in a searchable way.
A key part of selecting an enterprise search solution that's right for you is figuring out what capabilities you'll use. Regli says she often sees smaller companies buying powerful and expensive enterprise search engines from companies, such as Autonomy or Endeca, that provide simply more software than is needed. "The real message is you don't want to buy a six-burner stove to boil water," she says. "At the same time, you don't want to use an electric hot plate to cook a six-course gourmet meal."
Some vendors have developed niches that make their solutions ideal for certain types of companies, Regli says. For instance, Recommind is designed for law firms and specializes in searching legal documents; Endeca is very strong for businesses with catalogued merchandise.
According to Regli, many businesses are familiar with enterprise search, but willingness to jump in and start looking for a solution depends on size. Major Fortune 500 corporations, for the most part, have already embraced search--for some, in fact, it's become an integral part of doing business. Smaller businesses may know about enterprise search, but they find the cost and implementation process to be daunting, despite some relatively low-cost and easy-to-use solutions on the market for SMBs.
Once the system is selected, implemented, and up and running, though, it's usually easy for everyone in the enterprise to put it to use. Many enterprise search engines are based on the familiar, Google-style white text box to enter a search term, Brown says, so "if people can use the Google search box, they can certainly use enterprise search."
THE USES
At its heart, enterprise search is pretty straightforward: You need a document, and the search product's job is to help you find it. But enterprise search is more than just finding the single document that's right for a single situation: It's part of an overall strategy to make information more accessible and usable throughout the enterprise. Quebec City--based enterprise search vendor Coveo has several implementations where the flow of information generated by search helped the enterprise function better.
CA, one of Coveo's biggest customers, used the company's enterprise search solution to more quickly respond to customer questions and problems that come into the contact center. Haley & Aldrich, the engineering firm that designed Boston's Zakim Bridge, used Coveo's software to distribute information to the people working on the project. Another major implementation was with the U.S. Navy, which used Coveo's software to automate repair manuals for its fleet of aircraft. Manuals for aircraft are monstrously complex and can be an acute challenge to sort through when a plane comes in with a specific problem. While repairing military vehicles may not be the first use that comes to mind for search, the technology allows technicians to quickly find the information they need, all the way down to videos of how the job needs to be carried out.
Eric Negler, Coveo's executive vice president for business development, says that one customer described the benefits of enterprise search to him this way: "Good search is like electricity--our system doesn't run without it."
Enterprise search has a variety of practical applications beyond the finding and sharing of documents in the clutter of a computer system. Search has become an important part of e-discovery, the process by which documents are searched, indexed, and secured for the purpose of legal proceedings. If a company gets sued and needs critical documents, it is an extremely laborious process to have technology staff going from machine to machine digging them out. Search can turn up relevant documents with just a few keystrokes. Several analysts note that recent revisions to the Federal Rules of Civil Procedure mandate stricter record-keeping of electronic documents, driving more companies to start looking into enterprise search.
Other uses include online retailers that put enterprise search to work helping customers find products on their Web sites, and offering suggestions based on what other people have searched for--a feature that will be familiar to anyone who has shopped on Amazon.com or similar sites. (The popularity of those sites has shaped expectations among new enterprise search users in business: If it's not as accurate as Google or as user-friendly as Amazon.com, users may declare a search product to be a failure.) Media sites use similar technology to help readers find the stories and videos they're looking for. E-government initiatives also rely heavily on enterprise search to make documents available online, from city council agendas to federal reports. Enterprise search can also be used to manage document storage, searching for duplicate copies, or updating newly created documents.
Some companies are learning how to use search as a way to make business decisions. A Boston-based company called Compete has a product that allows businesses to keep an eye on competitors by tracking what customers are searching for and clicking on. (Compete's catchy slogan: "Track your rivals. Then eat their lunch.") Analyst Creese offers the example of a car company using the application to see what models are becoming more or less popular based on searches. That can help determine inventory. "It's not supremely accurate, but it's better than flying by the seat of your pants," he says.
THE MARKET
Analyst Matthew Brown hasn't done a formal survey of the market, but he estimates that the top seven enterprise search companies (including Autonomy, Endeca, and Fast Search & Transfer--which was snapped up by Microsoft in January) have a combined revenue of just shy of $1 billion, and have seen double-digit revenue-growth rates.
According to Brown, there are three tiers of enterprise search: the client market, midmarket, and entry level. The client market includes hardware-based search engines with a wealth of functions that can search billions of records, and is dominated by players like Autonomy and Fast. The average price of a deal to bring in a client-market server is between $300,000 and $400,000, and usually requires professional-services support to configure metadata and create custom interfaces.
Midmarket enterprise search offers less functionality--searching a smaller number of records for smaller businesses--and includes vendors familiar for their general software products: Microsoft SharePoint, SAP, and Google are among the players. A midmarket installation can cost between $30,000 and $40,000; extensive professional services may not be needed, but you'll require at least an IT administrator to make sure connections are secure.
The entry-level market includes engines with considerably less functionality, which can often be downloaded for free and installed by skilled analysts on their own, without major technical support, in a few hours. Google pioneered this market five years ago, and now offers two versions of enterprise search: combined hardware and software solutions (the Google Search Appliance, intended for larger companies and starting at $30,000; the Google Mini Search Appliance, for SMBs and starting at $3,000); and hosted software (Custom Search Business Edition) that allows Google technology to be added to the search function on a company's Web site.
Other companies have been putting pressure on Google with lower-cost or free products with somewhat less functionality than the upper tiers, including IBM's OmniFind Yahoo! Edition and Microsoft Search Server 2008 Express, both of which are downloadable for free. "The entry-level market has been growing very quickly for the last couple of years," Brown says, since Google's move into enterprise search in 2003. The arrival of major vendors seems to be a defensive response to Google's attempts to penetrate the software market, something Microsoft and IBM have an interest in preventing. "I don't think they care if they make any money on these products," Brown says.
THE FUTURE
A lot of large businesses bought into enterprise search over the last decade, when the technology was fairly immature. With midmarket and entry-level options growing more sophisticated, those businesses are now shopping around to see if they can get the same or better technology for less money. "That's a big problem for companies like Autonomy," Brown says. "It's potentially very disruptive."
The January purchase of Fast Search & Transfer by Microsoft for $1.2 billion not only shined a spotlight on the enterprise search market, but is likely to set off a round of consolidation in the high-end search market, Brown believes. Major vendors are more likely to see enterprise search as a viable business proposition and try to acquire the leading high-end search companies, which are typically small and ripe for takeover. Brown sees Endeca as a possible acquisition target, although industry leader Autonomy has enough diversity in its offerings that it should be able to remain on its own for the time being. As the enterprise search market consolidates and smaller companies are bought up by large vendors, "in the long run, search is going to go the way workflow management went: It started out as a standalone service and got absorbed," Creese says.
There are a few major issues enterprise search will be addressing as the technology matures. For instance, simply finding ways to refine results--making sure people get precisely the documents they're searching for--will become more important, as will increasing scalability (the number of documents that can be searched at once). The ability to search audio and video files is also emerging; some companies, including Coveo, already have solutions on the market.
Finally, one goal of enterprise search going forward will be the ability to generate answers to questions, rather than simply returning documents. When people search, they often are not hoping to find a certain document; they want to find specific information that the document contains. For these sorts of information-seeking queries, some responses to the search request, "When does my license with Microsoft expire?" are better than others: A result of "March 2010" is more useful than being directed to the license itself and having to read through it to find the expiration date.
Enterprise search typically arrives through one department that has a problem and thinks a search solution can help. After several departments each buy their own products, the topic tends to bubble up to the IT department, Creese says: Someone notices that the company's paying for six different search engines, and starts looking to consolidate.
"It's relatively rare for companies to worry about enterprise search as a strategic move," Creese adds. "It's still pretty technical, so I don't see a huge number of people diving in right now. But I do see a lot more enterprises using it in the future. Information overload is killing people."
SIDEBAR: Contact Center Search
A very different animal than its enterprise counterpart, contact center search provides its own set of benefits.
The general concept of enterprise search has gotten a lot of attention as of late, but one of its most specific applications has been acknowledged by experts as important for some: search within the
contact center. Essentially, contact centers are non-stop data-generating factories, with every call that comes in providing vital information about customer sentiments, problems, and trends--provided that all that data can be effectively managed.
One of the leading players in contact center search is Autonomy, whose Autonomy etalk unit employs enterprise search capabilities in ways specific to the contact center, such as understanding customer trends and behaviors by analyzing call data. Other vendors in enterprise search (such as Knova and Kana) and contact center infrastructure (such as Avaya) offer contact center--specific search products. CA used Coveo's enterprise search products specifically to improve the ability of contact center agents to answer customer queries. Astute Solutions recently released a new product called Agent Assist, which allows agents to enter a customer's question verbatim and get a reply that cites specific information within a document, then guides the agent through questions to ask the customer.
"The same things anyone wants to do in search are what you want to do in the contact center," says Donna Fluss, founder and president of DMG Consulting. The idea of contact center search "is not as new as people think it is. It's just a matter of how you think about it."
According to Ian Jacobs, a strategic analyst with Frost & Sullivan, the key component of successful contact center search is that when an answer isn't found immediately,
the search engine is able to prompt the agent with further questions to ask the customer in order to get better results. Also, contact center search functions typically have the ability to "learn" from the questions they are asked, creating files of answers to previously asked questions for future reference, including Web links or attachments to pass on to the customer, if needed.
"It's certainly not anything new. Contact centers do this sort of thing all the time," Jacobs says. "Although, in the past, a lot of the tools they used were homegrown." New innovations in contact center search include features that help agents pitch products to customers who call in with a problem--the vaunted goal of "turning the contact center into a profit center." Contact center search solutions can quickly provide agents with ideas for sales based on what the customer is seeking, such as extended warranties to other products that are appropriate to the caller.
Another recent innovation in contact center technology is the ability to link a company's Web site with its contact center. Online smart FAQs powered by search technology can guide customers through many problems, but if they can't come up with an answer, they can direct the customer to call an agent who has available all the information the caller entered online. The caller doesn't have to explain the problem all over again, and the agent can jump right in with an appropriate response. These types of services will drive more interest in enterprise search for the contact center, Jacobs believes. "I think you're going to see a greater push for CRM vendors to get into this," he says.
Contact the editors at editor@destinationCRM.com.