Relevance Tuning and Competitive Advantage via Search Analytics

Here are two cool things about Search Analytics that I’d like to point out.  The slides are stolen from our Search Analytics presentation at Enterprise Search Summit 2011 in Washington DC.

Search Analytics for A/B testing, relevance tuning and improvements

This slide shows how Search Analytics can be used to help with A/B testing.  Concretely, in this slide we see two Solr Dismax handlers selected on the right side.  If you are not familiar with Solr, think of a Dismax handler as an API that search applications call to execute searches.  In this example, each Dismax handler is configured differently and thus each of them ranks search hits slightly differently.  On the graph we see the MRR (see Wikipedia page for Mean Reciprocal Rank details) for both Dismax handlers and we can see that the one corresponding to the blue line is performing much better.  That is, users are clicking on search hits closer to the top of the search results page, which is one of several signals of this Dismax handler providing better relevance ranking than the other one.  Once you have a system like this in place you can add more Dismax handlers and compare 2 or more of them at a time.  As the result, with the help of Search Analytics you get actual, real feedback about any changes you make to your search engine.  Without a tool like this, you cannot really tune your search engine’s relevance well and will  be doing it blindly.

A/B Testing with Search Analytics

Note: while in this slide we see two Solr Dismax handlers, Sematext Search Analytics is search vendor agnostic - the same thing can be done with search powered by FAST/Microsoft Search, Attivio, Endeca/Oracle, Autonomy/HP, Vivisimo, Dieselpoint, Coveo, ElasticSearch, vanilla Lucene, Xapian, or Sphinx, or …

Gaining Competitive Advantage with Search Analytics

As you can see, the only way to fix or improve things, and in this case we are talking about various aspects of search experience, is by having something with which you can measure this search experience and your changes.  You need something to tell you when it’s time to improve things, and you need something that gives you feedback about your changes: Did the key metrics improve after you changes?  If so, how much?  Did any metrics degrade? etc.

Search Analytics Key Takeways

I can’t emphasize enough how important Search Analytics is and how few organizations use it or use it well and consistently.  While this may be a bit mind boggling for those of us who live and breath search, from your perspective this is a great thing – it means that if you are smart about using Search Analytics to improve your search engine and your users’ search experience, you will gain competitive advantage and be ahead of your competitors who still don’t have or don’t use Search Analytics!

If you have any questions of feedback about Search Analytics in general, please leave a comment and we’ll follow up as soon as possible!

– @sematext