• We’ve added a Standalone Monitor.  So far the only way to monitor Solr, ElasticSearch, HBase, Sensei, or JVM with SPM was by running our SPM Monitor in-process, as a Javaagent.  Starting with this version you have an additional option of running the monitor in a separate process.
• SPM URLs are now sharable. Just copy the URL from your browser while using SPM and give it to anyone who has access to the same SPM App and they’ll see the exact same view as you – this means seeing the same report, same graph, same filter selection(s), and the same time range!  Because we use SPM with a lot of our Solr and ElasticSearch consulting clients, this is huge for us (and them!), as it helps us all see the exact same view.
• We have simplified the SPM client installation a lot and have simplified the Collectd config a bit, too.
• Both SPM Sender and SPM Monitor have been reworked. Monitor has the ability to register new applications and Sender has the ability to pick that up.  Sender should also be running with ionice if you have ionice available and a bit of unnecessary work was removed from Monitor, so it should consume even fewer resources than before.
• We’ve added more info about SPM to the new SPM Wiki Space.

In addition to that, we’re working on:

• Hadoop monitoring. This includes performance reports for both HDFS and MapReduce – NameNode, JobTracker, TaskTracker, and DataNode.
• SPM client packaging.  This means you’ll soon be able to install SPM client as a Deb package or RPM, and then automate with Puppet or Chef.
• …

There are a few more interesting things in the works, but we’ve got to leave something for later.  If you have not tried SPM yet, you should!  User feedback has been awesome and there are a number of good things on the 2013 roadmap!

In the mean time, here is a small sample of pretty self-explanatory reports from SPM for Hadoop, so you can get a sense of what’s available.  There are, of course, a number of other Hadoop-specific reports included, as well as server reports, filtering, alerting, multi-user support, report sharing, etc. etc.

Please don’t forget to tell us what else would you like us to monitor - select your candidates - and if you like what you see and want a good monitoring tool for your Hadoop cluster, please sign up for private beta now.

Click on any graph to see it in its full size and high quality.

Hadoop NameNode Files

.

Hadoop DataNode Read-Write

.

Hadoop JobTracker MapReduce Runtime

.

Hadoop TaskTracker Tasks

.

What else would you like us to monitor with SPM?  Please select your candidates!

For announcements, promotions, discounts, service status, milk, cookies, and other goodies follow @sematext.

Noisy Jack Nicholson

Unlike in real life, you can’t really hear your noisy neighours in virtualized worlds.  This is kind of good – if you don’t hear them, they won’t bother you, right? Wrong! Oh yes, they will bother you, it’s just that without proper tools you won’t really realize when they’ve become load, how loud they got, and how much their noise is hurting you! So while it’s true you can’t hear these neighbours, you can see them!  Have a look at this graph from SPM:

Noisy neighbour(s) stealing your CPU. Click for a larger and sharper image.

What we see here is a graph for CPU “steal time” for one of our HBase servers.  Luckily, this happens to be one of our HBase masters, which doesn’t do a ton of CPU intensive work.  What we see is that somebody, some other VM(s) sharing the same underlying host, is stealing about 30% of the CPU that really belongs to us.  Sounds bad, doesn’t it?  What exactly does that mean?  It means that about 30% of the time, applications on this instance (i.e., in our VM) try to use the CPU and the CPU is not available.  Bummer. Of course, this happens at a very, very low level, so from the outside, without this sort of insight, everything looks OK — it’s impossible to tell whether applications are not getting the CPU cycles when they need them by just looking at applications themselves.

So, do you know how noisy your virtual neighbours are?  Do you know how much they steal from you?

If you want to see what your neighbour situation is, whether on AWS or in some other virtualized environment, this is what you can do:

1. Get SPM (pick “Java” for your SPM Application type once you get in even if you don’t need to monitor any Java apps, yes)
2. Run the installer, but don’t bother with the “monitor” (aka SPM Monitor) piece – all you need to know are your CPU metrics and for that we don’t need the monitor piece to be running at all actually.
3. Go to http://apps.sematext.com/ and look at the “CPU & Mem” tab
4. Unselect all metrics other than “steal”, as show in the image above.  Select each server you want to check in the filter right of that graph (not shown in the image) to check one server at a time.
5. Make use of SPM alerts and set them up so you get notified when the CPU steal percentage goes over a certain threshold that you don’t want to tolerate. This way you’ll know when it’s time to consider moving to a new VM/instance.

What do you do if you find out you do have noisy neighbours?

There are a couple of options:

• Be patient and hope they go to sleep or move out
• Pack your belongings, launch a new EC2 instance, and move there after ensuring it doesn’t suffer from the same problem
• Create more noise than your neighbour and drive him/her out instead. Yes, I just made this up.

In this particular case, we’ll try the patient option first and move out only when the noise starts noticeably hurting us or we run out of patience.  Happy monitoring!

What portion of Solr 4.x users use SolrCloud?

Let’s find out!  Please tweet this to help us get more votes and better stats.

Please vote only if you are using Solr 4.x.  Please do NOT vote if you are using 1.x or 3.x version of Solr.

• Solr vs. ElasticSearch: Part 1 – Overview
• Solr vs. ElasticSearch: Part 2 – Data Handling
• Solr vs. ElasticSearch: Part 3 – Searching
• Solr vs. ElasticSearch: Part 4 – Faceting
• Solr vs. ElasticSearch: Part 5 – Management API Capabilities
• Solr vs. ElasticSearch: Part 6 – User & Dev Communities Compared

Users Community

Let’s start by discussing the user activity around both ElasticSearch and Apache Solr.

Users Activity

We started working on this post right before the Christmas break of 2012. During that time we decided to see how active the user base was for both ElasticSearch and Apache Solr. To do that we used our handy search-lucene.com service and we compared the number of email messages sent to both user list. So let’s see how they stack up.

Apache Solr

As you can see, Solr user activity varies slightly from month to month which is perfectly understandable. Each bar on the chart represents two weeks. We can see the number of messages ranges from about 390 mails to about 770 per two weeks, which gives us between 800 to 1600 mails per month is we do a bit of rounding up. Quite impressive I must say!

ElasticSearch

Now let’s discuss the ElasticSearch side. First a few words of explanation. If you look at the above chart you might think that ElasticSearch mailing list was silent and then users started posting on October 2012. That’s clearly not true – it is just that we didn’t add ElasticSearch to search-lucene.com until recently.  However, you may see that the number of messages during the same period of time is quite similar – both Solr and ElasticSearch saw about 670 – 730 messages during a two weeks period. This gives us 2 emails per hour on average.

Distinct Users

Email volume is one thing, but I was always curious about how many different people write emails on the mailing lists. Having such number would give us an additional understanding of the structure of the community around a particular search engine, new users, etc. However, we should not look only at this number, but also on things like most active people on the mailing lists. In both cases we’ve looked at the same period from 1 to 30 December 2012. We’ve used the data we index for search-lucene.com to calculate these numbers.

Apache Solr

In case of Apache Solr there were 234 unique users sending mail to the users mailing list. Almost 8 unique users per day on average, nice

ElasticSearch

In case of ElasticSearch there were 271 unique users sending mail to the users mailing list. This gives us about 9 unique users per day on average which is even nicer.

Resources Available

As far as resources available, both ElasticSearch and Solr have great documentation. On Solr wiki site (http://wiki.apache.org/solr/) you can find information about most of the components and of course the tutorial for beginners. ElasticSearch is very similar, with tutorial and very good description of functionality available at http://www.elasticsearch.org/. In addition to that, there are three books published about Apache Solr (in English) and more (e.g. my Apache Solr 4 Cookbook) coming soon. As of now, there are no published books about ElasticSearch, but…. stay tuned

Search Trending

We also decided to use uncle Google to look at trends about Apache Solr and ElasticSearch. Let’s look at the following diagram:

As you can see, until early 2010 there was no interest in ElasticSearch at all, at least looking from the point of view of users searching about it. Note that we published the interview with Shay Banon over two and a half years ago – back in May 2010 – before ElasticSearch registered on Google’s search trends radar! SolrCloud didn’t exist back then, so people slowly started looking for information on SolrCloud later in 2010.  The volume of searches mentioning SolrCloud is very small even today – perhaps because people tend to search for Solr and not SolrCloud.  And while SolrCloud is still a new kid around the block, searches for Solr dwarf searches for ElasticSearch despite the buzz surrounding ElasticSearch.

Of course, the above doesn’t say anything about the number of users of both search engines, but it definitely shows some information about the interest in these technologies.

Developers and the Code

If you are familiar with ElasticSearch and Solr you’ll probably know that ElasticSearch is much younger than Apache Solr. Apache Solr was created by Yonik Seeley in 2004 and donated to Apache Software Foundation. On the other hand, the first version of ElasticSearch was released by Shay Banon in 2010. This is quite important to say before we can talk about differences about contributors and the code itself. But getting to the point – we thought that it may be interesting to see both Apache Solr and ElasticSearch look from the Bird’s Eye perspective. To do that we’ve used the statistics and charts from ohloh.net. So, let’s see what they look like.

Apache Solr

Code Statistics

If we look at the current statistics, at the beginning of January 2013 Solr had more than 212k lines of code, with almost 7000 commits and 38 contributors. However, keep in mind that contributors are people that committed the code, not necessarily the ones that actually implemented it and provided the patch, so the actual number of contributors is much higher. The chart looks like this:

Top Contributors

If we look at top contributors we see Mark Miller on top, followed by Yonik Seeley and Robert Muir in the third place

Active Contributors

One more interesting thing is the number of contributors that were actively involved during a given period of time. Looking at Apache Solr since 2006 we can see the following: I think that we can say that we had a stable growth of active contributors starting from 2006 until June 2012 with a bit of downfall shortly after that. However I don’t think that the number active contributors will be dropping, it’s more likely due to a bit of exhaustion of releasing Apache Lucene and Solr 4.0

ElasticSearch

Code Statistics

Current code statistics for ElasticSeach shows that the code base just hit the 240k LOC  with about 4.2k commits and 87 contributors.

Top Contributors

As we’d expect, Shay Banon is the top contributor to ElasticSearch. In the second place on the podium we have Martijn van Groningen and Igor Motov in the third place:

Active Contributors

And finally the active contributors. We don’t have the same time frame comparing to Apache Solr, which is understandable as ElasticSearch is younger, but still we can see what is happening. As you can see from the first quarter of 2011 there was a number of active contributors varying from 5 to about 10 with the top at the same time as in Solr – 12 active contributors in June 2012.

Summary

As everything in this post indicates, both projects’ development and user communities are strong, active, and about equal. 2013 will be an interesting year for both projects.

We are nearing the end of our SolrCloud vs. ElasticSearch series.  What else would you like us to cover?  Please use the comments to let us know!

- @kucrafal, @sematext

• Solr vs. ElasticSearch: Part 1 – Overview
• Solr vs. ElasticSearch: Part 2 – Data Handling
• Solr vs. ElasticSearch: Part 3 – Searching
• Solr vs. ElasticSearch: Part 4 – Faceting
• Solr vs. ElasticSearch: Part 5 – Management API Capabilities
• Solr vs. ElasticSearch: Part 6 – User & Dev Communities Compared

Input/Output Format

ElasticSearch

As you probably know ElasticSearch offers a single way to talk to it – its HTTP REST API – JSON structured queries and responses. In most cases, especially during query time, it is very handy, because it let’s you perfectly control the structure of your queries and thus control the logic.

Apache Solr

On the other hand we have Apache Solr. If you are familiar with it you know that in order to send a query to Solr one needs to send it using URL request parameters.  This makes communication much less structured compared to ElasticSearch JSON format. In response you can get multiple response formats that are supported out of the box, like the default XML, JSON, CSV, PHP serialized, or Ruby.

Statistics API

Most of the time your search cluster will be fine and you won’t have any problems with it. However, there are times where you may need to see what is happening inside Apache Solr or ElasticSearch to diagnose problems, such as performance problems (hello SPM!), stability issues, or anything like that. In such cases, both search engines provide some amount of statistics.

Apache Solr

In Solr we can use JMX or HTTP requests to retrieve information about handler usage, cache statistics or information about most Solr components.

ElasticSearch

ElasticSearch was designed to be able to return various statistics about itself. With the REST API calls we can get information from the simplest ones like cluster health or nodes statistic, to extent information like the detailed ones about indices with merges, refreshes. The same stats are available via JMX, too.

Settings API

ElasticSearch

ElasticSearch allows us to modify most of the configuration values dynamically. For example, you can clear you caches (or just specific type of cache), you can move shards and replicas to specific nodes in your cluster. In addition to that you are also allowed to update mappings (to some extent), define warming queries (since version 0.20), etc. You can even shut down a single node or a whole cluster with the use of a single HTTP call. Of course, this is just an example and doesn’t cover all the possibilities exposed by ElasticSearch.

Apache Solr

In case of Apache Solr we do not (yet) have the possibility of changing configuration values (like warming queries) with API calls.

Index / Collection Administration Capabilities

In addition to the capabilities mentioned above both ElasticSearch and Apache Solr provide APIs that allows us to modify our deployment when it comes to collections and indices.

Apache Solr

Pre 4.0 we were able to manipulate cores inside our Solr instances. We could create new cores, reload them, get their status, rename, swap two of them, and finally remove a core from the instance. With Solr 4.0, a new API was introduced that is built on top of core admin API – the collections API. It allows us to create collections on started SolrCloud cluster, reload them and of course delete them. As the collections API is built on top of the core admin API,  if you create a new collection all the needed cores on all instances will be created. Of course, the same goes for reloading and deleting – all the cores will be appropriately informed and processed.

ElasticSearch

In case of ElasticSearch we can create and delete indices by running a simple HTTP command (GET or DELETE method) with the index name we are interested in. In addition to that, with a simple API call we can increase and decrease the number of replicas without the need of shutting down nodes or creating new nodes. With the newer ElasticSearch versions we can even manipulate shard placement with the cluster reroute API. With the use of that API we can move shards between nodes, we can cancel shard allocation process and we can also force shard allocation – everything on a live cluster.

Query Analysis

Apache Solr

If you’ve used Apache Solr you probably come across the debugQuery parameter and the explainOther parameter. Those two allows to see the detailed score calculation for the given query and documents found in the results (the debugQuery parameter) and the specified ones (the explainOther). In addition, we can also see how the analysis process is done with the use of analysis handler or by using the analysis page of the Solr administration panel provided with Solr.

For example this is how debug information returned by Solr can look like:

<?xml version="1.0" encoding="UTF-8"?>
<response>
.
.
.
<lst name="debug">
 <str name="rawquerystring">ten</str>
 <str name="querystring">ten</str>
 <str name="parsedquery">(+DisjunctionMaxQuery((prefixTok:ten)~0.01) ())/no_coord</str>
 <str name="parsedquery_toString">+(prefixTok:ten)~0.01 ()</str>
 <str name="QParser">DisMaxQParser</str>
 <null name="altquerystring"/>
 <null name="boostfuncs"/>
 <lst name="timing">
  <double name="time">2.0</double>
  <lst name="prepare">
   <double name="time">1.0</double>
   <lst name="org.apache.solr.handler.component.QueryComponent">
    <double name="time">1.0</double>
   </lst>
   <lst name="org.apache.solr.handler.component.FacetComponent">
    <double name="time">0.0</double>
   </lst>
   <lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
    <double name="time">0.0</double>
   </lst>
   <lst name="org.apache.solr.handler.component.HighlightComponent">
    <double name="time">0.0</double>
   </lst>
   <lst name="org.apache.solr.handler.component.StatsComponent">
    <double name="time">0.0</double>
   </lst>
   <lst name="org.apache.solr.handler.component.DebugComponent">
    <double name="time">0.0</double>
   </lst>
 </lst>
 <lst name="process">
  <double name="time">1.0</double>
  <lst name="org.apache.solr.handler.component.QueryComponent">
   <double name="time">0.0</double>
  </lst>
  <lst name="org.apache.solr.handler.component.FacetComponent">
   <double name="time">0.0</double>
  </lst>
  <lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
   <double name="time">0.0</double>
  </lst>
  <lst name="org.apache.solr.handler.component.HighlightComponent">
   <double name="time">0.0</double>
  </lst>
  <lst name="org.apache.solr.handler.component.StatsComponent">
   <double name="time">0.0</double>
  </lst>
  <lst name="org.apache.solr.handler.component.DebugComponent">
   <double name="time">1.0</double>
  </lst>
 </lst>
</lst>
<lst name="explain">
 <str name="Ten mices">
1.3527006 = (MATCH) sum of:
 1.3527006 = (MATCH) weight(prefixTok:ten in 35158) [DefaultSimilarity], result of:
 1.3527006 = fieldWeight in 35158, product of:
 1.4142135 = tf(freq=2.0), with freq of:
 2.0 = termFreq=2.0
 6.1216245 = idf(docFreq=6355, maxDocs=1065313)
 0.15625 = fieldNorm(doc=35158)
 </str>
</lst>
</lst>
</response>

As you can see, we can get information about timings of each of the used components. In addition to that, we see the parsed query and of course the explain information showing us how the document score was calculated.

ElasticSearch

ElasticSearch exposes three separate REST end-points to analyze our queries, documents and explain the documents score. The Analyze API allows us to test our analyzer on a specified text in order to see how it is processed and is similar to the analysis page functionality of Solr. The Explain API provides us with information about the score calculation for a given documents. Finally, the Validate API can validate our query to see is it is proper and how expensive it can be.

For example, this is what Explain API response looks like:

{
 "ok" : true,
 "_index" : "docs",
 "_type" : "doc",
 "_id" : "1",
 "matched" : true,
 "explanation" : {
   "value" : 0.15342641,
   "description" : "fieldWeight(_all:document in 0), product of:",
   "details" : [ {
     "value" : 1.0,
     "description" : "tf(termFreq(_all:document)=1)"
   }, {
     "value" : 0.30685282,
     "description" : "idf(docFreq=1, maxDocs=1)"
   }, {
     "value" : 0.5,
     "description" : "fieldNorm(field=_all, doc=0)"
   } ]
 }
}

You can see the description about score calculation that is returned from the Explain API.

Before We End

There are a few words more we wanted to write before summarizing this comparison. First of all the above mentioned APIs and possibilities are not all that it is available, especially when it comes to ElasticSearch. For example, with ElasticSearch you can clear caches on the index level, you can check index and types existence, you can retrieve and manage your warming queries, clear the transaction log by running the Flush API, or  even close an index or open those that were closed. We wanted to point some differences and similarities between Apache Solr and ElasticSearch, but we didn’t want to make a summary of the documentation. So, if you are interested in some functionality and you don’t know if it exists, just send a mail to Apache Solr or ElasticSearch mailing list or leave a comment here, and we will be glad to help.

Summary

When we first started the Solr vs ElasticSearch series we planned to initially divide the series into five posts, which are now published. However after seeing the popularity of the series and the amount of feedback we’ve received, we decided to extend the series. You can soon expect the next part, which will be dedicated to non-technical, but deeply important and interesting aspects of both search servers. After that, we’ll get back to the technical details with the subsequent post  dedicated to score influence capabilities, describing how we can change the default Lucene scoring and influence it from configuration, during indexing time and finally during querying.

If you liked this post, please tweet it!

- @kucrafal, @sematext

We recently published 0.1.0 version of the library to Sonatype public maven repository. Thus, integration in your project became much easier:

  <repositories>
    <repository>
      <id>sonatype release</id>
      <url>https://oss.sonatype.org/content/repositories/releases/</url>
    </repository>
  </repositories>

  <dependency>
    <groupId>com.sematext.hbasewd</groupId>
    <artifactId>hbasewd</artifactId>
    <version>0.1.0</version>
  </dependency>

  <repositories>
    <repository>
      <id>sonatype release</id>
      <url>https://oss.sonatype.org/content/repositories/releases/</url>
    </repository>
  </repositories>

  <dependency>
    <groupId>com.sematext.hbasehut</groupId>
    <artifactId>hbasehut</artifactId>
    <version>0.1.0</version>
  </dependency>

  <dependency>
    <groupId>com.sematext.hbasehut</groupId>
    <artifactId>hbasehut</artifactId>
    <version>0.1.0-hadoop-2.0</version>
  </dependency>

Discount Codes

Since holiday season is coming up, we thought we’d offer some discounts every week between now until the end of the year.  Each of the following discounts can be used only during “its week” specified below.  There is a limit to the number of people who can use each discount, so if you want it, don’t waste too much time.  Each discount will reduce the price of SPM SaaS for 365 days after you’ve used it, which effectively means you will get discount until the end of 2013.  Note that when you register for SPM you do not need to enter your credit card information.  You also don’t need to provide it when you create the SPM application for the system you want to monitor.  And it is when you create your SPM application that you can enter the discount code.

• 20% for the remainder of this week until the end of this Sunday, December 9: NY201320
• 15% for the week of December 10, 2012: NY201315
• 10% for the week of December 17, 2012: NY201310
• 5% for the week of December 24, 2012: NY201305

Note that each discount code expires on Sunday at 00:00 UTC.

SPM Flavours

The above discounts are good for our SPM SaaS.  However, if you’d rather run SPM on your own servers, we do offer SPM on Premises – please get in touch if you are interested in the on premises version.  You can also vote for SPM SaaS vs. On Premise and that way tell us which version you prefer or want.

SPM Plans

There are a few different subscription plans available in SPM SaaS:

• Basic plan that is free and shows the last 30 minutes of performance data
• Standard plan that shows the last 30 days of data and costs $0.035/server/hour
• Pro plan that shows the last 60 days of performance data and costs $0.070/server/hour

If you have not used SPM before, here is what you can expect to see – click on the image to see a large, non-fuzzy version:

We hope you will find SPM useful and fun to use.  We are always looking for feedback – just email spm-support@sematext.com or ping @sematext and tell us what you like or don’t like about SPM.