ElasticSearch Poll

We use ElasticSearch more and more here at Sematext (we have a number of ElasticSearch projects right at this moment, some of them quite massive in terms of data and/or query volume).  In our work we typically have only 1 ElasticSearch instance/only 1 JVM running ElasticSearch on each server in the cluster.

How about you?  Do you run multiple ElasticSearch instances/JVMs per box in production?

Thank you for your feedback!

6 Responses to ElasticSearch Poll

  1. Andrei Savu says:

    What’s the reasoning for running multiple instances? Is this is an effective way of avoid GC related issues on machines with a lot of memory? How big is the overhead from a processing perspective? Do you use dedicated disk / EBS drives for each instance to keep the IO performance predictable? How about network bandwidth – can multiple ES saturate that or this is not even a problem worth considering? Is the ideal number of instances related to the number of CPU cores? Is this really a good idea from a HA point of view? (you can end-up having all the replicas on a single machine) Is ES supporting replica placement policies (machine, rack awareness)?

    • sematext says:

      Man Andrei, you sure you asked enough questions there? ;)

      • Yes, one might consider running multiple instances on boxes with lots of RAM and large heaps for GC reasons, though with JVM and GC evolving, that may be less of an issue in the (near?) future.
      • Whether one would point different ES instances to different disks I think depends on whether and how much disks are used in the first place, but yes, when a disk starts becoming a problematic bottleneck, then one could utilize multiple disks.
      • Network bandwidth – I don’t know. I suspect this will be hard to saturate because other pieces of the system may become the bottleneck before network does.
      • I don’t think the number of ES instances is related to the number of CPU cores, no, because even with a single instance in a 24-core system (for example), all cores can be kept 100% utilized given high enough query rate.
      • Re replicas being places on the same physical box – just solved: https://github.com/elasticsearch/elasticsearch/issues/1680
      • Yes, ES does provide the knobs needed to control shard and replica allocation. We are making heavy use of that in a new ES cluster where the number of nodes has several trailing zeros…
      • Andrei Savu says:

        Thanks for answering them. I hope that other readers will find them useful. Also I find it quite amazing that an important feature just got added a few hours ago.

        • sematext says:

          Andrei: Indeed. However, another way to look at that important feature that got implemented just now is that it got implemented within 24 hours of us opening the issue. That’s hot. :)

  2. The single reason for running more than one instance on a single machine is a machine with a lot of memory (north of 100gb) and possibly wanting to run several instances with smaller heap sizes (28-30gb, so compressed oops are stil in effect).

    So, that feature that was added is not really that important (though I was happy to add it it was simple enough). I never heard of people needing to run more than 1 instance on a machine.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 1,713 other followers

%d bloggers like this: