Beyond POC: Processing Metrics, Logs and Traces … at Scale

For those of you attending next week’s DevOps Summit event in New York City (part of the larger Cloud Computing Expo) with an interest in topics like performance monitoring and processing metrics, log management, and distributed transaction tracing — at scale, no less! — then Sematext founder Otis Gospodnetić will be speaking your language on Wednesday, June 10.

Talk Summary

Application metrics, logs, and business KPIs are a goldmine. It’s easy to get started with the ELK stack (Elasticsearch, Logstash and Kibana) — you can see lots of people coming up with impressive dashboards, in less than a day, with no previous experience. Going from proof-of-concept to production tends to be a bit more difficult, unfortunately, and it tends to gobble up our attention, time, and money. In this talk Otis will share the architecture and decisions behind our services for handling large volumes of performance metrics, traces, logs, anomaly detection, alerts, etc. Attendees will follow data from its sources, its collection, aggregation, storage, and visualization. The talk will also cover the overview of some of the relevant technologies and their strengths and weaknesses, such as HBase, Elasticsearch, and Kafka.

  • Date: Wednesday, June 10
  • Time: 3:30 pm to 4:30 pm

Panel Discussion: Microservices and IoT Power

Otis will also be participating in a lunchtime panel discussion, also on June 10 (from 12:45 pm to 1:45 pm) with other tech industry experts called “Microservices and IoT Power” that dives deep into the important architectural principles behind implementing IoT solutions for the enterprise. Let’s face it, as remote IoT devices and sensors become increasingly intelligent, they become part of our distributed cloud environment, and we must architect and code accordingly.  It promises to contain buzzwords galore!

DevOps_Summit

Let’s Talk About Elasticsearch, ELK Stack, Solr, Spark, Kafka, APM, Centralized Log Management, and…

We’ll be at Booth #230 in the DevOps Summit section of the floor, so stop by and say hello.  We’ll be demo-ing SPM performance monitoring, Logsene Log Management and Analytics and Site Search Analytics, along with our usual interest in discussing Search and Big Data consulting topics and more.  Or just drop us an email or DM us if you’re not going to be in the Big Apple from June 9-11 but have interest in chatting.

Hope to see you in NYC next week!

Elasticsearch Training in Berlin – Wednesday, June 3

For those of you interested in some comprehensive Elasticsearch training taught by experts from Sematext who know it inside and out, we’re running an Elasticsearch Intro workshop in Berlin on Wednesday, June 3 (the day after Berlin Buzzwords ends). This full-day, hands-on training workshop will be taught by Sematext engineer — and author of several Elasticsearch booksRafal Kuc.  The workshop is open to anyone, not just folks who attended Berlin Buzzwords.

ES_intro_2

Here are the details:

  • Date:  Wednesday, June 3
  • Time:  9:00 a.m. to 5:00 p.m.
  • Location:  idealo internet GmbH, Ritterstraße 11, 10969 Berlin, Germany (less than 2 km from Buzzwords site)
  • Cost:  EUR 400 (early bird rate, valid through May 25) – EUR 500 afterward – 50% off 2nd seat!
  • Food/Drinks:  Light breakfast, lunch and post-workshop snacks & beverages

Register_Now_2 In this training workshop attendees will go through a series of short lectures followed by exercises and Q&A sessions covering the many aspects of Elasticsearch.  There will also be plenty of opportunities to get production tips & tricks that make things smoother. We are also considering an Elasticsearch Advanced class to be taught simultaneously at the same location.  If this is of interest to you and/or your colleagues, please drop us a line and it could happen! Lastly, if you can’t make it…watch this space.  We’ll be adding more Elasticsearch training workshops in the US, Europe and possibly other locations in the coming months.  We are also known worldwide for our Elasticsearch consulting services and production support if you need help asap. Hope to see you in Berlin!

Elasticsearch Training at GeeCON 2015

[Note: Early Bird pricing ends on Tuesday, May 5!]

For those of you interested in some comprehensive Elasticsearch training taught by experts (and authors of several Elasticsearch books!) who know it inside and out, you are in luck if you are attending — or considering — the GeeCON conference taking place in Krakow from May 13-15.

There will be two full-day training workshops held on May 12 — Elasticsearch Intro and Elasticsearch Advanced — run by Sematext engineers Radu Gheorghe and Rafał Kuć.

You can find the details for each session here, including costs and topics covered:

Elasticsearch Intro

ES_intro_2

Elasticsearch Advanced

ES_advanced_2

In both training workshops attendees will go through a series of short lectures followed by exercises and Q&A sessions covering the many aspects of Elasticsearch.  There will also be plenty of opportunities to get production tips & tricks that make things smoother.

If you can’t make it…watch this space.  We’ll be adding more Elasticsearch training workshops in the US, Europe and possibly other locations in the coming months.  We are also known worldwide for our Elasticsearch consulting services and production support if you need help asap.

Hope to see you in Krakow!

Poll Results: HBase Version Distribution

The results for HBase version distribution poll are in.  Thanks to everyone who took the time to vote!

The distribution pie chart is below, but we could summarize it as follows:

  • A big chunk of HBase clusters, about 30%, are still “stuck” on HBase 0.94.x
  • Over 37% of the HBase clusters are on 0.98.x that, until very recently, was the latest stable version
  • Only about 7% of clusters are on the 0.96.x and we can assume these clusters will soon migrate to either 0.98.x or 1.0.x
  • Somewhat surprisingly, almost 20% of HBase clusters are already on HBase 1.0.0 even though 1.0.0 was released only a few weeks ago

It’s great to see so many clusters moving to 1.0.0 so quickly! As for why there are still so many clusters using 0.94.x, which is several years old, see this comment on the HBase mailing list.  Here at Sematext we make heavy use of HBase and were on 0.94.x version for a long time, too.  A few months ago we’ve moved to 0.98.x and have been enjoying all its benefits.  Furthermore, we’ve recently updated SPM for HBase to monitor a pile of new HBase metrics that provide interesting new insights about our HBase clusters though some of the new metric charts.  For example, we are now able to see the dramatic impact of major compactions on data locality (and thus HBase performance!) — see for yourself – https://apps.sematext.com/spm-reports/s/VhOltU14Cy, or the number and size of HLog files over time — https://apps.sematext.com/spm-reports/s/7LU1qvs7ur.

HBase version distribution

Apache HBase Version Distribution

You may also want to check out the results of our other polls about big data technologies.

HBase Poll: Version You Run?

We are updating SPM for HBase to make sure SPM collects all the key HBase metrics that were added in 0.98, we thought it would be good to see which HBase versions are being used in the wild.  We’re on 0.98 after being on 0.94 for a long time.  How about you?

Please tweet this poll and help us spread the word, so we can get a good, statistically significant results.  We’ll publish the results here and via @sematext (follow us!) in a week.

Please tweet this poll and help us spread the word, so we can get a good, statistically significant results.  We’ll publish the results here and via @sematext (follow us!) in a week.

Poll Results: Kafka Version Distribution

The results for Apache Kafka version distribution poll are in.  Thanks to everyone who took the time to vote!

The distribution pie chart is below, but we could summarize it as follows:

  • Only about 5% of Kafka 0.7.x users didn’t indicate they will upgrade to 0.8.2.x in the next 2 months
  • Only about 14% of Kafka 0.8.1.x users didn’t indicate they will upgrade to 0.8.2.x in the next 2 months
  • Over 42% of Kafka users are already using 0.8.2.x!
  • Over 80% of Kafka users say they will be using 0.8.2.x within the next 2 months!

It’s great to see Kafka users being so quick to migrate to the latest version of Kafka!  We’re extra happy to see such quick 0.8.2 adoption because we put a lot of effort into improving Kafka metric, as well as making all 100+ Kafka metrics available via SPM Kafka 0.8.2 monitoring a few weeks ago, right after Kafka 0.8.2 was released.

Apache Kafka Version Distribution

Apache Kafka Version Distribution

 

You may also want to check out the results of our recent Kafka Producer/Consumer language poll.

 

Kafka Poll: Version You Use?

UPDATE: Poll Results!

With Kafka 0.8.2 and 0.8.2.1 being released and with the updated SPM for Kafka monitoring over 100 Kafka metrics, we thought it would be good to see which Kafka versions are being used in the wild.  Kafka 0.7.x was a strong and stable release used by many.  The 0.8.1.x release has been out since March 2014.  Kafka 0.8.2.x has been out for just a little while, but…. are there any people who are either already using it (we are!) or are about to upgrade to it? Please tweet this poll and help us spread the word, so we can get a good, statistically significant results.  We’ll publish the results here and via @sematext (follow us!) in a week.

Please tweet this poll and help us spread the word, so we can get a good, statistically significant results.  We’ll publish the results here and via @sematext (follow us!) in a week.

Follow

Get every new post delivered to your Inbox.

Join 169 other followers