Video and Slides: Centralized Logging with Logstash and Elasticsearch

Sematext engineer and Elasticsearch / Logstash expert Rafal Kuc gave a well-received talk at the recent DevOps Days Warsaw event.  The talk was titled “From Zero to Hero – Centralized Logging with Logstash & Elasticsearch” and you can watch the video here:

 

And check out the slides here:

 

Brief Summary

Rafal talked about the common problem of digging through logs to find one particular event — or group of them.  And going even further into this pain point — what if you have lots of servers and you don’t have a single place to look for logs?  Do you really want to ssh to one or more servers and grep log files?  Of course not!  It’s 2014 and there are tools and services that help you spend less time hunting around for problems and more time actually fixing them.

To help solve this problem Rafal guided the audience through the basics of using Logstash and Elasticsearch together as the perfect combination for handling logs from multiple applications.  Attendees also learned how to set up Logstash, how to configure it to parse logs and, finally, how to send them to an Elasticsearch cluster.

Rafal also discussed tuning Elasticsearch for log management and centralized logging purposes, and showed how to easily switch between shipping logs to a self-hosted solution like Elasticsearch / Logstash / Kibana (aka ELK) and instead ship logs to Logsene Log Management and Analytics by changing a single line in Logstash configuration.

See also:

Enjoy!  And thanks to everyone who attended Rafal’s talk in person and stopped by the Sematext booth.

Apache Spark Monitoring in SPM

Apache Spark is an open-source, large-scale data processing engine built on top of the Hadoop Distributed File System (HDFS) and enables applications in Hadoop clusters to run up to 100x faster in memory, and 10x faster even when running on disk.  So it’s not surprising the usage of Spark is booming as this Google Trends graph shows.

And while Spark usage has been going through the roof, Engineers and DevOps handling Spark have not had a good monitoring tool at their disposal.  Well, that is, until now.  By releasing the first Spark monitoring product to market Sematext has, with the addition of Spark monitoring to SPM Performance Monitoring, Alerting and Anomaly Detection, just filled a big hole in the Spark ecosystem.

Having just been added — along with other goodies — to the latest SPM release, SPM for Spark monitors all Spark metrics.  It includes alerting, anomaly detection, log correlation, custom dashboards, events graphing, custom metrics, and a ton more.  SPM can be installed On Premises or one can use the Cloud version run by Sematext, in which case the setup takes less than 5 minutes before graphs with performance metrics start appearing in real-time.

Enough with the words – Show me what Spark Monitoring looks like!

Have a look at a few screenshots to see how we graph Spark metrics in SPM.  While we don’t use Spark at Sematext at this time and thus don’t have a live demo to show you, you can check out SPM’s live demo and see some other types of apps we monitor, such as Hadoop, HBase, Cassandra, Kafka, Storm, ZooKeeper, Elasticsearch, Solr, NGINX and NGINX Plus, Apache, MySQL, Redis, Java webapps and generic Java applications, as well as custom metrics.

Screenshot – Spark Executor metrics [click to enlarge]

Spark_screenshot_Executor_3

 

Screenshot – Spark Worker metrics  [click to enlarge]

Spark_screenshot_Worker_2

And One More Thing…

SPM now works hand-in-hand with Logsene Log Management and Analytics.  This makes the integration of performance metrics, logs, events and anomalies more robust for those of you looking to combine performance monitoring and centralized log management in one place — not only knowing that SOMETHING affected performance of your Spark cluster when you look at your performance metrics graphs or get an alert, but also exactly WHAT happened with the cluster by having immediate access to all relevant Spark event logs right there!

Take a Test Drive — It’s Easy and Free to Get Started

Like what you see here?  Sound like something that could benefit your organization?  Then try SPM and/or Logsene for Free for 30 days by registering here.  There’s no commitment and no credit card required.

Job: Sematext is hiring – Elasticsearch Engineer

The Sematext team is more distributed than your average Elasticsearch cluster and, trust me, we’ve seen a a good portion of the world’s Elasticsearch clusters.  The thing with Elasticsearch clusters is they often get new nodes added and they keep expanding to handle more data and more queries.  Similarly, we are looking to add a new node to the Sematext team so we can reshard our work a bit, distribute it more evenly, and scale further.  In plain English, we are looking for an Engineer who loves working with Elasticsearch, who loves large volumes of data, and a wide variety of projects and challenges involving large scale data processing, high volume indexing, high query rates, who likes working with our clients, and wants to make Logsene and SPM the killer log management and monitoring platforms.  Advanced knowledge of Elasticsearch is less important than passion to learn and build, positive attitude, ability to make decisions, work both independently and with the rest of the team, communicate well, and simply be a good person.  We can teach you everything about Elasticsearch and turn you into a bonsai tree loving Elasticsearch samurai, but we need you to be all these other things.

As a member of our team you will get to:

  • Work with world-class search experts
  • Design and implement systems (both our own and our clients’) that process 10s of thousands of queries per second and handle billions of documents, logs, data points, etc.
  • Interact with clients and customers world-wide
  • Provide guidance, architecture design, implementation, and production support around Elasticsearch
  • Participate in and contribute to open-source (we’ve contributed to Solr, Lucene, HBase, Flume, rsyslog, Logstash, etc.)
  • Share your knowledge with clients, at conferences and under-conferences, online community, etc.

This position:

  • Offers a lot of independence, learning, and growth
  • Is open to applicants “west of New York City” (this could be South, Central, or North America, of course), though we’ll happily make an exception if you persuade us we should make an exception for you!

Our search team members have written several books about search, regularly give talks at conferences, blog, and participate in open-source projects.  For more info, see 19 things you may like about Sematext.

Interested? Please send your resume to jobs@sematext.com.

For other job openings please see Jobs @ Sematext or even our previous job listings.

Correlating Metrics and Logs — Use Case: Elasticsearch Indexing

Here’s one way users can benefit from the SPM Performance Monitoring, Alerting and Anomaly Detection and Logsene Log Management and Analytics integration we just announced in the latest release.

Problem – CPU Utilization hits 95%!

  • You get an alarm about a CPU usage jump to 95% (note: using classic threshold-based alerts for CPU usage is a little crazy.  SPM’s anomaly detection feature would be a much better thing to use for CPU usage metrics).
  • You wonder, naturally, why this is happening and investigate immediately.
  • Without access to log graphs — like you would have with an SPM and Logsene combination — you would not be able to tell right away that the indexing rate increased.  It could be anything.  So you would need to connect, via ssh or VPN, to a server (or servers) where the CPU jumped and start looking around and see which process has been using the most CPU.  You’d run tools like top, vmstat, etc., but of course they’d have no historical data.
  • Even knowing which process uses the most CPU is not detailed enough.  You need to start looking at logs — either in another vendor’s log management tool which does not work seamlessly with your monitoring tool or manually “grepping” through one or more potentially very large log files on one or more servers — and try to determine what this application is doing more of now than it did before.  Not surprisingly, this is error-prone, time-consuming, and needlessly manual.  Most people have better things to do and want better tools.

Solution: Use SPM and Logsene Together to Triage

With a dashboard like the one you see here you can quickly tell what happened — i.e., why CPU usage went up.   In this particular case it is because the Elasticsearch indexing rate increased.  Now that the problem has been identified you can move on to taking action to fix it if a fix is needed.  Note:  You can even access the actual logs via Logsene so you can really be sure that there is no increase in some errors that are related to higher CPU usage.

test_dashboard_SPM_Logsene

We hope you found this use case helpful.  Got other performance monitoring, centralized log management or search-related use case ideas you’d like to see?  Drop us a line!

Sematext in GooglePlus

Quick shout-out to all G+ fans — you can find us in G+, too, and follow us there if you prefer that over the more traditional blog subscription: https://plus.google.com/+SematextGroup

Of course, @sematext is an option, too!

Announcement: New Functionality in SPM and Logsene

Summer is all but officially over, yet our work with SPM Performance Monitoring, Alerting and Anomaly Detection and Logsene Log Management and Analytics is not.  While lots of us took a well-deserved break over the last 1-2 months, we added a few goodies to both SPM and Logsene.  More interesting stuff is coming in the next release.

New in SPM

With SPM, the most notable addition is monitoring for Apache Spark.  We’ll have a separate post about Spark monitoring with SPM next week with all the details, including screenshots.  But that’s not the only new goodness; other additions include:

Integration with Nagios

  • You can now tell SPM where your Nagios lives and SPM will push all your Alerts to Nagios.  If you use PagerDuty, SPM can push your Alerts there, too.

Lowered SPM agent overhead

  • Those sending large volumes of metrics will see the most benefit.  The new agent makes use of Apache Flume to transport metrics.

Switched to sending metrics over HTTPS by default

These additions to SPM, along with recently announced monitoring support for NGINX Plus and NGINX make it an even more effective solution for organizations who are paying the unfortunate price of having a mish-mash of monitoring and alerting tools bolted together in an uneasy coexistence.

If you haven’t seen SPM yet, we have a live SPM demo so you can see it for yourself.  The demo shows Hadoop, HBase, Kafka, Elasticsearch, Solr, MySQL, Redis, and other types of apps being monitored.

New in Logsene

Until now you could create an unlimited number of Dashboards with SPM graphs, and now you can do that with Logsene graphs, too.  Moreover, you can place Logsene log graphs alongside SPM’s performance graphs, on the same Dashboard, and correlate your performance with your application logs!

This makes the integration of performance metrics, logs, events and anomalies more robust for those of you looking to combine performance monitoring and centralized log management in one place — not only knowing that SOMETHING happened when you look at your performance metrics graphs, but also exactly WHAT happened by having immediate access to relevant logs right there!

Screenshot – Dashboard with SPM Performance Graphs & Logsene Log Graphs  [click to enlarge]

test_dashboard_SPM_Logsene

Take a Test Drive — It’s Easy and Free to Get Started

Like what you see here?  Sound like something that could benefit your organization?  Then try SPM or Logsene for Free for 30 days by registering here.  There’s no commitment and no credit card required.

Talk: Using Logstash and Elasticsearch Together

Sematext engineer and Elasticsearch / Logstash expert Rafal Kuc is giving a talk about using Logstash and Elasticsearch together at DevOps Days Warsaw on September 26.  The talk is titled “From zero to hero: easy log centralization with Logstash and Elasticsearch” and it will be livestreamed and available afterward as a video. Details will be posted here and @sematext in late September.

Here is the common logging problem that Rafal is going to address: digging through logs to find one particular event — or group of them. And going even further into this pain point — what if you have lots of servers and you don’t have a single place to look for logs?  Let’s face it, you can end up spending hours digging through log files to find events you are interested in and manually correlating them with events from different sources.  If you have read this far into this post then chances are you have been there, done that.

And here is how Rafal’s talk could benefit people working with Logstash and Elasticsearch: he is going to guide attendees/viewers through the basics of using Logstash and Elasticsearch together as the perfect combination for handling logs from multiple applications.  So if you want to see how Logstash + Elasticsearch can help you, this is a great opportunity to do so.  Attendees/viewers will learn how to set up Logstash, how to configure it to parse logs and, finally, sending them to an Elasticsearch cluster.

Rafal will also discuss tuning Elasticsearch for log management and centralized logging purposes, and show how to easily switch between shipping logs to a self-hosted solution like Elasticsearch / Logstash / Kibana (ELK) and instead ship logs to Logsene Log Management and Analytics by changing a single line in Logstash configuration.

Last but not least — Rafal should be at the event both days (September 25 & 26).  So if you are going to attend in person and have some Logstash and/or Elasticsearch topics you’d like to discuss with an expert, then reach out to him on Twitter – @kucrafal

Logging Expertise If You Need It

Logging is in our DNA.  If you could use some help with log management and centralized logging, Sematext engineers provide logging expertise with Elasticsearch, Logstash, Syslog, Flume and other logging-related tools. Drop us a line if you could use a logging expert!

 

 

Follow

Get every new post delivered to your Inbox.

Join 1,683 other followers