Apache Spark Monitoring in SPM

Apache Spark is an open-source, large-scale data processing engine built on top of the Hadoop Distributed File System (HDFS) and enables applications in Hadoop clusters to run up to 100x faster in memory, and 10x faster even when running on disk.  So it’s not surprising the usage of Spark is booming as this Google Trends graph shows.

And while Spark usage has been going through the roof, Engineers and DevOps handling Spark have not had a good monitoring tool at their disposal.  Well, that is, until now.  By releasing the first Spark monitoring product to market Sematext has, with the addition of Spark monitoring to SPM Performance Monitoring, Alerting and Anomaly Detection, just filled a big hole in the Spark ecosystem.

Having just been added — along with other goodies — to the latest SPM release, SPM for Spark monitors all Spark metrics.  It includes alerting, anomaly detection, log correlation, custom dashboards, events graphing, custom metrics, and a ton more.  SPM can be installed On Premises or one can use the Cloud version run by Sematext, in which case the setup takes less than 5 minutes before graphs with performance metrics start appearing in real-time.

Enough with the words – Show me what Spark Monitoring looks like!

Have a look at a few screenshots to see how we graph Spark metrics in SPM.  While we don’t use Spark at Sematext at this time and thus don’t have a live demo to show you, you can check out SPM’s live demo and see some other types of apps we monitor, such as Hadoop, HBase, Cassandra, Kafka, Storm, ZooKeeper, Elasticsearch, Solr, NGINX and NGINX Plus, Apache, MySQL, Redis, Java webapps and generic Java applications, as well as custom metrics.

Screenshot – Spark Executor metrics [click to enlarge]

Spark_screenshot_Executor_3

 

Screenshot – Spark Worker metrics  [click to enlarge]

Spark_screenshot_Worker_2

And One More Thing…

SPM now works hand-in-hand with Logsene Log Management and Analytics.  This makes the integration of performance metrics, logs, events and anomalies more robust for those of you looking to combine performance monitoring and centralized log management in one place — not only knowing that SOMETHING affected performance of your Spark cluster when you look at your performance metrics graphs or get an alert, but also exactly WHAT happened with the cluster by having immediate access to all relevant Spark event logs right there!

Take a Test Drive — It’s Easy and Free to Get Started

Like what you see here?  Sound like something that could benefit your organization?  Then try SPM and/or Logsene for Free for 30 days by registering here.  There’s no commitment and no credit card required.

Correlating Metrics and Logs — Use Case: Elasticsearch Indexing

Here’s one way users can benefit from the SPM Performance Monitoring, Alerting and Anomaly Detection and Logsene Log Management and Analytics integration we just announced in the latest release.

Problem – CPU Utilization hits 95%!

  • You get an alarm about a CPU usage jump to 95% (note: using classic threshold-based alerts for CPU usage is a little crazy.  SPM’s anomaly detection feature would be a much better thing to use for CPU usage metrics).
  • You wonder, naturally, why this is happening and investigate immediately.
  • Without access to log graphs — like you would have with an SPM and Logsene combination — you would not be able to tell right away that the indexing rate increased.  It could be anything.  So you would need to connect, via ssh or VPN, to a server (or servers) where the CPU jumped and start looking around and see which process has been using the most CPU.  You’d run tools like top, vmstat, etc., but of course they’d have no historical data.
  • Even knowing which process uses the most CPU is not detailed enough.  You need to start looking at logs — either in another vendor’s log management tool which does not work seamlessly with your monitoring tool or manually “grepping” through one or more potentially very large log files on one or more servers — and try to determine what this application is doing more of now than it did before.  Not surprisingly, this is error-prone, time-consuming, and needlessly manual.  Most people have better things to do and want better tools.

Solution: Use SPM and Logsene Together to Triage

With a dashboard like the one you see here you can quickly tell what happened — i.e., why CPU usage went up.   In this particular case it is because the Elasticsearch indexing rate increased.  Now that the problem has been identified you can move on to taking action to fix it if a fix is needed.  Note:  You can even access the actual logs via Logsene so you can really be sure that there is no increase in some errors that are related to higher CPU usage.

test_dashboard_SPM_Logsene

We hope you found this use case helpful.  Got other performance monitoring, centralized log management or search-related use case ideas you’d like to see?  Drop us a line!

JOB: Devops Evangelist – Monitoring, Logging, Analytics

DEVELOPER / DEVOPS EVANGELIST

Sematext is looking for someone with a business and marketing bent and enough technical background to be able to put together bits of code and demos of SPM, Logsene, and other products we are working on.  A good fit for this role is a person who likes to teach and share, knows how to connect with people and their needs, is passionate and is considered (or wants to become) a thought leader in at least one area — Monitoring, Logging, Data Analytics and/or Business Intelligence.  Our ideal evangelist also enjoys the agility and challenge of a startup.  For a good description of the type of person we are looking for watch this video.  Sematext is a young, fast growing, highly distributed and agile team and our developer evangelist will work in many different capacities and contribute to the company’s success in a variety of ways.

 

RESPONSIBILITIES:

  • Create technical content and demos for publication on our blog and other channels to show developers, devops, and others how to implement specific solutions or use new technologies
  • Prepare and deliver presentations and webinars, speak at industry conferences, local meetups and other events
  • Build relationships with tech bloggers, open source contributors and product community leaders, journalists and analysts
  • Educate and empower developers, giving technical workshops and brown bags
  • Build partnerships with individuals, companies and organizations that serve the same communities we do (Elasticsearch, Solr, Kafka, Hadoop, Storm, Spark, etc.)
  • Gather and socialize product feedback that informs engineering, sales, and marketing decision making

 

REQUIREMENTS:

  • BS or higher in Computer Science or professional experience as a developer, sys admin, sales engineer or other technical role
  • Strong verbal and written communications skills with ability to write for engineers or high-level management
  • Entrepreneurial thinking and the ability to act effectively with only high-level direction

 

BONUS:

  • Participation in open-source community
  • Experience with other commercial and open source monitoring, logging, or analytics technologies
  • Experience working in a startup

 

You can check out our products, our services, our clients, and our team to get a better sense of what Sematext is all about.  Also worth a look are the 19 things you may like about Sematext.  Interested? Please send your resume to jobs@sematext.com.  For other job openings please see Jobs @ Sematext or even our previous job listings.

 

Logsene Log Management and Analytics Grows Up!

There are exciting changes on the immediate horizon for Sematext’s comprehensive logging solution: Logsene Log Management and Analytics.  We just announced the seamless integration of Logsene with SPM Performance Monitoring, Alerting and Anomaly Detection (more on that below) and we have a new release of Logsene just weeks away from production.  Here’s a glimpse of the new Logsene UI.  Watch for more log analytics and management goodies on our blog soon.

New Logsene UI sneak peek

logsene1366x768_blog_post

Of course, the new Logsene will keep the option to use the Kibana UI directly from Logsene.

New Logsene Functionality in Words

  1. We support multiple queries in a single view; each query’s data get added to the graph (hence 2 lines in this screenshot).
  2. Queries can be saved, converted to scheduled queries (get email with log graphs every day/week/month type of functionality), or to alert queries (i.e., “email me when we see log messages with XYZ in them”). Those little gray icons next to search fields are for that.
  3. Clicking on the icon to the left of the top search field will show more functionality.
  4. Filtering by various log attributes.
  5. Raw log events in what we internally call Logs Table (or LT). Multiple tabs on top correspond to multiple queries one can enter.
  6. On the left is a list of available fields that are shown in LT. Checking them adds them to LT. They can be dragged and reordered. Reordering them controls the order of columns in LT.
  7. Number behind field names shows the number of distinct values for that field.
  8. Logs can be downloaded as CSV or published to Github Gist or Pastebin services.
  9. Arrows in top-right of LT expand the LT into a “Kiosk mode.”
  10. Little icon in each row in LT can be clicked. It expands on click and lets you see context around the given log.
  11. Clicking on the two gray buttons in top right with down arrows will expose metrics from SPM and Events.

If you currently use Splunk or another log management tool then there is a good chance that you will find Logsene to be an easy and useful way to index, search and analyze your logs.

But Why Use Logsene?

In short: Logsene and SPM provide a single pane of glass for performance monitoring, centralized log management, alerts, anomalies, custom events, and custom KPIs.  Unlike competing products, SPM and Logsene together not only tell users that SOMETHING happened, it now tells them exactly WHAT happened.  No need for a mish-mash of individual open-source or commercial tools for monitoring, for alerting, for logging, for custom dashboards, etc., each with its own configuration, its own UI, etc.

Not only that, but Logsene offers a unique pricing plan that, unlike those of other log management products, does NOT charge for the amount of data and retention.  You only pay for the number of logs in the index – you don’t pay for data transfer or data retention.

We also offer Plan Auto-Upgrade, a unique feature that can automatically switch you to the next higher plan if you exceed your log threshold — preventing you from losing logs which could contain critical information, without forcing you to keep this higher plan.

Moreover, because Logsene usage is “by day”, when auto-upgrade happens the extra cost is really minimal — at any time after the auto-upgrade you can adjust retention and/or plan and go back to the old plan

Check out a Live Demo

See SPM and Logsene (note: old UI and Kibana UI, the new UI will be in the next release) for yourself by viewing a live demo.  You’ll also be able to poke around and see Storm, Kafka, Solr, Elasticsearch, Hadoop, HBase, MySQL, and other types of apps being monitored.

Try Logsene and/or SPM Performance Monitoring for Free for 30 days by registering here.  There’s no commitment and no credit card required.

We’re Hiring!

If you enjoy performance monitoring, log analytics, or search analytics, working with projects like Elasticsearch, Solr, HBase, Hadoop, Kafka, and Storm, then drop us a line.  We’re hiring planet-wide!  Front end and JavaScript Developers, Developer Evangelists, Full-stack Engineers, Mobile App Developers…get in touch!

Monitoring Kafka, Storm and Cassandra Together with SPM

Kafka, Storm and Cassandra — Big Data’s Three Amigos?  Not quite.  And while much less humorous than the movie, this often-used-together trio of tools work closely together to make in-stream processing as smooth, immediate and efficient as possible.  Needless to say, it makes a lot of sense to monitor them together.  And since you’re reading the Sematext blog you shouldn’t be surprised to hear that we offer an all-in-one solution that can be used for Kafka, Storm and Cassandra performance monitoring, alerts and anomaly detectionSPM Performance Monitoring.  Moreover, if you ship logs from any of these three systems to Logsene, you can correlate not only metrics from these three systems, but also their logs and really any other type of event!

Enough with all the words — here is an illustration of how these three tools typically work together:

Kafka-Storm-Cassandra

Of course, you could also be storing data into some other type of data store, like Elasticsearch or HBase or MySQL or Solr or Redis.  SPM monitors all of them, too.

So what do you get if you monitor Kafka, Storm, and Cassandra with SPM?

You get a single pane pane of glass, a single access point through which you can see visualizations of well over 100 metrics you can slice and dice by a number applications-specific dimensions.  For example, various Kafka performance metrics can be filtered by one or more topics, while some Storm performance metrics can be filtered by topology or worker.  Of course, you can set alerts on any of the metrics and you can even make use of SPM’s anomaly detection capabilities.

Kafka, Storm + Cassandra screenshot

 

Consolidate Your App Monitoring, Alerting, and Centralize Logging — It’s Easy!

Many organizations tackle performance monitoring with a mish-mash of different monitoring and alerting tools cobbled together in an uneasy coexistence that is often far from seamless.  SPM takes all that hassle away and makes it easy and comprehensive in one step.

Live Demo — See SPM for Yourself

Check out SPM’s live demo to see this monitoring for yourself.  You won’t find any demo apps showing Cassandra metrics because we don’t use it at Sematext yet, but you’ll be able to poke around and see Kafka, HBase, Elasticsearch, MySQL, and other types of apps being monitored.

Love the Idea of Monitoring Kafka, Storm & Cassandra Together?
Take a Test Drive — It’s Easy to Get Started.

Try SPM Performance Monitoring for Free for 30 days by registering here.  There’s no commitment and no credit card required.

We’re Hiring!

If you enjoy performance monitoring, log analytics, or search analytics, working with projects like Elasticsearch, Solr, HBase, Hadoop, Kafka, and Storm, then drop us a line.  We’re hiring planet-wide!  Front end and JavaScript Developers, Developer Evangelists, Full-stack Engineers, Mobile App Developers…get in touch!

 

Announcement: What’s New in SPM Performance Monitoring

A new SPM Performance Monitoring release was just pushed to production and it’s chock full of great new stuff to complement its proactive performance monitoring, alerting, anomaly detection, etc., available in the Cloud or On Premise.  Here is a run-down of the juicier additions. The slightly longer version can be found in the SPM Changelog.

Integration with Logsene Log Management and Analytics

SPM performance monitoring now gives users access to even more metrics by seamlessly integrating with event data and logs via Logsene Log Management and Analytics.  This enables correlation across performance metrics, alerts, anomaliesevents, logs, and provides a single pane of glass across any organization’s IT infrastructure.

Monitoring Support for More Applications

We’ve added native monitoring support for the following applications to complement monitoring for Solr, Elasticsearch, Hadoop, HBase, Storm, Redis, Kafka, ZooKeeper and many others.

Screenshots

Eager to see pictures instead of reading content?  Then jump below to see screenshots of these apps being monitored.

UI/UX Improvements

UI/UX improvements include: zooming and panning, client-side caching, wider and simpler metric charts, new filter slide-out panels with search capabilities, quick access to all dashboards and easier dashboard creation, and more.

Event Graphs

Events and event graphs are now integrated into SPM Performance Monitoring.  You can now correlate various types of events, such as alerts, anomalies, application deployments, restarts, releases, server reboots, etc., with performance metrics graphs, as well as with logs and log graphs.  Many of you will also be happy to hear that SPM can now turn Alerts into Events, and graph them as well.  Check out Event Integration if you want to publish your own Events.

More Powerful Storm Monitoring

SPM Storm monitoring now serves up more metrics, more graphs and provides more granular details.  This includes the addition of metric filters and the ability to monitor not just Spouts and Bolts, but also monitor Storm Workers.

Dashboard Enhancements

Creating and working with dashboards just got a lot more intuitive and flexible.  This includes:

  • creating new dashboards via an intuitive “build your own dashboard” tool
  • easier navigation via Miller Columns (think column-oriented view in OSX Finder)
  • adding whatever graphs you want to an existing or brand new dashboard from within that dashboard
  • a pull down menu to select specific dashboards for much quicker access to a specific dashboard

 

Screenshot – SPM Dashboard (one of many possible views; click to enlarge)

Dashboard_test

 

Screenshot – Cassandra Overview  (click to enlarge)

cassandra_overview

 

Screenshot – MySQL Overview  (click to enlarge)

MySQL Overview

Screenshot – Memcached Overview  (click to enlarge)

memcached-overview

Screenshot – Apache Monitoring Overview  (click to enlarge)

Apache Overview

 

Screenshot – AWS CloudWatch EBS Read/Write Bandwidth  (click to enlarge)

AWS_EBS Read:Write Bandwidth

 

Live Demo

Check out SPM’s live demo to see it for yourself.  You won’t find any demo apps showing Cassandra or Memcached metrics because we don’t use them at Sematext yet, but you’ll be able to poke around and see other types of apps being monitored — like Solr, Kafka, Hadoop and HBase, for example — along with MySQL, AWS, and Apache.

Consolidate Your App Monitoring — It’s Easy!

Many organizations tackle performance monitoring with a mish-mash of different monitoring and alerting tools cobbled together in an uneasy coexistence that is often far from seamless. SPM takes all that hassle away and makes it easy and comprehensive in one step.

Try SPM for Free for 30 Days

Try SPM Performance Monitoring for Free for 30 days by registering here.  There’s no commitment and no credit card required.

We’re Hiring!

If you enjoy performance monitoring, log analytics, or search analytics, working with projects like Elasticsearch, Solr, HBase, Hadoop, Kafka, and Storm, then drop us a line.  We’re hiring planet-wide!  Front end and JavaScript Developers, Developer Evangelists, Full-stack Engineers, Mobile App Developers…get in touch!

Announcement: Cassandra Performance Monitoring in SPM

Cassandra is a distributed database management system built to handle massive data sets while providing high availability without compromising performance.  That’s why many organizations use it for their mission-critical data. That being said, if you are running Cassandra then you’ll want to keep close tabs on it.  And now SPM Performance Monitoring can help you do just that as the latest release supports Cassandra performance monitoring.  Not to mention that all the usual SPM alerting, anomaly detection, etc., can be used with any of the Cassandra metrics.

Why Not Monitor More Than Just Cassandra?

Unlike some of the tools that only monitor Cassandra and nothing else, SPM Performance Monitoring covers Cassandra, the competing database HBase, and a lot more: Solr, Elasticsearch, Hadoop, MySQL, AWS CloudWatch, Memcached, Apache, and just about any other app you want to monitor.  SPM also monitors Storm and Kafka, which are often used together with Cassandra (considered their most popular data store).  Now all three can be monitored together!

Have a look at a few of the screenshots to see how we graph Cassandra metrics in SPM (a list of Cassandra metrics we monitor is listed further below).  You can also check out SPM’s live demo. You won’t find any demo apps showing Cassandra metrics because we don’t use Cassandra at Sematext yet, but you’ll be able to poke around and see other types of apps being monitored, like Solr, Kafka, Hadoop and HBase, for example.

Overview  (click to enlarge)

cassandra_overview

 

Compactions  (click to enlarge)

compactions

 

Pending Writes  (click to enlarge)

pending-writes

 

Write Requests  (click to enlarge)

write-requests

 

SSTable  (click to enlarge)

sstable

 

Bloom Filter  (click to enlarge)

bloom-filter

SPM now monitors Cassandra metrics like:

  • Write Requests (rate, count, latency)
  • Read Requests (rate, count, latency)
  • Pending Write Operations (flushes, post flushes, write requests, replication of write)
  • Pending Read Operations (read requests, read repair tasks, compactions)
  • Pending Cluster Operations (manual repair tasks, gossip tasks, hinterd handoff, internal responses, migrations, misc tasks, request responses)
  • Compactions
  • Row/Key Cache (hit ratio, requests count)
  • Local Writes (rate, count, latency)
  • Local Reads (rate, count, latency)
  • SSTable (size, count)
  • Bloom Filter (space used, false positives ratio)

Please tell us what you think – @sematext is always listening!  Is there something SPM Performance Monitoring doesn’t monitor that you would really like to monitor?

Consolidate Your App Monitoring — It’s Easy!

Many organizations tackle performance monitoring with a mish-mash of different monitoring and alerting tools cobbled together in an uneasy coexistence that is often far from seamless.  Think Graphite+Nagios, for example.  SPM takes all that hassle away and makes it easy and comprehensive in one step.

Try SPM for Free for 30 Days

Try SPM Performance Monitoring for Free for 30 days by registering here.  There’s no commitment and no credit card required.

We’re Hiring

If you enjoy performance monitoring, log analytics, or search analytics, working with projects like Elasticsearch, Solr, HBase, Hadoop, Kafka, and Storm, then drop us a line.  We’re hiring planet-wide!  Front end and JavaScript Developers, Developer Evangelists, Full-stack Engineers, Mobile App Developers…get in touch!

Follow

Get every new post delivered to your Inbox.

Join 1,683 other followers