Elasticsearch Training in London

3 Elasticsearch Classes in London

 

es-training-240x187

Elasticsearch for Developers ……. April 4-5

Elasticsearch for Logging ……… April 6

Elasticsearch Operations …….  April 6

All classes cover Elasticsearch 2.x

Hands-on — lab exercises follow each class section

Early bird pricing until February 29

Add a second seat for 50% off

Register_Now_2

Course overviews are on our Elasticsearch Training page.

Want a training in your city or on-site?  Let us know!

Attendees in all three workshops will go through several sequences of short lectures followed by interactive, group, hands-on exercises. There will be Q&A sessions in each workshop after each such lecture-practicum block.

Got any questions or suggestions for the course? Just drop us a line or hit us @sematext!

Lastly, if you can’t make it…watch this space or follow @sematext — we’ll be adding more Elasticsearch training workshops in the US, Europe and possibly other locations in the coming months.  We are also known worldwide for Elasticsearch Consulting Services, and Elasticsearch Production Support.
We hope to see you in London in April!

Video and Slides: Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker

Running Elasticsearch clusters on Docker? Thinking about it?  If “yes” then we’ve got a presentation for you that digs deep into the details.

(Note: we’ve also got a related blog post about monitoring the official Elasticsearch image on Docker that you might find useful)

Coming to you from the recent DevOps Days event in Warsaw and delivered by Sematext engineer Rafal Kuć, “Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker” is chock full of practical information that will no doubt answer many of the questions you might have about this process.

Presentation Topics

Some of the topics Rafal covers include:

  • Containers vs. Virtual Machines
  • Running the official Elasticsearch container
  • Container constraints
  • Good network practices
  • Dealing with storage
  • Data-only Docker volumes
  • Scaling, time-based data
  • Multiple tiers and tenants
  • Indexing with and without routing
  • Querying with and without routing
  • Routing vs. no routing
  • Monitoring

Here’s a Taste of What You’ll See

How do Containers stack up versus Virtual Machines? There are a lot of elements at play…

DevOps_Warsaw_preso

Elasticsearch “One-stop Shop”

Sematext is your “one-stop shop” for all things Elasticsearch: Expert Consulting, Production Support, Elasticsearch Training, and Elasticsearch Monitoring with SPM.

Docker Monitoring

Speaking of monitoring…SPM does both Docker monitoring in a sweet little container and Elasticsearch monitoring (and provides alerting and anomaly detection, too), along with many other integrations that DevOps folks find useful.

Enjoy!

Using Filebeat to Send Elasticsearch Logs to Logsene

One of the nice things about our log management and analytics solution Logsene is that you can talk to it using various log shippers.  You can use Logstash, or you can use syslog protocol capable tools like rsyslog, or you can just push your logs using the Elasticsearch API just like you would to send data to a local Elasticsearch cluster. And like any good DevOps team, we like to play with all the tools ourselves.  So we thought the timing was right to make Logsene work as a final destination for data sent using Filebeat.

With that in mind, let’s see how to use Filebeat to send log files to Logsene.  In this post we’ll ship Elasticsearch logs, but Filebeat can tail and ship logs from any log file, of course.

Continue reading “Using Filebeat to Send Elasticsearch Logs to Logsene”

Using Grafana with Elasticsearch for Log Analytics

Grafana is an open-source alternative to Kibana.  Grafana is best known as a visualization / dashboarding tool focused on graphing metrics from various data sources, such as InfluxDB. Even though Grafana started its life as a Kibana fork, it didn’t originally support using Elasticsearch as a Data Source.  Starting with version 2.5 Grafana added support for Elasticsearch as a Data Source — good news that we at Sematext got very excited about. Elasticsearch is typically not used to store pure metrics.  It is used more often for storing time series data like logs and other types of events (think IoT).  Grafana 2.5 was limited to the display of numerical values, but as of version 2.6 Grafana supports tabular display of textual data as well. Of course, most logs include numerical data, too, which means we can now use Grafana to render both logs and metrics from those logs stored in Logsene – perfect!

The Logsene API is compatible with Elasticsearch, which means you can use Grafana (from v2.6 and up) with your data in Logsene simply by using Grafana’s Elasticsearch Data Source and pointing it to Logsene. You only need to do two things:

  1. Create a Data Source
  2. Add a Table Panel to a Dashboard

Watch this short video to see Grafana and Logsene together in action:

We hope you like this new, alternative way to derive insight from your data in Logsene.  Got ideas how we could make it more useful for you?  Let us know via comments, email or @sematext.

Not using Logsene yet? Check out the free 30-day trial by registering here (ping us if you’re a startup, a non-profit, or educational institution – we’ve got special pricing for you!).  There’s no commitment and no credit card required.  Even better — combine Logsene with SPM to make the integration of performance metrics, logs, events and anomalies more robust for those looking for a single pane of glass.

Presentation: Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker

Running Elasticsearch clusters on Docker? Thinking about it?  If “yes” then we’ve got a presentation for you that digs deep into the details.

(Note: we’ve also got a related blog post about monitoring the official Elasticsearch image on Docker that you might find useful)

Fresh from DevOps Days in Warsaw and delivered by Sematext engineer Rafal Kuć, “Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker” is chock full of practical information that will no doubt answer many of the questions you might have about this process.

Presentation Topics

Some of the topics Rafal covers include:

  • Containers vs. Virtual Machines
  • Running the official Elasticsearch container
  • Container constraints
  • Good network practices
  • Dealing with storage
  • Data-only Docker volumes
  • Scaling, time-based data
  • Multiple tiers and tenants
  • Indexing with and without routing
  • Querying with and without routing
  • Routing vs. no routing
  • Monitoring

 

Here’s a Taste of What You’ll See

How do Containers stack up versus Virtual Machines? There are a lot of elements at play…

DevOps_Warsaw_preso

Elasticsearch “One-stop Shop”

Sematext is your “one-stop shop” for all things Elasticsearch: Expert Consulting, Production Support, Elasticsearch Training, and Elasticsearch Monitoring with SPM.

Docker Monitoring

Speaking of monitoring…SPM does both Docker monitoring in a sweet little container and Elasticsearch monitoring (and provides alerting and anomaly detection, too), along with many other integrations that DevOps folks find useful.

Enjoy!

Introducing Top Database Operations

If you run Elasticsearch, Solr, or any backend you communicate with using SQL (via JDBC), like SparkSQL, Apache Cassandra (CQL), Apache Impala, Apache Drill, MySQL, PostgreSQL, etc., you’ll like what we’ve just added to SPM.  We call it Database Operations and in SPM you can find it in the new Database report:

If you didn’t watch the video, here’s what Database Operations gives you:

  • Top 5 operation types across all your data stores or filtered to a specific data store type
  • Top 5 operation types by speed, throughput, or simply their volume
  • Time-series reports for volume, throughput, and latency broken down by operation type
  • Ability to view all collected operations, not just the slowest ones, filter by database type or by operation type, sorted by average or total duration, or throughput
  • Sparklines that show last 5 minute values and trends
  • Top 10 slowest individual operations and drill-in details

Integration with Transaction Tracing, so you can correlate slow data store operations with the actual transaction/request that triggered slow operations

Important:

  • To get this information add SPM agent to the application that is talking to a data store (e.g. Solr or Elasticsearch or MySQL or …). This is because the SPM agent captures operations at that client layer, not in the server itself.
  • To start capturing this information enable Transaction Tracing in your SPM agents

This, including Distributed Transaction Tracing, works for all Java applications

Database_ops_1

——-

Database_ops_graphic

Don’t forget – when you enable Database Operations you will also automatically get Transaction Tracing, as well as the cool AppMaps – enjoy! :)

Got ideas how we could make Database Operations better and more useful to you?  Let us know via comments, email or @sematext.

Grab a free 30-day SPM trial by registering here (ping us if you’re a startup, a non-profit, or educational institution – we’ve got special pricing for you!).  There’s no commitment and no credit card required.

Docker + Elasticsearch: How to Monitor the Official Elasticsearch Image on Docker

Update: Elasticsearch 2.x requires a setup with standalone monitors. Why? A very restricitve Java security policy got implemented in Elasticsearch 2.x. This security policy  forbids loading of 3rd party libraries – including in-process monitoring libraries from SPM. Here is an example with Docker Compose and SPM-Client standalone monitor, working with Elasticsearch 2.x.

The official Elasticsearch Image on Docker Hub has already generated more than 1.6 million pulls. It is probably the easiest way to get a development setup — which includes Elasticsearch — to the application stack. The reason for this crazy number? A rapidly growing number of organizations are using Elasticsearch and Docker in production. Needless to say, monitoring Elasticsearch is essential in production, and you can find a detailed analysis of this topic (including the “top 10 Elasticsearch metrics to watch”) in the free eBook: ElasticsearchMonitoring Essentials. Docker is disruptive in many ways, and there are many things that are slightly different and worth mentioning.  These include:

  1. Changed deployment for Elasticsearch and its monitoring tools using Dockerfile, Docker Compose or various Orchestration Tools
  2. There is a new Layer to monitor: Container Metrics and Events, see: Docker Events and Metrics monitoring and SPM for Docker
  3. Logging has changed: containers log to the console and logs needs to be retrieved from Docker-Daemon instead getting them from the Elasticsearch log file.  Check out our post on the subject: Innovative Docker Log Management
  4. Official Images may not provide options for monitoring (such as JMX).  However, the official Image for Elasticsearch provides an option to pass parameters to the Java Runtime Environment.  We we will use this option for Elasticsearch monitoring in this post. You should also be aware that the official Elasticsearch Image does not include any plugins, and commercial monitoring from Elastic can’t be distributed in this Image for licensing reasons.  Our monitoring tool of choice is SPM.  If you are not familiar with SPM — but have heard of it — or if you use Marvel, have a look at Marvel vs. SPM.

Next, I’m going to demonstrate a setup to monitor multiple Elasticsearch nodes on a single Docker Host. The final setup will provide the full Monitoring and Logging package:

  • Detailed Application Metrics for Elasticsearch, deployed on Docker
  • Detailed Container Metrics and Docker Events  
  • Centralized Logs for all Containers by SPM for Docker

So let’s first decide on one of the following options to monitor Elasticsearch on Docker.  You can:

  1. Build your own Elasticsearch container with the included monitoring components. I’m not going to go into details about this option today; rather, I’m going to focus on the official / trusted build.
  2. Use a standalone agent, which queries metrics from the Elasticsearch container. This requires a setup for JMX and Docker networking configurations for the monitor and Elasticsearch. The metrics, gathered by remote agents, are limited and, in the Docker context, running an external monitoring process plus Elasticsearch processes consumes more resources.  And the next option …
  3. Inject an SPM in-process monitoring agent into Elasticsearch. This option has the lowest resource usage and has support for advanced monitoring functions like Transaction Tracing and AppMap.

I chose to implement Option #3 in this blog post because it provides the best insights into Elasticsearch. This means the Elasticsearch container needs file-system access to the SPM monitoring agent. Sematext provides the SPM Client (which includes the monitoring agent and metrics sender) pre-installed in a Docker Image, referred as “SPM Client Image/Container” in the following instructions and published on Docker Hub as “sematext/spm-client”.  The main trick here is to mount a volume from SPM-Client Container into Elasticsearch Containers in order to load the monitoring library.

Let’s have a look at the desired setup and how to get there:

Elasticsearch-Monitoring-On-Docker

Monitoring Setup for Elasticsearch on Docker

Continue reading “Docker + Elasticsearch: How to Monitor the Official Elasticsearch Image on Docker”