Talk: Using Logstash and Elasticsearch Together

Sematext engineer and Elasticsearch / Logstash expert Rafal Kuc is giving a talk about using Logstash and Elasticsearch together at DevOps Days Warsaw on September 26.  The talk is titled “From zero to hero: easy log centralization with Logstash and Elasticsearch” and it will be livestreamed and available afterward as a video. Details will be posted here and @sematext in late September.

Here is the common logging problem that Rafal is going to address: digging through logs to find one particular event — or group of them. And going even further into this pain point — what if you have lots of servers and you don’t have a single place to look for logs?  Let’s face it, you can end up spending hours digging through log files to find events you are interested in and manually correlating them with events from different sources.  If you have read this far into this post then chances are you have been there, done that.

And here is how Rafal’s talk could benefit people working with Logstash and Elasticsearch: he is going to guide attendees/viewers through the basics of using Logstash and Elasticsearch together as the perfect combination for handling logs from multiple applications.  So if you want to see how Logstash + Elasticsearch can help you, this is a great opportunity to do so.  Attendees/viewers will learn how to set up Logstash, how to configure it to parse logs and, finally, sending them to an Elasticsearch cluster.

Rafal will also discuss tuning Elasticsearch for log management and centralized logging purposes, and show how to easily switch between shipping logs to a self-hosted solution like Elasticsearch / Logstash / Kibana (ELK) and instead ship logs to Logsene Log Management and Analytics by changing a single line in Logstash configuration.

Last but not least — Rafal should be at the event both days (September 25 & 26).  So if you are going to attend in person and have some Logstash and/or Elasticsearch topics you’d like to discuss with an expert, then reach out to him on Twitter – @kucrafal

Logging Expertise If You Need It

Logging is in our DNA.  If you could use some help with log management and centralized logging, Sematext engineers provide logging expertise with Elasticsearch, Logstash, Syslog, Flume and other logging-related tools. Drop us a line if you could use a logging expert!

 

 

Two Lucene/Solr Revolution 2014 Talks Accepted!

We recently got word from Lucene/Solr Revolution 2014 (in Washington, DC from Nov. 11-14) that talks submitted by two Sematext engineers were accepted as part of the Tutorial track!  They are:

In “Tuning Solr for Logs” Radu will discuss Solr settings, hardware options and optimizing the infrastructure pushing logs to Solr.

In “Solr Anti-Patterns” Rafal will point out common Solr mistakes and roads that should be avoided at all costs.  Each of the talk’s use cases will be illustrated with a before and after analysis — including changes in metrics.

You can see more details about both talks in this recent blog post.

The full agenda, including dates and times for the talks, will be available soon on the Lucene/Solr Revolution 2014 web site.

If you do attend one of these talks please stop by and say hello to Radu and Rafal.  Not only do they know Solr inside and out, but they are good guys as well!

Love Solr Enough to Even Want to Attend One of These Talks?

If you enjoy Solr enough to even think of attending these talks — and you’re looking for a new opportunity — then Sematext might be the place for you.  We’re hiring planet-wide and currently looking for Solr and Elasticsearch Engineers, Front end and JavaScript Developers, Developer Evangelists, Full-stack Engineers, and Mobile App Developers.

Announcement: NGINX and NGINX Plus Monitoring in SPM

The list of monitored stacks keeps growing!  SPM Performance Monitoring, Alerting and Anomaly Detection just added support for NGINX and NGINX Plus.  Now you can capture metrics like total requests and connections along with an overview report and a whole lot more.

NGINX is a high performance, open source web application accelerator that helps millions of the world’s busiest websites deliver more content, faster, to its users.  It is the #1 web server for the top 10k busiest websites in the world. NGINX Plus is the commercial version which adds advanced features, professional services, and shows more juicy metrics.

NGINX and NGINX Plus monitoring in SPM complements recently announced support for apps like MySQL, Cassandra, Memcached, Apache and AWS CloudWatch.  You can see a more complete list here (scroll down to “Monitored Apps”)

Here’s a glimpse into what SPM for NGINX and NGINX Plus provides – click on an image to see the full view or you can look at the actual SPM live demo showing SPM monitoring NGINX and NGINX Plus and their metrics.

Screenshot – NGINX Plus Overview  (click to enlarge)

NGINX+ overview_new

 

Screenshot – NGINX Plus Server Status  (click to enlarge)

NGINX+ server status

 

Screenshot – NGINX Plus Status Zones  (click to enlarge)

NGINX+ status zones 2

 

Screenshot – NGINX Plus Caches  (click to enlarge)

NGINX+ caches

 

Screenshot – NGINX Plus Upstreams  (click to enlarge)

NGINX+ upstreams

 

Screenshot – NGINX Overview  (click to enlarge)

NGINX overview

 

Screenshot – NGINX Status  (click to enlarge)

NGINX status

 

Live Demo — See SPM for Yourself

Check out SPM’s live demo to see NGINX and NGINX Plus monitoring for yourself.  You’ll also be able to poke around and see Kafka, HBase, Elasticsearch, Solr, MySQL, and other types of apps being monitored.

Love the Idea of Monitoring NGINX and NGINX Plus? Take a Test Drive — It’s Easy to Get Started.

Try SPM Performance Monitoring for Free for 30 days by registering here.  There’s no commitment and no credit card required.

JOB: Elasticsearch / Lucene Engineer (starts in the Netherlands)

In addition to looking for an Elasticsearch / Solr Engineer to join the Sematext team, we are also looking for an Lucene / Elasticsearch Engineer in EU for a specific project.  This project calls for 6 months of on-site work with our client in Netherlands.  After 6 months the collaboration with our client would continue remotely if there is more work to be done for the client or, if the client project(s) are over, this person would join our global team of Engineers and Search Consultants and work remotely (we are all very distributed over several countries and continents). This is a position focused on search – it involves working with Elasticsearch, but also requires enough understanding of Lucene to allow one to write custom Elasticsearch/Lucene components, such as tokenizers, for example. Here are some of the skills one should have for this job:

  •  knowledge of different types of Lucene queries/filters (boolean, spans, etc.) and their capabilities
  •  experience in extending out-of-the-box Lucene functionality via developing custom queries, scorers, collectors
  •  understanding of Lucene document analysis in the process of indexing, experience in writing custom analyzers
  •  experience in mapping advanced hierarchical data structures to Lucene fields
  •  experience in scalable distributed open-source search technologies such as Elasticsearch or Solr

The above is not much information to go by, but if this piqued your interest and if you think you are a good match, please fix up your resume and send it to jobs@sematext.com quickly.

Fall Internship at Sematext

This coming Fall 2014 we will have 2 open positions for enthusiastic students world-wide interested in spending a semester working with Sematext.  The positions will involve work on Performance Monitoring, Logging, Alerting, and Anomaly Detection, all of which are part of SPM and Logsene.  SPM and Logsene involve work with HBase and/or Elasticsearch, custom data processing components, metric aggregation, log aggregation, Kafka, Machine Learning and Data Mining algorithms, JavaScript, visualizations, reporting, and so on.

Sematext HQ is in Brooklyn, NY, USA, but we are a very geographically distributed organization whose members are spread over several countries and continents.  As such, we welcome students from all across the globe to do their internship with us from wherever they are.  Internship positions are available year-round, but are subject to student demand and our capacity.

You can check out our products, our services, our clients, our team, and if you’d like to be a part of it, please contact us and tell us about yourself.

JOB: Elasticsearch / Solr Engineer

We’ve grown nicely this year.  Our team has a new UI Developer, a new Solr/Elasticsearch Engineer, a new Marketing person, a new Automation Engineer, and this summer we have the first ever Intern.

Like all healthy organizations, we keep growing, and we are now looking for good Search Engineers who know Elasticsearch and/or Solr to join our geographically distributed search consulting team.  You will work remotely, from wherever you are, with smart people spread out across the planet and with an amazing array of companies world-wide on projects that range from just a week or two to several months.

At Sematext, we’ve built several exciting products – from smaller, search-focused products that work with Solr and Elasticsearch, to larger ones like SPMSearch Analytics, and most recently Logsene.  While not building products and running services, we help organizations world-wide with their search and big data needs – from fixing issues and providing production support to building complex search systems from scratch.  Our client list is long with a number of household names on it – from Instagram (Facebook) and Tumblr (Yahoo), Etsy and Shutterstock, to The BBC, Elsevier, Lockheed Martin, Reuters, Library of Congress, etc.  We did this without raising any money.  The demand for our products and services is growing and we are looking for good engineers and good people to join our adventure!

More formally:

Sematext is looking for a responsible, professional individual to join our team of search engineers.

Sematext is a New York-based startup with people spread over multiple continents and several hundred customers from Instagram and Tumblr, Etsy and Shutterstock, to The BBC, Elsevier, Lockheed Martin, Reuters, Library of Congress, etc. We’ve built systems handling over 10,000 QPS and have worked with multi-billion document indices. Our core products are:

In addition to the above products we offer consulting services around open source search and big data.

We are looking for a person who is:

  • Enthusiastic and positive
  • Driven, independent, and professional
  • A good communicator, both written and oral
  • Good with Solr and/or Elasticsearch and is hungry to learn more
  • Enjoys helping organizations make the best out of search

As a member of our search team you will get to:

  • Interact with clients world-wide
  • Provide guidance, architecture design, implementation, and support
  • Participate in Solr, Lucene, and Elasticsearch user and development communities
  • Work on Sematext’s search and data analytics products and participate in open-source search projects

This position:

  • Offers a lot of independence, learning, and growth
  • May require a bit of travel here and there, typically in the US and Europe
  • Is open world-wide

Our search team members have written several books about search, regularly give talks at conferences, blog, and participate in open-source projects.
For more info, see 19 things you may like about Sematext.

Interested? Please send your resume to jobs@sematext.com.

For other job openings please see Jobs @ Sematext or even our previous job listings.

Join Sematext at AWS Summit in New York City on July 10

Live, work or occasionally travel to New York City?  Going to be in our sleepy little town next Thursday, July 10th?  If the answer is “yes” (coupled with the fact that you’re reading our blog) then you might be interested to know that Sematext will be exhibiting at the AWS Summit in New York City on Thursday, July 10.  We’ll be in booth 323 AND we’re holding an iPad giveaway!

We encourage you to stop by, say hello, and check out the new goodies in SPM and the new Logsene Log Management and Analytics UI and functionality.   Now that SPM and Logsene are seamlessly integrated, you can not only find out that SOMETHING happened with AWS (or just about any other app), but also exactly WHAT happened.

If you are going to AWS Summit and want to set up a personal demo on site, drop us a line at mick.emmett@sematext.com.  Hope to see you there!

Not Going to AWS Summit in NYC?  You Can Still Try SPM and Logsene for Free

Try Logsene and/or SPM Performance Monitoring for Free for 30 days by registering here.  There’s no commitment and no credit card required.  You can also check out a live demo and see Storm, Kafka, Solr, Elasticsearch, Hadoop, HBase, MySQL, and other types of apps being monitored.

We’re Hiring!

If you are planning to attend AWS Summit New York — and even if you’re not — and you enjoy performance monitoring, log analytics, or search analytics, working with projects like Elasticsearch, Solr, HBase, Hadoop, Kafka, and Storm, then stop by our booth and meet us in person (or drop us a line)!  We’re hiring planet-wide!  Front end and JavaScript Developers, Developer Evangelists, Full-stack Engineers, Mobile App Developers…get in touch!

Follow

Get every new post delivered to your Inbox.

Join 1,654 other followers