Talk: Using Logstash and Elasticsearch Together

Sematext engineer and Elasticsearch / Logstash expert Rafal Kuc is giving a talk about using Logstash and Elasticsearch together at DevOps Days Warsaw on September 26.  The talk is titled “From zero to hero: easy log centralization with Logstash and Elasticsearch” and it will be livestreamed and available afterward as a video. Details will be posted here and @sematext in late September.

Here is the common logging problem that Rafal is going to address: digging through logs to find one particular event — or group of them. And going even further into this pain point — what if you have lots of servers and you don’t have a single place to look for logs?  Let’s face it, you can end up spending hours digging through log files to find events you are interested in and manually correlating them with events from different sources.  If you have read this far into this post then chances are you have been there, done that.

And here is how Rafal’s talk could benefit people working with Logstash and Elasticsearch: he is going to guide attendees/viewers through the basics of using Logstash and Elasticsearch together as the perfect combination for handling logs from multiple applications.  So if you want to see how Logstash + Elasticsearch can help you, this is a great opportunity to do so.  Attendees/viewers will learn how to set up Logstash, how to configure it to parse logs and, finally, sending them to an Elasticsearch cluster.

Rafal will also discuss tuning Elasticsearch for log management and centralized logging purposes, and show how to easily switch between shipping logs to a self-hosted solution like Elasticsearch / Logstash / Kibana (ELK) and instead ship logs to Logsene Log Management and Analytics by changing a single line in Logstash configuration.

Last but not least — Rafal should be at the event both days (September 25 & 26).  So if you are going to attend in person and have some Logstash and/or Elasticsearch topics you’d like to discuss with an expert, then reach out to him on Twitter – @kucrafal

Logging Expertise If You Need It

Logging is in our DNA.  If you could use some help with log management and centralized logging, Sematext engineers provide logging expertise with Elasticsearch, Logstash, Syslog, Flume and other logging-related tools. Drop us a line if you could use a logging expert!

 

 

Two Lucene/Solr Revolution 2014 Talks Accepted!

We recently got word from Lucene/Solr Revolution 2014 (in Washington, DC from Nov. 11-14) that talks submitted by two Sematext engineers were accepted as part of the Tutorial track!  They are:

In “Tuning Solr for Logs” Radu will discuss Solr settings, hardware options and optimizing the infrastructure pushing logs to Solr.

In “Solr Anti-Patterns” Rafal will point out common Solr mistakes and roads that should be avoided at all costs.  Each of the talk’s use cases will be illustrated with a before and after analysis — including changes in metrics.

You can see more details about both talks in this recent blog post.

The full agenda, including dates and times for the talks, will be available soon on the Lucene/Solr Revolution 2014 web site.

If you do attend one of these talks please stop by and say hello to Radu and Rafal.  Not only do they know Solr inside and out, but they are good guys as well!

Love Solr Enough to Even Want to Attend One of These Talks?

If you enjoy Solr enough to even think of attending these talks — and you’re looking for a new opportunity — then Sematext might be the place for you.  We’re hiring planet-wide and currently looking for Solr and Elasticsearch Engineers, Front end and JavaScript Developers, Developer Evangelists, Full-stack Engineers, and Mobile App Developers.

Announcement: NGINX and NGINX Plus Monitoring in SPM

The list of monitored stacks keeps growing!  SPM Performance Monitoring, Alerting and Anomaly Detection just added support for NGINX and NGINX Plus.  Now you can capture metrics like total requests and connections along with an overview report and a whole lot more.

NGINX is a high performance, open source web application accelerator that helps millions of the world’s busiest websites deliver more content, faster, to its users.  It is the #1 web server for the top 10k busiest websites in the world. NGINX Plus is the commercial version which adds advanced features, professional services, and shows more juicy metrics.

NGINX and NGINX Plus monitoring in SPM complements recently announced support for apps like MySQL, Cassandra, Memcached, Apache and AWS CloudWatch.  You can see a more complete list here (scroll down to “Monitored Apps”)

Here’s a glimpse into what SPM for NGINX and NGINX Plus provides – click on an image to see the full view or you can look at the actual SPM live demo showing SPM monitoring NGINX and NGINX Plus and their metrics.

Screenshot – NGINX Plus Overview  (click to enlarge)

NGINX+ overview_new

 

Screenshot – NGINX Plus Server Status  (click to enlarge)

NGINX+ server status

 

Screenshot – NGINX Plus Status Zones  (click to enlarge)

NGINX+ status zones 2

 

Screenshot – NGINX Plus Caches  (click to enlarge)

NGINX+ caches

 

Screenshot – NGINX Plus Upstreams  (click to enlarge)

NGINX+ upstreams

 

Screenshot – NGINX Overview  (click to enlarge)

NGINX overview

 

Screenshot – NGINX Status  (click to enlarge)

NGINX status

 

Live Demo — See SPM for Yourself

Check out SPM’s live demo to see NGINX and NGINX Plus monitoring for yourself.  You’ll also be able to poke around and see Kafka, HBase, Elasticsearch, Solr, MySQL, and other types of apps being monitored.

Love the Idea of Monitoring NGINX and NGINX Plus? Take a Test Drive — It’s Easy to Get Started.

Try SPM Performance Monitoring for Free for 30 days by registering here.  There’s no commitment and no credit card required.

JOB: Devops Evangelist – Monitoring, Logging, Analytics

DEVELOPER / DEVOPS EVANGELIST

Sematext is looking for someone with a business and marketing bent and enough technical background to be able to put together bits of code and demos of SPM, Logsene, and other products we are working on.  A good fit for this role is a person who likes to teach and share, knows how to connect with people and their needs, is passionate and is considered (or wants to become) a thought leader in at least one area — Monitoring, Logging, Data Analytics and/or Business Intelligence.  Our ideal evangelist also enjoys the agility and challenge of a startup.  For a good description of the type of person we are looking for watch this video.  Sematext is a young, fast growing, highly distributed and agile team and our developer evangelist will work in many different capacities and contribute to the company’s success in a variety of ways.

 

RESPONSIBILITIES:

  • Create technical content and demos for publication on our blog and other channels to show developers, devops, and others how to implement specific solutions or use new technologies
  • Prepare and deliver presentations and webinars, speak at industry conferences, local meetups and other events
  • Build relationships with tech bloggers, open source contributors and product community leaders, journalists and analysts
  • Educate and empower developers, giving technical workshops and brown bags
  • Build partnerships with individuals, companies and organizations that serve the same communities we do (Elasticsearch, Solr, Kafka, Hadoop, Storm, Spark, etc.)
  • Gather and socialize product feedback that informs engineering, sales, and marketing decision making

 

REQUIREMENTS:

  • BS or higher in Computer Science or professional experience as a developer, sys admin, sales engineer or other technical role
  • Strong verbal and written communications skills with ability to write for engineers or high-level management
  • Entrepreneurial thinking and the ability to act effectively with only high-level direction

 

BONUS:

  • Participation in open-source community
  • Experience with other commercial and open source monitoring, logging, or analytics technologies
  • Experience working in a startup

 

You can check out our products, our services, our clients, and our team to get a better sense of what Sematext is all about.  Also worth a look are the 19 things you may like about Sematext.  Interested? Please send your resume to jobs@sematext.com.  For other job openings please see Jobs @ Sematext or even our previous job listings.

 

JOB: Elasticsearch / Lucene Engineer (starts in the Netherlands)

In addition to looking for an Elasticsearch / Solr Engineer to join the Sematext team, we are also looking for an Lucene / Elasticsearch Engineer in EU for a specific project.  This project calls for 6 months of on-site work with our client in Netherlands.  After 6 months the collaboration with our client would continue remotely if there is more work to be done for the client or, if the client project(s) are over, this person would join our global team of Engineers and Search Consultants and work remotely (we are all very distributed over several countries and continents). This is a position focused on search – it involves working with Elasticsearch, but also requires enough understanding of Lucene to allow one to write custom Elasticsearch/Lucene components, such as tokenizers, for example. Here are some of the skills one should have for this job:

  •  knowledge of different types of Lucene queries/filters (boolean, spans, etc.) and their capabilities
  •  experience in extending out-of-the-box Lucene functionality via developing custom queries, scorers, collectors
  •  understanding of Lucene document analysis in the process of indexing, experience in writing custom analyzers
  •  experience in mapping advanced hierarchical data structures to Lucene fields
  •  experience in scalable distributed open-source search technologies such as Elasticsearch or Solr

The above is not much information to go by, but if this piqued your interest and if you think you are a good match, please fix up your resume and send it to jobs@sematext.com quickly.

Fall Internship at Sematext

This coming Fall 2014 we will have 2 open positions for enthusiastic students world-wide interested in spending a semester working with Sematext.  The positions will involve work on Performance Monitoring, Logging, Alerting, and Anomaly Detection, all of which are part of SPM and Logsene.  SPM and Logsene involve work with HBase and/or Elasticsearch, custom data processing components, metric aggregation, log aggregation, Kafka, Machine Learning and Data Mining algorithms, JavaScript, visualizations, reporting, and so on.

Sematext HQ is in Brooklyn, NY, USA, but we are a very geographically distributed organization whose members are spread over several countries and continents.  As such, we welcome students from all across the globe to do their internship with us from wherever they are.  Internship positions are available year-round, but are subject to student demand and our capacity.

You can check out our products, our services, our clients, our team, and if you’d like to be a part of it, please contact us and tell us about yourself.

JOB: Elasticsearch / Solr Engineer

We’ve grown nicely this year.  Our team has a new UI Developer, a new Solr/Elasticsearch Engineer, a new Marketing person, a new Automation Engineer, and this summer we have the first ever Intern.

Like all healthy organizations, we keep growing, and we are now looking for good Search Engineers who know Elasticsearch and/or Solr to join our geographically distributed search consulting team.  You will work remotely, from wherever you are, with smart people spread out across the planet and with an amazing array of companies world-wide on projects that range from just a week or two to several months.

At Sematext, we’ve built several exciting products – from smaller, search-focused products that work with Solr and Elasticsearch, to larger ones like SPMSearch Analytics, and most recently Logsene.  While not building products and running services, we help organizations world-wide with their search and big data needs – from fixing issues and providing production support to building complex search systems from scratch.  Our client list is long with a number of household names on it – from Instagram (Facebook) and Tumblr (Yahoo), Etsy and Shutterstock, to The BBC, Elsevier, Lockheed Martin, Reuters, Library of Congress, etc.  We did this without raising any money.  The demand for our products and services is growing and we are looking for good engineers and good people to join our adventure!

More formally:

Sematext is looking for a responsible, professional individual to join our team of search engineers.

Sematext is a New York-based startup with people spread over multiple continents and several hundred customers from Instagram and Tumblr, Etsy and Shutterstock, to The BBC, Elsevier, Lockheed Martin, Reuters, Library of Congress, etc. We’ve built systems handling over 10,000 QPS and have worked with multi-billion document indices. Our core products are:

In addition to the above products we offer consulting services around open source search and big data.

We are looking for a person who is:

  • Enthusiastic and positive
  • Driven, independent, and professional
  • A good communicator, both written and oral
  • Good with Solr and/or Elasticsearch and is hungry to learn more
  • Enjoys helping organizations make the best out of search

As a member of our search team you will get to:

  • Interact with clients world-wide
  • Provide guidance, architecture design, implementation, and support
  • Participate in Solr, Lucene, and Elasticsearch user and development communities
  • Work on Sematext’s search and data analytics products and participate in open-source search projects

This position:

  • Offers a lot of independence, learning, and growth
  • May require a bit of travel here and there, typically in the US and Europe
  • Is open world-wide

Our search team members have written several books about search, regularly give talks at conferences, blog, and participate in open-source projects.
For more info, see 19 things you may like about Sematext.

Interested? Please send your resume to jobs@sematext.com.

For other job openings please see Jobs @ Sematext or even our previous job listings.

Follow

Get every new post delivered to your Inbox.

Join 1,652 other followers