5-minute Logstash: Parsing and Sending a Log File

We like Logstash a lot at Sematext, because it’s a good (if not the) swiss-army knife for logs. Plus, it’s one of the easiest logging tools to get started with, which is exactly what this post is about. In less than 5 minutes, you’ll learn how to send logs from a file, parse them to extract metrics from those logs and send them to Logsene, our logging SaaS (basically, ELK Stack in the Cloud, though you can get an On Premises version, too, if you really want)

NOTE: Because Logsene exposes the Elasticsearch API, the same steps will work if you have a local Elasticsearch cluster.

NOTE: If this sort of stuff excites you, we are hiring world-wide for positions from devops and core product engineering to marketing and sales.

Overview

As an example, we’ll take an Apache log, written in its combined logging format. Your Logstash configuration would be made up of three parts:

  • a file input, that will follow the log
  • a grok filter, that would parse its contents to make a structured event
  • an elasticsearch output, that will send your logs to Logsene via HTTP, so you can use Kibana or its native UI to explore those logs. For example, with Kibana you can make a pie-chart of response codes:

pie_chart

The Input

The first part of your configuration file would be about your inputs. Inputs are modules of Logstash responsible for ingesting data. You can use the file input to tail your files. There are a lot of options around this input, and the full documentation can be found here. For now, let’s assume you want to send the existing contents of that file, in addition to the new content. To do that, you’d set the start_position to beginning. Here’s how the whole input configuration will look like:

input {
  file {
    path => "/var/log/apache.log"
    type => "apache-access"  # a type to identify those logs (will need this later)
    start_position => "beginning"
  }
}

The Filter

Filters are modules that can take your raw data and try to make sense of it. Logstash has lots of such plugins, and one of the most useful is grok. Grok makes it easy for you to parse logs with regular expressions, by assigning labels to commonly used patterns. One such label is called COMBINEDAPACHELOG, which is exactly what we need:

filter {
  if [type] == "apache-access" {   # this is where we use the type from the input section
    grok {
      match => [ "message", "%{COMBINEDAPACHELOG}" ]
    }
  }
}

If you need to use more complicated grok patterns, we suggest trying the grok debugger.

The Output

To send logs to Logsene (or your own Elasticsearch cluster) via HTTP, you can use the elasticsearch output. You’ll need to specify that you want the HTTP protocol, the host and port of an Elasticsearch server.

For Logsene, those would be logsene-receiver.sematext.com and port 80. Another Logsene-specific requirement is to specify the access token for your Logsene app as the Elasticsearch index. You can find that token in your Sematext account, under Services -> Logsene.

The complete output configuration would be:

output {
  elasticsearch {
    host => "logsene-receiver.sematext.com"
    port => 80
    index => "your Logsene app token goes here"
    protocol => "http"
    manage_template => false
  }
}

Wrapping Up

To start sending your logs, you’d have to download Logstash and put the three configuration snippets above in a file (let’s say, /etc/logstash/conf.d/logstash.conf). Then start Logstash. Once your logs are in, you can start exploring your data by using Kibana or the native Logsene UI.  Remember, Logsene is free to play with and it frees you up from having to manage your own Elasticsearch cluster.

13 Responses to 5-minute Logstash: Parsing and Sending a Log File

  1. Ivan says:

    Great writeup. The three logstash links in the overview section are incorrect because they are relative (lack the http protocol).

  2. Thanks for the writeup! It was super helpful.

  3. Pingback: Video and Slides: Centralized Logging with Logstash and Elasticsearch | Sematext Blog

  4. Pingback: Parsing and Centralizing Elasticsearch Logs with Logstash | Sematext Blog

  5. Pingback: Using Elasticsearch Mapping Types to Handle Different JSON Logs | Sematext Blog

  6. Ravi says:

    how to start logstash so that new .log files are updated to elasticsearch and then access through kibana?

    • Hi Ravi,

      The file input (as described above) supports wildcards, so you can say something like path => “/var/log/*” and when a new file comes up, Logstash will tail it.

      Then, if you want to upload to Logsene (which includes Kibana out of the box), the config is written above. If you want to upload to your own Elasticsearch cluster, just specify the host (and I’d also recommend setting ‘protocol’ to ‘http’ because it’s easier to upgrade). Once your logs are in, install Kibana and it should work out of the box, provided that you point it to the same Elasticsearch cluster.

      • Ravi says:

        Hi Radu,

        Thank you very much for your response. I am very new to elk stack and here although I made changes in logstash config file, I don’t see any changes at my kibana dashboard. It might be simple but I am asking you, how to trigger them.

      • Ravi says:

        when i run logstash using this command:
        logstash# /usr/java64/latest/bin/jaa -jar logstash-1.3.2-flatjar.jar agent -f logstash-1.3.2.conf &

        I see, port settings: my ip:9300
        prompting….

        my config file:
        file {
        type => “apps1″
        add_field => [ “componentName”, “ScheduleUI” ]
        path => “/tmp/test1.log”
        sincedb_path => “/root”
        }
        }
        filter {
        }
        output {
        elasticsearch {
        host => “myip”
        port => 9300
        cluster => “CLPcluster”
        }
        }

        ######
        I dont see any change in my kibana dashboard

  7. Pingback: Recipe: Reindexing Elasticsearch Documents with Logstash | Sematext Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 166 other followers

%d bloggers like this: