A more secure logstash install

We will installing logstash in a more secure fashion on CentOS 6 as Part 1 of our series on Monitoring your systems with logstash and Graylog2. ::Part 2::Part 3::Part 4::, but our the methodology is applicable to other distributions as well. We’ll be using Tanuki Software’s open source version of their “Java Service Wrapper” to facilitate running the program as a system service.

Our goals are simple:

  • Run as an unprivileged user
  • Consistent installation, logging, and configuration locations
  • Ability to scale to securely monitor other remote systems
  • Automating the installation
IMPORTANT NOTE BEFORE WE PROCEED!
It is a very bad idea to expose your system logs to anyone who can attach to your computer with a browser.
PLEASE MAKE SURE YOUR FIREWALL IS WORKING
and dropping incoming connections. YOU’VE BEEN WARNED

The installation overview:

  1. Download the logstash package
  2. Create the installation environment
  3. Configure the logstash options
  4. Modify Access to our log files
  5. Test the logstash Install
  6. Download the Java Service Wrapper
  7. Configure the Java Service Wrapper
  8. Test running Logstash via the JSW
  9. Verify everything works
  10. Tie up loose ends

We will assume you know the basics of system administration (ie.file permissions, ownership, etc.).
So let’s get this working!

Our first step will be to download logstash. You’ll likely want to download the monolithic package. While you’re at the logstash site, take a look at the available documentation.

Our next step is to create the installation environment. I’ve created a shell script to create the directories we’ll be needing, as well as to create our unprivileged user. I’ll wait while you download and look over the script. You’ll need to run the script as root, but can test it as a normal user quite easily (with some minor modifications to the script).

The command to run the script is:

"./prog_dir_setup logstash"

Our script will create the following directories:

~ create “/usr/local/bin/logstash” to hold our logstash executables
       then create “bin” and “lib” directories underneath
~ create “/var/spool/logstash” for our logs, pid, and elasticsearch database
       then create “log”, “pid” and “data” directories underneath
~ create “/etc/logstash” for our logstash config files

Our script will then create a system user named “logstash”, and
modify directory permissions so “logstash” can write to those locations.

~ Create user “logstash” and group “logstash” as a system account
       pointing $HOME to “/var/spool/logstash”
~ Change ownership of directories under “/var/spool/logstash” to “logstash.root”

We should next unpack the logstash archive to “/usr/local/bin/logstash”, creating a “jar” archive like “logstash-X.X.XX-monolithic.jar”. We’ll then create a symbolic link to the file via root shell, which will enable us to update the installation very easily (Note: change “X.X.XX” to reflect your logstash version).

To create logstash.jar symbolic link, as root:

"cd /usr/local/bin/logstash && ln -s ./logstash-X.X.XX-monolithic.jar ./logstash.jar"

We will next create a simple configuration file for our test. Using your favorite editor, create a “mylogstash.conf” in “/etc/logstash”. My example is below and you can also download it via this link.

input {
   file {
      type => "linux-syslog"
      path => [ "/var/log/messages" ]
   }
   file {
      type => "apache-access"
      path => "/var/log/httpd/access_log"
   }
   file {
      type => "apache-error"
      path => "/var/log/httpd/error_log"
   }
}
output {
   stdout {
   }
   elasticsearch {
      embedded => true
   }
#   gelf {
#      chunksize => 1420
#      facility => "logstash-gelf"
#      host => "127.0.0.1"
#      level => "INFO"
#      port => 12201
#      sender => "%{@source_host}"
#   }
}

You’ll note the section titled “GELF” is commented out. We’ll be uncommenting that section in a later tutorial to connect to Graylog. So if you’re creating your own, you can discard the section.

Now we’ll need to modify some permissions to allow our “logstash” group to be able to read “/var/spool/messages” and our apache logs. We’ll accomplish this task by modifying ACLs on the necessary files and directories, using the “setfacl” command as root.

setfacl -m g:logstash:x /var/log/httpd
setfacl -m g:logstash:r /var/log/messages

We will now test our installation to verify that it functions as we expect before we configure the Java Service Wrapper.
As “root” you should “su” to become user “logstash” and test that our inputs are readable and if successful we will attempt to launch logstash.

su logstash
tail /var/log/httpd/access_log
tail /var/log/httpd/error_log
tail /var/log/messages

If no errors were reported we were successful, and can now launch logstash as user “logstash” with the following command (spread over multiple lines):

/usr/bin/java -Des.path.data="/var/spool/logstash/data/" \
-jar "/usr/local/bin/logstash/logstash.jar" agent -vvv \
-f "/etc/logstash/mylogstash.conf" \
-l "/var/spool/logstash/log/logstash.log" \
-- web --backend elasticsearch://127.0.0.1/?local

This command tells logstash to:

  • Use “/var/spool/logstash/data” to store our elasticsearch database
  • Use super verbose logging “-vvv”
  • Use “/etc/logstash/mylogstash.conf” as our config file
  • Log to “/var/spool/logstash/log/logstash.log”
  • Start the web interface to elasticsearch on the local host

Give logstash a couple minutes to get going, and see if we can view some data.
Point your browser at http://127.0.0.0:9292
Once the interface comes up, restart apache and/or plug in a USB device to generate some events. Then put an asterik in the “Query” box and click the “Search” button and you should see some logged events.

If everything seems to be working as we hoped, we now configure logstash to run under the “Java Service Wrapper” which will allow us to run logstash as a service, launching as root and forking to run as user “logstash”.

We should now download the Community version of the Java Service Wrapper and unpack it to ~/wrapper (ie. “/home/username/wrapper”), and we will then copy some files from the ~/wrapper directory for our logstash config.

~ Copy ‘wrapper.conf” to /etc/logstash
 "cp ~/wrapper/src/conf/wrapper.conf.in /etc/logstash/wrapper.conf"
~ Copy Shell File and rename it to something meaningful
 "cp ~/wrapper/src/bin/sh.script.in /usr/local/bin/logstash/bin/logstash0_wrapper"
~ Copy “wrapper” executible
 "cp ~/wrapper/bin/wrapper /usr/local/bin/logstash/bin/"
~ Copy wrapper/lib/libwrapper.so
 "cp ~/wrapper/lib/libwrapper.so /usr/local/bin/logstash/lib/"
~ Copy wrapper/lib/wrapper.jar
 "cp ~/wrapper/lib/wrapper.jar /usr/local/bin/logstash/lib/"

Edit the below lines in /usr/local/bin/logstash/bin/logstash0_wrapper with your favorite editor.

 APP_NAME="logstash0"
 APP_LONG_NAME="Logstash Local Collector"
 WRAPPER_CONF="/etc/logstash/wrapper.conf"
 PIDDIR="/var/spool/logstash/pid"
 RUN_AS_USER="logstash"

Edit the below lines in /etc/logstash/wrapper.conf with your favorite editor (Note: I’ve hard coded some perfectly valid variables contained in Tanuki’s provided script).

wrapper.java.command=/usr/bin/java

wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperJarApp

wrapper.java.classpath.1=/usr/local/bin/logstash/lib/wrapper.jar

wrapper.java.classpath.2=/usr/local/bin/logstash/logstash.jar

wrapper.java.library.path.1=/usr/local/bin/logstash/lib

wrapper.java.additional.1=-Des.path.data=/var/spool/logstash/data/

wrapper.app.parameter.1=/usr/local/bin/logstash/logstash.jar
wrapper.app.parameter.2=agent
wrapper.app.parameter.3=-vvv
wrapper.app.parameter.4=-f
wrapper.app.parameter.5=/etc/logstash/mylogstash.conf
wrapper.app.parameter.6=-l
wrapper.app.parameter.7=/var/spool/logstash/log/logstash.log
wrapper.app.parameter.8=--
wrapper.app.parameter.9=web
wrapper.app.parameter.10=--backend
wrapper.app.parameter.11=elasticsearch://127.0.0.1/?local

wrapper.logfile=/var/spool/logstash/log/wrapper.log

You should now be able to launch logstash as root and have it run as logstash.
In a terminal window as root:

"/usr/local/bin/logstash/bin/logstash0_wrapper console"

You can again check everything is running as expected at:
http://127.0.0.1:9292

We will now edit the logstash account to remove it’s bash shell
you’ll want to do this after logstash is working as expected.

usermod -s /bin/false logstash

If you run the wrapper script without any parameters, you’ll see it has a number of useful options.

[root@mylaptop sphughes]# /usr/local/bin/logstash/bin/logstash0_wrapper

Usage: /usr/local/bin/logstash/bin/logstash0_wrapper [ console | start | stop | restart | condrestart | status | install | remove | dump ]

Commands:

console     Launch in the current console.
start       Start in the background as a daemon process.
stop        Stop if running as a daemon or in another console.
restart     Stop if running and then start.
condrestart Restart only if already running.
status      Query the current status.
install     Install to start automatically when system boots.
remove      Uninstall.
dump        Request a Java thread dump if running.

That’s a wrap. We can now resume normal logging instead of verbose. We’ll be expanding on our logstash usage in future articles. For the lastest news and information, I suggest you subscribe to the logstash users google group.

12 thoughts on “A more secure logstash install

  1. Great guide, I’ve put it to use on two Logstash machines I’ve built.

    My only comment: Setting the unprivileged logstash user’s shell to /bin/false seems to make the service wrapper unable to execute logstash!

    Perhaps I’ve overlooked something, but making the usermod -s change rendered my service wrapper inoperable.

    Otherwise, great information.

    • Thanks for the head’s up Wes! I’ve modified the post and will see if I can find the work around.

  2. Thanks for the Guide. This answered my question on how to set Logstash up as a service.

    I found one issue with your script. I am using CentOS 5 and my useradd does not have -U. I had to update the script to do a groupadd and modifed useradd to use -g to add the group.

    • Glad to hear the instructions were helpful and thanks for the feedback.

  3. Got up to a successful manual launch with this – but couldn’t get it to work with wrapper. Logs say:

    WrapperSimpleApp Error: Unable to locate the class : java.lang.ClassNotFoundException

    From what I can see in WrapperSimpleApp source the empty space between “Unable to locate the class” and “:” should be filled with args[0] … but I can’t tell what is failing to provide an argument there …

  4. when I run setfacl -m g:logstash:x /var/log/httpd I get an error on centos 6:
    setfacl: Option -m: Invalid argument near character 3

    whats going on ?

  5. Hi you will also need to set the HOME variable for logstash to work properly, particularly with the input file thread.

    You can do this in your wrapper conf with something like:
    set.HOME=/opt/logstash

    or wherever your install path for your logstash is …

  6. I’ve recently tried following your manual, but after I try running /usr/local/bin/logstash/bin/logstash0_wrapper console

    I’m getting this error:

    Running Logstash local Collector…
    wrapper | –> Wrapper Started as Console
    wrapper | Java Service Wrapper Community Edition 64-bit 3.5.15
    wrapper | Copyright (C) 1999-2012 Tanuki Software, Ltd. All Rights Reserved.
    wrapper | http://wrapper.tanukisoftware.com
    wrapper |
    wrapper | Launching a JVM…
    jvm 1 | WrapperManager: Initializing…
    jvm 1 | WrapperJarApp:
    jvm 1 | WrapperJarApp Error: Encountered an error running main:
    jvm 1 | WrapperJarApp Error: org.jruby.exceptions.RaiseException: (LoadError) library `fiber’ could not be loaded: java.lang.ClassNotFoundException: org.jruby.ext.fiber.ThreadFiberLibrary
    wrapper | <– Wrapper Stopped

  7. this doesn’t work for me at all on a fresh CentOS-6.2 install with in minute or so of attemtping to query the web interface the application exits.

    {“message”:”heartbeat”,”level”:”debug”,”file”:”/usr/local/bin/logstash/logstash-1.1.1-monolithic.jar!/logstash/agent.rb”,”line”:”472″,”method”:”run_with_config”}
    {“message”:”heartbeat”,”level”:”debug”,”file”:”/usr/local/bin/logstash/logstash-1.1.1-monolithic.jar!/logstash/agent.rb”,”line”:”472″,”method”:”run_with_config”}
    {“message”:”heartbeat”,”level”:”debug”,”file”:”/usr/local/bin/logstash/logstash-1.1.1-monolithic.jar!/logstash/agent.rb”,”line”:”472″,”method”:”run_with_config”}
    SystemCallError: Unknown error – Connection reset by peer
    sysread at org/jruby/RubyIO.java:2726
    read at file:/usr/local/bin/logstash/logstash-1.1.1-monolithic.jar!/gems/ftw-0.0.19/lib/ftw/connection.rb:230
    read_http_message at file:/usr/local/bin/logstash/logstash-1.1.1-monolithic.jar!/gems/ftw-0.0.19/lib/ftw/protocol.rb:30
    handle_connection at file:/usr/local/bin/logstash/logstash-1.1.1-monolithic.jar!/gems/ftw-0.0.19/lib/rack/handler/ftw.rb:125
    run at file:/usr/local/bin/logstash/logstash-1.1.1-monolithic.jar!/gems/ftw-0.0.19/lib/rack/handler/ftw.rb:107
    Exception `SystemExit’ at org/jruby/RubyThread.java:509 – Unknown error – Connection reset by peer
    {“message”:”heartbeat”,”level”:”debug”,”file”:”/usr/local/bin/logstash/logstash-1.1.1-monolithic.jar!/logstash/agent.rb”,”line”:”472″,”method”:”run_with_config”}

    ==> /var/spool/logstash/log/wrapper.log <==
    STATUS | wrapper | 2012/09/13 16:48:26 | <– Wrapper Stopped

    • I have removed the -vvv in the wrapper.conf and the service is now stable.

  8. Your setfacl is wrong (at least on centos 6). Should be:

    setfacl -m g:logstash:rx /var/log/httpd

  9. Also, you are missing some directory permission changes in /var/spool/logstash/data/elasticsearch/*

    Had to change ownership of the directories as they where all created and owned as root.

Comments are closed.