A more secure logstash install

We will installing logstash in a more secure fashion on CentOS 6 as Part 1 of our series on Monitoring your systems with logstash and Graylog2. ::Part 2::Part 3::Part 4::, but our the methodology is applicable to other distributions as well. We’ll be using Tanuki Software’s open source version of their “Java Service Wrapper” to facilitate running the program as a system service.

Our goals are simple:

  • Run as an unprivileged user
  • Consistent installation, logging, and configuration locations
  • Ability to scale to securely monitor other remote systems
  • Automating the installation
It is a very bad idea to expose your system logs to anyone who can attach to your computer with a browser.
and dropping incoming connections. YOU’VE BEEN WARNED

The installation overview:

  1. Download the logstash package
  2. Create the installation environment
  3. Configure the logstash options
  4. Modify Access to our log files
  5. Test the logstash Install
  6. Download the Java Service Wrapper
  7. Configure the Java Service Wrapper
  8. Test running Logstash via the JSW
  9. Verify everything works
  10. Tie up loose ends

We will assume you know the basics of system administration (ie.file permissions, ownership, etc.).
So let’s get this working!

Our first step will be to download logstash. You’ll likely want to download the monolithic package. While you’re at the logstash site, take a look at the available documentation.

Our next step is to create the installation environment. I’ve created a shell script to create the directories we’ll be needing, as well as to create our unprivileged user. I’ll wait while you download and look over the script. You’ll need to run the script as root, but can test it as a normal user quite easily (with some minor modifications to the script).

The command to run the script is:

"./prog_dir_setup logstash"

Our script will create the following directories:

~ create “/usr/local/bin/logstash” to hold our logstash executables
       then create “bin” and “lib” directories underneath
~ create “/var/spool/logstash” for our logs, pid, and elasticsearch database
       then create “log”, “pid” and “data” directories underneath
~ create “/etc/logstash” for our logstash config files

Our script will then create a system user named “logstash”, and
modify directory permissions so “logstash” can write to those locations.

~ Create user “logstash” and group “logstash” as a system account
       pointing $HOME to “/var/spool/logstash”
~ Change ownership of directories under “/var/spool/logstash” to “logstash.root”

We should next unpack the logstash archive to “/usr/local/bin/logstash”, creating a “jar” archive like “logstash-X.X.XX-monolithic.jar”. We’ll then create a symbolic link to the file via root shell, which will enable us to update the installation very easily (Note: change “X.X.XX” to reflect your logstash version).

To create logstash.jar symbolic link, as root:

"cd /usr/local/bin/logstash && ln -s ./logstash-X.X.XX-monolithic.jar ./logstash.jar"

We will next create a simple configuration file for our test. Using your favorite editor, create a “mylogstash.conf” in “/etc/logstash”. My example is below and you can also download it via this link.

input {
   file {
      type => "linux-syslog"
      path => [ "/var/log/messages" ]
   file {
      type => "apache-access"
      path => "/var/log/httpd/access_log"
   file {
      type => "apache-error"
      path => "/var/log/httpd/error_log"
output {
   stdout {
   elasticsearch {
      embedded => true
#   gelf {
#      chunksize => 1420
#      facility => "logstash-gelf"
#      host => ""
#      level => "INFO"
#      port => 12201
#      sender => "%{@source_host}"
#   }

You’ll note the section titled “GELF” is commented out. We’ll be uncommenting that section in a later tutorial to connect to Graylog. So if you’re creating your own, you can discard the section.

Now we’ll need to modify some permissions to allow our “logstash” group to be able to read “/var/spool/messages” and our apache logs. We’ll accomplish this task by modifying ACLs on the necessary files and directories, using the “setfacl” command as root.

setfacl -m g:logstash:x /var/log/httpd
setfacl -m g:logstash:r /var/log/messages

We will now test our installation to verify that it functions as we expect before we configure the Java Service Wrapper.
As “root” you should “su” to become user “logstash” and test that our inputs are readable and if successful we will attempt to launch logstash.

su logstash
tail /var/log/httpd/access_log
tail /var/log/httpd/error_log
tail /var/log/messages

If no errors were reported we were successful, and can now launch logstash as user “logstash” with the following command (spread over multiple lines):

/usr/bin/java -Des.path.data="/var/spool/logstash/data/" \
-jar "/usr/local/bin/logstash/logstash.jar" agent -vvv \
-f "/etc/logstash/mylogstash.conf" \
-l "/var/spool/logstash/log/logstash.log" \
-- web --backend elasticsearch://

This command tells logstash to:

  • Use “/var/spool/logstash/data” to store our elasticsearch database
  • Use super verbose logging “-vvv”
  • Use “/etc/logstash/mylogstash.conf” as our config file
  • Log to “/var/spool/logstash/log/logstash.log”
  • Start the web interface to elasticsearch on the local host

Give logstash a couple minutes to get going, and see if we can view some data.
Point your browser at
Once the interface comes up, restart apache and/or plug in a USB device to generate some events. Then put an asterik in the “Query” box and click the “Search” button and you should see some logged events.

If everything seems to be working as we hoped, we now configure logstash to run under the “Java Service Wrapper” which will allow us to run logstash as a service, launching as root and forking to run as user “logstash”.

We should now download the Community version of the Java Service Wrapper and unpack it to ~/wrapper (ie. “/home/username/wrapper”), and we will then copy some files from the ~/wrapper directory for our logstash config.

~ Copy ‘wrapper.conf” to /etc/logstash
 "cp ~/wrapper/src/conf/wrapper.conf.in /etc/logstash/wrapper.conf"
~ Copy Shell File and rename it to something meaningful
 "cp ~/wrapper/src/bin/sh.script.in /usr/local/bin/logstash/bin/logstash0_wrapper"
~ Copy “wrapper” executible
 "cp ~/wrapper/bin/wrapper /usr/local/bin/logstash/bin/"
~ Copy wrapper/lib/libwrapper.so
 "cp ~/wrapper/lib/libwrapper.so /usr/local/bin/logstash/lib/"
~ Copy wrapper/lib/wrapper.jar
 "cp ~/wrapper/lib/wrapper.jar /usr/local/bin/logstash/lib/"

Edit the below lines in /usr/local/bin/logstash/bin/logstash0_wrapper with your favorite editor.

 APP_LONG_NAME="Logstash Local Collector"

Edit the below lines in /etc/logstash/wrapper.conf with your favorite editor (Note: I’ve hard coded some perfectly valid variables contained in Tanuki’s provided script).









You should now be able to launch logstash as root and have it run as logstash.
In a terminal window as root:

"/usr/local/bin/logstash/bin/logstash0_wrapper console"

You can again check everything is running as expected at:

We will now edit the logstash account to remove it’s bash shell
you’ll want to do this after logstash is working as expected.

usermod -s /bin/false logstash

If you run the wrapper script without any parameters, you’ll see it has a number of useful options.

[root@mylaptop sphughes]# /usr/local/bin/logstash/bin/logstash0_wrapper

Usage: /usr/local/bin/logstash/bin/logstash0_wrapper [ console | start | stop | restart | condrestart | status | install | remove | dump ]


console     Launch in the current console.
start       Start in the background as a daemon process.
stop        Stop if running as a daemon or in another console.
restart     Stop if running and then start.
condrestart Restart only if already running.
status      Query the current status.
install     Install to start automatically when system boots.
remove      Uninstall.
dump        Request a Java thread dump if running.

That’s a wrap. We can now resume normal logging instead of verbose. We’ll be expanding on our logstash usage in future articles. For the lastest news and information, I suggest you subscribe to the logstash users google group.

Learning the BASH shell

Today we’ll explore the BASH shell a bit.

BASH is extremely powerful, yet succinct.

To view the built-in BASH commands available on a Linux system try:
“man -k bash”

To save these to a file in your home directory for later review try:
“man -k bash |awk ‘{print $1}’ > ~/bash_commands_available.txt”

And to view the available options for each command try:
“help command_I_want_to_learn”
replacing “command_I_want_to_learn” with something like “cat”

To learn about the help system try:
“help help”

Monitor your systems with logstash and Graylog2

I begin a series on configuring a secure local CentOS 6 installation of logstash and Graylog2 which are open source alternatives to commercial packages such as Splunk or Logscape. Using these tools you can see detailed and historical machine data such as WWW and FTP logs, as well as aggregate system logs.

We’ll also make use of a number of other open source tools in our installation, MongoDB, Java Service Wrapper, elasticsearch, and Ruby on Rails running on Apache via Passenger.

Our initial configuration will be a stand alone system, but in future posts we’ll show you how to expand our monitoring of remote systems.

While these tools are not fully mature, and you might not want to use them in an enterprise environment, they are almost certainly an improvement over the method you currently use (I suspect you are not fully reviewing your log files daily).

Our configuration goals in building this test system shall be:

  • Consistent Installation and configuration
  • Running the processes as an unprivileged user
  • Ability to scale to securely monitor other remote systems

So let’s get started!

Timestamp your bash history

In Memoriam: Dennis Ritchie

I want to begin this blog with something incredibly useful, yet simple.

When you find yourself trying to remember when you made that all important, late night configuration change, it really helps to have timestamps in your bash history.

Try adding the lines below to your “.bashrc”

       export HISTTIMEFORMAT