Install Graylog2 web interface on CentOS 6

We will be installing Graylog2 Web interface on CentOS 6 as Part 4 of our series on Monitoring your systems with logstash and Graylog2. ::Part 1::Part 2::Part 3::

Please Note! This configuration is for the 0.9.5 version of Graylog2 Server and has not been verified to function with the changes implemented in version 0.9.6, although most of the implementation should be similar.

We will be installing a Ruby on Rails web application framework driven by Passenger which runs as an Apache module. This means you’ll obviously need Apache installed as well as the g++ compiler (try “yum list gcc*” if you’re stuck) and typical development tools.

The installation overview:

  1. Install required packages
  2. Download Graylog2 Web Interface
  3. Create directory and copy files
  4. Configure Ruby
  5. Configure Bundler
  6. Edit “yml” config files
  7. Install Passenger
  8. Configure Passenger
  9. Configure Apache for Passenger
  10. Restart Apache and check for errors
  11. Check SELinux Status
  12. Set SELinux to permissive mode
  13. Configure database
  14. Create pid directory
  15. Test Ruby install with Brick
  16. Launch Web Interface

So let’s get started!

 
 

We’ll need to install some other packages found in the table below.

ruby-static - Static libraries for Ruby Devel
ruby-libs - Libraries Necessary to run Ruby
ruby-gems - Ruby Standard for packaging libraries
rubygem-rake - Ruby based make like utility
rubygem-hoe - rake/rubygems helper for Rakefiles
rubygem-gem_plugin - Plugin System based on Rubygems
ruby-docs Manuals and FAQS
ruby-devel - Ruby development environment
ruby - Interpreter
ruby-irb - Interactive Ruby
compat-readlines - Library for editing typed command lines
ruby-rdoc - tool to generate docs from Ruby source
g++ - GNU C++ compiler
libcurl-devel - Curl development headers with SSL support
openssl-devel - OpenSSL development headers
zlib-static - Zlib development headers
httpd-devel - Apache 2 development headers
apr-devel - Apache Portable Runtime (APR) development headers
apr-util-devel - Apache Portable Runtime Utility (APU) development headers

We’ll next need to download and unpack the Graylog2-web-interface to our home directory.

Next we’ll create a graylog2-web directory and copy the files. As root:

mkdir /var/www/graylog2-web
cd ~/graylog2-web-interface-X.X.X && cp -R ./* /var/www/graylog2-web/
chown -R apache.apache /var/www/graylog2-web/*

We’ll next configure Ruby, so as root run the commands below:

cd /var/www/graylog2-web && gem update
gem install git rake bundler

We next configure bundler

cd /var/www/graylog2-web && bundle install

Edit all “*.yml” files in /var/www/graylog2-web/config as appropriate (I needed to remove the comments in general.yml that were after the directives). Make sure your mongoid.yml matches your graylog2.conf and mongoDB settings as below.

production:
  host: 127.0.0.1
  port: 27017
  username: gluser
  password: grayloguser-password
  database: graylog2

Now we will install passenger to connect to apache. As root:

cd /var/www/graylog2 && gem install passenger

We next will Create and populate /etc/httpd/conf.d/passenger.conf

echo "LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-3.0.11/ext/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.11
PassengerRuby /usr/bin/ruby" > /etc/httpd/conf.d/passenger.conf

Our next step is to configure apache so passenger can run. You’ll need to edit your “/etc/httpd/conf/httpd.conf” creating a virtual host “gray.localhost”.

ServerName 127.0.0.1:80
#
DocumentRoot "/var/www/html"
#
<Directory "/var/www/html">
#
NameVirtualHost *:80
#
<VirtualHost *:80>
  ServerName gray.localhost
  DocumentRoot /var/www/graylog2-web/public
  RailsEnv production
  ServerAlias gray.localhost.localdomain
  ErrorLog logs/graylog2-error_log
  CustomLog logs/graylog2-access_log common
   <Directory /var/www/graylog2-web/public>
    Allow from all
    Options -MultiViews
   </Directory>
</VirtualHost>

We next restart apache and check for errors.

/etc/init.d/httpd/restart
tail --lines=20 /var/log/httpd/error_log

We likely notice errors that passenger does not start. If your configuration is solid, this is likely caused by SELinux. To see if SELinux is being enforced, as root:

getenforce

We can temporarily set SELinux to permissive mode. This will log the errors that are generated which will allow us to figure out which modules are being blocked. We can then tighten up the SELinux configuration at a later time. As root, run the following command:

echo 0 > /selinux/enforce

Restart apache again and see if passenger gets loaded this time. If you’re still having problems you will need to check your configuration.

We next create indexes and configure the database to our needs:

cd /var/www/graylog2-web
bundle exec rake db:mongoid:create_indexes RAILS_ENV=production --trace

We need to create a pid directory beneath the script directory and assign rights to apache

mkdir -p /var/www/graylog2-web/script/tmp/pids
chown -R apache.apache /var/www/graylog2-web/*

We can now to start Rails “brick” server as root from the application base directory. We do this to test if our rails setup is working correctly as brick gives us some nice status messages. If all seems good you can kill brick via Ctrl-C.

cd /var/www/graylog2-web
./script/rails server -e production

We should then be able to launch the interface via browser using passenger

http://gray.localhost

Hopefully everything is working as we expect and we are presented with the initial Graylog2 interface. Don’t forget that you’ve probably set SELinux to Permissive mode and will need to configure our file security settings to allow our application to run under SELinux.

A more secure Graylog2 server install

We will installing and configuring Graylog Server on CentOS 6 as Part 3 of our series on Monitoring your systems with logstash and Graylog2. ::Part 1::Part 2::Part 4::

Please Note! This configuration is for the 0.9.5 version of Graylog2 Server and will be updated to reflect the changes in the message store implemented in version 0.9.6

Our goals are simple:

  • Run as an unprivileged user
  • Consistent installation, logging, and configuration locations
  • Automating the installation

The installation overview:

  1. Download Graylog2 Server
  2. Create Intallation environment
  3. Unpack the Archive
  4. Configure Server Parameters
  5. Test installation
  6. Download and install the Java Service Wrapper
  7. Configure the Java Service Wrapper
  8. Test the Java Service Wrapper

We will assume you know the basics of system administration (ie.file permissions, ownership, etc.).
So let’s get this working!

Our first step will be to download the graylog2 server to our home directory.

Our next step is to create the installation environment. I’ve previously created a shell script for also installing logstash which will create the directories we’ll be needing, as well as to create our unprivileged user. You’ll need to run the script as root, but can test it as a normal user quite easily (with some minor modifications to the script).

The command to run the script is:

./prog_dir_setup graylog2

Our script will create the following directories:

~ create “/usr/local/bin/greylog2” to hold our greylog2 executables
       then create “bin” and “lib” directories underneath
~ create “/var/spool/greylog2” for our logs and pid
      then create “log”, “pid” and “data” directories underneath
         Note: we will not be using the data directory
~ create “/etc/greylog2” for our greylog2 config files

Our script will then create a system user named “graylog2”, and
modify directory permissions so “graylog2″ can write to those locations.

~ Create user “graylog2” and group “graylog2” as a system account
       pointing $HOME to “/var/spool/graylog2”
~ Change ownership of directories under “/var/spool/graylog2” to “graylog2.root”

We should next unpack the graylog2 server archive to “/usr/local/bin/graylog2” as root from our home directory.

We next copy the config file to “/etc/graylog2″

cp /usr/local/bin/graylog2/graylog2.conf.example /etc/graylog2/graylog2.conf

Our next step will be to configure our graylog2 server config file. Using your favorite editor edit /etc/graylog2/graylog2.conf such that it resembles the entries below. Remember to use the same username and password from setting up MongoDB

#since we will be getting syslog local from logstash
#
syslog_listen_port = 9514
# Or any other port over 1024 that you wish to use
#
#
#MongoDB Configuration
mongodb_useauth = true
mongodb_user = gluser
mongodb_password = grayloguser-password
mongodb_host = 127.0.0.1
mongodb_database = graylog2
mongodb_port = 27017

Now we can test if graylog2 server will run with our current configuration. “su” as user “graylog2″ and run the following command.

java -jar -DconfigPath=/etc/graylog2/graylog2.conf \
/usr/local/bin/graylog2/graylog2-server.jar

Now see if you are logging anything to Mongo in another terminal window as root.

tail -f /var/spool/mongo/log/mongod.log

If everything seems to be working as we hoped, we now configure graylog2 to run under the “Java Service Wrapper” which will allow us to run graylog2 as a service, launching as root and forking to run as user “graylog2″.

If you had not done previously, we should now download the Community version of the Java Service Wrapper and unpack it to ~/wrapper (ie. “/home/username/wrapper”).
We will then copy some files from the ~/wrapper directory for our graylog2 config.

~ Copy ‘wrapper.conf” to /etc/graylog2
 "cp ~/wrapper/src/conf/wrapper.conf.in /etc/graylog2/wrapper.conf"
~ Copy Shell File and rename it to something meaningful
 "cp ~/wrapper/src/bin/sh.script.in /usr/local/bin/graylog2/bin/graylog2_wrapper"
~ Copy “wrapper” executible
 "cp ~/wrapper/bin/wrapper /usr/local/bin/graylog2/bin/"
~ Copy wrapper/lib/libwrapper.so
 "cp ~/wrapper/lib/libwrapper.so /usr/local/bin/graylog2/lib/"
~ Copy wrapper/lib/wrapper.jar
 "cp ~/wrapper/lib/wrapper.jar /usr/local/bin/graylog2/lib/"

Edit the below lines in /usr/local/bin/graylog2/bin/graylog2_wrapper with your favorite editor.

 APP_NAME="graylog2"
 APP_LONG_NAME="Graylog Server"
 WRAPPER_CONF="/etc/graylog2/wrapper.conf"
 PIDDIR="/var/spool/graylog2/pid"
 RUN_AS_USER="graylog2"

Edit the below lines in /etc/graylog2/wrapper.conf with your favorite editor (Note: I’ve hard coded some perfectly valid variables contained in Tanuki’s provided script).

wrapper.java.command=/usr/bin/java
#
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperJarApp
#
wrapper.java.classpath.1=/usr/local/bin/graylog2/lib/wrapper.jar
wrapper.java.classpath.2=/usr/local/bin/graylog2/graylog2-server.jar
#
wrapper.java.library.path.1=/usr/local/bin/graylog2/lib
#
wrapper.java.additional.1=-DconfigPath=/etc/graylog2/graylog2.conf
#
wrapper.app.parameter.1=/usr/local/bin/graylog2/graylog2-server.jar
#
wrapper.logfile=/var/spool/graylog2/log/wrapper.log

You should now be able to launch graylog2 as root and have it run as graylog2.

In a terminal window as root:

"/usr/local/bin/graylog2/bin/graylog2_wrapper console"

If you run the wrapper script without any parameters, you’ll see it has a number of options available for our use.

[root@mylaptop sphughes]# /usr/local/bin/logstash/bin/logstash0_wrapper

Usage: /usr/local/bin/logstash/bin/logstash0_wrapper [ console | start | stop | restart | condrestart | status | install | remove | dump ]

Commands:

console     Launch in the current console.
start       Start in the background as a daemon process.
stop        Stop if running as a daemon or in another console.
restart     Stop if running and then start.
condrestart Restart only if already running.
status      Query the current status.
install     Install to start automatically when system boots.
remove      Uninstall.
dump        Request a Java thread dump if running.

That’s a wrap! We’ll next be posting instructions for setting up the Graylog2 web interface for viewing your logs.

Installing MongoDB for Graylog2

We will installing and configuring MongoDB on CentOS 6 as Part 2 of our series on Monitoring your systems with logstash and Graylog2. ::Part 1::Part 3::Part 4::, in order to support our Graylog2 server.

Our goals are simple:

  • Run as an unprivileged user
  • Consistent installation, logging, and configuration locations
  • Ability to scale to securely monitor other remote systems
  • Automating the installation
IMPORTANT NOTE BEFORE WE PROCEED!
You may be concerned about the installation of unsigned RPMs onto your system from the 10gen site. It this is the case, you should build from source.
YOU’VE BEEN WARNED

The installation overview:

  1. Add 10gen as a yum repository
  2. Download and install MongoDB
  3. Configure MongoDB
  4. Start MongoDB
  5. Create Admin User
  6. Create Greylog2 database
  7. Create Greylog2 user

We will assume you know the basics of system administration (ie.file permissions, ownership, etc.).
So let’s get this working!

We’ll first configure yum to check the 10gen repository for the lastest MongoDB
We’ll just echo the meager contents to create our file as root.
Below is for x86_64 CentOS

echo "[10gen]
name=10gen Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64
gpgcheck=0" > /etc/yum.repos.d/10gen.repo2

And below would be for the i386 32 bit CentOS

echo "[10gen]
name=10gen Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/i686
gpgcheck=0" > /etc/yum.repos.d/10gen.repo2

Next we’ll download and install MongoDB using yum, again as root:

yum info mongo*
yum install mongo-10gen*

Next we edit the MongoDB config file “/etc/mongod.conf” with our favorite editor
We want to set the following options

 logpath=/var/spool/mongo/log/mongod.log
 dbpath=/var/spool/mongo/data
 port = 27017
 auth = true

Now we will create the data directory and set permissions so mongod can write to the database. Again as root:

mkdir -p /var/spool/mongo/data
mkdir -p /var/spool/mongo/log
chown mongod.root /var/spool/mongo/*

Now we’re ready to start MongoDB. As root:

/etc/init.d/mongod start

Our next step is to create our default admin user. As normal user:

mongo
~ mongo now changes to a “>” shell prompt
>use admin
>db.addUser('admin', 'secure-password')

We next first authenticate, then create the graylog2 database.

>db.auth('admin', 'secure-password')
>use graylog2

That’s all there is to creating the database. Just too easy!

Next we create the graylog2 user and password for connecting to the database. Note the credentials for configuring the graylog2 server

>db.addUser('gluser', 'grayloguser-password')
>exit

We are now finished with configuring MongoDB, I hope it took just a few minutes. If you’d like to read up on MongoDB security, just head on over to the documentation.

A more secure logstash install

We will installing logstash in a more secure fashion on CentOS 6 as Part 1 of our series on Monitoring your systems with logstash and Graylog2. ::Part 2::Part 3::Part 4::, but our the methodology is applicable to other distributions as well. We’ll be using Tanuki Software’s open source version of their “Java Service Wrapper” to facilitate running the program as a system service.

Our goals are simple:

  • Run as an unprivileged user
  • Consistent installation, logging, and configuration locations
  • Ability to scale to securely monitor other remote systems
  • Automating the installation
IMPORTANT NOTE BEFORE WE PROCEED!
It is a very bad idea to expose your system logs to anyone who can attach to your computer with a browser.
PLEASE MAKE SURE YOUR FIREWALL IS WORKING
and dropping incoming connections. YOU’VE BEEN WARNED

The installation overview:

  1. Download the logstash package
  2. Create the installation environment
  3. Configure the logstash options
  4. Modify Access to our log files
  5. Test the logstash Install
  6. Download the Java Service Wrapper
  7. Configure the Java Service Wrapper
  8. Test running Logstash via the JSW
  9. Verify everything works
  10. Tie up loose ends

We will assume you know the basics of system administration (ie.file permissions, ownership, etc.).
So let’s get this working!

Our first step will be to download logstash. You’ll likely want to download the monolithic package. While you’re at the logstash site, take a look at the available documentation.

Our next step is to create the installation environment. I’ve created a shell script to create the directories we’ll be needing, as well as to create our unprivileged user. I’ll wait while you download and look over the script. You’ll need to run the script as root, but can test it as a normal user quite easily (with some minor modifications to the script).

The command to run the script is:

"./prog_dir_setup logstash"

Our script will create the following directories:

~ create “/usr/local/bin/logstash” to hold our logstash executables
       then create “bin” and “lib” directories underneath
~ create “/var/spool/logstash” for our logs, pid, and elasticsearch database
       then create “log”, “pid” and “data” directories underneath
~ create “/etc/logstash” for our logstash config files

Our script will then create a system user named “logstash”, and
modify directory permissions so “logstash” can write to those locations.

~ Create user “logstash” and group “logstash” as a system account
       pointing $HOME to “/var/spool/logstash”
~ Change ownership of directories under “/var/spool/logstash” to “logstash.root”

We should next unpack the logstash archive to “/usr/local/bin/logstash”, creating a “jar” archive like “logstash-X.X.XX-monolithic.jar”. We’ll then create a symbolic link to the file via root shell, which will enable us to update the installation very easily (Note: change “X.X.XX” to reflect your logstash version).

To create logstash.jar symbolic link, as root:

"cd /usr/local/bin/logstash && ln -s ./logstash-X.X.XX-monolithic.jar ./logstash.jar"

We will next create a simple configuration file for our test. Using your favorite editor, create a “mylogstash.conf” in “/etc/logstash”. My example is below and you can also download it via this link.

input {
   file {
      type => "linux-syslog"
      path => [ "/var/log/messages" ]
   }
   file {
      type => "apache-access"
      path => "/var/log/httpd/access_log"
   }
   file {
      type => "apache-error"
      path => "/var/log/httpd/error_log"
   }
}
output {
   stdout {
   }
   elasticsearch {
      embedded => true
   }
#   gelf {
#      chunksize => 1420
#      facility => "logstash-gelf"
#      host => "127.0.0.1"
#      level => "INFO"
#      port => 12201
#      sender => "%{@source_host}"
#   }
}

You’ll note the section titled “GELF” is commented out. We’ll be uncommenting that section in a later tutorial to connect to Graylog. So if you’re creating your own, you can discard the section.

Now we’ll need to modify some permissions to allow our “logstash” group to be able to read “/var/spool/messages” and our apache logs. We’ll accomplish this task by modifying ACLs on the necessary files and directories, using the “setfacl” command as root.

setfacl -m g:logstash:x /var/log/httpd
setfacl -m g:logstash:r /var/log/messages

We will now test our installation to verify that it functions as we expect before we configure the Java Service Wrapper.
As “root” you should “su” to become user “logstash” and test that our inputs are readable and if successful we will attempt to launch logstash.

su logstash
tail /var/log/httpd/access_log
tail /var/log/httpd/error_log
tail /var/log/messages

If no errors were reported we were successful, and can now launch logstash as user “logstash” with the following command (spread over multiple lines):

/usr/bin/java -Des.path.data="/var/spool/logstash/data/" \
-jar "/usr/local/bin/logstash/logstash.jar" agent -vvv \
-f "/etc/logstash/mylogstash.conf" \
-l "/var/spool/logstash/log/logstash.log" \
-- web --backend elasticsearch://127.0.0.1/?local

This command tells logstash to:

  • Use “/var/spool/logstash/data” to store our elasticsearch database
  • Use super verbose logging “-vvv”
  • Use “/etc/logstash/mylogstash.conf” as our config file
  • Log to “/var/spool/logstash/log/logstash.log”
  • Start the web interface to elasticsearch on the local host

Give logstash a couple minutes to get going, and see if we can view some data.
Point your browser at http://127.0.0.0:9292
Once the interface comes up, restart apache and/or plug in a USB device to generate some events. Then put an asterik in the “Query” box and click the “Search” button and you should see some logged events.

If everything seems to be working as we hoped, we now configure logstash to run under the “Java Service Wrapper” which will allow us to run logstash as a service, launching as root and forking to run as user “logstash”.

We should now download the Community version of the Java Service Wrapper and unpack it to ~/wrapper (ie. “/home/username/wrapper”), and we will then copy some files from the ~/wrapper directory for our logstash config.

~ Copy ‘wrapper.conf” to /etc/logstash
 "cp ~/wrapper/src/conf/wrapper.conf.in /etc/logstash/wrapper.conf"
~ Copy Shell File and rename it to something meaningful
 "cp ~/wrapper/src/bin/sh.script.in /usr/local/bin/logstash/bin/logstash0_wrapper"
~ Copy “wrapper” executible
 "cp ~/wrapper/bin/wrapper /usr/local/bin/logstash/bin/"
~ Copy wrapper/lib/libwrapper.so
 "cp ~/wrapper/lib/libwrapper.so /usr/local/bin/logstash/lib/"
~ Copy wrapper/lib/wrapper.jar
 "cp ~/wrapper/lib/wrapper.jar /usr/local/bin/logstash/lib/"

Edit the below lines in /usr/local/bin/logstash/bin/logstash0_wrapper with your favorite editor.

 APP_NAME="logstash0"
 APP_LONG_NAME="Logstash Local Collector"
 WRAPPER_CONF="/etc/logstash/wrapper.conf"
 PIDDIR="/var/spool/logstash/pid"
 RUN_AS_USER="logstash"

Edit the below lines in /etc/logstash/wrapper.conf with your favorite editor (Note: I’ve hard coded some perfectly valid variables contained in Tanuki’s provided script).

wrapper.java.command=/usr/bin/java

wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperJarApp

wrapper.java.classpath.1=/usr/local/bin/logstash/lib/wrapper.jar

wrapper.java.classpath.2=/usr/local/bin/logstash/logstash.jar

wrapper.java.library.path.1=/usr/local/bin/logstash/lib

wrapper.java.additional.1=-Des.path.data=/var/spool/logstash/data/

wrapper.app.parameter.1=/usr/local/bin/logstash/logstash.jar
wrapper.app.parameter.2=agent
wrapper.app.parameter.3=-vvv
wrapper.app.parameter.4=-f
wrapper.app.parameter.5=/etc/logstash/mylogstash.conf
wrapper.app.parameter.6=-l
wrapper.app.parameter.7=/var/spool/logstash/log/logstash.log
wrapper.app.parameter.8=--
wrapper.app.parameter.9=web
wrapper.app.parameter.10=--backend
wrapper.app.parameter.11=elasticsearch://127.0.0.1/?local

wrapper.logfile=/var/spool/logstash/log/wrapper.log

You should now be able to launch logstash as root and have it run as logstash.
In a terminal window as root:

"/usr/local/bin/logstash/bin/logstash0_wrapper console"

You can again check everything is running as expected at:
http://127.0.0.1:9292

We will now edit the logstash account to remove it’s bash shell
you’ll want to do this after logstash is working as expected.

usermod -s /bin/false logstash

If you run the wrapper script without any parameters, you’ll see it has a number of useful options.

[root@mylaptop sphughes]# /usr/local/bin/logstash/bin/logstash0_wrapper

Usage: /usr/local/bin/logstash/bin/logstash0_wrapper [ console | start | stop | restart | condrestart | status | install | remove | dump ]

Commands:

console     Launch in the current console.
start       Start in the background as a daemon process.
stop        Stop if running as a daemon or in another console.
restart     Stop if running and then start.
condrestart Restart only if already running.
status      Query the current status.
install     Install to start automatically when system boots.
remove      Uninstall.
dump        Request a Java thread dump if running.

That’s a wrap. We can now resume normal logging instead of verbose. We’ll be expanding on our logstash usage in future articles. For the lastest news and information, I suggest you subscribe to the logstash users google group.

Monitor your systems with logstash and Graylog2

I begin a series on configuring a secure local CentOS 6 installation of logstash and Graylog2 which are open source alternatives to commercial packages such as Splunk or Logscape. Using these tools you can see detailed and historical machine data such as WWW and FTP logs, as well as aggregate system logs.

We’ll also make use of a number of other open source tools in our installation, MongoDB, Java Service Wrapper, elasticsearch, and Ruby on Rails running on Apache via Passenger.

Our initial configuration will be a stand alone system, but in future posts we’ll show you how to expand our monitoring of remote systems.

While these tools are not fully mature, and you might not want to use them in an enterprise environment, they are almost certainly an improvement over the method you currently use (I suspect you are not fully reviewing your log files daily).

Our configuration goals in building this test system shall be:

  • Consistent Installation and configuration
  • Running the processes as an unprivileged user
  • Ability to scale to securely monitor other remote systems

So let’s get started!