Setting up a central logging server with rsyslog

Rsyslog is an enhanced form of syslog service that has support for sending log files over TCP and UDP and also supports the more reliable RELP protocol.

This article will show you the following:

  • Set up a central logging server that receives log messages from another server
  • Set up a web server that uses rsyslog to send log messages to the central server

I am using Ubuntu 13.10 for this example. You can download the latest Ubuntu image from here.

Ubuntu already has rsyslog installed and running. To check that rsyslog is running on your machine:

### bash Check rsyslog is running
$ ps -ef | grep rsyslog
root      6106     1  0 02:12 ?        00:00:00 rsyslogd -c5

You can check the version of rsyslog you have running

### bash Version of rsyslog
$ /usr/sbin/rsyslogd -v
rsyslogd 5.8.11, compiled with:
FEATURE_REGEXP:                         Yes
FEATURE_LARGEFILE:                      Yes
GSSAPI Kerberos 5 support:              Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported:      Yes
64bit Atomic operations supported:      Yes
Runtime Instrumentation (slow code):    No

We want to use RELP protocol which is not installed by default so we need to install this module.

### bash Install RELP for rsyslog
$ sudo apt-get install rsyslog-relp

Central Logging Server

Now we can set up the rsyslog.conf configuration file for the central logging server.

Rsyslog configuration file: rsyslog.conf

Line 27 loads the RELP module which allows the server to receive messages in the RELP format. The ‘im’ means Input Module.

Line 28 specifies the port number to listen for incoming messages.

Line 59 defines the template for the output file based on some parameters in the message. We use the host name and program name to generate the file name.

Line 60 maps incoming messages to the output template to use. All messages use the HostProgramName template.

Web Server

Now we have the central logging server set up and configured we will now set up an rsyslog server on the web server machine and forward nginx logs to the main server.

To install nginx:

sudo apt-get install nginx
sudo nginx
ps -ef | grep nginx # check nginx is running

The rsyslog.conf configuration file for the web server machine.

Rsyslog configuration file for web server: rsyslog.conf

On lines 39-40 we have to change the user from syslog otherwise the service will not have permission to access the nginx log files. Line 52 loads the output module for the RELP protocol. This allows the service to send messages in the RELP format. Line 53 maps all messages to the output module for RELP specifying the ip address and port number to send messages to.

Line 50 uses the directive $IncludeConfig. This will include all configuration files in the /etc/rsyslog.d. We will place the configuration for the nginx server logs in this directory. Here is the config file.

nginx configuration for rsyslog: 40-nginx.conf

This configuration file will cause rsyslog to monitor the nginx log files every 10 seconds, sending any changes over the network (as we have previously configured this behaviour in the main configuration file).

Restart the rsyslog service

sudo service rsyslog restart

Test that messages are sent correctly

logger 'Test Message from web-server'

This command can be included in any shell script that we want to be able to send messages.

Here is how to send a message from within a python script.

import syslog

syslog.syslog('hello from python!')

This message, after a short delay, should appear in the log files of the central logger service.

$ tail /var/syslog/web-server/python.log
Dec 16 02:00:45 web-server python: hello from python!

To check nginx messages are being sent to the server we use the curl command.

$ curl -I http://web-server/
HTTP/1.1 200 OK
Server: nginx/1.4.1 (Ubuntu)
Date: Sun, 15 Dec 2013 20:32:12 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Mon, 06 May 2013 10:26:49 GMT
Connection: keep-alive
ETag: "51878569-264"
Accept-Ranges: bytes

Nginx messages received from the web server machine.

$ tail /var/syslog/web-server/nginx.log
Dec 16 04:32:12 web-server nginx: - - [16/Dec/2013:04:32:12 +0800] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8y zlib/1.2.5"