Web server logs: convert to JSON and upload to logentries

logentries banner

Logentries is a company that gives you insight into your log data. It collects log data from servers using either rsyslog or it’s own agent software and uploads the data to their servers to help you visualize your log data. In this article I will first show you how to switch your apache or nginx web server to convert to JSON formatting, which is the best format to use when sending data to logentries, and then show you how to access and query your log data on logentries.

Some big companies use logentries to visualize their logs files. An example is mailchimp, which will be well known service for anybody doing email marketing:


mailchimp logo

Change your web server logging to use JSON formatting

Apache 2 Web Server

We need to edit the apache2.conf file, this file is usually found in /etc/apache2 but may be in a different location depending on your setup.

Add the following line after the existing LogFormat directives:

LogFormat "{ \"time\":\"%t\", \"remoteIP\":\"%a\", \"host\":\"%V\", \"request\":\"%U\", \"query\":\"%q\",
\"method\":\"%m\", \"status\":\"%>s\", \"userAgent\":\"%{User-agent}i\",
\"referer\":\"%{Referer}i\" }" leapache

The LogFormat section should look something like this:

Once you have done that you need to edit each website configuration file. Here is an example virtual host configuration file:

Let’s change the access log to use JSON formatting:

If we load the home page and look at the access_company.log file we see:

{ "time":"[17/Mar/2015:11:39:27 +0800]", "remoteIP":"1.2.3.4", "host":"company", "request":"/index.php", "query":"", "method":"GET",
"status":"200", "userAgent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36",
"referer":"-" }

Apache is now logging requests in JSON format!

Nginx Web Server

Changing the log configuration for nginx is similar. Add the following lines to the log section of your nginx configuration file:

Restart nginx:

sudo service nginx restart

Request a page on the server and check the access log, it should look like this:

"time": "2015-03-17T01:30:07-04:00", "remote_addr": "1.2.3.4", "remote_user": "-", "body_bytes_sent": "7020", "request_time": "0.732", "status": "200", "request": "GET / HTTP/1.1", "request_method": "GET", "http_referrer": "-", "http_user_agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36"

We have now set up JSON logging for nginx!

Sending log files to Logentries

If you don’t already have a account with Logentries then you need to create one first. It’s free.

After you have an account set up we need to install the agent on our web server machine to collect the logs and send them to logentries.

Install Logentries Agent

First we need to ensure we have python set up tools installed:

sudo apt-get install python-setuptools

Then we run this following command:

wget https://raw.github.com/logentries/le/master/install/linux/logentries_install.sh && sudo bash logentries_install.sh

$ sudo apt-get install python-setuptools
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  python-setuptools
0 upgraded, 1 newly installed, 0 to remove and 119 not upgraded.
Need to get 230 kB of archives.
After this operation, 830 kB of additional disk space will be used.
Get:1 http://mirrors.digitalocean.com/ubuntu/ trusty/main python-setuptools all 3.3-1ubuntu1 [230 kB]
Fetched 230 kB in 0s (581 kB/s)
Selecting previously unselected package python-setuptools.
(Reading database ... 88699 files and directories currently installed.)
Preparing to unpack .../python-setuptools_3.3-1ubuntu1_all.deb ...
Unpacking python-setuptools (3.3-1ubuntu1) ...
Setting up python-setuptools (3.3-1ubuntu1) ...

$ wget https://raw.github.com/logentries/le/master/install/linux/logentries_install.sh && sudo bash logentries_install.sh
--2015-03-17 02:32:55--  https://raw.github.com/logentries/le/master/install/linux/logentries_install.sh
Resolving raw.github.com (raw.github.com)... 103.245.222.133
Connecting to raw.github.com (raw.github.com)|103.245.222.133|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://raw.githubusercontent.com/logentries/le/master/install/linux/logentries_install.sh [following]
--2015-03-17 02:32:56--  https://raw.githubusercontent.com/logentries/le/master/install/linux/logentries_install.sh
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 103.245.222.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|103.245.222.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9006 (8.8K) [text/plain]
Saving to: ‘logentries_install.sh’

100%[======================================================================================================================>] 9,006       --.-K/s   in 0s

2015-03-17 02:32:57 (41.1 MB/s) - ‘logentries_install.sh’ saved [9006/9006]


***** Step 1 of 3 - Beginning Logentries Installation *****
Updating packages...(This may take a few minutes if you have a lot of updates)
Installing logentries package...


***** Step 2 of 3 - Login *****
Account key is required. Enter your Logentries login credentials or specify the account key with --account-key parameter.
Email: <enter your email address>
Password:
Registered <host name>
Installing logentries daemon package...
Logentries Install Complete

The Logentries agent is now monitoring /var/log/syslog


***** Step 3 of 3 - Adding Logs to follow *****
9 logs will be followed.


Monitoring all logs
following: /var/log/messages
following: /var/log/dmesg
following: /var/log/auth.log
following: /var/log/boot.log
following: /var/log/daemon.log
following: /var/log/dkpg.log
following: /var/log/kern.log
following: /var/log/lastlog
following: /var/log/mail.log
following: /var/log/user.log
following: /var/log/Xorg.x.log
following: /var/log/alternatives.log
following: /var/log/btmp
following: /var/log/cups
following: /var/log/anaconda.log
following: /var/log/cron
following: /var/log/secure
following: /var/log/wtmp
following: /var/log/faillog


***** Install Complete! *****
To view your logs in real time, go to your Logentries account and select Live-Tail.

We want to follow the web server logs we just converted to JSON format. In this example I will show you how to follow the nginx log files just change the file path to follow apache log files instead.

$ sudo le follow /var/log/nginx/access.log
Will follow /var/log/nginx/access.log as access.log
$ sudo le follow /var/log/nginx/error.log
Will follow /var/log/nginx/error.log as error.log
$ sudo service logentries restart
 * Restarting Logentries monitoring agent logentries

Run the ‘le whoami’ command to verify the agent is following the nginx logs:

$ sudo le whoami
name = 
hostname = 
key = 
distribution = Ubuntu
distver = 14.04

Xorg.x.log
access.log
alternatives.log
anaconda.log
auth.log
boot.log
btmp
cron
cups
daemon.log
dkpg.log
dmesg
error.log
faillog
kern.log
lastlog
mail.log
messages
secure
syslog
user.log
wtmp
22 logs

The nginx logs files are on lines 9 and 20.

The Logentries agent will now automatically follow your web server logs!

Use Logentries to run queries on the web server logs

From the “Hosts & Logs” panel on the left hand side of the screen click on the Web Server Host Name.

You should see the imported log files:

logs imported into logentries

If you click on the access.log file link you will see the individual log entries as they are uploaded. If you want to see all of the JSON you can click on the Options button and check Expand JSON and Wrap Text.

Now we have some imported logs we can do some calculations. Type the following into the ‘Search events’ box at the top of the screen:

request_time>0 CALCULATE(AVERAGE)

Here is a screenshot of the result:

do calculation on log files

The next step is to create a Dashboard to get a quick overview of what is happening on the server. Logentries has community packs that are pre-configured queries for different servers.

community packs for logentries

Select the nginx community pack:

logentries nginx pack

Download the pack to your machine.

From the sidebar select Add Community Pack:

logentries add community pack

Select the queries you want to import and what log file(s) to use. Then you should see the widgets added to your dashboard:

logentries dashboard


See Also: