Splunk 3.1: Log-Monitoring Revisited - page 2

New Features

We generally find that existing infrastructures already have central syslog servers, but in case you don't, here's a quick rundown of what it is all about.

Even the old Unix syslogd program is capable of sending syslog entries to a remote server. The configuration looks something like: *.err @loghost.domain

Unfortunately, the classic syslog daemon will only send logs to a single place. If you wanted to leave a copy of some logs locally, you were out of luck. With syslog-ng, available on most Unix and Linux platforms today, you can (among other fanciness) speficy multiple destinations for each facility.severity specified. For example:

*.err  /var/log/messages
*.* /var/log/syslog

The above will send any err severity messages to /var/log/messages, yet still log everything to one mondo-log file, /var/log/syslog. In fact, you can do this as many times as you like. Each server on your network should be configured to send *.* to a central log server. Instead of a file name, simply put @hostname as the destination.

For Splunk's purposes, it's best to simply add another line on the log server, if you want everything sent to splunk, saying: *.* |/var/run/splunk-pipe. What's all this, you ask? Well it's a named pipe, or FIFO. You can create the FIFO with mkfifo /var/run/splunk-pipe.

A FIFO is much more resource-friendly than constantly reading your text log files. A FIFO is a buffer that can be written to by one program, and read from by another in a First In Last Out fashion. To make this work, you simply need to configure Splunk to read from the FIFO.