Splunk 3.1: Log-Monitoring Revisited
New FeaturesMany moons have risen since I last gushed about Splunk, so what better way to reinvigorate our personal buzz than to install the latest version and write a how-to. After talking about a few neat features, we will briefly discuss how to set up central syslogging and how to install Splunk, before a tangent into "working around the free version's crippled interface."
First off, the latest Splunk version is a bit more polished. That may seem strange to say, but diamonds can, indeed, shine more if you find a new, better way to polish them. Splunk version 3.1.x, most noticeably, has a new front page. Ignoring the annoying "yes I want to use the free license" splash page, the front page is really the dashboard. You are no longer limited to just one dashboard, though!
The default graph shows how many total log entries have been processed in the last few days. Any other Saved Search can also be configured to display a graph, which is extremely handy. For example, say we're interested in the number of viruses our mail servers have found recently. If you use ClamAV, you simply perform a search to get the results you want, in this case:
clamav FOUND will do it. It says that 1,316 events´┐Ż in the past 24 hours were found--sounds correct. You probably want to click on the
sourcetype:sendmail_syslog text to add that search term as well; it immensely speeds up search times. Save the search, click the box to display it in the default dashboard, and--ta-da--the dashboard now has a graph. At a glance, we can see how many viruses per hour we have blocked in the last 24 hours. In the past, this type of information was only available by scripting something that scanned log files.
That's just one example, and there are tons more--the amount of spam rejected, failed login attempts, OSPF adjacency changes--the list goes on. You will want to schedule the search to run every few minutes for the most up-to-date dashboard information, which is easily accomplished via the Saved Search settings. Given too many saved searches, the dashboard will start loading slowly, which is surmountable by creating purpose-specific dashboards. Creating these wonderful graphs is highly addicting, so allocate a few hours before you start playing with it.