Splunk Notes

list available indexes:

| eventcount summarize=false index=* | dedup index | fields index

host="10.35.12.1" | stats count by action, host

Fortigate by country:

host="10.35.12.1" | stats count by src_country

log lines by time

host="10.35.12.161" | chart count by _time

grep -v

host="10.35.12.161" NOT "slapd"

log data per day:

index=_internal source=*metrics.log splunk_server="*" group="per_index_thruput" | eval MB=kb/1024 | chart span=1d sum(MB) over _time

hosts that have not logged in recent time:

| metadata index=* type=hosts | eval age = now()-lastTime | where age > (1) | sort age d | convert ctime(lastTime) | fields age,host,lastTime

where age > (1)

the “(1)” is in seconds
can calculate days in seconds by input value (4*86400)
86400 seconds in 1 day

using lookups

host="myserver" | lookup lookup_priority_FacilitySeverity priority as priority OUTPUT FacilitySeverity as FacilitySeverity | search FacilitySeverity!="Local4.Debug" | chart count by _time , FacilitySeverity  span=1h

where a csv file was created :

priority,FacilitySeverity
0,Kernel.Emergency
1,Kernel.Alert
2,Kernel.Critical
3,Kernel.Error
4,Kernel.Warning
5,Kernel.Notice
6,Kernel.Info
7,Kernel.Debug
8,User.Emergency
9,User.Alert
10,User.Critical

and uploaded to splunk and complete with lookup definitions.

sending alerts to scripts

http://wiki.splunk.com/Community:Use_Splunk_alerts_with_scripts_to_create_a_ticket_in_your_ticketing_system

To do this, set up your saved search, put it on a schedule, and set the action to trigger a shell script you've written whenever the number of events you're interested in is> 0.

Put your script (not the Remedy script) in /opt/splunk/bin/scripts.

This script should call the Java program that Remedy uses to generate tickets and pass it data from the Splunk alert. Splunk alerts support the following variables:

• $1 = number of events returned
• $2 = search terms
• $3 = fully qualified search string
• $4 = name of the saved search
• $5 = the reason the action/script was triggered (for example, the number of events returned was >1)
• $6 = a link to the saved search in Splunk +
• $7 = a list of the tags belonging to this saved search (this option was removed starting in Splunk 3.6)
• $8 = path to a file where raw results of this search are located (as opposed to passing the actual results into the ticket--this could be a lot of data).

The following example script passes the reason the script was triggered, a link to the saved search, and the path to the search results file into the ticket that the generateRemedyTicket Remedy script creates when it's run.