How to tap into Splunk’s big data analytics engine
Splunk takes machine data in the form of log files, or other computer-generated text files, and turns it into operational intelligence.
Machine data (log files), as data goes, is not particularly exciting as compared to say, human genomic information, but Splunk turns our “digital exhaust” into electronic gold for those who are willing to spend a little time to examine it. In fact, machine data is the fastest growing area of big data in enterprises; it includes every user transaction, every system message, every suspicious activity, and every machine-to-machine interaction. And believe it or not, it’s exciting to see what’s happening on your network. Splunk’s powerful Search Processing Language (SPL), originally based on the Unix pipeline and SQL, makes data analysis much more fun and much less complex than you’d think.
The big data part of Splunk comes from the log files, events, and other information that you send to it either directly or via forwarders. The Enterprise version begins its entry level licensing at 5 GB of data per day. For some enterprises that is a huge amount of data and you might never reach that 5 GB limit. In larger enterprises that host system farms, 5 GB might be an hourly average for log files, so you’ll have to increase your licensing limit to accommodate the volume of data you need to ingest.
Once you have Splunk Enterprise up and running and acquiring data, now it’s time to tap into its big data analytics engine. To begin using Splunk’s analytics, you must acquire data by adding so-called data inputs into your Splunk system. You may acquire data by uploading, monitoring, and forwarding. Some data types do not work using the upload feature, such as Windows event logs (.evt or evtx); Windows events are best configured for ingest by installing the universal forwarding client.
Though your total amount of collected data can be very large, Splunk’s SPL provides data users with a Google-like search experience that includes search term auto-completion and a highly configurable filtering capability. For example, you can enter a single keyword, such as a host’s name (adserver1), into the New Search field and view all results that match your keyword. A broad search can yield thousands or hundreds of thousands of results.
To filter these results, enter more specific information into the New Search field, such as host=”adserver1″ sourcetype=”activedirectory”. This refined search will narrow returned results into a more manageable dataset that only includes Active Directory-related events.
