By Tom West

The Challenge

Splunk is a great tool for combining data from various silos into one place, regardless of structure. However, once available, this amount of data can be difficult to navigate. Translating it into something meaningful can be challenging. In order to do so, Splunk Searches are needed, but they come with their own issues:

  • They have to be written almost perfectly in order to work. This isn’t easy, especially if you’re unfamiliar with the data you’re working with.
  • They often don’t work as expected, perhaps due to a typo or irregular data.
  • Failed Searches can be tricky to diagnose with the tools currently available within Splunk.

When trying to work out what’s gone wrong with a Search, you can use the manual debug mode, where you have the options to break the search apart, or look through the performance figures within the job inspector.

I work with Splunk every day at Converging Data. My colleagues and I often found this process frustrating, slow, and prone to error in itself. We knew there must be a better way.

The Solution

I recently created a free Splunk tool, SPL Rehab. This new app helps you quickly and easily identify issues with your Search. It breaks down your results and displays them in a single dashboard, so you can work out where the problem lies much faster.

You can view the state of your Search through various states of execution, helping to place the point at which issues arose. SQL Rehab even suggests ways in which you can improve the performance, such as running a tstats command.

Keep reading for more detail about how the app works, or:

Splunk Search breakdown results

We often found it hard to identify which line in a search hadn’t worked as intended. To help with this, SPL Rehab includes a table that displays every line of your search, including:

• Command
• Statement
• Duration
• Result Count
• Range Indicator

This makes it very easy to identify any lines which may need further investigation, as you can scan the results to see where errors occurred or where a result is unexpected.

Search Breakdown results

Partially running a Splunk Search

Selecting any line of the table on the Search Breakdown will execute the search up to the line you have selected. You can then view information about the search up to that point, including performance statistics, result statistics, and physical results.

Results can be displayed in either event, tabular, or field summary form. This gives you more control of what you see and when, depending on your investigation. There’s even an option to display the search that was executed and copy it into a separate search window if required.

Field summary results of partially selected search

View Splunk Search Stats

With SPL Rehab, you can view the overall stats of the originally executed Search. These are separated into Result statistics and Performance Statistics.

Result Statistics displays:

• Overall result count
• A timechart for the results (where available)
• Field Summary

Result stats

Performance Statistics shows:

• Search duration
• EPS (events per second)
• Lispy (the filter sent to the indexer)
• NEW FEATURE FOR V1.2 – a timeline for each entry in search.log
• A timeline for each command in the search
• A charted comparison of time spent using the following figures:
– Getting the data location information
– Getting the data from the journal
– Performing the search-time field extractions

Some of the Performance Stats mentioned above

All of this information is designed to make troubleshooting and improving your searches as easy as possible by putting everything together in a single dashboard. With it, you have control of seeing just what you want to see, when you want to see it! 

Make Search troubleshooting simpler today and download SPL Rehab.

Useful links



Support/bug reporting

About Tom

Tom West is a Solutions Architect at Converging Data. He has over 7 years’ experience in software development, with eight apps currently available on Splunkbase. Tom also runs the Splunk Yorkshire group.