Kibana dashboards, from the Elasticsearch project, can help you visualise activity and incidents in log files. Here I show 3 different types of use cases for dashboards and how each can be used to answer different questions depending on the person.  Video and details follow.

Text Search Dashboard

The first example is the simplest UI I could imagine: a query/search box, a histogram, and a table.  In this instance any user, at any level of curiosity, can find textual data in the logs using a keyword match.

Kibana 3 Dashboard: Text Search

Then they can also see the relative number of records that occur at given times within the time window of all the data available.  These are aggregate counts of all records that have some match to the query keyword or algebra.

Likewise, the table reflects the subset of data provided by the records, with the ability to only show fields of interest.

Process Details

A slightly more advanced use is to focus on a particular process (i.e. application) running on a machine that’s being logged.  Here we can then take a particular metric, i.e. CPU usage, and graph it instead of just a simple histogram.

Kibana 3 Dashboard: Process Details

A typical user may be in charge of a particular set of services in a system.  Here they can see how they perform and yet still dig into the details as desired.

I also do some cool “markers” to subtly show when events coincide with other process metrics.

All Events

The data example shown here has process, performance and event logging information.  I combine multiple queries and having them drive different parts of the dashboard – a pie chart, summary table, histogram, sparkline and other charts based on numeric data.

Kibana 3 Dashboard: All Events

These can then all be filtered based on time windows that are interactively selected.  This is really the typical picture of a dashboard – giving more densely packed information about a variety of metrics, ideal for system managers to get a handle on things.

Streaming Pipeline

The data are generated by Windows servers using a custom C# application that pushes data in a Kafka topic in a Hadoop cluster running in EC2. The data stream is then read from the topic using Actian DataFlow platform and pushed into Elasticsearch for Kibana to use at the end of the pipeline.  There are other reasons I have this kind of pipeline – namely that DataFlow can simultaneously also feed other outgoing part of the pipeline – RDBMS, graph, etc.   More on that in a future video/post.

Next Steps

  • My next plans are to show you Kibana version 4 in action, replicating some of what I’ve shown here.
  • If you haven’t seen it already, see this link and my video with some tips and tricks for using Kibana 3.
  • Tell me more about your interests in dashboards and I’ll consider focusing on them too.

About Tyler Mitchell

Director Product Marketing @ OmniSci.com GPU-accelerate data analytics | Sr. Product Manager @ Couchbase.com - next generation Data Platform for System of Engagement! Former Eng. Director @Actian.com, author and technology writer in NoSQL, big data, graph analytics, geospatial and Internet of Things. Follow me @1tylermitchell or get my book from http://locatepress.com/.