Five steps to tackle big graph data visualization

Understanding big graph data requires two things: a robust database and a powerful graph visualization engine. That’s why hundreds of developers have combined a graph database with KeyLines to create effective, interactive tools to explore and make sense of their graph data.

But humans are not big data creatures. Given most adults can store between 4 and 7 items only in their short term memory, loading an overwhelming quantity of densely-connected items into a chart won’t generate insight.

That presents a challenge for those of us building graph analysis tools. How do you decide which subset of data to present to users? How do they find the most important patterns and connections? That’s what we’ll explore in this blog post. You’ll discover that, with some thoughtful planning, big data doesn’t have to be a big problem.

For many organizations, ‘big data’ means collecting every bit of information available, then figuring out how to use it later. One of the many problems with this approach is that it’s incredibly challenging to go beyond aggregated analysis to understand individual elements.

To provide your users with something more useful, you need to think about the data funnel. Through back-end data management and front-end interactions, the funnel reduces billions of data points into something a user can comprehend.

Let’s focus on the key techniques you’ll apply at each stage.

There’s no point visualizing your entire database instance. You want to remove as much noise as possible, as early as possible. Filtering with database queries is an incredibly effective way to do this.

KeyLines’ flexibility means you can give users some nice visual ways to create custom filtering queries, like sliders, tick-boxes or selecting from a list of cases. In this example, we’re using queries to power a ‘search and expand’ interaction in KeyLines:

There’s no guarantee that filtering through search will be enough to keep data points at a manageable level. Multiple searches might return excessive amounts of information that’s hard to analyze. Filtering is effective, but it shouldn’t be the only technique you use.

Once filtering techniques are in place, you should consider aggregation. There are two ways to approach this.

Firstly, there’s data cleansing to remove duplicates and errors. This can be time-consuming but, again, queries are your friend. Functions like Cypher’s ‘count’ make it really easy to aggregate nodes in the back end.

Secondly, there’s a data modelling step to remove unnecessary clutter from entering the KeyLines chart in the first place. Can multiple nodes be merged? Can multiple links be collapsed into one?

It’s worth taking some time to get this right. With a few simple aggregation decisions, it’s possible to reduce tens of thousands of nodes into a few hundred.

Already by now, you should have reduced 1,000,000+ nodes to a few hundred. This is where the power of visualization really shines.