Creating a Graph Analytics Data Application Using a Supercomputer
We recently had to prototype a data application over a supercomputer tuned for graph analysis. We built a proof-of-concept leveraging multiple APIs, Cray’s Urika-GX and Graph Engine (CGE), and a handful of programming languages in less than a week.
The application consumes and visualizes the Panama Papers, over 11.5 million documents detailing international financial and legal misconduct and covering relationships between over 500,000 companies, foundations, and trusts. Users build a list of suspected bad actors by interacting with a high-performance network diagram.
This was just a proof of concept, but it had to prove some worthwhile capabilities and address a few interesting challenges.
First, our industry partners wanted to allow multiple users to build and manage a list of bad actors together. We decided to use data.world to host this list. The xap (what we call a data application made with the Exaptive Studio) would be able to query the Panama Papers on Cray’s supercomputer and the bad actor’s list data.world API and it would write back to the data.world as users identified more potential bad actors. (If you’re not familiar with data.world, here’s a nice intro.)
Our next worry was the performance of the visualization. A network diagram seemed to be the inevitable place to start to visualize the myriad of people and corporate entities implicated in the data. But big network diagrams are notoriously bad performers in the browser. Networks with more than 1,000 nodes and edges tend to be useless hairballs, and the point here was to create a useful visualization of 50,000 nodes and edges. So we opted to use a WebGL network diagram rather than the more popular d3 variety. (There is a public WebGL network diagram component for the Studio and a quick-start version with some built-in configuration.)
Lastly, the Cray CGE uses RDF and the native format of the Panama Papers database is Neo4j. We thankfully found a website that had already done the conversion to RDF. We just had to create some SPARQL components to work on the data.
We use a visual, data flow programming environment to build applications — a data flow between code modules — “components” — that perform operations on the data.
