Graph Databases Go Mainstream
Every decade seems to have its database. During the 1990s, the relational database became the principal data environment, its ease of use and tabular arrangement making it a natural for the growing needs to power the data web. While relational databases remained strong, the 2000s saw the emergence of XML databases, and NoSQL, the idea that databases didn’t need to be structured in a purely tabular form, began to get hot. During the 2010s, JSON databases gained traction, along with the spectacular rise and ultimate fall of Hadoop as a data platform. Given that, there is compelling evidence that graph databases will be the go-to database of the 2020s,
Graph databases have been around in one form or another since the early oughts, but they were generally slower, more complex to work with, and more limited in terms of their applicability than relational databases. They were also seen primarily as academic databases, as one of the earliest use cases for such databases was to build logical analysis systems, and because of its academic associations, the database stayed well under the commercial radar for a significant number of years.
By mid-2014, however, several key advances had all come together to give graph database technology its first of several boosts. Neo4J, an early and still heavily used graph database, had begun to gain enough maturity and penetration that it began to be used for certain classes of mathematical graph processing. Hardware (through cloud computing) had also become fast enough that many of the key early challenges of performance could be overcome. A graph query language (SPARQL) came out with a second edition that solved many of the problems that an earlier version had brought with it, including the introduction of an update capability that meant that a consistent mechanism could emerge for adding content dynamically. The emergence of JSON data stores such as MongoDB and CouchDB had put a lot more focus on the challenges of indexing non-traditional databases, which has, in turn, led to significant optimization in handling joins, one of the core requirements of any database but especially important for graph databases.
As important, several companies began experimenting with graph databases to solve problems that were beginning to become vexing at the corporate level – management of enterprise metadata, master data management, natural language processing, knowledge navigation and other related problems where search technology by itself had bottomed out. Ironically, machine learning, which typically uses a data clustering approach to text analysis as a brute force alternative has recently increasingly become just another mechanism to help build graph databases, to the extent that the most recent graph databases are now incorporating machine-learning algorithms and tools as part of their core suites.
Finally, graph databases are increasingly taking advantage of a completely unexpected development – the growing power and sophistication of graphics processing units, or GPUs, those specialized multi-core computers that are dedicated to creating sophisticated three-dimensional graphics. These GPUs work by creating meshes broken down into triangles – millions or even billions of triangles. By assigning properties to the points (nodes) and lines (edges) that make up these meshes, the GPUs can calculate color, lighting, hardness and softness of edges, displacement and dozens of other properties, and can do so very, very quickly because of the pipeline architecture that they employ.
It turns out that a graph database is really not that much different (in fact they are nearly identical) to the meshes used in graphics processing, especially from the standpoint of assigning properties to nodes and edges. Each node represents a concept, each edge represents a relationship.
