Building & exploring a music knowledge graph
It is always a valuable opportunity to understand our product better and recognize user needs. At GraphAware, building Hume, a graph-powered insight engine, we are proud of making an impact on our customers’ success. However, we use Hume also to support our processes and help our own needs. In the case of the event that took place throughout December, we were also able to have great fun and integrate the team.
Have you heard about the 30 Day Song Challenge? This is a game that cheered us up on short European December days. Every day we proposed songs on Slack according to a new theme. In this way, we collected almost 400 songs in various categories.
The game was well received, and it was met with a good response from the team.
We realised that creating a database of our music tastes would be interesting and fun. Having enriched it from different data sources, our data science engineers will open almost limitless possibilities for analysis. As a result of the experiment, we can build an interesting knowledge graph that can be an exciting experiment for exploring the capabilities of Hume.
As we are a graph-aware company, there were no second thoughts to import data into Neo4j and build a new knowledge graph in Hume. Thanks to regular and intense activity of Luanne, using the “physical protein interface,” we collected data in the Google spreadsheet. The first Orchestra pipeline we built was regularly reading the spreadsheet and ingesting data into Neo4j.
The workflow is straightforward. The timer component triggers the data pipeline periodically every 30 minutes, and Generic Enricher component fetches spreadsheet data using Opensheet API. Then a response is transformed using the Json File Processor component, and it is used as the input for creating a cipher query. Finally, every song is stored in the Neo4j database.
Very quickly, we could see our first music song graph. Our imagination has exploded at this point.
Keeping in mind that we are building a music knowledge graph, we realised it’s time to prepare for New Year’s eve and create a Spotify playlist. The Spotify API is very well documented so a new workflow was created quickly:
Neo4j is the source and the target database for workflow data processing. First, the workflow reads songs provided by participants (i.e., without Spotify’s identifier), and at the end, it enriches the songs in the graph with additional information.
Three major points of the workflow are Generic Enricher components that use Spotify API in order to complete the following tasks:
Message Transformer components make sure JSON documents processed by the pipeline are correctly transformed using Python scripting. This is one of the features that every data scientist/engineer will surely find essential. In order to improve the performance, it uses a Batch Processor that allows aggregating a stream of messages into a single message of defined size. When the job is done, and all songs are analysed, the Idle Watcher component is triggered, stopping the workflow.
Finally, we were able to build and share our first GraphAware playlist with the world, which consists of more than 360 songs and is over 24 hours long.
Are you already excited as we were? Because it’s just the beginning of the story.
Neo4j is a schema-free graph database. Therefore, it’s very easy to expand and enrich our database with data obtained from other structured data sources, thus starting the process of transforming an ordinary graph into a Knowledge Graph.
We enriched the graph with additional data:
In order to ingest such information, another set of workflows was built similarly to the previous one. Here, the advantage of having an orchestration tool really shines: with the low coding effort, we can build a robust fault-tolerant pipeline just from a UI interface.
We wanted to know how fans and the music industry recognized songs. Therefore, the next step was to supply information about the Grammy Awards.
