Graph Algorithms: Make Election Data Great Again
Editor’s Note: This presentation was given by John Swain at GraphConnect San Francisco in October 2016.
In this presentation, learn how John Swain of Right Relevance (and Microsoft Azure) set out to analyze Twitter conversations around both Brexit and the 2016 U.S. Presidential election data using graph algorithms.
To begin, Swain discusses the role of social media influencers and debunks the common Internet trope of the Law of the Few, rechristening it as “the Law of Quite a Few.”
Swain then dives into his team’s methodology, including the OODA (observe, orient, decide, and act) loop approach borrowed from the British Navy. He also details how they built the graph for the U.S. Presidential election and how they ingested the data.
Next, Swain explains how they analyzed the election graph using graph algorithms, from PageRank and betweenness centrality to Rank (a consolidation of metrics) and community detection algorithms.
Ryan Boyd then guest presents on using graph algorithms via the APOC library of user-defined functions and user-defined procedures.
Swain then puts it all together to discuss their final analysis of the U.S. Presidential election data as well as the Brexit data.
What we’re going to be talking about today is how to use graph algorithms to effectively sort through the election noise on Twitter:
John Swain: Let’s start right off by going to October 2, 2016, the date we published our first analysis of the data we collected on Twitter conversations surrounding the U.S. Presidential Election.
On that day, the big stories were Hillary Clinton’s physical collapse and her comment about the “basket of deplorables” — which included talk about her potentially resigning from the race. It was a very crowded conversation covered intensely by the media. We wanted to demonstrate that, behind all the noise and obvious stories, there were some things contained in this data that were not quite so obvious.
We analyzed the data and created a Gephi map of the 15,000 top users. One of the clusters we identified included journalists, the most prominent of whom was Washington Post reporter David Fahrenthold. Five days later, Fahrenthold broke the story about Donald Trump being recorded saying extremely lewd comments about women.
We’re going to go over how we discovered this group of influencers which, even though there was a bit of luck involved, we hope to show that it wasn’t just a fluke and is in fact repeatable.
In this presentation, we’re going to go over the problem we set out to solve and the data we needed to solve that problem, how we processed the graph data (with Neo4j and R), and how Neo4j helped us overcome some scalability issues we encountered.
I started this as a volunteer project about two years ago with the Ebola crisis, which was a part of the Statistics Without Borders project for the United Nations. We were looking for information like the below in the Twitter conversation about Ebola to identify people who were sharing useful information:
Because there was no budget, I had to use open-source software and started with R and Neo4j Community Edition.
I quickly ran into a problem. There was a single case of Ebola that hit the United States in Dallas, which happened to coincide with the midterm elections. The Twitter conversation about Ebola got hijacked by the political right and an organization called Teacourt, all of whom suggested that President Obama was responsible for this incident and that you could catch Ebola in all kinds of weird ways.
This crowded out the rest of the conversation, and we had to find a way to get to the original information that we were seeking. I did find a solution, which we realized we could apply to other situations that were confusing, strange, or new — which pretty much described the 2016 U.S. Presidential election.
