To make poorly labeled videos easier to discover, Manhattan-based video analysis startup Dextro is launching a platform that analyzes and tags the contents of publicly available videos, using algorithms to identify common scenes, objects, and speech. Mic, a news site aimed at millennials, has partnered with Dextro and will use the platform, called Sight, Sound & Motion (SSM), to discover newsworthy videos that may otherwise be difficult to find.
"Many of the biggest news stories over the past year had their origin in citizen video, but the sheer volume of user-generated video makes it hard for human eyes to isolate those that are newsworthy," Marcus Moretti, project manager of data at Mic, said in a statement. "Dextro's SSM platform allows us to parse the deluge of video into a stream of relevant, important video that can supplement or generate coverage for Mic."
The SSM platform relies on algorithms that recognize patterns associated with particular activities and events. Dextro founder and CEO David Luan told Fast Company the technology could easily identify videos of the Pope speaking in America this week by detecting scenes of crowds of onlookers, images of Pope Francis himself, and spoken words associated with him.
"For the first time, for every video, we can pick out not just what’s on screen, but also the sound and motion, and put it into one major model of what’s happening in the video," Luan said. "This takes away all of the need for human-generated metadata, which is incredibly noisy and not that indicative of what the content is."
On Thursday, Dextro published a website that allows visitors to see how the SSM platform analyzes videos recently uploaded to Twitter. The backend of the site, available only to staff at Mic, allows reporters to search for social videos by term, such as "President Obama" or "refugees." (See images in the slideshow above.) Luan said Dextro currently has no plans to expand the SSM dashboard to other news organizations.
In May, Dextro used its video analysis technology to scan and categorize all public Periscope streams in real time. The company's main business is helping clients who own large amounts of video content to index and search their archives, along with educating brands on how they are mentioned in social media videos. With the launch of SSM, the company is for the first time incorporating audio recognition into its video analysis platform.
