How AI picks the highlights from Wimbledon fairly and fast

Tennis during the Wimbledon Championships fortnight occurs on 18 courts with over 147,000 points contested. In many cases, fans watch one match at a time and catch up on the rest of the tournament by viewing preselected highlights that are generally about popular players.

Wimbledon’s digital content team seeks to serve up the best content to fans around the world who are looking for the latest news and scores. With great tennis played simultaneously across 18 courts during The Championships, it is a significant effort for video editors to watch all of these matches and create highlight videos using traditional tools. By reimagining the workflow of digital editors, Wimbledon uses IBM’s digital and AI capabilities to speed up the creation of match highlights. This leaves editors to choose which highlights to publish that fit their narrative—and more time to create additional content to serve up to fans on their digital channels.

For the 2019 Wimbledon Championships, IBM built an AI system that uses real-time clips of individual tennis scenes from 10 courts of live-produced match footage. Each scene is assigned a fair excitement score, which is a measurement of how notable this scene could be for downstream usage. Every highlight is ranked so that the most exciting points of the tournament can be discovered while minimizing the influence of player rank and crowd size. After the match, this system selects the most exciting tennis scenes and creates highlight videos, all within minutes of match completion.

In order to achieve this, IBM Watson was trained to better recognize acoustics and understand and remove inadvertent AI bias. The result is a higher-quality selection of sports highlights—and more of them.

Let’s take a closer look at what happens behind the scenes.

Video streams from Wimbledon courts are ingested and understood by machine multimedia comprehension algorithms. The computing techniques condense video from full-length tennis matches into a clipped highlight by using computer vision and sound to determine scene boundaries. The camera angle and transitions between different angles are helpful in selecting scenes. However, computer vision alone produces false positives when a player performs an unexpected action before or after the point, or if the broadcaster needs to change the viewing perspective to keep the content compelling.

The visual aspect of a tennis match, and any live sporting event, is highly-diverse with a variety of angles, variable looks, lighting, contrast and colors. While visual analysis works well, the incorporation of AI sound analysis was a natural progression. Generally, the sound in tennis is relatively stable and consistent in contrast to vision. For the 2019 Wimbledon Championships, we developed a system that detects events such as ball hits and point boundaries in a tennis match.

To classify the acoustic events from a stream of sounds in a tennis match, the audio needs to be segmented into small windows. We implemented a peak detection approach that finds the sound of interest and isolates it within a single window. When a sound peak is found, the audio is segmented over 0.5 seconds on both sides for a total duration of one second. The 1-second signals are input into an acoustic event recognition pipeline.

At the feature extraction phase, the 1-second sound clip is stratified into 20-ms frames. Mel-Frequency Cepstral Coefficients (MFCC) are extracted along with Delta Coefficients before being input into a Convolutional Neural Network (CNN). The classifier recognizes sounds such as announcer, applause, bounce, feet, racket hit, non-play noise and the line judge making a call. Overall, we maintained an F1-score of 87.76 percent over all sound recognition events. An F1 score is a measure of the accuracy of an algorithm in meeting success criteria.