YouTube is using AI and machine learning to replace video background

Google is using its AI algorithm to replace video background without green screen within YouTube Stories.

  • Published: March 7, 2018 3:17 PM IST

Google is bringing a new video-editing technology that aims to eliminate the green screen background, one of the most common tools in video industry. The technology dubbed as mobile real-time video segmentation uses artificial intelligence and machine learning to let users replace the background on their video in real-time.

Google has already begun testing the technology within YouTube and it works as simple as applying filters to a still photo. In a research blog, Google goes into great length to explain how the technology works. “Our new segmentation technology allows creators to replace and modify the background, effortlessly increasing videos’ production value without specialized equipment,” Valentin Bazarevsky and Andrei Tkachenka, Software Engineers, Google Research explain in the blog.

The feature is being beta tested via YouTube app with a limited set of content creators. The background replacement feature primarily applies to YouTube’s new “Stories” video format, which is similar to the disappearing stories on the Snapchat and Instagram. Google says the feature works by using a semantic segmentation task using convolutional neural networks.

It also notes that in order to make the machine learning algorithm work seamlessly on mobile processing device, it provided high-quality data of tens of thousands of annotated images. The machine learning pipeline is also tasked with the computation of binary mask separating foreground from the background for every input frame of the video.

The training process further involves feeding the algorithm with the annotated ground truth of each photo in several ways and then use it as a previous frame mask. The idea here is to offer an empty mask that acts as a previous mask for the transformed ground truth mask leading to a transformed image that offers proper segmentation.

Since the data set is limited at this time, the effect might not be as perfect as the one achieved with manual process. However, Google notes that the limited rollout only involves testing the technology on an initial set of effects. “As we improve and expand our segmentation technology to more labels, we plan to integrate it into Google’s broader Augmented Reality services.”

Google already uses AI and machine learning algorithm to blur the background of photos captured on its Pixel 2 smartphones. The technology has only gotten better over time and this video-editing feature is likely to get better as more users start feeding the algorithm with their video samples.

  • Published Date: March 7, 2018 3:17 PM IST