Google Pixel 2 and Pixel 2 XL offer the best smartphone camera experience in the premium segment right now. What sets them apart from other premium devices is the portrait mode feature, which uses only one lens and relies on machine learning and artificial intelligence to blur the background.
Despite having single lens, Google Pixel 2 produces among the best portrait shots, and it does so consistently. Now, Google is sharing some of the secrets behind that dramatic result with the rest of the tech world. The search giant has decided to open source the code for DeepLab-v3+, an AI-based image segmentation technology that helps Pixel 2 separate the foreground and the background.
In a blog post, Google Research engineers explain how semantic image segmentation assigns a semantic label to every pixel in an image, and train the neural network to detect the outlines of foreground objects. This process helps the image sensor classify objects, which the user cares about and the one you might ignore.
“We hope that publicly sharing our system with the community will make it easier for other groups in academia and industry to reproduce and further improve upon state-of-art systems, train models on new datasets, and envision new applications for this technology,” Liang-Chieh Chen and Yukun Zhu, Software Engineers, Google Research write in their blog post.
The open-source nature of the new segmentation algorithm does not mean that new phones or camera apps will match the performance of Pixel 2’s portrait mode. Google, in fact, clarifies that this isn’t the exact technology powering Pixel 2’s portrait mode, but the underlying technology principle can reproduce results similar to those of the Pixel 2.
Google actually wants both academics and industry people to further improve this technology, and develop new use cases not yet discovered by the search company. The segmentation technology could be used for various applications like object detection, which are a core utility in the areas of autonomous vehicles or for something more simpler like spotting boundaries. Smartphone brands have built their own AI technologies to mimic Pixel 2’s portrait mode, and it needs to be seen whether any of the tech companies bring this feature to their phone cameras.