First Look at Android 10’s Live Caption feature from the Google Pixel 4

Besides the next-gen Google Assistant, one of the coolest new features for Android 10 that Google showed off at I/O this year was Live Caption. It’s part of Google’s push to make Android more accessible for users with disabilities like hearing impairment, and what it does is transcribes audio that’s playing on the device into captions that float on the screen. It works offline and with most apps that play audio on your phone, including videos and podcasts. Google demoed the feature in a pre-recorded video played at I/O, but rather than making it immediately available like Live Transcribe, they’ll be rolling it out to “select phones running Android Q later this year.” Thanks to an unreleased APK we obtained from the Pixel 4 XL, we managed to get Live Caption working ahead of the feature’s general availability. Here’s our first look at the new feature.

Thanks to Nextrift, we got our hands on a treasure trove of APKs from the Pixel 4, including the Motion Sense APK and the new live wallpapers. One of the APKs they also sent us is an unreleased version of “Device Personalization Services.” This app is responsible for showing app shortcuts in the recent apps overview and Pixel Launcher app drawer, and it’s also what allows Pixel owners to select text and images in the recent apps overview. Version 1.6.255525616 of the app is technically older than what’s publicly available—the latest on APKMirror is version 2.0.268948072—but it’s never been publicly released. With this APK installed on my Pixel 2 XL and some other tweaks, I was able to get Live Caption to show up in Android 10’s settings.

As you can see in the first screenshot on the left shown below, the main settings page for Live Caption has 3 options. The first option is to change the language, though it’s currently limited to English since that’s the only supported language at launch. There aren’t any hints about what other languages might be supported in the future, sadly. Next, we can see a toggle to mask profanity. This will replace curse words with asterisks in the caption. Finally, there’s an option to show a Live Caption toggle in the volume panel. Besides these three toggles, there’s also a short video demo of the feature and text on the bottom explaining how it works.

You’ll see a dialog appear when the feature detects audio is being played for the first time. You can stop captioning by swiping down the notification panel, expanding the “Live Caption” notification, then tapping on the “stop” button. You can also drag the caption down to dismiss it like you can with picture-in-picture windows or bubbles.

By default, the captions appear in the middle of the screen in portrait mode and slightly above the bottom in landscape mode. You can tap and drag the caption up or down to reposition it, or you can double tap the caption to expand it.

I tested the feature in YouTube, Google Podcasts, Google Photos, Amazon Prime Video, and Netflix, and it worked in all these apps. It was accurate enough for me to understand what was being said without having to turn the volume up, but I don’t want to fully review or test the feature until it’s officially released.

Live Caption doesn’t work in phone calls, voice calls, or video calls because of a limitation in the API that is used by this feature. This same API is what allows for internal audio to be recorded in Android 10, though since Device Personalization Services is a system app, it has fewer limitations than screen recording apps.

Here’s a screen recording showing off the feature working on my Pixel 2 XL. As you can see, it generally captures what TK is saying in his OnePlus 7T unboxing video, though there are a few errors here and there.

Google says Live Caption will be available for “select, higher-end devices” due to memory and space constraints. The company planned to release a list of devices that will support the feature as we got close to the stable Android 10 release, but that obviously never happened. I’m betting that Google will debut the feature on the Pixel 4, but it’s strange that they’ve been advertising it on the official webpage for Android 10 when no devices have the feature yet. Once the feature does roll out for users, we’ll let you know.

The post First Look at Android 10’s Live Caption feature from the Google Pixel 4 appeared first on xda-developers.