Nvidia Makes Its 3D AI Voice Animation Tech Open Source

Nvidia has made its AI-powered 3D lip syncing animation software open source, according to a press release from the company on Friday. The animation model, named 'Audio2Face' is designed to generate 3D facial animation models with "human-like" expressions.

"Open sourcing technology allows developers, students, and researchers to learn from and build upon state-of-the-art code. This creates a feedback loop where the community can add new features and optimize the technology for diverse use cases," the release said.

It works by analyzing acoustic elements in audio such as the intonation of a voice to create a stream of animation data.

The data is then mapped to a character’s facial "poses" or expressions. The software can source audio tracks already saved on the users' devices for this or also stream audio in real time, having the model interpret the nuances of the voice track and emote them as it is being streamed.

"This data can be rendered offline for pre-scripted content or also be streamed in real-time for dynamic, AI-driven characters, providing accurate lip-sync and emotional expressions," the release said.

The software has been primarily used in video game development and in the customer service industry. Some of the clients that the company listed as its customers were NetEase (developer of Marvel Rivals), Streamlabs, UneeQ Digital Humans, Codemasters and more.

The company's software was used by Reallusion, which is a firm that facilitates users to create their own 3D character models. The firm told Nvidia that they used the software to create "expressive multilingual facial animations."

It was also used by Survios, the developers of Alien: Rogue Incursion Evolved Edition to fast rack their character models' facial animations and lip syncing, according to the release.

Farm51, creators of Chernobylite 2 also spoke glowingly of the technology saying that it helped them implement ideas they thought were not possible into their game.