The Faceboook Reality Labs have launched a new one-year blog series in which they provide step-by-step insights into the research work of the AR and VR development units. The first blog post deals with the lifelike avatars of the team in Pittsburgh. In it, the codec avatar technology is demonstrated, which should allow real-time conversations thanks to life-like alter egos in virtual climes.
Facebook Reality Labs – Lifelike avatars for AR and VR
The codec avatar technology is intended to overcome boundaries within social VR environments in the future. Lifelike avatars enable real-time communication in the virtual space, eliminating the need to make a difference between face-to-face conversation versus digital meeting. This is made possible by a 3D capture technology in combination with AI systems. To implement this technology of the future, the Pittsburgh team of the Facebook Reality Labs is working on the further development of the promising representation in the VR.
To write the responsible in their own blog:
“Codec avatars are still an active research project today, but it could radically change the way we connect to VR headsets and AR goggles tomorrow. It’s not just about cutting-edge graphics or advanced motion tracking. It’s about making the interaction with other people in virtual reality seem so natural and effortless, as if the person were standing right in front of you. The biggest challenge is creating authentic interactions in artificial environments. “
A social sense of presence must also be used to ensure realistic communication. In order to successfully implement this, the team relies on two tests to fulfill the criteria. In the so-called “ego-test”, the respective person in self-perception must be content with his digital image in order to identify with it. In the second step, the “mother test”, the alter ego must be so realistic that even the mother of the person would communicate with the avatar in a comfortable, authentic and unrestricted way, just as in real life. The alien perception of the avatar must also convince. The model for social communication within the VR is still the so-called “holodeck” known from the sci-fi series Star Trek.
The codec avatars are meant to take this goal a giant step closer to achieving the goal in the long term:
“The true potential of augmented reality and virtual reality is that it allows us to spend time with whoever we want and build meaningful relationships, no matter where people live.”
For this, even the smallest micromovements and facial expressions that make up a person have to be captured. The key to realistic implementation is the capture of all physical details and peculiarities of individuals.
To make this possible, the research team is currently on complex recording studios with numerous cameras, microphones and other hardware for encoding and decoding. While one of the studios is used alone for the pictures of the faces (see cover picture), the second is responsible for body shots.
This creates three-dimensional profiles of the respective persons, which generate enormous amounts of data. 1 GB per second is recorded per camera to capture the numerous subtle movements of the subjects. The recorded data should help AI systems to work easier and faster in the future. In the long term, however, portable VR glasses should be capable of capturing and generating realistic avatars.
The technology is far from mature and, according to those responsible, will take several more years to become suitable for the consumer market. Nevertheless, the Facebook team is currently working continuously on a possible implementation.
In addition to the technical challenges, there are also ethical complications, such as the use for deepfakes for discussion. In order to prevent such exploitation of the technology, discussions are currently being conducted on security systems through authentication for the lifelike avatars. This would personalize the digital alter egos and would only be usable by the actual owners.