Pixel Codec Avatars: One Step Closer To The Singularity?
Pixel Codec Avatars: One Step Closer To The Singularity?
Biometric scans: heart rate, retina scans, skin temperature, brain waves. How much of your private info are you willing to sacrifice in the name of communication, productivity and/or convenience?
Mark Zuckerberg, the individual behind the transformation of Facebook into Meta in October 2021, participated in his third interview with Lex Fridman, marking his inaugural conversation within the realm often referred to as the "Metaverse."
At the start of the interview, Fridman is clearly impressed by the lifelike quality of the virtual reality (VR) interview environment. Both Fridman and Zuckerberg willingly underwent extensive face scanning procedures to create the remarkably realistic avatars that were showcased during their session at Meta's Pittsburgh Reality Lab. This research project, known as Pixel Codec Avatars, is currently being developed by Meta.
"Codec Avatar allows us to have ultra-realistic copies of ourselves in this mixed reality." - Lex Fridman
Zuckerberg explains:
“The idea is that instead of our avatars being cartoony, and instead of actually transmitting a video, what it does is we’ve sort of scanned ourselves and a lot of different expressions, and we’ve built a computer model of sort of each of our faces and bodies and the different expressions that we make and collapse that into a Codec that then when you have the headset on your head, it sees your face, it sees your expression, and it can basically send an encoded version of what you’re supposed to look like over the wire. So, in addition to being photorealistic, it's also actually much more bandwidth efficient than transmitting a full video or especially a 3D immersive video of a whole scene like this.”
2021 Pixel Codec Avatars Supplemental Material via Meta/Facebook Reality Labs:
Zuckerberg further explains the company would need to “kind of over-collect expressions when we’re doing the scanning because we haven’t figured out how much we can reduce that down to a really streamlined process and extrapolate from the scans that have already been done.”
He goes on to describe one of the goals of the project is to be able to take a very quick scan with your cell phone. “Where you just take your phone, kind of wave it in front of your face for a couple of minutes, you know, say a few sentences, make a bunch of expressions, but overall, have the whole process just be two to three minutes and then produce something that’s of the quality of what we have right now.” In the interview with the pair, Zuckerberg said it took hours at the scan facility to produce the quality seen on the YouTube video.
In the future, Zuckerberg says Meta will not only be used in video calls but Codec Avatars will be seamlessly integrated into meetings where people are physically in the room but also have other participants show up in a photorealistic form, superimposed on the physical environment itself. Essentially, live holograms.
“You know, maybe by the end of the decade we’ll be living in a world where there are kinda as many holograms when you walk into a room as there are physical objects,” Zuckerberg says.
“A lot of people have this phrase where they call the physical world the real world. And you know, I kind of think increasingly, yeah, the physical world is super important, but I actually think the real world is the combination of the physical world and the digital worlds coming together. But until this technology they were sort of separate, right? It’s like you access the digital world through a screen, right? And you know, maybe it’s a small screen that you carry around or it’s a bigger screen when you sit down at your desk and you know, strap in for a long session. But they’re kind of fundamentally divorced and disconnected. And I think part of what this technology is gonna do is bring those together into a single coherent experience of what the modern real world is, which is, it’s gotta be physical because we’re physical beings. So the physical world is always gonna be super important, but increasingly, I think a lot of the things that we kind of think of can be digital holograms. I mean, any screen that you have can be a hologram. You know, any media in any book, art, you know, can basically be just as effective as a hologram, as a physical object. So I think that it’s actually an amazing thought experiment of like, how many physical things we have today that could actually be better as interactive holograms.”
Zuckerberg goes a step further than just the thought of physical reality and holograms blending together when it comes to humans, he describes meeting spaces where you have a fusion of both, but also, “Bob the AI, who’s an engineer on your team who’s helping with building things, and he can now be embodied as a realistic avatar as well, and just join the meeting in that way.”
META AI
In the Metaverse, you'll not only have the opportunity to engage with others through your personal digital twin, but you'll also soon have the capability to generate an artificial intelligence representation of yourself. Meta AI has already developed several celebrity AIs.
“We partnered with cultural icons and influencers to play and embody some of these AIs. They’ll each have profiles on Instagram and Facebook, so you can explore what they’re all about,” reads the Meta AI website.
“We’re going to start rolling these out in beta in the United States today. We’ll add new characters in the coming weeks played by Bear Grylls, Chloe Kim, and Josh Richards among others.”
Meta AI recently introduced the Artificial Intelligence (AI) studio which they say will be available for both coders and non-coders to build their own AIs. They assert that this will facilitate improved communication, not just among consumers, but also among businesses that opt to employ AI for customer engagement.
Content creators are also said to find great benefits in creating AIs of themselves to interact with followers, but where does that end?
As AI strives to emulate human behavior perfectly, it lacks emotions and at times could voice opinions or statements that diverge from the user’s own beliefs, leaving you with limited control over your AI avatar's expressions. Dangerous ground indeed.
Metaverse Immersive Products
Quest 3 will be the first mixed reality product from Meta that will give users the experience of being in a physical environment from a separate remote location. The overlay will also allow immersive experiences in live sports, gameplay, and different meeting spaces.
To attain a fully immersive experience, Quest 3 necessitates the incorporation of multiple exterior cameras on the headset to precisely capture the 3D environment. Therefore, in addition to gathering biometric data during the initial face/body scan, you are now also required to grant them access to the physical environment where you use the headset, allowing them to scan for objects and create a mapped representation from the user's viewpoint.
Meta promises users can “connect with people when you aren’t physically in the same place and get us even closer to that feeling of being together in person.”
“The metaverse will include familiar 2D experiences, as well as ones projected into the physical world and fully immersive 3D ones too,” the site says. “You’ll be able to access the metaverse on different devices like your phone or computer, as well as virtual reality devices where you’re fully immersed.”
In order to tap into the immersive experience of Meta, there will be some sacrifices. You will first need to have access to an internet connection, a device to connect to (phone, tablet, PC), headphones/earbuds, purchase a VR headset, and possibly even some cryptocurrency or NFTs since some metaverses require these to interact in the space.
Ray-Ban has also joined Meta with the production of smart glasses that users can wear to not only interact with spaces within the Metaverse like games or meetings but also live-stream straight from the user’s viewpoint. In addition, the glasses allow the user to take pictures and record video, make and answer calls, and send and receive WhatsApp, Messenger, and SMS text messages without picking up their phone.
The logical evolution of smart glasses would be biological implants, particularly within the retina or brain (Ahhem, Neurolink?) to where wearables are integrated into the human body and the user is now indeed, fully immersed in the experience. Connected at a level that comes with every bit as many problems as it does conveniences, from government overreach to who has access to your biometric information.
Biometrics
Fridman makes the comment, “This feels like just an incredible transition to something else, a different kind of communication. [It] breaks all barriers, like geographical physical barriers.” The program itself also breaks another barrier, into biometrics.
Biometrics are measurements including retinal, iris, palmprints, brain waves, vein pattern recognition, fingerprints and finger geometry (position and size of fingers), voice recognition, DNA, heart rates, facial recognition, and digital signature.
The collection of biometrics for identification purposes is a well-established concept, but the advancement of identification verification has seen a rapid acceleration in recent decades. Many individuals commonly associate biometrics with fingerprinting or DNA samples. However, in recent times, identity verification has evolved to encompass technologies such as retina scans, hand geometry analysis, and full face recognition devices. These innovations aim to revolutionize our world by providing convenient and rapid verification methods.
Fridman brings up “hand tracking,” used for gesture recognition, to which Zuckerberg says, “Yeah, we’ve been able to get way further on hand recognition in a shorter period of time than expected.”
However, what is being done with our personal data is a different story.
I give a quick rundown on how LifeLog came into existence, and why its brief run seemed to perfectly coincide with the release of Facebook, down to the day here:
DARPA’s 2003 HUMAN ID or HUMAN IDENTIFICATION at a DISTANCE (HID) vs. Meta Connect
According to DARPA’s Report to Congress regarding the Terrorism Information Awareness Program on May 20, 2003:
“The HumanID Program seeks to develop automated, multimodal biometric technologies with the capability to detect, recognize, and identify humans at a distance. DARPA believes that automated biometric recognition technologies could provide critical early warning support against terrorist, criminal, and other human-based threats. Obtaining this information may prevent or decrease the success rate of such attacks and provide more secure force protection of DoD operational facilities and installations. The HumanID Program seeks to develop a variety of individual biometric identification technologies capable of identifying humans at great distances in DoD operational environments. Once these individual technologies are developed, HumanID will develop methods for firing these technologies into an advanced human identification system. This system will be capable of multimodal fusion using different biometric techniques with a focus on body parts identification, face identification, and human kinematics. Biometric signatures will be acquired from various collection sensors including video, infrared, and multispectral sensors. These sensors will be networked to allow for complete coverage of large facilities. The goal of this program is to identify humans as unique individuals (not necessarily by name) at a distance, at any time of the day or night, during all weather conditions, with noncooperative subjects, possibly disguised and alone or in groups.”
Meta users must agree to facial scans, for now, but in time they will require full body scans in order to recreate their digital avatar in the program. The Pixel Codec Avatars create photorealistic avatars describing collecting the user’s data points of the face testing on Oculus Quest 2 VR headsets in a 2020 paper: “We capture multiview video data for each identity using two face capture systems: Subject 1-4 are captured with 40 cameras with 50mm focal length, while Subject 5 and 6 are captured with 56 cameras at 35mm focal length. We use images of size 2048×1334 for training and testing.”
“The data of each identity consists of expressions, range of facial motion, and reading sentences. We randomly select expressions and sentence readings as testing data, leading to ∼12K frames for training and ∼1K frames for testing per identity. The total number of images is roughly the number of frames multiplied by the number of cameras.”
Former President Obama stated in 2015, “It’s important to recognize that you can’t have 100% security, and also have 100% privacy and zero inconvenience.”
Considering the fact that Facebook/Meta collects very personal information from users who voluntarily give their most private details up in order to use the platform, and then proceed to post images, videos, etc., that is collected and stored is something that can’t be overlooked when comparing the purpose and intent behind LifeLog.
While citizens are up in arms over data being collected by military-developed programs, they will willingly share their most personal and private information with social media companies run by private corporations.
Julian Assange once told RT, “Facebook in particular is the most appalling spying machine that has ever been invented.”
The question remains, just how much of these private corporations have contracts with the government? How much of the information provided by users is being used not only in marketing but also to predict perceived threats and applied in social engineering programs?
If you have enough data, it all becomes predictive to an extent, with only a small margin of error, left up to chance of course. When everything is being tracked from keystrokes and web searches to biometric data being collected in real-time, users willingly give up their expectation of privacy all in the name of convenience, safety, and connectivity.
With new technologies promising to make your life and work easier, your relationships easier to sustain requiring so much of your private information, at what point do we all agree intrusiveness does not equal convenience or safety?
What does this kind of technology, and future technology mean for the human experience and the experience of consciousness? It would be ridiculous to not assume both would be wildly altered.
The internet and social media were sold to us as a way to connect with others and to strengthen relationships across vast distances. Many would say we’ve achieved the exact opposite. People are more isolated than ever, with social media fueling little more than ego.
Dayz of Noah goes far deeper on the subject of the Singulation Theory and Singulation Hypothesis and directly ties it to how Codec Avatars, the Metaverse and Web 3 (among other upcoming tech experiences) are going to be used to normalize the blend between virtual reality and that which we know as the physical realm here:
We’re literally planning our own demise. Only humans seems hellbent on eradicating on eradicating one another.
Making our life easier, therein lies the very problem. We’ve become the fish chasing a bright shiny lure only to find we’ve been hooked for a far more nefarious purpose. When we lose touch with our reality, then we lose touch with humanity.