Leading eye-tracking company Tobii has some ideas about why this is the next great leap for immersive tech.
A new wave of VR headsets is expected this year: the PlayStation VR 2, Meta’s pro-level VR headset called Project Cambria and maybe Apple’s long-rumored device. There’s one thing all of them will likely have in common: eye tracking.
The kind of consumer VR headsets you could buy for everyday use haven’t had eye tracking before. Yes, there are headsets that can use infrared cameras to track eye movement, but these have been more expensive, business-focused VR and AR headsets like the HTC Vive Pro Eye, the Pico Neo 2 Eye and the Microsoft HoloLens 2.
Eye tracking on a consumer device you can get in a store is a whole other step, and one that makes a lot of people uncomfortable.
I’ve used eye-tracking tech before and I know its benefits. You can more easily control things in VR by looking right at them instead of trying to use your hands, and it could be the gateway to better-looking VR graphics and smaller, smarter headsets. Eye tracking could also open up ways of having your VR avatar make more human-like eye contact with other avatars in future metaverse spaces.
But eye tracking also brings a ton of questions about data privacy and how big companies will manage that extra data responsibly.
There are no clear answers to these questions, but I spoke with Anand Srivatsa, CEO of Tobii, which is the largest manufacturer of eye-tracking components for VR, AR, PCs and assistive technologies. He sees eye tracking as the key steppingstone for where headsets and metaverses need to go next.
Being a cartoon avatar in a VR metaverse world right now is a forced compromise, offering more immersion in some ways while trading off actual eye contact. It can get off-putting fast. Facebook has promised that its next-gen VR headset, coming this year, will use eye and face tracking for its avatars, meaning avatars will be able to see eye to eye. Will that be better, or more uncanny?
I’ve used eye tracking for some avatar-based demos, and it does seem to make a big difference in helping it feel like someone’s actually intending to talk to me. Srivatsa sees social metaverse uses as the other big driving force for VR eye tracking right now in 2022.
“There are these two aspects. One is to fool your senses to say this is real, so you want to spend time there. And then the second thing is to go and have interaction that feels like real life, that really allows you to be immersed and not feel like this is something weird — because you still have the other parts of the awkwardness, like a headset. But if we can get really high visual fidelity, and you can get really natural interaction, then maybe that allows you to forgive that, hey, I do have something on my head.”
Srivatsa acknowledges that Tobii, as a manufacturer of the eye tracking tech, isn’t building social experiences: that will be up to companies like Meta, Sony and Apple to solve. But the potential is there, he thinks, to use eye contact as a way to focus conversations and audio, too, which would help for games, larger-scale social spaces and virtual workplace apps. Srivatsa sees scenarios where looking at a person could focus audio and even tune out others having other conversations. “I think it can be really powerful in enabling the kind of interactions we may not expect, where digital can even transcend, in some ways, what you can do in physical.”
Scott Stein/CNET
Further off, eye tracking could help bring an end to VR’s currently useful but bulky game controllers. Oculus Quest 2 controllers are fine for casual home games, but as many companies are striving to create headsets and glasses that could be worn around all the time, they’re useless.
No one’s solved a perfect smart glasses controller yet, but hand tracking (through cameras or some sort of wristband, watch, ring or gloves) plus eye tracking could maybe get to a future where controls could start feeling like a form of mind-reading. This is what Meta’s already trying to solve with its future AR glasses. Eye tracking in VR headsets could start to bridge how companies start to solve those next-wave controller challenges.
Srivatsa sees eye tracking as a way to get past the weird literal style of current hand tracking in VR, which assumes you’re able to grab things in literal arm’s reach. If the technology sees you looking hundreds of feet away, the hand tracking could “leap” there, sort of like how joysticks let us teleport quickly in VR now. It could also open up a new gesture language for VR and AR. Headsets like the HoloLens already do this to some extent, but there isn’t a consistent language yet.
“It’s like pinch and zoom. Years ago, nobody would understand what you were doing. Now, you can do that on the touchpad, and people understand that this is a gesture. There has to be a little bit of teaching and user behavior change,” Srivatsa says, but this type of use may not come into play for a while.
Eye tracking and face tracking have raised a number of serious privacy concerns about how companies handle an expanding set of user data. These concerns have existed already for smartwatches, smart speakers, video doorbells and tons of mobile apps, but VR and AR would be crossing into a whole new unexplored territory with eye tracking.
Srivatsa sees the concerns, but doesn’t necessarily have clear answers. “For us, this is super important. We have taken a very strong stance here, which is we think that users need to be in control of their data.” Srivatsa sees two levels of eye tracking data. The first is where eye control is basically used like a mouse, which he considers less concerning. “That information isn’t recorded, it’s just driving a particular interaction. For those kinds of things, we see very little impact in terms of stored data about the user.”
The other level, which involves possible recording of “heat maps” of where someone’s looking, or capturing images of people’s eyes, is the bigger unknown. “We think that, of course, that can be perceived as very invasive. We have a data transparency policy. And the intention of that is to make sure that the users have control and understand what their data is being used for if it is being stored,” Srivatsa says. “We expect our customers to be in compliance with our data transparency policy. So if they are storing eye tracking data, they have to go and get the user consent, they have to explain what they’re doing with it. And again, this is very much in the spirit of being transparent, allowing the users to opt in.”
Srivatsa says Tobii expects companies using their tech to do so responsibly and ethically. Enforcing that will be the bigger challenge. Srivatsa sees transparency about data use as the answer. I ask about secure local storage of data, but that isn’t required as part of how headsets use Tobii’s tech.
“I think there are realms of government regulation around data privacy and protection, and I think that’s sort of not where we are playing,” Srivatsa says. “We are really in the realm of data transparency. What we’re talking about is actually a little bit outside of the purview of where the governments are legislating today. We would expect, of course, that the governments in general would have some kind of requirements around biometric data that’s stored, however you collect it. What we want to be very clear about is, we know this is a new type of data that’s being collected. And that if you intend to store it, we want to make sure that users feel like there’s a huge amount of transparency and openness to include the user and the decision to use the data, or to use the technology and to understand what that data is being used for.”
That sounds like companies will be free to make their own decisions on how to use eye tracking tech. And it also means that keeping an eye, so to speak, on how Meta and others incorporate the data will be a big part of how the next wave of VR and AR play out. While Meta has already begun shifting its language on data privacy to prepare for future AR glasses, we don’t know whether those policies will change or be upheld. But it’s something that will likely be in play with your next VR headset.