My 25 years of interacting with the technology and its failures to go mainstream
|Jun 26|| 2|
Virtual reality (VR) isn’t what most people want. It’s often called a party trick, but party tricks ask little of their watchers. VR is more like a bounce house. Something you might want to rent and have for a social event, that’s exhausting and fun, but not something you want to be jumping in all day long. Even a summer camp where I directed, which offered bounce houses as a daily activity offering, would see enthusiasm from kids eventually wane. Or sometimes they’d break their arms.
I have an Oculus Go on my desk, a product which Facebook recently announced it was killing. I use it daily—not the device—but the box. To raise my laptop higher so I can attend video calls. This is more useful to my daily needs than anything the device can offer. Why do so many VR devices similarly continue to gather dust and fail to capture the mainstream?
In 2020, there’s a long pattern of consumer behavior over the previous 25 years that I’ll use alongside my personal experiences to ultimately show that most people don’t and won’t want VR. The reasons that VR continues to fail are that it:
Has conflicting imaginaries (Part 2)
Competes zero sum with other media options (Part 3)
Let’s start with Nintendo.
Nothing’s Free in Waterworld
At eight years old, I owned a virtual reality device, and it was terrible.
In 1995, Toys ‘R Us was my playground. My mom ran a daycare in our home, which meant that toys were central to her business and our lifestyle. We couldn’t afford a lot, but rotating through some toys a few times a year kept the toddlers from revolting.
I still remember the way Nintendo’s Virtual Boy console looked on display when I first saw it there. Behind translucent plastic and lit brightly, it looked like an artifact stolen from the future and put in a museum. My mom must have felt similar, because we brought one home. We could only afford it because it was already on sale, $50 instead of the original retail of $179.99. It came bundled with Mario’s Tennis and I chose Waterworld as an extra title. A bizarre choice given I was too young to experience the cinematic masterpiece that is still a Universal Studios show.
Nintendo’s Virtual Boy isn’t always considered true virtual reality the way that hardware engineers describe it today. Modern VR relies on a ubiquitous system of tracking and rendering. That is, tracking how a user physically moves and rendering to a display fast enough for it to feel real. Virtual Boy didn’t really do that, instead you placed it on a surface like a desk and peered into its display while holding a controller. Its 3D display was monochromatic, which means it rendered red pixels against a sea of black. Its limitations didn’t really matter, to me it still felt different than anything else at the time. Especially with Mario’s Tennis, where I could perceive the depth of the court.
I’d count the Virtual Boy as virtual reality because of how different it felt from other games at the time. It didn’t matter that there were less colors, or that there was no tracking. Peering into it still occluded the outside world. Waterworld felt like my world, a fate I don’t wish on any other child.
A mere week into my Nintendo Virtual Boy ownership, my eyes started hurting. The eye strain it created concerned my mom enough to return the device. My experience wasn’t unique, the device was a gigantic flop. It only sold 770,000 units, making it Nintendo’s worst selling console of all time.
The experience of owning a Virtual Boy was memorable, but it didn’t hugely impact my childhood. Not in the way that opening up a Nintendo 64 on Christmas or crushing on a girl or learning to swim did. I wouldn’t even think about VR again until I ended up building it myself in college.
In 2006, as an undergraduate at UC Santa Barbara, I was utterly bored. In an effort to combat the lack of inspiration my classes offered, I became a Research Assistant at the Research Center for Virtual Enviornments and Behavior (ReCVEB). My duties were to design VR worlds using Vizard, a software suite which combined Python scripting with OpenGL graphics and commercial VR hardware management. It did all the work that real-time engines like Unity and Unreal do now, but Vizard’s parent company Worldviz served customers that were mainly academic labs, not game developers.
I programmed my first VR world to do the thing I feared the most: to cross a sea of spiders. Using a simple 3D room model, I filled it with large, animated spiders triggered by user proximity. In a dark room, illuminated by a spotlight, a user could cross the room only through the spiders, which would scurry away. Also, I made the spiders HUGE, like Arachnophobia movie-ending huge. My psychology textbooks called this “stimulus threat” and it taught me what VR excels at: facing a fear. This was vastly more interesting than my Intro Psych classes.
My statistics professor, Nancy Collins, was running a study at our lab looking at social support in couples. A couple would come in, we’d separate them into different rooms, then we would put one in VR. We lied and told the subject that his or her partner was also in VR, networked from the other room. He or she would have to cross a cliff edge and we would manipulate the level of support the “virtual” partner offered in the task. As the published study describes it, “Perceptual cues were added to create a more enhanced experience. For example, rocks fell off the cliff when participants stepped on or near their location, and localized sound was added (e.g., the sound of wind swooping up from the canyon or rocks falling).”
I designed and programmed those perceptual cues. My virtual rocks (find them above) scared the shit out of people! It felt weird, to know something I created caused a real fear in people. Not to mention how pissed couples were at each other when I reunited them after the study concluded. Even though we debriefed them quickly and told them the truth about the deception, I wondered if our study was doing more relationship damage than we thought.
Without fully knowing it yet, after building those spiders and rocks, I was hooked on my very first charisma machine.
My colleague Morgan Ames’s wonderful book and body of research explores certain technology products as charisma machines. While difficult to pinpoint (she does a much better job than I), these types of products usually promise some major transformation through implementation of the technology.
Much of this lies at the hands of the charismatic leaders that push or lead design on the products themselves. Think of Elizabeth Holmes hawking Edison devices with Theranos, or Adam Neumann selling glorified real-estate at WeWork. They share a pattern of promise that confirms an ideology, often with biases and a narrow outlook on the world or its cultural nuances. Elon Musk for example, sells charisma machines and builds a cult of ideologically aligned followers only rivaled by Steve Jobs.
Charisma usually comes at a cost, and many of these pioneers mentioned employed questionable values to achieve their vision. They sometimes bring big innovations, like sending the first humans into space on a commercial rocket. But the cost is human, in the people treated poorly or exploited along the way (or killed, in Holmes’s case).
Virtual reality is the ultimate charisma machine. Both in the way leaders have promoted its salvation but also in the way the technology itself works. VR is an exploitation of the human perceptual system. To create convincing VR, we need to first understand the brain that it deceives.
All Reality is Virtual
Jim Blascovich, my social psychology professor at UCSB that ran ReCVEB, used to always joke that “all reality is virtual.” What he meant was that our brains create a construction of reality based on our experiences and best information at the time. We don’t see things as they are, instead we do a complex series of pattern recognition where our brain fills in the gaps. Think about optical illusions, which are exploits of the visual perceptual system. Vice covered some of these as explained by Facebook’s Chief Scientist Michael Abrash. Abrash, a programmer by trade, not a brain scientist, borrows decades of psychology and cognition research to point to why it matters for VR. It matters because VR is a glorified perceptual illusion.
Most VR hardware systems of the last couple decades work by tracking how a user moves their head, and then rendering an updated image quickly to a display. These displays are usually plastered to your face, awarding it glorious terms like FaceToaster or CyberGoggles or FaceComputer. Popular Science even covered the widely shared side effect of Oculus Face. Nobody looks better while wearing one of these. Especially not me, which is unfortunate given that if you Google Image search my name this comes up:
Another way to describe VR is not as a system of hardware but as a feeling. Psychologists and cognitive scientists call this “presence,” a confusing term to kind of describe how real something feels. There are categories too, like “environmental presence” and “self presence” and “social presence” all to describe the accuracy of a digital illusion.
But feelings are a tricky way to describe technology. This argument says that anytime I feel like I’m somewhere else, it counts as virtual reality. This is problematic given how frequently the word immersive is used to describe literally anything now. The act of dreaming you might also count here as the ultimate VR. At night, your mind becomes an imagery factory, churning out nightmares and fantasies as a closed loop.
It’s too frustrating to say that everything is VR because that also means that nothing is VR. We can’t just throw our hands up at philosophers and cognitive scientists. Especially when there are products that need to market and sell.
Let’s compromise and say that:
VR is the replacement of enough perceptual stimulus with digital ones to feel convincingly real
This lets us count Nintendo Virtual Boy, any Oculus product, as well as a whole range of other digital devices as VR.
After barely finishing my bachelor’s, I’d go on for the next six years to study VR and watch it transform from academic to consumer technology (again).
The Dog and Pony Show
In 2010, Jeremy Bailenson hired me to manage Stanford University’s Virtual Human Interaction Lab. I suspect it was because I had experience with Vizard from my UCSB days, and that he and Jim Bloscovich were close colleagues writing a book together. The lab at the time looked a lot like other university VR labs did, ReCVEB included, featuring a few drab windowless rooms with lots of wires and duct tape.
As part of his tenure offer, Bailenson negotiated a few million dollars from Stanford to build a more state-of-the-art laboratory space. Most of the money went to the architecture firm Kornberg Associates as well as the contractors to build the space. I project managed all of the hardware installations, learning everything from server racks to electrical diagrams to higher-order ambisonics (it’s as confusing at it sounds). We mostly just spent a lot of time on granite counter-tops, and what goes wrong with their sealing.
Bailenon’s vision for a more attractive demo space to showcase VR to visitors played a key part in its current resurgence. With VHIL up and running, using the best systems available in 2011, we went on an outreach blitz. With the undergraduate Joshua Bostick at my side, we toured thousands of people through the space over the next few years. I built a reservation system to entirely open VR to the public, something no other academic lab had attempted, not even USC’s Institute for Creative Technologies.
I absolutely loved outreach. Bringing in diverse groups of people, talking about VR, and then showing them it was incredibly rewarding. Especially a support group of elderly local widowers, who I watched flirt with each other while their minds spun around the technology’s leap from science fiction to reality.
We toured more high-profile visitors, too. Like Wired founder Kevin Kelly, NBA commissioner Adam Silver, and Prince of Spain Felipe VI. The Huffington Post wrote in 2013 after a visit to us, “And one thing is absolutely clear, Virtual Reality at this level isn’t going to stay in the labs much longer.” They even accurately captured my extreme anxiety in someone breaking our sole $45k headset every time somebody wore one.
A former Navy SEAL once shook my hand on his way out and thanked me for the Dog and Pony Show. At the time, I didn’t think of VR as a charisma machine. I didn’t have the perspective yet to wonder if that in all these tours, that our promise of a better humanity through VR was indeed confirming that it would eventually fail. The bounce house of VR was too fun, bringing people in and watching them leave exhilarated. Bailenson wrote another book that captures this weird time in deeper detail.
But no show was more important than the one that happened in 2014, when we got an email from Mark Zuckerberg.
The Room Where it Happened
When I met Mark Zuckerberg, he stuck out his hand and said, “Hi, I’m Mark,” which felt like an impressively polished move. He was shorter than I expected, as famous people sometimes are. He was accompanied by Cory Ondrejka, co-founder of Second Life and current VP of Engineering at Facebook.
Bailenson and our staff were stressing about the visit for a while. Zuckerberg’s personal security detail already did a prior walk-through of the space. I’m not sure what they were looking for, but I do remember a bodyguard counting the number of exits (we had two).
The tour went as they usually do. A playlist of files showing different worlds and research studies. We made visitors walk a plank, fly through a city, chop down a tree, and look at themselves in a virtual mirror. The tracking system was more jittery than usual, despite my extra calibration efforts. Infrared (IR) tracking, the way we did it then, sucked because of reflections. We used security cameras with IR filters that looked for IR lights, but sometimes those lights reflected off shiny things like watches or clothing. We couldn’t always fully predict how the system would perform. Today, it was not performing well.
In the demo room, Bailenson performed his most charismatic pitch, yet. I sat on the other side of the glass, at the keyboard, pulling the strings at his command. This Wizard of Oz system let us control the pace of action in demos and tours, as opposed to giving the visitor full control of their virtual world.
In this way, for a brief moment with Zuckerberg immersed in our gear, I realized my keyboard could mediate his brain. Like the rocks and spiders back at UCSB, my actions could equally make him feel threatened or elated. I could manipulate Zuckerberg’s senses, and his brain would have to work hard to convince himself it wasn’t real. Sometimes this is called the reverse suspension of disbelief. Kind of like that moment in a movie theater where you sit down and say to your brain: “I know this isn’t real, but I’m ready to be in the narrative.” VR is the opposite of that.
I was wielding a charismatic machine, and I hated the way it felt. Who was I to push the buttons of someone else’s brain?
Soon after their visit, Facebook purchased the hardware startup Oculus VR for $2.3 billion dollars, with Ondrejka leading the acquisition. In retrospect, Zuckerberg might have only visited us to do diligence that there wasn’t better VR out there than what Oculus was building at their office in Irvine.
There really wasn’t, we had an early prototype (the DK1) in our hands and it was impressively comfortable. Their next prototype (the DK2) made a few giant leaps. One was the hardware, which used a high refresh rate (75fps) with low-persistence, which means it would turn off pixels to eliminate motion blur. Another was the software, which used something called asynchronous time warp to eliminate junk tracking data and render images more optimally to the display. Mobile computing made all of the pieces cheaper, and took away the insane leverage and markup current VR manufacturers like NVIS had over the market. VR now could cost $300, not $45,000. Facebook made a hardware investment that had consumer reach.
More importantly though, I wondered if Zuckerberg bought VR to be his own charisma machine. It seemed likely. What’s cooler than Facebook in two dimensions? Three.
Religion on Contact
In the way a bounce house feels exhausting hours in, having your perceptions constantly mediated is only appealing for a select few. If cognitive load didn’t matter in entertainment, the video game market would be much bigger. Instead, we learned that most people want to watch other people play video games.
Former Oculus CTO John Carmack once described VR as “religion on contact” and I can’t imagine anything more dangerous. The internet has already proved itself an absolute mess the last few years. What happens when it becomes more persuasive?
Facebook has lately used the technology as a recruiting tool, indoctrinating executives to its revolution using the Oculus Quest. This is where charismatic machines show their largest edge, in building a fleet of evangelists to believe the same. It’s not unlike how the Trump administration has handled its recruitment and silence of non-believers. Or how Holmes built her inner circle at Theranos.
In this way, VR is overly charismatic. It will fail because there will be a rising voice of opposition that reject its persuasiveness. Already, they’re flagging the ways that the technology collects data by a company that really hasn’t deserved to collect it. What happens when that data collected is more telling than keystrokes, like with tracked movement or physiological data? Can we protect it like Bailenson suggests?
Facial recognition is rightly already up against this as its own dangerous charisma machine. Will VR be next?
Next up, in Part 2, I’ll explain how charismatic and conflicting ideologies translate to VR content creation. I’ll talk about my time building STRIVR out of a Menlo Park townhome, premiering my thesis work at the Tribeca film festival, and what to make of the promises of VR as the “ultimate empathy machine.”
Please subscribe if you’re enjoying the adventure and would like to read Part 2 fresh off the digital press. See you then!
Special thanks to Jordan Kutzer and Piera Von Glahn for early edits of this post!
The views represented above are my own, not of my company Blue Trot Group or any of its partners, affiliates, or clients.