Athletes show us a glimpse of promise for VR using 360 video
|Sep 14|| 1|
In my first post, I argued that nobody wants virtual reality because its hardware is overly charismatic. It exhausts your brain with a bounce house of stimuli, hijacking your perceptual system like a long night out.
But do you know who’s into that type of thing? ATHLETES. Professional athletes.
If you want to reject my hypothesis that nobody wants VR, athletes are a good place to start. They thrive in the sensory world and aren’t afraid to use bulky, specialized equipment to improve their performance. The pseudoscience of Tom Brady aside, an increasing number of athletes are using data with wearables to push the bounds of their training. Even my wife, a former collegiate swimmer and gadget skeptic, is using smart goggles right now to train for the Paralympics.
It would seem that all of the stars align to make VR an essential device for the athletic and sports industry. But there’s one major thing standing in the way: the business of selling software.
Let’s start with how I found myself Chief Technical Officer of sports startup and media darling STRIVR Labs.
The Pixel Quilt
Shortly before I started graduate school in 2014, another student Derek Belch came to my advisor Jeremy Bailenson wanting to study how VR could help better train athletes. More specifically, he wanted to use it with the Stanford Football team, where Belch was also a student coach.
Athletic Intelligence, a sort of combo of pattern recognition and cognitive ability, matters in football. Belch and Bailenson believed that using a VR device to replay patterns over and over would be a more effective training solution. Academic VR labs and athletic companies like Nike have long used data-driven movement metrics for insights. But in the coaching and recruitment world, especially football, reviewing video was still the assessment standard. VR gave us the potential to build huge, movement datasets by collecting data from the device’s tracking sensors.
Creating an interactive VR scene takes a ton of work and resources. A real-time rendering engine like Unity or Unreal must have digital 3D assets in the right places, at the right levels of detail, with the right defining properties. If you don’t have the budgets of big video game or movie studios, realism of interactive worlds is challenging. It was a challenge for us in academia for the same reasons.
If we went the rendering engine route for Belch’s training solution, it risked looking like a budget version of the video game Madden, which other companies were already attempting. See EON Reality’s video below for what existed on the market then:
To maximize behavioral benefits, we didn’t want it to feel like a cheap video game. Athletes are super perceptive, they’ll notice all the subtle movement details that might be wrong. The football player in the product video above doesn’t even hold the football! Instead of rendering a fully 3D scene, we needed something more photo-realistic.
On the side of my lab duties, I was moonlighting as a commercial photographer. Even though I knew little about football, I was adept at the world of cameras, lighting, and editing.
When I did some early need finding with Belch, I learned that football staff shoot video at every practice and game, then turn it around quickly for review. They typically use camcorders and clunky media player software like DV Sport (or the newer Hudl) to do this. This training workflow presented an additional technical challenge: creating or updating new VR content rapidly.
Enter 360-degree video, which Disney really pioneered in the 50’s. In 2014 it wasn’t all that different, you needed multiple cameras with wide angles that produced images that slightly overlapped. The combining of multiple videos into a single one was the tricky part, and took a lot of manual work called “stitching.” This process sews the overlapping edges of each video together like an elaborate pixel quilt. Finally, the image is warped so that it can project on the inside of a 3D sphere.
Here’s an example 360 image of the Stanford campus Main Quad after final stitching and warping:
I’m more social scientist than engineer, which is why I adore map projections (I have a tattoo of my favorite, hah). Have you seen a flat world map? That’s called equirectangular, it’s what 360 uses. It warps things a bit, which capture the struggle humans have in representing the 3D world in 2D ways.
There are major drawbacks to 360. While wearing a VR headset, you can look all around as if in a virtual planetarium, but you can’t move beyond where the cameras physically recorded. Meaning you’re stuck at a point in space.
Then there’s the issue of stereoscopy, which gives videos the illusion of 3D. It’s easy to do in photography, but video introduces all kinds of movement or depth disparity that strain your eyeballs. Its risks don’t usually justify its negligible realism gains. Even Oculus VR leadership Palmer Lucky and John Carmack (arguably, the hardware experts) used to joke about their inability to tell the difference between monoscopic 2D and stereoscopic 3D. Brains are weird, they fill in the gaps!
Most logistically challenging was the need to put the camera on the field to capture plays. To get a good angle, the 360 camera needed to be as close as possible to a player but just far enough away that they wouldn’t trip or collide with it.
These all seemed like okay tradeoffs to make, especially if it meant enabling sports teams to generate their own VR content libraries quickly. With a decided capture process in place, Stanford’s football coach David Shaw, granted us a single April practice to film.
It worked. Along with the help of Bailenson’s lab, I built a software wrapper that acted like a rudimentary media player. Athletes, especially the quarterback, could review plays in VR from the perspective of the field. We captured the usage data and output it to a text file to eventually summarize trends for coaches.
By the end of the Stanford Cardinal 2014 football season, our training prototype had reached enough players, collected enough data, and gathered enough attention to signal Belch to take it bigger.
Changing the Game
STRIVR Labs, Inc. (Sports TRaining in Virtual Reality) was cofounded in January 2015 by Derek Belch, Jeremy Bailenson and NFL retiree Trent Edwards. Because its early stage development came from Stanford technology (me + VHIL), they went through Stanford’s Office of Technology Licensing to get proper invention approvals. Stanford deferred on ownership and took no equity. An ideal outcome for startups emerging out of academic labs. I opted to be Chief Technology Officer (CTO) in the beginning while still remaining a graduate student, until we hit a scale where that would be unreasonable.
Much of the time spent early on in STRIVR was in sales and pipeline optimization. I built out our early patents in making the 360 video process faster and scalable. We automated the manual steps it took to offload, organize, stitch, and upload media to the cloud. One of the biggest problems we solved was predictive media organization. Camera companies like GoPro hadn’t yet released software tools that could match up clips shot at the same time. We built an algorithm that did it from a media hub and uploaded it as fast as possible to the cloud. I called the tool the dumbest thing I could come up with: Upload Blitz.
We experimented with better camera rigs that didn’t use GoPro, a company we came to loath through through their unreliable hardware and our partnership attempts. Here’s a prototype 360 rig, using rubber bands to attach Garmin cameras around a Philz coffee cup in what might be the MOST Silicon Valley thing ever:
We also experimented with making the rig less obstructive to players on the field, to avoid injuries. The software promised to reduce injuries on the practice field, so we couldn’t have it create more! Here’s early employee Jordan Kutzer adjusting a boom arm in the backyard of our new company headquarters (see: townhome) in Menlo Park:
There’s a sense of immediacy in seasonal sports which made everything URGENT. With only so many weeks in a season, every minute of practice and training is precious. Kutzer and I would stay up at night, fixing bugs and promising customers media issues would be resolved by the next morning. I lived at our office, attended class during the days, and slept little.
Things accelerated when the Dallas Cowboys (coaches in football all know each other) agreed to a trial session at their training camp in Oxnard, CA. I spent days sweating in a Cowboys trailer with players and staff to make sure everything was running smoothly. When Tony Romo, the Cowboys QB, took off the headset and said he loved it, we cheered at our first commercial sale.
To sign more NFL and NCAA team customers, we decided to use off-season or recently released athletes as our account managers. This was deliberate to gain the trust of coaching staff. They’d rather have a person that knew football first, and camerawork second. I built training and education efforts around using our camera hardware and software so that account reps on the ground could own all aspects of support. This meant teaching them how to debug camera and VR hardware issues, which were plentiful.
I highlighted these challenges during my Stanford e-Corner interview, which called STRIVR “user-centric because it has recruited actual players to help build its products”:
With VR becoming the next big thing in venture capital, STRIVR elevated in hype. Much of our PR came naturally, thanks to the culture of sports. Both Harvard Business School and Stanford Graduate School of Business would write case studies on this period in STRIVR’s early market trajectory.
As STRIVR slowly grew its sports customer base, scale limits became clear. With a fixed number of professional teams, even with league deals, growth would stagnate. So it made sense to start exploring other revenue streams for inevitable venture rounds.
One of those revenue streams was creative services.
Get Off My Field!
Madison Square Garden, the New York entertainment empire, called us with an idea. They wanted us to build a custom VR visitor experience for one of their venues. This is called an “activation” in advertising. Brands usually award these to large creative agencies like MediaMonks because they require specialized creative teams. With STRIVR’s deep expertise in sports and athletes, it gave us an advantage. When we launched the New York Rangers goalie fan experience for MSG, it opened up a new line of business we called STRIVR Studios.
I agreed to move to Los Angeles upon graduation along with my talented brother and cinematographer Kit Karutz to run our production efforts. Kit and I couldn’t find a quick place to live in LA, so we ended up renting a sailboat in Marina Del Ray, working long days and getting shaken by seals sleeping on our deck at night.
With the Hollywood vet Logan Mulvey working as our executive handling clients, Kit and I became a VR production powerhouse. We teamed up with the experienced studio Talespin, who taught me most of what I know about formal production work. Together we made commercials for Bank of America, Visa, Lowe’s, a bizarre skincare brand called Murad, and Stephen Curry’s personal brand SC30.
My favorite was an insane warm-up routine for PGA Tour Champions, which Kit and I edited from an Atlanta hotel room while taking breaks only to eat a nearby Waffle House. Don’t miss Belch getting lowwww in the background:
Productions put us in a weird position since we often promised to interfere little with athletes’ practice time. At one point while filming a commercial for Visa, Chicago Bears QB Jay Cutler looked me in the eye while I held a camera and yelled, “Get the fuck off my field!” Visa execs looking onward merely shrugged. Kit and I worked our magic and activated the experience at an NFL game in Chicago. I stole a rare media quote, claiming that STRIVR Studios can:
“Create for each fan the sense that they are intimate with these players that they support and this team that they support.”
Our production efforts were scrappy and innovative, which I loved. We competed and won bids against larger creative agencies. In few other creative career paths would I have so quickly directed Stephen Curry, worn a Rangers uniform, or took models on a Malibu beach picnic. It was even extra cool to be chosen for the prestigious Cannes Lion Startup Academy because of our creative efforts.
Athletes aren’t the saviors for bringing virtual reality mainstream. It’s not that it couldn’t benefit them. It’s more that selling software to coaches is a pain that doesn’t scale.
It’s hard to say how much of the resistance I saw over the years was the VR technology itself versus something new for a community hesitant to change. Just look at the way NFL was forced to use Microsoft Surface tablets, which leapfrogged the black and white photos they had previously! The NFL didn’t gradually ramp up their tech, they went from something completely analog to something overwhelmingly digital.
Leapfrogging happens a lot in technology. Certain communities have barriers in cost or culture that cause them to hold out until a perfect storm of things forces them to adopt.
Initially, I was stoked at the opportunity to make the sports field more quantitative. Coaches never really used the data as I had imagined it, which was not a problem unique to STRIVR. Sports are facing the same challenges with data that most private companies are encountering. More data becomes useless when it isn’t understood, or doesn’t exist in a culture that promotes it. Professional sports, for all their dollars, are a long way from systemic Moneyball.
I often wondered if our solution for sports training was a charisma machine. Something Arizona Cardinals QB Carson Palmer, an early STRIVR user, said during an interview to Sports Illustrated stuck with me over the years:
I’m archaic, and I thought, There is no way this can change the way I play quarterback. But I am all in on this. I’ll watch pressures on this during the week, and I swear I have flashbacks from a game, seeing the same pressures.
The mental flashbacks are a promising anecdote, but then he goes on to say:
The other thing that is cool about it is watching mechanics, because I can put the camera behind me. So if I’m in the pocket standing one way, I can put the camera behind me, I can have it to my right and to my left, and I can watch my feet.
According to Palmer, maybe it’s the wide, close camera angle helping visualization. I’m not convinced VR mattered as much as we thought. I think more likely, 360 video was a novelty that made watching practice tape less boring.
Software success is measured in sales, not efficacy. Customer growth matters more to investors than science. With its Seed venture round of $5 million, STRIVR pivoted to enterprise training for Fortune 500 companies, like of Walmart employees.
The pressures of venture capital force startups to find routes to grow fast. In software, those routes seek to keep costs low and replicate customer gains for huge returns. This makes sense if we’re talking about file sharing or social media startups. That’s because it’s 2020 and we’re all familiar with a web browser. It makes less sense if we’re talking about a new computing platform, which requires time and literacy to understand.
Many interesting VR startups of 2016 buckled under the slow consumer adoption and heavy weight of venture capital. What we’re left with is companies like STRIVR, Talespin, and Upskill. And remember Worldviz? They’re still around, building VR training solutions for companies like it’s 2002.
As a result, our possible future is now that we’ll use immersive devices for a long time to train for more mundane things. That doesn’t mean that I’ll want to. Safety training is mostly boring because my dumb lizard brain can’t comprehend future risks!
New technologies like VR need the space to breathe and find more unexpected use cases. More interesting is what STRIVR is doing lately, by finally using data in innovative ways for its customers. Like promoting workers to management by assessing their skills in VR.
Get used to intelligent systems using novel inputs like VR or spatial microphones to improve social dynamic outcomes. Like this product we prototyped in school, meant to diagnose teams and offer actionable ways to improve teamwork:
I think we’ll look back at the media and entertainment formats of the 20th century as these flat windows. The 360 pipeline we innovated at STRIVR gave it a scalable edge, and opened up a whole new level of photo-realistic training at a much cheaper cost. We helped open the window.
With the closing of our Seed round, I didn't want to sell enterprise software. Which is why I departed STRIVR to found my current venture Blue Trot in early 2017 and become my own worst nightmare: an artist.
Next up in Part 3, I’ll talk about the “empathy machine” so often quoted by artists and technologists. I’ll try to sort through Hollywood’s continual efforts and promises to make VR the next big thing for eyeballs everywhere, and my experiences of it through globally touring my graduate thesis and VR experience The Crystal Reef.
Special thanks to Jordan Kutzer and Piera von Glahn early edits of this post! Especially Jordan, for keeping me sane at the STRIVR HQ during those many sleepless nights. No thanks go to Jay Cutler, who I’m glad is finally retired.
The views represented above are my own, not of my company Blue Trot Group or any of its partners, affiliates, or clients. As an early founder, I also own shares in STRIVR, Inc.