NFL Super Bowl 2017 was recently telecast live on Fox Sports app, wherein the best moments of the game could be viewed for free using Google Cardboard or Gear VR. However, the full live-coverage of the match was unavailable due to technical constraints and exclusive copyrights being sold to the Fox Sports TV channel.
Every year, it has been a missed opportunity for all avid Super Bowl fans out there, as they do not have full match live-streaming access on their VR headsets. Here at IBTimes SG we try to explain the future of AR/VR tech in an exclusive interview with Todd Richmond, the esteemed member of Institute of Electrical and Electronics Engineers (IEEE).
Todd who is also the 'Director of the Mixed Reality Lab/Studio' at USC Institute for Creative Technologies, has whole heartedly accepted our request to discuss the future of AR/VR tech in sports and explain how Super Bowl is "primed for spectator change" in future wherein anybody can watch the game from the comfort of their couch, anywhere in the stadium.
In his brief email interview with us, Todd has clearly explained why we can't live-stream football games through VR headsets yet and the technical challenges that have seized this opportunity till date. Furthermore, the IEEE expert in AR/VR tech also takes us to a whole new level of detail, describing the impacts of VR/AR on sports and its pros and cons.
Here are the excerpts of my interview (Q&A) with the key member of IEEE, who has hands-on experience studying emerging disruptive technologies and immersive visualisation for decades:
1. Why can't we live stream games through VR headsets yet and what are the technical/sociological challenges to marry content with tech?
- It really depends on what experience you're trying to present to the user. If the goal is to provide a 360-degree video feed where the user can choose where to look, we can stream that now. If it is over the network, there will be some latency, but as long as you're not trying to coordinate with other actions or events, the user shouldn't notice (assuming that content is cached to avoid dropouts, etc).
- If you want to include more interactivity than choice of where to look, things get complicated, especially if you have multiple users. The biggest challenge is trying to sync different streams of information. This can be a challenge for a single user, and gets exponentially harder for multiple users in shared experiences.
- From a technical standpoint, we need to get resolution of the displays higher, as we're used to very good screen displays (4K TVs, etc). We also need to get VR headsets fully wireless, though that is starting to show up. The technology is somewhat the easy part – mostly we know what we want and what "better" looks like (wireless, low latency, high resolution, etc)
- From a content and social standpoint, there are a lot of very hard problems where we don't necessarily know what "better" looks like. The idea of 50 people sitting in a bar with headsets is one that likely won't get traction. The concept of 50 people sitting at home but together in a virtual bar where they can see the game and each other likely will get traction and is a goal. In this case the issues around synchronization and latency rear their head.
- More broadly there are questions around what the "social" experience looks and feels like. Do we replicate the real-time physical world, or do we create unreal worlds and enable actions that you can't do in the physical world – levitate, stop time, become giant-sized or ant-sized, etc. Those are open questions that will require a lot of experimentation.
- And that experimentation will require not just technologists but artists, writers, social scientists, etc. Also critical to these efforts are organizations like IEEE which help set standards to ensure interoperability and availability. In addition, there are myriad ethical issues that arise from mixed reality experiences, and IEEE is active in those areas as well.
2. What are the impacts of VR/AR on sports, the pros and cons, if you will; how will tech revolutionize games and the fan experience?
- You can look at it from two different perspectives – how will mixed reality (AR/VR) impact the sport for players, coaches, etc, and how will it impact the viewer's experience.
- For the teams, there are a lot of possibilities, and there is already exploration. SportVision, the company that pioneered the "virtual first down marker" for televised football as well as the virtual strike zone for baseball and other broadcast AR capabilities has worked with teams who are interested in using big data to understand player performance.
- The ability to track all players' location and other data during practice and games provides an opportunity to better understand team strategy and management of players. At this point AR and VR are just beginning to be explored for player and coaching use. But the combination of big data (tracking player performance – location, outcomes, etc, as well as biometrics) with AR/VR visualization will become part of the standard coaching tool set, just as video has.
- For viewers, it is a more complicated situation. There still isn't a compelling example of why a viewer would want 360 video other than for brief periods to get an "experience" of what it is like to be on the playing field, or on the diving platform, etc. Fast motion is usually not a good idea in VR, especially when the user has no control over it – it leads to motion sickness. So the idea of a 360 camera mounted to a player isn't really something that you want to immerse yourself in for long periods of time.
- We currently have very well done sport broadcasts for the TV screen, and some good "2nd screen" experiences on tablets. The director curates the view from a variety of cameras, and the story of the game plays out usually in a well-orchestrated manner. The combination of camera views, narration, and graphics helps to tell the story of the action, and those decisions are made by people who are knowledgeable in the sport, so they are able to explain what is going on. The 2nd screen experiences where a user is able to pick camera angles (e.g. Formula 1 or NASCAR apps) and get real-time telemetry data are great for highly knowledgeable viewers. This often small population knows what they want to see and when they want to see it. So, providing them control over the content makes sense.
- For viewers that do not know the particular sport inside/out, giving them control will likely lower the quality of their experience. Effective use of choice (user agency) requires knowledge as sports are often fast moving and a viewer needs to anticipate what is happening to be able to follow the action. That is the strength of current broadcast approaches – the director knows the sport and can anticipate what will happen next, having the coverage ready to present the action.
- I can imagine a future where there are different levels of sports broadcast. One would be tailored for people who don't know the sport (e.g. most Super Bowl viewers – they are often not big football fans). In that case the camera views and overlay information are curated by an expert. This could be a combination of flat screen and immersive tech.
- A second category could be the "typical fan" – someone who follows the sport but doesn't know a lot of the details or strategies. In this case the narrative would be less on obvious information and more trying to expose the underlying decisions and strategies to them, so they can become more sport savvy. Then at the top level you have viewer with deep knowledge, and in that case they may be able to completely curate the feeds (cameras, sensors, big data analysis) for their experience.
3. What other sensory technologies, either visual or audio or otherwise, that will heighten engagement in the future?
- The big two sensory areas will be audio and haptics. In addition, "big data" driven by biosensors on the players will be another area. Spatial audio is beginning to become easier to capture and resent, and this helps significantly with the end user feeling immersed in the action. The proliferation of body mics and microphone arrays will help to create truly immersive aural environments. Sound plays a big part in human sense-making, and can be used to heighten drama and excitement with viewers. There still are technical issues to be sorted through, but the use of spatial audio will be part of the landscape in the near future.
- Haptics is a harder technological reach. Gloves, controllers, and other devices have been used with limited success. Today we can provide certain types of haptic feedback, but making that seamless and meaningful to the user is still years away. We also don't necessarily know how haptics would play into something like a sports experience. For doing virtual tasks, we know that haptics is often important (grabbing objects, pushing buttons, etc). But for viewing a sport, beyond feeling physical contact (which may not be desirable), there aren't a lot of obvious direct correlations with the play. There could however be haptic feedback used in the user interfaces for the sports experiences that can heighten engagement.
- Biosensors and big data will likely be a significant part of the user experience. We're already seeing big data play out in fantasy sports leagues. Players are currently using biosensors for performance measure in practice, with the data used by the teams. There are emerging questions around who owns this data and who gets to see it.
- How that type of information will extend to the viewing public is an open question, but one could imagine pulse, forces, etc being shown. Formula 1 broadcast currently do this type of data display, showing G forces on the driver as they go through turns. These types of data may help a non-knowledgeable viewer engage the sport as everyone understands being tired or feeling the force of a car going around a turn. These experiential touch points can create a way for new viewers to engage the story of the event, and then induce them to learn more.
4. How does virtual-reality technology compare to other game-viewing changes through time? (e.g., TV's, flat screens, HDTV, LED TV, 1080p, Curved etc.)
- Completely new medium as the screen has no borders. This essentially breaks 100+ years of film theory and practice creating narratives on screens. We really are in the middle of rethinking what determines "viewing" and how you craft content and experiences in immersive devices.
- Giving the user agency (e.g. they can chose where to look) complicates story-telling, and even sporting events have many inherent narratives. We have been experiencing with techniques around controlling and diverting attention (which isn't ideal as you're not using the immersive medium for its strengths), and also with controlling time and space. We can use pause and rewind with DVRs and related TV capabilities now, and are starting to see isolation and spatial manipulation in broadcasts.
- These types of abilities can translate to AR/VR giving the user complete control over the experience, but then we're back to the problem of knowing if the user actually wants that responsibility. Many viewers want curated content and want the story laid out for them – essentially they desire to be a passive viewer. Others want some level of interaction and control, but at some point will want to "reset" and get back to the action and a level of understanding. We don't know how to craft that type of experience yet, and it will require advances in Artificial Intelligence and deep learning. Essentially we'll need algorithms to keep track of what is going on, what the user is doing, then be able to guide the user back to the main "story" when desired.
- AR/VR/MxR (Mixed Reality – which is where this is all heading) is a new medium for communication and collaboration. And like any new medium it will require time and experimentation to get right. Imagine back to the early days of film or television. The content was not particularly compelling, and was largely a "port" of existing content from another medium (e.g. filming stage plays). The breakthroughs come through experimentation and a desire to deeply understand the medium, the messages, and the viewer/user.
- For instance, editing helped move film from a curiosity to an art form. Those types of conceptual breakthroughs have not yet happened for AR/VR/MxR. But now that the prices of the hardware and software are approachable for a wider audience, and the tools for creation are improving, we will hopefully begin to see interesting experiments turn into new standards. This is one reason we have both a lab and a studio – we experiment with the new technology (inventing some of our own along the way), work with students at the School for Cinematic arts to explore crafting new experiences, then back at the lab can perform rigorous user studies to understand "why" something works or fails. This combination of vigour and rigor is key to our approach, and helps us deeply understand not just what AR/VR/MxR experiences can be made, but how they can be applied and why anyone should care.
