Virtual and Augmented Reality

The Next Major Computing and Communication Platform

Tushar Nandwana
March 17, 2016

Executive Summary

Star Trek: The Next Generation introduced us to the holodeck. This was the ultimate augmented reality where Captain Piccard or other Starfleet members could seamlessly interact with virtual characters, locations and activities without specialized visual gear or computing hardware. The holodeck is the classic embodiment of science fiction technology. We may be nowhere near the amazing technology that powered the holodeck, but we are definitely taking steps, albeit baby steps, in that direction.

Today, the field of virtual reality (VR) and augmented reality (AR) is gaining steam with a multitude of companies designing and developing products to take us to the next level. This has been fueled by advances in technology such as increased hardware processing power, better software algorithms and displays that are sharper, lighter and more power efficient, coupled with more affordable technology. Facebook’s Mark Zuckerberg stated that VR is “the next major computing and communication platform."1According to a Digi-Capital report, VR and AR combined will generate $150 billion in revenue by 2020. $120 billion will come from the sale of AR devices and services such as hardware, commerce, data, voice services, film and TV, with the other $30 billion from VR hardware devices and gaming.2 

 

What are AR & VR and its History?

It is difficult to accurately pinpoint when and where the concept of VR and AR originated. In 1945, Vannevar Bush, a Manhattan Project engineer, “outlined his vision for a head-mounted camera attached to ‘a pair of ordinary glasses’ that would record comments, photographs, and data from scientific experiments…”3 Fast-forward to 1965, when Ivan Sutherland wrote his seminal essay on ARand 1968, when he co-developed the first AR head-mounted display called the “Sword of Damocles” at Harvard University.5

VR and AR consist of sophisticated software and hardware. Hardware includes processors, a wide array of sensors and high resolution displays or micro-projectors to generate a realistic simulation. 

  • VR – A closed and fully immersive system that puts the user inside of the virtual world – whether a world of a head-mounted display or inside a large simulator (e.g. aircraft training simulator or the more elaborate video games). Closed means that the user is confined to operate and play within the virtual environment. Technological advances have pushed this world from 2-D to 3-D through the use of stereoscopic screens to fool the brain into visualizing a 3-D environment.
  • AR – An open and partly immersive environment that places virtual things into the user’s real world. It is open in that the user is no longer confined within the virtual environment. Through the use of sophisticated tracking systems, accelerometers, gyroscopes and software, 3D holograms are embedded onto the user’s physical world. This “allows the user to see the real world, with virtual objects superimposed upon or composited with the real world."6 Ronald Azuma, a pioneer in the field of AR and currently working at Intel defines AR application as “one that (1) combines the real world with the virtual world, (2) is interactive and in real-time, and (3) is registered in three dimensions.”7

    In AR, a user can physically maneuver around his physical environment while simultaneously interacting with the virtual world. AR can be thought of as the next step in the evolution of VR.

Why does this concern us?

Many companies either exited or stayed out of the initial VR/AR push several years ago following the ill-fated Google Glass launch. That product’s limited functionality, as well as privacy and social implications limited consumer interest and ultimately caused Google to shut down Glass. However, the tide seems to have turned. Today there is a greater push to use VR/AR as a communications platform – the next step in the evolution of social media. People are apt to stay longer while socializing in a VR environment, delivering a captive audience that companies can monetize by serving directed ads and content. This partially explains Facebook’s acquisition of Oculus (a manufacturer of VR headsets) in 2013 for $2 billion, or Google’s $542 million investment in Magic Leap in October 2014. Imagine a fully immersive, VR Facebook world where “friends” can spend hours virtually interacting.

CCS Insights projects that the growth in VR and AR devices will increase from 2.5M in 2015 to 24M in 20188– an astonishing 860% increase in three years. The growth in device sales will correlate to growth in the supply of core elements needed to enable the VR/AR environment – newer software algorithms and applications, and hardware such as specialized chips, displays, sensors, components and small form factor power systems. Companies such as Cypress Semiconductor, Intel, Infineon and Qualcomm with their SnapDragon 820 chipsets and others are likely to enjoy the benefits of this growth. Startups with expertise in hardware, software, peripheral products and services are making their mark in this potentially lucrative field. 

Who are the players?

Numerous companies, both established and in startup mode, are venturing into this space.

  • Google – Google Glass was not a success but this has not stopped Google from venturing into this space on both the VR and AR front. With VR, Google developed Cardboard where consumers assemble a cardboard headset and use their smartphone as the screen. The kit includes a lens, Velcro to affix the screen to the headset and a specialized mobile application (loaded onto the user’s smartphone) that splits the screen for stereoscopic viewing. Google sells the kit for $10 and provides the application for free. Cardboard is an entry level VR product designed for short-term use. Due to its low price point, it can be readily used in schools as a teaching aide – think of it as a glorified View Master. Google already provides the stereoscopic software at no cost to developers and will use YouTube as the service platform for uploading 3-D videos. They have also developed an inexpensive, camera system called Jump 16 that uses 16 cameras and software to generate a 360 degree video for viewing on Cardboard. Google is stirring development in this field by giving away the design to commercial developers such as GoPro.
  • Facebook Oculus – A VR headset called Rift planned for release on March 28, 2016 for $599. An ancillary product to Rift is called Touch with haptic feedback that provides the user with force sensation. The planned for release is in the second half of 2016.
  • HTC Valve – A headset created by smartphone maker HTC and video game company Valve and planned for release on April 5th 2016 for $799.
  • Samsung Gear VR – Uses a simplified $100 version of the Oculus headset in conjunction with various new Samsung smartphones that projects the VR environment. This product has been available since late 2015.
  • Sony PlayStation VR – A VR system geared for its PlayStation platform and planned for release in the first half of 2016.
  • Magic Leap – Google’s investment in Magic Leap is for a highly sophisticated AR product that uses a technology called “dynamic digitized light field signal.” It closely simulates an AR world superimposed onto the real world through the potential use of retinal projection. Rather than projecting an image onto a screen which your eye views, in this application, specialized micro projectors are likely projecting the image directly onto the user’s retina. “Its digital light field encodes more information about a scene to help the brain make sense of what it is looking at, including the scattering of light beams and the distance of objects.”9 It also allows for a wider viewing angle that aids in seamlessly integrating the virtual and physical worlds together. This will be an integrated, head-worn display that would be wirelessly connected to the powering server. This should be released in a few years.
  • Microsoft HoloLens – This is a sophisticated, untethered AR headset that is similar to Magic Leap. It includes a variety of sensors (accelerometers, gyroscopes and magnetometers) for eye, head and position tracking, a built-in camera, advanced built-in speaker system to virtualize the location of sounds and specialized lens projection systems, all coupled with highly sophisticated software to eventually overlay virtual images onto physical objects. It includes systems for spatial and pattern matching that allows it to better capture the real world so that AR objects can be appropriately layered onto real world objects (e.g. floors, tables, walls, chairs, etc.). For example, placing a 3D ball in mid-air or virtual objects onto walls, tables and floors. Developer kits will be released on March 30th, 2016 for $3,000, with a consumer product several years away.
  • Microsoft Room2Room – This is a research project which uses their Kinect depth cameras and digital projectors to capture the 3D image of a person in one room and project a life-sized version of that person in real-time onto a piece of furniture in another room.”10 There would be a similar set-up in the other room that allows the two individuals to interact with their 3D virtual representation in real time. This takes a standard video conference call to the next level.
  • Others – There are other VR headset manufactures with product launches anticipated in 2016 such as FOVE VR, Zeiss VR One, Avegant Glyph, Razer OSVR, Freefly VR and numerous other lesser known brands.11

VR & AR Applications

As noted by Stanford assistant professor Gordon Wetztein, applications for AR “could be really seamlessly integrated into everyday tasks: communicating, working, information visualization, gaming…"12 Applications can include commercial, military, navigation, tourism/sightseeing, medical, educational, training, recreational, gaming and others.

  • Gaming – Major growth in VR/AR is likely to occur in this field since there is a captive and hungry audience that needs to be titillated with the next thrill – games seem somewhat realistic on a 2D screen but introducing VR/AR elevates this to 3D but more importantly, enables players to fully immerse themselves into the game, or experience the game incorporating part of their physical environment.
  • Military – HUD (heads up display) for aircraft and HMD (head-mounted displays) for ground troops are VR/AR technologies that have been in use for years. HUD allows a pilot to be more alert and responsive by providing all of the necessary data directly in front of the pilot’s eyes. HMD are used extensively in simulation and training ground troops.
  • Commercial – AR systems are already appearing in retail outlets such as Lego stores and makeup counters. Today, a child can hold up a Lego box near a store display with an integrated camera. The system can show a 3D representation of the finished toy on top of the box being held by the child and moves as the child moves. At the make-up counter, AR systems can show how you would look after applying certain products. Or at the fashion outlet, you can preview how you would look in a specific outfit by virtually overlaying your face/body on a display. Such uses will continue to expand into other areas of commerce to enticing customers through highly personalized marketing, and possibly generating a sale.
  • Medical – The typical application is to use VR/AR for education where surgeons train on virtual simulations and models of the human body. Going forward, such systems can be coupled with other imaging systems (MRI, CAT) to enable a neurosurgeon to see a virtual 3-D image of a patient’s brain that is overlaid onto the patient’s actual anatomy. Through this process, surgeons may be able to perform surgeries that are currently impossible.

    VR tools can also be used to help patients in a multitude of ways such as using exposure therapy to treat phobias, treating military personal with PTSD, managing pain for burn victims, alleviating pain from phantom limb syndrome, assessing and rehabilitating those with brain damage, providing opportunities to explore different worlds for those who are disabled or homebound, and training young adults with autism.13

    James Blaha is a game developer with a severe lazy eye (a condition that affects two to three percent of the world’s population) has used a virtual reality based training tool to cure this disease in 30-minute sessions over three to four weeks.14
  • Navigation – Traditional GPS navigation systems can be enhanced through the use of VR/AR by overlaying directions directly onto HUD or a smartphone or navigational device. GPS directions would be overlaid directly onto a real-time video being captured by the device’s camera. Not only is this a safer option as the driver’s eyes remain on the road, but it is also easier to use.
  • Tourism – The use of VR/AR systems can enhance a user’s sightseeing experience whether it is in a museum or an outdoor historical site. Imagine wearing AR-enabled glasses and walking through the Coliseum in Rome – not only seeing the sight but also learning about the specific areas you are viewing, or appreciating what the Coliseum would have looked like when first built by overlaying an AR model. Some types of VR/AR systems already exist in certain museums where interactive kiosks help educate and entertain guests.
  • Educational/Training – AR systems allow digital content to be added to printed material, geographic locations and objects, making for a more rewarding and participatory educational experience. In turn, more engaged students are more inclined to learn. VR can allow students to take virtual journeys to remote parts of the world without having to leave the classroom. They can also build and manipulate objects that are otherwise expensive, dangerous or fragile, and make mistakes without consequences. Imagine using VR/AR-based training to learn surgical procedures, operate heavy construction or other equipment. AR can “bridge the gap between the theoretical and practical and focus on how the real and virtual can be combined together to fulfilling different e-learning objectives.”15

Pitfalls & Issues

There are currently a variety of technological and human-related issues that limit the mass appeal of VR and AR systems as well as what these systems can render. These issues need to be resolved before these systems can become mainstream and ubiquitous.

  • Technology Issues – We are in the evolution stage and these issues will be addressed over time. However, for now, users will have to make do.
    • Tethered – Due to the significant processing power that is required for VR/AR systems, systems tend to be tethered (physically connected to computing hardware), which greatly limits mobility and use. Consequently, current use is primarily for research and industrial applications. Microsoft and Magic Leap claim that their products will be untethered but they have yet to be released. Other VR products such as Oculus Rift, HTV Vive, and Sony VR need to be tethered to a gaming system to operate. They also require PCs with substantial processing power.
    • Power – Because of the higher processing requirements, power and how the headset receives it become factors. A balance needs to be achieved regarding the amount of power required, the duration and how much the power supply will weigh. If it weighs too much, the headset may become uncomfortable to use for longer durations.
    • FoV (Field of View) – One of the major concerns is the FoV of VR and AR headsets. The typical human eye has a 180 degree FoV that increases to 270 degrees if the eyes rotate. Most of the VR headsets have FoV that ranges from the low 90s to 110 degrees.16 A few have higher but many are lower. HoloLens appears to have a 40 degree FoV but this may change over time. There are claims that Magic Leap’s product has 180 degree FoV due to use of Light Field technology but this has not been formally confirmed by the company. The smaller FoV with most of these products creates gaps in the AR environment since the user is able to view outside of the FoV by simply moving their head or eyes. This makes the virtual reality world less immersive. It may also lead to short-term health concerns such as eye strain, headaches and nausea.
  • Human-Related Issues:
    • Motion Sickness – Vection is the feeling of movement where the user sees movement because the visual field is moving but the user is stationary. When using VR headsets, the user is stationary but the entire visual field is in motion. The brain sees the movement but can’t correlate or synch to actual movement as registered by the inner ear’s balance system. This can cause motion sickness-related ailments such as nausea or vomiting. Using VR systems continuously for longer durations can exacerbate these conditions potentially discouraging users from using VR systems for an extended time.

      Since AR systems incorporate both the physical and virtual worlds, they are likely to be less prone to these problems.
    • Cybersickness/Sim Sickness – For some, extended use of VR platforms could potentially lead to seizures, double vision, altered vision, loss of awareness and other symptoms. Samsung has developed a lengthy disclaimer17 for its Gear VR product line. To protect themselves, OEMs are likely to develop similar disclaimer language as the true effects of long-term VR immersion on the human body are uncertain.

      In a recent study on rats conducted by researcher Mayank Mehta at UCLA18, they recorded the rat’s brain activity as the rats traversed a 2-D virtual maze while on a small treadmill. In animals, the hippocampus region of the brain is involved in mapping one’s location in space along with other functions. Neurons with GPS-type capability are formed as an animal moves through its physical environment, which produces a mental map of the environment. Indicators such as sound, smell, visual, distance, environment and others are used to form this mental map. By placing the rats in a virtual environment, the study found that “60 percent of the neurons simply ‘shut down"19 and did not form the standard mental map of the environment. A possible reason is the lack of other indicators which are not present in a virtual environment.

      Whether long-term use of VR affects the human brain similarly remains undetermined at this time. More research is needed.   

    • Other – The extended use of screens placed so close to the eyes can result in eye strain or eye twitching that can be acute or become chronic. Samsung advises 15-minute breaks for every 30 minutes of use. Other conditions noted by Samsung in their disclaimer include dizziness, disorientation, visual abnormalities, impaired balance, light headedness, fatigue and others. Such conditions are likely to affect a minority of users, but OEMs should be aware of these factors and address accordingly.

Conclusion

VR and AR devices have had their fair share of hype when every year seemed to be their breakout year. There is definitely more excitement today than before, however, due to the major releases from major players in this field. There will definitely be a variety of products available but the question that remains is whether the products will live up to the hype. Will there be sufficient applications and uses for these virtual products outside of niche areas, and will the price point and ease of use allow them to become mainstream in 2016? It appears that we are definitely moving in that direction. VR/AR offer numerous benefits from recreational to commercial to medical, but there are hazards (both known and unknown) that OEMs need to remain cognizant about and address them in the design of their products.

Contact Us

To learn more about how OneBeacon Technology Insurance™ can help you manage online and other technology risks, please contact Dan Bauman, Vice President of Risk Control for OneBeacon Technology Insurance at dbauman@onebeacontech.com or 262.966.2739.

References

1 Moon, Mariella. (July 1, 2015). “Zuckerberg reveals Facebook AI, VR and Internet.org Plans.” Engadget.  Accessed December 2015. http://www.engadget.com/2015/07/01/zuckerberg-facebook-qna/ 

2 (April 2015). “Augmented/Virtual Reality to hit $150 billion disrupting mobile by 2020.” Digi-Capital. Accessed December 2015.  http://www.digi-capital.com/news/2015/04/augmentedvirtual-reality-to-hit-150-billion-disrupting-mobile-by-2020/

3 Kofman, Ava. (June 9, 2015). “Dueling Realities.” The Atlantic.  Accessed December 2015. http://www.theatlantic.com/technology/archive/2015/06/dueling-realities/395126/

4 Sterling, Bruce. (September 20, 2009). “Augmented Reality: ‘The Ultimate Display’ by Ivan Sutherland, 1965.” Wired. Accessed December 2015. http://www.wired.com/2009/09/augmented-reality-the-ultimate-display-by-ivan-sutherland-1965/

5 https://en.wikipedia.org/wiki/The_Sword_of_Damocles_(virtual_reality)

6 Azuma, Ronald, T. (August 1997). “A Survey of Augmented Reality.” In Presence:Teleoperators and Virtual Environments 6, 4 (August 1997), 355-385.  Accessed December 2015. http://www.cs.unc.edu/~azuma/ARpresence.pdf

7 Perdue, Tim. “Applications of Augmented Reality.” AboutTech. Accessed January 2016. http://newtech.about.com/od/softwaredevelopment/a/Applications-Of-Augmented-Reality.htm

8 “Augmented and Virtual Reality Devices to Become a $4 Billion-Plus Business in Three Years.”  CCS Insights.  Accessed December 2015.  http://www.ccsinsight.com/press/company-news/2251-augmented-and-virtual-reality-devices-to-become-a-4-billion-plus-business-in-three-years

Markoff, John.  (July 14, 2014).  “Real-life Illness in a Virtual World.”  New York Times.  Accessed December 2015.  http://www.nytimes.com/2014/07/15/science/taking-real-life-sickness-out-of-virtual-reality.html?_r=2

10 Metz, Rachel. (January 19, 2016). “Augmented Reality Study Projects Life-Sized People into Other Rooms.” MIT Technology Review. Accessed January 2016.  http://www.technologyreview.com/news/545466/augmented-reality-study-projects-life-sized-people-into-other-rooms/?utm_campaign=newsletters&utm_source=newsletter-weekly-mobile&utm_medium=email&utm_content=20160125

11 Oscillada, John, Marco (March 19, 2015).  “ List of All Virtual Reality Headsets under Development.”  VR Times.  Accessed January 2016.  http://www.virtualrealitytimes.com/2015/03/19/list-of-all-virtual-reality-headsets-under-development/

12 Ibid 3

13 Carson, Eric. (April 8, 2015). “10 ways virtual reality is revolutionizing medicine and healthcare.” TechRepublic.  Accessed January 2016.  http://www.techrepublic.com/article/10-ways-virtual-reality-is-revolutionizing-medicine-and-healthcare/

14 Stein, Joel.  (August 27, 2015). “Inside the Box.” Time Magazine, page 43.  Accessed December 2015.

15 Wang, Xianguy.  (October 2012). “Augmented Reality: A new way of augmented learning.” eLearn Magazine. Accessed January 2016.  http://elearnmag.acm.org/featured.cfm?aid=2380717

16 Oscillada, John, Marco. (May 24, 2015). “Comparison Chart of FOV (Field of View) of VR Headsets.”  VRTimes.  Accessed January 2016.  http://www.virtualrealitytimes.com/2015/05/24/chart-fov-field-of-view-vr-headsets/

17 Samsung Disclaimer - http://www.bestbuy.com/site/electronics/disclaimer/pcmcat364300050002.c?id=pcmcat364300050002

18 Lewis, Tanya. (December 5, 2014).  “Virtual Reality Affects Brain’s ‘GPS Cells’.”  Live Science.  Accessed January 2016.  http://www.livescience.com/49021-virtual-reality-brain-maps.html

19 Lewis, Tanya (February 3, 2015).  “Samsung Gear VR: Virtual Reality Tech May Have Nasty Side Effects.”  Live Science.  Accessed January 2016.  http://www.livescience.com/49669-virtual-reality-health-effects.html