Book Summary – Charlie Fink’s Metaverse (An AR Enabled Guide to VR & AR)

First, consider how to describe this new world. Immersive tech expert Stephanie Llamas worries that AR/VR terminology confuses potential users. Many still use the outdated, 1992 “virtuality continuum” of “Mixed Reality” (MR). Consider this rundown, for starters:

  • AR augments reality– AR acts as a “tool” in the lineage of other augmenting tools. The AR experience mingles digital simulation with real surroundings, so it can’t supply full user immersion.
  • VR replaces reality– VR transports you to some other place or time. The user wears a headset that provides an interactive, simulated setting. Creators work with VR’s “presence and agency” to create various narratives, be they entertaining, educational or beyond either one.
  • MR mixes the real and the virtual via a headset– MR augments you here and now. Microsoft added to the controversy about defining AR, MR, VR and XR [Cross Reality] with its Mixed Reality Spectrum. Its decision to use MR for its fully “occluded headset” increased confusion about terms.
  • XR is a blanket term for immersive technologies – XR is emerging as the favored term for the AR and VR ecosystem. As Apple, Google and Facebook (with its acquisition of Oculus Rift) move in to AR/VR, the need for clarity grows. Qualcomm tried to establish XR as “extended reality,” but “X” in XR still stands for a mutable variable.

Immersion Tech

To access connectivity’s wonders, current users must work with smartphones and tablets, but these familiar devices won’t keep up. Fink predicts that head-mounted displays (HMDs) will replace smartphones as the interface of choice. Later, people will adopt “wearables” and other new devices to make smartphones a relic and make even HMDs seem like stepping stones.

“As our machines get faster, they are going to get plenty fast enough to fool our senses and make us absolutely certain that we are in places as real as our waking lives.”

Pattie Maes of MIT’s Media Lab predicts an inevitable cyborgization of humankind as the digital world of the future will one day simply appear in your mind’s eye thanks to “brain computer interfaces” (BCI) or perhaps even “neural lace.” People crave VR’s full-immersion illusion. This wish for an almost “metaphysical” experience springs from ancient tales of trans-dimensional myths and magic, of “heaven and hell.” Today’s “escape rooms” attractions immerse users in live-action scenarios. Actors in New York’s Paradiso escape room interact with the audience. Some VR productions, such as Penrose Studios’ Allumette, let observers select their own perspective.

Cultural Influences

Michael Eichenseer, a VR Voice writer, highlights “pop culture’s” influence on VR, but that influence dates back in time. Leonardo da Vinci’s notes and sketches blended futuristic science and art. Science fiction – say, Mary Shelley’s seminal Frankenstein – often demonstrates a soaring futurism. Neal Stephenson’s 1992 Snowcrash introduced the term “Metaverse” for “beyond” the material world. The Matrix presented simulated reality wired into people’s brains. In Avatar, humans control bio-engineered alien bodies. Star Trek’s “Holodeck” inspires real-world “free roam location based VR” (LBVR). Holographic projection enabled Princess Leia’s message in Star Wars. The killer robot’s augmented display in The Terminator presaged blends of AI and AR. And Westworld raised issues about human interaction with VR characters.

“To get people to buy Pepsi, they have to know what soda is. For this reason, adoption may look more like personal computers, which took 15 years, than smartphones, which took two years.”

Nineteenth century stereoscopes took advantage of human “stereoscopic vision” to give the illusion of 3D depth. To get immersive 360-degree VR or AR, you need “optics” with light-bending lenses focused on an “infinite focal point.” In VR headsets, two screens create a 3D effect. Tracking follows the user’s head position and orientation. Poor tracking “latency” rates cause “VR sickness.” Machines offering “six degrees of freedom” (6DOF) track users’ movements better than smartphones’ limited 3DOF, but require tracking via either “inside out” – headset-mounted cameras – or “outside in” – external sensor “lighthouses.” Haptics provide sensory feedback; “VR gloves” provide sensory input. VR firms are striving to create whole body haptic feedback, like temperature, taste and smell.

“If people do indeed prefer presence in VR to the relative simplicity of social media, social VR could be a killer app.”

Star Wars Jedi Challenge AR gives you a “lightsaber,” a smartphone-mounting headset and three games. Acer’s sleek Windows MR headset provides a worthy VR experience. The dependable Oculus Rift, HTC Vive and lower-priced Playstation VR headsets offer “premier gaming.” In budget mobile VR, note Google’s Daydream View, Samsung’s Gear VR and the crazy-cheap Merge Cube VR, which enables even elderly smartphones. The ZapBox provides an MR kit including a cardboard headset and controllers.

The AR/VR “Inflection Point”

Expect much slower adoption of VR than of smartphones. Most users still see VR as a novelty. Infrastructure problems like low bandwidth plague VR, but Emergent VR’s Peter Wilkins says AR may offer greater “rewards” after several elements gel, like “spatial services” for 3D mapping of interior spaces. Building a worldwide “AR Cloud” will require mass consumer participation. Recognizing objects in space calls for artificial intelligence (AI) and “user interfaces” (UI) that enable intuitive 3D control.

“Essentially there is no one size fits all method for any platform.”

To reach consumers, AR firms must get their “value proposition” just right. Now, at $98 billion, the games industry dwarfs the $2 billion VR industry, but games offer monetization models for VR. On Google Play and the App Store, “free-to-play” games – with options to buy add-ons – earn the most money, but VR lacks mobile’s vast user base and easy payment systems. Letting users pay to skip ads may work when VR develops its user base and ad platforms. More content would drive the subscription model, which works best now for content distribution. Ticketing makes sense for LBVR, especially in theme parks. Hardware quality, the cost of content and “critical mass” will drive VR’s monetization.

Killer Apps and the AR Cloud

Ori Inbar of the AR fund Super Ventures thinks free AR-building tools from Apple and Google will encourage developers, but “mass adoption” will require the AR Cloud. When perfected, this “soft 3D copy of the world” will enable mass interactive collaboration. A “point cloud” of 3D coordinates will map the physical world to the AR Cloud. An “ultimate localizer” will tell AR devices where it is and will “pose” within a local point cloud. VR devices will lay “virtual content” over the real world. Now, AR isn’t interactive with the physical world. GPS can’t localize with sufficient accuracy, and the current “cloud” creaks under the strain of real-time streaming. But AR serial entrepreneur Matt Miesnieks sees the AR Cloud as a historic, vital development. And, 5G networks will help.

“Presence and agency in a photorealistic simulated world is the holy grail of immersion.”

Augmented reality changes the relationship between artist and audience. Coupled with Blockchain, it promises digital “scarcity,” letting artists limit or open up interaction with their work. New models and processes will emerge with open-source artistic collaboration, including “remote collaboration.” Jaron Lanier and others pioneered shared virtual experiences in the early 1990’s. Today, companies like Rec Room, Big Screen and AltspaceVR lead the pack. MR enables users to undertake complex, guided real-world tasks in remote locations.

“VR enthusiasts hope for a level of immersion as real as The Matrix, but fear the dystopian future of machines trapping the entire human race within a simulation.”

The business world adopts new tech earlier than home users, says Fink. He sees telepresence as an AR/VR killer app, and he lists several types: 2D Video Conference Systems, Robotic Telepresence (piloting remote ROVs), AR and VR Telepresence, and “True Holographic Telepresence,” using projection and mirrors to provide a virtual presence without HMDs. Big tech sees VR/AR domination as a major prize. One expert asks if the massively-backed newcomer Magic Leap could become AR/VR’s Apple? The big, heavyweight companies matter here: Microsoft, Google, Apple, Facebook and Amazon all have uses in play. Management expert Dr. Annika Steiber recognizes China’s drive to seize the AR/VR initiative, with Baidu, Alibaba and Tencent all on board. China’s government concentrates on the “innovation infrastructure,” start-up investment and education. Ambitious projects, including “VR towns,” are in its latest Five Year Plan.

Work, Medicine, Gaming and Social Media

Public relations executive Dirk Schart and producer Samuel Steinberger view VR/AR as transformative for work and education. They envision a near-term world of infographic holograms floating over cities and in-car virtual meetings. Virtual presences will assist you at work. Remote experts will train you via AR-overlaid interactive visualizations. Neuroscientist Dr. Walter Greenleaf says declining costs will make VR widely accessible in medicine. Current applications include treatment of PTSD with Skip Rizzo’s “Bravemind” VR system; Applied VR’s use of VR experiences in pain management; and Osso VR’s orthopedic surgery training system. Portable VR revolutionizes telemedicine by bringing virtual-presence doctors into homes.

“Man will be augmented. Man will merge with machines to become cyborgs.”

People need connection, hence social media’s success. The element of meeting other people – “social discovery” – attracts people, but most seek an AR/VR Facebook page – “social VR.” For some, Second Life fills this role. New social worlds are now online or in beta, including Oculus Rooms and Facebook Spaces.

“Our need to connect with other people follows us, no matter where technology takes us.”

Artists also see VR/AR’s potential, despite its current limitations. Sculptor/printmaker Zenka, painter Jane Lafarge Hamill, visual effects artist Kevin Mack and others seek to engage audiences interactively. Writer/director Tim Kashani celebrates the dissolution of the actor-audience divide. Moving from “cartoony” AR/VR to “cinematic” experiences requires multi-camera 3D “photogrammetry.” New software makes this accessible to creators and new 3D “volumetric capture” makes high-res avatars of people eerily believable. LBVR – in the form of “VRcades,” “casual VR,” “VR theaters” or “warehouse scale” – promises a lucrative new business model. Some small HTC Vive-based operators, use Springboard VR’s VRcade management system. A few IMAX VR centers operate at a larger scale, and warehouse-scale experiences also provide immersive play, but at high operating costs. The theme park pricing model works for AR/VR, but many operators struggle to turn a profit.

“I am a strong believer that calling [Pokemon GO] ‘AR’ is about as accurate as calling two slices of bread a sandwich.”

Lenders invested $2.3 billion in AR/VR in 2017. But invention and fast deployment remain challenging and big players compete by acquisition. In the “pyramid” of the AR/VR ecosystem, hardware is the base, with distribution and developer tools in the mid-layers. The killer apps will form the pinnacle. Artists and enterprises blaze the trail as AR and VR develop in tandem with related tech such as AI and robotics, but the mass audience wants a compelling social reason to grab AR/VR. The AR Cloud and other developments will make futuristic imaginings a reality but, as with Facebook, uncontrollable “Frankenstein” aspects will also emerge, creating new social problems.