The What’s Up with Science? blog series offers a deep dive into science, technology, and innovation topics on the minds of the public. The series matches technical explanations with relatable analogies to explain opportunities and answer the ultimate question: Why should we care?
My first visit to Hogwarts occurred during my high school class trip to Orlando, the theme park capital of the world. The thrill ride built inside that unmistakable castle was like nothing I had experienced before. I remember a virtual world splayed before me and being suddenly confronted by a massive, fuzzy, water-spitting spider. The crafty combination of wrap-around projection screens and life-size animatronics convinced me that I was flying through the Wizarding World, challenging my perception of reality.
That’s exactly the function of a group of technologies that fall under the umbrella of extended reality, or XR, which includes virtual reality, augmented reality, and other mixed realities. These XR variants differ in how they allow users to immerse themselves in sensory-based experiences. You can think of the different “flavors” of XR as points on a spectrum.
On one end of the spectrum is virtual reality (VR), which subjects a user to a completely virtual world. This is like how Harry Potter used Dumbledore’s Pensieve to “experience” people’s memories. Harry could experience the Hogwarts of decades ago through someone else’s eyes—he could see and hear what they saw and heard—but the world he experienced was virtual. Unlike the Pensieve, today’s VR devices use motion sensors to record the user’s movements and screens to project 3D images into the user’s line of sight.
On the other end of the XR spectrum is augmented reality (AR), which “enhances” the real world using computer-generated information or images. For example, you could look at your backyard using your cell phone camera and see a Pikachu standing in your garden (this, of course, is the premise of the popular mobile game Pokémon Go). AR overlays information on top of a real environment by using sensors to map the corners and edges of real-world objects. Harry Potter had an AR-like experience when he looked in the Mirror of Erised and saw his family surrounding him, even though they weren’t actually present.
You could also imagine combining different ratios of real and virtual components. This is mixed reality (MR) and comprises the middle section of the spectrum. The thrill ride inside Hogwarts castle in Orlando is a good example, since it supplements its screen-based, virtual world with real (and wet) objects.
Actually, the ride’s water-spitting spider highlights the need for XR experiences to stimulate senses beyond the visual. Currently, it’s fairly simple to incorporate sight, sound, and touch, while smell and taste are more difficult. Other challenges include user safety (experiencing a virtual or augmented world can be disorienting) and privacy (those sensors are collecting lots of data), as well as ensuring that devices have adequate processing power to run XR applications.
Overcoming those challenges gives us access to more than just awesome theme park rides. For example, architects and urban planners used AR to visualize building damage following the 2011 Christchurch earthquake. In the education sector, applications are being developed to help students visualize molecules or experience historical events. Medical professionals can use XR to see a patient’s blood vessels, organs, and tumors—or even guide them during surgery. These examples are just the tip of the iceberg, as XR also touches archaeology, art, engineering, entertainment, flight training, manufacturing, text translation, and even tourism.
XR is clearly changing the way we interact with the world around us, and the U.S. Government is finding both direct and indirect ways to use this technology to facilitate innovation and promote prosperity. Here at home, the Foreign Service Institute at the State Department has introduced VR as an experiential learning tool in certain trainings. Elsewhere, AR and VR training programs have been implemented to develop local workforces and manage wastewater. MR software and other immersive experiences are even helping people learn about our solar system.
The magic of XR lies in its ability to change our perception of reality, putting experiences in other countries, outer space, or even the Wizarding World at our fingertips. What would you do with XR technology?
About the Author: Aubrey R. Paris, Ph.D., is a Science and Technology Policy Adviser in the Office of the Science and Technology Adviser to the U.S. Secretary of State (STAS). She received her Ph.D. in Chemistry and Materials Science from Princeton University and B.S. in Chemistry and Biology from Ursinus College.