Welcome to Extended Reality (VR, AR, and MR)
While AI is becoming more and more common, accepted, and every-day in our lives, the introduction of extended reality (XR) is going to take the tech in our lives to a whole new level over the next few years.
With AI we already have fantastic systems to enrich our day-to-day (with many more on the horizon):
- Nest’s self-learning thermostat saves you money on your power bill without you having to do anything.
- Fitbit’s SmartTrack incredibly recognises and records your exercise for you, no more counting floors you’ve climbed and entering it manually!
- If you have Alexa, she might tell you that it’s time to get in the car if you want to get to your next appointment on time.
- Google maps redirects you when there’s a car accident on your route.
- Still emerging are things like self-driving cars.
But there is a whole new world of possibilities coming with extended reality (XR), which is the umbrella term that covers virtual reality (VR), augmented reality (AR), and mixed reality (MR). This is where experience comes into the user experience, with the emergence of these technologies, UX is moving to a whole new level.
Chances are, you have heard of all or some of these, but we will start at the beginning, by explaining what each of these are, and then we will look at design guidelines for including these technologies in your mobile games or apps.
We mentioned above, extended reality (XR) is the umbrella term that encompasses virtual reality (VR), augmented reality (AR), and mixed reality (MR) technology.
These technologies have things in common, which is why we have a term to include them all. The idea of XR has been around for decades, however, it’s really been a matter of waiting for technology to catch up with the ideas before they could become a reality and accessible to the general population.
It is not that hard to create tech in these areas, however, it is hard to create excellent tech in these areas. We will discuss some of the things you will need to consider during design and development later in the topic. Many people have tried early versions of virtual reality tech and learned first-hand about their tendency to produce “simulator sickness” in the user! Simulator sickness is the name given to the specific type of motion sickness people encounter when using simulators or VR tech if developers are not careful enough.
XR has already made a big impact in the education and medical fields and has the potential to revolutionise these and many other industries once the tech becomes more mainstream and accessible. Consider the impact of simulator training for pilots and astronauts, imagine that sort of tech being available across all industries for training. We’ve all heard that showing is better than telling, well, now doing is better than showing!
While learning about XR, you will no doubt hear reference to the
Watch Extended Reality: The Top Five Things You Need to Know (Inclusive Reality, 2018) to get a good picture of the scene we are learning about and then we’ll dive into virtual reality.
Virtual reality, or VR as it is commonly known, is where the user is immersed, through use of a special VR headset or goggles, into an artificial, computerised 3D environment, which they can interact with. A very key point here, that differentiates VR from other realities, is that you are not able to see any part of your real environment while you are experiencing VR.
It’s kind of like walking around in a movie or in that scene from your favourite novel.
- For job training or education, you might use a VR that looks like a normal everyday environment, like a hospital, factory, or office building – simulated environments and technology, provide a risk-free hands-on learning environment.
- For entertainment you might be looking for a VR that is ‘out of this world’, like being on another planet or walking around with dinosaurs.
- For relaxation or meditation, you might like to be sitting on a yoga mat on a tropical island looking out at a calm sea.
- For a fun night with friends, you might like to find your way out of a virtual escape room game together or go to a dance party in Cuba.
- For gaming, the world’s your oyster, or should we say out-of-this-world is your oyster? Your imagination (and wallet) is the only limit.
Have a look at the list of most anticipated movies of the year, and you will see all kinds of movies that revolve around virtual realities – this is a thing that once seemed futuristic and beyond reach which is now getting closer and closer, cheaper, and cheaper. And it is very much a hot topic!
Traditionally the cost of the gear was just so prohibitive for the public, however that is now changing. A few years ago, Google released a product called Google cardboard, which was a cardboard VR headset. Your own smartphone slips into the front of the headset and becomes the VR screen. No need for expensive gear, just a $20 carboard headset and your own smartphone. Suddenly, the public had affordable access to VR. You may not have heard of Google cardboard, and that’s probably because it’s not around anymore, however, google did make the instructions available for self-building, but what Google cardboard did do, was wake up the industry to the fact that VR doesn’t have to be super expensive and exclusive, and it has shifted the platform, making VR far more accessible. Soon after Samsung joined forces with the world leaders of VR tech – Oculus - to produce a headset with the latest VR technology called Gear VR. It requires one of the latest Samsung Galaxy smartphones along with the headset but will be much more stable as far as simulator-sickness thanks to the latest head-tracking software. The costs are slowly coming down.
Read How Rendever and AARP Used Virtual Reality to Grant the Wish of a Lifetime (Jaehnig, 2021) to discover how an older man’s travel dreams were realised using VR.
Read 10 Virtual Reality Trends to Look Out for in 2021 and Beyond (Sambol, 2021) to learn about different implementations of VR hitting the world stage.
Augmented reality is when you can see your real-world environment, but you can place/see digital objects in your real world. What sets it apart from the VR and MR is that you cannot interact with these digital objects.
Blending the real-world and digital objects means AR is a safer technology to use when outside the home or office. Although it is always best practice to begin with a warning screen for users.
AR can be much cheaper than VR as it usually doesn’t require any special tech, no headsets that block everything out to give you a fully immersive experience. Anyone with a smartphone can already download and use many AR apps.
At a kids’ hockey game, a parent on the side lines pulled out their phone to take a picture of the kids playing and those nearby heard a frog noise and then saw on the phone screen, a frog sitting on the hockey turf with kids running all around it. The other parents on the side lines were confused seeing a frog on the screen, but not being able to see it on the field, and were then delighted when they discovered that it was AR.
DHL, US Navy, and Boeing all deliver training using AR, reducing risk and first-time errors, and saving time.
Reading a storybook, kids can point their camera at a picture for it to pop up before them as a 3D image with a voice reading them the relevant part of the story.
As we saw in our earlier video, in the medical field, AR can overlay veins on the patient, so that it’s quicker and easier to draw blood. Who would have thought these things could be possible?
Watch HoloAnatomy app helps medical students learn anatomy (Case Western Reserve University, 2019) [1:53] to see the impact AR is having in the education field – you’ll notice that this AR app is using a headset, however this headset allows users to see the hologram in the real world, it is not immersive like VR, but allows the users to have their hands free to read and make notes as they are learning.
These examples really bring to life the potential gains from having more and more AR in our future.
Of course, one of the most well-known AR examples would have to be Pokémon Go. The goal of the creators was to get kids adventuring and exercising outside. Players use the app to catch and train Pokémon in real world locations, viewing them on the app, using their phone camera. It can be played anywhere.
In most of these cases, AR requires the user to view the real world and digital objects through a device camera, which makes this reality the most accessible to the public.
This is where things get really exciting, mixed reality is where the ‘extended’ comes into extended reality. Taking the best from both worlds, mixed reality combines aspects from both VR and AR and takes it to a new level.
Mixed reality is where we can interact with digital objects (holograms) that can be seen in our real world. This is the stuff that we’ve seen in movies and dreamed about for decades.
Watch What is mixed reality? A simple explanation for anyone (Bernard Marr, 2019) [3:32] to hear a clear explanation of all three realities.
So mixed reality combines the real world with a virtual world, VR with AR, and adds a bit more interactivity, making it sit a little further along the reality-virtuality (RV) continuum. You’ll see this explained in an upcoming video.
For example, mixed reality can bring to life all kinds of education in a visual and interactive way, which makes the learning so much more memorable. Read and learn about when The Weather Channel Launches Mixed Reality Broadcasts (Mileva, 2018), make sure you don’t miss the video at the end. Then we’ll delve into a quiz about what we’ve covered so far in the topic, before we move on to look at some design guidelines and principles for XR.
XR Design Principles
With extended reality, the stakes are high - it is both expensive and tricky to get XR right. But when it’s done right, it has incredible potential. The sky is the limit with XR. But as it is with AI, it is also important to grasp the responsibility designers have to the industry, it’s important that if you are going to tackle this, that you do it well, so as to aid the industy’s forward progress, not to harm it.
So what do we need to keep in mind while designing for XR? Watch Mike share valuable XR Design Theory and Practice for Digital Eyewear (Mike Alger, 2020) from his experience in the field.
Downs (2019) describes the three guiding principles that apply to all of XR as: space, motion, and scope.
Space
When dealing with 3D design, the creator needs to be completely aware of the space available and how all the elements fit together in that space, ensuring no element is oversized compared to the space available or to the other elements, and that no element is too big to be comfortable for the user. Ensure the interface is not overcrowded with text and that it remains clear and legible at all times.
Motion
We mentioned earlier ‘simulator sickness’ as the most important thing you need to avoid when designing for VR especially. The problem occurs when what the eyes see does not match the movement in the body. If users experience simulator sickness with your product, they will abandon it and look elsewhere. The users’ movements must be reflected by real time movement in the VR.
Read Best practices and virtual reality design principles (Mealy, 2018), an excerpt from Mealy’s book on Virtual & Augmented Reality to get some excellent practical tips on designing for VR, with a focus on motion and visual content. Mealy is a very experienced VR designer and author.
Scope
Ensure you don’t include functions or motion where it is not giving real value to your application – watch out for scope creep – where the customer or designers keep adding to the app, affecting the cost and timeline of the project.
In his article Downs also discusses a series of principles for each of the three reality types. Altogether, there are 18 design principles to learn about before designing for XR.
Scroll to the heading called ‘Common principles for AR, MR and VR design’ and read from there to the end of the article to learn 18 AR, MR and VR design principles (Downs, 2019).
XR Design case study
You might be wondering what your first XR project might be like – that of course will be affected by many factors, including whether you will be on your own or working in a team. But let’s look at how Ales progressed through his first XR project and what he learned along the way in Designing for AR - UX/UI Case Study (Ales Nesetril, 2019).
Once again, knowledge of the tools and shortcuts available to you will make an enormous difference. Often there is no need to reinvent the wheel, with a lot of the big businesses in the industry making way for others to come in and develop apps using their development tools, to:
- make it easier
- help ensure compatibility with their products (head gear etc)
- provide for a certain standard of performance (built-in head tracking etc).
Extended Reality summary
We hope that what you have learned here about VR, AR, and MR has inspired you to look to the future and aspire to be a part of the XR movement.
Finish this topic by watching Ryan share his 5 UX Design Tips for AR - Microsoft Designing Holograms App on HoloLens (RyanXR, 2020) for a practical look into designing with XR.
