AI Trends Weekly Brief: Convergence of AI with Augmented Reality and Virtual Reality


The convergence of virtual reality, augmented reality and AI is now becoming possible due to converging technical trends including: recent developments in AI that foster real-time image and speech recognition; increased availability and lower cost of local processing and storage; expanding network bandwidth to allow richer data streams; and the availability of AI in the cloud.

The impact of evolving AR/VR is through a range of industries beyond its arguably native field of gaming and entertainment, to include manufacturing, oil and gas, healthcare and education. 

To clarify terminology, augmented reality we define as technology that superimposes a computer-generated image on a user’s view of the real world, providing a composite view. Virtual reality is a computer-generated simulation of a three-dimensional image or environment that the user can interact with in a seemingly real or physical way, by using a helmet with a screen inside or gloves fitted with sensors.

To better understand how these new technologies can be applied to say, manufacturing, we can envision a continuum, with real environments at one end and wholly virtual environments at the other. In between lies augmented reality, as well as augmented virtuality which we can call mixed reality. This notion of a reality-virtuality continuum was first introduced over 20 years ago by Paul Milgram, an engineering professor at the University of Toronto, according to an account at 

To put this in a manufacturing context, think of the Real Environment as the factory floor, with workers operating machines based on information displayed on dials, gauges or screens. The Virtual Environment would be exemplified by use of virtual reality in factory floor planning. While few examples of VR in manufacturing can be cited, many examples of AR exist.

Roots of AR/VR in Gaming, Entertainment

The exact origins of virtual reality are disputed, partly because of how difficult it has been to formulate a definition for the concept of an alternative existence. If we skip to the 1990s, we saw the first widespread commercial releases of consumer headsets by game manufacturers including Sega and Nintendo. In 2010, the first prototype of Oculus Rift appeared, featuring a 90-degree field of vision previously unseen in the consumer market. In March 2014, Facebook purchased Oculus VR for $2 billion. By 2016, some 230 companies were developing VR-related products. Facebook has 400 employees focused on VR development. Google, Apple, Amazon, Microsoft, Sony and Samsung all have dedicated AR and VR groups.

The launch of Pokemon Go in the summer of 2016 was a breakthrough for the gaming industry and augmented reality itself. The game hit its peak in the summer of 2016 with nearly 45 million users. The game combined: mobility, for its ability to run on smartphones with AR software and tracking systems; meaningful (arguably) content; convincing and realistic interaction of the virtual and physical environment; and a unique value that went beyond what other technologies were delivering.

In this way, the game provided useful direction for future design of AR systems outside entertainment, such as marketing, fashion, tourism and retail.

AR/VR in Manufacturing

Manufacturers are only beginning to scratch the surface of what AR can do for their industry. Here we examine five areas of interest and selected startup companies working in the field.

Complex Assembly

Modern manufacturing involves putting together hundreds or thousands of components in a precise sequence as quickly as possible. This is true in the manufacture of smartphones or jet engines, and every new product requires a new set of assembly instructions.

“Your work instructions tend to be these PDFs that are hard to work through, plus they’re static documents, so they may be out of date,” commented Ash Eldritch, CEO and co-founder of Vital Enterprises, a developer of augmented reality software, in an interview with

“We take those instructions and make them glanceable in your field of view at all times, hands-free and voice-controlled,” Eldritch continued. “So we break down the work instructions along with associated technical drawings and even video from the last person who did the procedure and put all that onto the [AR] glasses. That means you can keep your hands on your task and you don’t need to walk over to a workstation to check something.”


In addition to helping with the assembly of manufactured products, augmented reality can be used to assist in the maintenance of manufacturing equipment. Mitsubishi Electric has been developing maintenance-support technology using augmented reality based on a 3D model that enables users to confirm the order of inspection on an AR display and then enter inspection results with their voice.

“We think that the most useful application of AR is maintenance in a manufacturing environment,” a representative from the Mitsubishi team developing this technology told “Currently, field workers confirm a maintenance target using a maintenance manual. The process is time-consuming and tiring for the workers because the manual and the maintenance target should be confirmed together.”

More specifically, Mitsubishi expects its AR system to be used for a variety of maintenance work, with some applications beyond manufacturing, including inspections of water-treatment plants and building electrical systems.

Last year, elevator manufacturer thyssenkrupp announced that its technicians would begin using Microsoft’s HoloLens technology as a tool in service operations. Using HoloLens, service technicians can visualize and identify problems with elevators ahead of a job, and have remote, hands-free access to technical and expert information when on site.

As the first fully self-contained holographic computer running Windows 10, Microsoft HoloLens has been helping companies and industries innovate in entirely new ways, said Scott Erickson, general manager for Microsoft HoloLens. “By utilizing the out-of-the-box Skype experience without any additional development required, thyssenkrupp’s 24,000 service engineers can now do their jobs safer and more efficiently. Triaging service requests ahead of the visit and getting hands-free remote holographic guidance when on site has reduced the average length of thyssenkrupp’s service calls by up to four times.”

Expert Support

Another fertile manufacturing application for augmented reality is expert support. This application also relates to maintenance, as Eldritch of Vital Enterprises explained:

“We’re doing a lot of field service, where the manufacturer of a machine also needs to provide service for it, and that involves travelling to the site where the machine is. For very distributed manufacturing operations, you have a number of inspectors and a number of technicians, but not enough experts for when things, inevitably, don’t go according to plan.”

“In those instances,” Eldritch continued, “you normally need to have your experts travel to the worksite, but by providing a ‘see what I see’ kind of telepresence, the expert can look through the eyes of the technician who’s actually doing the maintenance. They can also annotate on the field of view of the technician, so they can point out particular features of interest in what the technician is seeing. That allows experts to offer support and perform inspections from anywhere.”

This suggests that the much-lamented skills gap in manufacturing could have a technological solution in the form of augmented reality. Rather than training every technician or machinist, companies could use AR technology to supplement their employees’ existing knowledge with engineering expertise delivered via telepresence.

This also opens up new possibilities for training.

As Kocic of Hexagon Manufacturing Intelligence pointed out, “You can feed training information right on top of the actual parts and assemblies. So you can start mixing the execution with training, which ultimately makes the equipment easier to learn and use. That’s especially important given the general trend in manufacturing toward using less skilled labor than before. Before, metrology was done in air-conditioned rooms, but that’s shifted to the factory floor, and to people who are not specifically metrologists but instead are usually manufacturing engineers or operators. So how can you serve that information in a better way? AR seems to fit in that environment.”

Quality Assurance

Metrology in general, and quality assurance (QA) more specifically, offers numerous potential applications for augmented reality; some manufacturers have started exploring the opportunity.

Last year, SlashGear reported on a pilot program running at a Porsche assembly plant in Leipzig, Germany. A demonstration by the company showed Porsche technicians using augmented reality as a tool in the quality assurance process.

The basic idea is that quality professionals can take photos of parts or assemblies on vehicles under inspection, and then compare those images to ones provided by the company’s suppliers via an augmented reality overlay. Features that are out of specification can be highlighted by the overlay, enabling the Porsche technicians to identify the issue quickly and intuitively.

The eventual plan, as reported by SlashGear, is to link the cameras on the production floor directly to Porsche’s cloud-based parts database in order to enable real-time analysis of parts and assembled components. This could reduce inspection times significantly, though it would also seem to undermine the value of augmented reality in favor of automation.

Airbus has been using a “Mixed Reality Application,” or MiRA, to integrate digital mock-ups into production environments, giving assembly workers access to complete 3D models of the aircraft under production. According to Airbus, MiRA has been used on the A380 and A350 XWB production lines to check the integrity of secondary structural brackets, which hold hydraulics and other equipment in place.

“MiRA links a real object with its digital genome, transforming reality into an interactive world in which information about the object can be directly accessed,” said Nicolas Chevassus, head of industrial processes at Airbus Innovations. Airbus has reported that MiRA reduced the time required to inspect the 60,000 – 80,000 brackets in the A380 fuselage from 3 weeks to 3 days.


One school of thought is that AR is used to make the human worker smarter, forestalling replacement of the worker by total automation.

The manufacturer of the DAQRI smart helmet sees human workers as cooperating with automation.

“The DAQRI Smart Helmet cooperates with automation because it uses common protocols and standards, it is designed to integrate to DCSs, PLCs, SCADAs, Histograms, CMMSs and ERPs, making information previously constrained to the control room available right on the helmet, wherever the worker goes,” a DAQRI spokesman told

From the software side, Eldritch of Vital Enterprises offered a somewhat different perspective:

“All these technologies that are coming together around artificial intelligence are going to augment the capabilities of the worker and that’s very powerful. I call it Augmented Intelligence. The idea is that you can take someone of a certain skill level and by augmenting them with artificial intelligence via augmented reality and the Internet of Things, you can elevate the skill level of that worker.”

“So, I think that as more jobs get automated we’re also going to be pushing more workers up the skill-chain, which will make them more competitive in the market. So augmented reality and automation may compete in the near-term, but I think there will eventually be cooperation between automated machines and augmented human workers.”

VR/AR in Medical Diagnosis, Treatment and Therapy

With the advances in technology, VR, AR and artificial intelligence will create entirely new applications in healthcare. A 2017 global healthcare sector outlook by Deloitte cited in an account in Forbes, suggests the top ten technology innovations that will bring more value for less in health care are next-generation sequencing, 3D-printed devices, immunotherapy, AI, point-of-care diagnostics, VR, social media, biosensors and trackers, convenient care and telehealth.

The race to capture the three trillion dollar healthcare industry has already started with companies like Google and Apple acquiring health tech companies and fine-tuning their own platforms for smart glasses and other personal mobile diagnostic devices like a blood glucose monitors. At the World Economic Forum in Davos in January 2017, SAP CEO, Bill McDermott emphasized personalized, precision medicine through technology will be the future of the healthcare industry and moving it from a primarily analog experience to a digital one.

But instead of waiting for the future, there are already healthcare organizations using VR to improve the everyday lives of people. At Maplewood Senior Living in Connecticut, the residential facility is using VR to create immersive experiences to help residents with cognitive impairment and dementia unlock memories and stimulate emotions and interaction.

“We’re using VR for the sheer experiential value of creating these immersive 360-degree experiences,” said Greg Smith, CEO, Maplewood Senior Living. “We want to be able to provide a wow experience beyond the healthcare application because anyone can enjoy the experience of being within a virtual reality world.”

Mark Greget, co-founder, and CEO, NuEyes, wanted to create augmented reality (AR) smart glasses for people who have low vision or are vision impaired. He co-opted the technology from his time in the US Navy to make a prototype in 2013. Four years later, the NuEyes smart glasses are built on the ODG R-7 platform from Osterhout Design Group, with $1.2 Million in investment from angel investors under their belt. The company also has a major national insurance carrier that now covers up to 50 percent of the cost of the product which makes it more affordable for the visually impaired.

One of NuEye’s users is nine year old Felix, who is legally blind due to a rare disease. These smart glasses bring the outside world into focus for Felix. Felix can see spots of color, but he can’t see facial expressions or objects in a room. The NuEyes smart glasses let Felix see the faces of his classmates and the blackboard in the classroom, so he can have a meaningful learning experience like his classmates.

AR/VR Impact in Retail, Videoconferencing

Retail provides a number of exciting possibilities for a combined VR/AI approach, according to an account in TDWI. Virtual showrooms can provide enhanced product selection while a learning algorithm customizes both products and the buying experience itself. High-tech dressing rooms such as Ralph Lauren’s store in NYC provide another example. Sentiment analysis from social media combined with analysis of customer movement using big data and AI is already linking in-store marketing to customer’s smartphones.

Additional AR components could allow people to easily locate goods that could be enhanced with promotions such as virtual coupons. Combining an AR interface similar to Pokémon Go with a service such as the IBM Watson-based “Macy’s On-Call” in-store virtual assistant is likely to be the next target for retail marketers.

Another key business application is to improve videoconferencing. Videoconferencing suffers because telepresence — the sense of being in the room — is often missing. Immersive telepresence enhancements will come by adding elements of VR, AR, and AI to improve the collaborative experience. This may include improved camera tracking to naturally focus upon individual speakers or gestures and AR to provide real-time information on subject matter and participants in areas such as emergency response.

Funding and expertise is increasing in this area, including a new initiative from MIT. MIT Game Lab is creating Play Labs to accelerate start-ups in AR, VR, and AI. That gaming is an early focus of many of these efforts should surprise no one; gaming has historically been popular for trial runs and inspiration for much of today’s technology.

The spread of VR/AR is changing the way consumers and employees interact with the world, and AI and machine learning are allowing enterprises to deeply personalize their services. Business leaders need to ask themselves how AI combined with virtual and augmented reality can transform their business.

  • Compiled by John P. Desmond, AI Trends editor