Extended Reality, short: XR, has been a fascinating topic for experts and the interested public alike since the first XR headsets appeared. Recently, the closely linked vision of the metaverse hit the headlines, giving XR another boost. Eurescom message editor-in-chief Milon Gupta asked Prof. Dr. Didier Stricker, Head of the Augmented Vision department at the German Research Center for Artificial Intelligence (DFKI), what is behind the hype.

What is XR?

Didier Stricker: eXtended Reality can be understood as a hybrid real-virtual world at the convergence of Virtual Reality, Augmented Reality, and Wearables. The interaction of the user with the extended virtual-digital world plays a central role. The goal is to fully understand the user action and intention and, at the same time, to update the virtual representation of the real world so that it provides a consistent representation of it. eXtended Reality is the interface between the Physical World and its Digital Twin; XR fuses those two representations into one world, called eXtended reality.

How will XR technologies change the way we live and work in the future?

Didier Stricker: Our smartphones are today constantly with us. They can be considered as super-computers equipped with many sensors which accompany us in our everyday life, at work or at home. The information they provide is, however, still represented on the display, and interaction is limited to touch and voice. XR represents the next step: the deluge of data, understanding and interpretation through artificial intelligence, advanced sensing, and new seamless display technologies, such as XR glasses, will allow modeling complex facts and behaviours. XR will be ubiquitous and provide the user with the right information at the right time at any place. This will change the way we work, learn, and communicate.

What are the social and economic opportunities of XR for Europe?

Didier Stricker: Excellent research work is achieved in Europe in XR and related technologies. However, the main XR platforms are today provided by the big US IT companies. So, the focus should be on vertical applications related to the existing European industry, like machinery, automotive, or medical devices, and key social areas, such as education and demographic change. With very advanced expertise, as it is available in Europe, new modules or building blocks encompassing software and hardware can create large markets. So, Europe is well positioned, but investments related to innovation and start-ups, together with strong partnering with large industrial groups, will be necessary to create the required critical mass.

Which XR technologies and applications are you currently exploring at DFKI?

Didier Stricker: We focus today on two main areas in XR. First, the understanding of manual workflows such as assembly work in the industry. Our system is able to recognize and localize assembly parts in 3D and at the same time identify which is the assembly step in a complete assembly workflow. The interesting aspect is that the system is trained using CAD data only. No pictures of the real object are necessary. So, those new XR applications can be automated and deployed easily. The second technology we look at consists of the visual capture of the hands of the user from the camera mounted in the XR-glasses. We would like to extract a precise hand motion and interaction with objects. The idea is to digitalize human interaction with our surroundings and in this way to capture the expertise of people, for example surgeons, and out of it create new training simulators.

What are the toughest challenges in the development of XR technologies?

Didier Stricker: Several challenges are remaining, including real-time reality capture and human motion tracking. However, enormous progress has been achieved recently thanks to artificial intelligence, more precisely thanks to deep-learning techniques. Key will be at the end excellent ergonomics of the hardware and user acceptance, which will depend directly on the way we design our application. XR apps should support users, not distract them or even lead to unwanted effects such as a lower learning rate or full dependence on this kind of XR assistance.

Which XR applications do you expect to have the highest impact in the next five to ten years?

Didier Stricker: That is a good question! I think that remote collaboration will be the next key application of XR. Such applications can be implemented efficiently today and offer a real added value compared to simple teleconferencing or audio support. The return of investment can be proven easily, and no major barriers in terms of infrastructure are existing. The creation of meaningful supporting content and the design of an appropriate user experience will be key. Then, I do think that consumer apps will play a fundamental role. Examples are virtual try-on, virtual visit of an apartment, or virtual shopping of a car. The key here will be the quality of the user experience, which assumes first a perfect quality in terms of visualization and realism.