Next Generation virtual worlds – what opportunities may they bring? Human-Machine Interaction, Extended reality, digital twinning.

Publication date: April 16, 2024

For now, it is quite obvious, that advanced technology such as virtual worlds and metaverses is bringing a lot of opportunities in various fields. Education, health, manufacturing are only a few examples, where the technology of virtual worlds may be applied. To understand better the deeper meaning within these opportunities, it is worth to get to know how these developments will affect human-machine and human-human interactions.

Human-machine interaction

Next generation virtual worlds will connect various types of technologies – augmented and extended reality (VR, AR and XR) with digital twin and blockchain. These connections will formulate new types for the development of immersive environments interconnected with the real world. In this context, things such as Human-Machine Interaction (HMI), cognitive psychology and brain sciences are trying to understand how humans interact with and within the immersive environments and how those interactions impact human behaviour in both virtual and real worlds. Existing research indicates that virtual worlds have ability to change the way we connect, perceive and experience the world around us. Research in HMI includes two aspects: the interaction of the user with the interface and the impact of this process on the user’s behaviour. When it comes to virtual worlds, this process becomes more complex. The human experience associates not only with the virtual, real-time, multisensory interactions in realistic immersive simulations of the environment but also with the hybrid social environments such as AI-powered human-likes avatars.

From a HMI perspective, current research focuses on the input the system receives from humans, decisions the system generates (based on human input, DALL-E is a great example for this), the ways in which these decisions are communicated towards humans and the impact they have on our behaviour. With next generation virtual worlds, however, several emerging areas must be considered in order to better understand the cognitive, psychological and behavioural processes of the user.

HMI emerging areas

• Inputs the system perceives from humans

The key limitation of existing input devices (e.g., keyboards and touch screens) is that they cannot free the users’ hands and accurately reflect body movements. Recently, researchers have begun to study freehand manipulation that allows more intuitive and concrete interaction in immersive and hybrid environments. These emerging techniques often rely on computer vision and brain-computer interfaces.

• Interactivity of the system

Interactive technologies form the user experience in the virtual world, enabling users to create and exchange content, digital assets in virtual worlds, and move between different virtual locations. It is expected that the complexity of the emerging inter-connected systems will bring about new possibilities for human-machine interaction.

• Immersiveness of the system

Related to features such as the embodiment (users’ tendency to perceive the virtual body they control as their own biological body and their actions in VR as their own actions in real life), the interactivity, the navigability, sense-ability (users’ ability to sense, e.g. touch, smell, hear and taste in the virtual environment) and create-ability (ability to create aspects that do not exist in the physical world and recreate existing aspects of the physical world to diminish negative and enhance positive aspects of it in VR).

• Affordances that support collaboration

Particular choices in design can improve the quality of collaborative experiences among individuals from various geographical locations. This includes establishing a digital environment where individuals from different real-world locations can interact, cooperate, and engage in activities like shared discussions and simulated experiments to replicate real-life scenarios.

• Device-dependent and device agnostic behaviours

Although the devices facilitating entry into virtual realms appear to significantly influence human encounters and engagements, the human mind can readily acclimate to prevailing interfaces. The majority of individuals can become entirely absorbed within a simulated 3D setting, even in the absence of exclusive apparatus, relying solely on screen-oriented visual composition. Nonetheless, comprehending this human capacity remains an ongoing pursuit.

• Sense of human presence in multiple interconnected virtual worlds

Presence denotes an intricate psychological sensation of ‘being present’ within virtual reality (VR), encompassing the perception of physical existence and the capacity to engage and respond as though the user were physically situated in reality. In accordance with the Embodied Social Engagement hypothesis, avatars can effectively serve as intermediaries for interpersonal engagements within virtual domains. Within the framework of embodiment, the emergence of distinct modes of communication and interaction fosters a sensation of presence that emanates from human cognition, akin to genuine real-world interactions. At the heart of this theory lies the premise that in virtual environments, users—represented via avatars—initially establish a sense of their avatars’ existence, which subsequently evolves into a shared existence through avatar interactions with others, culminating in the cultivation of a sense of communal presence within the virtual sphere.

• Representation of the self with avatar(s)

Virtual worlds serve as incubators for novel social frameworks, within which individuals have the capacity to exhibit their distinct characteristics in digital domains, making deliberate choices concerning their avatar’s portrayal and the extent to which these attributes mirror their actual selves. The development of identities featuring unique digital imprints, forethought, and coherence can either maintain anonymity or establish links to the individual’s real-life persona. Conceivably, each individual can opt to showcase their preferences or personify themselves through a multitude of avatars, thereby enabling the presentation of numerous identities and personas per person. Within this context, it becomes intriguing to explore the types of experiences people will seek, the transformative impact of socialization within virtual realms on our interactions in the physical world, the evolution of occupational dynamics, and the extent to which individuals will invest in nurturing their virtual identity or identities.

The above-mentioned developments on the interaction of humans with and within virtual worlds, which blur the line between virtual and physical world, create the potential for a transformation in various fields of application.

Education and training

In the realm of the European Commission priority ‘A Europe fit for the Digital Age’, the Digital Education Action Roadmap (2021-2027) constitutes a strategic endeavour that outlines a shared aspiration for delivering top-notch, comprehensive, and easily accessible digital education across Europe. Its objective lies in bolstering the alignment of Member States’ educational and training systems with the demands of the digital era. Within this framework, advancements in technology and emerging innovations are under scrutiny to discern their implications within the realm of education.

Taking as a starting point that students’ learning is highly correlated with the associated tools for the scaffolding of their learning process, we hypothesise that next generation virtual worlds are expected to have a significant impact on education and training. Prior studies dissecting the attributes of immersive and interconnected virtual settings unveil the prospective prospects across diverse age brackets and academic disciplines. Majority of studies that explore the use of immersive technologies in education mainly focus on higher education and mostly elaborate on subject matters related to natural sciences, mathematics and engineering followed by studies on the role of immersive technologies on the arts and humanities. It is expected that the next generation of virtual worlds will integrate the variety of experiences that currently are connected to specific tools and modalities. The incorporation of cyber-physical encounters via varied methodologies is anticipated to introduce gamification elements into the educational landscape, thereby realigning educational objectives toward competency-focused learning.

A recent review study conducted and released by the European Commission (2023) provides a condensed overview of the possibilities offered by eXtended reality (XR) within the context of its application in the realm of education. More specifically, it refers to the following opportunities:

• Companies and organisations can employ XR for procedural training

in a cost-efficient way.

• XR allows training for the development of soft skills.
• Awareness-raising can be more effective using XR.
• XR allows for the expansion of creative opportunities in art and design.
• XR aids the development of computational thinking skills.
• Collaboration among students becomes more effective using XR.
• XR simulates a range of physical training.
• XR makes language learning more effective, engaging and accessible.

Highly social and immersive next generation virtual worlds have the potential to provide new directions for education that are different from those based purely on VR or AR.

Below, there are key opportunities that might emerge with next generation virtual worlds in relation to education and training:

• New forms of social learning

Human learning thrives through interpersonal engagements. Anticipated forthcoming virtual realms are poised to unveil fresh channels for social interaction, particularly within the framework of interconnected educational settings. These environments hold the potential to enhance prevailing modes of social learning. Within virtual spaces, educator-to-learner and peer-to-peer knowledge sharing can unfold unhindered by the constraints of physical boundaries. Such a context could potentially reshape existing educational curricula, amplifying emphasis on nurturing students’ global citizenship proficiencies.

• New forms of learning through experimentation and play in cyber-physical environments

Human learning is inherently facilitated by exploration, hands-on experimentation, and playful engagement. Tailored learning settings hold the potential to present novel avenues for project-centric exploration, innovation, and collaborative sharing, emerging as innovative educational instruments to bolster novel pedagogical approaches. These environments are poised to catalyze student motivation and stimulate creative aptitude. Meticulously crafted, they can ensure safety and alignment with developmental stages, enabling personalized learning journeys that hinge on individual curiosity. Simultaneously, these environments provide a controlled arena where students can undertake calculated risks, fostering growth within secure confines.

• New learning experiences and high immersion through virtualisation of connected spaces

Completely innovative encounters have the potential to be extended to students, thereby igniting their intrinsic drive to learn. Taking the instance of acquiring a second language, the objective transcends a mere course or learning endeavour. Instead, it strives to immerse students in an English-speaking ecosystem that encompasses work, education, social interactions, and entertainment, mirroring the experience of native English speakers. Consequently, two distinct learning encounters, namely conventional virtual reality (e.g. a focused VR game with constrained interactions) versus immersive social virtual realms, exhibit contrasting characteristics.

Health, medicine and well-being

Virtual worlds and related eXtended reality (XR) experiences offer many opportunities, both for healthcare professionals and patients (European Commission, 2023). Surgeons are among the healthcare professionals that are likely to benefit the most from such experiences, which have the potential to help them in their work.

Firstly, they can improve their performance during surgeries and reduce human errors. XR could enhance pre-operative planning, as the ability to view in 3D may help surgeons or radiologists to assess a patient’s condition more accurately before undertaking a surgical procedure. They could also be implemented in real-time scenarios. For instance, augmented reality (AR) displays worn on the head enable the projection of virtual elements onto the physical anatomy of a patient during a surgical procedure. This eliminates the necessity for external monitors, streamlining the process, and diminishing the cognitive load and errors associated with shifting contexts.

Moreover, hands-free image presentation and capture prove particularly advantageous in surgical settings, upholding sterility protocols and preventing infections. This approach also holds relevance in high-pressure situations like emergency surgeries, where less seasoned surgeons can receive remote guidance from more seasoned colleagues.

Secondly, virtual environments might also benefit medical analysis and diagnosis. Improved high-resolution 3D microscopic images of patients’ anatomy can enable a better understanding of data and diagnostics, leading to more accurate treatments. Patients can be diagnosed while observing them in virtual environments. For instance, VR devices can help to tackle the difficulties of testing human visual conditions, by providing a mechanism to reliably assess the core visual functions with standardised stimuli. The potential for diagnosing mental health, neurological, and cognitive disorders is especially encouraging. Interactive virtual environments provide a secure platform for simulating real-life scenarios, enabling the observation and treatment of patient behaviours and responses. Both VR and XR have already been used to diagnose and/or treat addictions, phobias, social anxiety, post-traumatic stress, psychosis, schizophrenia, attention-deficit or hyperactivity disorders, autism, eating disorders, and obsessive-compulsive disorders, Alzheimer’s and Parkinson’s disease.

Similar to various other domains, virtual worlds offer an avenue for enhanced medical skills training. For instance, extended reality (XR) provides surgical trainees with lifelike exposure to a 3D operating theatre and the opportunity to explore 3D models of human organs. This technology allows for practical training in real clinical procedures without jeopardizing patient safety. Moreover, XR in medical education has demonstrated privacy enhancement and reduced potential for embarrassment during learning, enabling students to practice on virtual patient simulations. The benefits of XR extend not only to healthcare professionals but also encompass patients and caregivers. XR presentations of medical procedures empower patients with greater awareness, confidence, and health literacy, fostering improved doctor-patient relationships and informed decision-making, such as consenting to surgical interventions. Caregivers often face emotional challenges related to a family member’s illness. XR can simulate the experience of living with specific medical conditions (e.g., dementia, visual impairment), cultivating empathy among caregivers.

Regarding therapies, immersive experiences have demonstrated the ability to alleviate acute and chronic pain such as that associated with burns, phantom limb and cancer. VR applications distract and entertain, blocking out noises from the physical world which may result in patients perceiving less pain. VR has gained significant recognition for its application in exposure therapy, a widely acknowledged method for addressing mental health conditions. This approach yields remarkable advantages for treating anxiety, eating disorders, substance use disorders, and post-traumatic stress disorder. Exposure therapy entails systematic and recurring exposure to anxiety-inducing triggers, leading to transformative shifts in patients’ thoughts, actions, and emotional and physical reactions. Similarly, XR interventions can be useful for people with neurological conditions, such as neurodevelopmental disorders and dementia, and persons recovering after a stroke or a traumatic brain injury.

Manufacturing

The emergence of virtual worlds within the framework of Industry 4.0 holds immense potential for enterprises, ushering in improved industrial operations, the creation of fresh employment prospects, and overall economic advancement. To illustrate, as outlined in a comprehensive 2022 market analysis by The Brainy Insights, the global metaverse market is anticipated to undergo substantial growth, escalating from €36.26 billion in 2021 to a projected €918.23 billion by the year 2030. The industrial applications emerging from next generation virtual worlds are expected to extend across various sectors, including logistics, engineering and manufacturing, and are already providing real world benefits for companies. This comprehensive integration leads to improvements in design and simulation processes, augmented operational efficiency (e.g., predictive maintenance, quality control, and supply chain management), and facilitates virtual training and remote collaboration. Furthermore, these applications have the potential to foster a more sustainable European industry by facilitating resource optimization and facilitating the adoption of innovative business models.

Digital twinning methods are also finding application in industrial contexts. These “digital twins” are virtual duplicates of tangible objects or systems, serving as tools to refine product design, elevate manufacturing methods, and enhance overall efficiency. Through the amalgamation of real-time data (sourced from cyber-physical systems and the Internet of Things) alongside sophisticated analytical tools (such as artificial intelligence and big data), digital twins empower predictive maintenance, continuous monitoring, performance enhancement, and decision-making based on simulations. This facilitates cost reductions and heightened productivity.

To leverage the capabilities presented by the next generation virtual worlds, it is imperative for businesses to embrace innovation, redefine their business models, and provide adaptable, personalized encounters for consumers. Nurturing the advancement and expansion of industrial virtual worlds plays a pivotal role in cultivating fresh avenues for business growth. Concurrently, ensuring the broad adoption of correlated technologies throughout industrial networks is of paramount importance, as it will catalyse efficiency enhancements and offer support to small and medium-sized enterprises (SMEs) in their endeavours toward sustainability.

Upcoming endeavours must fortify industrial virtual worlds by fostering a vibrant marketplace for emerging solution providers and heightening awareness among industrial users regarding the potential of these revolutionary technologies. Sustained involvement in ongoing conversations with industry participants will be pivotal to guaranteeing that European enterprises can wholeheartedly seize the available prospects.

Other fields

The impending surge of next-generation virtual worlds promises a paradigm-shifting array of prospects spanning education, healthcare, manufacturing, and public services. However, the scope of their adoption extends well beyond these domains, with numerous other fields and sectors poised to derive substantial advantages.

• Transport, hospitality and tourism

Virtual worlds possess the capability to democratize travel, hospitality, and tourism experiences by dismantling geographical and socio-economic limitations. This enables individuals who might lack the resources to physically journey to distant locations to virtually explore and encounter these sites. While virtual worlds will not supplant authentic hospitality and tourism encounters, they could reshape how we engage with associated products and services. For instance, travellers can enhance their trip planning by initially exploring virtual depictions of hotels and destinations. They can assess accommodations, amenities, and even engage with virtual guides. This aids in making well-informed choices and gaining a clearer perspective of what lies ahead before embarking on their actual journey. Engaging in activities like traveling, attending meetings, concerts, and visiting museums within virtual worlds has the potential to foster sustainable tourism by mitigating the carbon footprint linked to physical travel. Virtual travel experiences offer a viable alternative for individuals seeking to explore diverse destinations while avoiding greenhouse gas emissions. This approach can aid in offsetting the environmental effects of tourism and encourage the adoption of eco-friendly practices.

• Cultural heritage

The capacity of virtual worlds to conserve and reshape cultural heritage has gained widespread recognition. Virtual worlds can function as expansive digital archives, facilitating the conservation and availability of artworks, historical sites, and other cultural artifacts. Individuals from around the globe can virtually delve into and encounter cultural heritage that might be physically unreachable or susceptible to degradation.

The forthcoming era of virtual worlds will introduce avenues of heightened interactivity, including interaction with virtual renditions of historical figures, participation in reenactments of significant events, and even the simulation of experiences from various time periods. This augmented interactivity amplifies the educational and experiential dimensions of cultural heritage, rendering it more captivating and unforgettable for users. Furthermore, virtual worlds can facilitate cross-cultural interactions and collaborations. Individuals from disparate regions can convene in virtual realms to exchange and celebrate their cultural heritage, nurturing comprehension and admiration for diverse cultures. Artists, historians, and experts have the potential to collaborate on virtual undertakings aimed at reconstructing cultural artifacts that have been lost or harmed, amalgamating their insights and proficiency. Augmented reality warrants special consideration, as it presents a means to access cultural heritage content and information within real-world locales through overlays, delivering a layered encounter that seamlessly intertwines virtual and physical surroundings. This integration can enhance site visits to historical landmarks and museums, providing contextual information and interactive elements. An interesting recent initiative for virtual cultural heritage is ‘Time Machine Europe’, that focuses on the creation of AR/VR applications to simulate spatiotemporal 4D reconstructions with the aim of mapping the European social, cultural and geographical evolution across times.

• Labour market

Virtual worlds hold the potential to revolutionize remote work, elevating it to unprecedented levels. By crafting virtual workspaces, individuals can engage in collaboration and communication with colleagues across the globe, introducing fresh avenues for global talent recruitment and fostering greater work-life balance flexibility.

Moreover, the development of virtual environments necessitates a novel cadre of skilled professionals, ranging from virtual architects and designers to virtual event planners. This shift could usher in new employment roles and opportunities. Another advantage for the labour market lies in augmented collaboration and creativity. For instance, remote teams can amplify innovation by utilizing shared virtual spaces for brainstorming, ideation, and prototyping. Consequently, new virtual economies may emerge, enabling individuals to monetize their creativity and skills within virtual domains. This, in turn, could give rise to innovative business models and entrepreneurial ventures.

It is essential to underscore that virtual work possesses the capability to dismantle physical barriers, fostering inclusivity for individuals with disabilities, limited mobility, and diverse neurological profiles.

• Retail and customer experiences

The future holds the promise of immersive and socially interactive virtual shopping encounters. Within these experiences, customers will have the ability to explore virtual stores, try on virtual clothing and accessories, and engage directly with products before finalizing a purchase. Moreover, customers will be able to partake in social shopping escapades by virtually congregating with friends, family, or even strangers while perusing merchandise, exchanging opinions, and seeking advice. This social dimension aims to replicate the real-world shopping experience.

Another advantage lies in augmented product visualization. Customers will be empowered to perceive products in 3D, investigate various angles, and interact with virtual models or simulations. This heightened visualization will enable customers to make more well-informed buying choices by offering a comprehensive understanding of product attributes, functionality, and aesthetics.

Brands, too, will capitalize on this trend by crafting virtual experiences that extend beyond traditional retail settings. They can host virtual events, launch new products, or provide exclusive content and promotions. These immersive brand experiences will not only intensify customer engagement but also foster a deeper and more meaningful connection between customers and brands.

• Entertainment

The forthcoming users of virtual worlds are poised to actively partake in a plethora of VR/AR games, concerts, movies, virtual gatherings, sporting events, and a myriad of other experiences. They will be endowed with the ability to craft and distribute their own content, thereby contributing to the evolution of virtual worlds. This encompasses designing personal games, experiences, and sharing them with fellow users. This democratization of content creation ushers in a broad spectrum of entertainment choices, encouraging artistic expression.

The interconnection of various platforms and devices will foster seamless integration across diverse media and entertainment outlets. As an illustration, a film could be complemented by related tie-in games, merchandise, and interactive encounters, establishing a comprehensive entertainment ecosystem that forges fresh avenues for revenue generation.

Virtual worlds possess the capacity to heighten accessibility in entertainment, particularly for individuals with physical disabilities or residing in remote locales. They will be able to access and partake in entertainment experiences that were previously limited to specific geographic locations or physical capabilities.

Moreover, cross-cultural interactions will enable users from different regions and backgrounds to unite and share their cultures and perspectives. This convergence will engender a diverse and all-encompassing entertainment content that resonates with a global audience.