Why scientists are delving into the virtual world

Visiting Stephen Hilton’s laboratory is a hands-on experience. No staff passes, qualifications or lab coats are required here; guests are welcome to wander around and prod at the equipment, even spill chemicals on the floor if they fancy it, without repercussions. In fact, he encourages it. “Science should be interactive. It’s about being able to make a mess, make mistakes,” Hilton says, handing over a 3D shape he has just scribbled in mid-air.

Hilton’s lab is a digital one, experienced through virtual reality (VR) headsets, so the chaos being created is benign. But the environment mirrors the space and design of the real-life wet lab he runs in the School of Pharmacy at University College London — where his group researches synthetic chemistry, including drug discovery — albeit with brighter colours and some supernatural add-ons.

How virtual reality is helping to boost scientific engagement in rural Africa

The VR concept has existed since the 1960s, but any lingering perceptions of the technology as faddish — a toy with clunky graphics — are long outdated¹. In the past couple of years, accelerated by the COVID-19 pandemic and thanks in part to the falling cost of headsets, more researchers have been exploring the potential uses and benefits of incorporating extended reality (XR) into their work — including VR, augmented reality (AR) and mixed-reality technologies.

Brands such as Oculus Rift and Microsoft’s HoloLens made VR popular through gaming; now, Hilton and other researchers are deploying the same tools to monitor experiments, collaborate with international research partners and conduct immersive training programmes for students and colleagues.

Hilton uses the Oculus Quest 2 headset (now sold as Meta Quest 2) because it is relatively low in cost (around £300, or US$380) and allows him to have multiple headsets on the go at one time. The software also runs on a PC, enabling PC–headset interaction as well as PC–PC or headset–headset combinations. His virtual lab was built using the 3D design software Unreal Engine from US developer Epic Games, and runs on a local server for single users, switching to a cloud-based one for multiplayer and voice-connectivity programs.

During the pandemic, Hilton’s lab members found themselves struggling to conduct productive meetings with research partners in Germany and the United States using video calls alone. “At that time, existing commercial programs were expensive — between £20,000 and £70,000 — and not flexible to our needs, so we had to start from scratch,” Hilton explains. Now, his virtual lab features a 20-seat board room; in the real lab, the same space is a humble storage cupboard.

These days, he uses the VR environment mostly to train students, who can practise running experiments or complete assignments that are accessible either through a laptop or through VR headsets at home and on campus. Three AI assistants — life-sized, cartoonish human-like figures that respond when users speak to them — are on hand to offer support. Each assistant has a dedicated role and backstory, and draws on a programmed knowledge base as well as on the artificial-intelligence large language model ChatGPT to guide users around the virtual space and answer questions about safety, inventory and stock. Private customers, which include companies and research institutes, must pay a fee, bringing in cash for future lab projects. But Hilton’s team also uses the VR environment for educational outreach, hosting visits from secondary-school groups and collaborators from many countries in the global south.

This is helping to make the science more accessible: where once Hilton and his team could supervise only a handful of chemistry students and lab members at any one time, now, limitless numbers of learners can practise experiments, safety assessments or other scenes at home or through partner institutions elsewhere, he says. It also means Hilton spends less time repeating instructions or telling students where to find equipment, freeing him up to focus on other projects, including an expansion of the VR research programme.

“Just recently, we’ve created a tool for the virtual assistants to speak ten foreign languages,” he says. “The knowledge base translates in real time, meaning I could speak directly in English to answer a student’s query, and they’ll hear the answer in their local language. We see a big opportunity for this in many countries in the global south.”

Diverse avatars

One of the simplest ways to make virtual training more accessible could be by avoiding spoken languages altogether. “XR opens up the ability to use gestures and body language — showing over telling — without having to know the language very well,” says Koos De Beer, an XR solutions consultant and researcher at the University of Pretoria in South Africa.

Virtual-reality applications give science a new dimension

De Beer started using XR-based training tools for students at the university in 2010. Since he began working with the university’s Department of Mining Engineering in 2018, he has used the tools to enable students to ‘visit’ mines, tunnels and other potentially dangerous spaces remotely, through a virtual classroom. In recent years, his work has expanded to deliver VR programs more widely across engineering and in health care, particularly in poor areas of rural South Africa.

The avatars used to represent VR users “don’t have to look or speak a certain way,” says De Beer. They can simply be shown as robots, for instance, without the need for race or gender — although human biases present challenges when designing the figures. (For now, Hilton says, his program uses robot avatars that have masculine, muscular torsos, for example.)

Training tool

VR is also helping medical students to access learning opportunities sooner in their training. Simran Sharma, an obstetrics and gynaecology clinician at Cardiff University and the University Hospital of Wales in Cardiff, UK, was troubled by the lack of exposure to real-world practice in her early medical career. “The first time I ever came across a sepsis patient was as a qualified doctor — and that’s a common scenario for so many serious and rare conditions,” she says.

Using medical dummies to teach students how to diagnose and treat sepsis is considered the gold standard for training, she explains, but the pandemic made in-person training difficult. “It is also very expensive and resource-rich when you take into consideration the cost of the dummies — around £40,000 — the trainer’s time and the fact it is confined to specialist centres. And if you happen to miss a day’s training because you’re sick, it’s tough luck,” she adds.

Sharma was inspired to change this in 2019 after witnessing pregnant people become dangerously ill from sepsis during labour. While a research fellow at Cardiff University, she helped to develop a VR education tool for sepsis recognition as part of Project Sepsis, a UK research collaboration across medicine and science. Its VR project works by showing students a series of simulations, demonstrating the differences between sepsis symptoms and similar conditions. Unlike Hilton’s avatar-populated virtual lab, Sharma opted to film actors playing out realistic medical scenes, which students can watch and interact with using VR headset and handsets. The students are then tested on the knowledge gained during the experience through on-screen multiple-choice tests.

“Virtual reality offers a repeatable, standardized clinical training method that is not reliant on the quality of the trainer. It is evidence-based and has been shown to be a powerful tool in health care and its use is growing worldwide,” Sharma says. (See, for example, ref. ².)

Boosting productivity

Alongside VR’s potential for training and education, many scientists who use the technology hope that it will one day make their practical work more efficient.

In 2020, chemist Lee Cronin and his colleagues at the University of Glasgow, UK, designed a VR programme that allows him to control the robots in his lab remotely. Cronin and his research group use what he calls a “chemputer” — a series of robots that can be programmed to synthesize organic molecules automatically — as part of their quest to discover new molecules and generate artificial life forms. This automation makes the chemical experiments safer by minimizing the need for human handling of chemicals, and much faster, because the robots work 24–7.

The addition of the VR technology allows Cronin to move components and give instructions to the robots, wherever he is. “It also allows us to make shortcuts,” he explains. “I can be holding the handset and have a virtual conical flask in one hand. If I make a pouring motion, the robot knows to interpret that as moving liquid from hand a to hand b, and so it can turn on a pump and a valve. It’s not fully deployed yet, but the operating system is there,” he adds. “It’s very similar to the way a flight simulator works.”

He thinks that VR used in this way could have an important role in many other potentially dangerous scientific scenarios, including nuclear decommissioning and bomb-disposal research. And there are productivity benefits, too: “We need to invent more molecules if we’re going to solve big societal problems, like finding new drugs to target diseases. But chemical space is really hard to research,” he explains. “There are only about 2,800 small molecules approved in the clinic [for drugs] — to get there, we must have tried several billion molecules already. Cures for cancers are out there, but we don’t physically have the capacity to go through each chemical reaction one by one ourselves.”

According to Cronin, his Glasgow-based spin-out company, Chemify, has raised more than £33 million from international investors who have backed his plan to automate the design of new molecules to speed up drug development.

Smart mirror

The ability to visualize oneself inside an alternative reality is inspiring new avenues of research. Neuroscientist Olaf Blanke directs the Laboratory of Cognitive Neuroscience at the Swiss Federal Institute of Technology in Lausanne (EPFL). After years of research into prosthetics and cognitive therapies for people with amputations, he thinks that VR technologies are revolutionizing scientific understanding of the human brain and own-body perceptions.

“VR is essentially a very smart mirror,” he says. “As adults, it’s very easy for us to decide when we’re brushing our teeth that we’re not the image we see in the mirror — it’s a reflection. But once you have an avatar, you can play around with lots of different perspectives. It allows us to ask new questions about the visual self and self-consciousness.”

Overcoming the obstacles to invention

It’s possible that VR technology could one day replace, or at least improve, conventional methods for studying memory and trauma, he says. “Something like episodic memory used to be tested in the lab environment by asking participants to memorize a word list, which had obvious limitations. Through VR, we can recreate scenic 3D environments and then test memory alongside additional data” — such as tracking eye movements, heart rate and other vital signs.

He has also been experimenting with VR to study out-of-body experiences — a phenomenon estimated to be experienced by 10% of the global population, the reasons for which are not fully understood³.

Blanke hopes that furthering understanding in this field will lead to new and better forms of treatment for people who have neurological disorders. But there are still technological limitations. VR headsets can get heavy after a while and make users feel nauseous. And recruiting enough programmers is always a challenge, he says. ”It would be great to have more software that was easily adaptable for non-programmers, too. But scientists are a very small fraction of the market and so I don’t see that changing any time soon.”

Not so simple

Developing VR programs is no easy feat. When Hilton was planning his own virtual lab — a project that started in 2000 but was spurred on by the pandemic in 2020 — he says he was “a bit giddy with hubris”. He adds: “Having developed our own 3D printing software and PC software for digital flow, we thought that we could easily make our own VR software, too. However, it proved harder than we had initially imagined.” It has taken many years to bring Hilton’s software to a quality he is proud of, with plenty of help from his postdoctoral researchers and their 3D computer-aided design (CAD) skills.

“We have around ten people in the lab with a mix of backgrounds — biomedical, pharmacy and chemistry degrees, but no computing — and we train everyone from the ground up. First in CAD design of real equipment, then AI programming and then CAD to VR. All [lab members] run the sessions with users, so they then get used to VR training of others,” says Hilton. But not every lab has such time and resources.

For the most part, it seems the widest practical use for VR in science and research will be as a tool for training. There are clear opportunities for educational outreach, scaling up of courses and improving accessibility. But it is no silver bullet: barriers to access remain, despite the relatively low cost of headsets.

Such programs depend on a fast Internet connection, something many regions lack. In 2021, there were still 2.9 billion people without Internet, according to the United Nations. In South Africa, for instance, even major cities struggle with connectivity as a result of energy blackouts, and Internet access is never a given in rural areas. For De Beers, that means ensuring all the programs he helps to develop are designed to work with mobile-phone data and don’t rely on broadband Internet. But even this can be problematic, he says. “We often need to plan ahead to take in our own connectivity sources,” he says, for instance using satellites — although access to aerospace firm Space X’s low-latency satellite internet, called Starlink, has made this easier.

A lack of understanding around VR technologies also remains a fundamental barrier to wider adoption. As Sharma explains: “For many of the academic professors we demonstrated our product to, VR was seen as completely weird and unnatural. But for 20-something medical students we work with, it is really no big deal.”

For the cultural shift to accelerate, universities and other centres of learning can take a proactive role. “A lot of our work is also about exposing people to the technology — getting them used to it, because there’s no question that they will have to work with it at some point in their career,” De Beers explains. To this end, his team at the University of Pretoria have set up “the XR toy box: a unique space where students and staff can drop in and play with technology. It’s an exposure tool,” he explains.

Is VR an excuse for scientists to play with toys in the lab? “Of course,” Cronin admits. “But it’s also a lot more than that. Humans have always used tools to make new discoveries, and this is just the next tool in the box. I don’t think it takes away from human knowledge. But it will certainly change the nature of work, and hopefully lead to a better way of working.”