With the tap of a keyboard, Ryuji Hirayama brings a listless foam bead to life. The white speck jumps up and hovers with perfect stillness in space. Another tap, and the dot transforms into a luminous butterfly-like shape, which flaps its wings as it circles inside a black box. Diego Martinez Plasencia, Hirayama’s colleague at the University of Sussex in Brighton, UK, reaches into the box to show that there are no strings. The effect seems to be pure magic. “I showed it first to my daughters. They were like — ‘wow’,” Martinez Plasencia says, his eyes widening in child-like delight.
Behind the mid-air metamorphosis is a relatively simple set-up. Two slender arrays of 256 tiny speakers above and below the bead move it by generating ultrasound waves. The object darts so fast that all the eye sees is a continuously evolving 3D image a few centimetres across, drawn in the air as if by a high-speed Etch a Sketch. The same ultrasound speakers that create the image can also generate audio and tactile sensations. Reach towards the butterfly, and your finger might feel a flutter. In another case, a smiley face appears, accompanied by the strains of Queen’s ‘We Will Rock You’. Remarkably, most of the components used to create these effects are off-the-shelf.
“It’s an elegant and exciting platform,” says Daniel Smalley, a physicist at Brigham Young University in Provo, Utah, who last year unveiled a similar technique, using lasers to steer around a fleck of cellulose to produce images1. Until now, few physicists thought it would be possible to use sound to move a bead fast enough to create such a display, he says. In August, Tatsuki Fushimi, a physicist at the University of Bristol, UK, and his collaborators became the first to show that it was feasible. But their bead took longer to trace out shapes, meaning that only images smaller than 1 centimetre across could appear as a single, continuous object2. The Sussex team’s work is “a piece of engineering that makes us believe things we didn’t think were possible,” says Smalley.
The acoustic device, described in Nature3 on 13 November, is the latest example of a 3D-image-generation technology known as volumetric display, which differs in fundamental ways from technologies such as holograms, virtual reality and stereoscopes. Those more-familiar approaches use tricks of the light to create the illusion of depth, and can be life-sized and photorealistic. But holograms can be seen only from certain angles, virtual reality and stereoscopes require headgear, and all these techniques can cause eye strain. Free-space volumetric displays, by contrast, use lasers, electric fields, fog projections and other approaches to create truly 3D images that viewers can see from any vantage point. In that way, they’re the closest any display technology has come to Princess Leia’s SOS message in the 1977 film Star Wars.
Research into volumetric displays is even older than that film. And the approach has a crucial advantage over holograms, because it requires much less computational power. But despite decades of effort, free-space displays are still limited to small, crude drawings, and they have struggled to get off the ground commercially, says Smalley. Still, he is hopeful that work bringing together different and more-practical technologies, including acoustic levitation, will help volumetric display to find its killer app. It might be used in detailed interactive mock ups for medical trainees, perhaps, or to give people the ability to chat with distant relatives rendered in 3D. And the Sussex team’s acoustic method wouldn’t necessarily require a long phase of development to make its way out of the laboratory, says Smalley. “I would make a bet on this technology becoming commercial before many of the other technologies we work on.”
Lights and sound
Volumetric displays that are already on the market usually work by making 2D images 3D. The Voxon VX1, for example, projects photons onto a screen that vibrates rapidly up and down. Timed right, this creates a 3D image without the need for special eyewear. But the display’s complex mechanical parts mean it is locked behind glass and has so far found only niche uses, such as in museum displays.
In 2006, Hidei Kimura made one of the first attempts to draw images directly onto 3D space4. Kimura, chief executive of Burton, a firm based in Kawasaki City, Japan, and his academic collaborators developed a technique that used a laser to knock electrons off air molecules, causing them to glow. By moving the laser’s focal point at high speed, they could create luminescent dots of plasma that build to make a coarse image. “With nothing, we can create 3D images directly in the air,” says Kimura, who envisages using the technique to broadcast emergency information into the sky, or to project 3D replays above the pitch at a sporting event.
The plasma technique creates relatively stable images, but it faces some major limitations: it is low resolution (one laser blast equals one point in the image) and the laser is so intense that it could cause burns, says Yoichi Ochiai, a computer scientist and artist at the University of Tsukuba in Japan.
In 2016, Ochiai’s team adapted the plasma technique, using a lower-energy, shorter-pulse laser to make images that are safe to touch5. At a few millimetres across, the images are much smaller than the Kimura team’s. But by using lasers that pulse at a higher frequency and modulators to shape the beam into multiple focuses, the team can ramp up the resolution to 10–200 times higher than that used in Kimura’s work. This allows them to create more complex images, such as pinhead-sized fairies.
At Sussex, the acoustic 3D display started with another stalwart of science fiction: the tractor beam made famous in the 1960s television series Star Trek. Since 2012, Sriram Subramanian, who leads the team, has pioneered ways of crafting sound waves to create points of high pressure that can trap and move small objects6. But it was only when Hirayama came to the laboratory in 2018 that the team found a way to use sound to create images.
To appear as a solid picture, a bead needs to create each image frame in less than one-tenth of a second. Until now, acoustic levitation has tended to focus on holding objects as steady as possible; motion happens relatively slowly in a stop–start fashion, from one stable point to another. Hirayama’s innovation was to kick the bead before it comes to rest, calculating each new target point inside hardware designed especially for the computation. This meant that the team could change the field’s focus 40,000 times per second. The bead reaches speeds as high as 8.75 metres per second, which “looks like teleportation” when a 2-millimetre-wide bead crosses a few centimetres of space, says Hirayama. As the bead moves, a rapidly changing LED bathes the display in light to create colour.
The team was inspired by Smalley’s work using lasers to move and illuminate a speck of cellulose plant fibre1. Using the same number of particles and data, Smalley says his images are only one-tenth the size of the Sussex team’s, but have ten times the resolution.
The Sussex technique does have a drawback: it needs speakers on two sides of the display, which restricts a viewer’s ability to interact with the display and limits its size. But with hardware upgrades, Subramanian says it could be possible to use a different kind of acoustic wave to create images with speakers on just one side. The researchers are also working to improve their understanding of how the bead responds to the forces acting on it, which would allow them to move it faster, to draw more-complex images by levitating multiple beads at once and to integrate sight and touch more closely. In the current set-up, the tactile sensation and image don’t occur in exactly the same place, because the fields needed to create them can interfere with each other. Ochiai’s group has already found a way to bring together touch and sight by using fields that do not interfere: an acoustic field for tactile feedback and a laser to draw tiny images in plasma. The group has used the approach to draw braille dots in the air7.
Inevitably, any 3D display gets compared to Star Wars holograms. Sussex’s technique makes bigger images than previous similar methods and incorporates sound, so it brings us closer to recreating that, says Qiong-Hua Wang at Beihang University in Beijing, who works on 3D display devices. But the images are still tiny and far from photorealistic. Creating the kind of 3D effect in Star Wars by any means could take ten years, or even longer, she says.
But Barry Blundell, a physicist specializing in 3D technologies at the University of Derby, UK, cautions against trying to use volumetric technology to make rich, photorealistic displays. “No one would look at a sculpture and compare it to a painting,” he says. He adds that efforts to compete with holograms have often led to commercial dead-ends, and that the displays are best suited to applications that would be impossible in other media but don’t require high detail, such as interactive displays capable of showing complex 3D movements.
Interactivity could be powerful, Smalley says. Surgeons in training might use such displays to practise threading a catheter through the vessels of the heart, for instance. With one million moving particles, he adds, “you can have a disembodied face — do face-to-face telepresence”. Creating avatars of people in a space could give a stronger sense of presence than a photorealistic image seen through virtual reality, he says.
At the Sussex lab, a million-particle display seems a long way off. Only time will tell if the group’s approach will pave the way to such numbers. After showing off his sphere’s limited repertoire of tricks, Hirayama shuts off power to the speakers. The flapping butterfly vanishes, and the bead that created it drops and bounces gently on the display’s base. Hirayama picks it up and sets it in a box with hundreds of others, ready at any time to create magic in thin air.
Nature 575, 272-273 (2019)