Interactive Holographic Display Based on Finger Gestures

In this paper, we demonstrate an interactive, finger-sensitive system which enables an observer to intuitively handle electro-holographic images in real time. In this system, a motion sensor detects finger gestures (swiping and pinching) and translates them into the rotation and enlargement/reduction of the holographic image, respectively. By parallelising the hologram calculation using a graphics processing unit, we realised the interactive handling of the holographic image in real time. In a demonstration of the system, we used a Leap Motion sensor and a phase modulation-type spatial light modulator with 1,920 × 1,080 pixels and a pixel pitch of 8.0 µm × 8.0 µm. The constructed interactive finger-sensitive system was able to rotate a holographic image composed of 4,096 point light sources using a swiping gesture and enlarge or reduce it using a pinching gesture in real time. The average calculation speed was 27.6 ms per hologram. Finally, we extended the constructed system to a full-colour reconstruction system that generates a more realistic three-dimensional image. The extended system successfully allowed the handling of a full-colour holographic image composed of 1,709 point light sources with a calculation speed of 22.6 ms per hologram.

record and reconstruct the phase information of the light. This is problematic because the phase information significantly contributes to the resolution of the reconstructed 3D images 29 . On the contrary, holography can record and reconstruct phase information because it is based on wave optics. The electro-holographic images generated by Yamaguchi's interactive system 30 can be controlled by touch. However, this system cannot generate and reconstruct the electro-holographic images in real time. We have demonstrated an interactive electro-holography system in which the 3D image can be operated in real time using a keyboard 31 . However, using the keyboard as an interface to manipulate the images is non-intuitive. Hence, we have constructed an interactive system in which the observer can intuitively handle the electro-holographic image in real time through hand and finger movements, which are detected by a motion sensor (Fig. 1). This intuitive control mechanism provides a realistic, immersive environment for the observer. This type of intuitive handling of 3D images will likely be required in virtual-reality holographic displays in the future. In this paper, we demonstrate an interactive finger-sensitive system which enables an observer to handle electro-holographic images in real time using intuitive finger gestures. Figure 2a shows a schematic of the constructed system. First, the system detects the gestures of the observer using a motion sensor. Upon detection of a swiping gesture, the system rotates a 3D object composed of point clouds in a clockwise (leftward swipe) or anticlockwise (rightward swipe) direction. Here, the system performs the rotation process by multiplying the coordinate of each point by the rotation matrix. When the system detects a pinching gesture, the system enlarges (pinching out) or reduces (pinching in) the size of the 3D object. Here, the system performs the enlargement process by multiplying the coordinate of each point by the scale coefficient. Subsequently, the system calculates a CGH of the 3D object based on the rotation, enlargement or reduction operation. Finally, the system displays the CGH on the SLM and reconstructs a holographic 3D image by illuminating light to the SLM. The processing steps described above can be regarded as one frame of the electro-holographic image; thus, the interactive handling of electro-holographic motion pictures can be realised by repeating these processing steps. Figure 2b shows a schematic of the 3D object of a dinosaur composed of 4,096 point light sources. The 3D object was virtually placed 1 m away from the SLMs when calculating the CGHs. Figure 3 shows the experimental setup used to demonstrate the full-colour electro-holography reconstruction system. First, we used only a green laser (532 nm) and reconstructed monochromatic holographic images.   Figure 4 and supplementary movie M1 show the reconstructed images acquired using the constructed system. Figure 4a shows the reconstructed image being rotated by the observer using a swiping gesture. Figure 4b,c show the image being enlarged and reduced by the observer using pinching gestures, respectively. The average calculation time of the system was 27.6 ms per CGH, corresponding to 36.2 fps. Using only the CPU, the average calculation time was 1.39 s; thus, the parallelised system with both the GPU and CPU was 50.4 times faster than the system based on the CPU alone.

Results
As shown above, we successfully demonstrated that an observer can interactively handle the reconstructed image in real time using swiping and pinching gestures. The new interactive finger-sensitive system enables an observer to intuitively handle electro-holographic images in real time using finger gestures.

Discussion
To reconstruct more realistic 3D images close to an actual object, a system capable of full-colour reconstruction with red and blue light in addition to green light is needed. Thus, we extended the single-colour system to a full-colour reconstruction system to allow interaction with more realistic 3D images. Figure 5a shows a schematic of the full-colour interactive image reconstruction system. Using the single-colour interactive system described in the Results section, we only detected the swiping and pinching gestures. This indicates that we did not use the depth information detected by the motion sensor. We then implemented depth detection in the full-colour interactive image reconstruction system. This system detects finger touch gestures using a motion sensor and switches between the display and non-display of the red, green and blue components depending on the position of the finger gesture. Here, we set a virtual plane over the Leap Motion sensor and calibrated the depth position of the virtual plane as z = 0, as shown in Fig. 5b. The motion sensor detects the z-coordinate as well as the direction of the movement of the observer's finger position along the z-axis. The motion sensor regards the finger motion as the touch gesture only when the finger moves from the positive range (z > 0) to the negative range (z < 0). Additionally, the motion sensor detects the x-and y-coordinates on the virtual plane simultaneously. When the touch position is above the full-colour reconstructed image, the system switches to the red reconstructed image. When the position is to the left of the image, the system switches to the green reconstructed image. Finally, when the touch position is to the right, the system switches to the blue reconstructed image. In all cases, only the colour corresponding to the finger touch position changes; the other colours remain the same.
The experimental setup used for full-colour reconstruction is shown in Fig. 3. In addition to the green laser and one SLM for the green reconstructed image, we used a red laser (650 nm), a blue laser (450 nm) and two additional SLMs to display red and blue reconstructed images. The 3D object used in the demonstration (Fig. 6) consisted of an average of 1,709 point light sources for all frames. The full-colour object was virtually placed 1 m away from the SLMs when calculating the CGHs. Figure 7 and supplementary movie M2 show an operator switching between display and non-display full-colour images by touching the display. By touching the display to the top, left and right of the full-colour reconstructed image, the full-colour interactive system switched between display and non-display of the red, green and blue reconstructed images, respectively. The average calculation time of the system was 22.6 ms per CGH; thus, the system displayed the full-colour reconstructed image at more than 40 fps. The successful demonstration indicates that the developed system enables the interactive handling of full-colour reconstructed images using finger touch gestures.  In both the single-colour and full-colour reconstructed system, it is difficult to reconstruct and interactively handle more realistic and high-definition 3D images (e.g. a 3D image comprising tens of thousands of point light sources) in real time. Hence, it is necessary to speed up the CGH calculation in order to reconstruct 3D images composed of more point light sources. The computational complexity of a CGH is O (LN x N Y ), where L is the number of point light sources, N x is the number of pixels of the CGH along the horizontal direction and N Y is the number of pixels along the vertical direction.
One method used to speed up the CGH calculation is to distribute the calculation load over multiple GPUs 33,34 . For instance, a full-colour interactive handling system with three GPUs that distributes the CGH calculation of the each colour to one of the three GPUs will be able to calculate the CGH three times faster than the system demonstrated in this study with only one GPU. Using this method allows high-definition images with large numbers of point light sources to be reconstructed. Another method for speeding up the CGH calculation is to adopt a fast calculation algorithm such as wavelet shrinkage-based superposition (WASABI) 35 . The speed enhancement achieved by WASABI depends on the selectivity rate S of the representative wavelet coefficients. Even if S is reduced to 1%, WASABI can obtain almost the same reconstructed-image quality as that obtained without WASABI. In this case, WASABI accelerates CGH calculation by 100 times compared to the method with no fast calculation algorithm. Figure 8 shows a schematic of CGH calculation, which is expressed as follows:   where (x a , y a ) are the coordinates on the CGH, (x j , y j , z j ) are the coordinates of the j-th point light source, L is the number of point light sources, i is an imaginary unit, λ is the wavelength of light, A j is the field magnitude emitted from a point light source and U is the complex amplitude calculated by point light sources and the coordinates of the CGH. Because we used a phase modulation-type SLM, we calculated a phase modulation-type CGH pattern,φ(x a , y a ), using the following equation:

CGH calculation.
a a a a where arg(U) is an operator indicating the argument of a complex number. For a full-colour CGH, the wavelength in Eq. (1) is set to the red, green or blue wavelength and the CGH of each colour is calculated. Thus, the calculation time for the full-colour CGH is three times greater than that of the monochrome one. When constructing a full-colour optical system, red, green and blue CGHs can be displayed on one SLM [36][37][38][39] or on three different SLMs 40 . For the one-SLM system, there are several methods based on time-division multiplexing 36 , space-division multiplexing 37 , superposition of the red, green and blue images on a diffuser plate 38 and the use of SLMs with high dynamic phase modulation range 39 . Time-division multiplexing requires a high-speed SLM, such as a digital micro-mirror device, to obtain full-colour images without flickers caused by the switching of the reference lights to other colours. Space-division multiplexing decreases the resolution of each colour hologram because this method displays each colour hologram on the different regions of an SLM plane. A superposition method is proposed for a holographic projector 41,42 . This method is not suitable for the reconstruction of 3D images because it projects holographic images using a 2D diffuser plate. The last method requires a special phase-only SLM, which has high dynamic phase modulation range (i.e. 0-10π). Because the SLM used herein has the phase modulation range of 0-2π and a refresh rate of 60 Hz, we adopted the method using of three different SLMs.
To realise interactive handling without discomfort, the system requires real-time operation, which is difficult to achieve using only a CPU. Hence, we parallelised the processing of point clouds and the calculation of CGH by combining a CPU with a GPU. (1) and (2) are solved independently for each pixel, the system assigns one pixel calculation to one thread using the GPU. After the CGH calculation is complete for all pixels, the system sends the CGH result to the CPU. To calculate the full-colour CGH, the system solves Eqs (1) and (2) in the order of blue, green and red wavelengths within each thread. Finally, the system sends the full-colour CGH to the CPU.