Introduction

Recently, expectations for ultra realistic video technologies have increased and a great deal of research is being performed to realize ultra high-definition or three-dimensional (3D) implementations. Another aspect of ultra realistic video research involves 3D video images that provide a sense of reality that makes it seem as if an object were actually present. There are numerous methods of displaying 3D video, which can be classified into two-view display1 or multi-view display2,3, volumetric display4, or spatial image reproduction display5,6. Among these, the image reproduction 3D display method, in particular, can display easy-to-view 3D images naturally in a similar method as when an object is being directly observed since it is a method that reconstructs the image spatially. Although image reproduction display can be achieved using integral photographic and holographic methods, the holographic method is considered to be the ultimate 3D video display method since it can reproduce the wavefronts of the light reflected by an object and propagated through space.

A wide range of research has been conducted concerning electronic holography-type 3D displays for reproducing 3D videos based on holographic principles. This includes research directed towards increasing the image size7 and research directed towards real-time operation8.

The important topics of investigation for this kind of 3D display are the image size S of the 3D image and the viewing-zone angle θ indicating the range in which the 3D image can be seen. Naturally, it is desirable for both S and θ to be large. If we denote the number of spatial light modulator (SLM) pixels here by Nx in the horizontal direction and Ny in the vertical direction, the pixel spacing by px in the horizontal direction and py in the vertical direction and the wavelength of the reproduced light by λ, then the image size Sx and viewing-zone angle θx in the horizontal direction are given as follows:

Therefore, it is apparent that the product Sxθx is proportional to Nx as shown in equation (3):

equation (2) shows, viewing-zone angle θx and display bandwidth 1/px of SLM are equivalent, so equation (3) also turn out to be space-bandwidth product.

In other words, to simultaneously increase Sx and θx, the number of pixels Nx must increase. Note that we only provided an explanation for the horizontal direction here since the vertical direction is exactly the same. All subsequent explanations will also only be given for the horizontal direction.

Although it is desirable to increase the number of pixels Nx of an SLM, the maximum number is limited for a single SLM. Therefore, a space-division multiplexing method has been investigated for increasing the number of pixels using multiple SLMs and a time-division multiplexing method has been investigated for rapidly rewriting the hologram data that is displayed on an SLM to make it seem as if Nx were increased. With the space-division multiplexing method, if we denote the number of pixels in the horizontal direction of each SLM by Nx′ and denote the number of SLMs by Kx, then Nx = Nx′Kx. Some time-division multiplexing methods that have been attempted use shutters9,10 and others use galvano mirrors11. There is also a known technique that has devised a good way to use both space-division and time-division multiplexing together12. Since we want to discuss a method of using multiple SLMs here, we decided to deal with only a space-division multiplexing method.

Various methods for increasing the viewing zone by using a space-division multiplexing method have been proposed13,14,15. These methods are affected by the problem that part of the viewing-zone angle is missing because of the gaps between adjacent SLMs since it is physically impossible to place the SLMs very closely together. Therefore, finding a means of eliminating this missing segment problem is a major issue. A method have been proposed which overcome this problem by using beam splitter and multiple SLMs16. However, viewing-zone angle only scalable in horizontal direction and each gaps between SLMs are necessary equal to width of SLM.

Various methods of increasing the image size by using space-division multiplexing methods have also been proposed. Problems with these proposals are that they only provide parallax in the horizontal direction17,18 besides image size only scalable in horizontal direction19, or decrease display frame late which is inversely proportional to number of space-division multiplexing20. In addition, finding a means of eliminating missing image segments is also a major issue for these methods of increasing the image size just like it is for the methods of increasing the viewing zone.

In a previous work, we proposed a method of increasing the image size using space-division multiplexing by locating an optical system containing a lens array and other components in front of multiple SLMs21. This method not only can eliminate missing image segments, but can also control the balance between image size and viewing-zone angle by varying the focal length of lenses in the latter half within the optical system. In addition, it can exhibit parallax in both the horizontal and vertical directions unlike the methods mentioned earlier.

However, the technique for colourizing the system which was adopted by this method was difficult because the optical system is too complex to use a combination of three different colour light sources22. In this paper, we propose a colourization technique for electronic holography that reconstructs 3D images at a video rate based on time division multiplexing23,24,25,26,27,28,29 using three lasers (red, green and blue) and a set of three revolving wheel shutters synchronized with 60 Hz driven SLMs.

Results

We constructed a device with the parameters shown in Table 1. This device, which uses an optical system and 9 SLMs, corresponds to a SLM with approximately 74,600,000 pixels. Figure 1 and Video 1 show experimental setup. Figure 1(a) shows an exterior view of the constructed device and Fig. 1(b) shows its light source parts. We performed the following experiments using hologram data created by computer generated hologram (CGH) technology. Note that in this device, horizontal and vertical gap between adjacent SLMs Tx′ and Ty′ are three times the image size of each SLM Sx′ and Sy′ due to physical constraints of the SLMs. In other words, for a display area of one for the image part, the area of the missing segments is eight. Therefore, since the reproduced image cannot be directly observed straight on, an optical system like the one described here is required.

Table 1 System parameters
Full size table
Figure 1
figure 1

Experimental setup.

An exterior view of the constructed system (a) and its light source parts (b). H.S. took photographs.

Full size image

We confirmed that objects with different depths could be reconstructed in colour by using time division multiplexing. Figure 2 shows the experimental results. Figure 2(a) shows the design values of the displayed objects. The colours of the objects (blocks with a letter) are set as follows in the hologram data: “N” is green, “I” is yellow (red + green), “C” is cyan (green + blue) and “T” is magenta (red + blue). Figures 2(b) and 2(d) show photographs that were taken of the reconstructed objects focused on the “N”, “I” and “T” located 30 mm behind the hologram plane and the “C” located 30 mm in front of the plane, respectively. When the objects can be reconstructed at different depths, “C” should be blurry in Fig. 2(b) and “N”, “I” and “T” should be blurry in Fig. 2(d). From the experimental results and the description above, it is apparent that objects with different depths can be reconstructed. Figures 2(c), 2(d) and 2(e) show the views of the same displayed object from other angles. These figures show photographs that were taken of the reconstructed objects from locations 2.8 degrees to the right, directly in front and 2.8 degrees to the left, respectively. It is apparent that the reconstructed objects can be observed in the designed viewing zone and these missing segments that can be seen without the optical system are eliminated. We used method of computer generated hologram (CGH) to create hologram data for the earth and moon. Figure 3 shows the results of a similar experiment using this data.

Figure 2
figure 2

Experimental results of 3D objects located at different depth.

Design values of the displayed objects (a); photographs that were taken focused on the “C” in the 30 mm front (b); and the “N”, “I” and “T” in the 30 mm back from the hologram plane and that were taken from locations 2.8 degrees to the right (c), directly in front (d) and 2.8 degrees to the left (e). H.S. took photographs.

Full size image
Figure 3
figure 3

Experimental results of 3D objects located at different depth.

Design values of the displayed objects (a); photographs that were taken focused on the moon in the 30 mm front (b); and the earth in the 110 mm front from the hologram plane and that were taken from locations 2.8 degrees to the right (c), directly in front (d) and 2.8 degrees to the left (e). H.S. took photographs.

Full size image

We also confirmed that a video could be reproduced. Video 2 shows the experimental results. The letter “N” located 30 mm behind the hologram plane gradually moves forward until it is 30 mm in front of the plane and then gradually moves back until it is 30 mm behind the plane. The video sequence shows the letters “I”, “C” and “T” also sequentially moving forward and back in a similar manner. This experiment confirmed that this video sequence can be converted to a frame rate (with one set of R, G and B per frame) in time-division colour and played smoothly at a 20 frames per second (fps). From the experiments described above, it is apparent that a coloured moving image of 3D objects having an image size 55.3 mm × 31.1 mm which is nine times that of a SLM and a horizontal viewing-zone angle of 5.6 degrees can be played with a frame rate of 20 fps using the proposed method.

Figure 2, 3 and Video 2 show gaps between SLMs are eliminated. However boundaries of SLMs are observed. For example, observed boundaries are sparser in Fig. 3 (d) focused on earth, located far from the hologram plane, than boundaries in Fig. 3 (b), focused on moon, located near the hologram plane. And, difference of brightness between each small display area are also observed.

We also performed a display experiment in which we used a method30 to convert images captured by a 3D camera based on integral photography (IP) to hologram data. IP is a method of capturing many angle of light ray information in space at once by using lens array31. The camera system consists of high-resolution (4K) camera and lens array (240 × 135 lenses, 0.8 mm pitch). Figure 4(a) shows the foregoing camera system and reconstructed 3D images which converted to hologram from captured IP image. We calculated wavefronts propagation by IP image using computer to get hologram. We confirmed that the photographed images could be displayed by the foregoing system. Figures 4(b) and 4(c) show results of displaying hologram data which converted from images captured by the foregoing camera system. These pictures were taken focused on the panda and angel in left which is positioned in front of the panda, respectively. From these results, we confirmed that depth and surface texture of 3D images are reproduced truly.

Figure 4
figure 4

Experimental setup of integral 3D camera system (a) and results of displaying hologram data which converted from images captured by the foregoing camera system. Photographs that were taken focused on the panda (b) and angel in left which is positioned in front of the panda (c). H.S. took photographs.

Full size image

Discussion

A problem encountered when increasing the image size by using multiple SLMs is that some parts of 3D objects are missing due to the gap between SLMs. In this paper, we propose a time-division colourization technique for our previously proposed method of increasing the image size with no missing image segments by locating an optical system containing lens arrays and other components in front of multiple SLMs. Using the proposed optical system enables 3D images to be colourized while enjoying the advantages that parallax can be exhibited in both the horizontal and vertical directions, the balance between the image size and viewing-zone angle can be controlled, the device can be configured without regard to the ratio between the image size of each SLM and the gaps between SLMs, the image size can be increased by increasing the number of SLMs and unnecessary light can be eliminated.

We applied the proposed method to 9 SLMs to realize full parallax video holography at a frame rate of 20 fps with an image size of nine times that of one SLM and a viewing-zone angle of 5.6 degrees. Although the time-division colourization method had a frame rate of 20 fps because we used 60-fps SLMs, we could also reach 60 fps in colour, for example, if 180-fps SLMs were used.

When observing 3D image with focus on distant place from hologram plane in depth direction, boundaries of SLMs become diffuse and sparse, because these boundaries are exist on hologram plane. Actually, observed boundaries became sparser in Fig. 3 (d), focused on earth, than same one in Fig. 3 (b), focused on moon.

There is difference of brightness depends on difference of contrast between each display area in Fig. 2 or 3. There is no difference of contrast between each SLMs, so it is thought to be dominant that difference of optical specification of each beam splitter, which composes an incident optical system. Each beam splitter provides different contrast, it becomes a bad effect for viewing 3D image. It is possible to estimate that we can balance among the difference of display brightness in each display area, if we make luminance compensation for hologram data, which is displayed on each SLM.

Methods

Figures 5 and 6 show configurations of used optical systems in this experiment.

Figure 5
figure 5

Configuration of colourized light source system.

Full size image
Figure 6
figure 6

Configuration of optical system.

Full size image

The proposed method makes multiple SLMs appear to be a single SLM with no gaps between individual SLMs, which are the hologram planes. Of course, it is difficult to physically eliminate the gaps completely. For an enlargement-type 2D display system in which 2D displays are tiled, since the viewer's eyes are focused on the displays, no matter how precisely the individual displays are aligned, the spatial frequency components of the interference caused by the boundaries of the displays are easily perceived by the viewer since they often contain low-frequency components relative to the video signals themselves. However, for a 3D video display based on a holographic method, the reproduced image can be displayed at a depth location separated from the hologram plane. In this case, the viewer's eyes focus on the reproduced 3D image rather than the hologram plane. If the interference caused by the boundaries of the SLMs can be suppressed enough so that the eyes do not focus on the hologram plane, it can be driven out to the Fourier transform region with respect to the viewer's sense of sight. In other words, since it is dispersed to the entire field of vision, it cannot actually be perceived. As a result, this method can be expected to actually suppress the interference caused by tiling the SLMs for a 2D display.

Figure 5 shows the light source system that implements colourization based on time-division multiplexing also known as frame sequential method. This system consists of three lasers (red, green and blue), wheel shutters, spatial filters and a trichroic prism assembly. The shutters are rotating wheels driven by motors that are synchronized with the SLMs and controlled so that a beam of one of the three colours is selected to pass through the shutter every 1/60 second. After that, the laser beams pass through a spatial filter (convex lens and pinhole) so that the shape of the beams gradually becomes wider. Three beams are combined into one by a Phillips type trichroic prism assembly. Finally, the combined white beam is collimated by a large size convex lens to illuminate all of the SLMs.

Figure 6 only shows an example of the horizontal direction for the case when the number of SLMs Kx is 3. First, the previously described time-divided colour parallel light from a coherent light source as shown in Fig. 5 is incident perpendicularly on SLM C0 through beam splitter B0. Consequently, this is so-called in-line holography. The hologram data is displayed on C0. This data is synchronized with the wheel shutter and changed to accommodate the alternating wavelengths of the light sources (red, green, or blue) every 1/60 second. Note that the superscript at the top right of the letter represents the location within the array. Since the pixels are arranged in a grid pattern on a thin hologram, the light reflected on C0 contains a primary beam, conjugate beam, carrier beam and high-order beams of each of them. Since the object beam is generated as a primary beam, the other beams are unnecessary light and must be eliminated.

Next, the light is incident on lens , spatial filter F0 and lens . Spatial filter F0 is located on the common focal plane P1 of lenses and and the focal length f0 of and focal length f1 of are related as follows:

depending on the image size S′ of each SLM and the gap T′ between SLMs. Also, we set the distance a between C0 and as follows:

This enlarges the image due to C0 by (S′ + T′)/S′ and shifts it to (P2).

Here, let us simply denote the group of C0, B0, , F0 and by a partial optical system A0. By locating partial optical systems A0 and A2 adjacent to A1, the images due to C0, C1 and C2 will be tiled on plane P2 without gaps. As a result, the missing image segments due to the gaps between the SLMs will disappear.

The group of lenses , and forms the lens array L0 in practice and the group of lenses , and forms the lens array L1. F0 is a spatial filter for eliminating the conjugate beam, carrier beam and high order beams, which are unnecessary light32. The group of spatial filters F0, F1 and F2 forms the spatial filter array F in practice.

Finally, the light is incident on lens L2 and lens L3. By setting the focal length f2 of L2 and focal length f3 of L3 as follows:

the image that had been enlarged at P2 will be returned to its original size at plane P3.

This optical system can exhibit parallax in both the horizontal and vertical directions. It can also control the balance between the image size Sx and viewing-zone angle θx according to the values of f2 and f3. By varying the value of f3 starting from f0, you can configure the device with any ratio between the image size of each SLM and the gap between SLMs. The image size Sx can be enlarged by increasing the number of SLMs Kx and using a large-aperture lens for L2. Also, unnecessary light can be eliminated by spatial filter F0. Those are some advantages of this optical system.

Proposed method has scalability to increase the number of SLMs for further large image size. There are two factors of size limitation. One is size of lenses and the other one is vignetting of first order light by the outer edge of beam splitter, which was located at distant position from SLMs. A solution for latter problem is to reduce the number of steps of beam splitter to shorten light path in incident optical system.