Fatigue-free visual perception of high-density super-multiview augmented reality images

It is well known that wearing virtual reality (VR) and augmented reality (AR) devices for long periods can cause visual fatigue and motion sickness due to vergence-accommodation conflict (VAC). VAC is considered the main obstacle to the development of advanced three-dimensional VR and AR technology. In this paper, we present a novel AR high-density super-multiview (HDSMV) display technique capable of eliminating VAC in wide range. The designed binocular time-sequential AR HDSMV projection, which delivers 11 views to each eye pupil, is experimentally demonstrated, confirming that VAC is eliminated over a wide-range of viewer’s focus distance. It is believed that the proposed time-sequential AR HDSMV method will pave the way for the development of VAC-free AR technology.

www.nature.com/scientificreports/ with a real-world scene. The viewpoint provided by the moving slit at a certain instant is referred to as the view reference point (VRP); the synchronized operation of image projection and observation generates multiple VRPs and corresponding multi-view images. At a particular moment, the slit specifies a VRP, while the projection unit of the system displays a corresponding view image on the screen, updating this image as the slit position changes. By controlling the velocity of the slit and the framerate of the synchronized time-sequential projection, the SMV density can be adjusted.
The proposed system configuration produces unprecedented binocular VAC-free 3D images with an SMV density of eleven views. Figure 1b presents an example of the binocular 3D images achieved by the proposed system in the present study. The depth of the real object is indicated by AR images of numbers in a wire grid that indicate the diopter. To show the content behind the screen, we also display objects for a depth of 0.0, 1.0, and 2.0 diopters and the wire-grid object. The 0.0 diopter image represents a focus at infinity, while the 5.0 diopter

Results
Binocular time-sequential super-multiview projection. In the proposed projection system, a narrow slit with rapid linear movement produces the VRP. Although fast-moving VRP in front of the human eye can be implemented using various methods, we select the simple optical relay design shown in Fig. 2. The observation unit consists of a binocular optical relay with a symmetrical structure consisting of two identical binocular telescopes with a horizontally moving vertical one-dimensional (1D) slit. The purpose of this design is to produce binocular high-density VRP. The slit in the central plane of the symmetric binocular telescope structure creates symmetrical virtual VRPs on the entrance pupil and exit pupil planes. Each VRP is narrow enough and the depth of field is sufficiently long. As the slit moves in a horizontal direction, the viewer's eye perceives the directional views via the exit pupil plane VRPs. As a result, as viewers vary the focal length of their eyes, it is possible for objects at that distance in SMV content to be in focus. The operation sequence of the proposed BTS HDSMV method is provided in Visualization 2. The focal length of the objective lens f o is much longer than the focal length of the eyepiece lens f e . The projection screen is positioned away from the entrance pupil plane by the distance l . d o and d e are the distance between the eyepiece lens of the reverse telescope and the entrance pupil plane, and the distance between the eyepiece lens of telescope and the exit pupil plane, respectively. The demagnification of the slit is calculated as On the transverse axis, the position of a VRP v is determined by the position of the slit s and the magnification term −f e f o . The resulting demagnified slit image on the exit pupil plane determine the width of VRP (See  www.nature.com/scientificreports/ Supplementary Information). In addition, while the slit passes through the view synthetic region for each eye, there is no inter-pupillary crosstalk because the left and right-eye are completely segregated from each other by the outer wall of the fast-moving slit. Figure 3a presents the experimental setup for the proposed BTS HDSMV projection system. The binocular VRP optical relay contains two identical pairs of binocular telescopes (SV32-8, KOWA) with an optical chopper between them. The field of view of the binocular VRP optical relay is about 60 (deg.). The fast-moving slit for the www.nature.com/scientificreports/ proposed BTS HDSMV system is implemented by rapidly rotating the slit of the optical chopper in the observation unit. The objective focal length f o and the eye-piece focal length f e are 125.6 mm and 15.7 mm, respectively. The optical chopper has nine slits separated at 40 deg. intervals and the width of these slits is 2 mm. In order to ensure that the VRP motion is linear along the horizontal axis and to prevent inter-view crosstalk caused by the slight inclination of the slit, we place an anti-crosstalk filter on the slit plane. The large diameter rotating optical chopper with the anti-crosstalk filter achieves horizontal linear motion that is similar to the fast-linear moving slit shown in Fig. 2. Additional information about the anti-crosstalk filter is provided in Supplementary  Information and Visualization 3. The full-color DMD (V-7001, resolution 1024 × 768 pixels, ViALUX) is operated at a 12-bit color image generation framerate of 1440 Hz, and the optical chopper is synchronized to the DMDs. The distance from the projection screen to the entrance pupil plane l is set to 400 mm. The width and height of the projection screen are 100 mm and 75 mm, respectively. Figure 3b delineates the synchronization between the frames of the DMDs and the slits of the optical chopper. The viewer watches the scene via the very narrow directional ray-bundle arriving through the slits on the chopper. At any instant, only a single slit is placed within the eye-box region including the left and right eyes. If the minimum eye-box length is D o + PD , then the interval between the slits on the chopper should be wider than the eye-box length such that where PD , D o , M , and r are the inter-pupillary distance, the eye pupil diameter, the number of slits on the optical chopper, and the optical chopper radius, respectively. If Eq. (2) is not satisfied, at least two slits are present in the eye-box region at a given instant, leading to the inter-pupillary crosstalk. In our configuration, the maximum inter-pupillary distance PD is assumed to be 72 mm, and r and D o are set at 170 mm and 32 mm, respectively. According to Eq. (2), we employ nine slits ( M = 9 ) in the chopper (Fig. 3b). During a single rotation of the optical chopper, the slit set scans the view synthetic region of the binocular VRP optical relay system nine times. We refer to the time interval between two adjacent slits as one sequence. During a sequence, the DMD projector is programmed to shed N seq view images onto the screen. A single frame of an SMV image is composed of N seq view images. Under these operation condition, the number of views N assigned to an eye pupil area with diameter D o , is estimated as follows: The number of view images in one frame N seq is set at 48 (Fig. 3c) so that 48 views do not overlap in the eyebox region. By making the slit narrower, it is possible to increase N seq and vice versa. Among the 48 projected view images in a sequence, only 22 view images in yellow regions (Fig. 3c) are perceivable by two eyes. Therefore, one frame of SMV content has 22 view images and 26 empty images. We set the refresh rate for the SMV content υ to 30 frames per second (fps), which is a level at which human visual perception does not recognize flickering. Because υ is the result of multiplying M by the revolutions per second (rps) of the chopper, the optical chopper is set to have a stable rotation of 3.33 rps. In summary, a chopper with 9 slits rotating at 3.33 rps generates binocular SMV 3D images with a single eye SMV density of 11. The number of views N entering the pupil can be adjusted by changing the number of images in one frame, i.e., N seq . Binocular HDSMV 3D images. A gray scale AR image display from 5.0 diopters to 0.0 diopters is presented in Fig. 4.
For 3D content, view images are generated using a 3D modeling tool program, where the virtual camera is set to the VRP. We display virtual objects together with real objects to determine whether the proposed method accurately expresses the depth of the virtual objects. We construct a small room with the screen used as the window (Fig. 4a). The main real objects are slippers, a flowerpot on a table, a doll, and a wall. The distance between these objects and the entrance pupil plane are 5.0, 4.0, 3.0, and 2.5 diopters, respectively. We employ a lamp, a tulip, a dog, a window-frame, a parrot, a building, and a mountain as virtual objects using a 3D contents program as shown in Fig. 4b. The virtual objects are placed at 5.0, 4.0, 3.0, 2.5, 2.0, 1.0, and 0.0 diopters from the entrance pupil plane, respectively. The virtual objects located at 2.5 to 0.0 diopters beyond the window are seen as a part of the landscape outside of the window. The virtual objects at 5.0 to 2.5 diopters match the real objects on the inside of the room.
We determine whether each virtual object is in focus when the corresponding real object at the same depth is in focus by reconstructing the tulip in the flowerpot or reconstructing the dog next to the doll. Figure 4c shows the reconstructed VR images for the viewer's focus distance of 0.0, 3.0, and 5.0 diopters through the binoculars. When the viewer's focus distance is 0.0 diopters, the mountain and cloud are in focus and the other objects are blurred. The blurring of these objects depends on their depth. Conversely, if the focus distance is set to 5.0 diopters, the lamp and slippers are in focus and the other objects are blurred. The doll and dog are both in focus when the focus distance is 3.0 diopters. These results illustrate that the proposed method can sufficiently express the depth of objects from 5.0 to 0.0 diopters. A video of the experimental results in Fig. 4c is provided in Visualization 4.

Discussion
The accommodation effect for the 3D vertical lines is presented in Fig. 5. Figure 5a presents the right-eye image of a 3D vertical line located 100 mm (10.0 diopters) from the entrance pupil plane. The left panel in Fig. 5a shows the vertical line in focus. The middle panel in Fig. 5a shows the defocused blurred image when the viewer's focus www.nature.com/scientificreports/ distance is 1.7 diopters. As the viewer's focus distance increases, the defocused image separates into 11 vertical lines, as shown in the right panel of Fig. 5a when the view's focus distance is 1.0 diopter. Each line presents a view image corresponding to 11 VRP positions. The viewer recognizes the dispersed multiple lines instead of a single defocused blurred line. In this case, the viewer may experience the unnatural feeling that the voxel is not blurred but consists of multiple pixels. We distinguish natural-VAC-free and the pseudo-VAC-free zones based on whether the observed 3D image has natural defocus blur or multiple-overlapping blur with the separation of the view images.
To investigate the natural-VAC-free zone for the proposed BTS HDSMV projection system, we performed an experiment to observe vertical lines at several focus distances. As seen in Fig. 5b, three-line objects are arranged from 0.0 diopters to 5.0 diopters with 0.5-diopter intervals. Figure 5c presents the accommodation effect for a single-eye observation with respect to the viewer's focus distance z e . The left, middle, and right panels in Fig. 5c present binocular observation images taken by two cameras with z e of 0.0, 2.5, and 5.0 diopters. When z e is 0.0 This means that the proposed BTS HDMSV projection system has a potential to display natural 3D objects located from 5.0 to 0.0 diopters. The vertical lines are placed at different depths, but they are all centered horizontally. Therefore, due to binocular disparity, when the viewer watches the vertical lines with either the right or left eye, the vertical line at the point where the vergence does not match appears to shift horizontally in different directions. Discrimination between the natural-VAC-free zone and the pseudo-VAC-free zone depends on z e . The outof-focus images can be scored according to the amount of defocus blurring. Distinguishing the natural-VAC-free zone from the pseudo-VAC-free zone can be conducted using the two parameters specified in the retina plane: the maximum achievable blur width of one voxel c p , which is conceptually defined by the sum of the elemental view image blurring, and the desired natural true blurring of the same voxel, c o . c p is a function of z e and the number of VRPs N , while c o is the function of z e and the object distance d obj . The natural-VAC-free zone is judged by comparing c p (z e , N) and c o d obj , z e for a particular value of z e . When c p (z e , N) for a given z e is smaller than c o d obj , z e for a voxel at a certain d obj , the defocused image of the voxel appears as an overlapped image of separate pieces of elemental view images, as shown in the right panel of Fig. 5a, and the voxel is thus located in the pseudo-VAC-free zone. On the other hand, if c p (z e , N) is greater than or equal to c o d obj , z e , the blurring of the voxel located at d obj can be expressed without separation for a given z e . In this case, the voxel is located in the  N) . The other area is classified as the pseudo-VAC-free zone. d max and d min of the natural-VAC-free zone are a function of z e and N. Figure 6 presents defocused voxel images of four voxels (voxels 1, 2, 3, and 4) on the retina plane. Without a loss of generality, we assume that the number of monocular VRPs N is 3 in Fig. 6. The comparison of c p (z e , N) and c o d obj , z e is visualized with z e set to the far-field zone of 0.0 diopters and the near-field zone of 5.0 diopters, respectively, in Fig. 6a and b, in which c o is denoted by a green solid line in the retina image. Each red rectangle on the retina plane is the blurred pixel from the elemental view image and the maximum blur width c p is defined by the sum of the widths of the three rectangles on the retina plane. Voxel 2 is located at d min with the eye focus at z e = 0.0 diopters, and voxel 3 is located at d max with an eye focus of z e = 5.0 diopters. Voxels 1 and 4 are outside  Table 1 summarizes the derived equations. (See Supplementary Information for the full derivation of these equations). The system parameters for the proposed BTS HDSMV system are listed in Table 2, with the numerical values derived from numerical simulations. Plotting Eqs. (4)-(7) allows the dimensions of the natural-VAC-free zone to be determined (Fig. 7). It is assumed that both eyes can converge on the voxels in all ranges of d obj and z e . The red dotted lines represent d min from Eq. (6) and d max from Eq. (7), which specify the boundary of the natural-VAC-free zone in Fig. 7.
In practice, we can consider the resolution of one view image on the retina plane to be determined by the VRP width w v . The diffraction-limit eye-resolution affects c p , and it is possible to expand the depth range of the voxels to the limits specified by the blue lines in Fig. 7 (see Supplementary Information). It is seen that voxels 1 and 4 are not separated because they are located in the natural-VAC-free zone due to the eye-resolution limit. For example, voxels 1 and 4 in Fig. 6 are outside the VAC-free zone and perceived as a separated multi-pixel image according to the geometric natural-VAC-free zone analysis (Eqs. (6) and (7)). However, voxels 1 and 4 are located on the blue lines in Fig. 7, and are thus inside the VAC-free zone. The non-separation of voxels 1 and 4 is verified in the experimental results presented in Fig. 5c. Given that the binocular optical configuration of the BTS HDSMV system has the minimum depth 5.0 diopters in the voxel depth, the finalized VAC-free zone becomes a green rectangular area by in Fig. 6. The green rectangular area is the natural VAC-free zone of the proposed BTS HDSMV system.
The analysis and experimental results indicate that the proposed method widens the natural-VAC-free zone past the limit of previously reported SMV displays by implementing an unprecedentedly high SMV density. The VAC-free zone needs to be compared with the zone of comfort of the conventional stereoscopic 3D displays 8 . This zone is the depth range for 3D objects that supports convergence for both eyes while maintaining viewer comfort. In conventional stereoscopic 3D displays, the depth range of a comfortable 3D image is distributed along the relatively narrow blue zone presented in Fig. 7 because stereoscopic 3D displays provide only binocular disparity. Therefore, in the wide natural VAC-free zone created by the proposed BTS HDSMV method, viewers can see VAC-free binocular 3D images.

Conclusion
The main purpose of this study is the proof-of-concept validation of the BTS HDSMV projection mechanism. We have verified the principle of the BTS HDSMV method and demonstrated the BTS HDSMV operation with the SMV density of 11 to obtain a natural-VAC-free zone with a wide depth of 0.0 to 5.0 diopters. The numerical and experimental analysis has confirmed the technological feasibility and the potential of BTS HDSMV 3D displays. In the present system, to verify the feasibility, we choose a method of mechanically moving the position of the slit. If the eyepiece is implemented compactly based on electronic fast-moving slit devices such as ferroelectric liquid crystal display, it is expected to make a small form-factor prototype. We plan to research and develop a compact BTS HDSMV near-eye display in the near future.

Methods
Hardware development. The BTS HDSMV method requires high-speed display components such as a DMD to allow multiple view images to be rapidly projected onto the screen. The V-7001 DMD with a resolution of 1024 by 768 pixels (ViALUX) was used in the study and it was able to generate a 12-bit color image of 1440 Hz. Three DMDs were used to realize full-color images, with each DMD modulating red, green, or blue (RGB) images. The RGB images were combined using a trichroic prism and projected onto the screen by a projection lens. In the observation unit, SV32-8 telescopes (KOWA) were used. The diameter of the objective lens and the eye-relief were 32 mm and 15.5 mm, respectively. The angular magnification was 8x. We measured the focal length of the telescope lenses; the focal lengths of the objective lens, f o and the eyepiece lens, f e were 125.6 mm and 15.7 mm, respectively. The inter-pupillary distance can be adjusted from a minimum of 58 mm to a maximum of 72 mm. The optical chopper had nine slits and the angle between each slit was 40(deg.). The open width of the slit was 2 mm, and the radius of the optical chopper was 170 mm. The optical chopper was attached to the motor for the movement of the fast-moving slit. We implemented a control box to manipulate the speed of the motor and generated trigger signals to the synchronized projection unit. During one revolution of the motor, the nine slits scanned the view synthetic region of the binoculars. A single frame of SMV content was played back in the projection unit while each slit scanned the view synthetic region once. To reconstruct the SMV content at 30 fps, the motor was driven at 3.33 rps. www.nature.com/scientificreports/