JP7227095B2 - Hologram generation device and hologram generation method - Google Patents

Description

The present invention relates to a hologram generation apparatus and a hologram generation method, and more particularly, a hologram generation apparatus that generates a computer-generated hologram at high speed by reusing an object light wave on the hologram surface of the previous frame when a viewpoint shifts. and a hologram generation method.

Holography is a three-dimensional display technology that records and reproduces light from an object (object light) based on light interference and diffraction phenomena. A hologram is obtained by interfering with and recording the object light as interference fringes. In addition, by irradiating the interference fringes with reproduction illumination light, the light at the time of recording can be reproduced. Since holography can faithfully reproduce the light emitted from an object, it is regarded as an ideal three-dimensional display technology that satisfies all the physiological factors of human three-dimensional perception.

A hologram can be created by calculation using an electronic computer, and such a hologram is called a computer-generated hologram (CGH). CGH is a technology that calculates the propagation and interference of light waves necessary for hologram calculations as light wave simulation inside a computer, and outputs the interference fringes as electronic data such as images.

Compared to analog holograms, which are shot using a photographic dry plate, etc., there is no need for a complicated optical system for shooting, and animation can be easily created by switching the CGH displayed on the liquid crystal one after another. Because of its advantages, it is expected to be applied to next-generation TVs and XR (a generic term for technologies such as AR/MR/VR) devices.

On the other hand, there are many problems that need to be solved, such as the need for a display device with a pixel pitch on the order of the wavelength of light for wide-field viewing, and the long computation time required for light wave simulation.

Non-Patent Document 1 discloses a CGH calculation method capable of highly realistic rendering based on a CG ray tracing method. In addition, Non-Patent Document 1 discloses a technique for expanding the field of view using a Fourier Transform Optical System (FTOS) that expands the field of view by sacrificing the viewing zone.

This is a technology that expands the field of vision using a lens, but because it sacrifices the viewing zone, it is possible to view even in a narrow viewing zone like a head mounted display (HMD) type device. To some extent, it has been considered to have a high affinity with systems that can fix the viewpoint position.

FIG. 21 is a diagram showing the difference between the visual field and the visual field. (the range in which the eyes can be moved when viewing an image).

However, in the technique of Non-Patent Document 1, in addition to performing rendering based on the ray tracing method, which is said to take a relatively long calculation time among CG calculation methods, CGH originally takes a huge amount of time to calculate light wave simulation. is required, there is a problem that the generation processing time is long.

In particular, when applied to an HMD type device, it is necessary to calculate the CGH according to the viewpoint position and direction each time when the wearer moves by walking or the like. In this regard, when considering calculation by the method of Non-Patent Document 1, even if the movement of the viewpoint is slight, the CGH must be recalculated every time the position or orientation of the wearer changes. was efficient.

Non-Patent Document 2 assumes that the technique of Non-Patent Document 1 is applied to an HMD device, and proposes a high-speed calculation method when the wearer slightly moves the position or orientation of the viewpoint. Specifically, by reusing a portion of the object light wave on the hologram surface calculated in Non-Patent Document 1, a technique for high-speed calculation of the CGH of the next frame is proposed. In this technique, although there is quality deterioration due to a certain degree of approximation, since the object lightwave of the previous frame is reused, the calculation time and the like related to lightwave propagation can be greatly reduced.

T. Ichikawa, T. Yoneyama, and Y. Sakamoto, "CGH calculation with the ray tracing method for the Fourier transform optical system," Opt. Express 21, 32019-32031 (2013). R. Watanabe, K. Yamaguchi, and Y. Sakamoto, "Fast calculation method of computer generated hologram animation for viewpoint parallel shift and rotation using Fourier transform optical system," Appl. Opt. 55, A167-A177 Takamasa Nakamura, Yuji Sakamoto, "Comparison and verification of different hologram data calculation methods in Fourier transform optics," 3D Image Conference 2018, 1-1, (2018). Takuo Yoneyama, Eishin Murakami, Yuki Oguro, Hibiki Kubo, Kazuhiro Yamaguchi, Yuji Sakamoto, "Holographic head-mounted display with correct accommodation and vergence stimuli," Optical Engineering 57(6), 061619, (2018). Laurentini, A. "The visual hull concept for silhouette based image understanding.", IEEE Transactions on Pattern Analysis and Machine Intelligence, 16, 150-162 (1994). Ryosuke Watanabe, Takuya Sugawara, Yuji Sakamoto, "Fast calculation method of computer generated hologram using ray tracing method for Fourier transform optical system", 10th International Symposium on Display Holography, P-16, Jul. 2015.

In non-patent document 2, although it was demonstrated through experiments on subjects that there was a certain speed-up effect and little deterioration in quality, there were restrictions on the movement of the viewpoint that can be calculated at high speed. Specifically, Non-Patent Document 2 suggests that when the viewpoint moves (back and forth) in a direction in which the depth changes with respect to the viewed object, the object light wave of the previous frame cannot be reused.

In recent years, the term DoF (Degrees of Freedom) is sometimes used to express the number of directions of movement that can be supported when wearing an XR device. In general, it is called 6DoF (6 degrees of freedom) to correspond to movement in the xyz axis direction and rotation around the xyz axis. It can also be expressed as

However, even if it is in one direction, the movement that cannot be accelerated is mixed, which means that if there is even a slight forward or backward movement, the calculation must be redone using the technique of Non-Patent Document 1. , the practical constraints are very large.

In addition, in Non-Patent Document 2, distortion occurs even when parallel movement without back-and-forth movement is performed (in the present invention, parallel movement without back-and-forth movement is expressed as "parallel movement"). It is shown. In Non-Patent Document 2, only one point of depth where an object exists can be specified, and the object at that depth can be translated correctly. However, the further away the position of the object from the specified depth is, the more the movement will differ from the actual translation amount, so the translation algorithm cannot be applied well to scenes with depth differences There was a problem.

The object of the present invention is to solve the above technical problems, and when calculating CGH in an optical system that can expand the field of view with a lens, by incorporating a light wave simulation method based on a convergent spherical reference light wave, at any viewpoint movement, It is an object of the present invention to provide a hologram generation apparatus and a hologram generation method capable of realizing high-speed calculation of CGH by reusing object light waves.

In order to achieve the above objects, the present invention is characterized by having the following configuration in a hologram generating apparatus for generating a hologram used in an optical system that expands the field of view using a lens.

(1) Means for determining the acquisition method of the object light wave on the hologram surface to either the “calculation method” or the “reuse method” for each pixel according to the movement of the viewpoint, and each pixel determined by the “calculation method” means for reusing the object light wave of the corresponding pixel of the previous frame for each pixel determined by the "reuse method"; and based on the object light wave of each pixel and the converging spherical reference light wave and means for calculating the interference fringes on the hologram surface by light wave simulation.

(2) As for means for determining the acquisition method, the reuse method is determined when the movement amount of the viewpoint is less than a predetermined threshold, and the calculation method is determined when the amount is equal to or greater than the predetermined threshold.

(3) As for the means for reuse, when the forward and backward movement of the viewpoint is detected, the object light wave distribution of the previous frame is enlarged or reduced according to the movement direction and reused.

(4) The threshold for the amount of movement of the viewpoint when determining the acquisition method is made smaller for pixels that are farther from the center of the hologram plane.

(5) We increased the threshold for the amount of movement of the viewpoint when determining the acquisition method for pixels that record light waves from objects farther away from the viewpoint.

(6) The scaling factor of the object light wave distribution is made variable according to the distance from the hologram surface to the object point light source.

(7) In response to the parallel movement of the viewpoint, the means for determining the acquisition method determines the reuse method for the pixel area after the parallel movement that overlaps with the area before the translation, and determines the other pixel areas for the calculation method. I made it

(8) The means for reuse is to multiply the object light wave distribution before rotational movement by the phase difference caused by the difference in the optical path difference caused by the rotational movement of the viewpoint and reuse it as the object light wave distribution after rotational movement.

According to the present invention, the following effects are achieved.

(1) In a hologram (CGH) generation device used in an optical system that expands the field of view using a lens, the object light wave on the hologram surface acquired in the previous frame is moved and processed according to the movement of the viewpoint, and then the next Since the converging spherical reference light wave is used as the reference light used for the light wave simulation in reusing the frame, high-speed calculation of the hologram by reusing the object light wave can be realized while suppressing the quality deterioration.

(2) If the amount of movement of the viewpoint is less than a predetermined threshold and it is estimated that deterioration in viewing quality is small even if the "reuse method" is adopted, the "reuse method" is decided. High-speed calculation of reused holograms can be realized while suppressing deterioration in quality.

(3) When the forward/backward movement of the viewpoint is detected, the object light wave distribution in the previous frame is enlarged or reduced according to the movement direction, and is reused in the next frame, so that the hologram reflecting the backward/backward movement of the viewpoint can be calculated at high speed. become.

(4) When determining the generation method of the object light wave for each pixel according to the back and forth movement of the viewpoint and the amount of movement, the shorter the distance from the center of the hologram surface, the shorter the distance from the center of the hologram surface. Since the "reuse method" is determined even if the amount of movement is large, the object light wave of the next frame can be reused for pixels that are less deteriorated due to movement of the viewpoint. Therefore, it is possible to increase the rate of reuse of the object light wave on the entire hologram surface, and to perform high-speed calculation of the hologram reflecting the back and forth movement of the viewpoint.

(5) The threshold for the amount of movement of the viewpoint when determining the acquisition method is increased for pixels that display objects that are farther from the viewpoint. become.

(6) Since the expansion/contraction ratio of the object light wave distribution corresponding to the forward and backward movement of the viewpoint is made variable according to the distance from the hologram surface to the object point light source, there is little error regardless of the distance from the hologram surface to the object point light source. Holograms can be calculated at high speed.

(7) When determining the generation method of the object light wave for each pixel according to the parallel movement of the viewpoint and the amount of movement thereof, either the “calculation method” or the “reuse method”, the pixel area after the parallel movement that overlaps with the pixel area before the parallel movement is Since the reuse method is determined, the smaller the amount of movement, the higher the percentage of pixels determined to be the "reuse method", thereby enabling high-speed calculation of the hologram.

(8) When the viewpoint rotates, the object light wave distribution before the rotational movement is multiplied by the phase difference caused by the difference in the optical path difference caused by the rotational movement, and the object light wave distribution before the rotational movement is reused as the object light wave distribution after the rotational movement. High-speed calculation of holograms reflecting movement becomes possible.

1 is a side cross-sectional view showing a configuration example of an HMD to which a CGH device according to an embodiment of the invention is applied; FIG. 3 is a functional block diagram showing the configuration of main parts of the control device; FIG. FIG. 4 is a diagram showing the relationship between rotational movement of a viewpoint and rotational movement of a hologram plane; FIG. 11 is a diagram (part 1) showing a rendering method based on the ray tracing method; FIG. 11 is a diagram (part 2) showing a rendering method based on the ray tracing method; FIG. 10 is a diagram explaining movement of a hologram surface; FIG. 11 is a diagram (part 1) explaining parallel movement of the hologram surface; FIG. 11 is a diagram (part 2) explaining the rotational translation of the hologram plane; It is a figure explaining the rotational movement of a hologram surface. It is a figure explaining the back-and-forth movement of a hologram surface. FIG. 4 is a diagram showing an optical path difference caused by forward and backward movement of a hologram plane; It is a figure for demonstrating the calculation method of an optical path difference. FIG. 2 is a diagram (part 1) explaining a problem when scaling an object light wave distribution; FIG. 11 is a diagram (part 2) explaining a problem in scaling the object light wave distribution; FIG. 10 is a diagram showing an example in which a visible range expands when the viewpoint moves backward; FIG. 4 is a diagram showing an example of CGH to be output; 10 is a flow chart showing a procedure for determining an object light wave generation method for each pixel on the hologram surface; 4 is a flow chart showing a procedure for determining a method of generating an object light wave on a principle basis; 10 is a flow chart showing a procedure for determining an object light wave generation method on an error basis; FIG. 10 is a diagram schematically showing a method of generating an object light wave when forward and backward movement of a viewpoint is detected; It is the figure which showed the difference between a visual field and a viewing zone.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a side cross-sectional view showing a configuration example of the main part of an HMD to which a CGH generation device according to one embodiment of the present invention is applied. An optical lens 2 arranged on the surface to expand the field of view, a reconstruction illumination light source 3 for irradiating spherical waves as reconstruction illumination light, and a half mirror arranged on the optical axis O between the hologram surface and the reconstruction illumination light source 3. 4 and a control circuit 1 for driving the display device 5 are configured as a left and right pair, and a synchronization circuit ( not shown). The support members 6a, 6b, and 6c fix each component to the head H of the viewer (wearer) in a predetermined positional relationship.

FIG. 2 is a functional block diagram showing the configuration of main parts of the control device 1. As shown in FIG. A scene setting unit 101 defines 3DCG (3D Computer Graphics) of a scene for creating CGH within the apparatus. In this embodiment, the scene setting unit 101 inputs a 3DCG general format or the like, and arranges the 3DCG of the scene on the coordinate space within the apparatus.

The scene to be viewed is not necessarily limited to a virtual object. For example, a scene is set using 3DCG generated by reconstructing images captured by multiple cameras using the technology disclosed in Non-Patent Document 5. can be Using such technology, real objects can be recorded as CGH.

The viewpoint movement detection unit 102 includes a motion sensor and an acceleration sensor (not shown), and detects movement of the viewer's viewpoint based on outputs from these sensors. The method determining unit 103 selects the object light wave generation method for the next frame for each pixel on the hologram plane from either the “reuse method” or the “calculation method” based mainly on the result of the viewpoint movement detection performed by the viewpoint movement detection unit 102 . decide whether

In the "reuse method", the object light wave of each pixel generated in the previous frame is reused in the corresponding pixel of the next frame, so the amount of calculation related to object light wave propagation can be reduced. In the "calculation method", the object light wave is calculated for each pixel as before. In this embodiment, one of the generation methods is determined for each pixel based on the principle that if the amount of movement of the viewpoint is small, the “reuse method” is adopted. As for the first frame to be viewed, since there is no previous frame in which the object light wave can be reused, adoption of the "calculation method" is inevitably determined for all pixels.

On the other hand, when the object light wave generated in the previous frame is reused in the next frame, since the position of the hologram surface and the position of the sensor used for movement detection are different in this embodiment, when a rotational movement of the viewpoint is detected, , CGH corresponding to the rotational movement of the viewpoint cannot be reproduced even if the object light wave is reused by rotating the hologram surface.

As shown in FIG. 1, when an optical system including a magnifying lens 2 is used, the viewpoint is located near the focal length f of the lens. The viewpoint moves so as to draw an arc of distance f centered on the surface.

Therefore, in this embodiment, the method determination unit 103 is provided with a movement conversion unit 103a. Based on the relative positional relationship between the hologram surface and the sensor, the movement conversion unit 103a converts the change in the field of view due to the movement of the viewpoint into movement of the hologram surface ((a) parallel movement, (b) rotational movement, (c) forward/backward movement). or a combination of each movement), the combination of each movement (a), (b), and (c) corresponding to the detected viewpoint movement is determined, and the movement amount T (T1, T2 , T3).

The object light wave generation unit 104 includes a rendering unit 401 and a light wave propagation calculation unit 402 for generating the object light wave by the "calculation method", and an object light wave reuse unit 403 for generating the object light wave by the "reuse method".

The rendering unit 401 renders the scene defined by the scene setting unit 101 for each pixel of the next frame for which the method determination unit 103 has decided to adopt the “calculation method”, and selects an object point light source for propagation calculation. get.

In this embodiment, for the 3DCG scene set by the scene setting unit 101, in order to render the scene so that the video from the viewpoint of the viewer can be reproduced, elements such as those disclosed in Non-Patent Document 1 are used. The scene is rendered using a rendering method based on ray tracing from holograms.

In this rendering method, as shown in Fig. 4, the hologram surface is divided into small areas called elemental holograms, and from the center of each small area, the ray tracing method, one of the representative rendering techniques for CG, is used. do the rendering.

The ray tracing method is a technique that can realize realistic rendering that considers hidden surface removal, shading, etc., by shooting a ray from a certain point and coloring the screen with the color of the polygon that intersects with the ray. In this embodiment, the position and color of the intersection with the polygon obtained after the light ray is shot are obtained as points with three-dimensional position information (x, y, z) and color information (R, G, B). Record. Since many points can be obtained if the object is large, the rendering unit 401 obtains a Point Cloud when the target scene is viewed from a certain viewpoint.

The light wave propagation calculation unit 402 calculates light wave propagation from each object point obtained by the rendering unit 401 to the hologram surface. In this embodiment, the object is expressed as a mass of point groups, and the optical wave propagation when the spherical wave from this point group propagates to the hologram surface is calculated to obtain the object light wave distribution u(x, y ) is calculated based on the following equation (1).

Here, Ai is the brightness of the object point light source obtained by the rendering unit 401, ri is the distance between the object point light source and the hologram plane (x, y), k is the wave number calculated from the wavelength of light, and i is is the index of the object point light source, and N is the number of object point light sources that compose the scene. Also, j is the imaginary unit. If the object point has multiple color spaces such as RGB, the object light wave distribution u(x, y) is calculated for each color. However, since the wavelength of light changes depending on the color, the wavenumber k differs for each color.

In addition, Non-Patent Document 1 discloses a technique for cutting a part of the light wave distribution propagated on the hologram surface in order to delete the part related to the conjugate image in the propagated object light wave. The conjugate image may be removed by cutting a part of the object light wave distribution as shown.

In addition, since CGH is generally a technology that reproduces light, it requires a liquid crystal with a pixel pitch at the level of the wavelength of light (1 μm or less as a guideline), and a display device with a coarser pixel pitch is used. Then, in a high-frequency region exceeding the sampling frequency of the electronic device, aliasing noise may occur, causing problems such as the appearance of double images.

However, since it is difficult to prepare a display device (for example, a liquid crystal display) with such a fine pitch, it is necessary to calculate the object light wave distribution u(x, y) in consideration of the pixel pitch of the display device to be used in advance. A mechanism may be provided to prevent the occurrence of aliasing noise by cutting in advance a region that sometimes becomes a high-frequency component for each object point.

The object light wave reuse unit 403 refers to the object light wave on the hologram plane calculated in the previous frame for each pixel of the next frame for which the calculation method determination unit 103 has determined the “reuse method”, and The object light wave on the hologram surface of the next frame is approximately generated by appropriately applying deformation and processing according to the movement mode and amount of movement of and reusing it for the corresponding pixel of the next frame.

As a result, the number of pixels for which object light waves can be generated without performing rendering or light wave propagation calculation is increased, so that the speed of light wave propagation calculation can be greatly increased. In this embodiment, as shown in FIG. 5, by combining the object light waves on the hologram plane respectively generated by the light wave propagation calculation unit 402 and the object light wave reuse unit 403, the object light wave distribution of all pixels in the next frame is is obtained. A specific method of reusing the object light wave will be described in detail later.

The interference calculation unit 105 inserts a reference light wave as a light wave simulation on a computer into the object light wave distribution u(x, y) on the hologram plane generated by the light wave propagation calculation unit 402 and the object light wave reuse unit 403. Perform interference calculations. In this embodiment, a convergent spherical reference light wave that converges at the position of the focal length f of the lens of the reproduction optical system, disclosed in Non-Patent Document 3, is used as the reference light. The complex amplitude distribution R(x, y) of the light wave when the converging spherical reference light wave propagates on the hologram plane is expressed by the following equation (2).

Ro is the intensity of the reference beam, and r represents the distance from the position (0, 0, f) of the reference beam to the position (x, y, 0) on the hologram plane. The interference between the reference light wave and the object light wave is expressed by the following equation (3).

I(x, y) is the luminance distribution of CGH (interference fringes), and based on the value of this I(x, y), CGH is provided to CGH output section 106 in the subsequent stage. A CGH output unit 106 outputs the interference fringes calculated by the interference calculation unit 105 as image data. The output format of CGH is arbitrary, but generally, as shown in FIG.

On the other hand, the interference fringes calculated by the interference calculator 105 are often not normalized to this range. As for the CGHs to be output, each CGH may be output one by one as image data, or may be of a format in which images are displayed in a window at any time.

Next, a detailed description will be given of a method for the object light wave reuse unit 403 to reuse the object light wave of the previous frame for the next frame. In the following explanation, it is assumed that the xyz axes are set in the directions shown in FIG. 6, and the hologram plane displaying the object light wave distribution exists on the plane (x, y, 0) created by the xy axes. Assume that

Note that when the viewpoint movement is realized by combining (a) parallel movement, (b) rotational movement, and (c) forward/backward movement of the hologram surface, the order of each movement to be combined is arbitrary, but each movement is determined according to the movement order. may differ from each other. Therefore, it is desirable to determine in advance the order of each movement to be combined, and to calculate the movement amount of each movement on the premise of movement in that order.

A. First form of reuse: parallel movement Reuse by parallel movement of the object light wave distribution is roughly divided into two types: parallel movement in the xy-axis direction and rotational movement around the z-axis, but the basic idea is the same. is. In addition, in order to distinguish parallel movement due to rotational movement about the z-axis from rotational movement about the x-axis or y-axis, which will be described later, translation due to rotational movement about the z-axis may also be expressed as rotational translation. be.

If the detected viewpoint movement is a parallel movement of the hologram plane in the xy-axis direction or a combination of a plurality of movements including the parallel movement, the amount of movement is compared with a predetermined reference value. If the amount of movement is less than the reference value, the "reuse method" is determined, and as shown in FIG. 7, the object light wave distribution recorded on the hologram surface of the previous frame is translated as it is to the corresponding position of the next frame. reused. For other pixel regions, the "calculation method" is determined, and the light wave propagation is calculated by the cooperative operation of the rendering unit 401 and the light wave propagation calculation unit 402 to generate the object light wave distribution.

In Non-Patent Document 2, since the calculation must take into account the presence of a lens between the hologram surface and the object, there is a problem that distortion occurs when the object light wave distribution is simply translated. rice field. However, in this embodiment, convergent spherical reference light is used as reference light for light wave simulation, and there is no need to consider lenses, so such distortion does not occur.

On the other hand, if the detected viewpoint movement is a rotational parallel movement of the hologram plane centered on the z-axis or a combination of a plurality of movements including the rotational parallel movement, then as shown in FIG. is reused by translating the object lightwave distribution of the previous frame in the rotational direction, and lightwave propagation is calculated for non-overlapping pixel regions to generate the object lightwave distribution.

B. Second Reuse Form: Rotational Movement If the detected viewpoint movement is a rotational movement of the hologram plane about the x-axis or the y-axis or a combination of multiple movements including the rotational movement, as shown in FIG. , an optical path difference occurs in the propagating object light wave. Therefore, by multiplying the object light wave distribution u(x, y) on the hologram plane before rotation to be reused by the phase generated by the difference in the optical path caused by the rotational movement, the rotational movement can be approximately realized. For example, as shown in FIG. 9, when the hologram plane rotates about the x-axis, the object light wave distribution u _rot (x, y) on the hologram plane after rotation is represented by the following equation (4). where θ _rot is the angle of rotation.

C. Third Reuse Mode: Forward/Backward Movement If the detected viewpoint movement is forward/backward movement of the hologram plane in the z-axis direction or a combination of multiple movements including the forward/backward movement, the object light wave distribution in the next frame will be the forward/backward movement. It can be approximated by contraction or expansion of the object lightwave acquired in the frame. For example, as shown in FIG. 10, by reducing the object light wave distribution on the hologram plane, the reproduction position of the reproduced CGH image can be moved closer to the hologram plane. Conversely, by enlarging the object light wave distribution, the reproduction position of the reproduced CGH image can be moved farther.

Note that the forward/backward movement reproduction by enlarging or reducing the object light wave distribution is only an approximation, and several errors and problems occur as described in detail below. Therefore, it is desirable to calculate the wave propagation over the entire surface if the amount of movement exceeds the reference value. Problems caused by enlargement/reduction processing and solutions thereof will be described below.

As a first problem, simply enlarging or reducing the object light wavefront may cause a large error in a scene in which a plurality of objects with different depths exist. For example, if there is an object point light source A at a position of 50 cm from the hologram plane and an object point light source B at a position of 100 cm, if the hologram plane is reduced by 50%, an object with a depth of 100 cm will be located at a position of 50 cm and a distance of 50 cm. The movement is such that a deep object appears at a position of 25 cm.

However, in a normal forward/backward movement of the viewpoint, if the viewpoint moves forward by 50 cm, all objects must uniformly approach by 50 cm regardless of the depth. Therefore, in this embodiment, an error may occur in a scene with a difference in object depth.

In order to solve such a technical problem, in the present embodiment, the depth D at which the depth should be changed is set accurately. Also, if the depth changes to D′=D+ΔD when the viewpoint is moved by ΔD in the depth direction, the hologram surface is reduced/enlarged based on the ratio of D and D′.

For example, suppose you want to accurately represent an object at a depth of 100 cm in the Z direction when viewed from the hologram plane. At this time, if the viewpoint moves forward by 10 cm (ΔD=-10 cm), this object should appear at a depth of 90 cm. At this time, the hologram surface is reused by reducing the hologram surface to 90/100 with respect to the center of the hologram surface. The depth D, which is desired to change the depth accurately, may be manually set by the user in advance, or may be automatically set to the average depth of the object point light sources that make up the entire scene. A mechanism may be provided.

On the other hand, with respect to the hologram surface thus enlarged/reduced, as shown in FIG. 11, the closer to the end of the hologram surface, the more the difference in the optical path occurs before and after the movement. Therefore, in this embodiment, it is desirable to prevent the influence caused by the difference in optical path by multiplying the hologram surface after enlargement/reduction by the phase caused by the difference in optical path.

In this embodiment, as shown in FIG. 12, the distance to the pixel (x, y) on the hologram surface before reduction is r _h (x, y), and the pixel on the hologram surface after reduction (x _after , y _after ) The optical path difference r _d (x _after , y _after ) to be considered is represented by the following equation (5) when the distance to is r _h ' (x _after , y _after ).

Let u_b (x_after, y_after), the object light wave distribution u_after (x_after, y_after) is represented by the following equation (6).

As for the enlargement, it is possible to realize forward/backward movement by similarly calculating with consideration given to the optical path difference.

As a second problem, even if the depth D is set as described above, when enlarging or reducing the hologram surface with reference to the center of the hologram surface, an error occurs in the position reproduced during reuse. This is because, when enlarging/reducing the hologram surface with reference to the center of the hologram surface, as shown in FIG. This is because the object light wave is recorded at a position different from the correct position. In this way, in this embodiment, in principle, the positional difference is greater the further outside the hologram surface.

In order to solve such a technical problem, in the present embodiment, a mechanism is introduced in the method determination unit 103 that makes it easier for pixels outside the hologram surface to be determined to the "calculation method", thereby achieving natural reproduction. relatively easy to implement.

Also, as shown in FIG. 15, when the viewpoint moves backward, the visible range expands, so there is a case where a new object point becomes visible. In this case as well, since the object points are visible on the outside, it is desirable to make it easier for the "calculation method" to be determined on the outside.

Next, with reference to the flowchart of FIG. 17, a specific procedure for the method determining unit 103 to determine the object light wave generation method for the next frame for each pixel on the hologram surface to be the "reuse method" or the "calculation method" will be described. do.

In step S1, it is determined whether or not the viewpoint movement detection unit 102 has detected a viewpoint movement. When the viewpoint movement is detected, the process advances to step S2, and the movement method conversion unit 103a converts the viewpoint movement into a combination of (a) parallel movement, (b) rotational movement, and (c) forward/backward movement of the hologram plane. In step S3, the method determination unit 103 determines the object light wave generation method to be the "reuse method" or the "calculation method" for each pixel on the hologram surface on a principle basis.

FIG. 18 is a flow chart showing the procedure for determining the method of generating the object light wave on a principle basis. , and other pixels are provisionally determined to be the "reuse method".

In step S301, it is determined whether or not the next frame is the first frame, in other words, whether or not there is a previous frame in which the object light wave can be reused. If the next frame is the first frame, the "reuse method" cannot be applied in principle, so the process proceeds to step S308, and all pixels are determined to be the "calculation method".

On the other hand, if the next frame is not the first frame and there is a previous frame in which the object light wave can be reused, the process proceeds to step S302. In step S302, it is determined whether or not the detected viewpoint movement includes (a) parallel movement of the hologram plane. If translation on the plane created by the xy-axes (FIG. 5) or rotational translation about the Z-axis (FIG. 8) is included, the process proceeds to step S303.

In step S303, as described with reference to FIG. 5 or 8, it is determined for each pixel of the next frame whether or not there is an object light wave that can be reused by translation in the previous frame. For pixels for which no reusable object light wave exists in the previous frame, the "reuse method" cannot be applied in principle, so the "calculation method" is selected.

Note that when the detected viewpoint movement includes rotational movement of the hologram plane, as described with reference to FIG. There are no pixels to be determined. Similarly, when the detected viewpoint movement includes forward and backward movement of the hologram plane, there are no pixels to which the "reuse method" can be applied in principle as follows. does not exist.

In other words, when the viewpoint movement includes back-and-forth movement of the hologram plane, forward movement means that the object light wave distribution generated in the previous frame is reduced and reused. At this time, due to the reduction, there may be pixels in which there are no reusable object light waves. The object lightwave distribution generated in one frame contains all the information of the object visible in the next frame. Therefore, in forward movement, there are no pixels to which the "reuse method" cannot be applied in principle. Note that, in the present embodiment, the pixel value is set to 0 for pixels in which no object light wave exists, because there is no need to perform any particular processing.

Backward movement will expand the object lightwave distribution generated in the previous frame, and all pixels will be reused. Therefore, there is no pixel to which the "reuse method" cannot be applied in principle even in backward movement.

Returning to FIG. 17, in step S4, the object light wave generation method is determined to be the “reuse method” or the “calculation method” on an error basis for each pixel on the hologram surface. However, in this embodiment, the determination result based on the principle is given priority with respect to the application of the "calculation method". Calculation method” is maintained.

FIG. 19 is a flow chart showing an error-based determination method. In this embodiment, the "reuse method" , thresholds T1ref, T2ref, and T3ref of the amount of movement that can be judged that the approximation error is large and the predetermined viewing quality cannot be satisfied are set in advance. If there is, it will be determined as "calculation method".

In step S401, the movement amount T1 obtained by converting the detected viewpoint movement into parallel movement of the hologram plane is compared with a threshold value T1ref. If T1>T1ref, the process proceeds to step S402, and all pixels are determined to be the "calculation method".

In step S403, a movement amount (rotational angle) T2 obtained by converting the detected viewpoint movement into a rotational movement of the hologram plane is compared with a threshold value T2_ref. If T2>T2ref, the process proceeds to step S404, and all pixels are determined to be the "calculation method".

In step S405, the movement amount T3 obtained by converting the detected viewpoint movement into the front-rear movement of the hologram plane is compared with a threshold value T3ref. If T3>T3ref, the process proceeds to step S406, and all pixels are determined to be the "calculation method".

According to the inventors' verification, when the object light wave distribution of the previous frame is scaled and reused in response to the forward and backward movement of the viewpoint, even if the movement amount T3 is the same, the distance from the center of the hologram plane It has been confirmed that, as the distance is increased, an error occurs in the presentation position of the reconstructed image to be reconstructed, resulting in deterioration in viewing quality of the hologram.

Therefore, as shown in FIG. 20, different threshold values T3ref1, T3ref2, and T3ref3 (T3ref1>T3ref2>T3ref3) are provided according to the distances for the forward and backward movement of the hologram plane. On the other hand, if T3ref1≧T3>T3ref2, only the pixels of the intermediate area and the outer area excluding the inner area are determined as the "calculation method", and if T3ref2≧T3>T3ref3, the inner area and the intermediate area are determined. Only the pixels in the outer area excluding the area may be determined as the "calculation method", and if T3ref3≧T3, none of the pixels may be determined as the "calculation method".

In step S407, the weighted sum ΣT=αT1+βT2+γT3 (α, β, γ are the weight values of the respective movement amounts) of the respective movement amounts T1, T2, T3 is calculated and compared with the threshold value T123ref. If ΣT>T123ref, the process advances to step S408, and all pixels are determined to be the "calculation method".

In the above embodiment, the "reuse method" is adopted when the amount of movement is less than the predetermined threshold, and the "calculation method" is adopted when the amount of movement is equal to or greater than the predetermined threshold. As such a threshold, for example, based on the performance of the display device 5, the amount of movement exceeding the display limit can be set as the threshold.

For example, in parallel movement of the hologram plane, theoretically, if the movement exceeds the size of the hologram plane, the "reuse method" cannot be applied. For example, the pixel pitch of the liquid crystal device used in the optical system shown in Non-Patent Document 2 is 9.6 [μm], and the number of pixels is 1280 × 768 [pixels]. In this case, this size is 9.6 [μm]×1280≈12 [mm]. Therefore, the threshold can be set to 12 mm in the above case.

On the other hand, regarding the limit of forward and backward movement, the inventor placed an object at a depth of 75 cm from the viewpoint and moved the viewpoint position from the object to a depth of about 60 to 90 cm using the device described in Non-Patent Document 2. After creating the content and conducting a subjective evaluation experiment based on the DCR (Degradation Category Rating) method for more than 10 people, the result was that if the entire surface was recalculated every time it moved about 1 cm, deterioration would not be noticeable. was taken.

Regarding the occurrence of degradation, it is thought that it will change according to the nature of the scene and the speed of viewpoint movement, so it is difficult to indicate the limit in general, but if an object is at a depth of about 50 cm to 100 cm If it exists, it is considered that the entire surface should be recalculated for each movement of about 1 cm as a guideline.

Also, in general, regarding the forward/backward movement of the viewpoint, if an object is at a distance of 2m, it will move forward by 1cm, and if an object is at a distance of 50cm, it will move forward by 1cm. , the difference in the presentation position of the reconstructed image is more conspicuous because the visual change seems to be larger when the object exists at the position of 50 cm. Therefore, the application conditions of each method are changed according to the distance from the viewpoint. can be In this way, the number of pixels to which the "reuse method" can be applied can be increased while maintaining quality, so that high-speed calculation of holograms by reusing object light waves can be further promoted while suppressing deterioration in quality.

Furthermore, in the above embodiment, the amount of movement of the viewpoint is used as an index for selecting the "reuse method" or the "calculation method" as the method of acquiring the object light wave in the next frame. Also good.

For example, if the desired quality can be obtained by keeping the frame rate at 30 fps, the method selection based on the amount of movement of the viewpoint will reduce the number of pixels that can be reused and increase the calculation load of the object optical path. If it is not possible to maintain a frame rate of 30 fps, adopt the "reuse method". The pixel area may be expanded.

In this case, since the calculation time required to apply the "reuse method" is extremely small compared to the calculation time required to apply the "calculation method", the "calculation method" is adopted for the entire calculation time for one frame. It can be represented by the calculation time when Therefore, the threshold is set in advance or dynamically so that the "calculation method" is applied only to the pixel area of the ratio obtained by dividing the allowable processing time for one frame by the calculation time required for the "calculation method". can be

Alternatively, if a frame rate of 30 fps cannot be maintained if the "calculation method" is applied to the entire hologram surface, but if a frame rate of 30 fps can be maintained if the "calculation method" is applied to half the area, for example, the left half is 2n The frame rate of 30 fps may be maintained by applying the "calculation method" to the 2n+1 frame in the right half of the frame.

Furthermore, as shown in Non-Patent Document 6, based on the pixel pitch of the liquid crystal in advance, the object points that should be visible at the time of reproduction are calculated in advance, and the invisible object points are deleted in advance before the light wave in the latter stage. You may make it pass to a propagation part.

1...control circuit, 2...optical lens, 3...reproduction illumination light source, 4...half mirror, 5...display device, 6a, 6b, 6...support member, 101.. Scene setting unit 102...Viewpoint movement detection unit 103...Method determination unit 104...Object light wave generation unit 105...Interference calculation unit 106...CGH output unit 401.. Rendering unit 402 Light wave propagation calculation unit 403 Object light wave reuse unit

Claims (11)

In a computer- generated hologram generator used in an optical system that expands the field of view using a lens,
means for detecting movement of the viewer's viewpoint based on an output signal from a sensor that detects movement of the viewer's head in the hologram reproduction image;
means for determining, for each pixel on the hologram surface, an object light wave acquisition method to be either a "calculation method" or a "reuse method" according to the movement of the viewpoint;
means for calculating an object light wave propagating to each pixel determined by the calculation scheme;
means for reusing the object light wave of the corresponding pixel of the previous frame for each pixel determined for the reuse method;
and means for calculating interference fringes on the hologram surface by light wave simulation based on the object light wave and the converging spherical reference light wave of each pixel. 2. The method according to claim 1, wherein the means for determining the acquisition method determines the reuse method when the amount of movement of the viewpoint is less than a predetermined threshold, and determines the calculation method when the amount of movement of the viewpoint is equal to or greater than the predetermined threshold. hologram generator. 3. A hologram generating apparatus according to claim 2, wherein said means for reusing expands or contracts the object light wave distribution of the previous frame in accordance with the direction of movement when a back-and-forth movement of the viewpoint is detected for reuse. . 4. The hologram generating apparatus according to claim 3, wherein the predetermined threshold is smaller for pixels whose distance from the center of the hologram plane is greater than or equal to a predetermined reference distance than for pixels whose distance is less than the predetermined reference distance. 5. The hologram generating apparatus according to claim 3, wherein the predetermined threshold value is larger for a pixel that records light waves from an object that is farther from the viewpoint. 6. A hologram generating apparatus according to any one of claims 3 to 5, wherein the scaling factor of the object light wave distribution is variable according to the distance from the hologram surface to the object to be recorded. The means for determining the acquisition method is configured to determine a pixel region after the parallel movement that overlaps with the pixel region before the parallel movement to be the reuse method in response to the parallel movement of the viewpoint, and to determine the other pixel regions to be the calculation method. 7. The hologram generating apparatus according to any one of claims 1 to 6. The means for reusing multiplies the object light wave distribution before the rotational movement by the phase difference caused by the difference in the optical path difference caused by the rotational movement of the viewpoint, and reuses it as the object light wave distribution after the rotational movement. Item 8. A hologram generation device according to any one of Items 1 to 7. means for detecting viewpoint movement;
9. The hologram generating apparatus according to any one of claims 1 to 8, further comprising means for displaying said interference fringes on a hologram surface. In the method of generating a computer-generated hologram used in an optical system that expands the field of view using a lens,
A procedure for detecting movement of the viewer's viewpoint based on an output signal from a sensor that detects movement of the viewer's head in the hologram reproduction image;
a procedure of determining , for each pixel on the hologram surface, an acquisition method of an object light wave to be either a "calculation method" or a "reuse method" according to the movement of the viewpoint;
a procedure for calculating an object light wave propagating to each pixel determined by the calculation method;
a procedure of reusing the object light wave of the corresponding pixel of the previous frame for each pixel determined as the reuse method;
calculating interference fringes on the hologram plane by light wave simulation based on the object light wave and the converging spherical reference light wave of each pixel. 11. The method according to claim 10, wherein, in the step of determining the acquisition method, the reuse method is determined if the amount of movement of the viewpoint is less than a predetermined threshold, and the calculation method is determined if the amount is equal to or greater than the predetermined threshold. hologram generation method.

Images