JP2022050768A - Element image group generation device and program thereof - Google Patents

Abstract

To provide an element image group generation device capable of generating an element image group at a high speed.SOLUTION: An element image group generation device 1 comprises: a light path calculation device 3 which calculates a light path of a light beam reaching the pupil surface from each of pixels of a display through a microlens in advance, and calculates a tracking vectors by inverting a directional vector of the light beam reaching the pupil surface; and a light beam tracking calculation device 4 which performs, based upon head part movement information on a wearer of an HMD 2, light beam tracking processing to acquire color information at a part where an extension of the tracking vector comes into contact with a 3D object in a virtual space, and generates an element image group.SELECTED DRAWING: Figure 1

Description

The present invention relates to an element image group generator and a program thereof for generating an element image group displayed by a light field HMD.

In recent years, the use of virtual reality (VR) is spreading in various fields such as education, medical care, and entertainment. The optical system of a head-mounted display (HMD), which is a VR display device, has a simple structure in which an image of a display is magnified by an eyepiece. This HMD is a mechanism that gives a three-dimensional effect by displaying images with parallax on two displays, one for the left eye and the other for the right eye.

One of the problems that this HMD has not been able to solve is eye strain caused by the mismatch between the focal length and the convergence distance. This problem is caused by the fact that the position where the eyes focus by the adjustment function and the position where the line-of-sight directions of both eyes intersect due to the convergence function do not match. Then, this problem can be solved by using a light field HMD that virtually reproduces the light source of the 3D object.

The light field HMD includes a display, a microlens array, and an eyepiece (Non-Patent Documents 1 and 2). The light field HMD does not necessarily have an eyepiece, and may include only a display and a microlens array (Patent Document 1). Then, the light field HMD displays an element image group for generating a light source of the 3D object on the display.

In the light field HMD, the element image group is generated by the optical path calculation process from each pixel of the display through the optical element to reach the pupil surface, and the color information by reaching the 3D object arranged in the virtual space from the pupil surface. It is classified as a ray tracing process until it is acquired.

Japanese Unexamined Patent Publication No. 2020-07388

H. Huang and H. Hua, “Generalized methods and strategies for modeling and optimizing the optics of 3D head-mounted light field displays,” Opt. Express, 27 (18), 25154, (2019). D. Lanman and D. Luebke, “Near-Eye Light Field Displays,” ACM Trans. Graph. 32 (6), 1-10 (2013).

However, the light field HMD has a problem that a large amount of calculation is required to generate an element pixel group, and it takes time to generate the element pixel group.

Therefore, it is an object of the present invention to provide an element image group generation device capable of generating element image groups at high speed and a program thereof.

In order to solve the above problems, the element image group generation device according to the present invention is an element image group generation device that generates an element image group to be displayed on a light field HMD including an optical element array composed of a display and optical elements. Therefore, the configuration is provided with an optical path calculation means and a ray tracing calculation means.

According to such a configuration, the optical path calculation means preliminarily calculates the optical path of the light ray that passes through the optical element and reaches a predetermined pupil surface from each pixel of the display, and inverts the direction vector of the light ray that reaches the pupil surface. Calculate the tracking vector.
The ray tracing calculation means performs ray tracing processing to acquire color information of a place where an extension line of the tracking vector contacts an object in the virtual space based on the head motion information of the wearer of the light field HMD, and performs an element image. Generate a swarm.

In this way, the element image group generating device performs the optical path calculation from each pixel of the display to reach the pupil surface through the optical element in advance. Then, the element image group generation device can generate the element image group at high speed by performing the ray tracing process according to the head movement of the wearer of the light field HMD based on the result of the optical path calculation. That is, the element image group generation device can shorten the generation time of the element image group by performing the optical path calculation in advance and appropriately performing the ray tracing process according to the wearer's head movement.

The present invention can be realized by a program for making a computer function as the element image group generation device described above.

According to the present invention, an element image group can be generated at high speed.

It is a block diagram which shows the structure of the element image group generation apparatus which concerns on 1st Embodiment. It is explanatory drawing explaining the optical path calculation in 1st Embodiment. It is explanatory drawing explaining the ray tracing process in 1st Embodiment. It is a flowchart which shows the operation of the optical path calculation apparatus which concerns on 1st Embodiment. It is a flowchart which shows the operation of the ray tracing calculation apparatus which concerns on 1st Embodiment. It is a block diagram which shows the structure of the element image group generation apparatus which concerns on 2nd Embodiment. In the second embodiment, it is explanatory drawing explaining the setting of the unique number and the center position flag of an element image, (a) is the front view of the display, and (b) is the side view of the display.

Hereinafter, each embodiment of the present invention will be described with reference to the drawings. However, each embodiment described below is for embodying the technical idea of the present invention, and the present invention is not limited to the following unless otherwise specified. Further, the same means may be designated by the same reference numerals and the description thereof may be omitted.

(First Embodiment)
With reference to FIG. 1, the configuration of the element image group generation device 1 according to the first embodiment will be described.
The element image group generation device 1 generates and displays an element image group to be displayed on the light field type HMD2, and as shown in FIG. 1, the optical path calculation device (optical path calculation means) 3 and the ray tracing calculation. A device (ray tracing calculation means) 4 is provided. In this embodiment, it is assumed that the optical path calculation device 3 is mounted on the workstation and the ray tracing calculation device 4 is built in the HMD 2. Further, the HMD 2 and the optical path calculation device 3 are to perform wireless communication.

[HMD configuration]
The configuration of HMD2 will be described with reference to FIGS. 1 and 2.
The HMD 2 is a general light field type HMD, and includes a display 20, a microlens array (optical element array) 22, an eyepiece 24, and a storage means 25, as shown in FIGS. 1 and 2. ..

The display 20 is a flat display in which pixels 21 are arranged two-dimensionally, similar to a general smartphone. Further, the display 20 is composed of a display 20 _R for the right eye and a display 20 _L for the left eye, and displays a group of element images for the right eye and the left eye. Here, the element images for the right eye and the element images for the left eye may be displayed on the left and right sides of one display 20, respectively.

The microlens array 22 is a two-dimensional arrangement of microlenses (optical elements) 23. Further, the microlens array 22 is composed of a microlens array 22 _R for the right eye and a microlens array 22 _L for the left eye (see FIG. 3). For example, the microlens 23 is a minute biconvex lens.
The eyepiece 24 is a lens that magnifies an intermediate image formed after a light ray from an element image displayed by the display 20 passes through the microlens array 22. Further, the eyepiece lens 24 is composed of an eyepiece lens 24 _R for the right eye and an eyepiece lens 24 _L for the left eye (see FIG. 3). Here, the eyepiece 24 is located on the side of the wearer of the HMD2.

The pupil surface α is a virtual plane on which both eyes of the wearer of HMD2 are located.
The image plane β is a virtual plane on which an image of a 3D object (object) A is formed.

The storage means 25 is a storage device such as a memory or an SD card that stores various information necessary for generating an element image group. For example, the storage means 25 stores the information table I, the head movement information H, and the virtual environment information K. The details of various information stored in the storage means 25 will be described later.

[Configuration of optical path calculation device]
The optical path calculation device 3 performs optical path calculation in advance, and as shown in FIG. ₂ , calculates a tracking vector V3 in which the direction vector of the light ray reaching the pupil surface α is inverted. The optical path calculation is to calculate the optical path of a light ray passing through the microlens 23 and the eyepiece 24 from each pixel 21 of the display 20 and reaching the pupil surface α.

As shown in FIG. 1, the optical path calculation device 3 includes a microlens array corresponding means (optical element corresponding means) 30, an eyepiece refraction means 31, a pupil surface retroreflection means 32, and an information table generation means 33. ..

<Measures for Microlens Arrays>
The microlens array compatible means 30 performs microlens array compatible processing. As shown in FIG. 2, the microlens array compatible process corresponds to a pixel 21 and a microlens 23, and a vector after the microlens (post-optical vector) of a light ray passing from the pixel 21 to the center of the microlens 23. ) This is _a process for calculating V1. In FIG. 2, only one pixel 21 is associated with the microlens 23, but all the pixels 21 are associated with the microlens 23. Further, in FIG. 2, only one of the right eye and the left eye is shown, but both eyes have the same configuration.

Here, a parameter including a display pixel arrangement, a microlens array arrangement, a microlens array focal length, a microlens array pitch, and an eyepiece lens focal length is input to the microlens array corresponding means 30.

The display pixel arrangement represents the arrangement (i, j) of the pixels 21 constituting the display 20.
The microlens array arrangement represents the arrangement (k, l) of the microlenses 23 constituting the microlens array 22.
The microlens array focal length represents the focal length of the microlens 23 constituting the microlens array 22.
The microlens array pitch represents the pitch (interval) of the microlenses 23 constituting the microlens array 22.
The eyepiece focal length represents the focal length of the eyepiece 24.

Specifically, the microlens array-compatible means 30 uses the pixel 21 of the display 20 as a light source and calculates a vector of light rays passing through the microlens 23 associated with the pixel 21. As shown in FIG. 2, the starting point position of the light beam is the pixel position _ODij of each pixel 21 constituting the display 20. Further, the microlens rear vector V ₁ is a unit vector from the pixel position _ODij of the pixel 21 to the center position _OLAkl of the microlens 23 associated with the pixel 21.

Here, there are several methods for associating the pixel 21 with the microlens 23. For example, as the simplest method, there is a method in which the microlens 23 closest to the pixel 21 is associated with each pixel 21 of the display 20. In this method, among the above-mentioned parameters, the display pixel arrangement and the microlens array arrangement are used.

Further, as another method, there is a method in which the light beam from the pixel 21 of the display 20 passes through the microlens 23 and is incident on the wearer's eyeball, and then the microlens 23 is associated with the microlens 23. In this method, the pixel 21 is located within the radius of the microlens 23 from the center of the point where the straight line that is refracted by the eyepiece 24 and passes through the center of the microlens 23 touches the display 20 with the center of the pupil surface α as the starting point. To associate. In this method, among the above-mentioned parameters, the display pixel arrangement, the microlens array arrangement, the microlens array focal length, the microlens array pitch, and the eyepiece lens focal length are used.

After that, the microlens array corresponding means 30 outputs the display pixel arrangement and the microlens rear vector V ₁ to the eyepiece refraction means 31.

<Eyepiece refraction means>
_The eyepiece refraction means 31 inputs the display pixel arrangement and the microlens rearward vector V1 from the microlens array compatible means 30, and performs the eyepiece refraction processing.

As shown in FIG. 2, the eyepiece refraction process is to extend the posterior vector V ₁ of the microlens to obtain the posterior vector V ₂ of the light beam that has passed through the center of the microlens 23 and is refracted by the eyepiece 24. It is a process to calculate. That is, in the eyepiece lens refraction processing, the light beam emitted from the pixel 21 and passing through the microlens 23 is regarded as incident light, and the optical path of the light ray refracted by the eyepiece lens 24 is calculated. In this eyepiece refraction processing, the eyepiece focal length is used among the above-mentioned parameters.

The starting point position of the posterior eyepiece vector V ₂ is the contact position _OEP between the posterior microlens vector V ₁ and the eyepiece 24. Further, the posterior vector V ₂ of the eyepiece is a unit vector toward the direction refracted by the eyepiece 24.

After that, the eyepiece refraction means 31 outputs the post-eyepiece vector V ₂ to the pupil surface retroreflection means 32.

<Eye plane retroreflective means>
The pupil surface retroreflection means 32 inputs the eyepiece posterior vector V ₂ from the eyepiece refraction means 31 and performs the pupil surface retroreflection process.

As shown in FIG. 2, the pupil surface retroreflection process is a process of calculating a tracking vector V ₃ in which the direction of the eyepiece post-lens vector V ₂ is reversed on the pupil surface α. That is, in the pupil surface retroreflection process, the position where the light ray (the vector V ₂ after the eyepiece) that has passed through the eyepiece lens 24 reaches the pupil surface α is calculated from the contact determination with the plane corresponding to the pupil surface α. Here, the sign of the vector V ₂ after the eyepiece is inverted, and the tracking vector V ₃ = (u, v, w) for performing the ray tracing process. Further, the position where the vector V ₂ after the eyepiece reaches the pupil surface α is defined as the pupil surface position O _Eye = (x, y, z).

After that, the pupil surface retroreflection means 32 outputs the display pixel arrangement, the pupil surface position O _Eye , and the tracking vector V ₃ to the information table generation means 33.

<Information table generation means>
The information table generation means 33 performs information table generation processing. This information table generation process is a process of generating the display pixel arrangement, the pupil surface position O _Eye , and the tracking vector V3 input from the pupil surface retroreflection means ₃₂ as one file called the information table I. Then, the information table generation means 33 writes the information table I in the storage means 25.

[Structure of ray tracing calculator]
The ray tracing calculation device 4 performs ray tracing processing based on the wearer's head motion information H to generate an element image group. The ray tracing process is a process of acquiring color information at a point where the extension line of the tracking vector V ₃ contacts the 3D object A in the virtual space (FIG. 3). As shown in FIG. 1, the ray tracing calculation device 4 includes a binocular display means 40, a head movement means 41, a ray tracing means 42, and an element image group generating means 43.

<Binocular display means>
The binocular display means 40 refers to the information table I stored in the storage means 25 and performs the binocular display process. The binocular display process is a process of horizontally shifting the tracking vector V ₃ by a preset inter-eye distance d to generate tracking vectors V _3R and V _3L for the right eye and the left eye. Further, the distance between the eyeballs d represents the distance between the wearer's eyes and can be arbitrarily set. That is, in the binocular display process, in order to duplicate the information table I of both eyes from one eye, the pupil surface position _OEye is shifted ± d / 2 in the x direction according to the inter-eye distance d. The binocular display process may be performed only for the first time. In FIG. 3, the pupil surface positions _OEyeR1 and _OEyeR2 of the right eye, the pupil surface positions _OEyeL1 , _OEyeL2 of the left eye, the tracking vector V _3R1 and V _3R2 of the right eye, and the tracking vector V _3L1 of the left eye are shown. V _3L2 is illustrated.
After that, the binocular display means 40 outputs the information table I of the right eye and the left eye to the head movement means 41.

Hereinafter, before explaining the head movement means 41, the head movement information H and the virtual environment information K stored in the storage means 25 will be described.
The head movement information H is information representing the head movement of the wearer. The head motion information H includes the position difference of the head for each frame and the integrated value of the angular velocity. Here, it is assumed that the acceleration sensor (not shown) provided in the HMD 2 measures the head movement information H and writes it in the storage means 25.

The virtual environment information K is information necessary for drawing the 3D object A in the virtual space. The virtual environment information K includes mesh information (vertex coordinates, normal vector, color information) of 3D object A in virtual space and light source information (for example, position, color, brightness of light source B) in virtual space. And are included. For example, as the virtual environment information K, an OBJ format file can be mentioned. Here, it is assumed that the administrator of the element image group generating device 1 manually writes the virtual environment information K in the storage means 25.

<Head movement means>
The head movement means 41 inputs information tables I for the right eye and the left eye from the binocular display means 40, and performs head movement processing. This head movement processing is a processing in which a rotation matrix of the head determined from the head movement information H is applied to the tracking vector _{V3 of both eyes and the pupil surface position OEye with} reference to the position of the center of gravity of the head of the wearer. be.

Specifically, the head movement means 41 extracts the pupil surface positions _OEyeR , _OEyeL and the tracking vectors _V3R , _V3L from the information table I of the right eye and the left eye. Further, the head movement means 41 reads out the head movement information H from the storage means 25. As described above, the head motion information H includes the position difference of the head for each frame and the integrated value of the angular velocity. Therefore, the head motion means 41 has a rotation matrix R = [R _x , R _y , R of the head on each of the x-axis, y-axis, and z-axis from the translation matrix _TH of the position difference and the integrated value of the angular velocity. _z ] is calculated. Then, the head movement means 41 _applies a rotation matrix R to the pupil surface positions _OEyeR , _OEyeL and the tracking vectors V3R, _V3L with reference to the head center of gravity position OH = (x ₀ , y ₀ , z ₀ ). Apply.

After that, the head movement means 41 stores the pupil surface positions _OEyeR , _OEyeL and the tracking vectors V _3R , V _3L to which the rotation matrix R is applied in the information table I of the right eye and the left eye, and stores this information table I in the information table I. It is output to the ray tracing means 42.

<Ray tracing means>
The ray tracing means 42 performs ray tracing processing on the tracking vectors V _3R and V _3L to which the rotation matrix R is applied, and acquires color information at a point where the ray contacts the 3D object A.

Specifically, the ray tracing means 42 extracts the display pixel arrangement, the pupil surface positions _OEyeR , _OEyeL , and the tracking vectors _V3R , _V3L from the information table I input from the head movement means 41. Further, the ray tracing means 42 reads out the virtual environment information K from the storage means 25. As described above, the virtual environment information K includes the mesh information of the 3D object A in the virtual space and the light source information in the virtual space (see FIG. 3). Therefore, the ray tracing means 42 performs a ray tracing process with the tracking vectors V _3R and V _3L starting from the pupil surface positions _OEyeR and _OEyeL , and obtains a portion that comes into contact with the surface of the 3D object A. Further, the ray tracing means 42 calculates the pixel value (for example, RGB value) of each pixel 21 of the display 20 by allocating the color information of the portion in contact with the 3D object A to the display pixel arrangement. In FIG. 3, the color of the 3D object A is represented by the type of hatching.

After that, the ray tracing means 42 outputs the color information (pixel value) of each pixel 21 of the display 20 to the element image group generating means 43.

<Element image group generation means>
The element image group generating means 43 inputs the color information of each pixel 21 of the display 20 from the ray tracing means 42, and performs the element image group generating process. This element image group generation process is a process of storing the pixel value of each pixel 21 of the display 20 acquired by the ray tracing means 42 in each pixel of the element image and generating the element image group of both eyes.

After that, the element image group generation means 43 stores the pixel values for the pixels of all the element images, and then outputs the element image groups of the right eye and the left eye to the display 20. That is, the element image group generation means 43 outputs the element image group of the right eye to the display 20 _R for the right eye, and outputs the element image group of the left eye to the display 20 _L for the left eye.

Since the ray tracing process and the element image group generation process are described in References 1 and 2 below, detailed description thereof will be omitted.

Reference 1: Katayama, 3D model to integral stereoscopic conversion method, NHK Science & Technical Research Laboratories R & D / No. 128, July 2011, [online], [Search on July 9, 2011], Internet <URL: https://www.nhk.or.jp/strl/publica/rd/rd128/PDF/P04 -10.pdf/>
Reference 2: Technology for generating an integral stereoscopic image from a 3D model, NHK Science & Technical Research Laboratories R & D / No. 123, September 2010, [online], [Search on July 9, 2010], Internet <URL: https://www.nhk.or.jp/strl/publica/rd/rd123/PDF/P64 .pdf />

[Operation of optical path calculator]
The operation of the optical path calculation device 3 will be described with reference to FIG.
As shown in FIG. 4, in step S1, the microlens array corresponding means 30 associates the pixel 21 with the microlens 23, and the microlens posterior vector V of the light ray passing from the pixel 21 to the center of the microlens 23. ₁ is calculated.

In step S2, the eyepiece refracting means 31 calculates the post-eyepiece vector V ₂ of the light beam that has passed through the center of the microlens 23 and is refracted by the eyepiece 24 by extending the post-microlens vector V ₁ .
In step S3, the pupil plane retroreflective means 32 calculates the tracking vector V ₃ in which the direction of the eyepiece posterior vector V ₂ is reversed in the pupil plane α.
In step S4, the information table generation means 33 generates the information table I.

[Operation of ray tracing calculator]
The operation of the ray tracing calculation device 4 will be described with reference to FIG.
As shown in FIG. 5, in step S10, the binocular display means 40 horizontally shifts the tracking vector V ₃ by the preset inter-eye distance d, and the tracking vectors V _3R , V of the right eye and the left eye. Generate _3L .
In step S11, the head movement means 41 determines the head determined from the head movement information H in the tracking vectors V _3R , V _3L and the pupil surface positions _OEyeR , _OEyeL of both eyes with reference to the position of the center of gravity of the head of the wearer. The rotation matrix R of the part is applied.

In step S12, the ray tracing means 42 performs ray tracing processing on the tracking vectors V _3R and V _3L to which the rotation matrix R is applied, and acquires color information at a point where the ray contacts the 3D object A.
In step S13, the element image group generating means 43 stores the pixel value acquired in step S12 in each pixel of the element image and generates the element image group of both eyes.
In step S14, the element image group generating means 43 outputs the element image group generated in step S13 to the display 20.

In step S15, the element image group generating means 43 determines whether or not the generation of the element image group is completed in all frames.
If it does not end (No in step S15), the ray tracing calculation device 4 returns to the process of step S11.
When finished (Yes in step S15), the ray tracing calculation device 4 ends the process.

In this way, the ray tracing calculation device 4 repeatedly performs the processes of steps S11 to S14 for each frame to generate a moving image.

[Action / Effect]
As described above, the element image group generating device 1 according to the first embodiment preliminarily calculates the optical path from each pixel 21 of the display 20 through the microlens 23 and the eyepiece 24 to reach the pupil surface α. And generate the information table I. Then, the element image group generation device 1 can generate the element image group at high speed by performing the ray tracing process according to the wearer's head movement based on the information table I. As a result, the element image group generation device 1 can significantly shorten the image generation process in the light field type HMD2, and can shorten the time for generating the element image group for each frame. Further, if the configuration of the HMD 2 is the same, the element image group generating device 1 can use the information table I and omit the optical path calculation.

Here, since the light field method is similar to the integral method, the advantages when compared with the integral method will be described.
In the case of the integral method, since the three-dimensional images that can be seen from various viewpoints are displayed on the display, there is no need to regenerate the element image group even if the viewpoint position changes, and it is necessary to perform optical path calculation in advance. do not have. On the other hand, in the case of the light field method, only the stereoscopic image seen from the eyeball position is displayed on the HMD 2. Therefore, in the light field method, when the viewpoint position changes, it is necessary to generate an element image group according to the viewpoint position, and there is an advantage that the optical path calculation is performed in advance.

(Second Embodiment)
With reference to FIG. 6, the configuration of the element image group generating device 1B according to the second embodiment will be described as being different from the first embodiment.
The element image group generation device 1B performs the ray tracing process only in the vicinity of the central visual field of the wearer in advance, and partially generates the element image group to speed up the ray tracing process. As shown in FIG. 6, the element image group generating device 1B includes an optical path calculation device (optical path calculation means) 3B and a ray tracing calculation device (ray tracing means) 4B.

[HMD configuration]
The configuration of HMD2 will be described with reference to FIG.
As shown in FIG. 6, the HMD 2B includes a display 20, a microlens array (optical element array) 22, an eyepiece 24, a storage means 25, and a line-of-sight detection device 26. Further, the HMD 2B has a built-in ray tracing calculation device 4B.
Since the configurations other than the line-of-sight detection device 26 are the same as those in the first embodiment, the description thereof will be omitted.

The line-of-sight detection device 26 detects the line-of-sight direction of the wearer and outputs the line-of-sight direction vector G to the ray tracing calculation device 4B. Here, the line-of-sight detection device 26 can use a general line-of-sight direction detection method.

[Configuration of optical path calculation device]
As shown in FIG. 6, the optical path calculation device 3B includes a microlens array corresponding means (optical element corresponding means) 30B, an eyepiece refraction means 31, a pupil surface retroreflection means 32, and an information table generation means 33B. ..
Since the eyepiece refraction means 31 and the pupil surface retroreflection means 32 are the same as those in the first embodiment, the description thereof will be omitted.

<Measures for Microlens Arrays>
The microlens array corresponding means 30B performs the microlens array corresponding processing as in the first embodiment.
Further, the microlens array corresponding means 30B assigns a unique number of the element image to _each of the display pixel arrangements when calculating the vector V1 after the microlens. Specifically, the microlens array corresponding means 30B assigns the same unique number to the pixel 21 corresponding to the center position _OLAkl of the same microlens 23. That is, the pixels 21 to which the same unique number is assigned belong to the same element image. Further, the microlens array corresponding means 30B sets the center pixel flag to the pixel 21 located at the center among the pixels 21 to which the same unique number is assigned.

As shown in FIGS. 7A and 7B, the microlens array corresponding means 30B is assigned a unique number '1' because the 16 pixels 21 on the upper left belong to the same element image. Further, in the microlens array corresponding means 30B, among the pixels 21 having the unique number '1', the pixel 21 in the second column from the top and the second row from the left is located at the center, so that the center pixel flag is set for this pixel 21. Set. In FIG. 7, the pixel 21 in which the center pixel flag is set is shown by hatching.

Further, the microlens array corresponding means 30B assigns a unique number '2' to 16 pixels constituting the element image on the right side. Further, the microlens array corresponding means 30B sets the center pixel flag for the pixel 21 located at the center of the pixels 21 having the unique number '2' (the same applies to the unique numbers '3' to '16').

After that, the microlens array corresponding means 30B outputs the display pixel arrangement, the microlens rear vector V ₁ , the unique number of the element image, and the center pixel flag to the eyepiece refraction means 31.

<Information table generation means>
The information table generation means 33B generates an information table I including a unique number of the element image and a center pixel flag in addition to the display pixel arrangement, the pupil surface position _{OEye ,} and the tracking vector V3. Then, the information table generation means 33B writes the information table I in the storage means 25.

[Structure of ray tracing calculator]
As shown in FIG. 6, the ray tracing calculation device 4B includes a binocular display means 40, a head movement means 41, a ray tracing means 42B, and an element image group generating means 43.
Since each means other than the ray tracing means 42B is the same as that of the first embodiment, the description thereof will be omitted.

<Ray tracing means>
The ray tracing means 42B performs ray tracing processing with the tracking vector V3 in the direction closest to the line _- of-sight direction of the wearer, and acquires color information at a point where the ray contacts the 3D object A.

Specifically, the ray tracing means 42B extracts the display pixel arrangement, the pupil surface positions _OEyeR , _OEyeL , and the tracking vectors _V3R , _V3L from the information table I input from the head movement means 41. Further, the ray tracing means 42B reads out the virtual environment information K from the storage means 25. Further, the ray tracing means 42B inputs the line-of-sight direction vector G from the line-of-sight detection device 26.

Next, the ray tracing means 42B compares the tracking vector V ₃ of the pixel 21 having the central pixel flag with the line-of-sight direction vector G, and obtains the tracking vectors V _3R and V _3L most parallel to the line-of-sight direction vector G. Then, the ray tracing means 42B extracts the unique number of the element image included in the central visual field with reference to the unique number of the most parallel tracking vectors V _3R and V _3L . Further, the ray tracing means 42B performs ray tracing processing with the tracking vectors V _3R and V _3L of the pixel 21 having the extracted unique number.

The size of this central field of view can be set arbitrarily. For example, the size of the central visual field is 1/2 of the total visible field of view. If there is a margin in the drawing speed of the element image group, the size of the central visual field may be changed according to the speed of the calculation process, such as increasing the size of the central visual field.

In the example of FIG. 7, it is assumed that the unique numbers of the tracking vectors V _3R and V _3L most parallel to the line-of-sight direction vector G are '6'. Further, it is assumed that the size of the central visual field is a 3 × 3 element image. In this case, the ray tracing means 42B extracts the unique numbers '1 to 3, 5, 7, 9 to 11' of the surrounding one-element image based on the unique number '6'. Then, the ray tracing means 42B performs the ray tracing process with the tracking vectors V _3R and V _3L of the pixel 21 having the unique numbers '1 to 3, 5 to 7, 9 to 11'.

[Action / Effect]
As described above, the element image group generating device 1B according to the second embodiment uses the information table I and the line-of-sight direction vector G to generate only the element images corresponding to the line-of-sight direction. As a result, the element image group generation device 1B can reduce the amount of calculation and generate the element image group at a higher speed.

Although the embodiments of the present invention have been described in detail above, the present invention is not limited to this, and includes design changes within a range that does not deviate from the gist of the present invention.
In each of the above-described embodiments, the optical element array has been described as being a microlens array 22, but the present invention is not limited thereto.

In each of the above embodiments, the optical path calculation device 3 is mounted on the workstation and the ray tracing calculation device 4 is built in the HMD 2, but the present invention is not limited to this. For example, both the optical path calculator 3 and the ray tracing calculator 4 may be mounted on the workstation. In this case, the element image group generated by the ray tracing calculation device 4 is output to the HMD 2. Further, both the optical path calculation device 3 and the ray tracing calculation device 4 may be built in the HMD 2.

In each of the above-described embodiments, the HMDs 2 and 2B have been described as including the eyepieces 24, but the eyepieces 24 may not be provided. In this case, the eyepiece refraction means 31 does not perform the eyepiece refraction processing. Then, the pupil surface retroreflection means 32 performs the pupil surface retroreflection process by using the _microlens posterior vector _V1 instead of the eyepiece posterior vector V2. In this case, in the pupil plane retroreflection process, the tracking vector V ₃ in which the direction of the vector V ₁ after the microlens is inverted is calculated in the pupil plane α.

The present invention can also be realized by a program that operates the hardware resources such as the CPU, memory, and hard disk of the computer as the element image group generation device described above. These programs may be distributed via a communication line, or may be written and distributed on a recording medium such as a CD-ROM or a flash memory.

1,1B element image group generator 2,2B HMD
20 Display 21 Pixels 22 Microlens Array (Optical Element Array)
23 Microlens (optical element)
24 Eyepiece 25 Storage means 26 Line-of-sight detection device 3, 3B Optical path calculation device (optical path calculation means)
30,30B Microlens array compatible means (optical element compatible means)
31 Eyepiece refraction means 32 Eye retroreflection means 33, 33B Information table generation means 4, 4B Ray tracing calculation device (ray tracing calculation means)
40 Binocular display means 41 Head movement means 42, 42B Ray tracing means 43 Element image group generation means

Claims (6)

An element image group generator that generates an element image group to be displayed on a light field HMD including a display and an optical element array.
An optical path calculation means for calculating an optical path of a light ray passing through an optical element from each pixel of the display and reaching a predetermined pupil surface in advance, and calculating a tracking vector obtained by reversing the direction vector of the light ray reaching the pupil surface. ,
Based on the head motion information of the wearer of the light field HMD, a ray tracing process is performed to acquire color information of a place where an extension line of the tracking vector contacts an object in the virtual space, and the element image group is generated. Ray tracing calculation method and
An element image group generating device characterized by comprising. The optical path calculation means is
An optical element-compatible means for calculating the post-optical vector of a light ray passing through the center of the optical element from each pixel of the display.
In the pupil surface, a pupil surface retroreflection means for calculating the tracking vector obtained by reversing the direction of the vector after the optical element, and
The element image group generation device according to claim 1, wherein the element image group generation device is provided. The light field HMD further comprises an eyepiece.
The optical path calculating means further includes an eyepiece refracting means that extends the posterior vector of the optical element and calculates the posterior vector of the light beam that has passed through the center of the optical element and is refracted by the eyepiece.
The element image group generation device according to claim 2, wherein the pupil surface retroreflection means calculates the tracking vector in the pupil surface by reversing the direction of the vector after the eyepiece. The ray tracing calculation means is
A binocular display means for calculating the tracking vector for both eyes by horizontally shifting the tracking vector by a preset distance between the eyeballs.
A head movement means that applies a rotation matrix of the head determined from the head movement information to the tracking vector of both eyes with reference to the position of the center of gravity of the head of the wearer.
A ray tracing means that performs the ray tracing process with the tracking vector of both eyes to which the rotation matrix is applied and acquires color information of a place where the ray contacts the object.
An element image group generating means that stores the color information acquired by the ray tracing means in each pixel of the element image and generates the element image group.
The element image group generating apparatus according to any one of claims 1 to 3, wherein the element image group generating apparatus is provided. The ray tracing means is characterized in that the ray tracing process is performed with the tracking vector in the direction closest to the line-of-sight direction of the wearer, and the color information at the point where the ray contacts the object is acquired. Item 4. The element image group generation device according to Item 4. A program for causing a computer to function as the element image group generating device according to any one of claims 1 to 5.

Images