JP5632245B2 - Eyeglass field image display device - Google Patents

Description

  The present invention relates to a field-of-view image display device for glasses that displays a retinal image seen through a spectacle lens by simulation.

Conventionally, in a spectacle store, a spectacle user uses a sample lens to check the appearance and selects and orders a lens and a frame.
However, the types of sample lenses that can be prepared at an eyeglass store are limited. In particular, since there are so many kinds of progressive-power lenses, there are not always lenses suitable for spectacle users among sample lenses. Therefore, it is not clear how it will look until the ordered lens is completed and actually viewed.

In view of this, it has been proposed to display a retinal image that can be seen when wearing spectacles by simulation to show the appearance when a lens of a type not included in the sample lens is used (see, for example, Patent Document 1). ).
In the progressive-power lens, the distance portion and the near portion having different powers are smoothly connected to each other, so that distortion is generated, and when the face direction is changed, shaking is felt. By displaying a retinal image that can be seen when wearing spectacles by simulation, it is possible to easily show and explain this shaking and the like.

  In the configuration described in Patent Document 1, the range of the image is changed based on the change in the direction of the face, and distortion aberration image processing is performed on the image in the range.

In addition to the horizontal and vertical movements, the eyeball performs a rotational movement around the longitudinal axis.
And the method of measuring a motion of an eyeball also including a rotation motion is proposed (for example, refer to patent documents 2, a nonpatent literature 1, and a nonpatent literature 2).

Japanese Patent No. 3893760 JP 2005-66358 A

Yusuke Sakashita, Hironobu Fujiyoshi, Yutaka Hirata, “Three-dimensional eye movement measurement considering refraction in cornea”, Proceedings of 13th Image Sensing Symposium, 2007, IN2-25 Yusuke Sakashita, Hironobu Fujiyoshi, Yutaka Hirata, “3D Eye Movement Measurement by Image Processing”, Experimental Mechanics, Vol. 6, no. 3, 2006, p. 236-243

Astigmatism power, prism, astigmatism, and the like that exist in the axial direction need to be arranged in accordance with the eyeball to be corrected.
However, in a specific case, the eyeball may perform fusion rotation (torsion), which is rotation about the line-of-sight direction.
And when the case where this fusion rotation (torsion) is performed was examined, it became clear that it occurred when the line-of-sight directions such as convergence were not in a parallel state.

Although the fusion of rotation due to the convergence has individual differences, the influence is particularly prominent when the prescription value has a large astigmatism amount (for example, an astigmatism amount of 2.0 D or more).
For this reason, when the fusion of the glasses is performed (torsion), the conventional simulation method may cause a difference between the simulation result and the actual situation.

  In order to solve the above-described problems, the present invention provides a field-of-view image display device for glasses that can display a retinal image by simulation in response to fusion of the eyeball. .

  The eyeglass field-of-view image display device according to the present invention is configured to display a retinal image that is visible when the eyeglass lens is worn, by simulation, and in the field of view corresponding to the direction of the line of sight of the eyeball when the eyeglass lens is worn. A retinal image is created by performing processing that adds at least the amount of eyeball rotation, including fusion rotation, which is a rotational motion around the visual axis of the eyeball, corresponding to the eye gaze passing point in the spectacle lens. And an image processing unit for displaying the retinal image created by the image processing unit.

  In the eyeglass field-of-view image display device according to the present invention, the image processing unit includes a line-of-sight movement angle amount (horizontal and vertical movement of the eyeball) from a reference position at a distance eye point in the line-of-sight direction, a refractive power of the eyeglass lens, and / or Depending on the prism, the torsion (fusion rotation) angle of the eyeball can be calculated.

In the eyeglass field-of-view image display device according to the present invention, as an example, the image processing unit calculates the fusion rotation angles θ _L (left eye) and θ _R (right eye) based on the listing law. Vector Y _C, which is the outer product of vector Y _L (left eye) and vector Y _R (right eye), which are vectors orthogonal to the line-of-sight direction, and a unit vector in the line-of-sight direction of the left eye and a unit vector in the line-of-sight direction of the right eye From these, it can be set as the structure calculated by the following formula.

  This example is a method for calculating a fusion rotation angle when a vertical horopter is assumed to be the outer product direction of the left eye gaze direction and the right eye gaze direction. An actual vertical horopter is not necessarily the outer product direction of the left eye gaze direction and the right eye gaze direction. In that case, the fusion rotation angles of the left and right eyes should be determined so that the longitudinal direction of the eyeball (retinal) and the longitudinal horopter direction are in the same plane.

According to the eyeglass field-of-view image display device of the present invention described above, the image processing unit converts the line-of-sight of the eyeglass lens into the original image data in the field of view corresponding to the direction of the eyesight of the eyeball when the eyeglass lens is worn. A retinal image is created by performing a process of adding at least the amount of eyeball rotation including the fusion of rotation, which is a rotational motion around the visual axis of the eyeball, corresponding to the passing point.
As a result, it is possible to create an image of a retinal image that has been processed to add the amount of eyeball rotation by reflecting not only the change in the face direction and the change in the direction of the line of sight but also the fusion of the eyeball. Therefore, even when the fusion of rotation occurs, the gaze portion can be accurately simulated.
Therefore, the eyeglass field image display device of the present invention can realize the eyeglass field image display device capable of displaying a retinal image close to the actual appearance in correspondence with fusion rotation.

1 is a schematic configuration diagram (block diagram) of a visual field image display device (display system) for glasses according to an embodiment of the present invention. It is an example of the three-dimensional CG model used in the system of FIG. It is a flowchart which shows the process until it displays the image of the simulation in the apparatus of FIG. It is a coordinate system used in the simulation of one embodiment of the present invention. It is a figure explaining distortion of the light ray by refraction of a lens. It is a figure explaining the ray tracing for calculating | requiring PSF. A, B It is a figure which shows the division | segmentation method of an entrance pupil. It is a figure which shows a response | compatibility with the image formation position on a retina, and an incident angle. It is a coordinate system for demonstrating fusion rotation. A, B It is a figure explaining the gaze direction of the right and left eyes in the case of binocular vision. It is a figure which shows the y direction common to both eyes. It is a figure which shows an equal probability ellipse. It is a flowchart which shows the process until it displays the image of the simulation in the visual field image display apparatus (display system) of the spectacles of other embodiment of this invention.

Hereinafter, the best mode for carrying out the invention (hereinafter referred to as an embodiment) will be described.
The description will be given in the following order.
1. 1. Explanation of configuration of apparatus / system according to one embodiment of the present invention 2. Description of theory and method of moving image simulation 2-1. Purpose of simulation 2-2. Coordinate system used for simulation 2-3. Explanation of lens distortion 2-4. Explanation of blurring by lens 2-5. Explanation about fusion of rotation (torsion) 2-6. Synthesis of simulation image 2-7. Coordinate conversion when the center line of sight is designated 2-8. Spline interpolation approximation of ray data 2-9. Simplification of PSF 2-10. Speeding up multidimensional B-spline interpolation calculation 2-11. Summary 3. 3. Description of another embodiment of the present invention Modified example

<1. Description of Configuration of Apparatus / System of One Embodiment of Present Invention>
As an embodiment of the present invention, FIG. 1 shows a schematic configuration diagram (block diagram) of a field-of-view image display device (display system) for glasses.

The system shown in FIG. 1 includes an HMD (head mounted display) 11, a PC (personal computer) 15, two monitors 16 and 17, and a game pad 18 or a keyboard 19 as input devices.
The HMD 11 includes a head motion sensor (gyro sensor or the like) 12 and a line-of-sight motion sensor (line-of-sight tracking device or the like) 13.
The PC 15 includes a graphic board 21 for an image for the right eye, a graphic board 22 for an image for the left eye, and a USB 23. An HMD controller 14 is connected between the HMD 11 and the PC 15. Since the graphic boards 21 and 22 are provided, the PC 15 operates as an image processing unit of the eyeglass field-of-view image display device of the present invention.

The apparatus / system of the present embodiment is intended to allow the progressive power lens to be seen from the viewpoint of “distortion” and “blurring”.
Therefore, an image processing effect of “distortion” and “blurring” is given to a three-dimensional CG (computer graphics) movie to be viewed by real-time calculation to reproduce the appearance of the progressive addition lens.

In the system shown in FIG. 1, stereoscopic display is possible by preparing a right-eye image and a left-eye image.
The field of view to be reproduced is a 3D CG model according to the movement of the viewer's line of sight by the gyro sensor (head movement sensor 12) and the line-of-sight tracking device (line-of-sight movement sensor 13) mounted on the HMD 11. The display is made to follow the field of view.

In the PC 15, image processing as a “distortion filter” and a “blurring filter” is performed.
The “distortion filter” is a mechanism that returns the value of the output coordinate (T ′, C ′) on the image side from the B-spline ray database for each pixel of the original image.
The “blurring filter” is a mechanism that returns the values of the size (σ _μ , σ _v ) and the direction coefficient (ρ) of the blurred texture from the B-spline ray database for each pixel of the original image.
The output visible image takes all pixels of the original image as input, and adds (overlapping) the blurred texture image returned through the “blur filter” to the coordinate position in the output image returned through the “distortion filter”. Generated.

An example of a three-dimensional CG model used in the system of FIG. 1 is shown in FIG.
In the space, five objects 31, 32, 33, 34, and 35 having shapes such as a sphere, a cylinder, a rectangular parallelepiped, and a cube are arranged. When viewed from the eyeglass user 41, the distances to the objects 31, 32, 33, 34, and 35 are various, the spherical object 31 is relatively close, and the objects 32 and 34 are relatively far away. .
In FIG. 2, along with the arrangement of the three-dimensional CG model, the line-of-sight passing points 46A and 46B on the spectacle lens 50 are also shown in the two-direction main lines of sight 43A and 43B.

With reference to FIG. 2, it will be described that the position of the line-of-sight passing point on the spectacle lens 50 changes depending on the position of the visual field.
First, consider a state in which the eyeglass user 41 looks at the object 32 and the object 33 and the visual field 44A is visible. At this time, the head 42 of the eyeglass user 41 faces the visual field 44A although not shown. The main line of sight 43A is directed toward the visual field center 45A of the visual visual field 44A. Here, since the visual field 44A is relatively far from the spectacle user 41, the position of the line-of-sight passing point 46A on the spectacle lens 50 is slightly above the center.
Next, it is assumed that the eyeglass user 41 looks at the object 34 and the object 35 and the visual field 44B is visible. At this time, the head 42 of the eyeglass user 41 faces the visual field 44B although not shown. The main line of sight 43B is directed to the visual field center 45B of the visual visual field 44B. Here, since the visual field 44B is relatively closer to the eyeglass user 41 than the visual field 44A, the position of the line-of-sight passing point 46B on the eyeglass lens 50 is larger than the line-of-sight passing point 46A. Located below and near the center of the spectacle lens 50.
Thus, not only the direction of the head 42 of the eyeglass user 41 changes due to the change of the visual fields 44A and 44B, but also the line of sight on the eyeglass lens 50 corresponding to the distance to the visual fields 44A and 44B. The positions of the passing points 46A and 46B also change.
For this reason, if the line-of-sight passing point is fixed simply by making it correspond to the movement of the head as in Patent Document 1, it will be very different from the actual appearance.
In the present embodiment, a simulation image is displayed using the system shown in FIG. 1 so as to correspond to the change in the position of the line-of-sight passing points 46A and 46B on the spectacle lens 50.

Next, the process until the simulation image is displayed is shown in the flowchart of FIG.
The simulation process in the system of FIG. 1 will be described below with reference to FIG.

First, the position, orientation (orientation of the head 42), and eye-gaze direction of the eyeglass user 42 are detected by a gyro sensor (head movement sensor 12) or a gaze tracking device (gaze movement sensor 13) mounted on the HMD 11. Is detected. In step S1, detection for the right eye is performed, and in step S2, detection for the left eye is performed.
On the other hand, in step S3, as shown in FIG. 2, a CG virtual object (three-dimensional CG model) is prepared.
Next, in step S4, using the CG walk-through function, a visual field to be cut out from the three-dimensional CG model is obtained based on the face position, orientation, and line-of-sight direction detected in steps S1 and S2. This visual field is different for the right eye and the left eye.
Next, in step S5, an original image having no distortion or blur for the right eye is created from the visual field cut out in step S4. Similarly, in step S6, in step S5, an original image without any distortion or blur for the left eye is created from the visual field cut out in step S4.
On the other hand, in step S7, the prescription power, the addition power, and the lens type of the eyeglass user 41 are input by an input device (keyboard 19 or the like).
Next, shape data, layout data, and a right eye model of the right eye lens are created from the input content in step S8, and shape data, layout data, and a left eye model of the left eye lens are created in step S9.
Next, based on the shape data, the layout data, and the eyeball model created in steps S8 to S9, a three-dimensional spline interpolation coefficient of light ray data is generated in step S10.
Next, using the three-dimensional spline interpolation coefficient of the ray data generated in step S10, in step S11, the three-dimensional spline interpolation for the right eye, the outgoing ray direction, the PSF parameter, the lens passing point position, and other various parameters. Find the coefficient. Similarly, in step S12, the three-dimensional spline interpolation coefficients for the left eye for the outgoing light direction, PSF parameter, lens passing point position, and other various parameters are obtained.
Next, a simulation is executed in step S13 using the original image created in steps S5 and S6 and the parameters and interpolation coefficients obtained in steps S11 and S12. This simulation process includes using image processing hardware.
Next, in step S14, an image including distortion and blur for the right eye is created. Similarly, in step S15, an image including distortion and blur for the left eye is created.
The image including distortion and blur generated in this way is displayed on the display screen of the HMD 11, the right eye monitoring monitor 16, and the left eye monitoring monitor 17.
Through the process described above, an image including distortion and blur corresponding to the viewing direction is displayed on the display screen of the HMD 11.

<2. Explanation of theory and method of video simulation>
2-1. The purpose of the simulation The purpose of this moving image simulation is to express how it feels when wearing glasses with a still image or a moving image.
By using a combination of 3D CG, HMD11, gyro sensor, and eye tracking device, it is possible to present in real time an image that can be seen when turning the head or changing the eye in the virtual space. .
Further, as shown in the system of FIG. 1 and the flowchart of FIG. 3, binocular stereoscopic vision is also possible if separate images are presented to the left and right eyes, respectively.

2-2. Coordinate System Used for Simulation FIG. 4 shows a coordinate system used for the simulation of this embodiment.
As shown in FIG. 4, a three-axis orthogonal coordinate system of x-axis, y-axis, and z-axis is configured. The x-axis is taken in the direction from the front to the eye. The y-axis is orthogonal to the x-axis and faces upward. The z-axis is the horizontal direction from right to left. The x-axis-y-axis-z-axis directions are in accordance with the right-hand rule. The origin is placed at the center of rotation of the eyeball. The broken line in the figure schematically shows the eyeball and its cornea.

In the coordinate system shown in FIG. 4, an arbitrary point P (x, y, z) in space (here, x <0, that is, in front of the eyes) is the angles β and γ of light rays entering the eye and the rotation center point. It can be expressed again by the distance PO.
The position on the simulation image is tan β = y / x in the vertical direction and tan γ = z / x in the horizontal direction. In the case of glasses, it is convenient to represent the object distance by the reciprocal number instead of representing it as it is. Accordingly, the position of an arbitrary point in space can be expressed as follows.

2-3. Explanation of lens distortion When viewed through a lens, light rays are refracted.
That is, the object point in the direction of (ψ, ζ) with the naked eye moves to (ψ ′, ζ ′) when viewed through the spectacle lens.
This will be described in more detail with reference to FIG. FIG. 5 shows a spectacle lens 50, an eyeball 51, and a rear vertex spherical surface 52 corresponding to the concave surface of the spectacle lens 50.
At the arbitrary point P shown in FIG. 5, the incident direction at the naked eye is PO, but when viewed through the spectacle lens 50 with glasses, the incident direction to the rotation center O of the eyeball 51 changes to RO.
Similarly, at point A shown in FIG. 5, the incident direction when the naked eye is AO is AO, but when viewed through the spectacle lens 50, the incident direction of the eyeball 51 to the rotation center O changes to BO.

Here, the position (ψ ′, ζ ′) of the object felt when wearing glasses can be expressed by a function of the position (D ₁ , ψ, ζ) when the eye is naked. That means
ψ ′ = ψ ′ (D _l , ψ, ζ)
ζ ′ = ζ ′ (D _l , ψ, ζ)
It can be expressed by the function The contents of this function can be determined by ray tracing described later.
In the case where there is a fusion rotation, it is necessary to perform coordinate conversion in accordance with the fusion rotation angle of the central line of sight. The principle of fusion will be described later.

2-4. Description of lens blur The cause of lens blur is that all rays from the object point do not converge on one point of the retina.
A light amount distribution in which light from the object point spreads in a range centered on the image point is formed. This distribution is called a point spread function (PSF).

A method for obtaining this function PSF will be described with reference to FIG.
When obtaining the PSF, first, the principal ray PQO passing through the point P is searched.
When the principal ray is determined, the entrance pupil is divided equally (for example, 400 divisions), the ray connecting from the point P to the center of each divided region is traced, and the point intersecting the retina is obtained.
In FIG. 6, each ray can be traced using the azimuth angle of the point P and the azimuth angle of the ray starting from the point P with respect to the principal ray.
Strictly speaking, the position of the entrance pupil is a conjugate point on the object side of the pupil, but it may be set to satisfy PO = PO ′ at one point O ′ on the extension line of the line PQ on the object side of the principal ray. .

In the case of PSF of the geometrical optical principle, the density of the retinal intersection is the PSF as it is.
When considering the effect of wave optics, the optical path difference of the light in each divided region is further calculated, and the PSF is obtained by Fresnel integration.

Next, various methods of dividing the entrance pupil can be considered.
As the main division method, there are two types, a square division shown in FIG. 7A and a helical division shown in FIG. 7B.
The square division shown in FIG. 7A divides an area vertically and horizontally and uses the center point of each area. In this case, it is simple and clear, but there are wasted parts at the four corners, and only about 70% of the planned number of rays can be traced.
On the other hand, the spiral division shown in FIG. 7B uses a point on a curve extending in a spiral shape from the center point of the entrance pupil. In this case, all the planned number of rays can be tracked.
Note that the spiral arrangement preferably includes at least two or more spirals. By including two or more spirals, the entrance pupil can be used more efficiently than an arrangement including only one spiral. In FIG. 7B, six spirals are included, and when the six spirals are included in this way, the entrance pupil can be used most efficiently.

The PSF thus obtained is the light density distribution on the retina surface, but the coordinates of the incident image are the coordinates (ψ, ζ) in the direction viewed from the center of rotation of the eyeball. Conversion with (ψ, ζ) is required.
Here, the relationship between the incident angle and the image height is shown in FIG. It is considered that the image height is small in the effective range of PSF, and sufficient accuracy can be obtained by paraxial calculation.
That is, ψ _m = y _m / f and ζ _m = z _m / f. In addition, f is a focal distance of eyes and changes with prescription frequencies.

In this way, the light amount distribution at the position on the retina can be converted into the light amount distribution in the incident light direction. That is, the eyes feel that light from an object point comes not only from the object point but also from a certain range of space centered on the object point.
The neighboring points affect each other, making it difficult to distinguish and appear blurry.

Naturally, the PSF when viewed through different positions on the lens is different.
Further, even when viewing through the same position on the lens, if the object distance is different, the PSF is also different.
Furthermore, even when looking at the same object distance point through the same position on the lens, the PSF is different if the eye adjustment state is different.
When there is a fusion, the positional relationship between the eyeball and the lens changes by the fusion angle, and the PSF also changes. In particular, in the case of an astigmatism prescription (an astigmatism component is included in the power of the eyeball), attention is required.

2-5. Description of Fusion Rotation (Torsion) As described above, when the line-of-sight direction is not in a parallel state, the eyeball may perform fusion rotation.
The rotation of the eyeball when looking far away is based on the Listing's Law. This listing rule is a rule that determines the posture when the eyeball is directed in a certain direction in space.
The posture of the eyeball refers to the horizontal direction and the vertical direction of the eyeball. If the posture of the eyeball is not determined, the top, bottom, left and right of the retinal image cannot be determined.
However, only by determining the direction of the line of sight, that is, the direction of the optical axis of the eyeball, the posture of the eyeball can take all directions rotating around the line of sight.

The listing law defines the posture of the eyeball in an arbitrary gaze direction at infinity.
For listing laws, see, for example, “Visual Information Processing Handbook” p. 405 states that "any rotation of one eye can be considered to occur about an axis in one plane (listing plane)".

This will be described using the coordinate system shown in FIG.
The coordinate system shown in FIG. 9 is similar to the coordinate system shown in FIG. 4, the X axis is the direction from the front (horizontal front) to the eye, the Y axis is the upward direction orthogonal to the X axis, the Z axis is the horizontal direction, It is said. The X-axis direction in FIG. 9 is also called the first eye position.
And the eyeball and the point R of the center of rotation of the eyeball are shown. The YZ plane is the above-described listing plane.

The posture after the eyeball rotation in an arbitrary direction is the same as the rotation around the straight line in the listing plane including the point R. In FIG. 9, an example of a straight line serving as the rotation axis is shown between the Y axis and the Z axis. The rotation axis is perpendicular to both the first eye position (X-axis direction) and the line-of-sight direction after rotation.
If the eyeball is rotated to a direction vector (L, M, N) (not shown), the X-axis, Y-axis, and Z-axis vectors of the rotated eyeball coordinate system are expressed by the following equation (1). Calculated.

The listing law is correct for determining the posture of the eyeball with respect to an object with one eye at infinity.
Further, for example, when an object at infinity is viewed and the body is tilted, the left eye and the right eye have the same eyeball posture, and the eyeball rotation is the same.
On the other hand, when an object that is not infinitely far away is viewed with both eyes, the posture of the eyeball may be different between the left eye and the right eye.
Here, FIGS. 10A and 10B are diagrams for explaining the gaze directions of the left and right eyes in the case of binocular vision.

In the case of binocular vision, if the object is infinitely far away, the left eyeball 51L and the right eyeball 51R are directed in the same viewing direction as shown in FIG. 10A. The posture is also the same.
At this time, there is no difference between the retinal images of the left and right eyes.
In FIGS. 10A and 10B, an eyeball 55 obtained by averaging the left eye and the right eye is indicated by a broken line at the center between the left eyeball 51L and the right eyeball 51R.

On the other hand, binocular convergence is required for an object (point A) of a finite distance, as shown in FIG. 10B.
Therefore, the viewing direction of the left-eye eyeball 51L and the viewing direction of the right-eye eyeball 51R are different, and the amount of rotation of the eyeball is different between the left and right eyeballs 51L and 51R. In FIG. 10B, since the point A is on the left front, the amount of rotation of the right-eye eyeball 51R is larger than the amount of rotation of the left-eye eyeball 51L.

In eyeball rotation based on the listing law, the eyeball posture after rotation, that is, each direction vector of the y-axis and z-axis after rotation depends on the viewing direction vector shown in Expression (1).
If the viewing direction vectors of the left eye and the right eye are different, the direction vectors of the y-axis and the z-axis after the rotation do not match with the left and right eyes, and a rotation shift of the retinal image occurs.
In order to eliminate this rotational deviation, rotation around the line of sight is required for the left and right eyes. This rotation is called fusion rotation.

In order to determine the angle of fusion (fusion rotation angle), it is necessary to determine the y direction (vertical direction) common to the left and right eyes.
The common y direction is naturally a direction that is perpendicular to both the left eye gaze and the right eye gaze. That is, the direction is perpendicular to a plane including a straight line between the center of the left eyeball 51L and the point A and a straight line between the center of the right eyeball 51R and the point A in FIG. 10B.
Accordingly, as shown in FIG. 11, the y-direction common to both eyes, the vector x _L in the left eye gaze direction and the vector x _{R in the} right eye gaze direction are respectively common to the y-direction vectors in the vertical direction. there is a y _C. This direction is also called a theoretical longitudinal horopter.

このとき、共通のｙ方向のベクトルｙ_Ｃは、下記の式（３）で表される。

If line-of-sight direction unit vector x _L and x _R of the right and left eyes, respectively, and is as the formula (2) below.

In this case, the vector y _C a common y-direction is expressed by the following formula (3).

ベクトルｙ_Ｌとベクトルｙ_Ｃとの角度が、左眼球の融像回旋角度となり、同様にベクトルｙ_Ｒとベクトルｙ_Ｃとの角度が、右眼球の融像回旋角度となる。 The vectors y _L and y _{R in} the y direction based on the listing rule when the left and right eyes are rotated in the respective line-of-sight directions are expressed by the following equation (4).

Angle between the vector y _L and the vector y _C becomes a fusion torsion angle of the left eye, as well as the angle between the vectors y _R and the vector y _C, the fusion torsion angle of the right eye.

Furthermore, when wearing spectacles, the refractive action of the spectacle lens must be considered.
In this case, the common y direction y _C itself does not change, but the y direction after passing through the left and right lenses is obtained by ray tracing, and the common y direction obtained by ray tracing and the right and left eye listing laws are used. When the angle with the vector in the y direction is obtained, the fusion angle is obtained.

And vector Y _C common y direction obtained by ray tracing, vector Y _L in the y-direction of the right and left _eyes, and a Y _R, by the following formula, the left and right eye image fusion line angle theta _L, a theta _R Can be calculated.

  Note that the above expression is an expression for obtaining a fusion angle when the vertical direction common to both eyes, that is, the vertical horopter is the outer product direction of the left and right eye gazes. The actual vertical horopter may differ from the cross product direction of the left and right eye gazes. The fusion angle in that case should be determined so that the longitudinal direction of the eyeball and the actual longitudinal horopter direction are in the same plane.

2-6. Synthesis of simulation image So far, distortion, blurring and fusion of the lens have been described.
When distortion and blur are combined by an image processing technique, it is possible to simulate an image viewed through a spectacle lens. Furthermore, not only still images but also moving images are possible.
Distortion can be realized by obtaining the corresponding point object side of all pixels in the image side field of view and applying the luminance information of the original image.
Blurring can be achieved by “distributing” the luminance of each pixel to the surrounding pixels based on its PSF and reconstructing the luminance of all pixels in the image.
The blur process is also called a convolution operation. The difference from the general convolution operation is that the PSF is not constant.
The fusion (torsion) can be realized by performing processing for adding the amount of eyeball rotation including fusion to the original image data in the field of view.

More preferably, with regard to fusion rotation (torsion), the amount of movement of the line of sight from the reference position at the distance eye point in the line of sight (horizontal and vertical movement of the eyeball) and the refractive power of the spectacle lens and / or The angle of fusion (torsion) of the eyeball is calculated according to the prism.
In FIG. 10B, the reference position at the distance eye point in the line-of-sight direction is the direction of the broken line extending from the left and right eyeballs 51L and 51R, and from the eyeball 55 that averages the left eye and the right eye, it is an angle from the front. Left by φ. In FIG. 10B, the sum of the movement angle amounts of the left and right eyes is an angle α.

More preferably, the angle of fusion rotation theta _{L (the} left eye) and theta _{R (right} eye), a vector orthogonal to the viewing direction of both eyes, which is calculated from the listing's Law, vector Y _{L (left} eye) and vector Y _R (right eye) and the vector Y _C which is the outer product of the unit vector in the line of sight of the left eye and the unit vector of the line of sight in the right eye are calculated by the above-described equation.

Note that fusion rotation (torsion) does not occur when the left and right eye gaze directions are parallel, so when the left and right eye gaze directions are parallel depending on the position of the subject, fusion rotation is performed. There is no processing to add the amount.
Fusion rotation (torsion) occurs in near vision except far vision looking at infinity, and occurs when the left and right eye gaze directions are different (not parallel but intersecting). is there.
In addition, with the movement of the eyeball inversion (ie, convergence) when the object being watched approaches, or the movement of the eyeball abduction when the object being watched away (ie, spread), A process of fusion (torsion) is added.

2-7. Coordinate transformation when the central line of sight of the field of view is specified As described above, in the coordinate system (hereinafter referred to as the global coordinate system) in which the direction from the front to the eye is the x axis, the distortion information (from the object side viewing direction to the image side) By using the conversion to the viewing direction and blur information (specific viewing direction, PSF at the viewing distance), the entire visual field can be simulated.

（ここで、ａ，ｂ，ｃは、中心視線方向のグローバル座標における方向単位ベクトル（ａ ｂ ｃ）の各軸成分である。）
そして、グローバル座標の任意一点（ｘ，ｙ，ｚ）のローカル座標（ｘ'，ｙ'，ｚ'）は、下記の式（６）で変換される。

逆に、ローカル座標の任意一点（ｘ'，ｙ'，ｚ'）のグローバル座標（ｘ，ｙ，ｚ）は、下記の式（７）で変換される。

However, in actual simulations, the central line of sight is not necessarily directly in front.
For example, when it is desired to confirm how the near portion is visible, the central line of sight needs to pass through the lens near portion. In this case, the central line of sight hits the diagonal direction of the global coordinates. A simulation is performed in a local coordinate system having the oblique direction as the x ′ axis.
At this time, how to determine the y ′ axis and the z ′ axis of the local coordinate system becomes a problem. Here, it is decided according to the listing law, which is one of the laws of eyeball rotation. According to the listing rule, the vertical direction and the left-right direction when the central line of sight is facing directly in front change to a predetermined direction in response to the eyeball rotating and the central line of sight moving. Even when the center line of sight moves and the line of sight changes, the vertical and horizontal directions of the actual object change so as to be the vertical and horizontal directions even in the retinal image.
A coordinate matrix conversion matrix is expressed by the following equation (5).

(Here, a, b, and c are each axis component of the direction unit vector (a b c) in the global coordinates in the central line-of-sight direction.)
Then, the local coordinates (x ′, y ′, z ′) of an arbitrary point (x, y, z) in the global coordinates are converted by the following expression (6).

Conversely, global coordinates (x, y, z) of an arbitrary point (x ′, y ′, z ′) of local coordinates are converted by the following equation (7).

  By using the coordinate conversion formula as described above, it is possible to realistically simulate distortion when the line-of-sight direction passing through an arbitrary point on the lens is the central line-of-sight.

2-8. Spline Interpolation Approximation of Ray Data The optical principle and image processing method for simulating the appearance through a spectacle lens were established as described above.
However, when you start the simulation, you suffer from a huge amount of calculations. The shape of the spectacle lens is not a simple spherical surface, and in particular, the progressive lens is a free-form surface.
For ray tracing of a complicated surface such as a spectacle lens, a method of repeatedly converging is adopted. This takes at least several times longer than simple spherical ray tracing.
In addition, the large number of pixels in the image to be simulated has spurred an increase in the amount of ray tracing calculation.
Ray tracing (principal ray tracing) for finding out which pixel of the original image corresponds to all pixels of the image as a result of all simulations must be executed. In addition, a number of (for example, 100) rays emanating from the corresponding object points are traced to determine the PSF, and a spot on the retina is obtained. In order for all of these ray tracing to correspond to aspherical surfaces, it is necessary to adopt a convergence method repeatedly, which is a tremendous calculation burden.
With the current computing ability of one personal computer, it takes several days to process one image (one frame of moving image) with such a method.

つまり、下記の関数
ψ'＝ψ'（Ｄ_ｌ，ψ，ζ）
ζ'＝ζ'（Ｄ_ｌ，ψ，ζ）
が成立する。
しかも、変数（Ｄ_ｌ，ψ，ζ）に対し、（ψ'，ζ'）は連続変化することも、容易に想像できる。このような関数は、スプライン補間に適している。 On the other hand, in a state in which the lens shape, eyeball parameters, and the positional relationship between the lens and the eyeball are all determined, the image side (ψ ′, ζ ′) is uniquely determined with respect to any one point on the object side described below.

That is, the following function ψ ′ = ψ ′ (D _l , ψ, ζ)
ζ ′ = ζ ′ (D _l , ψ, ζ)
Is established.
Moreover, it can be easily imagined that (ψ ′, ζ ′) continuously changes with respect to the variable (D _l , ψ, ζ). Such a function is suitable for spline interpolation.

Therefore, a finite number of sample points are set in the domain of each variable. For example, the sample points of the reciprocal object distance D ₁ are (−0.2, 0.0, 0.2, 0.5, 0.8, 1.1, 1.4, 1.7, 2.0, 2.3, 2.6, 2.9, 3.2, 3.6, 4.0), and samples of the tangent ψ of the vertical angle are (−1.5, −1.2, − 1.0, -0.8, -0.6, -0.4, -0.2, 0.0, 0.2, 0.4, 0.6, 0.8, 1.0, 1.. 2, 1.5), and samples of the tangent ζ of the left and right angles are (−1.5, −1.2, −1.0, −0.8, −0.6, −0.4, -0.2, 0.0, 0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.5).
For all combinations of these sample points, ray tracing is performed to find a true function value.
A method of interpolating function values for other variable values (values between sample points) using true function values at the sample points has been established. There are many interpolation methods according to the purpose, but the B-spline method is most suitable as an interpolation method when the true value at the sample point is known.

  The number and interval of sample points are related to the interpolation accuracy. In general, the interpolation accuracy is small where the sample point interval is small. However, if the interval is reduced, the number of samples for covering the entire definition area increases, and it is necessary to secure a large amount of memory in the program. Since recent PCs and OSs can be equipped with a lot of memory, the limit on the number of samples has been relaxed, and high-precision results can be obtained.

を、少ない計算量のスプライン補間で得られるようになる。式中、Ｃは補間係数であり、Ｎは各次元の節点に基づいた基底多項式関数である。 In this way, a function representing distortion information

Can be obtained with a small amount of spline interpolation. In the equation, C is an interpolation coefficient, and N is a basis polynomial function based on the nodes of each dimension.

2-9. Simplification of PSF As described above, in order to obtain the PSF of an object point exactly, the light beam emitted from the object point and passing through a large number of points that equally divide the entrance pupil is traced to obtain a spot on the retina. Further, a spot density distribution function is obtained.
However, with this method, even if the number of light beams is increased, the accuracy cannot be improved as expected.
In addition, when the aberration is small, the spot is concentrated, and there are cases where the light ray hardly passes other than the image point, and when the power error is large, there is a case of uniform distribution in a certain region. Change is intense.
On the other hand, in the case of simulation and lens performance evaluation, an accurate PSF is not necessarily required. For example, in the case of visual acuity, it represents the closest distance (viewing angle) at which two points can be distinguished. In this case, the precise shape of the PSF function is not required, and the size of the range covered by the PSF is an important parameter. Accordingly, even if the PSF is boldly simplified, it can be said that the role played in the lens performance evaluation is not greatly affected.
On the contrary, if the PSF is assumed to be a continuous function in advance and the parameters are applied using the ray tracing data, the PSF can be expressed with a small number of parameters. These parameters can be obtained by spline interpolation (three-dimensional) like the above-described distortion function.

（ここで、μ，νはそれぞれｙ、ｚ方向の主光線からの偏移角、σ_μ，σ_ν，ρは正規分布のパラメータである。これらのパラメータはσ_μ＞０，σ_ν＞０，−１＜ρ＜１を満たす。）
下記の式（９）で表される楕円の線上すべての点（μ，ν）において、次の式（１０）が成り立つ。

そして、その等高線楕円内の積分は、下記の式（１１）となる。

この場合の等確率楕円を、図１２に示す。 As the shape of the simplified function, a two-dimensional normal distribution is considered appropriate so that the PSF of astigmatism with a power error or any axial angle can be approximated. In other words, the simplified function is expressed by the following equation (8). .

(Where μ and ν are shift angles from the principal rays in the y and z directions, respectively, and σ _μ , σ _ν and ρ are parameters of a normal distribution. These parameters are σ _μ > 0 and σ _ν > 0. , −1 <ρ <1 is satisfied.)
The following equation (10) holds at all points (μ, ν) on the elliptical line represented by the following equation (9).

The integral within the contour ellipse is given by the following equation (11).

The equiprobability ellipse in this case is shown in FIG.

Thus, the two-dimensional normal distribution function can represent the extent of spread (σ _μ , σ _ν ), the degree of astigmatism blur (equal probability ellipse major / minor axis ratio), and the angle (major axis angle).
Of course, near-infinite change due to the state of the optical system of PFS cannot be faithfully expressed, but it is effective as a simplified function for expressing PSF.

である。ここで、Ｎは光線数で、(μ_ｉ，ν_ｉ)は交点座標である。 Considering the method of obtaining the parameters σ _μ , σ _ν , and ρ of the two-dimensional normal distribution function from the ray data, the intersections of many rays scattered on the (μ, ν) plane (each intersection point is each division point on the entrance pupil) Can be considered as a method of _calculating the statistical value of the corresponding to σ _μ , σ _ν , and ρ. That means

It is. Here, N is the number of rays, and (μ _i , ν _i ) is the intersection coordinates.

As described above, the PSF distribution function at an arbitrary point (D ₁ , ψ, ζ) on the object space can be approximated by a two-dimensional normal distribution function having parameters σ _μ , σ _ν , and ρ. Furthermore, σ _μ , σ _ν , and ρ can be expressed as a function of (D _l , ψ, ζ). That means
σ _μ = σ _μ (D _l , ψ, ζ)
σ _ν = σ _ν (D _l , ψ, ζ)
ρ = ρ (D _l , ψ, ζ)
These functions can also be obtained by spline interpolation in the same manner as the distortion information. That is, it can be obtained as follows.

It should be noted here that the function value may exceed the domain due to spline interpolation error. For example, although -1 <ρ <1, when it is obtained by interpolation, a result such as ρ = 1.002 may be obtained, and an ellipse may not exist. As a solution to this problem, it is effective to interpolate sin ⁻¹ ρ instead of ρ, and perform sine operation on the obtained result to obtain ρ.

In addition to distortion and blur parameters, useful parameters can be obtained by spline interpolation. For example, the lens convex surface passing point position ( _yconvex , _zconvex ) of the principal ray, the concave surface passing point position ( _yconvave , _zconveve ), and the like can be mentioned. These parameters can be calculated as follows.

  The lens passing point position of the principal ray is useful for simulating the distortion and blur distribution in the local coordinate system with the transmitted ray at the specific lens position as the central line of sight.

で表される。ここで、ｉは各次元の節点番号、Ｃ_ｉはその係数、ｎは標本点数である。Ｎ_ｉ（ｘ)は、ｉ番節点に対応する基底関数であり、階数Ｍの場合、ｉ番節点とｉ＋Ｍ番節点との間の範囲でゼロでない値を持ち、隣接節点間はｍ−１次多項式で表される（基底関数の局部性のため）。
言い換えると、ｘの定義域内の任意点ａにおいては、ゼロでない値のＮ_ｉ（ｘ）が最多でもＭ個しか存在しない。
従って、補間式は一見するとｎ項あるように見えるが、ｘ＝ａにおいては実質Ｍ項であり、Ｍ回の掛け算とＭ回の足し算でＦ（ａ）が得られる。 2-10. Speeding up multidimensional B-spline interpolation calculation One-dimensional spline interpolation is

It is represented by Here, i is the node number of each dimension, C _i is its coefficient, and n is the number of sample points. N _i (x) is a basis function corresponding to the i-th node, and in the case of rank M, N _i (x) has a non-zero value in the range between the i-th node and the i + M-th node, and the m−1 order between adjacent nodes Expressed in polynomial form (due to locality of basis functions).
In other words, at an arbitrary point a within the domain of x, there are only M non-zero values N _i (x) at most.
Therefore, the interpolation equation seems to have n terms at first glance, but when x = a, it is substantially M terms, and F (a) is obtained by M multiplications and M additions.

ここで、ｉ，ｊ，ｋは各次元の節点番号であり、それぞれ標本点数だけ変化する。
つまり、項の数は各次元の標本点数の積になる。しかし、上述の基底関数の局部性により、ある一点については、ゼロでない項の数は、各次元の階数の積である。
各次元のスプライン階数が４の場合、項の数は４^３＝６４である。つまり、一回の補間演算では、足し算を６４回、掛け算を６４×３＝１９２回行うことになる。
一般的には、ｎｊ次元のＭ階スプライン補間演算に必要な掛け算の回数は、ｎｊ×Ｍ^ｎｊであり、次元数が大きくなるにつれて、急激に計算負担が増える。 Three-dimensional spline interpolation is expressed by the following equation (12).

Here, i, j, and k are node numbers of each dimension, and change by the number of sample points.
That is, the number of terms is the product of the number of sample points in each dimension. However, due to the locality of the basis functions described above, for a certain point, the number of non-zero terms is the product of the rank of each dimension.
If the spline rank in each dimension is 4, the number of terms is 4 ³ = 64. That is, in one interpolation operation, addition is performed 64 times and multiplication is performed 64 × 3 = 192 times.
In general, the number of multiplications required for the nj-dimensional M-order spline interpolation calculation is nj × ^Mnj , and the calculation load increases rapidly as the number of dimensions increases.

この式（９）は、１次元の補間のネスト構造（入れ子構造）であり、次元の順番は自由に変えることができる。掛け算と足し算は、共に４＋４×(４＋４×４)＝８４回であり、ほぼ１／２の計算時間で済む。
一般的には、ｎｊ次元のＭ階スプライン補間演算に必要な掛け算の回数は、下記式（１４）で表される。

However, if the above equation (12) is rewritten as the following equation (13), the number of calculations can be slightly reduced.

This equation (9) is a one-dimensional interpolation nested structure (nested structure), and the order of dimensions can be freely changed. The multiplication and the addition are both 4 + 4 × (4 + 4 × 4) = 84 times, and the calculation time is almost ½.
In general, the number of multiplications required for the nj-dimensional M-th order spline interpolation calculation is expressed by the following equation (14).

2-11. Summary The technical aspects of moving image simulation using 3D CG have been described. The amount of calculation is enormous because a three-dimensional external world viewed with a progressive lens is simulated with a moving image.
This enormous amount of calculation can be reduced by spline interpolation and PSF parameterization, and can be achieved to a realizable level.
When the present invention is applied, the time required for the calculation can be shortened to about 1/100 to 1/1000 compared to the case where the simulation calculation is performed without applying the present invention.

Actually, a spline coefficient database of left and right binocular distortion and blur parameters was prepared in advance, and a simulation image was generated using a high-performance personal computer and an image graphic board.
When a real-time walkthrough was executed in the CG with an HMD equipped with a gyro, it was realized at a speed of 10 frames / second.

3. Description of Other Embodiments of the Present Invention The present invention is not limited to a configuration that displays images on both the HMD 11 and the monitors 16 and 17 as in the above-described embodiments. For example, only one of the display units may be provided and a retinal image may be displayed on the display unit, for example, the monitor may be omitted and displayed only on the HMD.
In the present invention, the display unit for displaying a retinal image is not limited to an HMD or a monitor. The head movement sensor is not limited to the gyro sensor.

Here, as another embodiment of the present invention, a configuration of a field-of-view image display device (display system) for glasses will be described.
In this embodiment, instead of the HMD 11 and the monitors 16 and 17 shown in FIGS. 1 and 3, a 3D (stereoscopic) display such as a 3D television or a 3D projector is used as a display unit. Then, after a moving image is created in advance, a simulation image is created frame by frame, a moving image when a spectacle lens is applied is created from the simulation image, and left and right eye images are displayed on the 3D display.

The process until the simulation image is displayed in the apparatus of the present embodiment is shown in the flowchart of FIG.
First, in step S21, a CG virtual object (three-dimensional CG model) is prepared as shown in FIG.
In step S22, a moving image story created in advance is prepared.
Next, in step S23, a frame of a moving image is created from the viewpoint and viewing direction of the left and right eyes from the CG virtual object (three-dimensional CG model) and the moving image story.
Next, in step S24, the luminance (RGB) and the distance from the viewpoint of each pixel of the original image having no distortion or blur for the right eye are obtained from one frame of the moving image created in step S23. Similarly, in step S25, the luminance (RGB) and the distance from the viewpoint of each pixel of the original image having no distortion or blur for the left eye are obtained from one frame of the moving image created in step S23.
On the other hand, in step S26, the prescription power, the addition power, and the lens type of the spectacle user are input by an input device (keyboard 19 or the like).
Next, from the input content, shape data and layout data of the right eye lens and a right eyeball model are created in step S27, and shape data, layout data and left eyeball model of the left eye lens are created in step S28.
Next, based on the shape data, layout data, and eyeball model created in steps S27 to S28, a three-dimensional spline interpolation coefficient for the light ray data is generated in step S29.
Next, using the three-dimensional spline interpolation coefficient of the ray data generated in step S29, in step S30, the three-dimensional spline interpolation of the right ray for the outgoing ray direction, the PSF parameter, the lens passing point position, and other various parameters. Find the coefficient. Similarly, in step S31, three-dimensional spline interpolation coefficients for the left eye, the direction of outgoing light, the PSF parameter, the lens passing point position, and other various parameters are obtained.
Next, a simulation is executed in step S32 using the brightness and distance from the viewpoint of each pixel of the original image created in step S24 and step S25, and the parameters and interpolation coefficients obtained in steps S30 and S31. To do. This simulation process includes using image processing hardware.
Next, in step S33, an image including distortion and blur for the right eye is created. Similarly, in step S34, an image including distortion and blur for the left eye is created.
The image including distortion and blur created as described above is displayed in a 3D display on a 3D display.
Through the process described above, a stereoscopic image including distortion and blur corresponding to the viewing direction is displayed on the 3D display screen.

  In the present embodiment, other configurations other than those described above are the same as those in the previous embodiment, and thus redundant description is omitted.

4). In the above-described embodiment, in the process of adding distortion, the luminance information of the original image is applied to the corresponding point object side of all the pixels in the image side field of view.
In the above-described embodiment, in the process of adding blur, the luminance of each pixel is “distributed” to the peripheral pixels based on the PSF to reconstruct the luminance of all the pixels in the image. Furthermore, the normal distribution function and parameters represented by the equation (1) are used.
In the present invention, the processing method for adding distortion and blur is not limited to the method described in the above embodiment, and other methods can be used.
The present invention also includes a configuration in which a method other than the method described in the above-described embodiment is used for one or both of the process of adding distortion and the process of adding blur.

Further, in the above-described embodiment, the right-eye image and the left-eye image are respectively created to enable stereoscopic viewing.
The present invention also includes a configuration in which only one of the right-eye image and the left-eye image is created.

In the above-described embodiment, the present invention is applied to the case where a progressive-power lens simulation is performed, and the processing for adding the amount of eyeball rotation including fusion is performed to create a retinal image.
In the present invention, not only the progressive power lens but also other spectacle lenses can be used to create a retinal image by performing processing for adding the amount of eyeball rotation including fusion.

  The present invention is not limited to the above-described embodiment, and various other configurations can be taken without departing from the gist of the present invention.

11 HMD, 15 PC, 16, 17 Monitor, 41 Eyeglass wearer, 42 Head, 43A, 43B Main line of sight, 44A, 44B Visual field of view, 46A, 46B Line of sight, 50 Eyeglass lens

Claims (3)

A visual field image display device for spectacles, which displays a retinal image that is visible when a spectacle lens is worn by simulation,
The original image data in the field of view corresponding to the direction of the line of sight of the eyeball in the state where the spectacle lens is applied is at least a rotational movement around the visual axis of the eyeball corresponding to the passing point of the line of sight in the spectacle lens. An image processing unit that creates the retinal image by performing processing to add an eyeball rotation amount including a certain fusion rotation;
And a display unit for displaying the retinal image created by the image processing unit.   The image processing unit performs eye torsion (in accordance with the amount of eye movement angle (horizontal and vertical movement of the eyeball) from the reference position at the distance eye point in the eye direction and the refractive power of the spectacle lens and / or the prism. 2. The visual field image display device for spectacles according to claim 1, wherein an angle of fusion is calculated. The image processing unit is a vector Y _L (left eye) that is a vector orthogonal to the line-of-sight direction of both eyes calculated from the listing law, with the angles θ _L (left eye) and θ _R (right eye) of the fusion rotation. ) And a vector Y _R (right eye) and a vector Y _C that is the outer product of the unit vector in the line-of-sight direction of the left eye and the unit vector in the line-of-sight direction of the right eye, Item 2. A field-of-view image display device for glasses according to Item 1.

Images