JP2009003812A - Generation method and generation device of defocus image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a means to generate a defocus image efficiently and display it on a real time basis in three-dimensional CG. <P>SOLUTION: The method is for generating a two-dimensional image by performing perspective transformation from a viewpoint of an observer to an object arranged in view volume. This method comprises the following steps: (a) a step for dividing the space in the above-described view volume into a plurality of voxels more finely divided in proportionate to adjacency degree to the viewpoint of the observer; (b) a step for calculating the amount of the blur on retina of the above-described observer by tracing an optical path from the center of the above-described voxel to the above-described viewpoint; (c) a step for calculating and recording range of shifting the vertex in the voxel corresponding to the amount of the blur on the above-described retina; (d) a step for actually shifting the vertex responding to the above-described vertex shifting range; and (e)the step for generating the above-described two-dimensional image seen from the above-described viewpoint by the perspective transformation. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a method and apparatus for generating a defocused image in the field of three-dimensional computer graphics (3DCG). That is, the present invention relates to a method and apparatus capable of generating a defocused image when generating a two-dimensional image by performing perspective transformation on an object placed in a three-dimensional space.

  Reflection and refraction display is one of the important themes in computer graphics research. Conventionally, these processes have been realized by ray tracing, but the processing time is enormous. Therefore, in recent years, a method of approximately displaying reflection / refraction in real time using environment mapping, which is a function of graphics hardware, is generally used.

  Ray tracing is originally a method for optical system design and is still used to simulate accurate refraction. Approximation methods are not currently used for designing optical systems that require accuracy.

  However, for the purpose of simulating how a human can see through a spectacle lens, it can be expected that there is no practical problem with the approximation method. Real-time display not only increases the efficiency of design verification, but also enables applications such as presentation to customers at dealers.

  The related research on the technology to display in consideration of real-time reflection and refraction is described below. In addition, related research on human vision correction simulation will be introduced.

(Real time reflection / refraction display)
The environment mapping proposed by Blinn to approximate reflection (Non-Patent Document 1) can be implemented by hardware (Non-Patent Documents 2 and 3) and is therefore widely used for real-time display applications. Since environment mapping involves the premise that the object to be reflected is at infinity, there is a drawback that the error is particularly large for a nearby (local) object.

  In recent years, methods for improving this have been devised. Hakura et al. Solved this problem by using a separate layer for local objects and preparing individual environment maps by preprocessing by ray tracing (Non-Patent Document 4). However, in this method, when the reflecting / refracting object moves and the local object frequently changes, the cost of the preprocessing is high. Szirmay-Kalos et al. Found a cube map that also has a depth value z for each pixel, and approximated the intersection of the ray and the object in the reflection / refraction direction by iterative calculation, so that more accurate reflection / refraction of the local object was achieved. Realized in real time (Non-Patent Document 5). Since the purpose of this method is a game, refraction was aimed at creating an atmosphere of a transparent object placed in the scene.

  In order to verify the state of the entire scene viewed through the lens, a more accurate method is required. In view of this, the present inventors have proposed a method of performing the intersection calculation of the ray and the scene through each vertex of the lens and realizing the same accuracy as ray tracing (see Japanese Patent Application No. 2006-124656).

(Simulation of vision correction)
As a drawing technique that takes into account the characteristics of the human eye, simulation of depth of field has been performed for some time (Non-Patent Documents 6 and 7).

  A simulation model that considers the eyeball as a lens system and considers correction of vision has been proposed by Mostafawy et al. (Non-patent Document 8). Loos et al. Used wavefront tracing, which is a wave-optical approach, to simulate visual acuity correction using a progressive focus lens (a bifocal lens with no boundary) (Non-Patent Document 9). Barsky introduced the concept of Vision Realistic Rendering and introduced depth of field (Non-Patent Document 11).

  Since all these conventional methods perform processing using distributed ray tracing, the display speed is slow and real-time processing is impossible.

Further, the present inventors have proposed a method of storing a vertex shift amount for each voxel and generating a blurred image based on the vertex shift amount (Non-patent Document 12). According to this method, it is possible to generate a blurred image almost in real time. However, with this approach,
・ Voxel is assumed in the space outside the view volume, so the calculation efficiency is bad. ・ Voxel is generated by dividing the space at equal intervals, so there is a slight difference from the actual appearance. There was room for improvement.
1) JF Blinn and ME Newell.Texture and reflection in computer generated images.Communications of the ACM, Vol. 19, No. 10, pp. 542-547, 1976. 2) P. Haeberli and M. Segal. Texture mapping as A fundamental drawing primitive. In Fourth Eurographics Workshop on Rendering, pp. 259-266, 1993. 3) D. Voorhies and J. Foran. Reflection vector shading hardware. In Proc. SIGGRAPH '94, pp. 163-166, 1994. 4) ZS Hakura, JM Snyder, and JE Lengyel.Parameterized environment maps.In SI3D '01: Proceedings of the 2001 symposium on Interactive 3D graphics, pp. 203-208, 2001. 5) L. Szirmay-Kalos, B. Aszodi, I. Lazanyi, and M. Premecz. Approximate Ray-Tracing on the GPU with Distance Impostors.Computer Graphics Forum (Proc.Eurographics 2005), Vol.24, No.3, pp. 685-704, 2005. 6) RL Cook, T. Porter, and L. Carpenter. Distributed Ray Tracing. In Proc. SIGGRAPH'84, pp. 137-146, 1984. 7) P. Haeberli and K. Akeley. The accumulation buffer: hardware support for high-quality rendering. In Proc. SIGGRAPH '90, pp. 309-318, 1990. 8) S. Mostafawy, O. Kermani, and H. Lubatschowski. Virtual Eye: Retinal Image Visualization of the Human Eye. IEEE CG & A, Vol. 17, No. 1, pp. 8-12, January-February 1997. 9) J.Loos, Ph. Slusallek, and H.-P. Seidel.Using wavefront tracing for the visualization and optimization of progressive lenses.Computer Graphics Forum (Proc.Eurographics 1998), Vol.17, No.3, pp. 255-263, 1998. 10) J. Amanatides and A. Woo. A fast voxel traversal algorithm for ray tracing. In Proc. Eurographics 1987, pp. 3-10, 1987. BARSKY BA: Vision-realistic rendering: simulation of the scanned foveal image from wavefront data of human subjects.In Proc.APGV '04 (2004), pp. 73-81. Information Processing Society of Japan Vol.2006, No.91, pp.25-30 "Display of refractive correction results using vertex-based ray tracing"

  The present invention has been made in view of the above situation. An object of the present invention is to provide means for efficiently generating a defocused image and enabling display in real time.

  In order to solve the above-described problems, the present invention has a configuration described in any of the following items.

(Item 1)
A method of generating a two-dimensional image by performing perspective transformation from an observer's viewpoint on an object placed in a view volume, and further comprising the following steps:
(A) dividing the space in the view volume into a plurality of voxels that are made finer as the viewpoint from the observer is closer;
(B) calculating a blur amount on the retina of the observer by following an optical path from the center of the voxel to the viewpoint;
(C) calculating and recording a vertex shift range in the voxel corresponding to the amount of blur on the retina;
(D) actually shifting the vertex according to the vertex shift range;
(E) generating the two-dimensional image viewed from the viewpoint by perspective transformation;

  According to the invention according to item 1, the vertex shift range in the voxel corresponding to the amount of blur on the retina is calculated. Here, the vertex is an element constituting the polygon. Using the obtained vertex shift range, the vertex is actually shifted from the initial position. Thereby, the polygon which has the said vertex in the position shifted | deviated from the original position is produced | generated. In this state, perspective transformation is performed. By repeating this operation, in the image obtained by the perspective transformation, the objects whose positions are shifted are displayed in an overlapping manner, so that the objects appear to be blurred. That is, a defocused image can be generated thereby.

  Further, according to the invention according to item 1, the closer to the observer's viewpoint, the more the voxels are divided, so the blur amount (defocus amount) for each voxel can be finely changed. For this reason, the actual appearance can be simulated more accurately. In addition, such voxel setting is a process having high affinity with the perspective transformation essential for image generation using the three-dimensional modeling space, and thus it is possible to shorten the processing time required for image generation.

(Item 2)
further,
(F) determining an optical path from the viewpoint to the center of the voxel, this step being performed before the step (b);
The image generation method according to item 1, wherein the step (b) is executed by tracing back the optical path obtained in the step (f).

(Item 3)
Item 3. The image generating method according to Item 1 or 2, wherein a wavefront tracking method is used as a method of tracing the optical path from the center of the voxel to the viewpoint in the step (b).

(Item 4)
An apparatus for generating a two-dimensional image by performing perspective transformation on an object arranged in a three-dimensional space,
The apparatus includes a storage unit and an image generation unit,
The storage unit stores data and computer software necessary for processing,
The image generation unit performs the following processing:
(A) dividing the space in the view volume into a plurality of voxels that are made finer as the viewpoint from the observer is closer;
(B) calculating a blur amount on the retina of the observer by following an optical path from the center of the voxel to the viewpoint;
(C) calculating and recording a vertex shift range in the voxel corresponding to the amount of blur on the retina;
(D) a step of actually shifting the vertex according to the vertex shift range;
(E) generating the two-dimensional image viewed from the viewpoint by perspective transformation;

(Item 5)
A method of generating a two-dimensional image by performing perspective transformation on an object arranged in a three-dimensional space, and further comprising causing a computer to execute an image generation method including the following steps: Computer program:
(A) dividing the space in the view volume into a plurality of voxels that are made finer as the viewpoint from the observer is closer;
(B) calculating a blur amount on the retina of the observer by following an optical path from the center of the voxel to the viewpoint;
(C) calculating and recording a vertex shift range in the voxel corresponding to the amount of blur on the retina;
(D) a step of actually shifting the vertex according to the vertex shift range;
(E) generating the two-dimensional image viewed from the viewpoint by perspective transformation;

  According to the present invention, it is possible to provide means for efficiently generating a defocused image and enabling real-time display.

  Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

(Image generation device)
First, an image generation apparatus used for implementing this embodiment will be described with reference to FIG. The image generation apparatus includes an input unit 10, an input interface 11, a display unit 20, an output interface 21, a storage unit 30, an image generation unit 40, and a bus 50. These components can be configured by hardware itself such as a workstation or a computer, or by incorporating a computer program in the hardware.

  The input unit 10 is connected to the bus 50 via the input interface 11. The input unit 10 is an appropriate input device such as a keyboard, a mouse, or a touch pad, for example, and can be used to input a command from an operator.

  The display unit 20 is connected to the bus 50 via the output interface 21. The output unit 20 is an appropriate output device such as a display or a printer. The output unit 20 is configured to receive the image generated by the image generation unit 40 (two-dimensional image obtained by perspective transformation) and present it to the user.

  The storage unit 30 is a device that can store and read data (including computer programs). As the storage unit 30, an appropriate device such as a magnetic storage device (for example, a hard disk), a semiconductor storage device (for example, a memory board), or an optical storage device (for example, a storage device using an optical disk such as a DVD) can be used. . The storage unit 30 may be built in a workstation or computer, or may be connected to the outside via an interface device. Furthermore, the storage unit 30 may be a storage device that can be used via a network. The storage unit 30 may be physically configured by a plurality of storage devices.

  The storage unit 30 according to the present embodiment stores in advance a computer program for configuring the image generation unit 40 using a workstation or a computer, and data used for performing the method of the present embodiment. In addition, the storage unit 30 can store data generated in accordance with the execution of the method of the present embodiment.

The image generation unit 40 is configured to perform the following processing.
(A) dividing the space in the view volume into a plurality of voxels that are made finer as the viewpoint from the observer is closer;
(B) calculating a blur amount on the retina of the observer by following an optical path from the center of the voxel to the viewpoint;
(C) calculating and recording a vertex shift range in the voxel corresponding to the amount of blur on the retina;
(D) a step of actually shifting the vertex according to the vertex shift range;
(E) generating the two-dimensional image viewed from the viewpoint by perspective transformation;

  Details of these processes will be described in the description (described later) of the image generation method according to the present embodiment. The image generation unit 40 is configured as a functional element by executing a computer program on a workstation or a computer.

  The bus 50 is a transmission path that enables data exchange between the above-described units. The bus 50 is not limited to a configuration in which one bus is shared, and a configuration in which a dedicated bus is separately used. Further, the transmission path may be wireless instead of wired. In short, the transmission path only needs to be able to exchange data, and the physical configuration and the protocol to be used are not limited.

(Image generation method according to this embodiment)
Next, an image generation method according to the present embodiment will be described. In the following, vertex-based ray tracing used for refraction processing for accurately displaying distortion caused by a lens will be described first. In the description of the present embodiment, all target objects are polygon models.

(algorithm)
The processing procedure performed for each frame is as follows. A description will be given in correspondence with FIG. This algorithm is the same as that described in Non-Patent Document 12 by the inventors.
(1) Obtain all reflection / refraction vectors at the vertex 1a of the reflection / refraction object surface 1;
(2) Obtain virtual viewpoint 2 and field of view and draw environment map;
(3) For each vertex of a reflective / refractive object:
(a) Find the point (intersection) 4 where the extension of the reflection / refraction vector intersects the object 3 in the scene;
(b) Find a straight line 5 connecting the intersection 4 and the virtual viewpoint 2;
(c) Find the intersection of the straight line 5 and the environment map projection plane 6;
(d) The coordinates of the intersection 4 on the environment map image are set as the texture coordinates of the vertex 1a;
(4) Draw the reflection / refraction object and the entire scene from the original viewpoint.

  The virtual viewpoint 2 in (2) above is obtained as a point closest to the extension line by selecting several reflection / refraction vectors. This algorithm basically guarantees that the exact position of each vertex of the reflective / refractive object is reflected.

  As an exception, there may be a discrepancy between occluded objects due to a shift between the vertex and the virtual viewpoint. This is almost limited to cases where the reflective / refractive object is large and the ray spread is extremely large, and this is not a problem in lens simulation.

  Although an example of reflection is shown in FIG. 2, in the case of a refracting object such as a lens, a similar procedure is performed by obtaining a ray after two refractions of incident and outgoing.

(Reflection and refraction display results)
Display examples of vertex-based ray tracing reflection are shown in FIGS. As a comparison, FIGS. 3C and 3D show examples of reflection display by conventional cube mapping. It can be seen that the reflection of an object close to the viewpoint, which is displaced in cube mapping, is accurately processed in vertex-based ray tracing. In these figures, an image reflected on the door mirror of the car is displayed. In the images of FIG. 3B and FIG. 3D, mirror wireframe models are superimposed. Many spheres were placed at the intersection of the reflection vector and the scene. In vertex-based ray tracing, the intersection is accurately reflected in the vertex. Further, FIG. 4 shows a display example of refraction through a lens. This figure shows the display result of refraction through a concave lens. FIG. 4B is an enlarged view of the vicinity of the right end of FIG. 4A, in which the lens vertex is displayed in a wire frame, and a small sphere is arranged at the intersection of the refraction vector and the scene. Also in this figure, it can be seen that the intersection of the refraction vectors is accurately reflected at the apex.

(Method of this embodiment)
Hereinafter, a method for generating a blurred image according to the present embodiment will be described with reference mainly to FIG. This method is roughly divided into preprocessing and rendering. In the preprocessing, for each voxel, a vertex shift range is calculated, and this is associated with the voxel and recorded in the storage unit 30. At the time of rendering, drawing processing is performed by displacing each vertex of each polygon according to the vertex shift range. This will be described in detail below.

(Step SA-1 in FIG. 5)
First, voxel separation will be described with reference to FIG. In this example, the view volume 2 from the viewpoint 1 is set. The view volume is an area where perspective transformation is performed in the modeled 3D space. Here, in the present embodiment, the lens 3 is disposed between the viewpoint 1 and the view volume 2.

  In the present embodiment, voxels are generated by dividing the space in the view volume 2. That is, a set of voxels constitutes a view volume in the present embodiment. Here, in this embodiment, the space in the view volume 2 is divided more finely as it is closer to the viewpoint 1 of the observer (see FIG. 6A). That is, in any of the depth direction, the up-down direction, and the left-right direction (that is, the XYZ direction), the voxel is divided so as to become finer (smaller) as the viewpoint is closer. Such division can be realized by dividing the normalized device coordinate system (NDC) at equal intervals. For this reason, there is an advantage that a necessary amount of calculation can be reduced (described later).

(Step SA-2 in FIG. 5)
Next, an optical path from the viewpoint 1 to the center of the voxel 4 is obtained. The detailed procedure of this method will be described with reference to FIG.

(Step SB-1 in FIG. 7)
First, as shown in FIG. 8, the model is arranged at the center of the voxel 4. Specifically, a group of voxels having a certain depth is first set as a target. A rectangular model 5 is arranged at each center (center in the depth direction) of the voxel group. These rectangular models have different colors depending on the voxels.

(Step SB-2 in FIG. 7)
Then, in this state, each rectangular model 5 is drawn (perspective transformation) in consideration of refraction at the lens 3. An example of the drawing result is shown in FIG. In this drawing result, since different colors are assigned to the respective voxels, the voxels can be specified on the basis of the colors.

(Step SB-3 in FIG. 7)
Thereafter, the center position (center in the XY plane) of each rectangular model on the projection plane is obtained from the drawing result image (see FIG. 9).

(Step SB-4 in FIG. 7)
Further, a ray (Ray) from the viewpoint toward the position at the center of the rectangle on the projection surface is obtained. In other words, this ray goes from the viewpoint to the center of the voxel.

  In the present embodiment, the amount of blur on the retina is calculated by tracing back the optical path thus obtained (described later).

(Step SA-3 in FIG. 5)
Next, the amount of blur on the retina is calculated. The outline of this calculation is that the optical path from the viewpoint to the center of the voxel obtained as described above is followed from the voxel to the retina through the viewpoint (pupil). In the present embodiment, the wavefront tracking method is used as a method of tracing the optical path from the voxel center to the retina via the viewpoint.

(Step SA-4 in FIG. 5)
Next, the vertex shift range in the voxel corresponding to the amount of blur on the retina is calculated and recorded.

(Steps SA-5 and SA-6 in FIG. 5)
Next, a two-dimensional image viewed from the viewpoint is generated by perspective transformation. The generation of the two-dimensional image can be basically performed by the vertex-based ray tracing method described in Non-Patent Document 12.

  According to the method of the present embodiment, the perspective transformation is performed a plurality of times while shifting each vertex according to the method described in “Implementation of rendering by GPU” described later according to the obtained vertex deviation range. For this reason, in an image obtained by perspective transformation, an object composed of a plurality of polygons whose positions are shifted within the vertex shift range appears blurred. Thereby, a defocused image can be generated.

  Further, according to this method, the closer to the observer's viewpoint, the finer the voxels are divided, so that the blur amount (defocus amount) for each voxel can be finely changed. For this reason, the actual appearance can be simulated more accurately. In addition, such voxel setting is a process having high affinity with the perspective transformation essential for image generation using the three-dimensional modeling space, and thus it is possible to shorten the processing time required for image generation.

(Specific example of this embodiment)
Hereinafter, a specific example of generating a blurred image using the method of the present embodiment will be described. First, details of the wavefront tracking method employed in step SA-3 of the present embodiment will be described, and then a blur image generation method using the same will be described.

(Wavefront tracking method)
The wavefront tracking method itself has conventionally existed as a method of ray tracing. The wavefront at a point is represented by the direction of wave motion. For example, the normal vector N, the two main directions e ₁ and e ₂ and the corresponding main curvatures κ ₁ and κ ₂ indicate the direction of wave motion. The two main directions e ₁ and e ₂ are orthogonal to each other. Curvature is negative when the center of curvature is on the opposite side of the wave motion.

As it diffuses or travels at distance d, the wavefront simply deforms as follows.
Here, the vectors N, e ₁ and e ₂ do not change.

This transformation is guided by Snell's law, which is well known, so only the results are shown here. The direction vector N is
N ′ = μN + γN ^(s) Here, N ^(s) is a unit normal vector of the refractive surface, and μ = n ₁ / n ₂ and γ = −μcosφ + cosφ ′. n ₁ and n ₂ are the refractive indices of the medium before and after the surface, respectively. φ and φ ′ denote an incident angle and a refraction angle, respectively.

Unit vector
think of. This is a common tangent vector for both the incident wavefront and the surface. The curvature corresponding to the vector u is obtained as follows.

Consider another unit vector v = N × u. Further, consider the angle ω between the vector u and the main direction vector e ₁ . That is,
At this time, according to Euler's formula,
Which determines the u, v directional curvature κ _u , κ _v and torsion κ _uv for the incoming wave. Due to refraction, the curvature of the wavefront in the u and v directions changes as follows:

Here, κ _u ^(s) , κ _v ^(s), and κ _uv are the curvature and torsion in the u and v directions of the refractive surface at the incident point, respectively. These can be calculated in the same way as the wavefront by using Equation 3. In order to represent the wavefront after refraction, it is necessary to find the main curvature and the main direction by modifying Equation 3.

  Table 1 shows the usage of the wavefront tracking operation in the proposed method. That is, this table represents which operation is used at which stage in the calculation of the blur field for each voxel.

  Table 1 shows the pre-calculation process for each voxel. These operations are performed for double refracted rays from the source of each voxel to the viewpoint.

(Eye model)
In order to simulate the appearance due to refractive error including astigmatism, a technique using a wavefront tracking method and distributed ray tracing is conventionally known (see Non-Patent Document 9). In this embodiment, a new wavefront tracking method is used in the intraocular region. In this model, the wavefront in the eye lens is deformed to form a revolved conoid shape that resembles a cone, and this shape is clipped at the retina location to produce the exact amount of blur, ie the point spread shape. To win. This blur amount is stored in the storage unit 30 as a blur field value.

  In this embodiment, the eye as the lens mechanism is regarded as a thin toric lens. This lens was dependent on the direction and had a uniform refractive power over the lens surface. From the viewpoint of ophthalmology, this eyeball model can handle direct astigmatism and oblique astigmatism in addition to myopia, hyperopia and presbyopia.

(Wave deformation by thin toric lens)
In general, wavefront deformations in the eye are based on complex laws. Therefore, in this embodiment, in addition to approximating the eye to a thin toric lens, it is further assumed that the direction of the incident light beam is orthogonal to the lens. This assumption is reasonable for the following reasons. That is, the eyeball rotates toward the area of interest and the light rays converge at the fovea of the retina.

  In the following, a state in which the wavefront is refracted and converges on the retina by introducing a simple deformation of the incident wavefront will be described.

The state after being refracted by the spectacle lens and translated toward the surface of the cornea is described as a wavefront having main curvatures κ ₁ and κ ₂ and main directions e ₁ and e ₂ , respectively. As described in FIG. 10, here, the coordinate system of the eye lens is the right-handed coordinate system, and the origin is the incident portion. In addition, the z-axis was matched with the wavefront normal vector. The x axis was in contact with both the wavefront and the lens and was horizontal. For simplicity, the positive direction of the x axis is determined so that the y axis faces upward.

  Since every tangent vector in the incident wavefront is also a tangent vector for the lens, the x-axis can be chosen as a common tangent, and is denoted here by the tangent vector u. Thereby, it is possible to describe a wavefront subjected to deformation by a toric lens.

Let ω _{1 be} the angle between the x-axis and the first main direction e ₁ . Then, by assigning ω = −ω ₁ to Equation 3, the directional curvatures κ _x , κ _y and κ _ky can be obtained.

  Due to the deformation of the wavefront on the surface of the eye lens, here we introduce the vergence formula instead of Snell's law. Here, the direction-dependent refractive power in the toric lens is introduced as a set of input parameters in astigmatism.

First, given the basic refractive powers P ₁ and P ₂ and the angle between the x-axis and the first main direction ω ₁ ^(e) , the direction-dependent refractive power P _x (xz plane). ), P _y (in the yz plane), and P _xy are obtained by applying Equation 3. Here, κ ₁ = P ₁ , κ ₂ = P ₂ , and ω = −ω ₁ ^(e) are substituted.

In optics, the basic valence law is
It is represented by Here, V is the fluence of the input light, and can be expressed as V = nκ by definition. Here, the refractive index n of the medium on the input side and the curvature κ of the input point are used. Further, P is the refractive power of the lens, and V ′ is the fluence of the output light. In this embodiment, this equation is extended to directional curvature and induced laws, leading to wavefront deformation due to astigmatism. here,
It is. Here, μ = n ₁ / n ₂ , n ₁ is the refractive index of air, and n ₂ is the refractive index of the vitreous humor.

In this way, the wavefront finally converges on the retina according to equation 5. Here, κ _u = κ ′ _x , κ _v = κ ′ _y , and κ _uv = κ ′ _xy , the main curvatures κ ′ ₁ and κ ′ ₂ , and the first main direction ω ′ ₁ = −ω. Get the angle.

(Evaluation of the spread of light in the retina)
In general, light as described above does not converge to one point when refracted by a toric lens. Light of a certain size passing through a circular pupil has a shape resembling a cone. This shape is known as Sturm's conoid. Here, two focal lines are orthogonal to each other (see FIG. 11). Therefore, in the following, (1) the conoid of the starm is analyzed, and (2) the adjustment by the human eye is examined. Thus, the shape of the spread of light projected on the retina when the pupil and light from a certain point light source that forms a conoid are given. Next, we will consider eye adjustment.

In order to briefly examine the Starm conoid, for the time being, the rotation in the z direction is ignored. That is, ω ′ ₁ = 0. In order to simulate the spread of light projected on one point, attention is paid to the cross-section of the conoid in a plane perpendicular to the z-axis. Let z be the constant parameter z _e between the cornea and the retina. As a result, the following elliptic formula is obtained.
R _p is the radius of the pupil. The above a and b are obtained by using the similarity of a right triangle along the z axis (for example, ΔAOC and ΔBPC in FIG. 11). The parameters κ ′ ₁ and κ ′ ₂ are determined by how much adjustment by the human eye is incorporated (described later).

In the retina, z _e = z _r , and the elliptical shape described by a, b and ω ′ ₁ represents the point spread of the light source. The light wave from the wave source at the voxel is tracked and an ellipse at the retina can be obtained. This shape is a blur pattern (ie, “amount of blur on the retina”). This pattern is generated by each light source from each voxel. This blur pattern is recorded in association with each voxel. However, a, instead of recording the b and omega _'1, record this equivalent 2 × 2 matrix as voxel value. This matrix converts unit circles into oriented ellipses. The matrix for each voxel, precalculated in this way, becomes the blur field element.

  To determine the scaling factor of this matrix, it is necessary to consider the geometric relationship between the retina and the normalized device coordinate system (NDC). The scale in NDC is equivalent to the voxel space in this embodiment.

Let s _h and s _{v be} the scaling factor from the retina to the NDC in the horizontal and vertical directions, respectively. Further, final blur field values B _{vx, vy, vz} (that is, “vertex deviation range”) in the voxels (v _x , v _y , v _z ) are as follows.

Figure 12 shows how s _h is how they are computed. The small horizontal length h _r in the retina is assigned to the length h _NDC in the normalized device coordinate system (NDC). Scaling factor _{s h is} calculated by _h NDC / _{h r.} The vertical scaling factor s _v can be calculated similarly.

  Here, it should be noted that any of the variables on the right side in Equation 9 can change depending on the respective voxel. The result of the 2 × 2 matrix multiplication is stored corresponding to each voxel.

(Adjustment by eye)
As is well known, the human eye attempts to adjust the refractive power so that it is the smallest circle of confusion with a blurred pattern of retina. _Let zr be the distance between the thin lens and the retina in the eyeball model. This condition is satisfied by setting a = −b in Equation 8. Therefore, the following formula is obtained.

Extending the above law with equations 3, 7, 4 and 5, and using the fact that n ₁ = 1.0, we get:
here,
It is. Optically, the κ bar is the average curvature of the wavefront incident on the eye, and the P bar is the average refractive power of the eye.

Let P _min bar be the refractive power at the minimum (relaxed) accommodation in a standard, non-astigmatic eye, and ΔP be the accommodation power limited by the maximum accommodation capacity Δ _max . That is,
And This
It becomes.

From the equations 11 and 12, the following equation is obtained.

The meaning of the above adjustment formula is as follows. The average curvature κ bar of incident light is usually negative. When this becomes smaller (that is, when the −κ bar becomes larger), the light wave comes from a position close to the eye, and the effort ΔP required for adjustment increases. The reverse is also true. Typical values for _Δmax are shown in Table 2. This value decreases with age.

Let A be an astigmatism component defined by the user. That is, A = P _{1, min} −P _{2, min} . Here, P _{1, min} and P _{2, min} are the basic refractive powers of the eye with minimum adjustment, which is as follows.

Therefore,
It is. And by definition, it is possible to obtain the basic refractive power of the eye taking account of adjustment.

The wavefront tracking module receives the above two values along with the user defined parameter ω ₁ ^(e) to enable the tracking described above. That is, in the present embodiment, it is possible to set the parameters of the eye lens in consideration of the eye adjustment function.

  Table 2 above is a standard eye parameter. Table 3 shows examples of parameters that need to be controlled in order to simulate the appearance.

  As described above, the blur amount on the retina can be calculated as an elliptical range having an inclination using the wavefront tracking method. The vertex shift range corresponding to each voxel is stored as a blur field value (2 × 2 matrix of equation (9)). The operation of actually shifting the vertices is performed in the rendering described below.

(Gender rendering implementation)
In the main rendering loop, the scene is drawn repeatedly according to the number of samplings specified by the user. Then, the drawing results are mixed to generate a blurred image.

  In each sampling, one displacement seed vector (hereinafter referred to as a seed vector, FIG. 13A) is generated and supplied to the vertex shader. The seed vector is a 2D vector, starting from the center of the unit circle and ending at a point randomly generated within the unit circle.

In the vertex shader, the seed vector is converted into an actual 2D displacement vector corresponding to the vertex position (FIG. 13B). For this conversion, the 2 × 2 matrix B _{vx, vy, vz} (Equation 9, FIG. 13C) that is the voxel value of the blur field obtained above is used. This voxel value is referenced by the NDC coordinate of the vertex. Here, the relationship between the viewpoint coordinate system (camera coordinate system) and the NDC coordinate system is shown in FIG. Such a voxel separation method can be realized by dividing the NDC coordinate system, which is an intermediate result when the vertex processing is performed by the GPU, at equal intervals. By doing so, there is not only a quality advantage that the vicinity of the eye can be made dense, but also a great advantage in processing performance that intermediate data of each object vertex coordinate at the time of display processing can be used for displacement. . Since the vertex in this coordinate system (NDC) is generally always calculated in the display process, the voxel to which it belongs can be easily found based on the NDC coordinates. If the amount of displacement at the voxel is also stored in advance in the value of the NDC coordinate system, the displacement process is completed simply by adding to the intermediate coordinates. That is, there is an advantage that unnecessary coordinate conversion is not required.

  In reality, the vertex shader always calculates the homogeneous coordinates (x, y, z, w) one step before the NDC coordinates. In this embodiment, each of x, y, and z is divided by w of the homogeneous coordinates to obtain the NDC coordinates, and the vertex is displaced by the process of adding the 2D displacement vector and returning to the homogeneous coordinates again. .

The blur field values B _{vx, vy, vz} are given to the vertex shader as a 2D texture in which 3D textures or partial 2D textures are arranged. The typical size of the blur field is 32 × 32 voxels in the x and y directions and 128 voxels in the z (depth) direction. Each field value is a 2 × 2 matrix and can be expressed as a 4-element texel (texture pixel).

  An example of vertex shader code for displacement is shown below. This uses the OpenGL shader language. By this operation, the vertex of the polygon can be actually shifted and drawn. A blurred image can be generated by performing drawing a plurality of times while superimposing the obtained images while changing the amount of vertex shift.

  FIG. 15 is an explanatory diagram showing the entire processing in this embodiment. As shown here, in the perspective transformation, the first pass and the second pass are respectively drawn, and a two-dimensional image obtained by combining them is presented to the user. Since the process of drawing for each pass is a well-known technique, detailed description thereof is omitted.

(Results and performance)
Performance is affected by the number of all polygons included in the scene. In particular, the complexity of the mesh in the refractive lens is influential. This is because the ray is traced for each vertex. Here, a polygon model that is divided so as to have 1000 vertices is used. This can generate an image that is substantially as accurate as ray tracing. The following experiments were performed under the following conditions.
3.2GHz Pentium4 CPU
Using NVIDIA GPU GeForce 7800GTX Graphics
Output image size of 1024 x 768 pixels

  Rendering twice refracted while passing through the lens achieved 30-40 fps (frame per second) for a scene with approximately 10,000 polygons. When chromatic aberration was taken into account, the throughput dropped to 15 fps (see FIG. 16). FIG. 16 is an example of drawing in consideration of chromatic aberration. Here, the three colors of RGB are drawn separately. Also, different refractive indexes are set for each color. For example, (R: 1.60, G: 1.61, B: 1.62). The image obtained by superimposing the images obtained in this way is presented to the user as a two-dimensional image. Thereby, the state of chromatic aberration can be known.

  The defocus display processing is performed by combining a plurality of drawing results obtained by sampling each vertex within the movement range. Its performance is affected by the number of samplings. Here, a 10-sampled blurred image is used. From experience, this is considered sufficient for verification of the lens design. When there is no refraction by the lens, the frame rate of the blurred image shown on the left side of FIG. 17 is approximately 30 fps. In the case of rendering that takes into account the effects of refraction due to the spectacle lens, the value is 7 fps to 20 fps or higher. However, this is affected by the complexity of the scene, the window size, and the number of samples. The center and right images in FIG. 17 were 20.1 fps respectively. The conditions at this time were 1024 × 768 windows with 10 samplings. In the image of FIG. 17, the diameter of the pupil is set to 2 cm, and the result of defocus is emphasized. The value of the blur field is proportional to the pupil size rp. This is clear from equations 8 and 9.

  FIG. 18 shows the verification result of the progressive lens under design. The upper half of the lens is designed to correct myopia and the lower half is designed to correct presbyopia in the center of the lens. Although the left and right sides of the lower half are not corrected well, this indicates that it is difficult to design a progressive lens.

  An example of astigmatism is shown in FIG. Here, the voxel information in a part of the blur field is shown as an ellipse group. FIG. 19 is a drawing result by simulation for astigmatism of the naked eye. The newspaper that is the object is placed quite close (6.1 cm in the lower left corner and 10.7 cm in the upper right corner). At this distance, adjustment by the human eye is difficult and defocusing occurs. Because of the astigmatism setting, the out-of-focus range, i.e. the blur field, becomes elliptical. A part of the blur field (a group of voxels at approximately the same depth as the newspaper) is displayed as an ellipse for each voxel. Here, it should be noted that the ellipses of each blur field itself are displayed out of focus within the range of their own size. The elliptical shape differs depending on the distance from the viewpoint. This is based on the phenomenon described as Sturm's conoid.

  Observations for all blur fields are shown in FIG. FIG. 20 shows a state in which a blur field of 8 × 8 × 128 voxels is observed (upper right). The result of applying the blur field is also shown (lower right). Each small circle represents a blur field value for a voxel. Near the viewpoint, the voxel spacing becomes very narrow, so each circle looks close to each other and looks like a lump. The size of the circle indicates the apex deviation range at that point. This is as shown in FIG. Here, strong myopia and presbyopia are simulated, and the focus is only on the right computer display model. At the point of focus, the blur field value is zero and no circle is displayed.

  In this embodiment, wavefront tracking is used for the pre-computation to generate the blur field. This pre-calculation took 300 milliseconds for a blur field of 8 × 8 × 128 voxels. It took 1 second if the blur field was 16 × 16 × 128, and 7 seconds if it was 32 × 32 × 256.

(Conclusion)
As described above, according to the method of the present embodiment, it is possible to provide a high-speed rendering method in consideration of human visual acuity. The features of this method are as follows.
-Refraction and blur can be expressed at an almost real-time frame rate.
・ A fine eyeball model that can simulate the visual acuity of various symptoms and refraction by the eyeball lens is adopted.
・ Bokeh information is spatially distributed and pre-calculated and stored as a blur field.

  Blurfield is a new concept that can be used for high-speed defocusing. It can be easily implemented using a programmable GPU, the data itself is simple and the capacity is not very large. The blur field can be applied in a wide range of fields as long as it is an application field that requires displacement of an object according to a position in space. Blurfield can also be combined with distributed ray tracing if the programmer adds vertex displacement routines.

  When using the blur field technique, the object needs to be divided into meshed polygons. Also, this polygon needs to have a granularity comparable to the blur field voxel size.

  In the above example, an example using a spectacle lens is shown, but the present invention can be applied to a scene where it is necessary to generate a visual appearance without using a spectacle lens.

  It should be noted that the scope of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the scope of the present invention.

  For example, each component described above may exist as a functional block, and may not exist as independent hardware. As a mounting method, hardware or computer software may be used. Furthermore, one functional element in the present invention may be realized by a set of a plurality of functional elements. A plurality of functional elements in the present invention may be realized by one functional element.

  Moreover, the functional element may be arrange | positioned in the position physically separated. In this case, the functional elements may be connected by a network.

It is a schematic functional block diagram of the image generation apparatus which concerns on this embodiment. It is explanatory drawing for demonstrating the image generation method by vertex-based ray tracing. FIGS. 3A and 2B are diagrams showing results drawn by the method of FIG. 2, in which FIGS. 1A and 1B show results of drawing by vertex-based ray tracing, and FIGS. 2C and 2D show drawing by conventional cube mapping. Results are shown. It is a figure which shows the drawing result in the refraction through a concave lens. FIG. 2B is an enlarged view of the vicinity of the right end of FIG. It is a flowchart for demonstrating an example of the image generation method in this embodiment. It is explanatory drawing for demonstrating an example of the image generation method in this embodiment. In this embodiment, it is a flowchart for demonstrating an example of the method of calculating | requiring a lens path | route. It is explanatory drawing for demonstrating the method of calculating | requiring a lens path | route. FIG. 9 is an explanatory diagram illustrating an example of a result of drawing a rectangular model in the example of FIG. 8. It is a figure for demonstrating the calculation method of the wavefront in the wavefront tracking method. The figure (a) shows the input wave, the figure (b) shows the refractive power of the astigmatic eye, and the figure (c) shows the refracted wave. It is a figure which shows the conoid of the starm formed with a toric lens. It is explanatory drawing for demonstrating the calculation method for converting the blur amount on a retina into the vertex shift | offset | difference amount in a voxel. It is explanatory drawing for demonstrating calculation of the amount of vertex shifts in a voxel. FIG. 4A is a diagram showing an example of a seed vector common to all voxels. FIG. 5B is a vector diagram (NDC coordinate system) showing an actual displacement given to a vertex in a certain voxel using a seed vector. FIG. 5C is a calculation example showing the value of the blur field in a certain voxel. It is explanatory drawing for demonstrating the procedure converted from a viewpoint coordinate system to a normalized device coordinate system. It is explanatory drawing which shows the detail of the drawing procedure in an Example. It is a figure which shows the example of drawing which considered chromatic aberration. It is a figure which shows the example of a drawing obtained by simulating myopia and hyperopia. It is an example of the image which simulated the state which looked at the target object via the progressive focus lens. It is a simulation result of how to see with the naked eye of astigmatism. FIG. 5A shows an example of an image when no blur is considered, and FIG. 5B shows an example of an image when blur is taken into consideration. It is explanatory drawing for demonstrating the example of a blur field.

Explanation of symbols

1 viewpoint 2 view volume 3 lens 4 voxel 5 rectangular model 10 input unit 11 input interface 20 display unit 21 output interface 30 storage unit 40 image generation unit 50 bus

Claims (5)

A method of generating a two-dimensional image by performing perspective transformation from an observer's viewpoint on an object placed in a view volume, and further comprising the following steps:
(A) dividing the space in the view volume into a plurality of voxels that are made finer as the viewpoint from the observer is closer;
(B) calculating a blur amount on the retina of the observer by following an optical path from the center of the voxel to the viewpoint;
(C) calculating and recording a vertex shift range in the voxel corresponding to the amount of blur on the retina;
(D) actually shifting the vertex according to the vertex shift range;
(E) generating the two-dimensional image viewed from the viewpoint by perspective transformation; further,
(F) determining an optical path from the viewpoint to the center of the voxel, this step being performed before the step (b);
The image generation method according to claim 1, wherein the step (b) is executed by tracing back the optical path obtained in the step (f).   The image generation method according to claim 1, wherein in step (b), a wavefront tracking method is used as a method of tracing an optical path from the center of the voxel to the viewpoint. An apparatus for generating a two-dimensional image by performing perspective transformation on an object arranged in a three-dimensional space,
The apparatus includes a storage unit and an image generation unit,
The storage unit stores data and computer programs necessary for processing,
The image generation unit performs the following processing:
(A) dividing the space in the view volume into a plurality of voxels that are made finer as the viewpoint from the observer is closer;
(B) calculating a blur amount on the retina of the observer by following an optical path from the center of the voxel to the viewpoint;
(C) calculating and recording a vertex shift range in the voxel corresponding to the amount of blur on the retina;
(D) actually shifting the vertex according to the vertex shift range;
(E) generating the two-dimensional image viewed from the viewpoint by perspective transformation; A method of generating a two-dimensional image by performing perspective transformation on an object arranged in a three-dimensional space, and further comprising causing a computer to execute an image generation method including the following steps: Computer program:
(A) dividing the space in the view volume into a plurality of voxels that are made finer as the viewpoint from the observer is closer;
(B) calculating a blur amount on the retina of the observer by following an optical path from the center of the voxel to the viewpoint;
(C) calculating and recording a vertex shift range in the voxel corresponding to the amount of blur on the retina;
(D) actually shifting the vertex according to the vertex shift range;
(E) generating the two-dimensional image viewed from the viewpoint by perspective transformation;