JP5452801B2 - Stereoscopic image generating apparatus and program thereof - Google Patents

Description

  The present invention relates to a technique for generating a three-dimensional image from a three-dimensional shape model to be displayed in a three-dimensional display method such as an integral method or a lenticular method.

  In recent years, research and development of 3D display devices have progressed, and autostereoscopic 3D display devices such as an integral method and a lenticular method that allow a viewer to view a 3D image without using special glasses have been developed. . For example, an integral-type stereoscopic display device includes a high-definition image display panel and a two-dimensional lens array disposed on the front surface thereof, and a stereoscopic image is reproduced when an element image is displayed on the high-definition image display panel. Light rays from each pixel of the element image are emitted in a direction determined by the pixel position and the position of the element lens of the lens array. That is, as many rays as the number of pixels of the element image are emitted from one element lens. Here, when the condition that the distance between the element lenses is an integral multiple of the pixel size of the element image, the direction of the light rays emitted from each element lens is finite. Therefore, the orthogonal projection is performed by collecting the light rays in the same direction. It can be represented as an image. That is, the light ray group emitted from the stereoscopic display device can be represented by a finite number of orthographic images.

  Here, techniques for converting a three-dimensional shape model obtained by computer graphics (CG) or a range sensor into an integral stereoscopic image have been proposed (for example, Non-Patent Documents 1 and 2). These techniques described in Non-Patent Documents 1 and 2 are basic techniques for converting a three-dimensional shape model into an integral stereoscopic image. For example, in the technique described in Non-Patent Document 1, a three-dimensional shape model of computer graphics is used as a subject, a lens array and a lens for depth control are installed in a three-dimensional virtual space. A three-dimensional image is generated.

  In other words, in the techniques described in Non-Patent Documents 1 and 2, if orthographic projection can be used, a display image can be generated from a three-dimensional shape model by rendering light rays of the number of element lenses of the lens array once. Can do. In this case, in the techniques described in Non-Patent Documents 1 and 2, it is possible to generate a display image from a three-dimensional shape model with a smaller amount of computation than in the case where center projection is used.

Athineos, Spyros S ,: “Physical modeling of a microlens array setup for use in computer generated IP”, Proceedings of the SPIE, Volume 5664, pp.472-479 (2005). Huy Hoang Tran, et al: “Interactive 3D Navigation System for Image-guided Surgery”, Journal of Virtual Reality, 8 (1), pp. 9-16 (2009)

  However, in the techniques described in Non-Patent Documents 1 and 2, when the distance between the element lenses does not satisfy the condition that the element size is an integer multiple of the pixel size of the element image, the direction of the light rays is different for each element lens. It cannot be represented by one orthogonal projection image. Since this orthographic projection is an effective means for generating a display image with a small amount of calculation, there is a strong demand from users to generate a display image by orthographic projection even when the above conditions are not satisfied.

  Therefore, the present invention solves the above-described problem, and provides a stereoscopic image generation apparatus and program for generating a display image by orthographic projection even when the interval between element lenses does not satisfy the condition that the element size is an integer multiple of the pixel size. The purpose is to provide.

  In order to solve the above-described problem, the stereoscopic image generating device according to the first invention of the present application is a three-dimensional shape as a display target to be displayed on a stereoscopic display device including a lens array in which element lenses are two-dimensionally arranged and image display means. A stereoscopic image generating apparatus that captures a model with a virtual camera that is a virtual imaging camera and generates a display image for the stereoscopic display apparatus, the parameter input means, a normalized image pixel size calculating means, a normal image It comprises a normalized image pixel position calculating means, a virtual camera position / posture calculating means, a virtual captured image generating means, a normalized image generating means, and a display image generating means.

  According to such a configuration, the stereoscopic image generating apparatus receives the focal length of the element lens, the interval of the element lens, and the pixel interval of the image display unit by the parameter input unit. Further, the stereoscopic image generating apparatus is configured such that the normalized image pixel size calculating unit divides the element lens interval by the pixel interval of the image display unit and multiplies the element lens interval by a preset constant. In the normalized image having a pixel size that is a fraction of an integer with respect to the distance between the element lenses, the distance between the element lenses is divided by the number of pixels between the element lenses. The pixel size of the normalized image is calculated. Then, the stereoscopic image generating apparatus calculates the pixel coordinates of the normalized image by dividing the pixel interval of the image display unit by the pixel size of the normalized image by the normalized image pixel position calculating unit. In this way, the stereoscopic image generating apparatus prepares a normalized image whose pixel size is 1 / integer with respect to the interval between the element lenses.

  Further, the stereoscopic image generating device may multiply the element set by multiplying a distance set in advance so as to indicate the position from the lens array to the position of the virtual camera by the virtual camera position / posture calculating unit and the pixel size of the normalized image. The position of the virtual camera is calculated based on the value divided by the focal length of the lens, and the reverse direction of the unit vector from the pixel position of the element image to the center position of the element lens in the normalized image is taken by the virtual camera. And calculating the orientation of the virtual camera using the shooting direction of the virtual camera. That is, the three-dimensional image generation apparatus calculates, as a shooting direction of the virtual camera, a unit vector from the center position of the element lens toward the pixel position of the element image in the normalized image by the virtual camera position / posture calculation unit. In addition, the stereoscopic image generation device uses the virtual camera position / orientation calculation means to generate a unit vector indicating the shooting direction of the virtual camera and a unit vector indicating the vertical direction of the plane of the lens array (a direction orthogonal to the row of the lens array). From the above, the attitude of the virtual camera is calculated.

In addition, the stereoscopic image generating apparatus orthographically projects the three-dimensional shape model onto a virtual plane based on the position and orientation of the virtual camera calculated by the virtual camera position / orientation calculating unit by the virtual captured image generating unit. Then, a virtual photographed image obtained by photographing the three-dimensional shape model with the virtual camera is generated. Then, the stereoscopic image generating apparatus generates the normalized image by assigning the pixel value of the pixel of the virtual photographed image to the pixel of the normalized image using a predetermined coordinate conversion formula by the normalized image generating unit. In other words, the normalized image pixels and the virtual captured image pixels are arranged on a straight line by the above-described orthographic projection, and the stereoscopic image generating device uses the prepared normal values of the pixel values of the virtual captured image according to the straight line. Assigned to each pixel of the digitized image. Therefore, the stereoscopic image generating apparatus forms an element image on the normalized image.
Furthermore, the stereoscopic image generating apparatus generates the display image from the normalized image generated by the normalized image generating unit by the display image generating unit.

  In the stereoscopic image generating device according to the second invention of the present application, in the virtual photographed image, a plurality of filter application regions surrounded by a midpoint of a line segment connecting pixels corresponding to the center position of the element lens are set. It is further characterized by further comprising first aliasing noise reduction means for performing low-pass filter processing for each filter application area.

Here, the observer of the stereoscopic display device sees the stereoscopic image through the lens array. This is equivalent to sampling a three-dimensional image according to the distance between the element lenses of the lens array. For this reason, in the lens array, aliasing noise (folding distortion) due to sampling occurs.
Therefore, the stereoscopic image generating apparatus regards the pixel corresponding to the center of the element lens of the virtual photographed image (orthographic projection image) as the sample point by the first aliasing noise reduction unit, and applies the filter application region centered on the sample point. A low-pass filter process is performed every time. Thereby, the stereoscopic image generating apparatus can reduce aliasing noise caused by sampling in the lens array.

  In addition, the stereoscopic image generating apparatus according to the third invention of the present application further includes filter application area setting means for setting the filter application area to the virtual photographed image using a matrix indicating the attitude of the virtual camera. And

  According to such a configuration, the stereoscopic image generating apparatus can set the filter application area determined by the arrangement of the element lenses without depending on the hand.

In order to solve the above-described problem, the stereoscopic image generation program according to the fourth invention of the present application is a 3 as a display object to be displayed on a stereoscopic display device including a lens array in which element lenses are two-dimensionally arranged and image display means. In order to capture a three-dimensional shape model with a virtual camera, which is a virtual imaging camera, and generate a display image for the stereoscopic display device, a computer is provided with parameter input means, normalized image pixel size calculation means, and normalized image. It functions as a pixel position calculating means, a virtual camera position / posture calculating means, a virtual captured image generating means, a normalized image generating means, and a display image generating means.

  According to such a configuration, the focal length of the element lens, the interval between the element lenses, and the pixel interval of the image display unit are input to the stereoscopic image generation program by the parameter input unit. Further, the stereoscopic image generation program is configured such that the normalized image pixel size calculation unit divides the interval between the element lenses by the pixel interval of the image display unit and multiplies a predetermined constant between the element lenses. In the normalized image having a pixel size that is a fraction of an integer with respect to the distance between the element lenses, the distance between the element lenses is divided by the number of pixels between the element lenses. The pixel size of the normalized image is calculated. Then, the stereoscopic image generation program calculates the pixel coordinates of the normalized image by dividing the pixel interval of the image display unit by the pixel size of the normalized image by the normalized image pixel position calculating unit. In this way, the stereoscopic image generation program prepares a normalized image whose pixel size is 1 / integer with respect to the interval between the element lenses.

  Further, the stereoscopic image generation program multiplies the distance set in advance to indicate the position from the lens array to the position of the virtual camera by the virtual camera position / posture calculation means and the pixel size of the normalized image. The position of the virtual camera is calculated based on the value divided by the focal length of the lens, and the reverse direction of the unit vector from the pixel position of the element image to the center position of the element lens in the normalized image is taken by the virtual camera. And calculating the orientation of the virtual camera using the shooting direction of the virtual camera. That is, the stereoscopic image generation program calculates the unit vector from the center position of the element lens toward the pixel position of the element image in the normalized image as the shooting direction of the virtual camera by the virtual camera position / posture calculation means. In addition, the stereoscopic image generation program uses the virtual camera position / orientation calculation means to generate a unit vector indicating the shooting direction of the virtual camera and a unit vector indicating the vertical direction of the plane of the lens array (a direction orthogonal to the arrangement of the rows of the lens array). From the above, the attitude of the virtual camera is calculated.

Further, the stereoscopic image generation program projects the three-dimensional shape model onto the virtual plane based on the position and orientation of the virtual camera calculated by the virtual camera position / orientation calculation unit by the virtual captured image generation unit. Then, a virtual photographed image obtained by photographing the three-dimensional shape model with the virtual camera is generated. Then, the stereoscopic image generation program generates the normalized image by allocating the pixel value of the pixel of the virtual photographed image to the pixel of the normalized image using a predetermined coordinate conversion formula by the normalized image generating unit. In other words, the normalized image pixels and the virtual photographed image pixels are arranged on a straight line by the above-mentioned orthographic projection, so the stereoscopic image generation program uses the prepared normal values for the pixel values of the virtual photographed image pixels according to this straight line. Assigned to each pixel of the digitized image. Therefore, the stereoscopic image generation program forms an element image on the normalized image.
Further, the stereoscopic image generation program generates the display image from the normalized image generated by the normalized image generation unit by the display image generation unit.

The present invention has the following excellent effects.
The first and fourth inventions of the present application prepare a normalized image whose pixel size is 1 / integer with respect to the interval between the element lenses. In the first and fourth inventions of the present application, orthographic projection is performed with the center of the element lens corresponding to the pixel of the virtual photographed image, and the pixel value of the pixel of the virtual photographed image is assigned to each pixel of the normalized image. As a result, the first and fourth inventions of the present application can generate a display image by orthographic projection even when the interval between the element lenses does not satisfy the condition of an integral multiple of the pixel size of the element image, so that the amount of calculation can be reduced. .

In the second invention of the present application, aliasing noise caused by sampling in the lens array can be reduced, so that the display image can be of high quality.
According to the third invention of the present application, the filter application region determined by the arrangement of the element lenses can be appropriately set without depending on the hand, so that accurate low-pass filter processing can be performed .

It is the schematic which shows the outline of the three-dimensional display apparatus in this invention, (a) is a figure which shows a mode that a three-dimensional image is displayed, (b) shows the positional relationship of a lens array and the display surface of an image display panel. FIG. It is a block diagram which shows the structure of the stereo image production | generation apparatus which concerns on embodiment of this invention. It is a conceptual diagram which shows modeling of the three-dimensional display apparatus in this invention, and is a figure which shows the positional relationship with an element image, an image display panel, and the pinhole of a lens array. In this invention, it is a figure which shows the positional relationship of the normalized image and the pinhole of a lens array. It is a block diagram which shows the structure of the virtual camera imaging | photography means of FIG. In this invention, it is explanatory drawing explaining imaging | photography with a virtual camera, and is a figure which shows the positional relationship of a lens array, a normalized image, and a three-dimensional shape model. In this invention, it is explanatory drawing explaining imaging | photography with a virtual camera, and is a figure which shows the positional relationship of the sample point on a lens array, and a normalized image. In this invention, it is explanatory drawing explaining imaging | photography with a virtual camera, and is a figure which shows the positional relationship of a normalized image, a lens array, and a virtual camera. It is explanatory drawing explaining the low-pass filter process by the 1st filter of FIG. It is explanatory drawing explaining the production | generation of the display image by the 2nd filter of FIG. 3 is a flowchart illustrating an overall operation of the stereoscopic display device of FIG. 2. It is a flowchart which shows operation | movement of the virtual camera imaging | photography means of FIG. In this invention, it is a figure which shows the structure of the three-dimensional video generation system which produces | generates the integral three-dimensional video of a dynamic three-dimensional shape model. It is an image explaining the Example of this invention, (a) is a picked-up image in a comparative example, (b) is a picked-up image in an Example, (c) is an enlarged image of (a), ( d) is an enlarged image of (b).

[Outline of stereoscopic display device]
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate. In each embodiment, means having the same function are denoted by the same reference numerals and description thereof is omitted.
First, an outline of the stereoscopic display device 2 according to the present invention will be described as a premise of the stereoscopic image generating device 1 (see FIG. 2) according to the embodiment of the present invention.

As shown in FIGS. 1A and 1B, the stereoscopic display device 2 integrates a display image I (an image in which a plurality of element images G are arranged vertically and horizontally) generated by the stereoscopic image generating device 1 into an integral system. is intended to be displayed by includes an image display panel (image display means) 21 such as a liquid crystal display, and a lens array L _a placing the element lenses L _P two-dimensionally.
First, the stereoscopic display device 2 displays the display image I of the display target (three-dimensional shape model) generated by the stereoscopic image generation device 1 on the display surface of the image display panel 21. In this case, the element lenses L _P corresponding to the element image G is uniquely determined. Then, the stereoscopic display device 2 forms a stereoscopic image Z and projected by the element lenses L _P elements image G in the air. In other words, the viewer A, through element lenses L _P of the lens array L _A, by viewing each element image G, it is possible to visually recognize a stereoscopic image Z.
Thus, the display image I for displaying on the stereoscopic display device 2 is required. Therefore, the display image I is generated using the stereoscopic image generating apparatus 1 according to the embodiment of the present invention.

[Configuration of stereoscopic image generation apparatus]
Hereinafter, with reference to FIG. 2, the configuration of the stereoscopic image generating apparatus 1 according to the embodiment of the present invention will be described.
As illustrated in FIG. 2, the stereoscopic image generation device 1 is configured to generate a display image of the stereoscopic display device 2 by photographing a three-dimensional shape model as a display target to be displayed on the stereoscopic display device 2 with a virtual camera. Stereoscopic display device modeling means (parameter input means) 11, normalized image setting means (normalized image pixel size calculating means) 12, filter parameter setting means (filter application area setting means) 13, and virtual camera photographing Means 14, a first filter (first aliasing noise reduction means) 15, a normalized image generation means 16, and a display image generation means 17 are provided.

The stereoscopic display device modeling means 11 receives parameters (F, R _H , R _V , P) for modeling the stereoscopic display device 2 and outputs these parameters to the normalized image setting means 12. These parameters are defined as shown in FIG.

F: focal length of the element lenses _{L P} (distance between the image display panel 21 and the lens array _{L A)}
R _H: the lens array _L horizontal spacing element lenses _{L P} in _{A R V:} lens array _L vertical spacing element lenses _{L P} in _A P: pixel spacing of the image display panel 21

As in this embodiment, the element lens L _P are offset for each row, a √3 / 2 times the horizontal spacing R _H of vertical spacing R _V elements lens L _P element lenses L _P explain.
In addition, it is _{assumed that} the horizontal interval _RH of the element lens L _{P and} the vertical interval R _{V of the} element lens L _P are not an integral multiple of the pixel interval of the element image G.
In the stereoscopic display device 2, it is regarded the center position of the element lenses L _P pinhole H. That is, the center position of the element lenses L _P is the position of the pinhole H.

Since the interval R _V vertical horizontal gap R _H and element lenses L _P element lenses L _P is not an integer multiple of the pixel spacing element image G, taken using a virtual camera (orthographic projection camera) Even in this case, it is not possible to directly acquire the light beam at the pixel position of the element image G. Therefore, as shown in FIG. 4, in the process of shooting a virtual camera (calculation process), to define the pixel spacing element image G, the vertical direction in the horizontal direction between R _H and element lenses L _P element lenses L _P distance R _V of preparing the image surface as an integral multiple of the pixel spacing element image G. Hereinafter, the image on this image plane is referred to as a normalized image I ′.

It also defines world coordinates that indicate the physical location. The world coordinates, element lenses L _P lens center which is located in the center of the lens array L _A (pin holes) as the origin, X-axis lateral of the lens array L _A, the longitudinal direction of the Y-axis and a lens array surface The vertical direction is the Z axis.
Further, (i, j) is defined as the image coordinates of the entire element image G. It is assumed that the origin of the image coordinates is the pixel closest to the origin of the world coordinates.

Normalized image setting unit 12, the normalized image I ', as represented by the following formula (1), the vertical gap horizontal gap R _H and element lenses L _P element lenses L _P R by dividing the _V pixel number M between elements lens L _P, to calculate the vertical pixel size P _V of the 'horizontal pixel size P _H and the normalized image I' of normalized image I.

Then, the normalized image setting unit 12 outputs these parameters (F, R _H , R _V , P, P _H , P _V ) to the filter parameter setting unit 13. In this case, the normalized image setting means 12, as represented by the following formula (2), the constant by dividing the horizontal gap R _H of the element lenses L _P in the pixel spacing P of the image display panel 21 K based on the value obtained by multiplying the calculated number of pixels M between the element lenses L _P.

Here, in formula (2), “<a>” means rounding off a to an integer. _{That, "<R H / P>} " becomes the number of pixels between the element lenses L _P in FIG. 3, for low-pass filtering in a second filter 17a to be described later, to K times the value. Here, the constant K can be set to an arbitrary value. Further, when it is set in the even that the constant K exceeds 2, the pixel size of the normalized image I 'is less than half the pixel size of the image display panel 21, and the horizontal spacing of the element lenses L _P the integral submultiple against R _H and element lenses _{L P} of vertical spacing _{R V.} Thereby, the low-pass filter processing by the second filter 17a described later can be easily applied.

The filter parameter setting unit 13 receives parameters (F, R _H , R _V , P, P _H , P _V ) from the normalized image setting unit 12 and a preset constant N. The constant N is a value that indicates that sampling by a constant multiplying the number of pins holes in the lens array L _A. Then, the filter parameter setting means 13 uses the following equation (8) as the origin (0, 0) of the local coordinates (u, v) described later on the virtual photographed image (image of the virtual camera Vc). A filter application area A _R (see FIG. 9) is set.

Thereafter, the filter parameter setting means 13 outputs the parameters (F, R _H , R _V , P, P _H , P _V ) to the virtual camera photographing means 14 and sets the set filter application area _AR to the first. Output to the filter 15.
The filter application area A _R is for indicating a range in which the first filter 15 to be described later to perform low pass filtering, the details will be described later.

  The virtual camera photographing means 14 generates a virtual photographed image by virtually photographing a display target with the virtual camera Vc (see FIG. 6). For this reason, as shown in FIG. 5, the virtual camera photographing means 14 includes a normalized image pixel position calculating means 141, a pinhole position calculating means 142, a virtual camera position / posture calculating means 143, and a virtual photographed image generating means. 144.

The normalized image pixel position calculation unit 141 receives parameters (F, R _H , R _V , P, P _H , P _V ) from the filter parameter setting unit 13. Here, using equation (1), the (world coordinates) of the normalized image I ′ can be expressed by the following equation (3).

Therefore, the normalized image pixel position calculation unit 141 sets the pixel interval P of the image display panel 21 to the pixel size P _{H in} the horizontal direction of the normalized image I ′ and the normalized image I ′ as represented by the following equation (4). 'it is divided by the respective vertical pixel size P _V of the normalized image I' of the image I to calculate the pixel coordinates of the. That is, the pixel coordinates of the display image I can be converted into the pixel positions of the normalized image I ′ using Expression (4).
In equation (4), (i ′, j ′) represents the pixel coordinates of the normalized image I ′.

Then, the normalized image pixel position calculating unit 141 uses the parameters (F, R _H , R _V , P, P _H , P _V ) and the calculated pixel coordinates of the normalized image I ′ as the pinhole position calculating unit 142. Output to.

The pinhole position calculating unit 142 receives parameters (F, R _H , R _V , P, P _H , P _V ) and the calculated pixel coordinates of the normalized image I ′ from the normalized image pixel position calculating unit 141. Is done. Then, the pinhole position calculating means 142 calculates the position of the nth pinhole H in the horizontal direction and the mth pinhole H in the vertical direction using the following equation (5).
In Expression (5), “%” is a symbol indicating a remainder operation, and H (n, m) is the position of the pinhole H.

Then, the pinhole position calculation means 142 uses the pixel coordinates of the normalized image I ′, the position of the pinhole H, and the parameters (F, R _H , R _V , P, P _H , P _V ) as the virtual camera position. Output to attitude calculation means 143.

The virtual camera position / posture calculation unit 143 receives the pixel coordinates of the normalized image I ′ from the pinhole position calculation unit 142, the position of the pinhole H, and the parameters (F, R _H , R _V , P, P _H , P _V ). The virtual camera position / orientation calculating unit 143 generates a unit vector to be described later, and then calculates the position and orientation of the virtual camera Vc.
Here, the pixel of the element image G corresponding to the pinhole position H (n, m) is represented as I ′ _n , _m (u, v). In addition, (u, v) is the local image coordinates on the normalized image I ′ with H (n, m) as the origin.

First, as shown in FIG. 6, the virtual camera position / orientation calculating means 143 sets the pinhole position to H (0,0) in the normalized image I ′, and the element image G corresponding to this H (0,0). pixel I _{'0, 0} (u, v) for each pixel in the, using equation (6) below, the pixel I' position of _{0, 0} (u, v) from the pinhole H (0,0) Unit vectors (i _x , i _y , i _z ) are generated.
Here, the pixels I ′ ₀ , ₀ (u, v) = (X _P , Y _P , −F) = (uP _H , vP _V , −F). Further, the pinhole position H (0, 0) = (X _R , Y _R , 0) = (0, 0, 0).
This unit vector indicates the direction in which the pixel I ′ ₀ , ₀ (u, v) is visually recognized on the extension line.

Next, the virtual camera position / posture calculation unit 143 calculates the position and posture of the virtual camera Vc based on the generated unit vector and the distance Length (see FIG. 8). Specifically, the virtual camera position / orientation calculation unit 143 multiplies the distance Length by the pixel size (P _H , P _V ) of the normalized image I ′ as represented by the following equation (7). The position of the virtual camera Vc is calculated based on the value divided by the focal length F of the element lens.

The distance Length is the distance from the origin lens array _{L A} to the virtual camera Vc (e.g., 2 meters).
The virtual camera Vc is a virtual photographing camera that photographs a display target, and represents a viewing distance when the observer A views the stereoscopic display device 2. At this time, the distance from the virtual camera Vc to the stereoscopic display device 2 is assumed to be equal to the distance Length. Further, the virtual camera Vc from the origin lens array L _A, assumed that apart than 3D geometric model is displayed.

Then, the virtual camera position / orientation calculation unit 143 calculates the reverse direction r _z (see FIG. 6) of the generated unit vector as the shooting direction of the virtual camera Vc. Further, the virtual camera position and orientation calculation unit 143, to guide the image caught on the virtual camera Vc, the correspondence between the pinhole H of the lens array L _A, determine the attitude of the virtual camera Vc. Specifically, the virtual camera position / orientation calculation unit 143 obtains a unit vector orthogonal to both the reverse direction r _z of the unit vector and the Y axis of the world coordinate system as the horizontal axis r _x of the virtual camera Vc. Next, the virtual camera position / orientation calculation unit 143 obtains a unit vector orthogonal to both r _z and r _x as the vertical axis r _y of the virtual camera Vc. That is, in the element lens L _P with (n, m) = (0, 0), if the world coordinates of a pixel (u, v) in the element image G are (X _P , Y _P , −F), the virtual The matrix R indicating the attitude of the camera Vc can be expressed by the following equation (8).

  Thereafter, the virtual camera position / orientation calculation unit 143 outputs the calculated position and orientation of the virtual camera Vc to the virtual captured image generation unit 144.

  The virtual captured image generation means 144 receives the position and orientation of the virtual camera Vc from the virtual camera position / orientation calculation means 143, and orthogonally projects the display target on the virtual plane based on the position and orientation of the virtual camera Vc. Thus, a virtual photographed image obtained by photographing the display target with the virtual camera Vc is generated.

<Shooting with a virtual camera>
Hereinafter, with reference to FIGS. 6 to 9, photographing by the virtual camera Vc will be described in detail (see FIG. 5 as appropriate).
In FIG. 6, only a part of the surface of the three-dimensional shape model (display target) Obj is illustrated for the sake of simplicity.
In FIG. 7, “●” and “△” indicate sample points, “●” is a sample point at the pinhole H, and “△” is an increased sample point other than “●”. is there. In FIG. 7, “■” is a pixel to be calculated in the same relative position with respect to the pinhole H in each element image G. That is, the virtual captured image generation unit 144 can represent the pixel position of each element image G by local coordinates (u, v) with the pinhole H as the origin.
Further, in FIGS. 7 and 8, the number of pixels the virtual camera, the lens array L _A "●" are and "△" the sample points and the same number of combined. Further, in FIG. 8, each of the light rays is indicated by an arrow.

Specifically, the virtual photographic image generating unit 144, as shown in FIG. 6, about the pin hole H present in the reverse direction r _z unit vector, including the lens array L _A total of pinhole H The stereoscopic display device 2 virtually arranged at the angle of view is photographed virtually. That is, the virtual photographic image generating unit 144 as the center of the virtual photographic image position of the pinhole H present in the reverse direction r _z unit vector, with each pixel position of the element image G and the optical principal point of the virtual camera Vc A virtual captured image is generated by orthogonally projecting the color of the surface of the three-dimensional shape model Obj existing on the straight line connecting the two to the virtual plane (the focal position of the virtual camera Vc).

At this time, virtual-photographed-image generating unit 144, as shown in FIGS. 7 and 8, in imaging by a virtual camera Vc, which collectively sample the light passing through the pinhole lens array L _A. Here, the virtual photographic image generating means 144, in order to apply the first filter to be described later, samples the number of rays that the number of pinholes H N times the lens array L _A (e.g., N = 4) . Then, the virtual captured image generation unit 144 acquires a light beam corresponding to the pixel located at the local coordinates (u, v) of each element image G. Thus, in the virtual photographic images, the spacing element lenses L _P is to meet the integral multiple of the pixel size of the elemental image G, the direction of the light beam emitted from each element lens L _P is finite, It is possible to collect rays in the same direction. In other words, the virtual photographed image generating unit 144 can represent the emitted light group by a finite number of virtual photographed images.

Other words, by using the above-mentioned matrix R (see equation (8)), a pinhole H of the lens array L _A, can be orthogonally projected on the image plane of the virtual photographic image. In this case, assuming that the three-dimensional position of the pinhole is (X _R , Y _R , 0), the image coordinates (k, l) of the point orthogonally projected on the virtual photographed image are expressed by the following equation (9) and It is calculated | required using Formula (10).

  Here, (W ′, H ′) is the vertical and horizontal sizes of the virtual photographed image, and (K, L) is the number of pixels of the virtual photographed image. In this case, when the sample points “●” and “Δ” in FIG. 7 are projected onto the virtual photographed image, the projected points are as shown in FIG.

As described above, the virtual-photographed-image generating unit 144 may use Equation (9) and (10), and orthogonal projection of an arbitrary point on the lens array L _A plane virtual camera Vc. Thereafter, the virtual captured image generation unit 144 outputs the generated virtual captured image to the first filter 15.

<Reduction of aliasing noise caused by lens array sampling>
Hereinafter, with reference to FIG. 9, according to a first filter 15, it will be described the reduction of aliasing noise due to sampling at the lens array L _A (see properly Figure 2).
Here, the filter parameter setting means 13, in the virtual photographic image will be described the example of setting the filter application region A _R as shown in FIG. That is, the filter application area A _R is around a pixel corresponding to a pinhole H1, the line segment connecting the pixels corresponding to the other of the pinhole H adjacent to the pin hole H1 (one-dot chain line in FIG. 9) and a region which is surrounded by the middle point _C P 1 to 4 (e.g., diamond). That is, the shape of the filter application area A _R will depend on the attitude of the arrangement and the virtual camera Vc element lenses L _P.

The first filter 15, the filter application area A _R from the filter parameter setting unit 13 is inputted, the virtual captured image is inputted from the virtual camera photographing means 14. The first filter 15 performs low-pass filtering in the filter applied in the area A _R in the virtual photographic image. Specifically, the first filter 15, as a low-pass filtering, to calculate an average value of all pixel values existing in the filter application area A _R, the center of the filter the average coverage area A _R The pixel value of the “●” pixel located at. Moreover, the 1st filter 15 may obtain | require a weighted addition value or a median value instead of calculating | requiring an average value as a low-pass filter process. Thereafter, the first filter 15 outputs the virtual photographed image subjected to the low-pass filter processing to the normalized image generation means 16.

The filter parameter setting means 13, also with the pixel corresponding to another pinhole H except pinholes H1, sets the filter application area A _R Likewise. The first filter 15 is naturally subjected to low-pass filter processing for each the filter application area A _R.

The normalized image generation means 16 receives the virtual photographed image from the first filter 15 and assigns the pixel value of the pixel of this virtual photographed image to the pixel of the normalized image I ′ by a predetermined coordinate conversion formula. A normalized image I ′ is generated. Specifically, the normalized image generating means 16 uses the sample point (“●” in FIG. 9) of the virtual photographed image that has been subjected to the low-pass filter processing as the calculation target pixel (in FIG. 7) of the normalized image I ′. Assigned to “■”). Of course, the normalized image generating means 16 needs to obtain pixel coordinates on the virtual photographed image of the pinhole H whose position is represented by (n, m). Therefore, the normalized image generation unit 16, the physical location H (n, m) on the lens array L _A plane by the equation (5) is obtained using equation (9) and (10) Obtain the image coordinates of the virtual photographed image. Further, the pixel coordinates of the normalized image I ′ corresponding to these coordinates are expressed by Expression (11). In this case, the first term on the left side of Expression (11) indicates the position of the pinhole H.

Thereafter, in the stereoscopic image generating device 1, each pixel of the element image G is designated as local coordinates (u, v), and shooting processing with a virtual camera by the virtual camera shooting unit 14, low-pass by the first filter 15. The pass filter processing and the pixel value assignment processing by the normalized image generation means 16 are executed for all the pixels of the element image G. Therefore, in the stereoscopic image generating apparatus 1, it is only necessary to repeat these processes for the number of pixels of the element image G, and it is not necessary to repeat the process for all the pixels existing in the image as in the conventional case.
In this way, the normalized image generating means 16 can obtain all pixel values of the normalized image I ′. Thereafter, the normalized image generating unit 16 outputs the generated normalized image I ′ to the display image generating unit 17.

  The display image generation unit 17 receives the normalized image I ′ from the normalized image generation unit 16 and generates the display image I from the normalized image I ′. For this reason, as shown in FIG. 2, the display image generation means 17 is provided with the 2nd filter (2nd aliasing noise reduction means) 17a.

<Reduction of aliasing noise caused by sampling on the image display panel>
Hereinafter, with reference to FIG. 10, reduction of aliasing noise caused by sampling on the image display panel 21 by the second filter 17 a will be described (see FIG. 2 as appropriate).
The second filter 17a performs low-pass filter processing on the normalized image I ′ because the normalized image I ′ has M-times pixels in the vertical and horizontal directions of the display image I displayed on the image display panel 21. . Specifically, as shown in FIG. 10, the second filter 17 a includes a pixel value of one pixel of the display image I in a region corresponding to one pixel of the display image I as a low-pass filter process. The average value of all the pixels of the normalized image I ′ is processed. Furthermore, the second filter 17a may obtain a weighted addition value or a median value as a low-pass filter process instead of obtaining an average value.

  That is, since the second filter 17a can perform coordinate conversion by the above-described equation (4), the display image I is generated by performing the low-pass filter processing represented by the following equation (12). Then, the second filter 17a outputs the generated display image I to the stereoscopic image generating device 1 (see FIG. 1).

In Expression (11), I ′ (i ′ + s, j ′ + t) represents the pixel value of the pixel (i ′ + s, j ′ + t) of the normalized image I ′, and I (i, j) Indicates the pixel value of the pixel (i, j) of the display image I. D is the number of I (i ′ + s, j ′ + t) that satisfies the condition of Σ, and is the number of pixels that contributed to the sum. Moreover, s and t is an integer from 0 to half the number of pixels between the element lenses L _P.

[Operation of stereoscopic image generation device]
Hereinafter, the overall operation of the stereoscopic image generating apparatus 1 in FIG. 2 will be described with reference to FIG. 11 (see FIG. 2 as appropriate).
In the stereoscopic image generating device 1, the parameters (F, R _H , R _V , P) of the stereoscopic display device 2 are input by the stereoscopic display device modeling means 11 (step S1).

In addition, the stereoscopic image generating apparatus 1 sets the normalized image I ′ by using the normalized image setting unit 12 (Step S2). That is, the stereoscopic image generating apparatus 1, the normalized image setting unit 12, the formula (1) and (2) using the normalized image I 'horizontal pixel size P _H and the normalized image I' of the to calculate the vertical pixel size P _V

In addition, the stereoscopic image generating apparatus 1 sets the filter parameter by the filter parameter setting unit 13 (step S3). That is, the stereoscopic image generating apparatus 1, the filter parameter setting unit 13 sets the filter application area A _R on the virtual captured image by using the matrix R representing the orientation of the virtual camera Vc.

  In addition, the stereoscopic image generating apparatus 1 uses the virtual camera photographing unit 14 to photograph with the virtual camera Vc to generate a virtual photographed image (step S4). Details of the operation of the virtual camera photographing means 14 will be described later.

In addition, the stereoscopic image generating device 1 performs low-pass filter processing on the virtual photographed image using the first filter 15 (step S5). That is, the stereoscopic image generating apparatus 1, the first filter 15 calculates the virtual photographic image, the average value of all pixel values existing in the filter application area A _R, the weighted addition value or the median value, the the pixel values of all pixels existing values in filter application region a _R.

  In addition, the stereoscopic image generation apparatus 1 uses the normalized image generation unit 16 to normalize the pixel values of the pixels of the virtual captured image using the expressions (5), (9), (10), and (11). By assigning to the pixels of the normalized image I ′, a normalized image I ′ is generated (step S6).

  Thereafter, the stereoscopic image generating apparatus 1 uses the normalized image generating unit 16 to perform the imaging process with the virtual camera by the virtual camera imaging unit 14 and the low-pass filter using the first filter 15 for all pixels of the element image G It is determined whether the process and the pixel value assignment process by the normalized image generation means 16 have been executed (step S7).

Here, when the process is not executed on all the pixels of the element image G (No in step S7), the stereoscopic image generating apparatus 1 returns to the process of step S4.
On the other hand, when the process is executed on all the pixels of the element image G (Yes in step S7), the stereoscopic image generating apparatus 1 proceeds to the process of step S8.

  In addition, the stereoscopic image generating apparatus 1 performs low-pass filter processing on the normalized image I ′ using the second filter 17a (step S8). That is, the stereoscopic image generating apparatus 1 uses the second filter 17a as a low-pass filter process to convert the pixel value of the pixel of the display image I to the average value of all the pixels of the normalized image I ′ corresponding to the pixel. , A weighted addition value or a median value is processed.

Hereinafter, the operation of the virtual camera photographing means 14 in FIG. 5 will be described with reference to FIG. 12 (see FIG. 5 as appropriate).
The virtual camera photographing unit 14 uses the normalized image pixel position calculating unit 141 to calculate the pixel position of the normalized image I ′ using the equations (3) and (4) (step S41).

  Further, the virtual camera photographing means 14 calculates the position of the pinhole H by using the formula (5) by the pinhole position calculating means 142 (step S42).

Further, the virtual camera photographing means 14 uses the virtual camera position / orientation calculating means 143 to obtain the normalized image I ′ from the pixel I ′ ₀ , ₀ (u, v) to the position of the pinhole H (0, 0). Generate a unit vector. Then, the virtual camera photographing unit 14 calculates the position and orientation of the virtual camera Vc based on the unit vector and the distance Length generated by the virtual camera position / orientation calculating unit 143 (step S43).

  In addition, the virtual camera photographing unit 14 uses the virtual photographed image generation unit 144 to project the display target onto the virtual plane based on the position and orientation of the virtual camera Vc, thereby capturing the virtual image obtained by photographing the display target with the virtual camera Vc. A captured image is generated (step S44).

As described above, the stereoscopic image generating apparatus 1 according to an embodiment of the present invention, the normalized image setting unit 12, the normalized image I pixel size becomes integral submultiple respect to the spacing element lenses L _P ' Prepare. Then, the stereoscopic image generating apparatus 1, by the virtual camera photographing means 14 performs the orthogonal projection such as the center of the element lenses L _P corresponds to a pixel of the virtual photographic image, the pixel values of the pixels of the virtual photographic image, was prepared Assigned to each pixel of the normalized image I ′. Thereby, a stereoscopic image generating apparatus 1, even if the distance between the element lenses L _P does not satisfy the condition that an integral multiple of the pixel size of the elemental image G, it is possible to generate a display image I by orthogonal projection, less the amount of calculation can do. More specifically, the stereoscopic image generating apparatus 1, as compared with the conventional technique of performing the projection conversion for all the pixels present in the image, can be reduced by the amount of computation the number of 1 / element lenses L _P .

Further, the stereoscopic image generating apparatus 1, the first filter 15, so reducing the aliasing noise due to sampling at the lens array L _A, it is possible to display image I high quality. At this time, the stereoscopic image generating apparatus 1 can set the filter application region by the filter parameter setting unit 13 without human intervention, and enables accurate low-pass filter processing. Then, the stereoscopic image generating device 1 reduces the aliasing noise caused by sampling on the image display panel 21 by the second filter 17a, so that the quality of the stereoscopic image can be improved.

  Here, even in the conventional technique for performing projective transformation (for example, Japanese Patent Application Laid-Open No. 2009-175866), the aliasing noise reduction filter is used. Since the calculation is performed, the calculation amount is increased. However, since the first filter 15 and the second filter 17a can be applied to orthographic projection in the stereoscopic image generating device 1, the amount of calculation required for reducing the aliasing noise is less than that of the conventional aliasing noise reduction filter. can do.

In the embodiment of the present invention, the stereoscopic image generating apparatus 1 has been described as including the first filter 15 and the second filter 17a. However, only one or both of them may be included. Good.
Here, the case of providing only the first filter 15, the stereoscopic image generating apparatus 1, since reducing the aliasing noise due to sampling at the lens array L _A, to obtain a display image I of practically sufficient quality it can. As described above, when only the first filter 15 is provided (that is, when the second filter 17 a is not provided), the stereoscopic image generating apparatus 1 displays the center pixel of the normalized image I ′ by the display image generating unit 17. A surface image may be generated by using pixels of the elementary image.
Further, when only the second filter 17a is provided, the stereoscopic image generating device 1 can reduce the aliasing noise caused by the sampling on the image display panel 21, and thus can obtain a display image I having a practically sufficient quality. .

  In the embodiment of the present invention, the example applied to the integral method has been described, but the present invention is not limited to this. For example, by setting the vertical direction of the display image I to one pixel, the present invention can also be applied to the lenticular method.

  In the present invention, when the ratio between the pixel size of the element image G in the normalized image I ′ and the pixel size of the image display panel 21 is not an integer, the second filter 17a performs an interpolation process to obtain a sampling position. The pixel value of the display image I according to the above may be calculated.

  In the embodiment of the present invention, the stereoscopic image generating apparatus according to the present invention has been described as an independent apparatus. However, in the present invention, a general computer is realized by a stereoscopic image generating program that functions as each of the above-described units. You can also. This stereoscopic image generation program may be distributed via a communication line, or may be distributed by writing in a recording medium such as a CD-ROM or a flash memory.

[Application to dynamic 3D shape model]
Hereinafter, a method of generating a stereoscopic image (integral stereoscopic video) of a dynamic three-dimensional shape model using the stereoscopic image generating device 1 will be described with reference to FIG.

As shown in FIG. 13, the stereoscopic video generation system S includes a three-dimensional modeling device 3 and a stereoscopic image generation device 1.
The three-dimensional modeling apparatus 3 arranges a plurality of cameras so as to surround a display target, and generates a dynamic three-dimensional shape model from the plurality of camera images. As the three-dimensional modeling apparatus 3, for example, a three-dimensional modeling apparatus disclosed in Japanese Patent No. 4014140 can be used.

  Accordingly, the stereoscopic video generation system S generates a dynamic three-dimensional shape model for each video frame from a plurality of camera images in the three-dimensional modeling device 3, and the stereoscopic image generation device 1 generates the dynamic three-dimensional shape model from the dynamic three-dimensional shape model. Generate an integral 3D image. Accordingly, the stereoscopic image generating apparatus 1 can generate a moving image (integral stereoscopic video) by generating an integral stereoscopic image for each video frame.

Hereinafter, as an example, the effect of reducing aliasing noise by the first filter 15 will be described with reference to FIG. 14 (see FIG. 2 as appropriate).
In this embodiment, the stereoscopic image generating device 1 in FIG. 2 is configured not to include the second filter 17a (hereinafter referred to as “stereoscopic image generating device 1B”), and the stereoscopic image generating device 1B displays the display image I. Generated. Then, to display the display image I on the stereoscopic display device 2, were taken stereoscopic image Z observed through the lens array L _A stereoscopic display device 2 with a digital camera (not shown). As a comparative example, to generate a stereoscopic image Z of the same display target in the conventional stereoscopic image generation apparatus, and a stereoscopic image Z observed through the lens array L _A of the stereoscopic display device 2 taken with a digital camera.

  The captured image in the comparative example is shown in FIG. 14A, and the enlarged image is shown in FIG. In the photographed image of FIG. 14A, spot-like noises with extremely different values are observed in a rough state due to the viewpoint movement. In addition, in the enlarged image of FIG. 14C, roughness was felt so as to correspond to the captured image of FIG.

  Further, a photographed image in the embodiment is shown in FIG. 14B, and an enlarged image thereof is shown in FIG. In the captured image of FIG. 14B, it was found that spot noise was reduced as compared to FIG. In addition, in the enlarged image of FIG. 14D, it was found that the high frequency (roughness) was removed by the low-pass filter processing as compared with FIG. 14C. From the above, it has been found that in the stereoscopic image generating device 1B, the display image I having a practically sufficient quality can be obtained by the first filter 15.

DESCRIPTION OF SYMBOLS 1 3D image production | generation apparatus 11 3D display apparatus modeling means (parameter input means)
12 Normalized image setting means (normalized image pixel size calculating means)
13 Filter parameter setting means (filter application area setting means)
14 Virtual camera photographing means 141 Normalized image pixel position calculating means 142 Pinhole position calculating means 143 Virtual camera position / orientation calculating means 144 Virtual photographed image generating means 15 First filter (first aliasing noise reducing means)
16 Normalized image generating means 17 Display image generating means 17a Second filter (second aliasing noise reducing means)
2 stereoscopic display device 21 image display panel (image display means)
L _A lens array _{L P} element lenses

Claims (4)

A 3D model as a display object to be displayed on a stereoscopic display device including a lens array in which element lenses are arranged two-dimensionally and an image display means is photographed by a virtual camera which is a virtual photographing camera, and the stereoscopic display device A stereoscopic image generating device for generating a display image for use,
Parameter input means for inputting a focal length of the element lens, an interval between the element lenses, and a pixel interval of the image display means;
The number of pixels between the element lenses is calculated based on a value obtained by dividing the element lens interval by the pixel interval of the image display means and multiplying by a preset constant, and is an integer with respect to the element lens interval. Normalized image pixel size calculation means for calculating a pixel size of the normalized image by dividing the interval between the element lenses by the number of pixels between the element lenses in a normalized image having a pixel size of 1 ,
A normalized image pixel position calculating unit that calculates pixel coordinates of the normalized image by dividing a pixel interval of the image display unit by a pixel size of the normalized image;
The position of the virtual camera is calculated based on a value obtained by multiplying the distance set in advance so as to indicate the position from the lens array to the position of the virtual camera and the pixel size of the normalized image and dividing by the focal length of the element lens. And calculating the reverse direction of the unit vector from the pixel position of the element image to the center position of the element lens in the normalized image as the shooting direction of the virtual camera, and using the shooting direction of the virtual camera, the virtual camera Virtual camera position / posture calculating means for calculating the posture of
Based on the position and orientation of the virtual camera calculated by the virtual camera position / orientation calculation means, the 3D shape model is orthogonally projected onto a virtual plane, thereby taking a virtual image of the 3D shape model captured by the virtual camera. Virtual captured image generation means for generating an image;
A normalized image generating means for generating the normalized image by assigning a pixel value of the pixel of the virtual photographed image to a pixel of the normalized image by a predetermined coordinate transformation formula;
Display image generating means for generating the display image from the normalized image generated by the normalized image generating means;
A stereoscopic image generating device comprising: In the virtual photographed image, a plurality of filter application regions surrounded by the midpoints of the line segments connecting the pixels corresponding to the center positions of the element lenses are set, and a low-pass filter process is performed for each filter application region. Aliasing noise reduction means,
The stereoscopic image generating apparatus according to claim 1, further comprising: Filter application area setting means for setting the filter application area in the virtual captured image using a matrix indicating the attitude of the virtual camera;
Stereoscopic image generating apparatus according to claim 2, further comprising a. A 3D model as a display object to be displayed on a stereoscopic display device including a lens array in which element lenses are arranged two-dimensionally and an image display means is photographed by a virtual camera which is a virtual photographing camera, and the stereoscopic display device Computer to generate display images for
Parameter input means for inputting a focal length of the element lens, an interval between the element lenses, and a pixel interval of the image display means,
The number of pixels between the element lenses is calculated based on a value obtained by dividing the element lens interval by the pixel interval of the image display means and multiplying by a preset constant, and is an integer with respect to the element lens interval. In a normalized image having a pixel size that is a fraction, a normalized image pixel size calculating unit that calculates the pixel size of the normalized image by dividing the interval between the element lenses by the number of pixels between the element lenses;
Normalized image pixel position calculating means for calculating pixel coordinates of the normalized image by dividing the pixel interval of the image display means by the pixel size of the normalized image;
The position of the virtual camera is calculated based on a value obtained by multiplying the distance set in advance so as to indicate the position from the lens array to the position of the virtual camera and the pixel size of the normalized image and dividing by the focal length of the element lens. And calculating the reverse direction of the unit vector from the pixel position of the element image to the center position of the element lens in the normalized image as the shooting direction of the virtual camera, and using the shooting direction of the virtual camera, the virtual camera Virtual camera position / posture calculating means for calculating the posture of
Based on the position and orientation of the virtual camera calculated by the virtual camera position / orientation calculation means, the 3D shape model is orthogonally projected onto a virtual plane, thereby taking a virtual image of the 3D shape model captured by the virtual camera. Virtual captured image generation means for generating an image,
A normalized image generating means for generating the normalized image by assigning a pixel value of the pixel of the virtual photographed image to a pixel of the normalized image by a predetermined coordinate transformation formula;
Display image generating means for generating the display image from the normalized image generated by the normalized image generating means;
A three-dimensional image generation program characterized by functioning as