JP2020028114A - Stereoscopic image generation device and program thereof - Google Patents

Abstract

To provide an element image generation device capable of efficiently calculating a cost volume.SOLUTION: In a stereoscopic image generation system 1, an element image generation device 3 includes a multi-view image input means 30, parameter setting means 31, sampling point setting means 32, cost volume calculation means 33 for calculating a cost volume by image-based rendering, cost volume interpolation means 34 for calculating the shift amount on a depth layer and interpolating the cost volume according to the calculated shift amount, light color information generation means 35 for generating light color information from a cost volume by image-based rendering, and element image generating means 36 for generating an element image from light ray color information.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a three-dimensional image generation device that generates a three-dimensional image by image-based rendering and a program therefor.

  Research and development of future three-dimensional image media such as three-dimensional television have been actively conducted. A plurality of methods for realizing this are known, and one of those methods is an integral three-dimensional method (Non-Patent Documents 1 and 2). The feature of the integral three-dimensional system is that special glasses are not required, and there is a vertical and horizontal motion parallax. The reason is that light rays from various directions can be acquired and displayed by using a lens array for photographing and displaying. Here, an image used for the display in the integral three-dimensional system is called an element image. That is, in the integral three-dimensional imaging, an element image may be generated.

  As an integral three-dimensional imaging technique, there are various techniques other than a technique using a lens array. For example, a method is known in which a three-dimensional model is constructed by a plurality of cameras and then rendered again to generate an element image (Non-Patent Document 3). There is also known a method of directly calculating light rays necessary for generating an element image without photographing with a plurality of cameras and constructing a three-dimensional model (Non-Patent Document 4).

  Consider the case where an element image is generated by the latter method. In this conventional method, light color information required for generating an element image can be generated by using a technique called image-based rendering (IBR) (Non-Patent Document 5). In the image-based rendering, an arbitrary viewpoint image is generated by photographing a multi-view image using a camera array in which cameras are arranged two-dimensionally and inserting light rays into the multi-view image.

  Hereinafter, the image-based rendering will be described. In order to generate an arbitrary viewpoint image, the ray color information may be known. Therefore, in order to obtain light ray color information, several depth layers are set in the photographing space, and it is estimated from which depth layer the light ray originated. For the estimation of the depth layer, a general cost such as a color consistency cost is used. The sum of the costs for all pixels is called a cost map (two-dimensional array of costs). Further, the cost map is extended in the depth direction, and a collection of the cost maps in all depth layers is called a cost volume (a three-dimensional array of costs). If this cost volume can be calculated, it is possible to estimate the depth layer in which each light ray has occurred from the cost volume, and calculate the light color information from the depth layer.

G. Lippmann, “Epreuves reversibles Photographies integrals,” Comptes-Rendus Academie des Sciences 146, 446-451 (1908). F. Okano, H. Hoshino, J. Arai, and I. Yuyama, “Real-time pickup method for a three-dimensional image based on integral photography,” Appl. Opt. 36 (7), 1598-1603 (1997) . Yuichi Iwadate, Miwa Katayama, "Integral stereoscopic image generation method by oblique projection", IEICE Technical Report, Vol.34, No.43, 3DIT2010-66, IDY2010-63, IST2010-52, 2010, p. 17-20 Y. Taguchi, T. Koike, K. Takahashi and T. Naemura, "TransCAIP: A live 3D TV system using a camera array and an integral photography display with interactive control of viewing parameters." IEEE Transactions on Visualization and Computer Graphics 15.5 ( (2009): 841-852. Yuichi Taguchi, Keita Takahashi, Ken Naemura, "Real-Time All-Focus Free Viewpoint Video Synthesis System Using Network Camera Array", IEICE Technical Report, vol. 107, no. 539, PRMU2007-258, pp. 79 -86 (2008)

However, in the conventional method, an enormous amount of light color information is required to generate an element image, and there is a problem that a calculation amount of a cost volume becomes enormous.
Furthermore, in the conventional method, there is a problem that depth estimation cannot be performed with high accuracy even if simple depth estimation processing is performed using a cost volume due to the influence of occlusion and the like. This is because the accuracy is low in a cost volume generated from one viewpoint.

Therefore, an object of the present invention is to provide a three-dimensional image generation device and a program for efficiently calculating a cost volume.
Still another object of the present invention is to provide a stereoscopic image generation device and a program for calculating a cost volume with high accuracy so that simple and accurate depth estimation processing can be performed.

  In view of the above-described problem, the stereoscopic image generation device according to the present invention represents light rays from a subject by image-based rendering using a multi-view image of a subject captured by a camera array in which two or more photographing cameras are arranged. A three-dimensional image generating apparatus that generates light information and generates a three-dimensional image from the light information, comprising: a parameter setting means, a cost volume calculating means, a cost volume interpolating means, a light information generating means, and a three-dimensional image generating means. Means.

According to such a configuration, the parameter setting unit is configured such that the optical element array provided in the stereoscopic display device for displaying the stereoscopic image is parallel to the virtual optical element array arranged in the imaging space where the subject is located, and at a predetermined interval in the depth direction. Are set in advance, a depth layer arranged in the shooting space, a viewpoint position, and an interpolation position between the viewpoint positions.
The cost volume calculation means converts the cost volume, which is a three-dimensional array of costs representing the similarity or dissimilarity between images when the sampling points on the depth layer are projected onto the multi-viewpoint image by image-based rendering, to the viewpoint position. It is calculated every time.
The cost volume interpolation unit calculates the shift amount between the viewpoint position and the interpolation position on the depth layer, and calculates the interpolation position from the cost volume of each viewpoint position calculated by the cost volume calculation unit according to the calculated shift amount. Interpolate the cost volume of
As described above, since the three-dimensional image generation device interpolates the cost volume according to the shift amount, the cost volume can be efficiently calculated.

The ray information generation unit estimates the depth layer in which the ray has occurred from the cost volume of the viewpoint position and the interpolation position by image-based rendering, and generates ray information in the estimated depth layer.
The three-dimensional image generating means generates a three-dimensional image from the light beam information.

Here, the stereoscopic image generation device may correct the calculated cost volume instead of newly interpolating the cost volume. That is, the stereoscopic image generation device may include a cost volume correction unit instead of the cost volume interpolation unit.
The cost volume correction unit calculates the shift amount between the viewpoint positions to be corrected on the depth layer, and corrects the cost volume to be corrected according to the calculated shift amount.
As described above, since the three-dimensional image generation device corrects the cost volume according to the shift amount, it is possible to calculate the cost volume with high accuracy.

  Note that the stereoscopic image generation device according to the present invention can also be realized by a program that causes a general computer to perform a cooperative operation as the above-described units.

According to the present invention, since the cost volume is interpolated according to the shift amount, the cost volume can be calculated efficiently, and the enormous amount of calculation required for calculating the cost volume can be reduced.
According to the present invention, since the cost volume is corrected according to the shift amount, a highly accurate cost volume can be calculated, a simple and accurate depth estimation process can be performed, and a high-quality stereoscopic image can be generated.

It is a block diagram showing composition of an element picture generation device concerning a 1st, 3rd embodiment. FIG. 3 is an explanatory diagram illustrating a shooting space in the first embodiment. In the first embodiment, (a) and (b) are explanatory diagrams illustrating a ray group necessary for calculating a cost volume. In the first embodiment, (a) and (b) are explanatory diagrams illustrating a light beam area. In the first embodiment, (a) is an explanatory diagram illustrating a depth layer, and (b) is an explanatory diagram illustrating a sampling point. In the first embodiment, (a) and (b) are explanatory diagrams illustrating the shift amount for each depth layer. FIG. 4 is an explanatory diagram illustrating calculation of a shift amount in the first embodiment. FIG. 4 is an explanatory diagram illustrating a shift direction in a depth layer in the first embodiment. FIG. 4 is an explanatory diagram illustrating a calculation formula described in MATLAB in the first embodiment. FIG. 4 is an explanatory diagram illustrating interpolation of a cost volume in the first embodiment. FIG. 4 is an explanatory diagram illustrating generation of an element image in the first embodiment. 3 is a flowchart illustrating an operation of the element image generation device of FIG. 1. It is a block diagram showing composition of an element image generation device concerning a 2nd, 4th embodiment. FIG. 13 is an explanatory diagram illustrating correction of a cost volume in the second embodiment. 14 is a flowchart showing the operation of the element image generation device of FIG. In the third embodiment, (a) and (b) are explanatory diagrams illustrating a light beam structure of a perspective projection lens shift. FIG. 13 is an explanatory diagram illustrating sampling points in a third embodiment. FIG. 13 is an explanatory diagram illustrating an interval between sampling points in a third embodiment. FIG. 13 is an explanatory diagram illustrating calculation of a shift amount in a third embodiment. FIG. 13 is an explanatory diagram illustrating calculation of a shift amount in a third embodiment. FIG. 14 is an explanatory diagram illustrating interpolation of a cost volume in the third embodiment.

  Hereinafter, each embodiment of the present invention will be described in detail with reference to the drawings as appropriate. In each embodiment, the same means is denoted by the same reference numeral, and the description is omitted.

(1st Embodiment)
[Overview of 3D image generation system]
An overview of a stereoscopic image generation system 1 according to the first embodiment will be described with reference to FIG.
The stereoscopic image generation system 1 generates an integral stereoscopic element image (stereoscopic image) by image-based rendering. As shown in FIG. 1, the stereoscopic image generation system 1 includes a camera array 2 and an element image generation device (stereoscopic image generation device) 3.

  Note that “image-based rendering” is to synthesize an image at an arbitrary viewpoint position based on a framework that describes visual information in a three-dimensional space as light rays using a multi-view image obtained by photographing a subject at a different viewpoint position. Technology. For example, image-based rendering is described in detail in Non-Patent Document 5 described above.

The camera array 2 is configured by arranging the photographing cameras 20 two-dimensionally. The camera array 2 generates a multi-viewpoint image in which a subject is photographed at different viewpoint positions, and outputs the generated multi-viewpoint image to the element image generation device 3. For example, the camera array 2 includes a total of 64 photographing cameras 20 arranged at equal intervals in the vertical and horizontal directions, and generates a multi-viewpoint image for 64 viewpoints. Further, the camera array 2 has the photographing camera 20 mounted on a frame (not shown) so that the photographing camera 20 is located on the same plane (the XY plane in FIG. 2). For example, a general network camera can be used as the photographing camera 20.
In the camera array 2, since the camera calibration may be performed, the photographing cameras 20 do not have to be strictly located on the same plane. Since the details of the camera array 2 are described in detail in Non-Patent Document 5 described above, further description is omitted.

[Configuration of Element Image Generation Device]
Hereinafter, the configuration of the element image generation device 3 will be described.
The element image generating device 3 generates light color information (light information) representing the color of the light from the subject by image-based rendering using the multi-view image from the camera array 2 and uses the element image from the light color information. Is generated. As shown in FIG. 1, the element image generating device 3 includes a multi-view image input unit 30, a parameter setting unit 31, a sampling point setting unit 32, a cost volume calculation unit 33, a cost volume interpolation unit 34, A light color information generating means (light information generating means) 35 and an element image generating means (stereoscopic image generating means) 36 are provided.

  The multi-viewpoint image input unit 30 receives a multi-viewpoint image from the camera array 2 and outputs the input multi-viewpoint image to the cost volume calculation unit 33.

  The parameter setting means 31 is for setting various parameters necessary for generating the element image in advance. For example, a user of the three-dimensional image generation system 1 sets various parameters in the parameter setting unit 31 via operation units such as a mouse and a keyboard (not shown). Then, the parameter setting means 31 outputs the set various parameters to the sampling point setting means 32.

  The various parameters include the position and pitch of a virtual lens array (virtual optical element array) described later, the number of virtual element lenses (virtual optical elements), the imaging space, the depth layer, the viewpoint position, the interpolation position, and the like. Is raised. Details of these parameters will be described later.

The sampling point setting means 32 is for presetting parallel sampling points on a depth layer in the photographing space. For example, the sampling point setting means 32 sets an intersection point between a plane of a depth layer described later and a ray vector (6 degrees of freedom in position and direction) as a sampling point. Then, the sampling point setting means 32 outputs the set sampling points and various parameters to the cost volume calculation means 33.
Note that the parallel projection is a projection in which light rays at each viewpoint position are parallel.

<Setting of parameters and sampling points>
The setting of parameters and sampling points will be described in detail with reference to FIGS. In FIG. 2, the horizontal direction is shown as an X axis, the vertical direction is shown as a Y axis, and the depth direction is shown as a Z axis.
In the present embodiment, as shown in FIG. 2, the three-body object 9 (9 1 _{to 9 3)} is to be disposed within the imaging space alpha. In the depth direction, the object 9 _to 93 ₃ are aligned from the front side of the imaging space alpha in order, subject 9 ₂ is positioned at the center of the imaging space alpha. Further, in the horizontal direction, the object ₉₁ to 93 ₃ are arranged offset to the left and right.

  Hereinafter, the “integral stereoscopic display device” is abbreviated as “IP stereoscopic display device”. This IP stereoscopic display device includes a lens array in which a predetermined number of element lenses are arranged two-dimensionally. Therefore, the virtual lens array L is set at a position in the photographing space α that will be the lens array surface of the integral stereoscopic display device.

The virtual lens array L is one in which the lens array of the IP stereoscopic display device is arranged in the photographing space α. That is, the virtual lens array L is a virtual one of the lens array of the IP stereoscopic display device in the photographing space α. This virtual lens array L includes a predetermined number of virtual element lenses 1 arranged two-dimensionally, like a real lens array. Here, the pitch (element spacing) between the virtual element lenses 1 is the pitch. The virtual lens array L is located at the center of the imaging space α in the depth direction, and is parallel to the depth layer D (FIG. 5) (located on the XY plane).
In consideration of the above, the parameter setting unit 31 may set the position and direction of the virtual lens array L, the pitch and the number of the virtual element lenses l.

The range in which the stereoscopic image is reproduced is determined by the extent to which the viewing range and the depth of the IP stereoscopic display device are reproduced, and the reproduction range is photographed by the camera array 2. Therefore, the imaging space α only needs to include the reproduction range of the stereoscopic image. In FIG. 2, the photographing space α is shown by hatching. In the depth direction, the imaging space alpha is around the virtual lens array L, in the range of from the front position alpha _F deep side position alpha _B. In the horizontal direction, the photographing space α is a range extending from the principal point of the outermost virtual element lens 1 by the viewing angle of the IP stereoscopic display device.
In consideration of the above, the parameter setting unit 31 may set the photographing space α.

In FIG. 3A, light rays necessary for calculating the cost volume in the parallel projection are shown by arrows. Ray groups of the same line type correspond to the same viewpoint when photographed by a parallel projection camera. That is, each of the solid-line, dashed-dotted line, and dashed-line ray groups corresponds to a different viewpoint position.
In consideration of the above, the parameter setting unit 31 may set a viewpoint position necessary for calculating the cost volume and an interpolation position which is a viewpoint position at which the cost volume is to be interpolated.

  FIG. 3B shows a light beam region formed by the light beam group of FIG. 3A, and FIG. 4A shows only the light beam region of FIG. . The “light ray region” is a region in the imaging space α where the light source group corresponding to the same viewpoint passes through the virtual lens array L. In the actual integral three-dimensional system, since the light beam group is denser, FIG. 4B shows a light beam region in FIG. In FIG. 4B, a dotted ray region is added between the dashed line and the dashed line in FIG. 4A, and a two-dot chain line is added between the dashed line and the solid line. .

As shown in FIG. 5 (a), it sets the depth layer _{D (D} 1 ~D ₅₎ in the imaging space alpha. This depth layer D is parallel to the virtual lens array L. The depth layer D is located at equal intervals in the imaging space α with respect to the virtual lens array L in the depth direction. For example, the depth layers D _{1 to} D ₅ are arranged at regular intervals from the near side to the far side of the imaging space α. Depth layer D ₁ overlaps the front side position alpha _F of the imaging space alpha. Depth Layer D ₃ is located in the center of the imaging space in the depth direction, it overlaps the virtual lens array L. Depth layer D ₅ overlaps the back side position alpha _B the imaging space alpha.
In consideration of the above, the parameter setting unit 31 may set the depth layer D. Note that the depth layers D do not necessarily have to be at equal intervals in the depth direction.

In parallel projection, a viewpoint, for example, a viewpoint position corresponding to a group of light rays indicated by solid lines in FIG. From this viewpoint, as shown in FIG. 5 (b), the intersection of the ray indicated by the solid line arrow and the depth layer D is the sampling point SP, and the cost described later is calculated around the sampling point SP. That is, the sampling point SP functions as a representative of the small area illustrated by dots.
In consideration of the above, the sampling point setting unit 32 may set the intersection of each ray and the depth layer D as the sampling point SP for each viewpoint position.

Returning to FIG. 1, the description of the configuration of the element image generation device 3 will be continued.
The cost volume calculation means 33 calculates a cost volume by image-based rendering for each viewpoint position. In the present embodiment, the cost volume calculation means 33 calculates a cost volume for the sampling point SP set by the sampling point setting means 32.

The “cost” represents the similarity (or dissimilarity) between the images when the sampling point SP on the depth layer D is projected on the multi-view image.
A two-dimensional array of costs in the horizontal and vertical directions is a “cost map”, and a three-dimensional array of costs in the horizontal, vertical, and depth directions is a “cost volume”.

Here, the cost volume calculation unit 33 can calculate the cost volume using general image-based rendering. For example, the cost volume calculation unit 33 obtains a color consistency cost from the sum of the variances of the RGB colors.
The cost volume calculation unit 33 outputs the calculated cost volume, the multi-viewpoint image, and various parameters to the cost volume interpolation unit 34.

The cost volume interpolation means 34 calculates the shift amount between the viewpoint position and the interpolation position on the depth layer D in the parallel projection. Then, the cost volume interpolation means 34 interpolates the cost volume at the interpolation position from the cost volumes already calculated at each viewpoint position according to the calculated shift amount. Thereafter, the cost volume interpolation means 34 outputs the cost volume at the viewpoint position and the interpolation position to the light color information generation means 35.
Note that “shift” is a displacement between the viewpoint position and the interpolation position on the depth layer D, and “shift amount” is the displacement amount.

<Interpolation of cost volume in parallel projection>
The interpolation of the cost volume in the parallel projection will be described with reference to FIGS.
The viewpoint position corresponding to the solid ray region is referred to as “viewpoint A”, the viewpoint position corresponding to the one-dot chain line ray region is referred to as “viewpoint C”, and the viewpoint position corresponding to the two-dot chain line ray region is referred to as “viewpoint B”. In FIG. 6A, reference numerals A to C are given to light beam regions corresponding to the viewpoints A to C, respectively. In addition, the cost volumes of the viewpoints A and C have already been calculated by the cost volume calculation unit 33, and the cost volumes of the viewpoint B are interpolated.

  When the depth layer D is set to a light-converging surface F (a surface in which light rays at each viewpoint are concentrated at one point), and the subject 9 (not shown in FIG. 6) is completely present only on the depth layer D, That is, consider a case where the subject 9 having no thickness is located on the depth layer D. In this case, the interpolation of the cost volume only needs to shift and combine the cost maps for each depth layer D. This calculation process is simple and requires a small amount of calculation. However, in practice, this is often not the case, so it is an approximate interpolation.

As shown in FIG. 6 (b), the depth layer _D 1 to D _5, the shift amount s is different. Since the depth Layer D ₃ located at the center of the imaging space α in the depth direction, the shift amount s is 0. The farther from the depth Layer D _3, the shift amount s is increased. That is, the shift amount s of the depth layers D ₁ and D ₅ is larger than the shift amount s of the depth layers D ₂ and D ₄ .

A specific method of calculating the shift amount s will be described with reference to FIG.
As shown in FIG. 7, in the virtual lens array L, virtual element lenses 1 are arranged at a pitch p. A lens array coordinate system is set on the basis of the lens array surface LP of the virtual lens array L, and the lens array coordinate system is used. Here, a ray vector V _A of the viewpoint A on the interpolating side and a ray vector V _B of the viewpoint B on the interpolating side are defined. The two light vectors V _A and V _B are normalized such that the component in the depth direction (Z-axis direction) becomes 1, and the difference vector V _{A−B in} the horizontal direction (X-axis direction) is defined as v. . At this time, the cost volume interpolation means 34 can calculate the shift amount s by the following equation (1). In Expression (1), the position of the depth layer D in the depth direction is d.

As shown in FIG. 8, the boundary of the depth layer D ₃ virtual lens array L is situated, the shift direction is reversed. Here, while shifting to the left in the depth layer D ₁ of the front side, shifted to the right in the depth layer D ₅ on the back side. In consideration of the reversal of the shift direction, the cost map of the viewpoint A is moved by the shift amount s for each depth layer D to fill the cost map of the viewpoint B. Since the cost map is a two-dimensional array, interpolation can be performed using the following equation (2) or equation (3). That is, the cost volume interpolation unit 34 can perform the interpolation of the cost volume by performing the calculation of the expression (2) or the expression (3) on each depth layer D.

In Equations (2) and (3), s _i = round (s), and round () is a rounded function.
In Expressions (2) and (3), for ease of explanation, the notation of MATLAB described in the following reference is used.
References: "MATLAB", [Search on June 15, 2018], Internet <URL: https://jp.mathworks.com/products/matlab.html>

Specifically, 'A' and 'B' represent the cost maps of viewpoint A and viewpoint B. In addition, “()” indicates an element of the cost map in the order of the vertical direction and the horizontal direction, and “:” indicates that the element is specified in a range. For example, “A (−, 1: U-s _i )” represents a range from 1 to U-s _i in the horizontal direction in the cost map of viewpoint A. Here, since only the horizontal direction is considered, the vertical direction is indicated by "-".

That is, Equations (2) and (3) are obtained by moving a fixed range (shown by hatching) in the cost map CM _A of the viewpoint A by the shift amount s as shown in FIG. _B represents interpolation. Actually, since the shift is necessary in the vertical direction, the cost volume interpolation means 34 also calculates the shift amount s in the vertical direction as in the horizontal direction.

Here, how to use the viewpoint C will be described. As shown in FIG. 6, in the depth layers D ₁ and D ₂ on the near side, the ray region of the viewpoint A is located on the right side of the ray region of the viewpoint B. On the other hand, in the depth layers D ₄ and D ₅ on the far side, the ray region of the viewpoint A is located on the left side of the ray region of the viewpoint B. As described above, a part of the light beam region of the viewpoint B deviates from the light beam region of the viewpoint A. That is, the cost volume of the viewpoint B has a range that cannot be interpolated only by the cost volume of the viewpoint A (illustrated by dots). Therefore, the cost volume interpolation means 34 similarly performs the interpolation of the cost volume by using the cost volume of the viewpoint C opposite to the viewpoint A when viewed from the viewpoint B.

  Note that, as shown in FIG. 6, there is a range where the ray regions of the viewpoints A and C overlap (shown by hatching). In this overlapping range, the cost volume interpolation means 34 can interpolate the cost volume of the viewpoint B using the plurality of cost volumes of the viewpoints A and C. In this overlapping range, interpolation is performed by an arbitrary method such as adopting the average cost value of the viewpoints A and C, adopting the lower cost in the viewpoints A and C, and performing weighting according to the shift amount s.

Thus, the cost volume interpolation unit 34, as shown in FIG. 10, according to the shift amount s, viewpoint A, C costs volume _CV A of the CV _C, it can be interpolated the cost volume CV _B viewpoint B . Although only three viewpoint positions A to C have been described here, the present invention can be extended to more viewpoint positions.

Returning to FIG. 1, the description of the configuration of the element image generation device 3 will be continued.
The light color information generation unit 35 generates light color information at the viewpoint position and the interpolation position from the cost volume input from the cost volume interpolation unit 34. As shown in FIG. 1, the light color information generation means 35 includes a depth layer estimation means 35A and a light color information acquisition means 35B.

  The depth layer estimating means 35A estimates the depth layer D where the light rays of the subject 9 are generated by the image-based rendering from the cost volume for each viewpoint position and interpolation position. For example, the depth layer estimating unit 35A searches for the minimum value of the cost included in the cost volume, and allocates the depth layer D having the minimum cost to each ray. In this way, an appropriate depth layer D is estimated for each ray.

The light color information acquiring means 35B acquires light color information on the depth layer D estimated by the depth layer estimating means 35A by image-based rendering. For example, the ray color information acquiring unit 35B acquires ray color information for each viewpoint position and interpolation position by bilinear interpolation of the color of the ray in the optimal depth layer D. In addition, the ray color information acquiring unit 35B may project the depth of the interpolation position onto a multi-viewpoint image and acquire ray color information of the depth. It should be noted that the “light color information” indicates the color of a light ray from a subject, and is equivalent to an arbitrary viewpoint image of the subject.
The ray color information acquiring unit 35B outputs the ray color information acquired at each viewpoint position to the element image generating unit 36 as an arbitrary viewpoint image at the viewpoint position and the interpolation position.

  The element image generation means 36 generates an element image from the light color information input from the light color information acquisition means 35B. For example, the element image generation unit 36 extracts pixels at the same position from an arbitrary viewpoint image at the viewpoint position and the interpolation position, and generates an element image composed of the extracted pixels. Then, the element image generating means 36 outputs the generated element image to the outside (for example, an IP stereoscopic display device).

<Generation of element image>
The generation of the element image E will be described with reference to FIG.
As shown in FIG. 11 (a), an image _{P 11} of the upper left perspective, right-side perspective image _{P 21} of the image _{P 11,} that the image _{P 12} of the lower viewpoint picture _{P 11,} consider the image P of the 3 perspectives. Also, order to simplify the description, the image P _11, denoted by "● 1" in the upper left pixel, labeled "● 2" to the right pixel, denoted by "● 3" in the lower pixel. Similarly, the image P ₂₁ are denoted by the "▲ 1" - "▲ 3", the image P _12, denoted by "■ 1" - "■ 3".

In this case, as shown in FIG. 11B, the element image generating means 36 extracts the upper left pixels ● 1, ▲ 1, ■ 1,... From the images P ₁₁ , P ₁₂ , P ₂₁ and outputs the upper left element image. to generate the E _11. In other words, the elemental image E ₁₁ is the top left pixel of the same position ● 1, ▲ 1, and is configured ■ 1, in .... Further, the elemental image _{E 11,} the upper left pixel ● 1, ▲ 1, ■ 1 , ... arrangement of the image _P 11 obtained by extracting them _pixels, corresponding to the viewpoint position of P _{12, P 21.} That is, in the element image E _11, pixel ● 1 is disposed on the upper left, pixel ▲ 1 is arranged on the right side, the pixel ■ 1 is disposed on the lower side.

Further, the element image generating means 36 generates an element image E ₂₁ composed of right pixels ● 2, ▲ 2, ■ 2,... Extracted from the images P ₁₁ , P ₁₂ , P ₂₁ similarly to the element image E _11. I do. Further, like the element images E ₁₁ and E ₂₁ , the element image generation unit 36 generates an element image composed of lower pixels ● 3, ▲ 3, ■ 3,... Extracted from the images P ₁₁ , P ₁₂ , P _21. to generate the E _12.

Here, the element image E is arranged at a position corresponding to the pixel position of each pixel constituting the element image E. For example, the element image E ₁₁ is a top left pixel ● 1, ▲ 1, ■ 1 , which is configured by ..., it is arranged in the upper left. Further, the elemental image E ₂₁ is right pixel ● 2, ▲ 2, ■ 2 , which is configured by ..., it is disposed on the right side of the element image E _11. Further, the elemental image E ₁₂ is lower pixel ● 3, ▲ 3, ■ 3 , since it is constituted ... in, it is arranged below the element image E _11.

  Note that, depending on the configuration of the optical system of the IP stereoscopic display device, an inverted stereoscopic image may be displayed. In this case, the element image generating means 36 may generate the element image E by inverting the position of each pixel vertically and horizontally so that an erect image of a stereoscopic image is displayed.

[Operation of Element Image Generation Device]
The operation of the element image generation device 3 will be described with reference to FIG.
As shown in FIG. 12, the parameter setting means 31 sets in advance various parameters necessary for generating an element image. These various parameters include the position and pitch of the virtual lens array, the number of virtual element lenses, the imaging space, the depth layer, and the viewpoint position (step S1).
The multi-view image input unit 30 receives a multi-view image from the camera array 2 (step S2).

The sampling point setting means 32 sets a sampling point on the depth layer in the photographing space in advance. For example, the sampling point setting means 32 sets an intersection between each ray and the depth layer as a sampling point for each viewpoint position (step S3).
The cost volume calculation means 33 calculates a cost volume by image-based rendering for each viewpoint position. For example, the cost volume calculation means 33 calculates a cost volume for the sampling points set in step S3 (step S4).

The cost volume interpolation means 34 calculates the shift amount on the depth layer by using the above-described equation (1). Then, the cost volume interpolation means 34 interpolates the cost volume at the interpolation position from the cost volume calculated in step S4 according to the calculated shift amount (step S5).
The depth layer estimation unit 35A estimates the depth layer where the light rays of the subject 9 have been generated by the image-based rendering from the cost volumes obtained in steps S4 and S5. For example, the depth layer estimating unit 35A searches for the minimum value of the cost included in the cost volume, and assigns the depth layer with the minimum cost to each ray (step S6).

The light color information acquiring unit 35B acquires light color information on the depth layer D estimated in step S6 by image-based rendering. For example, the light color information obtaining unit 35B obtains light color information by bilinear interpolation of light colors at the optimum depth. In addition, the ray color information acquiring unit 35B may project the depth of the interpolation position on a multi-viewpoint image to acquire ray color information of the depth (step S7).
The element image generation unit 36 generates an element image from the light ray color information acquired in step S7. For example, the element image generation unit 36 extracts pixels at the same position from the light ray color information (arbitrary viewpoint image), and generates an element image composed of the extracted pixels (step S8).

[Action / Effect]
As described above, the element image generating apparatus 3 according to the first embodiment interpolates the cost volume according to the shift amount in the light beam structure of the parallel projection, so that the cost volume is efficiently calculated and the cost volume is calculated. An enormous amount of calculation required for calculation can be reduced.

(2nd Embodiment)
[Configuration of Element Image Generation Device]
With reference to FIG. 13, a different point of the element image generation device 3B according to the second embodiment from the first embodiment will be described.
The element image generating apparatus 3B is different from the first embodiment in that the cost volumes of all viewpoint positions are calculated and the cost volumes are corrected. As shown in FIG. 13, the element image generating apparatus 3B includes a multi-view image input unit 30, a parameter setting unit 31B, a sampling point setting unit 32, a cost volume calculation unit 33, a cost volume correction unit 37, a light beam A color information generation unit 35 and an element image generation unit 36 are provided.
Except for the parameter setting unit 31B and the cost volume correction unit 37, the configuration is the same as that of the first embodiment, and the description is omitted.

  In the parameter setting means 31B, all viewpoint positions for calculating the cost volume are set in advance as parameters. That is, since the parameter setting means 31B does not perform the interpolation of the cost volume, it is not necessary to set the interpolation position as a parameter. In other respects, the parameter setting unit 31B is the same as that of the first embodiment, and further description will be omitted.

  The cost volume correction means 37 calculates the shift amount between the viewpoint positions to be corrected on the depth layer, and corrects the cost volume to be corrected according to the calculated shift amount. Then, the cost volume correction unit 37 outputs the corrected cost volume to the light color information generation unit 35.

<Cost volume correction>
The cost volume correction will be described with reference to FIG.
As described above, the ray regions at two adjacent viewpoint positions overlap (the hatched area in FIG. 6B), and the cost volume also overlaps in this overlapping range. Therefore, the cost volume is corrected within this overlapping range. . Here, it is assumed that the viewpoint positions to be corrected are viewpoints A and B adjacent to each other.

As shown in FIG. 14, between the viewpoint A, the cost volume _CV A of B, CV _B, shifted by the shift amount s. Therefore, similarly to the first embodiment, it can be defined difference vectors V _A-B between the light vectors V _B light vectors V _A and the viewpoint B of the viewpoint A. The pitch p and depth vector V _d virtual element lens can also be defined as in the first embodiment. Therefore, the cost volume correction unit 37 can calculate the shift amount s using the above-described equation (1).

Then, the cost volume correction section 37, according to the calculated shift amount s, the correction object viewpoint A, the cost volume _CV A of B, and corrects the CV _B. Here, the cost volume correction unit 37 calculates an average value of the costs in a range where the cost volumes of the viewpoints A and B overlap, and performs correction using the average value. At this time, if the costs differ from each other by a predetermined value or more in a range where the cost volumes overlap, the cost volume correcting unit 37 may perform the correction only on the cost. This value can be set arbitrarily, assuming that costs are greatly different from each other.

[Operation of Element Image Generation Device]
The operation of the element image generation device 3B will be described with reference to FIG.
Note that the processing of steps S2 to S4 and S6 to S8 is the same as that of the first embodiment, and thus the description is omitted.

  As shown in FIG. 15, various parameters required for generating an element image are set in the parameter setting unit 31B in advance. Here, since the parameter setting means 31B does not perform interpolation of the cost volume, there is no need to set the interpolation position as a parameter (step S1B).

  The cost volume correction unit 37 calculates the shift amount on the depth layer using the above-described equation (1). Then, the cost volume correction unit 37 corrects the cost volume calculated in step S4 according to the calculated shift amount. Here, the cost volume correcting unit 37 calculates an average value of the costs in a range where the cost volumes overlap, and performs correction using the average value. At this time, if the costs differ from each other by a predetermined value or more in a range where the cost volumes overlap, the cost volume correcting unit 37 may perform the correction only on the cost (step S5B).

[Action / Effect]
As described above, since the element image generating apparatus 3B according to the second embodiment corrects the cost volume according to the shift amount in the parallel projection light beam structure, it calculates a high-precision cost volume and outputs a high-quality element. Images can be generated.

(Third embodiment)
Hereinafter, as a premise for describing the third embodiment, a light beam structure of the light beam reproducing type display system will be described.
The light beam structure of the light beam reproducing type display system can be roughly classified into the parallel projection (FIG. 3) described in the first and second embodiments and the perspective projection lens shift (FIG. 16) described in the third embodiment. In the parallel projection, all rays are parallel as shown in FIG. On the other hand, in the perspective projection lens shift, as shown in FIG. 16A, all light rays are condensed at a focal point T at a certain distance in the depth direction.

  Note that, in FIG. 16A, the light beam group in the perspective projection lens shift is illustrated by arrows. Ray groups of the same line type correspond to the same viewpoint when photographed by the oblique projection camera. That is, each of the solid-line, dashed-dotted line, and dashed-line ray groups corresponds to a different viewpoint position. FIG. 16B illustrates a light beam region formed by the light beam group of FIG. 16A.

  Here, as shown in FIG. 17, the intersection of the depth layer D and each light ray is set as a sampling point SP. Then, as shown in FIG. 18, in the perspective projection lens shift, if the depth layer D is the same, the interval between the sampling points SP is the same regardless of the viewpoint position. That is, similar to the parallel projection, the perspective projection lens shift is characterized in that the shift amount is the same if the depth layer D is the same. By utilizing this feature and calculating the shift amount for each depth layer D and interpolating the cost volume, the cost volume can be calculated efficiently.

[Configuration of Element Image Generation Device]
With reference to FIG. 1, a configuration of an element image generating device 3C according to the third embodiment will be described, focusing on differences from the first embodiment.
As shown in FIG. 1, the element image generating apparatus 3C includes a multi-view image input unit 30, a parameter setting unit 31, a sampling point setting unit 32C, a cost volume calculation unit 33, a cost volume interpolation unit 34C, A light color information generator 35 and an element image generator 36 are provided.
The configuration other than the sampling point setting unit 32C and the cost volume interpolation unit 34C is the same as that of the first embodiment, and a description thereof will be omitted.

  The sampling point setting means 32C pre-sets a sampling point SP of a perspective projection lens shift at which a light beam at each viewpoint position converges on one converging point T on a depth layer in the photographing space.

In the perspective projection lens shift, a ray group at a certain viewpoint is considered. From this viewpoint, as shown in FIG. 17, the intersection of the light ray and the depth layer D is the sampling point SP, and the cost is calculated around the sampling point SP. That is, the sampling point SP functions as a representative of the small area illustrated by dots.
In consideration of the above, the sampling point setting unit 32C may set the intersection between each ray and the depth layer D as the sampling point SP for each viewpoint position.

  The cost volume interpolation means 34C calculates the shift amount between the viewpoint position and the interpolation position on the depth layer D. In the present embodiment, the cost volume interpolation means 34C calculates the shift amount s in the perspective projection lens shift using Expressions (4) and (5) described later. In other respects, the cost volume interpolation means 34C is the same as in the first embodiment.

<Interpolation of cost volume in perspective projection lens shift>
The interpolation of the cost volume in the perspective projection lens shift will be described with reference to FIGS.
A lens array coordinate system is set on the basis of the lens array surface LP of the virtual lens array L, and the lens array coordinate system is used. As shown in FIG. 19, the position of the converging point T with respect to the virtual lens array L in the depth direction is W, and the position of the depth layer D with respect to the lens array plane LP in the depth direction is d. Here, two light beams passing through the adjacent virtual element lens 1 and converging at the same light condensing point T will be considered. Interval P _P of the two light beams is greater the closer the focal point T on lens array surface LP, it is expressed by the following equation (4).

Here, the lens pitch on the lens array plane LP is p. Furthermore, as shown in FIG. 20 think viewpoint intersects on the lens array plane LP A, ray vector _V A of B, and _{V B.} In Figure 20, the focal point _T A, _{T B} is the viewpoint A, B each focusing point T. Two light vectors _V A, normalized so Z-axis component of _{V B} becomes 1, the component in the horizontal direction (X axis direction) of the difference vector _{V A-B} v, the difference vector _{V A-B} Is | v |. In this case, the shift amount s of each depth layer D is represented by the following equation (5).

As shown in FIG. 21, the cost volume interpolation unit 34C performs the calculations of Expressions (4) and (5) on each of the depth layers D _{1 to} D ₅ to interpolate the cost volume. For example, it is assumed that the cost volume is known at the viewpoint positions corresponding to the ray groups of the solid line and the broken line. In this case, the cost volume interpolation means 34C can interpolate the cost volume at the viewpoint position corresponding to the dashed-dotted ray group from the known cost volumes for the two viewpoints.

[Action / Effect]
As described above, since the element image generating apparatus 3C according to the third embodiment interpolates the cost volume according to the shift amount in the light beam structure of the perspective projection lens shift, it efficiently calculates the cost volume, and An enormous amount of calculation required for calculating the volume can be reduced.

(Fourth embodiment)
[Configuration of Element Image Generation Device]
With reference to FIG. 13, a difference of the configuration of the element image generation device 3D according to the third embodiment from the second embodiment will be described.
The element image generating apparatus 3D differs from the second embodiment in that the light beam structure is a perspective projection lens shift.

As shown in FIG. 13, the elementary image generation device 3D includes a multi-view image input unit 30, a parameter setting unit 31B, a sampling point setting unit 32D, a cost volume calculation unit 33, a cost volume correction unit 37D, a light beam A color information generation unit 35 and an element image generation unit 36 are provided.
Note that the configuration other than the sampling point setting unit 32D and the cost volume correction unit 37D is the same as that of the second embodiment, and thus the description is omitted.

  In the sampling point setting means 32D, similarly to the sampling point setting means 32C of FIG. 1, the sampling point SP in the perspective projection lens shift is set in advance. Since the sampling point setting means 32D is the same as in the third embodiment, further description is omitted.

  The cost volume correction unit 37D calculates the shift amount between the viewpoint positions to be corrected on the depth layer D in the perspective projection lens shift. In the present embodiment, the cost volume correction unit 37D calculates the shift amount s in the perspective projection lens shift using the above-described equations (4) and (5), similarly to the cost volume interpolation unit 34C of FIG. . In other respects, the cost volume correction unit 37D is the same as that of the second embodiment, and thus further description is omitted.

  As described above, the element image generating apparatus 3D according to the fourth embodiment corrects the cost volume according to the shift amount in the light beam structure of the perspective projection lens shift. Element images can be generated.

(Modification)
As described above, the embodiments of the present invention have been described in detail. However, the present invention is not limited to each of the above-described embodiments, and includes a design change or the like without departing from the gist of the present invention.
In each of the embodiments described above, the number of depth layers is five, but the number of depth layers is not particularly limited.
In each of the embodiments described above, the shift amount in the horizontal direction has been described. However, the shift amount can be similarly calculated in the vertical direction.

  As shown in FIG. 3A, in the integral three-dimensional system, there is a light-condensing surface F on which light beams converge at predetermined intervals in front and rear from the virtual lens array L in the depth direction. FIG. 3A illustrates the position of the light collecting surface F in the depth direction. On this light condensing surface F, the solid line, the one-dot chain line and the broken line intersect and converge. Therefore, since the parameter setting means 31 can perform more accurate interpolation, it is preferable to set the depth layer D on the light-collecting surface F.

  In each of the embodiments described above, an example in which the present invention is applied to the integral three-dimensional system has been described, but the present invention is not limited to this. For example, the present invention can be applied to a three-dimensional system in which one viewpoint is a light beam of oblique projection (for example, a parallax barrier three-dimensional system).

  In the embodiment described above, the element image generation device is described as independent hardware, but the present invention is not limited to this. For example, the present invention can also be realized by a program that causes hardware resources such as a CPU, a memory, and a hard disk of a computer to cooperate as the above-described element image generation device. These programs may be distributed via a communication line, or may be distributed by writing to a recording medium such as a CD-ROM or a flash memory.

1, 1B-1D stereoscopic image generation system 2 Camera array 3, 3B-3D element image generation device (stereoscopic image generation device)
Reference Signs List 20 photographing camera 30 multi-viewpoint image input means 31, 31B parameter setting means 32, 32C, 32D sampling point setting means 33 cost volume calculation means 34, 34C cost volume interpolation means 35 light color information generation means (light information generation means)
35A Depth layer estimating means 35B Ray color information acquiring means 36 Element image generating means (stereoscopic image generating means)
37,37D cost volume correcting means _9,9 to 93 ₃ subjects A, B, C light region _CM A, CM _B cost map _{CV A, CV B, CV C} costs volume _{D, D} 1 ~D ₅ depth layer E, E ₁₁ , E ₁₂ , E ₂₁ Element image L Virtual lens array (virtual optical element array)
LP Lens array surface l Virtual element lens (virtual optical element)
_{P, P 11, P 12,} P 21 image SP sampling points _{T, T A, T B} focal point _V A, _{V B} light vector alpha imaging space alpha _B far side position alpha _F front position

Claims (9)

Using a multi-view image obtained by photographing a subject with a camera array in which photographing cameras are arranged two-dimensionally, ray information representing rays from the subject is generated by image-based rendering, and a stereoscopic image is generated from the ray information. An image generation device,
An optical element array included in a stereoscopic display device that displays the stereoscopic image is parallel to a virtual optical element array arranged in an imaging space where the subject is located, and in the imaging space so as to have a predetermined interval in a depth direction. A parameter setting unit in which an arranged depth layer, a viewpoint position, and an interpolation position between the viewpoint positions are set in advance;
By the image-based rendering, a cost volume that is a three-dimensional array of costs representing similarity or dissimilarity between images when the sampling points on the depth layer are projected onto the multi-viewpoint image, for each viewpoint position. Cost volume calculating means for calculating;
The shift amount between the viewpoint position and the interpolation position is calculated on the depth layer, and the interpolation position of the interpolation position is calculated from the cost volume of each viewpoint position calculated by the cost volume calculation unit according to the calculated shift amount. Cost volume interpolation means for interpolating the cost volume,
By the image-based rendering, a ray information generating unit that estimates a depth layer in which the light ray has occurred from the cost volume of the viewpoint position and the interpolation position, and generates the ray information in the estimated depth layer,
Stereoscopic image generating means for generating the stereoscopic image from the light beam information,
A stereoscopic image generation device comprising: On the depth layer, further includes a sampling point setting unit in which a sampling point of parallel projection in which light rays for each of the viewpoint positions are parallel is set in advance,
The parameter setting unit further sets a position and an element interval of the virtual optical element array, and sets the depth layer in a depth direction based on a position of the virtual optical element array,
The cost volume interpolation means includes a depth vector element d from the virtual optical element array to the depth layer, a difference vector element v between the viewpoint position ray vector and the interpolation position ray vector, and Using the following equation (1) including the element interval p of the virtual optical element array,

The three-dimensional image generation device according to claim 1, wherein the shift amount s is calculated. On the depth layer, further includes a sampling point setting unit in which a sampling point of a perspective projection lens shift in which the light beam for each of the viewpoint positions converges to one converging point,
The parameter setting unit further sets a position and an element interval of the virtual optical element array, and sets the depth layer in a depth direction based on a position of the virtual optical element array,
The cost volume interpolation means,
Includes an element interval p of the virtual optical element array, an element d of a depth vector from the virtual optical element array to the depth layer, and an element W of a vector indicating the position of the converging point with respect to the virtual optical element array. Using equation (4)

Calculating the distance P _P sampling points each other and the interpolation position and the view point position,
Using an element v of the difference vector of the ray vector of the interpolation position the ray vector of the viewpoint position, and the element d, the equation that contains the interval P _P (5),

The three-dimensional image generation device according to claim 1, wherein the shift amount s is calculated. Using a multi-view image obtained by photographing a subject with a camera array in which photographing cameras are arranged two-dimensionally, ray information representing rays from the subject is generated by image-based rendering, and a stereoscopic image is generated from the ray information. An image generation device,
An optical element array included in a stereoscopic display device that displays the stereoscopic image is parallel to a virtual optical element array arranged in an imaging space where the subject is located, and in the imaging space so as to have a predetermined interval in a depth direction. Parameter setting means for setting the arranged depth layer and the viewpoint position in advance;
By the image-based rendering, a cost volume that is a three-dimensional array of costs representing similarity or dissimilarity between images when the sampling points on the depth layer are projected onto the multi-viewpoint image, for each viewpoint position. Cost volume calculating means for calculating;
Cost volume correction means for calculating the shift amount between the viewpoint positions to be corrected on the depth layer, and correcting the cost volume to be corrected according to the calculated shift amount,
The image-based rendering, ray information generating means for estimating a depth layer in which the light ray has occurred from the cost volume corrected by the cost volume correcting means, and generating the ray information in the estimated depth layer,
Stereoscopic image generating means for generating the stereoscopic image from the light beam information,
A stereoscopic image generation device comprising: On the depth layer, further includes a sampling point setting unit in which a sampling point of parallel projection in which light rays for each of the viewpoint positions are parallel is set in advance,
The parameter setting unit further sets a position and an element interval of the virtual optical element array, and sets the depth layer in a depth direction based on a position of the virtual optical element array,
The cost volume correction unit includes a depth vector element d from the virtual optical element array to the depth layer, a light vector difference vector element v between the correction target viewpoint positions, and a virtual optical element array element. Using the following equation (1) including the interval p,

The three-dimensional image generation device according to claim 4, wherein the shift amount s is calculated. On the depth layer, further includes a sampling point setting unit in which a sampling point of a perspective projection lens shift in which the light beam for each of the viewpoint positions converges to one converging point,
The parameter setting unit further sets a position and an element interval of the virtual optical element array, and sets the depth layer in a depth direction based on a position of the virtual optical element array,
The cost volume correction means,
Includes an element interval p of the virtual optical element array, an element d of a depth vector from the virtual optical element array to the depth layer, and an element W of a vector indicating the position of the converging point with respect to the virtual optical element array. Using equation (4)

Calculating the distance P _P of sampling points at the viewpoint position between the correction target,
Above using an element v of the difference vector of the ray vector at the correction target viewpoint position between, said element d, the equation that contains the interval P _P (5),

The three-dimensional image generation device according to claim 4, wherein the shift amount s is calculated.   7. The cost volume correction unit according to claim 4, wherein, when the costs included in the cost volume are different from each other by a predetermined value or more, the cost volume correction unit corrects the cost. 8. The stereoscopic image generation device according to claim 1. The stereoscopic image is an integral stereoscopic element image,
The said three-dimensional image production | generation means extracts the pixel of the same position from each ray information, and produces | generates the said element image comprised by the extracted said pixel, The one of Claim 1 to 7 characterized by the above-mentioned. 3. The stereoscopic image generation device according to claim 1.   A program for causing a computer to function as the three-dimensional image generation device according to claim 1.

Images