JP2017228152A - Depth map generation device, depth map generation method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a depth map generation device configured to generate a parameter for generating a depth map at high speed and with high accuracy.SOLUTION: A depth map generation device comprises: an initialization part for initializing parameters of depth and normal information in each of viewpoint images at all viewpoints by generating a random number for each pixel; a space propagation processing part for propagating the parameters correspondingly to a predetermined first evaluation value between adjacent pixels in a target viewpoint image; a viewpoint propagation processing part for propagating the parameters correspondingly to a predetermined second evaluation value between pixels of the target viewpoint image and a nearby viewpoint image; and a depth map generation part for generating a depth map from the parameters of the viewpoint images. The space propagation processing part performs processing for propagating parameters from each end pixel of a pixel to which a parameter is propagated to a non-end pixel adjacent to the end pixel in parallel, and the viewpoint propagation processing part performs processing for propagating the parameter of each end pixel to a non-end pixel of the nearby viewpoint image in parallel.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a depth map generation device, a depth map generation method, and a program.

  3D restoration technology based on multi-viewpoint images is attracting attention not only in the computer vision research community, but also in a wide range of fields such as digital archives of cultural assets and the entertainment industry (see Patent Document 1).

JP 2013-019801 A

The three-dimensional restoration technique based on the above-described multi-viewpoint image uses a triangulation technique, and finally generates a three-dimensional coordinate point group by integrating depth maps at the respective viewpoints.
For this reason, when restoring the three-dimensional shape of the target object in the world coordinate system using the images (viewpoint images) at each viewpoint, the coordinate points of the three-dimensional coordinate points in the world coordinate system between two or more different viewpoint images It is necessary to detect the corresponding pixel of each viewpoint image corresponding to, and generate a depth map from each viewpoint.
Conventionally, a first straight line (epipolar straight line) is generated from the viewpoint P1 to the viewpoint image, and a second straight line that intersects the first straight line is generated from a viewpoint P1 different from the viewpoint P1.

  Here, it is assumed that the tolerance coordinate point at which the second straight line intersects the first straight line is sequentially changed, and it is assumed that the three-dimensional coordinate point of the target object exists at the crossed coordinate point. Projecting to each of the image and the viewpoint image of the viewpoint P2. Then, using the luminance value of the pixel at the position projected on each viewpoint image, the luminance value of each pixel between the viewpoint images is correlated, and the tolerance coordinate point having the highest correlation value is extracted as a three-dimensional coordinate point. The depth map is generated.

  However, in order to extract the intersection coordinate point corresponding to the three-dimensional coordinate point, it is necessary to obtain the correlation value between the viewpoint images every time the tolerance coordinate point is changed in fine increments in pixel units. For this reason, enormous calculation is required to generate the depth map for restoring the target object, and the calculation cost becomes very high. As a result, it takes a long time to restore the target object in the world coordinate system.

  The present invention has been made in view of such a situation, and a depth map that generates parameters (depth, inclination of normal line) for generating a depth map used for restoration of a three-dimensional coordinate point group at high speed and with high accuracy. A generation apparatus, a depth map generation method, and a program are provided.

  In order to solve the above-described problem, the depth map generation apparatus of the present invention is a depth map generation apparatus that generates a depth map used for three-dimensional image restoration using the patch match stereo method. Between the adjacent pixels in the target viewpoint image, which is the target viewpoint image, and an initialization unit that generates and initializes parameters including depth information and normal line information for each pixel with random numbers in each viewpoint image of the viewpoint The parameter is transmitted between pixels of a spatial propagation processing unit that propagates a parameter corresponding to a predetermined first evaluation value, and a target viewpoint image and a near viewpoint image that is an image of a near viewpoint that is a viewpoint near the target viewpoint. A viewpoint propagation processing unit that propagates corresponding to a predetermined second evaluation value, and the parameters of a plurality of different viewpoint images. A depth map generation unit that generates a depth map for each image, and the spatial propagation processing unit has not completed propagation adjacent to each of the end pixels from each of the end pixels that are the pixels to which the parameter has been propagated. Processing for propagating parameters to unfinished pixels is performed in parallel, and the viewpoint propagation processing unit performs processing for propagating the parameters of each of the ending pixels to unfinished pixels of the neighboring viewpoint image in parallel. It is characterized by performing.

  In the depth map generation device of the present invention, the spatial propagation processing unit repeatedly performs parameter propagation processing a predetermined number of times, and when the spatial propagation processing unit performs the propagation process an odd number of times, If the parameter is propagated from the pixel of the first vertex that is a predetermined vertex in the rectangular shape of the target viewpoint image, and the number of times of performing the propagation processing is an even number, the first vertex of the target viewpoint image A process for propagating a parameter from a pixel at the second vertex at a point-symmetrical position is performed.

  The depth map generation device according to the present invention generates a fine adjustment of a parameter transmitted to a pixel by random numbers for a pixel for which parameter propagation processing has been completed in each of the spatial propagation processing unit and the viewpoint propagation processing unit. Further, a parameter fine adjustment unit that performs the adjustment value is provided.

  The depth map generation apparatus of the present invention is characterized in that the spatial propagation processing unit performs parameter propagation processing on the plurality of target viewpoint images in parallel.

  The depth map generation method of the present invention is a depth map generation method for generating a depth map used for three-dimensional image restoration using a patch match stereo method, and the initialization unit uses all viewpoints used for generation of a depth map. An initialization process for generating and initializing parameters including depth information and normal information in each image by random numbers for each pixel, and a spatial propagation processing unit between adjacent pixels in the target viewpoint image that is the target viewpoint image A spatial propagation process for propagating the parameter in correspondence with a predetermined first evaluation value, and a near-point viewpoint image in which the viewpoint propagation processing unit is an image of a near-point viewpoint that is a viewpoint near the target viewpoint image and the target viewpoint. A viewpoint propagation process for propagating the parameter between pixels corresponding to a predetermined second evaluation value, and a depth map generator, A depth map generation process for generating a depth map for each viewpoint image from the parameters of the viewpoint image, wherein the spatial propagation processing unit performs the end from each of the end pixels that are the pixels to which the parameter has been propagated. A process of propagating parameters to unfinished pixels that have not been propagated adjacent to each pixel is performed in parallel, and the viewpoint propagation processing unit sets the parameters of each of the finished pixels to the unfinished pixels of the neighboring viewpoint image. The process of propagating the end pixel is performed in parallel.

  The program of the present invention is a program that causes a computer to execute processing of a depth map generation device that creates a depth map used for three-dimensional image restoration using the patch match stereo method. Between the adjacent pixels in the target viewpoint image that is an image of the target viewpoint, and an initialization unit that generates and initializes a parameter including depth information and normal line information for each pixel with random numbers in each viewpoint image of the viewpoint Spatial propagation processing means for propagating a parameter corresponding to a predetermined first evaluation value, and the parameter between pixels of a target viewpoint image and a near viewpoint image that is a near viewpoint image that is a viewpoint near the target viewpoint. , Viewpoint propagation processing means for propagating corresponding to a predetermined second evaluation value, and a plurality of different viewpoint images It is operated as a depth map generating means for generating a depth map for each viewpoint image from the parameters, and from each of the end pixels that are the pixels to which the parameter has been propagated, the propagation adjacent to each of the end pixels has not been completed. Processing for propagating parameters to end pixels is performed in parallel, and the viewpoint propagation processing means performs processing for propagating the parameters of each of the end pixels to unfinished pixels of the neighboring viewpoint image. This is a program to be executed in parallel by the processing means.

  As described above, according to the present invention, depth map generation capable of generating parameters (depth, inclination of normal line) for generating a depth map used for restoration of a three-dimensional coordinate point group at high speed and with high accuracy. An apparatus, a depth map generation method, and a program can be provided.

It is a figure which shows the structural example of the depth map production | generation apparatus by embodiment of this invention. It is a figure explaining the setting of the range which generates the random number used as a parameter of the three-dimensional coordinate point in the world coordinate system corresponding to the pixel of a viewpoint image. It is a figure which shows the structural example of the parameter table memorize | stored in the memory | storage part 19 in this embodiment. It is a figure which shows a response | compatibility with the object viewpoint image and the vicinity viewpoint image in each of space propagation processing and viewpoint propagation processing. It is a conceptual diagram which shows the arrangement | positioning relationship between the reference pixel and object pixel in a spatial propagation process. It is a figure explaining the thread | sled of the spatial propagation process of each pixel in the target visual line image Vk in FIG. It is a flowchart showing the operation | movement procedure of the parameter propagation process of a depth map production | generation apparatus. It is a figure which shows the position of each viewpoint of the imaging device which imaged the three-dimensional-shaped target object. As an example, a graph configuration is shown in the case where there are two neighboring viewpoint images in the viewpoint propagation processing. It is a figure which shows the start viewpoint image of the thread | sled in case the number of the near viewpoint images which propagate a parameter is two, and the number of threads is two. It is a figure which shows the start viewpoint image of the thread | sled in case the number of the near viewpoint images which propagate a parameter is two, and the number of threads is three.

  An embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram illustrating a configuration example of a depth map generation device according to an embodiment of the present invention. 1 includes a control unit 11, an initialization unit 12, a viewpoint selection unit 13, a thread generation unit 14, a spatial propagation processing unit 15, a viewpoint propagation processing unit 16, a fine adjustment unit 17, and a depth map generation unit. 18 and storage unit 19 are provided. Further, in the present embodiment, the parameters in the viewpoint image are estimated by the patch match stereo (PatchMatch Stereo) method.

The control unit 11 performs control of each unit of the depth map generation device and data transmission / reception processing with an external device.
The initialization unit 12 includes the depth (depth information) of the coordinate point in the world coordinate system corresponding to each pixel of the viewpoint image captured from all viewpoints, and the normal vector (normal information, normal inclination θVk (m described later). ) And a vector indicated by φVk (m)) are initialized with random numbers.
The viewpoint selection unit 13 selects a viewpoint image of a viewpoint to be processed in the parameter propagation process as a target viewpoint image.
The thread generation unit 14 generates a thread for performing parameter propagation processing in the row direction of pixels in the target viewpoint image Vk corresponding to each row.

For each thread corresponding to each row, the spatial propagation processing unit 15 performs parameter propagation processing from the adjacent reference pixel for which spatial propagation processing has been completed for the target pixel in each row.
The viewpoint propagation processing unit 16 selects a near viewpoint image (as a stereo pair for the target viewpoint) corresponding to the target pixel, which is another viewpoint image in the vicinity of the target viewpoint image, from the target pixel for which the spatial propagation processing in the target viewpoint image has been completed. The parameter propagation process is performed for the pixel of the selected viewpoint image.
The fine adjustment unit 17 finely adjusts the parameter of the target pixel for which the spatial propagation process has been completed, using an adjustment value generated by a random number.
The depth map generation unit 18 generates a depth map in each viewpoint image after the spatial propagation processing for each pixel by the spatial propagation processing unit 15 and the viewpoint propagation processing of the viewpoint propagation processing unit 16 are completed.
The storage unit 19 stores viewpoint images V1, viewpoint images V2, viewpoint images V3, ... included in the viewpoint image group V = {V1, V2, V3, ...}, and also stores a parameter table (described later). ing.

<Initialization process>
Hereinafter, an initialization process for pixels of each viewpoint image performed by the initialization unit 12 will be described. In this initialization process, the initialization unit 12 uses the stereo of the target viewpoint image used when performing pixel matching on the viewpoint image selected as the target viewpoint image (the viewpoint image selected from the viewpoint image group V). A near viewpoint image (a viewpoint image near the target viewpoint image Vk from the viewpoint image group V) that is a viewpoint image of a viewpoint near the target viewpoint to be paired is selected.

  When all of the viewpoint images included in the viewpoint image group V are set as the target viewpoint image Vk, a neighboring viewpoint image CVk that is a stereo pair with the target viewpoint image Vk is set. Here, when selecting the near viewpoint image CVk to be selected as a stereo pair, if the selected near viewpoint image CVk is separated from the target viewpoint image Vk by a predetermined parallax, the target viewpoint image Vk and the near viewpoint image CVk are Image deformation between the viewpoint images increases, and the accuracy of pixel matching between the target viewpoint image Vk and the neighboring viewpoint image CVk is reduced and becomes unstable. For this reason, a plurality of viewpoint images are selected in order from the shortest baseline length between the viewpoints with respect to the target viewpoint image Vk, and, for example, Npair of neighboring viewpoint images CVk are selected in order of the shortest baseline length from the selected viewpoint image. Make a selection.

  Then, for all viewpoint images included in the viewpoint image group V, the initialization unit 12 sets 3 in the world coordinate system corresponding to each pixel of image coordinates m = (u, v), which is a coordinate value in the viewpoint image. The parameters of the dimensional coordinate points, that is, the depth dVk (m) and the normal gradients θVk (m) and φVk (m) are initialized using uniform random numbers. A range for generating random numbers as initial values of the depth dVk (m) and the normal gradients θVk (m) and φVk (m) can be freely set according to each viewpoint image. In the present embodiment, random numbers are generated according to the setting range described below.

FIG. 2 is a diagram for explaining setting of a range for generating a random number used as a parameter of a three-dimensional coordinate point in the world coordinate system corresponding to the pixel of the viewpoint image. FIG. 2 illustrates a case where each of the vicinity viewpoint image C1 and the vicinity viewpoint image C2 is selected as the vicinity viewpoint image CVk for the target viewpoint image Vk.
FIG. 2A shows the positional relationship in the world coordinate system between the target viewpoint image Vk and each of the near viewpoint image C1 and the near viewpoint image C2 that are the near viewpoint images CVk. A viewpoint passing through the image center O in the target viewpoint image Vk is set as a viewpoint L, and the viewpoint L is projected onto each of the near viewpoint image C1 and the near viewpoint image C2 (two-dimensional viewpoint coordinates of the near viewpoint image C1 and the nearby viewpoint image C2). Coordinate conversion). The viewpoint L projected onto the near viewpoint image C1 is the projection viewpoint LC1, and the viewpoint L projected onto the near viewpoint image C2 is the projection viewpoint LC2.

FIG. 2B is a diagram showing a projected viewpoint LC2 projected on the near viewpoint image C2. Let w be the image width of the near viewpoint image C2 (the length of the long side C2W in the rectangular viewpoint image C2). Then, a line segment LL1 is generated in parallel to the short side C2H1 of the viewpoint image C2 at a position separated from the short side C2H1 by w / 4, which is a ¼ distance of the width w. Similarly, a line segment LL2 is generated in parallel with the short side C2H2 of the viewpoint image C2 at a position separated from the short side C2H2 by w / 4, which is a distance ¼ of the width w.
Here, the coordinates of the intersection between the projection viewpoint LC2 and the line segment LL1 are set as a coordinate value xC2_1, and the coordinates of the intersection between the projection viewpoint LC2 and the line segment LL2 are set as a coordinate value xC2_2. In addition, similar processing is performed on the near viewpoint image C1, and each of the coordinate value xC1_1 and the coordinate value xC1_2 is obtained.

  Next, each of the coordinate value coordinate value xC2_1, the coordinate value xC2_2, the coordinate value xC1_1, and the coordinate value xC1_2 is projected onto the viewpoint L in the world coordinate system. Then, it is assumed that the image center O, which is the center coordinate of the target viewpoint image Vk, corresponds to each of the coordinate value xC1_1 and the coordinate value xC1_2 on the near viewpoint image C1, that is, the pixel value of the coordinate value xC1_1 and the coordinate value xC1_2. Assuming that each corresponds to a three-dimensional coordinate point on the viewpoint L, a three-dimensional coordinate point ZC1_1 and a three-dimensional coordinate point ZC_2 are generated. Similarly, it is assumed that the image center O, which is the center coordinate of the target viewpoint image Vk, corresponds to the coordinate value xC2_1 and the coordinate value xC2_2 on the near viewpoint image C2, that is, the pixel of the coordinate value xC2_1 and the coordinate value xC2_2. , Corresponding to a three-dimensional coordinate point on the viewpoint L, a three-dimensional coordinate point ZC2_1 and a three-dimensional coordinate point ZC2_2 are generated.

  Next, at the three-dimensional coordinate point ZC1_1 and the three-dimensional coordinate point ZC1_2 projected onto the viewpoint L, the three-dimensional coordinate point ZC1_2 that is close to the coordinate center O is classified into the minimum position selection group, and the three-dimensional coordinate that is far from the coordinate center O is displayed. The coordinate point ZC1_1 is classified into the maximum position selection group. Similarly, in the three-dimensional coordinate point ZC2_1 and the three-dimensional coordinate point ZC2_2 projected onto the viewpoint L, the three-dimensional coordinate point ZC2_1 that is close to the coordinate center O is classified into the minimum position selection group, and the three-dimensional that is far from the coordinate center O. The coordinate point ZC2_2 is classified into the maximum position selection group. For example, even when there are three or more neighboring viewpoint images, the same processing as described above is performed, and each of the three-dimensional coordinate point ZCn_1 and the three-dimensional coordinate point ZCn_2 projected onto the viewpoint L is changed to the minimum position selection group, the maximum position. Sort and sort each selection group.

Then, the initialization unit 12 selects a coordinate point of a three-dimensional coordinate point on the viewpoint L farthest from the image center O in the world coordinate system in the minimum position selection group. In FIG. 2A, the three-dimensional coordinate point ZC2_1 of the near viewpoint image C1 is selected as the minimum three-dimensional coordinate point Zmin.
Similarly, the initialization unit 12 selects a coordinate point of a three-dimensional coordinate point on the viewpoint L closest to the image center O in the world coordinate system in the maximum position selection group. In FIG. 2A, the three-dimensional coordinate point ZC1_1 of the near viewpoint image C2 is selected as the maximum three-dimensional coordinate point Zmax.
Thereby, the depth from the image center O calculated | required with a random number is set with a random number within the range of the line segment set by the three-dimensional coordinate point ZC2_1 and the three-dimensional coordinate point ZC1_1. That is, the initialization unit 12 uses the three-dimensional coordinate point ZC1_1 as a reference value, obtains a numerical value within a range from 0 to Δd by a random number, and adds the numerical value obtained by the random number to the reference value, thereby obtaining the target viewpoint image. Depth dVk (m) at Vk is obtained. In the following, Expression (1) representing the range of the depth dVk (dVk (m)) by a random number is shown.

  In the equation (1), the depth dVk (dVk (m)) is determined by the three-dimensional coordinate point Zmin (minimum three-dimensional coordinate point) on the viewpoint L in the world coordinate system and the three-dimensional coordinate point Zmax (maximum three-dimensional coordinate). It is shown that it exists between the coordinate point). That is, the initialization unit 12 obtains Δd using a random number, and obtains the depth dVk (dVk (m)) in the target viewpoint image Vk within the range of the coordinate value of the three-dimensional coordinate point of the equation (1).

  The expression (2) is an expression indicating the minimum value of the depth dVk when viewed from the image center O. That is, ZCNmin having the maximum distance from the image center O in ZCNmin (minimum position selection group) of the neighboring viewpoint images Cn belonging to the set V of viewpoint images excluding the target viewpoint image Vk is represented as Zmin. ing.

  The expression (3) is an expression indicating the maximum value of the depth dVk in the expression (1) when viewed from the image center O. That is, ZCnmax having the smallest distance from the image center O among ZCnmax (maximum position selection group) of the neighboring images Cn belonging to the set of neighboring viewpoint images CVk of the target viewpoint image Vk is set as the maximum value Zmax. ing.

  Returning to FIG. 1, for example, in the present embodiment, the initialization unit 12 sets the generation range of random numbers of the normal slope θVk (m) and the normal slope φVk (m) to ± 60 degrees. In the present embodiment, the normal inclination θVk (m) indicates an angle formed by the normal and the z-axis, and the normal inclination φVk (m) is a two-dimensional plane formed by the normal and the x-axis y-axis. The angle formed by the projected normal and the x axis is shown.

When selecting the target viewpoint image Vk from the viewpoint images, the initialization unit 12 determines a near viewpoint image CVk for the target viewpoint image Vk, and sets the random number generation range of the depth dVk (m) expressed by the equation (1) as follows: Obtained for each target viewpoint image Vk.
Then, the initialization unit 12 sets the depth dVk (m) in the parameter of each pixel (pixel m) in the target viewpoint image Vk to a random number in the range indicated by the expression (1) corresponding to the target viewpoint image Vk as described above. (Δd) is determined at a predetermined step size. Further, when the initialization unit 12 obtains the depth dVk (m), each of the obtained normal gradient θVk (m) and normal gradient φVk (m) of the pixel m is ± 60 as described above. Obtained by random numbers in the range of degrees.
For each viewpoint image selected as the target viewpoint image Vk, the initialization unit 12 writes and stores the pixel parameters obtained by random numbers in the parameter table of the storage unit 19.

FIG. 3 is a diagram illustrating a configuration example of a parameter table stored in the storage unit 19 in the present embodiment. The parameter table is provided for each viewpoint image. For example, viewpoint image identification numbers (V1, V2, etc.) for identifying the viewpoint image are given.
In FIG. 3, the parameter table includes columns for pixel coordinates, parameters, a process end flag, and a process completion flag. Here, the pixel coordinate indicates the coordinate value of the pixel m in the viewpoint image. The parameters indicate numerical values of the depth vVk (m), the normal gradient θVk (m), and the normal gradient φVk (m) of the pixel m corresponding to the pixel coordinates. The process end flag identifies a pixel for which a spatial propagation process, which will be described later, has been completed, and is a flag that is set for each spatial propagation process. The processing completion flag is a flag for identifying a pixel that does not need to be subjected to subsequent propagation processing when an evaluation value for evaluating a parameter described later exceeds a predetermined threshold.

<Spatial propagation processing>
This spatial propagation process is a process of spatial propagation in the patch match stereo method.
The viewpoint selection unit 13 selects an unprocessed viewpoint image as a processing target of the spatial propagation process from all viewpoint images as the target viewpoint image Vk.
The thread generation unit 14 generates a thread for performing propagation processing of parameters of the viewpoint image selected as the target viewpoint image Vk in the target viewpoint image Vk selected by the viewpoint selection unit 13 (details of thread generation will be described later). .
The spatial propagation processing unit 15 performs processing for propagating the parameter of the pixel of the target pixel adjacent to the pixel m in the same target viewpoint image Vk and having undergone the spatial propagation processing.

Spatial propagation processing is based on the fact that, in many cases, adjacent pixels in the same viewpoint image are imaged on the surface of the same state in the same object, so that there is almost no change in parameters between adjacent pixels. It is said. That is, in the spatial propagation processing, parameters are propagated from neighboring pixels to neighboring pixels based on the above-described assumption of parameter continuity between neighboring pixels.
A first matching score (first evaluation value) is calculated when the pixel m = (u, v) in the target viewpoint image Vk maintains the same parameters as the neighboring pixel p. The first matching score is obtained by generating a three-dimensional coordinate point by converting the pixel m into the world coordinate system based on the luminance value of the pixel m and the parameter, and generating the generated three-dimensional coordinate point with respect to the viewpoint image Vk. The evaluation value is compared with the luminance value when coordinate conversion is performed, and the larger the evaluation value, the closer to the true parameter.

FIG. 4 is a diagram illustrating the correspondence between the target viewpoint image and the near viewpoint image in each of the spatial propagation process and the viewpoint propagation process. In FIG. 4, the parameter of the pixel p adjacent to the pixel m is propagated to the pixel m in the target viewpoint image Vx by the spatial propagation process.
Each of the three-dimensional coordinate point M in the world coordinate system corresponding to the pixel p and the three-dimensional coordinate point 200 adjacent to the three-dimensional coordinate point M is close to the coordinate point in the world coordinate system, and there is no significant change such as unevenness. It is considered to have a similar texture.

The spatial propagation processing unit 15 reads out from the parameter table of the storage unit 19 each of the parameter of the pixel m selected as the target pixel and the parameter of the pixel p that propagates the parameter to the pixel m.
For example, the parameter of the pixel m in the target viewpoint image Vk is pVk (m) = (dVk (m), θVk (m), φVk (m)). Further, the parameter of the pixel p = (u + δ, v) of the adjacent reference pixel is pVk (u + δ, v). The spatial propagation processing unit 15 calculates the first matching score using the parameter of the pixel m as Score (pVk (u, v), m).

In the spatial propagation processing unit 15, the function Score (*, m) is a function for calculating the first matching score. When * is pVk (u, v), the parameter pVk (m) of the pixel m is used as the parameter. The first matching score is calculated, and the first matching is calculated by the parameter pVk (u + δ, v) of the pixel p of the reference pixel adjacent to *.
The spatial propagation processing unit 15 then calculates the first matching score Score (pVk (u, v)), m) obtained when * is pVk (u, v) and * of the pixel p of the adjacent reference pixel. Comparison is made with the first matching score Score (pVk (u + δ, v), m) obtained for each of the parameters pVk (u + δ, v).

The spatial propagation processing unit 15
Score (pVk (u + δ, v), m)> Score (pVk (u, v)), m)
In the case of, the parameter of the pixel m is changed to the parameter of the pixel p which is an adjacent reference pixel. That is, the spatial propagation processing unit 15 propagates the parameter of the reference pixel as the parameter of the target pixel.
On the other hand, the spatial propagation processing unit 15
Score (pVk (u + δ, v), m) ≦ Score (pVk (u, v)), m)
In the case of, the parameter of the pixel m is not changed to the parameter of the pixel p which is the adjacent reference pixel, and the parameter of the pixel m is left as it is. That is, the spatial propagation processing unit 15 does not propagate the parameter of the reference pixel as the parameter of the target pixel.

  In addition, even when the pixel p = (u, v + δ) which is the reference pixel, the spatial propagation processing unit 15 obtains the first matching score Score (pVk (u, v + δ), m) and obtains the first matching score Score ( Comparison with pVk (u, v)), m). Then, as described above, the spatial propagation processing unit 15 determines whether or not to propagate the parameter of the reference pixel as the parameter of the target pixel by comparing the first matching score.

The spatial propagation process described above is performed for all the pixels in the pixel set Ω of the target viewpoint image Vk. Then, the spatial propagation processing unit 15 sets a processing end flag for the pixel (end pixel) for which the spatial propagation processing of the parameter in the parameter table of the storage unit 19 has been completed. For example, this processing end flag is set to “0” for a pixel for which spatial propagation processing has not been completed (unfinished pixel), and is set to “1” for a pixel for which spatial propagation processing has been completed. In addition, the spatial propagation processing unit 15 provides a predetermined completion threshold value, and when the parameter matching score of the pixel m exceeds the completion threshold value, the completion flag of the corresponding pixel in the parameter table in the storage unit 19 is changed from “0” to “ Change to 1 ”. When changing the parameter of the pixel m, the spatial propagation processing unit 15 refers to the parameter table in the storage unit 19 and does not perform the propagation process on the pixel m whose completion flag is “1”.
The spatial propagation process described above is performed a plurality of times for each viewpoint image.

FIG. 5 is a conceptual diagram showing an arrangement relationship between the reference pixel and the target pixel in the spatial propagation process. FIG. 5 shows an arrangement relationship between the pixel p of the reference pixel and the pixel m of the target pixel in the target viewpoint image Vk.
When the pixel m is the pixel b, the pixel p is the pixel a. When the pixel b is the pixel coordinate (u, v), the pixel a is the pixel coordinate (u-1, v).
Also, when the pixel m is the pixel c, the pixel p becomes the pixel a. When the pixel c is the pixel coordinate (u, v), the pixel a is the pixel coordinate (u, v−1).
On the other hand, when the pixel m is the pixel y, the pixel p is the pixel z. When the pixel y is the pixel coordinate (u, v), the pixel a is the pixel coordinate (u + 1, v).
In addition, when the pixel m is the pixel x, the pixel p is the pixel z. When the pixel x is the pixel coordinate (u, v), the pixel z is the pixel coordinate (u, v + 1).
That is, when the spatial propagation process is started from the pixel a, δ in pVk (u + δ, v) and pVk (u, v + δ) is “−1”, and when the spatial propagation process is started from the pixel z, pVk (u + δ , V) and pVk (u, v + δ) are “+1”.

As described above, spatial propagation processing is performed by Breyer's paper (M. Bleyer, C. Rhemann, and C. Rother, “PatchMatch Stereo-Stereo matching with slanted support windows,” Proc. British Machine Vision Conf., Vol. 11, pp. 1-11, 2011.) Similar to the patch match stereo spatial propagation processing in the multiple iterations, it starts from the pixel a of the upper left vertex at the odd number of times and starts from the pixel z of the lower right vertex at the even number of times. To do. That is, in an odd number of times, in a set of pixels arranged on a rectangle, a parameter spatial propagation process is performed from a pixel at a predetermined vertex toward a pixel at a vertex symmetrical to the predetermined vertex.
Further, in the target viewpoint image, it may be configured to start from the pixel z of the lower right vertex at odd times and to start from the pixel a of the upper left vertex at even times.
Further, instead of the pixel at the lower right vertex and the pixel at the upper left vertex, each of the pixel at the upper right vertex and the pixel at the lower left vertex of the target viewpoint image may be configured as an even-numbered pixel and an odd-numbered pixel, respectively. .

  On the other hand, in an even number of times, in a set of pixels arranged on a rectangle, a parameter spatial propagation process is performed from a vertex-symmetric pixel to the predetermined vertex toward the predetermined vertex. As a result, parameter propagation, which was insufficient in the spatial propagation processing from one direction, can be performed for all pixels by performing bidirectional transmission. For example, in only one direction, the parameter is not propagated to the vertex pixel at which the spatial propagation process is started. Further, in only one direction, even if a true parameter is obtained by a random number at a halfway pixel, the true parameter is not propagated to a pixel for which propagation processing is performed before that pixel.

<Viewpoint propagation processing>
This viewpoint propagation process is a view propagation process in the patch match stereo method.
This viewpoint propagation processing is based on the assumption that the depth map is consistent among many viewpoint images, and the parameters of the pixel m of the viewpoint image Vk are set to the neighboring viewpoint image CVk that has already been selected as the neighboring viewpoint image. This is a process for propagating to a pixel.
The viewpoint propagation processing unit 16 obtains a pixel in the near viewpoint image CVk corresponding to the pixel m of the target viewpoint image Vk.

The viewpoint propagation processing unit 16 performs coordinate conversion on the coordinates of the pixel m of the target viewpoint image Vk based on the depth dVk (m) in the parameter of the pixel m, using the conversion matrix of the captured viewpoint image Vk, and the target viewpoint image Vk. The three-dimensional coordinate point M in the camera coordinate system (three-dimensional) of the viewpoint is restored. Then, the viewpoint propagation processing unit 16 obtains a three-dimensional coordinate point M ′ in the camera coordinate system (three-dimensional) of the viewpoint of the near viewpoint image CVk by performing coordinate transformation on the restored three-dimensional coordinate point M.
The following formula (4) for obtaining the coordinate value of the three-dimensional coordinate point M ′ is shown below. In addition, the viewpoint propagation processing unit 16 obtains the coordinate value of the three-dimensional coordinate point M ′ by the equation (4).

In the equation (4), Mx ′, My ′, and Mz ′ indicate the x-coordinate value, y-coordinate value, and z-coordinate value of the three-dimensional coordinate point in the camera coordinate system at the viewpoint of the near viewpoint image CVk, respectively. . [*] ^T indicates a transposed matrix. m˜ (on the top of m˜) represents the homogeneous coordinates of m. Rc is a rotation matrix in the camera coordinate system of the viewpoint of the near viewpoint image CVk. RVk ⁻¹ is an inverse matrix of the rotation matrix at the viewpoint of the target viewpoint image Vk. tVk is a translation vector in the camera coordinate system of the viewpoint of the target viewpoint image Vk. tc is a translation vector in the camera coordinate system of the viewpoint of the near viewpoint image CVk. AVk ⁻¹ is an inverse matrix of camera parameters in the camera coordinate system of the viewpoint of the target viewpoint image Vk.

  Then, the viewpoint propagation processing unit 16 obtains the coordinate point of the pixel m ′ obtained by projecting the three-dimensional coordinate point M ′ onto the neighboring viewpoint image Cvk using the following equation (5).

In Expression (5), Ac is a matrix of camera parameters in the camera coordinate system of the viewpoint of the near viewpoint image CVk. s is a non-zero real number.
The viewpoint propagation processing unit 16 uses the equation (5) to determine the coordinates of the pixel m ′ that is the corresponding point of the three-dimensional coordinate point M ′ in the camera coordinate system of the viewpoint of the neighboring viewpoint image CVk on the neighboring viewpoint image CVk. Find a point.
Then, the viewpoint propagation processing unit 16 uses the parameter normal vector of the pixel m corresponding to the camera coordinate system of the viewpoint of the target viewpoint image Vk, and the parameter normal vector of the parameter corresponding to the camera coordinate system of the viewpoint of the neighboring viewpoint image CVk. Is converted using the following equation (6).

  In the equation (6), n ′ represents a normal vector of the parameter of the pixel m ′ corresponding to the camera coordinate system of the viewpoint of the near viewpoint image CVk. n indicates a normal vector in the parameter of the pixel m corresponding to the camera coordinate system of the viewpoint of the target viewpoint image Vk. Each of nX ′, nY ′, and nZ ′ indicates the x coordinate value, the y coordinate value, and the z coordinate value of the three-dimensional coordinate point of the camera coordinate system at the viewpoint of the near viewpoint image CVk.

  Next, the following expression (7) shows the parameter p ′ (m ′) of the pixel m ′ in the near viewpoint image CVk obtained from the above expression (6).

In the above equation (7), Mz ′ represents a parameter of the depth d at the pixel m ′. tan ⁻¹ (nX ′ / nZ ′) represents a normal gradient θCVk (m ′). tan ⁻¹ (nY ′ / nZ ′) represents a normal gradient φCVk (m ′).

The viewpoint propagation processing unit 16 obtains a matching score Score (p ′ (m ′), m ′) for the pixel m ′ in the near viewpoint image CVk using the above equation (7). This matching score corresponds to the multi-viewpoint image in the present embodiment, and details will be described later.
Here, the viewpoint propagation processing unit 16 calculates the matching score Score (p ′ (m ′), m ′) using the parameter p ′ (m ′) of the pixel m ′ based on the parameter of the pixel m. Further, the viewpoint propagation processing unit 16 reads the parameter pC (m ′) of the pixel m ′ from the parameter table of the near viewpoint image CVk in the storage unit 19 and calculates the matching score Score (pC (m ′), m ′). To do.

Then, the viewpoint propagation processing unit 16 compares Score (p ′ (m ′), m ′) with Score (pC (m ′), m ′).
Here, the viewpoint propagation processing unit 16
Score (p ′ (m ′), m ′)> Score (pC (m ′), m ′)
, The parameter of the pixel m ′ is converted from pC (m ′) to p ′ (m ′). At this time, the viewpoint propagation processing unit 16 rewrites the parameter of the pixel m ′ in the parameter table of the near viewpoint image CVk in the storage unit 19 to p ′ (m ′).

On the other hand, the viewpoint propagation processing unit 16
Score (p ′ (m ′), m ′) ≦ Score (pC (m ′), m ′)
In this case, the parameter of the pixel m ′ remains pC (m ′), and the parameter table of the near viewpoint image CVk in the storage unit 19 is not rewritten.
As described above, the viewpoint propagation processing unit 16 performs viewpoint propagation processing for propagating the parameter of the pixel m in the target viewpoint image Vk to the pixel m ′ in the neighboring viewpoint image CVk.

In the present embodiment, by performing the above-described viewpoint propagation processing, the parameters of the pixel m in the target viewpoint image Vk are set to the pixels m ′ of the neighboring viewpoint images CVk of all viewpoints in the vicinity of the viewpoint of the target viewpoint image Vk. Can be propagated.
However, when parameter propagation processing is performed on all neighboring viewpoint images CVk of neighboring viewpoints, parameter propagation processing needs to be performed Npair times corresponding to the number of stereo pairs Npair, which increases the calculation cost. End up. For this reason, in this embodiment, the near viewpoint image CVk is selected corresponding to one viewpoint, and the viewpoint propagation process is performed using the selected one near viewpoint image CVk as a stereo pair. Thereby, even if the number of stereo pairs Npair increases, the processing time of the viewpoint propagation process can be kept constant.

Further, the viewpoint propagation process of parameters from the pixel m of the target viewpoint image Vk may not be performed on all the pixels m ′ in the near viewpoint image CVk. However, when this neighboring viewpoint image CVk is selected as the target viewpoint image Vk, a pixel m ′ that has been subjected to the viewpoint propagation process is replaced with a pixel m ′ that has not been subjected to another viewpoint propagation process by the spatial propagation process. The parameter propagated in the viewpoint propagation process is propagated.
As a result, in this embodiment, parameters are propagated to each of all viewpoint images, and when viewpoint propagation processing is performed with all viewpoint images as respective neighboring viewpoint images. The same effect can be obtained.

That is, in FIG. 4, parameter propagation is performed from the pixel m in the target viewpoint image Vk to the pixel m ′ corresponding to the neighboring viewpoint image CVk by the viewpoint propagation process.
The pixel m and the pixel m ′, which are projections of the three-dimensional coordinate point M and the three-dimensional coordinate point M ′ in the world coordinate system, respectively correspond to the three-dimensional coordinate point M and the three-dimensional coordinate point M ′. Is assumed to have a parameter that is consistent with the texture.

In addition, as described above, in the present embodiment, since the near viewpoint image CVk that propagates the parameter from the target viewpoint image Vk is selected at random, a random viewpoint image CVk around the target viewpoint image Vk is randomly selected. A viewpoint propagation process equivalent to propagating the parameter of the pixel m of the target viewpoint image Vk is performed by providing an interval between the viewpoint images.
For this reason, according to the present embodiment, even when the neighborhood viewpoint image CVk that propagates the parameter of the pixel m of the target viewpoint image Vk is randomly selected, the neighborhood that becomes a stereo pair from the target viewpoint image Vk by the viewpoint propagation processing The parameter can be propagated with the same accuracy as when the parameter is propagated to all the viewpoint images. Further, according to the present embodiment, by selecting only one nearby viewpoint image CVk as a target for propagating parameters from the target viewpoint image Vk, and by performing viewpoint propagation processing on this nearby viewpoint image CVk, Even if the viewpoint images forming a stereo pair increase, it is possible to perform parameter propagation processing between viewpoint images while maintaining the calculation cost and the restoration accuracy.

<Matching score in viewpoint propagation processing>
In the present embodiment, using ZNCC (Zero-mean Normalized Cross Correlation), a matching score Score (p ′ (m ′), m ′).
In the present embodiment, when obtaining the ZNCC, the inclination of the normal line is used to correct the inclination of the surface of the evaluation target, so that the near viewpoint image matches the window shape cut out from the target viewpoint image Vk. The window shape cut out from CVk is changed. This makes it possible to perform matching with robustness for comparison between the target viewpoint image and a neighboring viewpoint image having an inclination with respect to the target viewpoint image.

  ZNCC is expressed by the following equation (8).

  In the above equation (8), f- (-) is represented by the following equation (9), and g- (-) is represented by the following equation (10).

As shown in both the equations (9) and (10), the average value of the pixel values of the entire image is subtracted from each of the pixel value f and the pixel value g obtained by the function, and f− (f -) And g- (-) in the upper part of g.
In the present embodiment, each of the pixel value f and the pixel value g in the equation (8) is defined as the following equations (11) and (12).

IVk indicates an image. m indicates the coordinates of the pixel. ω represents the number of pixels. Expression (11) indicates that a window of ω × ω pixels is cut out from the image IVk with the pixel m as the center.
Further, Trans (I, H) indicating an image in the expression (12) indicates that the image I is converted by the projective transformation matrix H. In the equation (12), when a parameter p (d (m), θ (m), φ (m)) of a certain pixel m is given, the target viewpoint image Vk and the neighboring viewpoint image CVk (simply C) The window that performs the matching is also modified using the following equation (13).

  In the above equation (13), R is a rotation matrix and is represented by the following equation (14).

In the above equation (14), the rotation matrix R is obtained from the rotation matrix RC at the viewpoint of the near viewpoint image CVk and the inverse matrix RVk ⁻¹ of the rotation matrix RVk at the viewpoint of the target viewpoint image Vk.
Further, t in the equation (13) is a translation vector, and is represented by the following equation (15).

In the above equation (15), the translation vector t is obtained from the result of multiplying the translation vector tC at the viewpoint of the near viewpoint image CVk and the translation vector tVk at the viewpoint of the target viewpoint image Vk by the rotation matrix R.
M in the equation (13) is a coordinate value of a three-dimensional coordinate point in the camera coordinate system at the viewpoint of the target viewpoint image Vk, and is obtained by the following equation (16).

In the equation (16), d is data of depth in the parameter of the target viewpoint image Vk. AVk ⁻¹ is an inverse matrix of the camera parameter matrix AVk at the viewpoint of the target viewpoint image Vk. u is the x coordinate of the pixel m in the target viewpoint image Vk. v is the y coordinate of the pixel m in the target viewpoint image Vk.
Further, n in the equation (13) is a normal vector, and is obtained by the following equation (17).

  In the above equation (17), each of θ and φ is an angle in the camera coordinate system at the viewpoint of the target viewpoint image Vk, and each is an angle formed by the normal line with the z axis, and the normal line is formed by the x axis and the y axis. Projected on a two-dimensional plane, the angle formed by the projected normal and the x axis is shown.

By using each of the above-described equations (11) and (12), a matching score between two viewpoint images (the target viewpoint image Vk and the neighboring viewpoint image CVk) can be calculated.
Since ZNCC is a matching score between two viewpoint images, the value is one. For this reason, the value of ZNCC can be used as a matching score obtained from the depth and the slope of the normal as it is.
On the other hand, in the present embodiment, since multi-viewpoint images are used, a plurality of ZNCC values corresponding to the number of neighboring viewpoint images CVk can be obtained in the target viewpoint image Vk and the plurality of neighboring viewpoint images CVk. However, a plurality of ZNCC values cannot be used as they are as matching scores for comparison between the target viewpoint image Vk and the plurality of neighboring viewpoint images CVk. Therefore, a matching score obtained by integrating a plurality of obtained ZNCC values is defined and used as a matching score for comparison between the target viewpoint image Vk and the plurality of neighboring viewpoint images CVk.

In addition, ZNCC in the comparison between the target viewpoint image Vk and the plurality of neighboring viewpoint images CVk may be a low value due to the influence of occlusion of the viewpoint images that form a stereo pair. As a result, when the obtained ZNCC values are integrated by a simple average, as described above, there may be the influence of occlusion, resulting in a low-reliability matching score.
For this reason, in the present embodiment, the confidence value is used as a matching score in order to integrate a plurality of ZNCC values obtained by comparing the target viewpoint image Vk and the plurality of neighboring viewpoint images CVk.

For each viewpoint image VkεV, a neighboring viewpoint image that is a stereo pair to be matched is defined as CVk = {C1, C2, C3,..., CNpair} ⊆V \ {Vk}. That is, a plurality of neighboring viewpoint images CVk are selected from viewpoint images excluding Vk selected as the target viewpoint image in the viewpoint image group V. Here, the number of stereo pairs Npair is the number of stereo pairs used in the calculation for obtaining the matching score.
At this time, the matching score Score (p, m) at the parameter p = {d, θ, φ} of the pixel m in the target viewpoint image Vk is expressed by the following equation (18).

In the above equation (18), th is a predetermined threshold value for determining the value of ZNCC. Further, f is the above formula (11), and g is the above formula (12).
Further, ZNCC ′ (f, g, th) in the equation (18) is obtained by the following equation (19).

  In equation (11), ZNCC ′ (f, g, th) becomes ZNCC (f, g) when ZNCC (f, g) is larger than the threshold th, and becomes the threshold th otherwise. That is, when this threshold value th is exceeded, ZNCC is calculated by the equation (11), assuming that the reliability value of ZNCC (f, g) is high.

<Fine adjustment process>
This fine adjustment process is a Plane refinement process in the patch match stereo method.
In the fine adjustment process, in the case of only the parameter propagation process by the spatial propagation process and the viewpoint propagation process described above, there are parameters that cannot be searched with high accuracy depending on the initial value obtained by a random number. Therefore, in this fine adjustment process, a parameter is updated by adding a small random number as an adjustment value to the parameter of the pixel m for which the propagation process has been completed for the target pixel, and comparing the matching score.

The fine adjustment unit 17 generates a random number ΔdVk, a random number ΔθVk, and a random number ΔφVk for each of the depths dVk (m), θVk (m), and φVk (m).
Then, the fine adjustment unit 17 sets Δp = (ΔdVk, ΔθVk, ΔφVk), and reads the parameter of the pixel m from the parameter table corresponding to the target viewpoint image Vk in the storage unit 19. The fine adjustment unit 17 calculates a matching score Score (pVk (m), m) and a matching score Score (pVk (m) + Δp, m).

  The fine adjustment unit 17 compares each of Score (pVk (m), m) and Score (pVk (m) + Δp, m) of the matching score. At this time, the fine adjustment unit 17 changes pVk (m) to pVk (m) + Δp when Score (pVk (m) + Δp, m)> Score (pVk (m), m). That is, the fine adjustment unit 17 sets pVk (m) = (dVk (m), θVk (m), φVk (m)) to pVk (m) in the parameter table corresponding to the target viewpoint image Vk in the storage unit 19. + P = (dVk (m) + ΔdVk, θVk (m) + ΔθVk, φVk (m) + ΔφVk).

Δp described above is generated as a random number for each pixel m selected from all the pixels in the set of pixels in the target viewpoint image Vk.
In the present embodiment, 1/4 of the range in which each random number is generated in initialization is set as the generation range of Δp. For example, the depth dVk (m) is generated by a random number with ΔdVk as an adjustment value with a finer step size than Δd in a quarter of the range shown in the equation (1). Further, with respect to the inclination of the normal line, the value of ± 60 degrees is set to ± 15 degrees, and each of the adjustment values of ΔθVk and ΔφVk is generated by a random number with a finer step size.

In addition, since this fine adjustment process generates an adjustment value using a random number, even if the adjustment value is generated only once using a random number, it is insufficient to obtain the effect of fine adjustment. For this reason, the fine adjustment unit 17 performs an update process of generating ΔP Nrefine times and changing it to pVk having the highest matching score.
Further, every time the fine adjustment process is repeated, in order to improve the accuracy of adjustment, the generation range of ΔP is gradually narrowed, for example, the generation range is decreased by ½.

<Parallel processing of parameter propagation in pixel units>
Hereinafter, an operation of parallelizing the spatial propagation processing, viewpoint propagation processing, and fine adjustment processing of the patch match stereo method described above in this embodiment in units of pixels will be described with reference to FIG. In the following description, the initialization of pixels in all viewpoint images has been completed.
In the patch match stereo method, as described above, the process of propagating the parameter p from the reference pixel to the target pixel is performed. For this reason, when selecting a target pixel to be subjected to spatial propagation processing, viewpoint propagation processing, and fine adjustment processing, the pixel for which propagation processing has been completed needs to be a reference pixel, and the order of performing the propagation processing is dependent. .

  Therefore, in the patch match stereo method, if the order of selection of the target pixel for performing parameter propagation processing in the pixel array is changed, a result of correct parameter propagation processing cannot be obtained. Therefore, in the conventional patch match stereo method, the propagation direction of the parameter is set to one direction, for example, so that the pixel subjected to the propagation process is a reference pixel and the pixel adjacent to the reference pixel is a target pixel. Limiting to the row direction (x-axis direction) shown in FIG. 5, parallelization is performed to perform propagation processing for each row.

However, since the parameter propagation direction is limited to one direction, parameter propagation by the patch match stereo method is not sufficiently performed, so that accurate parameter estimation is not performed and three-dimensional shape restoration accuracy is performed. There may not be.
For this reason, in this embodiment, in order to obtain the result of correct parameter propagation processing and improve the reconstruction accuracy of the three-dimensional shape, the propagation direction is not limited to only one direction, but not only in the row direction but in the column direction (y-axis). Parallel processing that propagates to (direction) is also performed.
In the present embodiment, for example, a hardware LSI that performs parallel processing of a plurality of threads such as a pixel engine LSI (Large Scale Integration) is used, and the system LSI is configured to perform a plurality of parameter propagation processes. . Alternatively, a configuration may be adopted in which a plurality of threads are executed using a high-speed CPU. In any case, in the present embodiment, each of the initialization unit 12, the viewpoint selection unit 13, the spatial propagation processing unit 15, the viewpoint propagation processing unit 16, and the fine adjustment unit 17 operates independently in each thread. The thread can be executed in parallel.

In FIG. 5, the thread generation unit 14 generates a thread for parameter propagation performed by the spatial propagation processing unit 15 corresponding to each row of pixels in the target viewpoint image Vk (corresponding to each pixel row).
In addition, the spatial propagation processing unit 15 selects one of adjacent pixels (adjacent pixels in the same row, adjacent pixels in a row immediately above or directly below) in each thread in the order in which the parameters of each row are propagated. Is detected with reference to the parameter table in the storage unit 19 to determine whether or not the parameter propagation processing has ended, that is, whether or not the processing end flag is set (is set to “1”). Then, when the parameter propagation process has been completed for any of the pixels adjacent to the leading pixel in the propagation direction, the spatial propagation processing unit 15 sets the leading pixel as the target pixel in the thread, and sets the parameter A parameter propagation process between rows is performed by using any of the adjacent pixels in the row immediately above or immediately below the propagation process as a reference pixel.

FIG. 6 is a diagram illustrating a thread for spatial propagation processing of each pixel in the target viewpoint image Vk in FIG. As shown in FIG. 6, the thread generation unit 14 generates each of the parameter propagation thread TH1, the thread TH2,... That is performed by the spatial propagation processing unit 15 corresponding to each row of pixels in the target viewpoint image Vk. Each of the threads TH1, TH2,... Is generated corresponding to the first row, the second row,... Of pixels in the target viewpoint image Vk in FIG.
For example, when the pixel a is the reference pixel, in the thread TH1, the spatial propagation processing unit 15 can perform the parameter propagation process by the spatial propagation process with the adjacent pixel b as the target pixel. In the thread TH1, when the parameter propagation processing for the pixel b is completed, the spatial propagation processing unit 15 becomes the reference pixel, the adjacent pixel d becomes the target pixel, and sequentially corresponds to the thread TH1. The parameter propagation process is performed. In the thread TH1, the spatial propagation processing unit 15 rewrites the end flag for the pixel b in the parameter table in the storage unit 19 from “0” to “1” when the parameter propagation processing for the pixel b is completed.

  When the pixel a is a reference pixel, the spatial propagation processing unit 15 in the thread TH2, the pixel e that is a pixel adjacent in the direction in which the propagation process is performed on the pixel c that is the target pixel, and the processing order of the thread Is determined whether or not the propagation process of the adjacent pixel a has been completed, excluding the pixel f which is an adjacent pixel in the row after the self. In the thread TH2, the spatial propagation processing unit 15 uses the pixel a as a reference pixel, and performs a parameter propagation process on the pixel c by the spatial propagation process. In the thread TH2, when the parameter propagation to the pixel c is completed, the spatial propagation processing unit 15 sets the target pixel as the pixel e and the adjacent pixel c as the target pixel. The spatial propagation processing unit 15 rewrites the end flag of the pixel c in the parameter table in the storage unit 19 from “0” to “1” when the parameter propagation processing for the pixel c is completed.

  In the thread TH2, the spatial propagation processing unit 15 adjoins the pixel h that is adjacent to the pixel e that is the target pixel in the direction in which the propagation processing is performed, in the row in which the processing order of the thread is after itself. It is determined whether or not the propagation process of each of the adjacent pixels b and c is finished except for the pixel i which is a pixel to be processed. Then, in the thread TH2, the spatial propagation processing unit 15 sets each of the pixel b and the pixel c as the reference pixel after the propagation processing of each of the pixel b and the pixel c is finished, and sets the parameter by the spatial propagation processing for the pixel e. Propagation processing is performed.

Here, in the thread TH2, the spatial propagation processing unit 15 sets the parameter having the higher matching score among the parameters of the pixel b and the pixel c as the parameter of the pixel e. In the thread TH2, when the spatial propagation processing for the pixel e is completed, the spatial propagation processing unit 15 rewrites the end flag for the pixel e from “0” to “1” in the parameter table of the storage unit 19.
As described above, in each thread, the spatial propagation processing unit 15 sequentially performs a spatial propagation process of parameters between adjacent pixels for each row in a thread unit.

In the thread processing described above, in the case where the pixel a is a parameter propagation start pixel in FIG. 5, the order of the upper pixel row to the lower pixel row is the order of the left pixel to the right pixel in each row. This corresponds to the case where spatial propagation processing is performed. At this time, when the coordinate value of the pixel m of the target pixel is (u, v), the coordinate value of the pixel p of the reference pixel is (u-1, v) or (u, v-1).
On the other hand, when the pixel z is set as the parameter propagation start pixel, spatial propagation processing may be performed in the order of the lower pixel row to the upper pixel row, and in the order of the right pixel to the left pixel in each row. is there. At this time, when the coordinate value of the pixel m of the target pixel is (u, v), the coordinate value of the pixel p of the reference pixel is (u + 1, v) or (u, v + 1).

As described above, in the present embodiment, in the parameter spatial propagation processing, whether or not the propagation processing for the pixel to be the reference pixel has been completed is confirmed by the end flag, so the parameter propagation order in each row Can keep. In addition, since the parameter having the higher matching score is propagated from the two reference pixels to the corresponding pixel, that is, not only in the row direction but also in the column direction (y-axis direction), the parameter estimation accuracy can be improved. it can.
Further, when the spatial propagation processing for the target pixel in the target viewpoint image Vk is completed for each thread, the viewpoint propagation processing unit 16 performs the viewpoint propagation processing on the pixel of the near viewpoint image CVk using the target pixel.

<Description of parameter propagation processing>
Next, an operation procedure of parameter propagation processing will be described.
FIG. 7 is a flowchart showing an operation procedure of parameter propagation processing of the depth map generation device.
Step S1:
The control unit 11 performs initialization so that the number δ indicating the number of repetitions of the propagation process is “1”. Further, the control unit 11 initializes all completion flags in the parameter table in the storage unit 19 to “0”.
The initialization unit 12 initializes parameters of all pixels in all viewpoint images stored in the storage unit 19 by generating random numbers.

Step S2:
The control unit 11 performs initialization to set the end flag in the parameter table in the storage unit 19 to “0”.

Step S3:
The viewpoint selection unit 13 selects a viewpoint image serving as a target viewpoint from viewpoint images stored in the storage unit 19.

Step S4:
The control unit 11 determines whether the current parameter propagation process is an odd number or an even number, that is, whether the number δ is an odd number or an even number. At this time, the control unit 11 advances the process to step S5 when the number of times δ is an odd number, and advances the process to step S6 when the number of times δ is an even number.

Step S5:
The thread generation unit 14 generates a thread to be performed by the spatial propagation processing unit 15 when the pixel a (see FIG. 5) at the top left of the target viewpoint image Vk is used as a start pixel. Here, since the start pixel is the pixel a, the thread generation unit 14 corresponds to the order of each row of the pixels from the first row at the top to the nth row at the bottom, and the thread TH1, the thread TH2,. Generate each of
Then, the spatial propagation processing unit 15 detects, in each thread, whether or not the propagation processing in other pixels adjacent to each pixel in the corresponding row is completed by referring to the parameter table in the storage unit 19, Propagation processing from the reference pixel to the target pixel is performed in accordance with the matching score as described above.

Step S6:
The thread generation unit 14 generates a thread to be performed by the spatial propagation processing unit 15 when the lower right top pixel z (see FIG. 5) of the target viewpoint image Vk is set as a start pixel. Here, since the start pixel is the pixel z, the thread generation unit 14 corresponds to the order of each row of pixels from the lowermost nth row to the uppermost first row, and the threads TH1, TH2,. Generate each of
Then, in each thread, the spatial propagation processing unit 15 detects whether or not the propagation processing in other pixels adjacent to each pixel in the corresponding row is completed by referring to the parameter table in the storage unit 19, Propagation processing from the reference pixel to the target pixel is performed in accordance with the matching score as described above.

Step S7:
The viewpoint selection unit 13 selects a propagation destination neighboring viewpoint image CVk that propagates parameters from the target pixel of the target viewpoint image Vk, from the viewpoint images excluding the target viewpoint image Vk in the storage unit 19.

Step S8:
In each thread, the viewpoint propagation processing unit 16 performs viewpoint propagation processing for propagating the parameter of the target pixel of the target viewpoint image Vk to the pixel of the neighboring viewpoint image CVk corresponding to the target pixel.

Step S9:
In each thread, the fine adjustment unit 17 performs fine adjustment processing of the parameter propagated to the target pixel of the target viewpoint image Vk by generating an adjustment value using a random number. Then, the fine adjustment unit 17 outputs the parameter of the target pixel for which the fine adjustment process has been completed to the spatial propagation processing unit 17.
Here, the fine adjustment unit 17 writes the parameter of the target pixel for which the fine adjustment process has been completed in the parameter field of the corresponding pixel record in the parameter table of the storage unit 19.

Step S10:
In each thread, the fine adjustment unit 17 rewrites and changes the end flag of the corresponding pixel in the parameter table of the storage unit 19 from “0”. At this time, when the matching score based on the parameter of the target pixel exceeds the completion threshold, the spatial propagation processing unit 17 changes the completion flag of this pixel in the parameter table of the storage unit 19 from “0” to 1.

Step S11:
The control unit 11 refers to the parameter table corresponding to the target viewpoint image Vk in the storage unit 19 and determines whether or not the end flags of all the pixels are “1”.
At this time, when the end flag of all the pixels in the parameter table is “1”, the control unit 11 finishes the parameter spatial propagation process and the viewpoint propagation process for all the pixels of the target viewpoint image Vk. And the process proceeds to step S12. On the other hand, when the end flag of all the pixels in the parameter table is not “1”, the control unit 11 determines that the parameter spatial propagation process and the viewpoint propagation process for all the pixels of the target viewpoint image Vk are not completed. Determination is made, and the process proceeds to step S4. At this time, the control unit 11 stores a viewpoint image stored in the storage unit 19 in a viewpoint image processing table (not shown) in which the viewpoint image and a flag indicating whether or not the viewpoint image has been subjected to the spatial propagation processing have been associated. On the other hand, a flag is set for the viewpoint image for which the parameter propagation processing for all pixels has been completed (from “0” to “1”).

  In addition, when the number of times δ is an odd number, the control unit 11 determines whether only the end flag of the pixel z is “1” because the start pixel is the pixel a and the end pixel to be processed last is the pixel z. On the other hand, if the number δ is an even number, since the start pixel is the pixel z and the end pixel to be processed last is the pixel a, only the end flag of the pixel a is “1”. It may be configured to determine whether or not.

Step S12:
The control unit 11 refers to the viewpoint image processing table in the storage unit 19 and determines whether or not the parameter spatial propagation processing for all viewpoint images has been completed, that is, whether or not the flags of all viewpoint images are “1”. Judgment is made.
At this time, if the parameter spatial propagation processing for all viewpoint images is completed, the control unit 11 advances the processing to step S13. On the other hand, if the parameter spatial propagation processing for all viewpoint images has not been completed, the control unit 11 advances the processing to step S3.

Step S13:
The control unit 11 increments the number δ (adds 1).

Step S14:
The control unit 11 determines whether or not the number of times δ exceeds the set number h. When the number of times δ exceeds the set number h, the control unit 11 ends the parameter propagation process. On the other hand, when the number of times δ does not exceed the set number h (less than the set number h), the control unit 11 advances the process to step S2. When the number δ exceeds the set number h, the control unit 11 determines that the parameters in each pixel of all viewpoint images are estimated.
Here, the set number h is, for example, three or more experimentally, and the number of repetitions in which sufficient parameter estimation is performed by propagation of parameters between pixels is made to correspond to the three-dimensional uneven shape. In advance, it is obtained and set by an experiment or the like.

<Parallel processing of parameter propagation for each viewpoint image>
Hereinafter, an operation of parallelizing the spatial propagation processing, viewpoint propagation processing, and fine adjustment processing of the patch match stereo method described above in this embodiment in units of viewpoint images will be described with reference to FIGS. 8 and 9. In the following description, the initialization of pixels in all viewpoint images has been completed.
FIG. 8 is a diagram illustrating the position of each viewpoint of the imaging apparatus that images a three-dimensional object. In FIG. 8, for example, the object 100 is imaged by an imaging device arranged in each of the viewpoint A to the viewpoint I, and a multi-viewpoint image that is a plurality of viewpoint images is acquired. The viewpoint near the viewpoint A is the viewpoint B, and the viewpoints near the viewpoint B are the viewpoint A and the viewpoint C.

FIG. 9 shows, as an example, the configuration of a graph when there are two neighboring viewpoint images in the viewpoint propagation process. In this graph, for example, when viewpoint A is a starting viewpoint image for starting viewpoint propagation processing, viewpoint A → view point B → view point C → view point D →... The line-of-sight propagation process of parameters between the viewpoint images is performed.
For example, when the viewpoint image of the viewpoint A is the target viewpoint image Vk, the neighboring viewpoint image CVk is the viewpoint image of each of the viewpoints B and C, and then the viewpoint image of the viewpoint B is the target viewpoint image Vk. In this case, the near viewpoint image CVk is a viewpoint image of each of the viewpoints A and C.

In the viewpoint propagation processing already described, when there are two neighboring viewpoint images CVk for parameter propagation, the processing for selecting the neighboring viewpoint image CVk for the target viewpoint image is performed as follows.
The number of threads to be parallelized for each viewpoint image is allocated in accordance with, for example, the number of CPUs. Here, the order of viewpoint images to be processed is set in a graph by each thread.

FIG. 10 is a diagram illustrating a thread start viewpoint image when there are two neighboring viewpoint images CVk that perform parameter propagation and the number of threads is two. In the start viewpoint image, the first thread is the viewpoint image at the viewpoint A, and the second thread is the viewpoint image at the viewpoint I.
In FIG. 10, the first thread is set as a target viewpoint image in order of viewpoint A → viewpoint B → viewpoint C → viewpoint D →..., And parameter line-of-sight propagation processing is performed between viewpoint images between adjacent viewpoints. On the other hand, the second thread is set as the target viewpoint image Vk in order of viewpoint I → viewpoint H → viewpoint G → viewpoint F →..., And parameter line-of-sight propagation processing is performed between the viewpoint images between adjacent viewpoints.

  As the starting viewpoint images in each of the first thread and the second thread, the viewpoint images of the farthest viewpoints are selected so that the neighboring viewpoint images are the same and the parameter propagation process does not interfere. That is, the thread generation unit 14 generates a graph (for example, the graph shown in FIG. 9) indicating the selection order of the target viewpoint image from the viewpoint array. Then, when the number of threads is 2, the thread generation unit 14 generates each of the first thread and the second thread. Here, the thread generation unit 14 sets the viewpoint images of the viewpoint A and the viewpoint I, which are both ends that are most distant from each other, as the start viewpoint images of the first thread and the second thread, respectively.

Each of the first thread and the second thread selects the viewpoint image of viewpoint A and the viewpoint image of viewpoint I as the target viewpoint image, respectively, and starts parameter propagation processing.
Further, in each of the first thread and the second thread, the parameter parallelization processing in units of pixels already described is performed independently. In order to prevent the neighboring viewpoint image CVk from interfering, viewpoint images separated by three or more are set as start viewpoint images, respectively.

FIG. 11 is a diagram illustrating a starting viewpoint image of a thread when there are two neighboring viewpoint images that perform parameter propagation and the number of threads is three. In the start viewpoint image, the first thread is the viewpoint image at viewpoint A, the second thread is the viewpoint image at viewpoint I, and the third thread is the viewpoint image at viewpoint E.
In FIG. 10, the first thread is set as a target viewpoint image in order of viewpoint A → viewpoint B → viewpoint C → viewpoint D →..., And parameter line-of-sight propagation processing is performed between viewpoint images between adjacent viewpoints. On the other hand, the second thread is set as the target viewpoint image Vk in order of viewpoint I → viewpoint H → viewpoint G → viewpoint F →..., And parameter line-of-sight propagation processing is performed between the viewpoint images between adjacent viewpoints. In addition, the second thread is set as the target viewpoint image Vk in order of viewpoint E → view point D → view point F → view point C →..., And parameter line-of-sight propagation processing is performed between viewpoint images between adjacent viewpoints.

  As the starting viewpoint images in each of the first thread, the second thread, and the third thread, the viewpoint images of the farthest viewpoints are selected so that the neighboring viewpoint images are the same and the parameter propagation process does not interfere. That is, as in the case of FIG. 10, the thread generation unit 14 generates a graph (for example, the graph shown in FIG. 9) indicating the selection order of the target viewpoint image from the viewpoint arrangement. Then, when the number of threads is 3, the thread generation unit 14 generates each of the first thread, the second thread, and the third thread. Here, the thread generation unit 14 has the viewpoint images of the viewpoint A, the viewpoint I, and the viewpoint E, which are the two end portions that are most distant from each other, as the start viewpoints of the first thread, the second thread, and the third thread, respectively. An image.

Each of the first thread and the second thread selects a viewpoint image of viewpoint A, a viewpoint image of viewpoint I, and a viewpoint image of viewpoint E as target viewpoint images, respectively, and starts parameter propagation processing.
In each of the first thread, the second thread, and the third thread, the parameter propagation parallel processing in units of pixels described above is performed independently. In order to prevent the neighboring viewpoint image CVk from interfering, viewpoint images separated by three or more are set as start viewpoint images, respectively.

<3D shape restoration processing>
The depth map generator 18 stores each of the 3D coordinate points in the 3D coordinate point group (group of 3D coordinate points M) in the world coordinate system based on the parameter table of each viewpoint image stored in the storage unit 19. A depth map for obtaining the coordinate value of is obtained. Then, the depth map generation unit 18 writes the depth map for each generated viewpoint image Vk into the storage unit 19 and stores it, or outputs it to an external three-dimensional shape restoration device (not shown).
In the depth map of the viewpoint image Vk (∈V), the three-dimensional shape restoration apparatus sets the depth of each pixel m to dVk (m), and sets the internal parameters of the imaging device (for example, camera) that images the viewpoint image Vk. When AVk is set and external parameters are RVk (rotation matrix) and tVk (translation vector), the coordinate value of the three-dimensional coordinate point M in the world coordinate system restored from the pixel m (coordinate value) of the viewpoint image Vk is It is represented by the following equation (20).

Then, the three-dimensional shape restoration device restores the coordinate values of the three-dimensional coordinate points M of the pixels m of all the viewpoint images Vk by the above equation (3). The three-dimensional shape restoration apparatus restores the three-dimensional shape by integrating the point groups of the three-dimensional coordinate points obtained from all the viewpoint images Vk.
In addition, the depth map generation device according to the present embodiment may be included in the above three-dimensional shape restoration device, and a depth map may be generated by inputting a viewpoint image, and the three-dimensional shape restoration device may be configured to restore the three-dimensional shape. good.

  According to the present embodiment, as described above, the initial value is generated as a random number, and the larger the number of pixels of the viewpoint image, the larger the number of random numbers given as the initial value, and the surrounding neighboring in the spatial propagation processing Since parameters are propagated to the pixels by parallel processing, a value close to the true value can be given to each pixel at high speed. Thereby, according to this embodiment, the parameter in the pixel in the viewpoint image can be estimated at high speed with high accuracy, and a depth map can be generated with high accuracy.

Further, according to the present embodiment, the initial value is generated as a random number, and as the number of multi-viewpoint images increases, a parameter closer to the true value given as the initial value in the target viewpoint image becomes a near viewpoint by the viewpoint propagation process. Probability of being propagated to the image increases, and parameter spatial propagation processing and viewpoint propagation processing are performed in parallel, so the parameter estimation results for each pixel converge at high speed with fewer iterations than the conventional method. be able to.
In addition, according to the present embodiment, the parameters of a plurality of viewpoint images are propagated by the parallel viewpoint propagation processing, so that the parameters can be quickly and highly accurately applied to an area where the texture change is small in the viewpoint image. It is possible to estimate.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  In addition, a program for realizing each device described above is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed, thereby executing an execution process. May be. Here, the “computer system” may include an OS and hardware such as peripheral devices.

  Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

Further, the “computer-readable recording medium” refers to a volatile memory (for example, DRAM (Dynamic) in a computer system serving as a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Random Access Memory)) that holds a program for a certain period of time is also included.
The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
The program may be for realizing a part of the functions described above.
Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

DESCRIPTION OF SYMBOLS 11 ... Control part 12 ... Initialization part 13 ... Viewpoint selection part 14 ... Thread generation part 15 ... Spatial propagation processing part 16 ... Viewpoint propagation processing part 17 ... Fine adjustment part 18 ... Depth map generation part 19 ... Storage part

Claims (6)

A depth map generation device that creates a depth map used for three-dimensional image restoration using a patch match stereo method,
An initialization unit that generates and initializes a parameter including depth information and normal information in each viewpoint image of all viewpoints used for generation of depth maps by random numbers for each pixel;
A spatial propagation processing unit that propagates the parameter corresponding to a predetermined first evaluation value between adjacent pixels in the target viewpoint image that is an image of the target viewpoint;
A viewpoint propagation processing unit for propagating the parameter in correspondence with a predetermined second evaluation value between pixels of a target viewpoint image and a neighboring viewpoint image that is a neighboring viewpoint image that is a viewpoint near the target viewpoint;
A depth map generation unit that generates a depth map for each viewpoint image from the parameters of a plurality of different viewpoint images, and
The spatial propagation processing unit performs in parallel the process of propagating the parameter from each of the end pixels, which are the pixels to which the parameter has been propagated, to an unfinished pixel that is adjacent to the end pixel and has not been propagated. In addition, the depth map generation device is characterized in that the viewpoint propagation processing unit performs in parallel the process of propagating the parameters of each of the end pixels to the unfinished pixels of the neighboring viewpoint image. The spatial propagation processing unit repeatedly performs parameter propagation processing a predetermined number of times, and when the spatial propagation processing unit performs the propagation process an odd number of times, the predetermined shape in the rectangular shape of the target viewpoint image When the parameter is propagated from the pixel of the first vertex that is the vertex of the target and the number of times of performing the propagation process is an even number, the second vertex at the point-symmetrical position of the first vertex of the target viewpoint The depth map generation apparatus according to claim 1, wherein a process of propagating a parameter from the pixel is performed. For the end pixel for which the parameter propagation processing has been completed in each of the spatial propagation processing unit and the viewpoint propagation processing unit, the parameter fine adjustment is performed by adjusting the parameter propagated to the pixel with an adjustment value generated by a random number. The depth map generating apparatus according to claim 1, further comprising: a unit. The depth map generation device according to any one of claims 1 to 3, wherein the spatial propagation processing unit performs parameter propagation processing on the plurality of target viewpoint images in parallel. A depth map generation method for creating a depth map used for 3D image restoration using a patch match stereo method,
An initialization process in which an initialization unit generates and initializes a parameter including depth information and normal information for each pixel by random numbers in each viewpoint image of all viewpoints used to generate a depth map;
A spatial propagation processing step in which the spatial propagation processing unit propagates the parameter corresponding to a predetermined first evaluation value between adjacent pixels in the target viewpoint image that is an image of the target viewpoint;
A viewpoint in which the viewpoint propagation processing unit propagates the parameter in correspondence with a predetermined second evaluation value between pixels of the target viewpoint image and a neighboring viewpoint image that is a neighboring viewpoint image that is a viewpoint near the target viewpoint. Propagation process,
A depth map generating unit that generates a depth map for each viewpoint image from the parameters of a plurality of different viewpoint images; and
In parallel, the spatial propagation processing unit performs the process of propagating the parameter from each of the propagation end pixels which are the pixels to which the parameter has been propagated, to the unfinished pixels which are adjacent to the end pixel and have not been propagated. The depth map generation method is characterized in that the viewpoint propagation processing unit performs in parallel a process of propagating the parameters of each of the end pixels to the unfinished pixels of the neighboring viewpoint image. A program that causes a computer to execute processing of a depth map generation device that creates a depth map used for three-dimensional image restoration using a patch match stereo method,
In the computer,
Initialization means for generating and initializing parameters including depth information and normal line information for each pixel by random numbers in each viewpoint image of all viewpoints used for generation of the depth map;
Spatial propagation processing means for propagating the parameter corresponding to a predetermined first evaluation value between adjacent pixels in the target viewpoint image that is an image of the target viewpoint;
Viewpoint propagation processing means for propagating the parameter in correspondence with a predetermined second evaluation value between pixels of a target viewpoint image and a neighboring viewpoint image that is a neighboring viewpoint image that is a viewpoint near the target viewpoint;
Operating as a depth map generating means for generating a depth map for each viewpoint image from the parameters of a plurality of different viewpoint images;
The process of propagating parameters from each of the end pixels, which are the pixels to which the parameter has been propagated, to unfinished pixels that have not been propagated adjacent to the respective end pixels is performed in parallel. A program for causing the spatial propagation processing means to perform in parallel the process of propagating the parameters of each of the end pixels to the unfinished pixels of the neighboring viewpoint image.

Images