JP2019220099A - Stereo matching processing device, stereo matching processing method, and program - Google Patents

Abstract

To provide a stereo matching processing device for easily selecting a pair of images capable of accurately calculating a depth value in a multi-viewpoint three-dimensional restoration technique.SOLUTION: The present invention relates to a stereo matching processing device for calculating a depth value of a target object by using multiple images obtained by imaging the target object at mutually different imaging positions. The stereo matching processing device comprises: a first processing section for calculating a first depth value that is a depth value for each pixel in a first reference image by performing stereo matching between the first reference image in the multiple images and a first neighborhood image that is different from the first reference image; a selection section for selecting a second neighborhood image that is different from the first reference image in the multiple images, on the basis of the first depth value; and a second processing section for calculating a second depth value that is a depth value for each pixel in the first reference image, by performing stereo matching between the first reference image and the second neighborhood image.SELECTED DRAWING: Figure 1

Description

  The present invention provides a stereo matching process for generating a three-dimensional shape with higher accuracy in a three-dimensional reconstruction method for generating a three-dimensional shape model using a plurality of images of a target object imaged from different viewpoints (imaging positions). The present invention relates to an apparatus and a stereo matching processing method.

Conventionally, there is a three-dimensional restoration method for generating a three-dimensional shape model of a target object using a plurality of images captured from different viewpoints.
The three-dimensional restoration method includes, for example, a method using a stereo camera. In a stereo camera, an image is captured by each of a plurality of calibrated cameras arranged at regular intervals, and stereo matching is performed using each of the captured images, based on the principle of triangulation. Calculate the depth value of each pixel in the image. Then, a three-dimensional shape model is created by converting the calculated depth value into a point group having three-dimensional position information.

  By the way, when performing the stereo matching, there is a condition of the viewpoint position of the other image in which the depth value can be accurately calculated according to the distance from the viewpoint in one image to the target object. For example, in Non-Patent Document 1, mutual rotation is performed by changing the imaging direction by moving the position of the camera, instead of rotating motion that changes the imaging direction by rotating the camera without moving the position. It is described that the image is captured such that the parallax angle between the viewpoint position of the image and the target object is about 15 degrees. This is because, due to the principle of stereo matching, when the parallax angle is small, the error of the depth value tends to increase. Conversely, if the parallax angle is too large, the mutual image and the target object will not face each other, making image processing difficult and tending to reduce the accuracy of the depth value.

  On the other hand, in a stereo camera, the relative positional relationships and angles (imaging directions) of a plurality of cameras are fixed so as not to be shifted. For this reason, the distance between the viewpoint from which the depth value can be obtained with high accuracy and the target object is limited. As a result, if the distance between the viewpoint and the target object is limited, the target object may not fit in the image captured by the stereo camera depending on the size of the target object, or the target object may be captured smaller than the image. In such a case, the accuracy of the depth value may decrease.

  As described above, in a stereo camera, the relative positional relationship and angles (imaging directions) of a plurality of cameras are fixed in advance. For this reason, the distance between the viewpoint from which the depth value can be obtained with high accuracy and the target object is limited.

  As another three-dimensional restoration method different from the stereo camera, there is a multi-view three-dimensional restoration method. In this method, a three-dimensional shape model is generated based on a plurality of images (hereinafter, referred to as multi-view images) of a target object captured from different viewpoints. In this method, the number of cameras is not limited, and the distance from the viewpoint to the target object is not limited. For example, the target object may be imaged from a different viewpoint while moving one camera. Alternatively, the target object may be imaged by a plurality of cameras from different viewpoints. Therefore, an image captured in an arbitrary direction from an arbitrary viewpoint can be used regardless of the size of the target object, and an image captured of the target object is selected so that the depth value can be calculated accurately. It is possible.

  On the other hand, since a pair of images used for stereo matching is not determined in advance as in the case of a stereo camera, it is necessary to select a pair of images used for stereo matching. Non-Patent Document 2 describes that, for a multi-viewpoint three-dimensional restoration method, an image captured from a viewpoint with a short distance to a target object is selected as a pair of images used for stereo matching.

Yasushi Yagi, et al. "Computer Vision Advanced Guide 5 (multi-view stereo, 3D reconstruction using structured light)" published by Adcom Media Co., Ltd., 2012 M. Goesele, B.S. Curless, S.M. M. See Seitz, "Multi-View Stereo Revised", Proc. of the IEEE 2006

  However, as described in Non-Patent Document 2, when an image captured from a viewpoint with a short distance to the target object is selected as a pair of images used for stereo matching, the parallax angle between the images may be small. is there. When the parallax angle is small, the error of the depth value increases. On the other hand, as described in Non-Patent Document 1, when a pair of images having an appropriate parallax angle relationship is selected, the distance to the captured target object may be different between the selected images. When the distance to the captured target object is short, the target object is imaged larger than the image, and when the distance is long, the target object is imaged smaller than the image. For this reason, when the distance to the captured target object is significantly different between the images, the appearance of the target object between the images changes, and the image is not suitable as a pair of images used for stereo matching.

  The present invention has been made in view of such a situation, and in a multi-view three-dimensional reconstruction method, a stereo matching processing apparatus capable of easily selecting a pair of images from which a depth value can be accurately calculated. , A stereo matching processing method, and a program.

  The stereo matching processing device of the present invention is a stereo matching processing device that calculates a depth value of the target object using a plurality of images of the target object captured from different imaging positions. The stereo matching processing device performs stereo matching between a first reference image in the plurality of images and a first neighboring image different from the first reference image, thereby obtaining a depth value for each pixel in the first reference image. A first processing unit that calculates a certain first depth value; a selection unit that selects a second neighboring image different from the first reference image in the plurality of images based on the first depth value; A second processing unit configured to calculate a second depth value that is a depth value for each pixel in the first reference image by performing stereo matching between a reference image and the second neighboring image.

  In the stereo matching processing device of the present invention, a search for determining a search range in which the first depth value is searched based on an imaging position and an imaging direction in each of the camera parameters of the first reference image and the first neighboring image. The image processing apparatus further includes a range determination unit, wherein the first processing unit calculates a first depth value in the range of the search range determined by the search range determination unit.

  In the stereo matching processing device of the present invention, the search range determination unit may determine the search range based on a range in which the imaging region of the first reference image and the imaging region of the first neighboring image are common. Features.

  In the stereo matching processing device according to the aspect of the invention, the search range determination unit may be configured such that an imaging position of the first reference image and a line-of-sight vector passing through an arbitrary pixel in the first reference image intersect with an imaging region of the first neighboring image. The search range is determined based on a region to be searched.

  In the stereo matching processing device of the present invention, the search range determination unit may include a pixel of the first reference image corresponding to a three-dimensional point group obtained by inputting the plurality of images to SfM (Structure from Motion). The search range is determined based on a depth value.

  In the stereo matching processing device of the present invention, when the search range determination unit sets a part of the imaging region of the first reference image as the search range, the first depth value calculated by the first processing unit When the other search range determined based on the range is smaller than the search range, the other search range is updated as the search range.

  In the stereo matching processing device of the present invention, the search range determination unit may be configured such that the line-of-sight passing through a depth upper limit point at which a depth value reaches a predetermined upper limit in an imaging position of the first reference image and an imaging region of the first reference image. When the parallax angle between the vector and the line of sight passing through the position corresponding to the depth upper limit point in the imaging position of the first neighboring image and the imaging region of the first neighboring image is less than a predetermined parallax angle threshold , The search range is set to no range.

  In the stereo matching processing device according to the aspect of the invention, the selection unit may include a three-dimensional point corresponding to the first depth value and an imaging position of the first reference image in a vector group passing through each imaging position of the plurality of images. An image picked up at an image pick-up position corresponding to a vector having an angle between the vector and a vector passing within the predetermined range is selected as the second neighboring image.

  In the stereo matching processing device of the present invention, the selection unit may include a distance from a three-dimensional point corresponding to the first depth value to an imaging position of the first reference image, and each of the plurality of images from the three-dimensional point. The second neighboring image is selected based on the distance to the imaging position.

  The stereo matching processing apparatus according to the present invention is characterized in that the selecting unit selects the second neighboring image for each pixel in the first reference image.

  The stereo matching processing apparatus according to the present invention is characterized in that the selecting unit selects the second neighboring image for each first reference image.

  In the stereo matching processing device of the present invention, the selection unit may select the second neighboring image based on a predetermined representative pixel in the first reference image and a predetermined representative depth value in the first reference image. It is characterized by.

  In the stereo matching processing device of the present invention, the selection unit sets the representative pixel as a pixel corresponding to an optical center in the first reference image, and sets the representative depth value to a plurality of the second pixels calculated in the first reference image. The second neighborhood image is selected as a median of one depth value.

  The stereo matching processing device of the present invention uses a second reference image different from the first reference image among the plurality of images, and a third neighboring image different from the second reference image included in the plurality of images. And a third processing unit that calculates a third depth value that is a depth value for each pixel in the second reference image by performing stereo matching.

  A stereo matching processing method according to the present invention is a stereo matching processing method for calculating a depth value of a target object by using a plurality of images of the target object captured from different imaging positions. In the stereo matching processing method, the first processing unit performs stereo matching between a first reference image in the plurality of images and a first neighboring image different from the first reference image, so that the first processing unit performs processing on the first reference image. A first processing step of calculating a first depth value, which is a depth value for each pixel, and a selection unit configured to determine, based on the first depth value, a second neighboring image different from the first reference image in the plurality of images. And the second processing unit performs stereo matching between the first reference image and the second neighboring image to obtain a second depth value that is a depth value for each pixel in the first reference image. It is characterized by including a second processing step of calculating.

  A program that causes a computer to operate as a stereo matching processing device that calculates a depth value of the target object by using a plurality of images captured from different imaging positions of the target object. A first processing unit that calculates a first depth value that is a depth value for each pixel in the first reference image by performing stereo matching between the first reference image and a first neighboring image different from the first reference image; Selecting means for selecting a second neighboring image different from the first reference image in the plurality of images based on the first depth value, and performing stereo matching between the first reference image and the second neighboring image. By doing so, a process for operating as second processing means for calculating a second depth value that is a depth value for each pixel in the first reference image is performed. A gram.

  According to the present invention, in a multi-view three-dimensional restoration method, it is possible to easily select a pair of images from which depth values can be calculated with high accuracy.

FIG. 2 is a block diagram illustrating a configuration example of a stereo matching processing device 1 according to the first embodiment. FIG. 3 is a diagram illustrating a process performed by a first processing unit 103 according to the first embodiment. FIG. 4 is a diagram illustrating a process performed by a selection unit 104 according to the first embodiment. FIG. 4 is a diagram illustrating a process performed by a selection unit 104 according to the first embodiment. FIG. 5 is a diagram for describing processing performed by a second processing unit 105 according to the first embodiment. 5 is a flowchart illustrating an operation example performed by the stereo matching processing device 1 according to the first embodiment. It is a block diagram showing an example of composition of 1 A of stereo matching processing units concerning a 2nd embodiment. FIG. 13 is a diagram illustrating a process performed by a third processing unit according to the second embodiment. It is a flow chart which shows an example of operation which stereo matching processing device 1A concerning a 2nd embodiment performs. It is a block diagram showing an example of composition of stereo matching processing device 1B concerning a 3rd embodiment. It is a figure explaining processing which search range deciding part 109 concerning a 3rd embodiment performs. FIG. 14 is a diagram illustrating a process performed by a search range determining unit according to a first modification of the third embodiment. It is a figure showing the example of selection of the pixel which calculates the depth value concerning modification 1 of a 3rd embodiment. FIG. 14 is a diagram illustrating a process performed by a search range determining unit according to a second modification of the third embodiment. FIG. 15 is a diagram illustrating a process performed by a search range determination unit according to a third modification of the third embodiment. FIG. 21 is a diagram for describing processing performed by a search range determination unit 109 according to Modification 4 of the third embodiment. FIG. 15 is a diagram illustrating a process performed by a search range determination unit according to Modification Example 5 of the third embodiment.

  Hereinafter, a stereo matching processing device according to an embodiment will be described with reference to the drawings.

<First embodiment>
First, a first embodiment will be described.
FIG. 1 is a block diagram illustrating a configuration example of a stereo matching processing device 1 according to the first embodiment. The stereo matching processing device 1 includes, for example, an image information acquisition unit 101, a camera parameter estimation unit 102, a first processing unit 103, a selection unit 104, a second processing unit 105, an output unit 106, and an image information storage unit 107.

The image information acquisition unit 101 acquires image information of a multi-view image, and causes the image information storage unit 107 to store the acquired image information.
The camera parameter estimating unit 102 estimates camera parameters using the image information of the multi-viewpoint image. The camera parameter estimating unit 102 obtains an estimated value of a camera parameter by, for example, inputting image information of a multi-view image into a Structure from Motion (SfM) that extracts a feature amount from an image and calculates a corresponding point. I do. The camera parameter estimation unit 102 stores the estimated camera parameters in the image information storage unit 107.

  In SfM, a feature amount of an image is extracted from each of the input multi-viewpoint images, and matching between the extracted feature amounts is performed between the images. In SfM, camera parameters are estimated by performing optimization by bundle adjustment using the matched feature amounts. The camera parameters estimated by SfM include internal parameters and external parameters. Here, the internal parameters are variables related to the inside of the camera, such as the focal length and the optical center of the camera. The external parameters are variables related to the installation environment of the camera, such as the viewpoint (imaging position) and imaging direction of the camera. In SfM, an estimated value of a three-dimensional point group is output based on the matched feature amount in addition to the camera parameters.

  The first processing unit 103 calculates a depth value for each pixel in each of the multi-viewpoint images by performing stereo matching using the multi-viewpoint image in which the camera parameters are specified.

  The first processing unit 103 selects an arbitrary first reference image from the multi-view image. The first reference image is a multi-view image for which a depth value is calculated for each pixel. Further, the first processing unit 103 selects a first neighboring image different from the first reference image from the multi-view image. The first neighboring image is an image on which stereo matching processing is performed as a pair with the first reference image. Here, the first processing unit 103 may select a plurality of first neighboring images. The first processing unit 103 calculates a depth value for each pixel in the first reference image by performing stereo matching using the first reference image and the first neighboring image. Note that, as a method of stereo matching, for example, there is a method using a normalized cross-correlation method, a phase-only correlation method, or the like, and any method may be used. Here, the depth value calculated by the first processing unit 103 is an example of a “first depth value”.

Here, the processing performed by the first processing unit 103 will be described with reference to FIG.
FIG. 2 is a diagram illustrating a process performed by the first processing unit 103 according to the first embodiment. FIG. 2 is a schematic diagram showing a bird's-eye view of the state in which the imaging region E1 is imaged from the viewpoint C1 and the imaging region E2 is imaged from the viewpoint C2.
In this example, an image in which the imaging region E1 is imaged from the viewpoint C1 is a first reference image, and an image in which the imaging region E2 is imaged from the viewpoint C2 is a first neighboring image.
In this example, an XYZ coordinate system is shown as a coordinate system common to each viewpoint. In this XYZ coordinate system, an arbitrary direction on the image plane of the first reference image is shown as an X axis, a direction perpendicular to the X axis is shown as a Y axis, and a depth direction of the first reference image with respect to the image plane is shown as a Z axis. ing. That is, in this case, the direction of the depth value calculated by the first processing unit 103 is the Z-axis direction.
Further, in this example, the depth value zp is calculated by the first processing unit 103 for an arbitrary pixel p in the first reference image, and the position coordinates of the three-dimensional point P corresponding to this pixel p are (Xp, Yp, Zp ). Also, among the multi-view images, the viewpoints C3 and C4 of images different from the first reference image and the first neighboring image are schematically shown.

  As shown in FIG. 2, the first processing unit 103 selects a first reference image and a first neighboring image from the multi-viewpoint images. The first processing unit 103 may arbitrarily select the first reference image and the first neighboring image from the multi-view image, but, for example, the distance between the first reference image and the viewpoint is within a predetermined range. An image having a viewpoint in the region is selected as a first neighboring image.

  Further, the first processing unit 103 uses, for example, normalized cross-correlation in a local region (patch) of the image, SSD (Sum Squared Difference) of the local region of the image, or a phase-only correlation method to determine the depth of the pixel p. A matching score is calculated for all candidate values that can be value candidates. The first processing unit 103 sets the depth value zp that indicates the highest score among the candidate values based on the calculated matching score of the depth value candidate value.

  Returning to FIG. 1, the selection unit 104 selects a second neighboring image different from the first reference image from the multi-viewpoint image based on the depth value calculated by the first processing unit 103. The second neighboring image is an image having an imaging condition for a pixel corresponding to the depth value calculated by the first processing unit 103 so that the depth value of the pixel can be accurately calculated. Here, the imaging condition is determined, for example, according to a relative positional relationship between the position of the viewpoint of the first reference image and the position of the three-dimensional coordinates corresponding to the depth value calculated by the first processing unit 103. The condition is such that an image having a viewpoint in the region is selected as the second neighboring image. The selection unit 104 outputs the identification information of the selected second neighboring image to the second processing unit 105.

Here, the processing performed by the selection unit 104 will be described with reference to FIGS. FIGS. 3 and 4 are diagrams illustrating the processing performed by the selection unit 104 according to the first embodiment.
3 and 4, among the viewpoint C1 (Xc1, Yc1, Zc1) of the first reference image, the viewpoint C7 (Xc7, Yc7, Zc7) of the second neighboring image, and the multi-view image, the first reference image and the first The viewpoints C5 and C6 of an image different from the two neighboring images are shown. Also, a three-dimensional point P (Xp, Yp, Zp) corresponding to a depth value calculated by the first processing unit 103 for a pixel in the first reference image is shown.

  As illustrated in FIG. 3, based on the position coordinates of the three-dimensional point P, the selection unit 104 generates, for example, an image having a viewpoint in a region where the parallax angle γn (n is an arbitrary natural number) is within a predetermined angle range. Select as the second neighborhood image. The parallax angle γn here is an angle between a vector passing through the viewpoint C1 and the three-dimensional point P and a vector passing through the viewpoint Cn and the three-dimensional point P in an arbitrary multi-viewpoint image. For example, the selection unit 104 calculates a line-of-sight vector V1 indicating the direction of the three-dimensional point P from the viewpoint C1, based on the position coordinates of the three-dimensional point P. Further, the selection unit 104 calculates a line-of-sight vector Vn indicating the direction of the three-dimensional point P from the viewpoint Cn (n is an arbitrary natural number other than 1) of another multi-viewpoint image. Then, when the parallax angle between the line-of-sight vectors V1 and Vn is within a predetermined angle range, the selection unit 104 sets the image having the viewpoint Cn corresponding to the line-of-sight vector Vn as the second neighboring image. Here, the line-of-sight vector Vn is an example of a “vector group”.

Alternatively, the selection unit 104 may extract a region where the parallax angle with the line-of-sight vector V1 is within a predetermined angle range, and set an image having a viewpoint inside the extracted region as the second neighboring image.
In this example, when the predetermined angle range includes the parallax angles γ6 and γ7 and is set to a range that does not include the parallax angle γ5, the image captured from the viewpoint C7 corresponding to the parallax angle γ7 and the parallax angle γ6 The image picked up from the viewpoint C <b> 6 is selected as the second neighboring image. Thus, the selection unit 104 may select a plurality of images as the second neighboring images.

  Further, as shown in FIG. 4, based on the position coordinates of the three-dimensional point P, the selection unit 104 sets the distance ratio between the distance L1 and the distance Ln (n is any natural number other than 1) to a predetermined distance ratio. An image having a viewpoint in an area serving as a range may be selected as the second neighboring image. Here, the distance L1 is a distance from the viewpoint C1 to the three-dimensional point P. The distance Ln is a distance from the viewpoint Cn to the three-dimensional point P in an arbitrary multi-viewpoint image. For example, the selection unit 104 calculates the distance Ln based on the position coordinates of the three-dimensional point P and the position coordinates of the viewpoint Cn. Then, when the ratio of the distance Ln to the distance L1 is within the range of the predetermined distance ratio, the selecting unit 104 sets an image captured from the viewpoint Cn corresponding to the distance Ln as a second neighboring image. In this example, assuming that the distances increase in the order of the distances L1, L7, L5, and L6, for example, the range of the predetermined distance ratio includes the ratio of the distance L5 to the distance L1, and includes the ratio of the distance L6 to the distance L1. In a case where the distance is set to a range that is not within the range, an image captured from the viewpoint C7 corresponding to the distance L7 and an image captured from the viewpoint C5 corresponding to the distance L5 are selected as the second neighboring images.

  Here, the case where the selecting unit 104 selects the second neighboring image based on the ratio of the distance Ln to the distance L1 has been described as an example, but the present invention is not limited to this. The selection unit 104 may select the second neighboring image, for example, when the difference between the distance L1 and the distance Ln is less than a predetermined difference threshold.

  Further, the selection unit 104 selects an image corresponding to the viewpoint Cn in which the parallax angle γn is within a predetermined angle range and the distance ratio to the distance L1 is within the predetermined distance ratio as a second neighboring image. You may. Accordingly, the selection unit 104 can select the second neighboring image satisfying the imaging condition in which the parallax angle is within the predetermined angle range and the distance ratio is within the predetermined distance ratio, and can determine the depth value with accuracy. It is possible to calculate well.

  Further, the selection unit 104 may select the second neighboring image for each depth value calculated by the first processing unit 103. In this case, for each three-dimensional point corresponding to the depth value, the selection unit 104 calculates a parallax angle and a distance based on the three-dimensional point for all multi-viewpoint images, and the parallax angle with the three-dimensional point is calculated. An image within a predetermined angle range and / or a distance ratio within a range of the predetermined distance ratio is selected as a second neighboring image. This makes it possible to select a second neighboring image for which a depth value can be calculated with high accuracy for each pixel in the first reference image.

  Alternatively, the selection unit 104 may select one second neighboring image for the first reference image for which the depth value has been calculated. In this case, the selection unit 104 determines the representative pixel and the representative depth value in the first reference image. The selection unit 104 selects an appropriate second neighboring image based on the position coordinates of the three-dimensional point corresponding to the representative pixel and the representative depth value, and replaces the selected second neighboring image with the second neighboring image of the first reference image. And Here, the representative pixel is, for example, a pixel corresponding to the optical center in the first reference image. The representative depth value is, for example, a median value of all the depth values calculated in the first reference image. However, the present invention is not limited to this. The representative pixel may be an arbitrary pixel in the first reference image, and the representative depth value may be calculated using a simple averaging value of all the depth values or other statistical methods. It may be a value calculated by calculation, or a depth value calculated for an arbitrary pixel.

  Returning to FIG. 1, the second processing unit 105 performs a stereo matching using the first reference image and the second neighboring image selected by the selection unit 104 to obtain a depth value for each pixel in the first reference image. calculate. The second processing unit 105 may perform stereo matching by the same method as the method by the first processing unit 103, or may perform stereo matching by a different method. Since the second processing unit 105 uses the second neighboring image that satisfies the imaging condition that allows the depth value to be calculated with high accuracy, the second processing unit 105 can calculate the depth value with high accuracy. Here, the depth value calculated by the second processing unit 105 is an example of a “second depth value”.

  Here, the processing performed by the second processing unit 105 will be described with reference to FIG. FIG. 5 is a diagram illustrating a process performed by the second processing unit 105 according to the first embodiment. FIG. 5 is a schematic view showing a bird's-eye view of the state where the imaging region E1 is imaged from the viewpoint C1 and the imaging region E7 is imaged from the viewpoint C7. In this example, the depth value zp # is calculated by the second processing unit 105 for an arbitrary pixel p in the first reference image, and the position coordinates of the three-dimensional point P corresponding to the pixel p are (Xp, Yp, Zp #). In addition, viewpoints C5 and C6 of images different from the first reference image and the second neighboring image in the multi-view image are schematically illustrated.

  As illustrated in FIG. 5, the second processing unit 105 searches for a depth value for the pixel p of the first reference image for which the depth value zp has been calculated by the first processing unit 103. Specifically, the second processing unit 105 calculates a correlation value between the pixel p and a pixel in the vicinity thereof for all the candidate values that can be candidates for the depth value of the pixel p, and among the candidate values, The one showing the highest correlation is defined as the depth value zp #.

  Here, the second processing unit 105 sets the range in which the depth value is searched to a predetermined range including the depth value zp calculated by the first processing unit 103, for example. Specifically, the second processing unit 105 sets a range included in the depth value zp ± the predetermined value α as a range in which the depth value is searched. The predetermined value α in this case is, for example, a value corresponding to an error amount included in the depth value zp calculated by the first processing unit 103. In this case, since the range of the depth value search is limited, the second processing unit 105 can calculate the correlation value in detail by reducing the depth interval to be searched. In this case, the depth value zp # calculated by the second processing unit 105 has a higher precision than the depth value zp calculated by the first processing unit 103, and Zp at the position of the three-dimensional point P corresponding to the pixel p The coordinate (Zp #) can be set to a highly accurate value.

Returning to FIG. 1, the output unit 106 uses each of the depth values for each pixel calculated by the second processing unit 105 and the camera parameters to calculate the three-dimensional position coordinates of the target object captured in the multi-viewpoint image. Convert to 3D point cloud and output. By using the point group converted by the output unit 106, a three-dimensional shape model of the target object can be generated.
The image information storage unit 107 stores image information of a multi-viewpoint image and camera parameters thereof.

  Here, an operation performed by the stereo matching processing device 1 according to the first embodiment will be described with reference to FIG. FIG. 6 is a flowchart illustrating an operation example performed by the stereo matching processing device 1 according to the first embodiment.

Step S1:
The image information acquisition unit 101 acquires image information of a multi-view image. The image information acquisition unit 101 stores the acquired image information in the image information storage unit 107.
Step S2:
The camera parameter estimating unit 102 estimates camera parameters based on the image information of the multi-view image. The camera parameter estimation unit 102 stores the estimated camera parameters in the image information storage unit 107 in association with the image information.

Step S3:
The first processing unit 103 selects an arbitrary first reference image from the multi-view image.
Step S4:
The first processing unit 103 selects a first neighboring image from the multi-view image. The first processing unit 103 selects, for example, an image having a viewpoint close to the viewpoint of the first reference image among the multi-view images, as the first neighboring image.
Step S5:
The first processing unit 103 calculates a depth value for each pixel in the first reference image by performing stereo matching using the first reference image and the first neighboring image. The first processing unit 103 outputs the calculated depth value (first depth value) to the selection unit 104.
Step S6:
The selecting unit 104 selects a second neighboring image based on the three-dimensional point corresponding to the depth value calculated by the first processing unit 103. For example, the selecting unit 104 determines that the parallax angle with respect to the three-dimensional point is within a predetermined parallax angle, and the distance from the viewpoint to the three-dimensional point is between the viewpoint C1 of the first reference image and the three-dimensional point. An image that is within a range of a predetermined distance ratio with respect to the distance L1 is selected as a second neighboring image.
Step S7:
The second processing unit 105 performs stereo matching using the first reference image and the second neighborhood image selected by the selection unit 104, thereby obtaining a depth value (second depth value) for each pixel in the first reference image. Is calculated.
Step S8:
The output unit 106 converts the pixels for which the depth values have been calculated by the second processing unit 105 into corresponding position coordinates of three-dimensional points.
Step S9:
The stereo matching processing device 1 determines whether to end the stereo matching. For example, when a predetermined end condition is satisfied, the stereo matching processing device 1 determines to end the stereo matching. The predetermined end condition is, for example, a case where a depth value has been calculated for all multi-viewpoint images. When determining that the stereo matching is not to be ended, the stereo matching processing device 1 returns to the process illustrated in step S3, and performs the process of again selecting the first reference image from the multi-view image.

  As described above, the stereo matching processing device 1 according to the first embodiment selects the second neighboring image based on the depth value (first depth value) calculated by the first processing unit 103, and The processing unit 105 calculates a depth value (second depth value) using the second neighborhood image. Thereby, the stereo matching processing device 1 according to the first embodiment can select the second neighboring image based on the depth value calculated by the first processing unit 103, and thus calculates a highly accurate depth value. It is possible to easily select a pair of images that can be created.

(Second embodiment)
Next, a second embodiment will be described. In the following description, only portions different from the above-described embodiment will be described, and the same portions will be denoted by the same reference numerals and description thereof will be omitted.

  In the present embodiment, depth values of a plurality of images are combined in order to reduce errors in camera parameters. FIG. 7 is a block diagram illustrating a configuration example of a stereo matching processing device 1A according to the second embodiment. The stereo matching processing device 1A includes, for example, a third processing unit 108.

  The third processing unit 108 selects a second reference image. The second reference image is an image whose depth value is to be calculated, and is different from the first reference image. The third processing unit 108 selects, for example, an image having a viewpoint near the viewpoint C1 of the first reference image as the second reference image. Alternatively, the third processing unit 108 may select the second reference image based on the depth value calculated by the first processing unit 103. In this case, the third processing unit 108 determines the visibility of the three-dimensional point corresponding to the depth value calculated by the first processing unit 103, and uses an image obtained by capturing the three-dimensional point as a second reference image. select.

  Further, the third processing unit 108 selects a third neighboring image. The third neighboring image is an image different from the second reference image in the multi-viewpoint image, and is an image on which stereo matching processing is performed in combination with the second reference image. The third processing unit 108 selects, for example, an image whose viewpoint is close to the second reference image as the third nearby image, as in the case where the first nearby image is selected by the first processing unit 103. Alternatively, the third processing unit 108 corresponds to the viewpoint position of the second reference image and the depth value calculated by the first processing unit 103, as in the case where the second neighboring image is selected by the selection unit 104. An image having a viewpoint at which the parallax angle γ and the distance L are appropriate may be selected as the third neighboring image according to the relative positional relationship between the positions of the three-dimensional coordinates.

  The third processing unit 108 calculates a depth value for each pixel in the second reference image by performing stereo matching using the second reference image and the third neighboring image. As a method of stereo matching, for example, there are a normalized cross-correlation method, a phase-only correlation method and the like, and any of them may be used. Here, the depth value calculated by the third processing unit 108 is an example of “third depth value”.

  Here, the three-dimensional point corresponding to the depth value calculated by the second processing unit 105 and the three-dimensional point corresponding to the depth value calculated by the third processing unit 108 are actually the same object. May not be the same position coordinates. This is because one of the first reference image, the first neighborhood image, the second neighborhood image, the second reference image, and the third neighborhood image is considered to include an estimation error in the estimated camera parameters. .

  Therefore, the output unit 106 determines the position coordinates of the three-dimensional point corresponding to the depth value for each pixel calculated by the second processing unit 105 and the cubic coordinates corresponding to the depth value for each pixel calculated by the third processing unit 108. A position coordinate of one three-dimensional point is generated by combining the position coordinates of the original point and the generated three-dimensional point is output. The output unit 106 generates position coordinates of one three-dimensional point, for example, by calculating an average value of both position coordinates. As described above, by using the average value, the error of the camera parameter is averaged, and the position coordinates of the three-dimensional point can be calculated with high accuracy.

  Note that the third processing unit 108 may select a plurality of third neighborhood images, or may use the first neighborhood image or the second neighborhood image already used for stereo matching by the first processing unit 103 or the second processing unit 105. The image may be selected as the third neighborhood image.

Here, a process performed by the output unit 106 according to the second embodiment will be described with reference to FIG. FIG. 8 is a diagram illustrating a process performed by the output unit 106 according to the second embodiment. In FIG. 8, a schematic view in which the imaging region E1 is captured from the viewpoint C1, the imaging region E7 is captured from the viewpoint C7, the imaging region E8 is captured from the viewpoint C8, and the imaging region E9 is captured from the viewpoint C9. FIG.
In this example, an image in which the imaging region E1 is captured from the viewpoint C1 is a first reference image, an image in which the imaging region E7 is captured from the viewpoint C7 is a second neighboring image, and the imaging region E8 is captured from the viewpoint C8. The image is defined as a second reference image, and the image obtained by capturing the imaging region E9 from the viewpoint C9 is defined as a third neighboring image.
In this example, the second processing unit 105 indicates a three-dimensional point P (Xp, Yp, Zp #) corresponding to a depth value of an arbitrary pixel in the first reference image. Also, the three-dimensional point P2 (Xp2, Yp2, Zp2) corresponding to the depth value of an arbitrary pixel in the second reference image is shown by the third processing unit 108.

As illustrated in FIG. 8, the output unit 106 outputs the three-dimensional point P corresponding to the depth value for each pixel calculated by the second processing unit 105 and the depth value for each pixel calculated by the third processing unit 108. The three-dimensional point P2 and the corresponding three-dimensional point P2 are combined to obtain one three-dimensional point position coordinate.
The output unit 106 combines the three-dimensional point P and the three-dimensional point P2 when, for example, the distance on the XY plane between the three-dimensional point P and the three-dimensional point P2 is less than a predetermined distance.
The output unit 106 generates a single three-dimensional point by combining the coordinate values of the three-dimensional points by simple averaging. In this case, the position coordinates of the three-dimensional point obtained by combining the three-dimensional point P (Xp, Yp, Zp #) and the three-dimensional point P2 (Xp2, Yp2, Zp2) are ((Xp + Xp2) / 2, (Yp + Yp2) / 2. , (Zp # + Zp2) / 2). Alternatively, the output unit 106 may generate a position coordinate of one three-dimensional point by performing weighting on each coordinate value of the three-dimensional point, and adding and averaging the weights to combine them.

  FIG. 9 is a flowchart illustrating an operation example performed by the stereo matching processing device 1A according to the second embodiment. The processing shown in steps S101 to S107 and step S112 in this flowchart is the same as the processing shown in steps S1 to S7 and step S9 in the flowchart of FIG.

Step S108:
The third processing unit 108 selects a second reference image based on the depth value (first depth value) calculated by the first processing unit 103.
Step S109:
The third processing unit 108 selects a third neighboring image different from the second reference image from the multi-view image. The third processing unit 108 selects, for example, an image having a viewpoint at a position close to the viewpoint of the second reference image as the third neighboring image.
Step S110:
The third processing unit 108 calculates a depth value (third depth value) for each pixel in the second reference image by performing stereo matching using the second reference image and the third neighborhood image.
Step S111:
The output unit 106 combines a plurality of three-dimensional points. The output unit 106 converts the pixel for which the depth value has been calculated by the second processing unit 105 into the position coordinates of the corresponding three-dimensional point. The output unit 106 converts the pixels whose depth values have been calculated by the third processing unit 108 into corresponding position coordinates of a three-dimensional point. Then, the third processing unit 108 combines these three-dimensional points to generate one three-dimensional point.

  As described above, the stereo matching processing device 1A according to the second embodiment combines the three-dimensional point of each pixel in the first reference image and the three-dimensional point of each pixel in the second reference image to generate one Generate three 3D points. For this reason, the accuracy of the three-dimensional points generally deteriorates when there is an estimation error in the camera parameters. However, the stereo matching processing device 1A according to the second embodiment combines the three-dimensional points calculated from a plurality of images. This makes it possible to reduce errors in camera parameters.

(Third embodiment)
Next, a third embodiment will be described. In the following description, only portions different from the above-described embodiment will be described, and the same portions will be denoted by the same reference numerals and description thereof will be omitted.

  In the present embodiment, in order to reduce the processing load caused by calculating the depth value, the range of the depth value searched by the first processing unit 103 (hereinafter, referred to as a search range) is limited. FIG. 10 is a block diagram illustrating a configuration example of a stereo matching processing device 1B according to the third embodiment. The stereo matching processing device 1B includes, for example, a search range determination unit 109.

  The search range determining unit 109 determines a search range based on camera parameters of an image on which stereo matching is performed. The search range determination unit 109 outputs the determined search range to the first processing unit 103.

  The search range determination unit 109 determines a depth value range (search range) based on the camera parameters of the first reference image and the first neighboring image selected by the first processing unit 103. By limiting the search range, it is possible to reduce the processing load involved in calculating the depth value.

  The first processing unit 103 searches for a depth value in the search range determined by the search range determination unit 109. Specifically, the first processing unit 103 assumes that the depth value of the pixel p in the first reference image is a predetermined candidate value in the search range, and calculates and calculates the matching score for all the assumed candidate values. A value indicating the highest score among all the assumed candidate values is defined as a depth value.

  In stereo matching, in principle, the finer (smaller) the width (interval) of the depth search, the higher the accuracy of the depth value can be. However, it is not realistic to make the search step width infinitely small and to perform a search in a wide search range because the amount of calculation becomes enormous. For this reason, for example, it is conceivable that the search range and the step size of the search are set as a trade-off relationship so as not to exceed the upper limit of the calculation amount. In this case, the narrower the search range, the finer the calculation of the step size, so that the accuracy of the depth value can be increased. In the present embodiment, the search range determination unit 109 limits the search range so that the search step width does not become too coarse, so that the depth value can be accurately calculated.

  Here, the processing performed by the search range determination unit 109 will be described with reference to FIG. FIG. 11 is a diagram illustrating a process performed by the search range determination unit 109 according to the third embodiment. FIG. 11 is a schematic diagram showing a bird's-eye view of a state in which the imaging region E1 is imaged from the viewpoint C1 and the imaging region E2 is imaged from the viewpoint C2. In this example, an image in which the imaging region E1 is imaged from the viewpoint C1 is a first reference image, and an image in which the imaging region E2 is imaged from the viewpoint C2 is a first neighboring image.

  As illustrated in FIG. 11, the search range determination unit 109 determines a search range based on, for example, an imaging region E12 that is a region where the imaging regions E1 and E2 are common. Specifically, the search range determination unit 109 calculates a maximum value Dmax (*, *, zmax) and a minimum value Dmin (*, *, zmin) in the z-axis direction (depth direction) in the imaging region E12. Here, “*” indicates an arbitrary coordinate value. The search range determination unit 109 determines a range from the calculated “zmin” to “zmax” as a search range. That is, the search range determination unit 109 determines the search range to be within a range of depth values that can be taken by the object imaged in both the first reference image and the first neighboring image.

  As described above, since the stereo matching processing device 1B according to the third embodiment includes the search range determination unit 109 that limits the search range searched by the first processing unit 103, the stereo matching processing device 1B includes all the candidates that can be depth values. On the other hand, there is no need to calculate a correlation value, and it is possible to reduce processing involved in calculating a depth value.

(Modification 1 of Third Embodiment)
Next, a first modification of the third embodiment will be described. In this modification, the search range determination unit 109 limits the search range of the first reference image based on the search range of the pixels of the first reference image.
The processing performed by the search range determination unit 109 of the present modification will be described with reference to FIGS. FIG. 12 is a diagram illustrating a process performed by the search range determining unit 109 according to the first modification of the third embodiment. FIG. 12 is a schematic diagram showing a bird's-eye view of a state in which the imaging region E1 is imaged from the viewpoint C1 and the imaging region E2 is imaged from the viewpoint C2.
In this example, an image in which the imaging region E1 is imaged from the viewpoint C1 is a first reference image, and an image in which the imaging region E2 is imaged from the viewpoint C2 is a first neighboring image.

  As illustrated in FIG. 12, the search range determination unit 109 sets a range in which the line-of-sight vector V1 intersects the imaging region E2 (hereinafter, referred to as an intersection range) as a search range. Here, the line-of-sight vector V1 is a vector that passes through the viewpoint C1 and an arbitrary pixel in the first reference image. Specifically, the search range determination unit 109 calculates a minimum value Dmin (*, *, zmin) and a maximum value Dmax (*, *, zmax) in the z-axis direction (depth direction) in the intersection range. The search range determination unit 109 determines a range from the calculated minimum value Dmin to the maximum value Dmax, that is, a range from “zmin” to “zmax” as the search range.

  Alternatively, if the range from the maximum value Dmax to the minimum value Dmin is too wide and it is difficult to calculate the correlation values in detail for the entire search range, the search range determination unit 109 sets the minimum value from the maximum value Dmax to the minimum value Dmax. A partial range up to Dmin may be determined as the search range. For example, the search range determining unit 109 determines a search range from D1 (*, *, zave-α) to D2 (*, *, zave + β), that is, a range from “zave-α” to “zave + β”. . Here, zave is the average value of the search range, and α and β are positive real values. The search range determination unit 109 arbitrarily determines α and β according to the processing capability of the calculation processing of the stereo matching processing device 1 (1A, 1B), the size of the range from the maximum value Dmax to the minimum value Dmin, and the like. You may.

  The search range determination unit 109 calculates the maximum value Dmax and the minimum value Dmin for all the pixels in the first reference image by the above-described method, and determines the search range. Accordingly, a search range suitable for each pixel in the first reference image can be determined, so that it is possible to accurately calculate a depth value for all pixels.

  In the above, the search range determination unit 109 has been described as an example in which the maximum value Dmax and the minimum value Dmin are calculated for all the pixels in the first reference image, but the present invention is not limited to this. For example, the search range determination unit 109 may calculate the maximum value Dmax and the minimum value Dmin for a specific pixel selected in the first reference image.

FIG. 13 is a diagram illustrating an example of a pixel for calculating a depth value according to Modification Example 1 of the third embodiment. FIG. 13 schematically shows the first reference image. In this example, a uv coordinate system is shown as a coordinate system indicating the position of an image. In this uv coordinate system, the upper left point in the first reference image is set as the origin, the horizontal direction of the image is set as the u axis, and the vertical direction of the image is set as the v axis.
In this case, as illustrated in FIG. 13, for example, the search range determination unit 109 determines the maximum depth value for each of the pixels p10 to p18 corresponding to the intersections of the straight lines that vertically and horizontally divide the first reference image into four. Dmax and the minimum value Dmin are respectively extracted. For example, the search range determination unit 109 determines a search range for each pixel based on an average value, a maximum value, and a minimum value of the extracted plurality of depth values. For example, the search range determination unit 109 sets the upper limit of the search range to ((average value) + (maximum value−minimum value) × variable) and sets the lower limit of the search range to ((average value) − (maximum value−minimum value)). × variable). The variable here is any positive real number of 0.5 or less. Thereby, even if the range from the minimum value to the maximum value is too wide and the depth value is calculated in the entire range, the depth value may not be calculated with high accuracy, the range is narrowed. This makes it possible to calculate the depth value with high accuracy.

  Then, the search range determination unit 109 sets the maximum value among the upper limit of the search range and the lower limit of the search range for each pixel calculated for each of the pixels p10 to p18 to the upper limit and the minimum of the search range in the first reference image. The value is the lower limit of the search range. Alternatively, a value obtained by adding a predetermined margin to each of the maximum value and the minimum value among the calculated values may be set as the upper limit and the lower limit of the search range. Thereby, the search range determination unit 109 can reduce the processing load for calculating the search range as compared with the case where the search range is determined for all the pixels in the first reference image.

  In the above description, a case where nine pixels in the first reference image are selected has been described as an example. However, the present invention is not limited to this, and the search range determination unit 109 may perform more than this example. Pixels may be selected, or a smaller number of pixels than in this example may be selected. Further, the search range determining unit 109 may select a pixel at an arbitrary position in the first reference image.

  As described above, the stereo matching processing device 1B according to the first modification of the third embodiment is based on the line-of-sight vector V1 indicating the direction of the pixel in the first reference image and the imaging area E2 of the first neighboring image. To determine the search range. Thus, the search range can be determined for each pixel in the first reference image, and the depth value can be calculated with high accuracy. Further, the entire search range of the first reference image can be determined based on the search range of the pixel selected in the first reference image, and the depth value can be accurately calculated without increasing the processing load. Becomes possible.

(Modification 2 of Third Embodiment)
Next, a second modification of the third embodiment will be described. In this modification, the search range determination unit 109 limits the search range based on a predetermined upper limit of the search range.
The processing performed by the search range determination unit 109 of the present modification will be described with reference to FIG. FIG. 14 is a diagram illustrating a process performed by the search range determination unit 109 according to the second modification of the third embodiment. FIG. 14 is a schematic diagram showing a bird's-eye view of a state in which the imaging region E1 is imaged from the viewpoint C1 and the imaging region E2 is imaged from the viewpoint C2, as in FIG. Further, an image in which the imaging region E1 is captured from the viewpoint C1 is a first reference image, and an image in which the imaging region E2 is captured from the viewpoint C2 is a first neighboring image.

As illustrated in FIG. 14, the search range determination unit 109 determines a search range with an upper limit Dmaxth (*, *, zmaxth) that is a predetermined upper limit of the search range as an upper limit.
Thereby, in the stereo matching processing device 1B according to the second modification of the third embodiment, the line-of-sight vector V1 corresponding to the optical center in the first reference image and the line-of-sight vector V2 corresponding to the optical center in the first neighboring image are obtained. Are substantially parallel, the upper limit value Dmaxth can be set as the upper limit even if the maximum value in the depth direction in the common region of the imaging regions E1 and E2 is infinite. Note that the upper limit value Dmaxth here may be arbitrarily determined according to the processing capacity of the calculation processing of the stereo matching processing device 1 (1A, 1B).

(Modification 3 of Third Embodiment)
Next, a third modification of the third embodiment will be described. In this modification, the search range determination unit 109 limits the search range based on the parallax angle.
The processing performed by the search range determination unit 109 of the present modification will be described with reference to FIG. FIG. 15 is a diagram illustrating a process performed by the search range determining unit 109 according to the third modification of the third embodiment. FIG. 15 is a schematic diagram showing a bird's-eye view of a state in which the imaging region E1 is imaged from the viewpoint C1 and the imaging region E2 is imaged from the viewpoint C2, similarly to FIG. In this example, an image in which the imaging region E1 is captured from the viewpoint C1 is a first reference image, and an image in which the imaging region E2 is captured from the viewpoint C2 is a first neighboring image.

  As illustrated in FIG. 15, the search range determination unit 109 determines a search range based on a parallax angle threshold θth that is a predetermined parallax angle threshold. Specifically, the search range determination unit 109 determines, for a point q at which the line-of-sight vector V1 (optical axis) corresponding to the optical center of the first reference image and the upper limit Dmaxth intersect, the line-of-sight vector V1, the viewpoint C2, When the angle (parallax angle) between the vector and the vector M passing through the point q is smaller than the parallax angle threshold θth, the search range is set to “no range”, that is, 0 (zero). Here, the point q is an example of the “depth upper limit point”.

  When the search range is 0 (zero), the first processing unit 103 does not calculate the depth value. That is, the first processing unit 103 does not perform a process of searching for a depth value. Note that the parallax angle threshold θth here may be arbitrarily determined according to the processing capability of the calculation processing of the stereo matching processing device 1 (1A, 1B).

  Thereby, the stereo matching processing device 1B according to the third modification of the third embodiment does not search for a depth value when satisfying a predetermined condition that makes it difficult to calculate a depth value with high accuracy. It is possible to reduce the processing load for searching for the depth value. Here, the predetermined condition means that, when a predetermined upper limit value Dmaxth is determined and the depth value of the pixel is located at the position of the upper limit value Dmaxth, the parallax angle of the pixel with respect to the viewpoints C1 and C2 is a predetermined value. This is the case where it is less than the parallax angle threshold θth. In this case, since the parallax angle is too small, it is difficult to accurately calculate the depth value.

(Modification 4 of Third Embodiment)
Next, a fourth modification of the third embodiment will be described. In this modification, the search range determination unit 109 limits the search range based on the result of SfM.

  The processing performed by the search range determination unit 109 of the present modification will be described with reference to FIG. FIG. 16 is a diagram illustrating a process performed by the search range determination unit 109 according to Modification 4 of the third embodiment. FIG. 16 is a schematic diagram in which an image obtained by imaging the imaging region E1 from the viewpoint C1 is used as a first reference image, and a point cloud of three-dimensional points output from SfM for the first reference image is overlooked.

  As illustrated in FIG. 16, the search range determination unit 109 determines the minimum values Dmin (*, *, Zmin) that are minimum in the depth direction (Z-axis direction) among the three-dimensional point group output from SfM. And the maximum value Dmax (*, *, Zmax) that becomes the maximum in the Z-axis direction is calculated. The search range determination unit 109 provides a predetermined margin between the calculated minimum value Dmin and the maximum value Dmax, that is, a range from “zmin” to “zmax” or a range from “zmin” to “zmax”. The determined range is determined as a search range. The margin here may be determined according to the processing capacity of the calculation processing of the stereo matching processing device 1 (1A, 1B), the size of the range from the maximum value Dmax to the minimum value Dmin, and the like. For example, the search range determination unit 109 sets the search range from s times “zmax” to 1 / s times “zmin”. Here, s is one or more arbitrary real numbers.

  As described above, the stereo matching processing device 1B according to the fourth modification of the third embodiment uses the result of SfM. Thereby, in the stereo matching processing device 1B according to the fourth modification of the third embodiment, it is possible to easily determine the search range.

(Modification 5 of Third Embodiment)
Next, a fifth modification of the third embodiment will be described. In this modification, the search range determination unit 109 updates the search range.

  The processing performed by the search range determination unit 109 of the present modification will be described with reference to FIG. FIG. 17 is a diagram illustrating a process performed by the search range determination unit 109 according to the fifth modification of the third embodiment. FIG. 17 shows a plurality of schematic diagrams in which a point group of three-dimensional points output as a result of stereo matching is overlooked for the first reference image captured from the viewpoint C1. The left side of FIG. 17 illustrates a point group of three-dimensional points output as a result of performing the stereo matching in the first search range F1. The center of FIG. 17 shows the updated search range F2. The right side of FIG. 17 shows a point group of three-dimensional points output as a result of performing stereo matching in the first search range F1 and as a result of performing stereo matching in the updated search range F2. .

  As shown in FIG. 17, the search range determining unit 109 first determines the search range F1. The search range determination unit 109 may determine the first search range F1 by using any of the methods described in the third embodiment and its modifications. In this case, the first processing unit 103 calculates the depth value in the search range F1 determined by the search range determination unit 109.

  As described above, the stereo matching processing device 1 (1A, 1B) repeatedly performs the calculation of the depth value by the first processing unit 103 until a predetermined end condition for ending the stereo matching is satisfied (step in FIG. 6). S9, and step S112 in FIG. 9).

  When the depth value is calculated by the first processing unit 103 after the second time, the search range determination unit 109 determines the point group of the three-dimensional point corresponding to the depth value calculated by the first processing unit 103 before the previous time. The search range is updated based on the position coordinates. The search range determination unit 109 sets, for example, a range obtained by adding a predetermined margin to the range from the minimum value to the maximum value of the position coordinates of the three-dimensional point group as the updated search range. In this case, the search range determination unit 109 may update the search range when the updated search range F2 is smaller than the first search range F1. In this case, the search range F2 is an example of “another search range”.

  As described above, the stereo matching processing device 1B according to the fifth modification of the third embodiment updates the search range. Accordingly, in the stereo matching processing device 1B according to the fifth modification of the third embodiment, the processing load involved in calculating the depth value is reduced or the accuracy of the calculated depth value is increased by narrowing the search range. It is possible to do.

  All or part of the stereo matching processing device 1 (1A, 1B) in the above-described embodiment may be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read and executed by a computer system. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, and a storage device such as a hard disk built in a computer system. Further, a “computer-readable recording medium” refers to a communication line for transmitting a program via a network such as the Internet or a communication line such as a telephone line, which dynamically holds the program for a short time. Such a program may include a program that holds a program for a certain period of time, such as a volatile memory in a computer system serving as a server or a client in that case. The program may be for realizing a part of the functions described above, or may be a program that can realize the functions described above in combination with a program already recorded in a computer system, It may be realized using a programmable logic device such as an FPGA.

  As described above, the embodiments of the present invention have been described in detail with reference to the drawings. However, the specific configuration is not limited to the embodiments, and includes a design and the like within a range not departing from the gist of the present invention.

1, 1A, 1B ... stereo matching processing device 101 ... image information acquisition unit 102 ... camera parameter estimation unit 103 ... first processing unit 104 ... selection unit 105 ... second processing unit 106 ... output unit 107 ... image information storage unit 108 ... Third processing unit 109: search range determination unit

Claims (16)

A stereo matching processing device that calculates a depth value of the target object using a plurality of images of the target object captured from different imaging positions,
Calculating a first depth value that is a depth value for each pixel in the first reference image by performing stereo matching between a first reference image in the plurality of images and a first neighboring image different from the first reference image. A first processing unit,
A selection unit that selects a second neighboring image different from the first reference image in the plurality of images based on the first depth value;
A second processing unit that calculates a second depth value that is a depth value for each pixel in the first reference image by performing stereo matching between the first reference image and the second neighborhood image. Stereo matching processing device. A search range determining unit that determines a search range in which the first depth value is searched, based on an imaging position and an imaging direction in each of the camera parameters of the first reference image and the first neighboring image,
The stereo matching processing device according to claim 1, wherein the first processing unit calculates a first depth value within a range of the search range determined by the search range determination unit. The said search range determination part determines the said search range based on the range where the imaging area of the said 1st reference image and the imaging area of the said 1st vicinity image are common. Stereo matching processing device. The search range determination unit is configured to determine the search range based on a region where an imaging position of the first reference image and a line of sight passing through any pixel in the first reference image intersect with an imaging region of the first neighboring image. The stereo matching processing device according to claim 2, wherein: The search range determination unit is configured to determine the search range based on a depth value of a pixel of the first reference image corresponding to a three-dimensional point group obtained by inputting the plurality of images to SfM (Structure from Motion). The stereo matching processing device according to claim 2, wherein: The search range determination unit is configured to perform another search determined based on the first depth value calculated by the first processing unit when a part of an imaging region of the first reference image is set as the search range. The stereo matching processing device according to claim 2, wherein when the range is smaller than the search range, the other search range is updated as the search range. The search range determination unit is configured to include: an imaging position of the first reference image; a line-of-sight vector that passes through a depth upper limit at which a depth value is a predetermined upper limit in an imaging region of the first reference image; When the parallax angle between the imaging position and the gaze vector passing through the position corresponding to the depth upper limit point in the imaging region of the first neighboring image is less than a predetermined parallax angle threshold, the search range is set to no range. The stereo matching processing device according to claim 2, wherein: The selecting unit may include, among a vector group passing through the three-dimensional point corresponding to the first depth value and the imaging position of each of the plurality of images, an angle between the vector and the vector passing through the imaging position of the first reference image. The stereo matching processing device according to any one of claims 1 to 7, wherein an image captured at an imaging position corresponding to a vector within a predetermined range is selected as the second neighboring image. The selecting unit is configured to determine a distance from a three-dimensional point corresponding to the first depth value to an imaging position of the first reference image and a distance from the three-dimensional point to an imaging position of each of the plurality of images. The stereo matching processing device according to any one of claims 1 to 8, wherein the second neighboring image is selected. The stereo matching processing device according to claim 1, wherein the selection unit selects the second neighboring image for each pixel in a first reference image. The stereo matching processing device according to any one of claims 1 to 9, wherein the selection unit selects the second neighboring image for each first reference image. The stereo matching processing device according to claim 11, wherein the selection unit selects the second neighboring image based on a predetermined representative pixel and a predetermined representative depth value in the first reference image. The selecting unit, the representative pixel as a pixel corresponding to the optical center in the first reference image, the representative depth value as a median of the plurality of first depth values calculated in the first reference image, The stereo matching processing device according to claim 12, wherein the second neighborhood image is selected. By performing stereo matching using a second reference image different from the first reference image among the plurality of images and a third neighboring image different from the second reference image included in the plurality of images, The stereo matching processing device according to any one of claims 1 to 13, further comprising: a third processing unit that calculates a third depth value that is a depth value for each pixel in the two reference images. A stereo matching processing method for calculating a depth value of the target object using a plurality of images of the target object captured from different imaging positions,
The first processing unit performs stereo matching between a first reference image in the plurality of images and a first neighboring image different from the first reference image to obtain a depth value for each pixel in the first reference image. A first processing step of calculating a first depth value;
A selecting unit that selects a second neighboring image different from the first reference image in the plurality of images based on the first depth value;
A second processing step in which the second processing unit calculates a second depth value that is a depth value for each pixel in the first reference image by performing stereo matching between the first reference image and the second neighboring image. A stereo matching processing method comprising: A program that operates a computer as a stereo matching processing device that calculates a depth value of the target object using a plurality of images of the target object captured from different imaging positions,
Said computer,
Calculating a first depth value that is a depth value for each pixel in the first reference image by performing stereo matching between a first reference image in the plurality of images and a first neighboring image different from the first reference image. First processing means for performing
Selecting means for selecting a second neighboring image different from the first reference image in the plurality of images based on the first depth value;
A program for operating as a second processing unit that calculates a second depth value that is a depth value for each pixel in the first reference image by performing stereo matching between the first reference image and the second neighboring image.

Images