JP2019091122A - Depth map filter processing device, depth map filter processing method and program - Google Patents

Abstract

To provide a depth map filter processing device which corrects a depth value of a cluster-shaped pixel, and corrects an incorrect depth value of a pixel near a surface of a body, without changing a correct depth value of a depth map into the incorrect depth value.SOLUTION: A depth map filter processing device comprises: a consistency determination unit which compares pixel information of corresponding pixels of each of different depth maps, and determines the presence or absence of consistency between the pixels; a penalty score integration unit which gives a penalty score to the pixel determined that the consistency does not exist; an integrated value determination unit which determines whether or not the integrated value of the penalty score exceeds a predetermined first threshold, after the consistency of the pixel information between the pixels in all combinations of each of the depth maps is determined, and generates a penalty score map indicating the penalty score of each of the pixels; and a depth value correction unit which corrects the depth value of the pixel of the integrated value exceeding the first threshold in the penalty score map.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a depth map filter processing apparatus for correcting an incorrect depth value (a depth value different from an actual value) in a depth map of a plurality of captured images in which an object is captured from different viewpoints, a depth map The present invention relates to a filter processing method and program.

  There is a three-dimensional reconstruction method of generating a three-dimensional shape model of an object using a plurality of captured images obtained by imaging the object. In this three-dimensional restoration method, a depth map in which distance information (depth value) to an object when observed from an arbitrary viewpoint (an imaging position of a captured image) is recorded as a two-dimensional image is used.

For example, as one of the three-dimensional reconstruction methods, a three-dimensional shape model of the object is obtained based on a plurality of two-dimensional captured images (hereinafter referred to as "multi-viewpoint images") obtained by imaging the object from different viewpoints. There is a multi-view three-dimensional reconstruction method to generate (see, for example, Patent Document 1).
In this multi-view three-dimensional reconstruction method, by performing stereo matching between multi-view images, a depth map at each view is generated, and a depth map at all the views is integrated to obtain a three-dimensional shape model of the object. Restore.

In addition, as a method of acquiring a depth map other than stereo matching, the depth map is directly obtained as a two-dimensional image by a depth camera, an RGB (red green blue) -D (depth) camera, or the like.
The RGB-D camera is a special camera that captures a combination of a color image and a depth map. Moreover, as a typical depth camera, a Time of Flight distance image sensor and an active stereo sensor can be mentioned. Each of the Time of Flight distance image sensor and the active stereo sensor corresponds to the object by irradiating the object with the laser light and the pattern light (or slit light) and observing the reflected light. Generate a depth map.

The distance to the object from the plurality of viewpoints is measured using such depth camera or RGB-D camera, and the depth map corresponding to the object is acquired.
The three-dimensional shape of the object can also be generated by integrating the depth maps corresponding to the object acquired at a plurality of viewpoints.

As described above, in the three-dimensional reconstruction method, when generating a three-dimensional shape model, a depth map may be used as intermediate data or acquired data in reconstruction of the three-dimensional shape model. For this reason, each of the accuracy of depth values in the depth map and the quality of the entire depth map is an important factor that affects the accuracy and quality of the three-dimensional shape model to be restored.
In order to restore a high-precision and high-quality three-dimensional shape model, it is necessary to generate or obtain a depth map with high accuracy and high quality.

  On the other hand, even if the depth map is acquired with high accuracy and high quality, the acquired depth map may include an inaccurate depth value. For example, in the multi-view three-dimensional reconstruction method described above, when stereo matching between multi-view images in a repetitive pattern area or textureless area fails, a large error occurs in the depth values of pixels in that area, and depth values different from actual You will get

  Further, in the depth map, it is difficult to obtain an accurate depth value for the contour portion indicating the object boundary of the target object in the captured image. For this reason, in the obtained depth map, the depth value in the object boundary region of the target object becomes an inaccurate numerical value, and the error of the depth value tends to be large.

  On the other hand, in the case of using a depth camera or an RGB-D camera, the object is viewed from the viewpoint by total reflection / diffuse reflection of light emitted by the object, or disturbance from the surrounding environment such as high illuminance such as sunlight. It is not possible to accurately measure the distance to For this reason, an error occurs in the depth value for the object.

As described above, due to various factors, errors may occur in depth values in the depth map, and the quality of the depth map is degraded.
As described above, in the case of stereo matching with a plurality of captured images, or using a depth camera or an RGB-D camera, it is difficult to obtain a high quality depth map that does not include inaccurate depth values.
Therefore, in the three-dimensional reconstruction method, when using a depth map for generation of a three-dimensional shape model, it is necessary to apply a filtering process to correct an incorrect depth value to depth maps obtained by various methods. .

As one of the filtering processes for correcting the incorrect depth value of the depth map, a denoising filter for denoising in a two-dimensional color image, such as a bilateral filter or a median filter, is used.
Also, as another filtering process for the depth map, there may be mentioned isolated point removal when each of the pixels in the depth map is projected onto a three-dimensional space. The isolated point removal generates a three-dimensional point group corresponding to the object using multi-view depth maps. Then, when the number of other three-dimensional points existing within a predetermined range centered on each of the three-dimensional points in the generated three-dimensional point group is less than a predetermined threshold, the three-dimensional point in the depth map Correct the depth value of the selected pixel to improve the quality of the depth map.

JP, 2017-027101, A

As described above, when applying a noise removal filter of a two-dimensional color image, when noise is superimposed on the depth value of any pixel that occurs suddenly in the depth map, for example, the depth value in this pixel unit is used. It can be corrected.
However, in the depth map, if noise is superimposed on the depth values of a plurality of pixels in a cluster (region including a plurality of adjacent pixels) units, these incorrect depth values can not be corrected.

  On the other hand, when the above noise removal filter is used, the correction is performed by comparing with the depth values of surrounding pixels, so even if it is a pixel with the correct depth value, it is greatly different from the depth values of the surrounding pixels in the depth map , This correct depth value will be changed to an incorrect depth value. In addition, since the noise removal filter corrects the depth values of all the pixels in the depth map, the depth values of the respective pixels tend to be smoothed, which may degrade the quality of the depth map.

In addition, when generating a three-dimensional point group and removing an isolated three-dimensional point, it is possible to correct the isolated three-dimensional point, but when noise is superimposed on a cluster-like three-dimensional point in three-dimensional space, Since it is not considered as an isolated point group, it is difficult to correct the depth value of the cluster-like pixel on which the noise is superimposed.
Therefore, by setting a large number of pixels as the threshold value, it is possible to correspond to the depth value generated in cluster-like pixels having a certain number of pixels. On the other hand, increasing the number of threshold pixels increases the possibility of changing the accurate pixel depth value to an incorrect depth value.

  In addition, in the removal of isolated points in the three-dimensional space, it is possible to cope with the correction of the depth value of the pixel at a large distance from the surface of the target object. However, it is not possible to cope with the correction of the depth value of a pixel (pixel near the surface) whose distance from the surface of the target object is small.

  The present invention has been made in view of such a situation, and it is possible to correct the depth value of the cluster-like pixel without changing the correct depth value in the depth map to the incorrect depth value, and the pixel near the object surface Abstract: A depth map filtering device, a depth map filtering method, and a program capable of correcting an incorrect depth value of

  In order to solve the problems described above, the depth map filter processing device of the present invention compares the pixel information of each corresponding pixel of different depth maps, and determines the presence / absence of consistency between the pixels. And a penalty score integration unit that gives a penalty score to pixels determined not to be consistent, and after the determination of the consistency of the pixel information between pixels in all combinations of the depth map is completed, An integrated value determination unit that determines whether or not the integrated value of the penalty score exceeds a predetermined first threshold, and generates a penalty score map in which the penalty score of each of the pixels is indicated; And a depth value correction unit configured to correct the depth value of the pixel of the integrated value exceeding the first threshold.

  In the depth map filtering device of the present invention, when the integrated value determination unit determines that the integrated value exceeds the first threshold, the determination of consistency between pixels according to the depth map after correction of the depth value is performed. It is characterized in that the process is performed on the consistency determination unit to continue the correction process of the depth map.

  The depth map filtering apparatus of the present invention generates a three-dimensional point from the depth value of the target pixel of one depth map in the pair to be compared and the camera parameter of the one depth map, and the three-dimensional point is generated in the other depth map. And a reference pixel extraction unit that extracts reference pixel information that is the pixel information of the corresponding reference pixel in the other depth map, and the consistency determination unit determines the reference pixel information and the target pixel. And comparing the target pixel information, which is the pixel information, and performing the determination of the consistency based on whether the reference pixel information and the target pixel information are less than a predetermined second threshold value. .

  The depth map filter processing device of the present invention is characterized in that the pixel information is a depth value of the pixel.

  The depth map filter processing device of the present invention is characterized in that the pixel information is a pixel value of the pixel.

  The depth map filtering apparatus of the present invention is characterized in that the pixel information is each of a depth value and a pixel value of the pixel.

  In the depth map filtering device of the present invention, the depth value correction unit refers to the penalty score map, and the pixel having the maximum integrated value in the pixels of the integrated value exceeding the first threshold is selected. It is characterized in that it is extracted as a maximum value pixel and the depth value of the maximum value pixel is corrected.

  In the depth map filtering device of the present invention, when the depth value correction unit corrects the depth value of the pixel, the depth value correction unit generates a three-dimensional point of the three-dimensional shape model from the depth map. Changing the depth value of the pixel to an incorrect value to be ignored in the calculation of generation.

  In the depth map filtering method according to the present invention, the consistency determining unit compares pixel information of corresponding pixels of different depth maps, and determines whether or not there is a consistency between the pixels, and a penalty. A penalty score integration process in which a score integration unit integrates a predetermined penalty score with respect to pixels determined to be inconsistent, and an integrated value determination unit determines between pixels in all combinations of the depth map. After completion of the determination of the consistency of the pixel information, it is determined whether or not the integrated value of the penalty score exceeds a predetermined first threshold to generate a penalty score map in which the penalty score of each of the pixels is indicated. In the integrated value determination process, the depth value correction unit determines the depth of the pixel of the integrated value exceeding the first threshold in the penalty score map. Characterized in that it comprises a depth value correction step of correcting the.

  A program according to the present invention compares a pixel information of corresponding pixels of different depth maps with a computer, and determines whether or not there is a match between the pixels. A penalty score integrating means for giving a penalty score to the image, and after the determination of the consistency of the pixel information among the pixels in all combinations of the depth map is completed, the integrated value of the penalty score is a predetermined first threshold value Integrated value determining means for determining whether or not the penalty score is exceeded and generating a penalty score map in which the penalty score of each of the pixels is indicated; the pixel of the integrated value exceeding the first threshold in the penalty score map It is a program for operating as depth value correction means which corrects a depth value.

  As described above, according to the present invention, without changing the correct depth value in the depth map to an incorrect depth value, the correction of the depth value of the cluster-like pixel, the incorrectness of the pixel near the object surface It is possible to provide a depth map filter processing device, depth map filter processing method and program capable of correcting depth values.

It is a block diagram showing an example of composition of a depth map filter processing unit by one embodiment of the present invention. FIG. 6 is a diagram showing an example of the configuration of a depth map table in a depth map storage unit 107. It is a conceptual diagram explaining inhibition of visibility of each of a three-dimensional point M and a three-dimensional point generated from a depth value Dk (xp). It is explanatory drawing of the process which adds the penalty by consistency determination with depth value ZM of object pixel xj, and depth value Dk (xp) of reference pixel xp. It is a figure which shows the structural example of the penalty score map table memorize | stored in the penalty score map memory | storage part 108. FIG. It is a flowchart which shows the operation example of the process which corrects the depth value of the outlier in the depth map which the depth map filter processing apparatus 1 by this embodiment performs. It is a figure explaining correction processing of a correction object pixel which has a penalty score which has the maximum value. It is a figure which shows the image which decompress | restored the three-dimensional point group of the target object based on the depth map produced | generated from the multi-viewpoint image of the target object.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing a configuration example of a depth map filter processing device according to an embodiment of the present invention. In FIG. 1, the depth map filter processing apparatus 1 includes a depth map input unit 101, a reference pixel extraction unit 102, a consistency determination unit 103, a penalty score integration unit 104, an integration value determination unit 105, a depth value correction unit 106, and a depth map. A storage unit 107 and a penalty score map storage unit 108 are provided. Here, the depth map is a camera parameter from the viewpoint position (viewpoint coordinates in a three-dimensional coordinate space) and the depth (depth) from the position in the viewpoint direction to the target object for each pixel from white (close to the viewpoint position) It is represented by gray scale to black (far from the viewpoint position).

The depth map input unit 101 inputs a depth map group used to generate a three-dimensional shape model of a target object, which is supplied from an external device. The depth map group is composed of depth maps corresponding to each of a plurality of camera parameters (described later) for the target object.
Further, the depth map input unit 101 adds depth map identification information, which is identification information for identifying the depth map, to each of the depth maps in the depth map group, and writes and stores the depth map identification information in the depth map storage unit 107.

  FIG. 2 is a view showing a configuration example of the depth map table in the depth map storage unit 107. As shown in FIG. The depth table in FIG. 2 is provided with fields of depth map identification information, camera parameters, and depth map index for each record. The depth map identification information is identification information for identifying each of the depth maps in the depth map group. The camera parameters are various parameters of an imaging device that captures a captured image (including an image acquired directly as a depth map) corresponding to a depth map, including viewpoint coordinates and a viewpoint direction. The viewpoint coordinates indicate the coordinate position in the three-dimensional coordinate space where the target object is located. The viewpoint direction indicates the observation direction from viewpoint coordinates in the three-dimensional coordinate space where the target object is located. The camera parameters include external parameters and internal parameters, and the external parameters are the center coordinates of the lens in the world coordinate system, the direction of the optical axis of the lens, etc. The internal parameters are the focal length, the image center, the resolution of the image (pixels Number, distortion aberration coefficient, and the like.

  Also, the depth map index indicates an address in the depth map storage unit 107 in which data of the depth map is stored. In the area of this address, a map in which a depth value (depth value) is described in pixel units is written and stored as depth map data. That is, the depth map is a two-dimensional map in which distance information (depth value) from the viewpoint to the target portion for each pixel is stored for image coordinates virtually set in each viewpoint. For example, the input depth map is a depth map generated by stereo matching in multi-view three-dimensional restoration, a depth map captured by a depth camera or an RGB-D camera, or the like.

  Referring back to FIG. 1, the reference pixel extraction unit 102 refers to the depth map table of the depth map storage unit 107, and the target depth map Di (corresponding to the viewpoint i), which is the depth map to be corrected A combination with a reference depth map Dk (corresponding to the viewpoint k) which is a depth map to be referred to for comparison with the map Di is extracted. Then, the reference pixel extraction unit 102 calculates the depth value Di (xj) of each of the target pixel xj (any of n pixels included in the target depth map Di, 1 ≦ j ≦ n) which is a pixel in the target depth map Di. And camera parameters to generate a three-dimensional point M corresponding to each pixel in the three-dimensional coordinate space.

In addition, the reference pixel extraction unit 102 projects the generated three-dimensional point onto the reference depth map Dk, and the reference pixels xp (n in the reference depth map Dk) are pixels in the reference depth map Dk at the projected position. 1), and 1 ≦ p ≦ n) is extracted.
Here, the reference pixel extraction unit 102 generates a three-dimensional point from the depth values of all the target pixels in the target depth map Di, and projects each of the three-dimensional points on the reference depth map, and is at the projected position. Extract reference pixels. Here, the reference pixel extraction unit 102 extracts each combination of all the depth maps stored in the depth map storage unit 107, and extracts each of the reference pixels xp. At this time, when the three-dimensional point generated from the depth value in the target depth map Di is not projected in the reference depth map, the reference pixel extraction unit 102 does not extract the reference pixel corresponding to the three-dimensional point not projected.

The consistency determination unit 103 obtains the distance between the three-dimensional point M of the target pixel xj and the viewpoint k, and sets this distance as the depth value ZM. That is, the depth value ZM indicates the distance between the three-dimensional point M of the target pixel xj and the viewpoint k.
Further, the consistency determination unit 103 determines the depth value Dk (xp) of the reference pixel xp extracted by the reference pixel extraction unit 102 corresponding to the target pixel xj as a depth which is the reference depth map Dk in the depth map storage unit 107. Read from map.

Then, the consistency determination unit 103 compares the depth value ZM of the target pixel xj with the depth value Dk (xp) of the reference pixel xp. For example, the consistency determination unit 103 obtains a depth value difference (ZM−Dk (xp)) which is a difference between the depth value ZM of the target pixel xj and the depth value Dk (xp) of the reference pixel xp. The consistency between the depth value ZM of the target pixel xj and the depth value Dk (xp) of the reference pixel xp is determined based on whether or not 0 (less than or equal to 0).
0 Z ZM-Dk (xp) (1)

  That is, when the depth value ZM is larger than the depth value Dk (xp), the depth value of the target pixel xj without interfering with the visibility of the three-dimensional point M generated from the depth value Dk (xp) It is determined that the consistency between Di (xj) and the depth value Dk (xp) of the reference pixel xp is obtained. On the other hand, if the depth value ZM is smaller than the depth value Dk (xp), the three-dimensional point M interferes with the visibility of the three-dimensional point generated from the depth value Dk (xp). It is determined that the consistency between the value Di (xj) and the depth value Dk (xp) of the reference pixel xp is not obtained.

  FIG. 3 is a conceptual diagram illustrating the inhibition of the visibility of each of the three-dimensional point M and the three-dimensional point generated from the depth value Dk (xp). In each of FIGS. 3A and 3B, the three-dimensional point M generated from the target pixel xj in the target depth map Di of the viewpoint i represents the visibility of the three-dimensional point N generated from the reference pixel xp. It is inhibiting. Here, in FIG. 3A, the depth value ZM of the target pixel xj is an outlier, and the three-dimensional point M is present at an incorrect position. On the other hand, in FIG. 3B, the depth value Dk (xp) of the reference pixel xp is an outlier, and the three-dimensional point N is present at an incorrect position. That is, as a case where the visibility is inhibited, an incorrect three-dimensional point inhibits the visibility of the correct three-dimensional point, and the correct three-dimensional point inhibits the visibility of the incorrect three-dimensional point. There is a case shown in FIG.

FIG. 4 is an explanatory diagram of a process of adding a penalty based on the consistency determination between the depth value Di (xj) of the target pixel xj and the depth value Dk (xp) of the reference pixel xp. FIG. 4A shows that depth values from the viewpoint k of the three-dimensional point M and the three-dimensional point N in the three-dimensional space are compared.
The depth value of the three-dimensional point M from the viewpoint k is ZM, and the depth value of the three-dimensional point N from the viewpoint k is Dk (xp). As shown in FIG. 4A, when each of the depth value ZM and the depth value Dk (xp) satisfies the relationship of the equation (1), as described later, the pixel xj of the viewpoint i by the penalty score integrating unit 104. And an accumulation process is performed on the penalty score of each of the pixels xp of the viewpoint k (in the case of FIG. 4A, the penalty score is incremented).
4B, as described later, when the determination of consistency is performed based on whether or not the depth value difference is equal to or less than the threshold (depth difference threshold Dth), if the depth value difference is equal to or less than the threshold It shows that it is determined to have a sex.

Moreover, although (1) Formula was used in the comparison of the consistency determination part 103, you may use the following (2) Formula.
Dth Z ZM-Dk (xp) (2)
In the equation (2), the depth difference threshold Dth is set, and if the depth value ZM is smaller than the depth difference threshold Dth with respect to the depth value Dk (xp), matching of the depth values of the target pixel xj and the reference pixel xp Sex is judged to be taken. When it is determined whether or not the depth value difference is 0 or less, if the depth value ZM is smaller than the depth value Dk (xp), the depth value Di (xj) of the target pixel xj and the reference pixel xp It is determined that there is no consistency with the depth value Dk (xp) of Therefore, when the depth value difference is within a small error range (a predetermined error range generated at the time of depth map generation), the depth value Di (xj) of the target pixel xj and the depth value Dk (xp) of the reference pixel xp In order to determine that there is consistency (in order to include robustness to the determination), the depth difference threshold Dth is set.

Further, in comparison with the consistency determination unit 103, the following equation (3) may be used in addition to each of the equations (1) and (2).
Dth ≧ | ZM−Dk (xp) | (3)
In equation (3), if the absolute value of the depth value difference between the depth value ZM and the depth value Dk (xp) is equal to or less than the depth difference threshold Dth, the depth value ZM of the target pixel xj and the depth value Dk of the reference pixel xp It is determined that there is consistency with (xp). On the other hand, when the absolute value of the depth value difference between the depth value ZM and the depth value Dk (xp) exceeds the depth difference threshold Dth, the depth value Di (xj) of the target pixel xj and the depth value Dk (xp of the reference pixel xp) It is determined that there is no consistency with

That is, in the case of the equation (3), it is assumed that the three-dimensional point M and the three-dimensional point generated from the depth value Dk (xp) restore the same surface of the target object. When the distance between a and a three-dimensional point generated from the depth value Dk (xp) is large, it is assumed that the same surface can not be restored, and consistency between the depth value ZM and the depth value Dk (xp) is not obtained .
The consistency determination unit 103 determines the depth value Di (xj) of the target pixel xj and the depth value Dk (xp of the reference pixel xp) using any of the above-described equations (1), (2), and (3). Determination of consistency with Here, the consistency determination unit 103 matches the consistency of the depth value of each of the reference pixels extracted by the reference pixel extraction unit 102 with the target pixel corresponding to the reference pixel for all combinations of depth maps. Make a decision.

  The penalty score integration unit 104 increments the penalty score of each pixel as a penalty for each of the target pixel xj determined as inconsistent by the consistency determination unit 103 and the reference pixel xp (adds “1”) Do the processing. The penalty score is given as a penalty score map corresponding to each pixel of the depth map. The penalty score map has the same configuration as the depth map, but the penalty score is given as data to the pixel, not the depth value.

FIG. 5 is a diagram showing a configuration example of the penalty score map table stored in the penalty score map storage unit 108. As shown in FIG.
In FIG. 5, in the penalty score table, fields of depth map identification information and penalty score map index are provided for each record. The depth map identification information is the identification information for identifying the depth map as described above. The penalty score map index indicates the address of the area where the data of the penalty score map in the penalty score map storage unit 108 is stored. The penalty score map is configured of pixels whose penalty score is described as data, which correspond to each of the pixels in the depth map. That is, in the penalty score map, penalty scores for each of the pixels in the depth map are described corresponding to each pixel.

Returning to FIG. 1, the penalty score integration unit 104 refers to the penalty score table in the penalty score map storage unit 108, and detects penalty score maps corresponding to the depth map identification information of each of the target depth map Di and the reference depth map Dk. Do.
Then, the penalty score integration unit 104 increments the penalty score of each of the target pixel xj and the reference pixel xp in the detected penalty score map that the consistency determination unit 103 has determined that there is no consistency, and performs penalty score integration processing. Do.
The integration process of the penalty score in the penalty score table described above is performed on each of the target pixel and the reference pixel in each of all the target depth maps and the reference depth maps determined to be inconsistent by the consistency determination unit 103. .

  The integrated value determination unit 105 refers to each of the penalty score maps stored in the penalty score map storage unit 108, and extracts a penalty score map including pixels having a penalty score equal to or higher than a preset penalty score threshold. Do. At this time, when no pixel having a penalty score equal to or higher than the penalty score threshold is detected in all the penalty score maps, the integrated value determination unit 105 ends the depth map filtering process by the depth map filtering apparatus. The integrated value determination unit 105 causes the depth value correction unit 106 to continue the filtering process when a pixel having a penalty score equal to or higher than the penalty score threshold is detected.

  Here, in the above-described process, in the penalty score map including the penalty score integrated for each pixel, the penalty score is larger as the pixel inconsistency is not obtained with respect to the corresponding other pixel in the different depth map. Therefore, the penalty score at each pixel of the penalty score map indicates the degree of inconsistency with the pixels of the depth map of other viewpoints. Since the pixels with a large penalty score are not consistent with the depth maps of many other viewpoints, majority logic can say that the depth value is likely to be incorrect.

  The depth value correction unit 106 detects a pixel with the largest penalty score (of which the penalty score is the maximum value) from pixels having a penalty score equal to or higher than the penalty score threshold value extracted by the integrated value determination unit 105, and corrects this pixel Extract as a pixel. Then, the depth value correction unit 106 performs a depth map correction process (filter process) by changing the extracted depth value of the correction target pixel to a predetermined numerical value. The predetermined numerical value is, for example, “0” or a negative numerical value which is determined as an incorrect value as a depth value used for generation in the generation process of the three-dimensional shape model.

In addition, the depth value correction unit 106 corrects the depth map by changing the average value of the depth values of the plurality of pixels adjacent to the correction target pixel in the depth map as the predetermined numerical value (filter processing ) May be used.
In addition, the penalty score integration unit 104 may use an integrated value of the depth value difference between the depth value of the target pixel and the depth value of the reference pixel (or the absolute value) as the penalty score instead of incrementing the penalty score. .

When the reference pixel extraction unit 102 generates a three-dimensional point of the target pixel xj in the target depth map Di, if the depth value of the target pixel xj is an incorrect value, the three-dimensional point generation of the target pixel xj Do not do.
When the reference pixel extraction unit 102 projects the generated three-dimensional point on the reference depth map Dk and detects a reference pixel xp that is a pixel in the reference depth map Dk at the projected position, the reference pixel xp If the depth value is an incorrect value, this reference pixel xp is not extracted.

  In the present embodiment, filtering processing of the depth map is repeatedly performed until the integrated value determination unit 105 detects no pixel having a penalty score equal to or higher than the penalty score threshold. For this reason, in the present embodiment, by adding the processing to the above-described incorrect value, as a process not already used to generate a three-dimensional shape by the previous filter processing, the filter is applied again to the pixels to which the incorrect value is written. It is assumed that the processing is not performed.

  FIG. 6 is a flowchart showing an operation example of a process of correcting the depth value of outliers in the depth map performed by the depth map filter processing device 1 according to the present embodiment. The flowchart shown below has shown the flow of the filter processing which the depth map filter processing apparatus 1 corrects the depth value of the outlier in the depth map.

Step S1:
The depth map input unit 101 inputs a depth map group used to generate a three-dimensional shape model of a target object, which is supplied from an external device.
Then, the depth map input unit 101 assigns, to each of the depth maps in the depth map group, depth map identification information that is identification information for identifying the depth map, and the depth map storage unit with the depth map identification information Write to 107 for storage.
Also, at this time, the penalty score integration unit 104 performs reset processing on the penalty score map with the penalty scores of the pixels in all the penalty score maps stored in the penalty score map storage unit 108 as “0”.

Step S2:
The reference pixel extraction unit 102 refers to the depth map table of the depth map storage unit 107, and the target depth map Di, which is the depth map to be corrected for the depth value, and the reference depth map Dk compared with the target depth map Di. Extract combinations of

Step S3:
Then, the reference pixel extraction unit 102 uses the depth value Di (xj) of each of the target pixels xj in the target depth map Di and the camera parameters to generate a three-dimensional point M corresponding to each pixel in the three-dimensional coordinate space. Generate each. At this time, when the depth value of the target pixel xj is an incorrect value, the reference pixel extraction unit 102 does not generate a three-dimensional point of the target pixel xj.

Step S4:
Further, the reference pixel extraction unit 102 projects the generated three-dimensional point on the reference depth map Dk, and extracts the reference pixel xp which is a pixel in the reference depth map Dk at the projected position. At this time, when the depth value of the reference pixel xp at the position where the generated three-dimensional point is projected onto the reference depth map Dk is an incorrect value, the reference pixel extraction unit 102 does not extract this reference pixel xp. Further, the reference pixel extraction unit 102 extracts each combination of all the depth maps stored in the depth map storage unit 107, and extracts each of the reference pixels xp. Here, the reference pixel extraction unit 102 generates a three-dimensional point from the depth values of all the target pixels in the target depth map Di, and projects each of the three-dimensional points on the reference depth map, and is at the projected position. Extract reference pixels.

Step S5:
The consistency determination unit 103 obtains the distance between the three-dimensional point M of the target pixel xj and the viewpoint k, and sets this distance as the depth value ZM.
Further, the consistency determination unit 103 refers to the depth map storage unit 107 with each of the depth values Dk (xp) of the reference pixels xp extracted by the reference pixel extraction unit 102 corresponding to the target pixel xj. , Read out from the depth map which is the stored reference depth map Dk. The depth value ZM indicates the distance from the viewpoint k of the three-dimensional point M.
Then, the consistency determination unit 103 compares the depth value ZM with the depth value Dk (xp), and determines whether the depth value difference (ZM−Dk (xp)) is 0 or less (the above (( 1) Based on the relationship of expression). At this time, when the depth value difference (ZM−Dk (xp)) is equal to or less than 0, the consistency determination unit 103 determines that the depth value is consistent. On the other hand, when the depth value difference (ZM−Dk (xp)) exceeds 0, the consistency determination unit 103 determines that the depth value is not consistent.

Step S6:
The penalty score integration unit 104 applies a penalty to each of the target pixel xj determined as inconsistent by the consistency determination unit 103 and the reference pixel xp.
That is, the penalty score integration unit 104 increments the penalty score of each of the target pixel xj in the target depth map Di in the penalty score map storage unit 108 and the reference pixel xp in the reference depth map Dk Do).

Step S7:
In the penalty score integration unit 104, the reference pixel extraction unit 102 creates a combination of all the depth maps, and whether or not the process of adding penalty to each of the target pixel and the reference pixel in the combination of the depth maps is completed. Make a decision on
At this time, the penalty score integration unit 104 proceeds with the process to step S8 when the process of adding the penalty to each of the target pixel and the reference pixel in the combination of all the depth maps is completed. On the other hand, when the penalty addition processing for each of the target pixel and the reference pixel in the combination of all the depth maps is not completed, the penalty score integration unit 104 proceeds the process to step S2.

Step S8:
The integrated value determination unit 105 refers to each of the penalty score maps stored in the penalty score map storage unit 108. Then, the integrated value determination unit 105 determines the presence or absence of a pixel having a penalty score equal to or greater than a preset penalty score threshold. At this time, when there is a pixel having a penalty score equal to or higher than the penalty score threshold value, the integrated value determination unit 105 advances the process to step S9 (that is, continues changing the depth value). On the other hand, when there is no pixel having a penalty score equal to or higher than the penalty score threshold, the integrated value determination unit 105 ends the depth map filter processing by the depth map filter processing device.

Step S9:
The depth value correction unit 106 detects a pixel having a maximum penalty score (integrated value) from pixels having a penalty score equal to or higher than the penalty score threshold value extracted by the integrated value determination unit 105, and corrects this pixel Extract as

Step S10:
The depth value correction unit 106 extracts the depth map including the correction target pixel in the depth map storage unit 107, and changes (rewrites) the extracted depth value of the correction target pixel to a predetermined numerical value to correct the depth map. Do the processing. The predetermined numerical value is changed to, for example, “0” or a negative numerical value which is determined as an incorrect value as a depth value used for generation in the generation process of the three-dimensional shape model.
Then, the penalty score integration unit 104 resets the penalty score of each pixel in all the penalty score maps in the penalty score map storage unit 108 to “0”, and the process proceeds to step S2.

FIG. 7 is a diagram for explaining the correction process of the correction target pixel having the penalty score having the maximum value. FIG. 7A illustrates the detection of the pixel having the maximum value as the penalty score. The pixel in the depth map DP5 of the viewpoint P4 corresponding to the three-dimensional point M2 and the pixel corresponding to the depth map DP1 of the viewpoint P5 corresponding to the three-dimensional point M1 have a maximum value at a penalty score of "4" There is.
Therefore, the depth value correction unit 106 performs a process of changing the depth value of each of the pixel corresponding to the three-dimensional point M1 and the pixel corresponding to the three-dimensional point M2 to an incorrect value.

  Thus, according to the present embodiment, after the correction process of the correction target pixel, the penalty score addition process from step S2 to step S7 is performed again to obtain an overall penalty score as shown in FIG. Change. By repeating this process, the penalty score of the pixel in all the penalty score maps becomes less than the penalty score threshold, and the outliers of the depth value can be reduced.

  Further, according to the present embodiment, a pixel having a depth value that hinders the appearance of a three-dimensional point of another pixel is detected, and processing is performed using the depth value of this pixel as an outlier as an incorrect value. Therefore, it is possible to reduce the outliers of the depth value without changing the correct depth value in the depth map to the incorrect depth value.

  Further, according to the present embodiment, each of the three-dimensional points corresponding to the depth value of the pixel is projected onto the depth map of the other viewpoint, and the depth value of this three-dimensional point is converted to the depth map of the other viewpoint Since (after calculating the depth value of the three-dimensional point in other viewpoints) and comparing with the depth value of the pixel at the projected position, each pixel is processed independently, so Correcting the depth value and correcting the incorrect depth value of the pixels near the object surface precisely correct the depth value with high score value, and obtain the depth map consistent with the depth map of other viewpoints It is possible.

Further, in the present embodiment, instead of using the depth value of the depth map in the determination of the consistency, the pixel value of the pixel in the captured image having a pixel corresponding to the position of the pixel of the depth map may be used. . In this case, the depth map input unit 101 reads the captured image corresponding to the depth map from the external device, assigns the same depth map identification information as the corresponding depth map, and writes and stores the same in the depth map storage unit 107 . At this time, the captured image index is written in the depth map table, and the data of the captured image is written and stored in the area of the address indicated by the captured image index.
As already described in the above embodiment, the reference pixel extraction unit 102 creates a three-dimensional point M from the depth value of the target pixel xj in the target depth map Di of the viewpoint i and the camera parameter. Then, the reference pixel extraction unit 102 projects the three-dimensional point M on the reference depth map Dk of the different viewpoint k, and extracts the reference pixel xp at the projected position.

Then, the consistency determination unit 103 compares the pixel value of the target pixel xj with the pixel value of the reference pixel xp. In this embodiment, a gradation value (RGB of a color image) which is a pixel value or a luminance value obtained from the pixel value is used. In the case of a black and white image, gray scale gradation is used.
In the following description, the consistency determination unit 103 refers to, for example, the color Ii (xj) of the target pixel xi of the target color image Ii corresponding to the target depth map Di and the reference of the target color image Ik corresponding to the reference depth map Dk. The comparison with the color Ik (xp) of the pixel xp is performed using the following relational expression (4).

| Ii (xj) −Ik (xp) |> Ith (4)
In equation (4), Ith is a preset pixel value threshold.
When the absolute value of the difference value between the color Ii (xj) and the color Ik (xp) exceeds the pixel value threshold Ith, the consistency determination unit 103 obtains consistency between the target pixel xi and the reference pixel xp. It is determined that there is no Then, the penalty score integration unit 104 increments the penalty score of each of the target pixel xj and the reference pixel xp.
Further, the penalty score integration unit 104 may be configured to integrate the absolute value | Ii (xj) −Ik (xp) | of the difference value with respect to the penalty score of each of the target pixel xj and the reference pixel xp. The correction process of the pixel depth value using the penalty score after the penalty score integration process is the same as the operation described above.

In addition, a combination of any one of the equations (1) to (3) and the equation (4) may be used for determination of performing an integration process of the penalty score of each of the target pixel xj and the reference pixel xp.
For example, the configuration may be such that the penalty score is integrated when there is no consistency in the depth value and no consistency in the color.
On the other hand, when there is no consistency in depth value or consistency in color, a configuration may be adopted in which the penalty score is integrated.
Further, in the present embodiment, when it is determined that there is no consistency between the target pixel and the reference pixel, it is described that the process of integrating the penalty scores of both the target pixel and the reference pixel is performed. It may be configured to perform only the score integration process.

  Further, in the above-described embodiment, by performing the correction process of the depth map, the consistency between the depth values of the corresponding pixels between the depth maps, and the consistency between the depth map and the color image change, so that the penalty The score map values also change. Therefore, in the present embodiment, the filtering process of the depth map is alternately repeated between the calculation of the penalty score map and the correction of the depth map until the penalty score of all the pixels in all the penalty score maps becomes "1" or less. Process to Also, set the penalty score threshold to a predetermined value other than “1”, and calculate the penalty score map and modify the depth map until the score value of all pixels in all penalty score maps is less than that threshold. It may be configured to be repeatedly and alternately processed.

FIG. 8 is a view showing an image in which the three-dimensional point group of the object is restored based on the depth map generated from the multi-viewpoint image of the object. FIG. 8A shows an example of a multi-viewpoint image at any of the viewpoints obtained by capturing an object from different viewpoints.
In the example of FIG. 8, camera parameters obtained by capturing the multi-viewpoint image of the imaging device with each of the six multi-viewpoint images captured from different viewpoints including the multi-viewpoint image shown in FIG. A depth map of each viewpoint was generated by the PatchMatch Stereo method using each of them, and a three-dimensional point group was generated from the depth map of each viewpoint and camera parameters.

FIG. 8B shows a case where a three-dimensional point group is generated by the as-generated depth map without performing the depth map filtering process in the present embodiment on the depth map generated by the PatchMatch Stereo method. The results are shown.
As can be seen from FIG. 8 (b), the three-dimensional point group restored using the as-generated depth map includes many incorrect three-dimensional points. The incorrect three-dimensional point group is generated because the incorrect (outlier) depth values included in the generated depth map are converted as they are into three-dimensional points.

FIG. 8C shows the result in the case where a three-dimensional point group is generated using the corrected depth map after the depth map filter processing according to the present embodiment is performed on the depth map generated by PatchMatch Stereo method. It shows.
As can be seen from FIG. 8 (c), in the case of the three-dimensional point group reconstructed using the corrected depth map, an inaccurate three-dimensional point group as contained in FIG. 8 (b) is generated As a three-dimensional shape model of the target object, the result is higher in accuracy than in the case of FIG.

  The result is that the depth map filtering process according to the present embodiment can correct the incorrect (outlier) depth values included in the depth map, which are caused by the incorrect depth values. It shows that it is suppressing that the incorrect three-dimensional point is generated. In particular, in the result of FIG. 8C, the incorrect three-dimensional point generated in the form of clusters in the result of FIG. 8B and the three-dimensional point generated near the surface of the target object are also obtained. It can confirm that it has been corrected. In addition, the 3D point group shape model by the restored 3D point group is correctly restored although the incorrect 3D point is corrected by comparison with the image of FIG. 8A. It can also be confirmed that is not unfairly corrected.

  A program for realizing the function of the depth map filter processing apparatus 1 of FIG. 1 in the present invention is recorded in a computer readable recording medium, and the program recorded in the recording medium is read into a computer system and executed. By doing this, processing may be performed to correct the depth value of each pixel in the depth map. Note that the “computer system” referred to here includes hardware such as an operating system (OS) and peripheral devices. The "computer system" also includes a WWW (World Wide Web) system provided with a homepage providing environment (or display environment). The "computer readable recording medium" is a portable medium such as a flexible disk, a magneto-optical disk, a ROM (Read Only Memory), a CD-ROM (Compact Disc-Read Only Memory), etc., and is incorporated in a computer system. It refers to a storage device such as a hard disk. Furthermore, “computer-readable recording medium” refers to volatile memory (RAM (Random Access (RAM) within a computer system serving as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line). Memory), etc., includes those which hold a program for a certain period of time.

  The program may be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the “transmission medium” for transmitting the program is a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Further, the program may be for realizing a part of the functions described above. Furthermore, it may be a so-called difference file (difference program) that can realize the above-described functions in combination with a program already recorded in the computer system.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and design changes and the like within the scope of the present invention are also included.

1 ... depth map filter processing device 101 ... depth map input unit 102 ... reference pixel extraction unit 103 ... consistency determination unit 104 ... penalty score integration unit 105 ... integrated value determination unit 106 ... depth value correction unit 107 ... depth map storage unit 108 ... Penalty score map storage unit

Claims (10)

A consistency determination unit that compares pixel information of corresponding pixels of different depth maps, and determines presence / absence of consistency between the pixels;
A penalty score integration unit that gives a penalty score to pixels determined to be inconsistent;
After the determination of the consistency of the pixel information between the pixels in all combinations of each of the depth maps is completed, it is determined whether the integrated value of the penalty score exceeds a predetermined first threshold value, and the pixels An integrated value determination unit that generates a penalty score map in which each penalty score of
A depth value correction unit which corrects the depth value of the pixel of the integrated value exceeding the first threshold value in the penalty score map. When the integrated value exceeds the first threshold, the integrated value determination unit causes the consistency determination unit to determine consistency between pixels according to the depth map after correction of the depth value. The depth map filtering device according to claim 1, wherein the correction processing of the depth map is continued. A three-dimensional point is generated from a depth value of a target pixel of one depth map in a pair to be compared and a camera parameter of the one depth map, and the three-dimensional point is projected onto another depth map A reference pixel extraction unit that extracts reference pixel information that is the pixel information of the corresponding reference pixel;
The consistency determination unit
The consistency is determined by comparing the reference pixel information with target pixel information that is the pixel information of the target pixel, and determining whether the reference pixel information and the target pixel information are less than a predetermined second threshold. The depth map filtering apparatus according to claim 1, wherein the determination is performed. The depth map filtering device according to any one of claims 1 to 3, wherein the pixel information is a depth value of the pixel. The depth map filter processing device according to any one of claims 1 to 3, wherein the pixel information is a pixel value of the pixel. The depth map filtering apparatus according to any one of claims 1 to 3, wherein the pixel information is each of a depth value and a pixel value of the pixel. The depth value correction unit
The pixel having the maximum value of the integrated value in the pixels of the integrated value exceeding the first threshold is extracted as a maximum value pixel with reference to the penalty score map, and the depth value of the maximum value pixel is corrected. The depth map filtering device according to any one of claims 3 to 6, characterized in that: The depth value correction unit
When correcting a depth value of the pixel, when generating a three-dimensional point of the three-dimensional shape model from the depth map, the depth value of the pixel is set to an invalid value ignored in calculation of generation of the three-dimensional point. The depth map filtering device according to any one of claims 1 to 7, wherein the depth map filtering device is changed. A consistency comparison step of comparing the pixel information of corresponding pixels of different depth maps, and determining the presence or absence of the consistency between the pixels;
A penalty score integration process in which a penalty score integration unit integrates a predetermined penalty score with respect to pixels determined to be inconsistent;
After the integrated value determination unit determines the consistency of the pixel information between pixels in all combinations of the depth map, it is determined whether the integrated value of the penalty score exceeds a predetermined first threshold value. An integrated value determination step of performing a determination and generating a penalty score map in which the penalty score of each of the pixels is indicated;
A depth value correction process of correcting the depth value of the pixel of the integrated value exceeding the first threshold value in the penalty score map. Computer,
Consistency comparison means for comparing pixel information of corresponding pixels of different depth maps, and determining presence or absence of consistency between the pixels;
Penalty score integrating means for giving a penalty score to pixels determined to be inconsistent,
After the determination of the consistency of the pixel information between the pixels in all combinations of each of the depth maps is completed, it is determined whether the integrated value of the penalty score exceeds a predetermined first threshold value, and the pixels Integrated value determination means for generating a penalty score map in which the penalty score of each of
A program for operating as depth value correction means for correcting the depth value of the pixel of the integrated value exceeding the first threshold in the penalty score map.