JPH06243258A - Depth detector - Google Patents

Abstract

PURPOSE:To realize correspondence of high correspondence position precision by reducing erroneous correspondence between two pictures regardless of a complicated scene. CONSTITUTION:Picture processing parts 3a to 3n and 3'a to 3'n subjects picture information, which are stored in picture memories 2a and 2b, to picture processings different from one another. Area correspondence candidate selecting parts 5a to 5n collate areas obtained as the result of picture processings to select correspondence candidates of individual small areas included in respective areas from collation results of the whole of areas. Area correspondence candidfates obtained by selecting parts 5a to 5n by each small area division result and their reliabilities are registered in an area correspondence reliability memory 6. A correspondence determining part collates segments, which are obtained by edge segment extracting parts 8a and 8b, or density correlations in correspondence candidate areas of individual small areas described in the area correspondence reliability memory 6 to determine the corresponding areas, thus improving the precision of correspondence positions.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a depth detecting device for obtaining the depth of a scene from an image.

[0002]

2. Description of the Related Art In recent years, an image is input from a TV camera and the T of a target object included in the image is used by using this image information.
Various methods for obtaining the distance (depth) from the V camera have been considered. Here, as a method of obtaining the depth of a scene from an image, monocular vision obtained from one image by utilizing knowledge to an object to some extent, stereoscopic vision obtained by triangulation by associating between two images, and three or more Multi-viewing using an image of, and a motion stereo using an image input while moving, and the like. Among them, the stereoscopic view has an advantage over other methods in that knowledge about the scene is not particularly required and the depth can be obtained by two stationary cameras. However, in stereo vision, since depth is calculated by triangulation, it is necessary to make correspondence between two images. As this method, conventionally, various methods such as area-based correspondence and shape-based correspondence are used. Is proposed.

First, the area-based correspondence is 2
A method of taking a density correlation between two images or an SSDA method is often used, but the processing time is enormous and the appearance changes greatly between the two images such as a portion where the depth changes discontinuously. However, there is a problem that it is not possible to collate well. Further, in order to increase the resolution for calculating the depth, it is necessary to reduce the size of the window for correlation, but there is a problem in that if the window is made smaller, there will be more erroneous correspondence.

Area division is performed using information such as color,
A method of associating each area has been proposed, but there is a problem that if the accuracy of the area division result is not high, the accuracy of depth calculation cannot be sufficiently met. There is also a problem that effective image processing means differs depending on the scene.

On the other hand, as the shape-based association, a means for extracting the edge segment from the image and performing the association by using the shape and connection relationship of each segment is often used. Therefore, there is a problem that the correspondence cannot be established unless the connection of the segments is collated in a wide range. Also,
If the individual segments are associated with each other independently, the correspondence may not be determined as in the case of a repeated pattern.

[0006]

As described above, in the conventional depth detecting apparatus using stereo vision, the widths of the area-based correspondence and the shape-based correspondence between two images are also accurate. There was a drawback that it could not be checked.

It is an object of the present invention to provide a depth detecting device capable of accurately aligning two images in fine units when determining the depth to an object using stereoscopic vision.

[0008]

SUMMARY OF THE INVENTION The present invention comprises a plurality of image input means for inputting image data, and a plurality of image memories for respectively storing the image data input by these image input means, Each of the image data stored in these image memories is divided into small areas, and a plurality of image processing means for respectively performing a plurality of types of image processing, and a plurality of image processing means corresponding to the one stored data are provided. A plurality of labeling means for respectively integrating the respective small areas of the image data according to the respective image processing results, and an integrated area of the respective image data obtained by the labeling means for each of the image processing results of the other image data. A plurality of area-corresponding candidate selecting means for selecting matching areas for each small area using the same type of image processing results, and matching selected by these area-corresponding candidate selecting means. An area-corresponding reliability memory in which each small area in the area is associated with each small area in the integrated area and stored together with the reliability, and collation corresponding to the integrated area according to the reliability stored in the area-related reliability memory Correspondence deciding means for deciding a region and performing alignment in units of each of these small regions, and distance calculating means for calculating a depth distance corresponding to the image data using the alignment result obtained by the correspondence deciding means. It is characterized by having.

[0009]

According to the present invention, since a plurality of different image processing means are provided and the image processing is performed independently, even if the effect of a part of the image processing is lowered depending on the type of scene, another image processing compensates for this. be able to. Further, since the correspondence candidate of each small area is obtained from the matching result of all areas adjacent to the small area and having the same image processing result, the error in the result of the image processing is strong. Further, since the correspondence candidates of the small area are independently registered in the area correspondence reliability memory by each area correspondence candidate selection means, even if a part of the area correspondence candidate selection means is leaked from the correspondence candidates, other areas are not registered. It can be supplemented from the result of the correspondence candidate selection means. In addition, by limiting the corresponding area candidates by area correspondence, erroneous correspondence due to segment matching and correlation matching is reduced, and the rough correspondence of the entire area including the segment is already taken. By considering the correspondence order of, it becomes possible to associate the part including the repeated pattern. In addition, since the corresponding areas are selected from the area correspondence candidates registered in the area correspondence reliability memory by segment matching or correlation matching, at the same time, by these matching, accurate positioning inside the small area can be performed. It is possible to compensate for the poor accuracy of the corresponding position included in the correspondence by area.

[0010]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 shows a schematic structure of the depth detecting apparatus in the embodiment, which is a TV camera 1a, 1b, image memories 2a, 2b, image processing units 3a, 3b, 3n, 3 '.
a, 3'b, ~ 3'n, labeling parts 4a, 4b ~ 4
n, area correspondence candidate selection units 5a, 5b to 5n, area correspondence reliability memory 6, correspondence determination unit 7, edge segment extraction units 8a, 8b, and distance calculation unit 9.

In such a structure, the TV camera 1a
Image data input from the
Image data input from the V camera 1b is stored in the image memory 2
Send to b. The image processing units 3a to 3n divide the image data stored in the image memory 2a into m × m size small areas as shown in FIG. 2, and apply different image processing algorithms to each small area independently. Label small areas. Further, the image processing data calculated during the image processing is stored for each small area. The image processing units 3'a to 3'n apply the same image processing algorithm as that of the image processing units 3a to 3n to each of the small areas obtained from the image data stored in the image memory 2b, and label each small area. Stores image processing data. In this regard, a case where n = 2, that is, a case where there are two image processing units will be described in detail.

The image processing unit 3a and the image processing unit 3'a label each small area in the image using the density and direction of the edge. FIG. 3 shows the flow of processing performed by the image processing units 3a and 3'a. First, the entire image stored in the image memories 2a and 2b is subjected to spatial filtering by a 3 × 3 mask (Sobel operator) shown in FIG. 5 to obtain an edge intensity image Is and an edge direction image Id. . Since the edges are sampled in four directions as shown in FIG. 6, each pixel of Id is 0 to 3 depending on the direction.
Is assigned the value of. As shown in FIG. 2, (i, j)
Let Rij be the second small region. In the area corresponding to Rij of Id, when the sum Mij of luminance is obtained in the area corresponding to Rij of Is, and Mij is larger than a preset threshold TH1, that is, when the edge density is high, the histogram H
Ask for. The distribution of the histogram H is collated with the distribution of a preset histogram group, and the label number indicating the closest distribution is added to the small area Rij. FIG. 7 shows a part of the preset histogram group. H is Figure 7
If it is close to the distribution of (a), label 0 is added to Rij, if it is close to the distribution of (b), label 1 is added, and if it is close to the distribution of (c), label n is added. All the small areas in Is are labeled by performing the above processing.
The value of Mij is stored in ij.

The image processing section 3b and the image processing section 3'b label each small area in the image using the color information. FIG. 4 shows the flow of processing performed by the image processing units 3b and 3'b. First, the hue and saturation of each pixel are calculated for the entire image stored in the image memory 2b, and the hue image Ih and the saturation image Ic are obtained. In the small area corresponding to Rij of Ic, the total luminance Lij is obtained, and when Lij is larger than a preset threshold TH2, that is, when the saturation is high, the color information is effective in this area. Judgment is made, and a histogram H'is obtained in the area corresponding to Rij of Ih. The distribution of this human gram H'is collated with the distribution of the preset histogram group, and the label number showing the closest distribution is added to the small region Rij. All the small areas in Ic are labeled by performing the above processing and L
Store the values of ij respectively.

The labeling units 4a and 4b are the image processing unit 3.
With respect to the labels of the small areas Rij obtained in a and 3b, the small areas having the same label and being adjacent to each other are grouped together. FIG. 8A shows an example in which the small areas Rij labeled by the image processing unit 3a are put together into four areas K1 to K4. Further, FIG. 8B shows an example in which the small areas Rij labeled by the image processing unit 3b are put together into three areas S1 to S3.

The area correspondence candidate selection section 5a is a labeling section 4.
Area Ki obtained by a (i = 1 to 1 in FIG. 8A)
4) is a small region Rij (1 <1 <obtained by the image processing unit 3'a.
Correspondence candidates are selected by collating in the search area Wi set on the labeling result of i <m, 1 <j <m). The search area changes depending on the installation state of the TV cameras 1a and 1b. However, when the optical axes of the TV cameras 1a and 1b are set to be parallel, the epipolar line is parallel to the horizontal axis of the image. The search area Wi is set at the same height as Ki (FIG. 9). It is assumed that the area Ki obtained by the labeling unit 4a is composed of small areas Rst (1 <s <m, 1 <t <m) as shown by the hatched areas in FIG. 9 (a). As shown in FIG. 10, Ki is raster-scanned on the small area Rij in Wi and the labels between the overlapping small areas are compared. If the labels are the same, the sum Mst and Rij of the edge densities of Rst are compared. The difference Sst = | Mst−Mij | of the sum Mij of the edge densities of is obtained. It is assumed that there are N1 small areas having the same label and the total sum of the respective Sst is Z1. At this time, the reliability δ1 and the degree of overlap γ1 are defined as δ1 = Z1 / N1 γ1 = N1 / P1. Here, P1 is the total number of small areas included in Ki. As .gamma.1 is closer to 1 and .delta.1 is closer to 0, each small region Rst of Ki overlaps with a similar small region Rij. Therefore, Ki
Δ1 and γ1 are raster-scanned in Wi, and when 0 <δ1 <Tδ1 (1) Tγ1 <γ1 <1 (2) is satisfied, the region correspondence reliability memory 6 corresponding to each small region Rst Ki The respective positions (i, j) of the overlapping small areas Rij and the values of δ1 and γ1 are registered in the respective positions (s, t) of. Here, Tδ1 and Tγ1 are threshold values set in advance. For example, as shown in FIG. 11, {Rst, Rs +
1t, Rs +2, Rs + 3t, Rs + 3t + 1, Rs +
4t + 1, Rs + 5t + 1, Rs + 6t + 1} Ki, and if δ1 and γ1 satisfy the conditions of (1) and (2) at the position of FIG. Information is registered. (S, t) ← (i, j) (s + 1, t) ← (i + 1, j) (s + 2, t) ← (i + 2, j) (s + 3, t) ← (i + 3, j) (s + 3, t + 1) ← (I + 3, j + 1) (s + 4, t + 1) ← (i + 4, j + 1) (s + 5, t + 1) ← (i + 5, j + 1) (s + 6, t + 1) ← (i + 6, j + 1)

Here, (s, t) ← (i, j) is the position (i, j) as the corresponding region candidate at the position (s, t) of the region reliability memory 6 and δ1 and γ1 at that time. It indicates that it is added. In the area correspondence candidate selection unit 5a, all the areas (K1 to k4 in the example of FIG. 8A) obtained by the labeling unit 4a are subjected to the above processing and are obtained from the TV camera 1a in the area reliability memory. The small area candidates in the image from the TV camera 1b corresponding to each small area in the image are registered.

The area correspondence candidate selection unit 5b is a labeling unit 4.
Area Si obtained from b (i = 1 to 1 in FIG. 8B)
3) is a small region Rij (1 <1 <obtained by the image processing unit 3'b.
Correspondence candidates are selected by matching in the search area set on the labeling result of i <m, 1 <j <m).
The search area Wi is set in the same manner as in the case of the area correspondence candidate selection unit 5a. Area Si obtained by the labeling section 4b
Is a small area Rst (1 <s
<M, 1 <t <m). Si is raster-scanned on the small area Rij in Wi, and the labels between the overlapping small areas are compared. If the labels are the same, the sum Lst of the saturation of Rst and the sum Lij of the saturation of Rij are compared. The difference Tst = | Lst−Lij | It is assumed that there are N2 small areas having the same label and the total sum of Tst is Z2. At this time, the reliability δ2 and the overlapping degree γ2 are defined as δ2 = Z2 / N2 γ2 = N2 / P2. Here, P2 is the total number of small areas included in Si. As .gamma.2 is closer to 1 and .delta.2 is closer to 0, each small region Rst of Si overlaps with a similar small region Rij. Therefore, Si
Δ2 and γ2 are raster-scanned within Wi, and if 0 <δ2 <Tδ2 (3) Tγ2 <γ2 <1 (4) is satisfied, the region correspondence reliability memory 6 corresponding to each small region Rst Si At each position (s, t) of (1), each position (i, j) of the overlapping small region Rij and the values of δ2 and γ2 are registered. Here, Tδ2 and Tγ2 are threshold values set in advance. For example, as shown in FIG. 13, {Rst, Rs +
1t, Rs + 2t, Rs + 3t} Si, and if δ2 and γ2 satisfy the conditions of equations (3) and (4) at the position of FIG. 13 during raster scanning, the following information is stored in the area correspondence reliability memory. be registered. (S, t) ← (i, j) (s + 1, t) ← (i + 1, j) (s + 2, t) ← (i + 2, j) (s + 3, t) ← (i + 3, j)

Here, (s, t) ← (i, j) is the position (i, j) in the area reliability memory 6 at the position (s, t) and the position (i, j) as the corresponding area candidate and δ2 and γ2 at that time. It indicates that it is added. In the area correspondence candidate selection unit 5b, all the areas (S1 to S4 in the example of FIG. 8B) obtained by the labeling unit 4b are subjected to the above processing and are obtained from the TV camera 1a in the area reliability memory. The small area candidates in the image from the TV camera 1b corresponding to each small area in the image are registered.

By the area correspondence candidate selection section 5a, the same area (i, j) has already been stored in (s, t) of the area reliability memory 6.
If is registered, the registered reliability δ1 and duplication degree γ1 are renewed by the linear form (k1, k2, k3, k4 are constants) δ = k1 ・ δ1 + k2 ・ δ2 γ = k3 ・ γ1 + k4 ・ γ2 The reliability δ and the redundancy γ are changed. On the contrary, the area candidate selection unit 5b has already set (s,
If the same area is registered in t), the area correspondence candidate selection unit 5a executes the same change.

In the correspondence determining unit 7, the area correspondence candidate selecting unit 5
The corresponding area is determined from the corresponding area candidates registered in the area correspondence reliability memory by a and 5b, and the detailed alignment is performed by the correlation and the matching of the edge segment. First, based on the type of label obtained by the image processing unit 3a, correlation is used to select a correspondence candidate registered in the area reliability memory, but whether an edge segment is used is determined. When (i, j) n (0 <n <k) is registered as a candidate corresponding to (s, t) in the area correspondence reliability memory, that is, the small area Rst in the image obtained from the TV camera 1a. Consider a case where k small areas Rij in the image obtained from the TV camera 1b are registered as the correspondence candidates of.
If the density of edges in the small area Rst is low and indicates a single direction, the edge segments inside Rst and Rij obtained by the edge segment extraction units 8a and 8b are compared. As shown in FIG. 14, with respect to a segment that crosses the small region Rst, the length and inclination of each segment, including the adjacent segment, and the connection relationship between adjacent segments are similar. Is present around Rij. Then, the region in which the segment most similar to the k correspondence candidate small regions exists is determined as the corresponding region of Rst, and the corresponding position of the segment is determined as the accurate corresponding position, and the sending distance is obtained to the distance calculation unit. Further, if the edge density in the small region Rst is high and the directivity is poor, the density correlation with Rst is taken around each of the k corresponding candidate small regions Rij, and the region having the highest correlation value is determined as the corresponding region of Rst. , The position at which the correlation has a peak is set as an accurate associating position, and the sending distance is obtained to the distance calculating unit.

Here, the case where the number of image processing units is two has been described, but as shown in FIG. 1, there is no limit to the number of image processing units and area correspondence candidate selection units, and the same processing as described above is realized. it can.

As described above, since the area correspondence candidate selection unit obtains the correspondence candidate of each small area from the collation result of the entire area including the small area, part of the labeling of the small area by the image processing unit is performed. Even if is incorrect, it is possible to avoid an incorrect association. Further, since each area candidate selection unit independently obtains the correspondence candidate and writes it in the area correspondence reliability memory, the reliability of the correspondence candidate of each small area registered in the area correspondence reliability memory is determined by each area candidate selection. It is possible to reflect each output result desired by the department. In other words, the reliability of the correspondence candidates of each small area registered in the area correspondence reliability memory is not only the candidate given the high reliability by many area candidate selection units, but also a part of the area candidate selection unit has a high reliability. Since the one to which the reliability is given is also registered, it is possible to prevent the small region corresponding to the true from leaking from the association candidates due to an error in some image processing units. Furthermore, the correspondence determination unit uses the labeling of the edges of the small areas to determine in advance whether the small area for which segment matching is advantageous is the small area for which density correlation matching is advantageous, and is registered in the area correspondence reliability memory. Since the correspondence candidate is selected, it is possible to avoid the disadvantages involved in the segment matching and the density correlation matching, and it is possible to refine the correspondence for each small area to the pixel level correspondence.

[0024]

According to the present invention, since a plurality of pieces of image processing information are used and the reliability of the association for each small area is obtained from the collation result of the area including the small area, the influence of image processing error is reduced. You can hold it down. Further, since the small areas are associated with each other and the precise alignment of the inside of the areas is performed by using the segment or the correlation, it is possible to reduce the association error even in the area where the texture is crowded. As described above, it becomes possible to perform accurate alignment between two images in a fine unit.

[Brief description of drawings]

FIG. 1 is a block diagram showing a schematic configuration of an embodiment of the present invention.

FIG. 2 is a diagram for explaining a small area according to an embodiment of the present invention.

FIG. 3 is a diagram showing an image processing unit 3a, 3'a according to an embodiment of the present invention.
FIG.

FIG. 4 is a diagram showing an image processing unit 3b, 3′b according to an embodiment of the present invention.
FIG.

FIG. 5 is an image processing unit 3a, 3'a according to an embodiment of the present invention.
For explaining the edge operator of FIG.

FIG. 6 is an image processing unit 3a, 3'a according to an embodiment of the present invention.
For explaining the direction of the edge of the.

FIG. 7 is an image processing unit 3a, 3'a according to an embodiment of the present invention.
For explaining the labeling of the small areas of the.

FIG. 8 is a diagram for explaining the integration of small labels of the labeling unit 4a according to the embodiment of the present invention.

FIG. 9 is a region correspondence candidate selection unit 5a according to an embodiment of the present invention.
For explaining the search area of FIG.

FIG. 10 is a region correspondence candidate selection unit 5 according to an embodiment of the present invention.
The figure for demonstrating the scan of the area | region of a.

FIG. 11 is a region correspondence candidate selection unit 5 according to an embodiment of the present invention.
The figure for demonstrating registration to the area | region corresponding reliability memory of a.

FIG. 12 is a region correspondence candidate selection unit 5 according to an embodiment of the present invention.
The figure for demonstrating the search area | region of b.

FIG. 13 is a region correspondence candidate selection unit 5 according to an embodiment of the present invention.
The figure for demonstrating registration to the area | region corresponding reliability memory of b.

FIG. 14 is a diagram for explaining segment matching according to an embodiment of the present invention.

[Description of Reference Signs] 1a, 1b ... TV camera 2a, 2b ... Image memory 3a, 3b-3n, 3'a, 3'b, -3'n ... Image processing unit 4a, 4b-4n ... Labeling unit 5a, 5b -5n ... Region correspondence candidate selection unit 6 ... Region correspondence reliability memory 7 ... Correspondence determination unit 8a, 8b ... Edge segment extraction unit 9 ... Distance calculation unit

Claims (1)

[Claims] 1. A plurality of image input means for inputting image data, a plurality of image memories for respectively storing the image data inputted by these image input means, and a plurality of image memories stored in these image memories. A plurality of image processing means for dividing each image data into small areas and performing a plurality of types of image processing respectively, and the image according to the image processing results of the plurality of image processing means corresponding to one image data. A plurality of labeling means for integrating each small area of data,
A plurality of region-correspondence candidates that select matching regions by using the same type of image processing result for each small region of the other image data for the integrated region of each image data obtained for each image processing result by the labeling means. The selection means, the area correspondence reliability memory that stores the small areas in the collation areas respectively selected by the area correspondence candidate selection means in association with the small areas in the integrated area together with the reliability, and the area correspondence Correspondence determining means for determining the collation area corresponding to the integrated area according to the reliability stored in the reliability memory, and performing alignment for each of these small areas, and the alignment result obtained by this correspondence determining means. And a distance calculating unit that calculates a depth distance corresponding to the image data by using the depth detecting apparatus.