JP2961272B1 - Object recognition apparatus and method using feature vector - Google Patents

Abstract

An object of the present invention is to perform distance estimation with high accuracy by correcting similarity in consideration of occurrence of occlusion (concealment of an object) and the like. Corresponding candidate point information extracting means for extracting local information such as brightness information near a corresponding candidate point of each image capturing means corresponding to each selected pixel of an image captured by an image capturing means for each assumed distance. To 608, and local information near the corresponding candidate point of each imaging unit extracted here as input, a feature vector generating unit 609 for generating a feature vector for accurately characterizing a local feature, and a generated feature vector Using a feature vector stabilizing unit 610 that performs information stabilization processing by accumulating the data of each of the elements in a position or time manner, and utilizing the relationship between the elements of the stabilized feature vector. By providing the distance estimating means 611 for estimating the distance to the object, information in a three-dimensional space is estimated.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus and a method for recognizing an object, and more particularly to distance information to an object to be recognized by using triangulation principles from image information obtained by a plurality of image pickup means arranged at different positions. More specifically, the present invention relates to an apparatus and a method which are preferably applied to a case where three-dimensional information of a recognition target object is calculated and an object is recognized using the three-dimensional information.

[0002]

2. Description of the Related Art Conventionally, as a method of measuring a distance to an object to be recognized based on an image pickup result of an image sensor as an image pickup means, a measurement method using stereo vision (stereo vision) has been widely known. Have been.

[0003] This measurement is a useful method for obtaining three-dimensional information such as distance, depth and depth from a two-dimensional image.

[0004] That is, two image sensors are arranged, for example, on the left and right, and the parallax generated when the same image of the object to be recognized is picked up by these two image sensors is calculated based on the principle of triangulation.
This is a method of measuring the distance to an object. The pair of left and right image sensors at this time is called a stereo pair, and is called a twin-lens stereo view because measurement is performed by two units.

FIG. 12 shows the principle of such two-lens stereo vision. As shown in the figure, in binocular stereo vision, images # 1 of the left and right image sensors 1 and 2 (imaging plane 1
a) and an image # 2 (obtained on the imaging surface 2a), it is necessary to measure a parallax (disparity) d which is a difference between the positions of the corresponding points P1 and P2. Generally, the parallax d is a point 50a (a recognition target object 5) in a three-dimensional space.
The relationship expressed by the following equation is established between the distance z to the point on 0).

Z = F · B / d (1) where B is the distance (base line length) between the left and right image sensors 1 and 2, and F is the lens 31 of the image sensor 1 and the lens of the image sensor 2. 32 focal length. Usually, since the base line length B and the focal length F are known, if the parallax d is known, the distance z can be uniquely obtained.

The parallax d can be calculated by searching each point between the images # 1 and # 2 to determine which point corresponds to which point. That of the other image # point on 2 P ₂ corresponding to one of the image # point P ₁ on 1 at this time and be referred to hereinafter as "corresponding points", to explore the corresponding point, hereinafter "corresponds Point search ". When assuming a distance to the object 50, it will be referred to as "candidate corresponding points" below that of one of the image # 1 on the point a point on the other image # 2 corresponding to P _1.

[0008] In the case of performing measurement by binocular stereo vision, if the true corresponding point P ₂ corresponding to the true distance z can be detected as a result of the corresponding point search, the true parallax d is calculated. Is obtained, and it can be said that the true distance z to the point 50a on the object 50 can be measured at this time.

By executing such processing for all pixels of one image # 1, an image (distance image) in which distance information is added to all selected pixels of image # 1 is generated.

The processing for finding the corresponding distance by searching for the corresponding point will be described in detail with reference to FIGS. 13, 14 and 17. FIG.

FIG. 17 is a block diagram showing the configuration of a conventional distance measuring device (object recognizing device) based on binocular stereo vision.

The reference image input unit 101 receives a reference image # 1 taken by the image sensor 1 serving as a reference when calculating the parallax d (distance z). On the other hand, the image input unit 1
In 02, a captured image # 2 of the image sensor 2 which is an image having a corresponding point corresponding to a point on the reference image # 1 is captured.

Next, the processing in the correspondence candidate point coordinate generation unit 103, the correspondence candidate point information extraction unit 104, the similarity calculation unit 105, the in-window addition unit 106, and the distance estimation unit 107 will be described with reference to FIG. The corresponding candidate point coordinate generation unit 103 stores and stores the position coordinates of the corresponding candidate points of the image # 2 of the image sensor 2 for each assumed distance z _n for each pixel of the reference image # 1. By reading this, the position coordinates of the corresponding candidate point are generated.

That is, reference image # of reference image sensor 1
1, a pixel P ₁ specified by the position (i, j) is selected, and a distance z _n to the recognition target object 50 is assumed. Then, the position coordinates of the corresponding candidate point P ₂ in the image # 2 of the other image sensor 2 corresponding to this assumption distance z _n is read.

Next, the corresponding candidate point information extracting unit 104 determines the corresponding candidate point P ₂ based on the position coordinates of the corresponding candidate point P ₂ of the image sensor 2 generated by the corresponding candidate point coordinate generating unit 103 in this manner. A process for extracting local information is executed. Here, the local information is image information of a corresponding candidate point obtained in consideration of a pixel near the corresponding candidate point. A representative example of the image information is the brightness of a pixel, and may be image information obtained by performing preprocessing on an image.

Furthermore, the similarity calculating unit 105, similar to the above image information of the corresponding candidate information extracting section corresponding candidate point P ₂ obtained in 104 (local information) and the reference image selected pixel image information P ₁ The degree is calculated. In calculating the similarity, information is generally stabilized by accumulating data that is close in position. An example is matching using area-based windows.

Therefore, in-window adding section 106
By pattern matching between a region around the selected pixel of the reference image # 1 and a region around the corresponding candidate point of the image # 2 of the image sensor 2, the regions of both images are compared with each other,
A stabilized similarity obtained by stabilizing the information of the similarity is calculated.

[0018] That is, as shown in FIG. 13, with the window WD ₁ around the position coordinates of the selected pixel P ₁ reference picture # 1 is cut, the image of the image sensor 2 #
Window WD ₂ around the second position coordinates of the corresponding candidate point P ₂ is cut out, by performing pattern matching for these windows WD _1, WD ₂ together, these stabilizing similarity is calculated. This pattern matching is performed for each assumed distance z _n . Then, the same pattern matching is performed for all pixels for each selected pixel of the reference image # 1.

In addition to the stabilization in the spatial area (window) described above, there is a method for stabilization in time as well as the stabilization.

FIG. 14 is a graph showing the correspondence between the assumed distance z _n and the reciprocal Qssad of the (stabilized) similarity.

[0021] the window WD ₁ in FIG. 13, as a result of assumptions distance was matching with the ₂ 'window WD and the center position of the corresponding candidate points when the _n' where z is similarity as shown in FIG. 14 and a large value is obtained as the reciprocal Qssad (is smaller similarity) is the window WD ₁ in FIG. 13, the window WD ₂ assumptions distance around the position of the corresponding candidate points when the z _nx It can be seen from the result of the matching that the inverse number Qssad of the similarity is small (the similarity is large) as shown in FIG.

In this way, from the correspondence between the hypothetical distance z _n and the reciprocal of the similarity Qssad, the point with the highest similarity (the point at which the reciprocal of the similarity Qssad has the minimum value) is determined. Assumed distance z corresponding to the point with higher degree
Finally, _nx is estimated as the true distance (the most probable distance) to the point 50a on the recognition target object 50. That is a candidate corresponding point P ₂ corresponding to the assumed distance z _nx in FIG. 13 is a corresponding point on the selected pixel P _1. As described above, the distance estimating unit 107 determines the one having the highest similarity among the similarities obtained by sequentially changing the assumed distance z _n for the selected pixel of the reference image # 1, and determines the most similar one. The hypothetical distance z _{nx at} which the degree becomes higher is estimated and output as the true distance.

Although the above description has been made of the case of the binocular stereo vision, three or more image sensors may be used. Performing distance measurement (object recognition) using three or more image sensors is referred to as distance measurement (object recognition) by multi-view stereo vision.

In the distance measuring apparatus (object recognizing apparatus) based on multi-view stereo vision, a plurality of image sensors are divided into a stereo pair consisting of two image sensors, and each of the stereo pairs is subjected to the above-described twin-lens stereo. It employs a system that applies the principle of vision repeatedly.

That is, a reference image sensor is selected from a plurality of image sensors, and a stereo pair is formed between the reference image sensor and another image sensor. Then, the processing in the case of binocular stereo vision is applied to each stereo pair. As a result, the distance from the reference image sensor to the recognition target present in the field of view of the reference image sensor is measured.

The relationship between stereo pairs in a conventional multi-view stereo will be described with reference to FIG. 15. In the twin-lens stereo shown in FIG. 13, the corresponding image paired with the reference image # 1 is # 2.
However, in the multi-view stereo, the pair of the reference image # 1 and the image # 2 of the image sensor 2, the pair of the reference image # 1 and the image # 3 of the image sensor 3, ..., the reference image # 1 Image sensor N
There are a plurality of stereo pairs such as the pair of image #N. Before performing the processing based on each stereo pair of the corresponding image and the reference image, it is necessary to consider the mounting distortion of the camera, which is an image sensor, and the correction processing by calibration is usually performed in advance.

The process of searching for a corresponding point by a multi-view stereo to obtain a true distance is the same as that of FIG.
This will be described in detail with reference to FIGS. 15 and 18 corresponding to FIGS.

FIG. 18 is a diagram for explaining the configuration of a conventional distance measuring device (object recognition device) based on multi-view stereo vision.

Each of the image sensors 1, 2, 3,..., N
Are arranged at predetermined intervals in the horizontal, vertical or diagonal direction (for convenience of explanation, FIG. 15 shows a case where they are arranged on the left and right at regular intervals).

The reference image input unit 201 receives a reference image # 1 captured by the image sensor 1 serving as a reference when calculating the parallax d (distance z). On the other hand, the image input unit 20
2, a captured image # 2 of the image sensor 2 which is an image having a corresponding point corresponding to a point on the reference image # 1 is captured. Also in the other image input units 203 and 204, the image # 3 of the image sensor 3 corresponding to the reference image # 1 and the image #N of the image sensor N corresponding to the reference image # 1 are captured.

[0031] In the corresponding candidate point coordinate generating unit 205, for each pixel of the reference image # 1, for each assumed distance z _n, of the image # 2 image sensor 2 position coordinates of the corresponding candidate points of the image sensor 3 Position coordinates of the corresponding candidate point of image # 3, image sensor N
The position coordinates of the corresponding candidate points of the image #N are stored and stored, and by reading these, the position coordinates of each corresponding candidate point are generated.

That is, the reference image # of the reference image sensor 1
1, a pixel specified by (i, j) existing at the predetermined position P ₁ is selected, and a distance z _n to the recognition target object 50 is assumed. Then, the position coordinates of the corresponding candidate point P ₂ in the image # 2 image sensor 2 corresponding to this assumption distance z _n is read. Similarly, the position coordinates of the corresponding candidate point P ₃ of the image # 3 of the image sensor 3 corresponding to the selected pixel P ₁ (i, j) and the assumed distance z _n of the reference image # 1 are read, and 1 selected pixel P ₁ (i, j), corresponding candidate point P _N of image #N of image sensor N corresponding to assumed distance z _n
Is read out. Then, similar reading is performed by sequentially changing the assumed distance z _n . Similar reading is performed by sequentially changing the selected pixels. Position coordinates (X ₂ corresponding candidate point P ₂ Thus,
Y ₂ ), position coordinates (X ₃ , Y ₃ ) of corresponding candidate point P ₃ ,...
The position coordinates (X _N , Y _N ) of the corresponding candidate point P _N are output from the corresponding candidate point coordinate generator 205.

Next, the corresponding candidate point information extracting unit 206 determines the corresponding candidate point P ₂ based on the position coordinates of the corresponding candidate point P ₂ of the image sensor 2 generated by the corresponding candidate point coordinate generating unit 205 in this manner. A process for extracting local information is executed. Similarly, in correspondence candidate information extracting unit 207, based on the position coordinates of the corresponding candidate point P ₃ of the image # 3 of the image sensor 3 generated by the corresponding candidate point coordinate generating unit 205, the candidate corresponding point P ₃ local information, the candidate corresponding point information extraction unit 208, based on the position coordinates of the corresponding candidate point P _N of the image #N image sensors N which is generated in the corresponding candidate point coordinate generating unit 205, the candidate corresponding points P _N Local information is obtained respectively.

Further, the similarity calculating section 209 calculates the similarity between the local information of the corresponding candidate point P ₂ obtained by the corresponding candidate point information extracting section 206 and the image information of the selected pixel P ₁ of the reference image # 1. Is calculated. Then, the in-window addition unit 21
In No. 2, the regions of the two images are compared with each other by pattern matching between the region around the selected pixel of the reference image # 1 and the region around the corresponding candidate point of the image # 2 of the image sensor 2, and stable. The similarity is calculated.

[0035] That is, as shown in FIG. 15, with the window WD ₁ around the position coordinates of the selected pixel P ₁ reference picture # 1 is cut, the image of the image sensor 2 #
Window WD ₂ around the second position coordinates of the corresponding candidate point P ₂ is cut out, by performing pattern matching for these windows WD _1, WD ₂ together, these similarities are calculated. This pattern matching is performed for each assumed distance z _n . FIG. 16A shows a hypothetical distance z _n and a stereo pair (reference image sensor 1 and image sensor 2).
It is a graph showing a relationship between a reciprocal Qs ₁ stabilizing similarity.

[0036] the window WD ₁ in FIG. 15, as a result of assumptions distance was matching with the ₂ 'window WD around the position coordinates of the corresponding candidate points when the _n' where z is 16
As shown in (1), a large value is obtained as the reciprocal Qs ₁ of the similarity (similarity is small), but FIG.
The window WD _1, matching the results of the window WD ₂ around the position coordinates of the corresponding candidate points when the assumptions distance z _nx is the reciprocal of the similarity, as shown in FIG. 16 (1) Qs _It can be seen that ₁ is smaller (similarity is larger).

Similarly, in the similarity calculating section 210, the corresponding candidate point P ₃ obtained by the corresponding candidate point information extracting section 207 is obtained.
Similarity between the local information and the reference image # 1 of the image information of the selected pixel P ₁ of is calculated. Then, the in-window addition unit 2
In 13, the window WD ₁ around the position coordinates of the selected pixel P ₁ reference picture # 1, the image # 3 image sensors 3
Window W centered on the position coordinates of the corresponding candidate point P ₃ of
Pattern matching between D ₃ is executed, these stabilizing similarity is calculated. Then, by performing pattern matching for each assumed distance z _n , the stereo pair (reference image sensor 1 and image sensor 3) is also assumed to have the assumed distance z _n and the stabilized similarity as shown in FIG. graph showing a relationship between a reciprocal Qs ₂ of sought. In the similarity calculation unit 211 in the same manner, the similarity between the local information and the reference image # 1 of the image information of the selected pixel P ₁ of the corresponding candidate points P _N obtained above corresponding candidate point information extraction unit 208 is calculated You. Then, the intra-window addition unit 214 calculates a corresponding candidate point P _N between the window WD ₁ centered on the position coordinates of the selected pixel P ₁ of the reference image # ₁ and the image #N of the image sensor _N.
Pattern matching between the window WD _N around the position coordinates of the runs, these stabilizing similarity is calculated. Then, since pattern matching is performed for each assumed distance z _n , the stereo pair (reference image sensor 1 and image sensor N) also has the assumed distance z _n and the reciprocal of the stabilized similarity shown in FIG. correspondence between the Qs _N is required.

In the final similarity calculating section 215, the correspondence between the assumed distance z _n obtained for each stereo pair and the reciprocal of the stabilized similarity is added for each assumed distance, and the assumed distance and the stabilized similarity are added. The correspondence relationship with the reciprocal of the degree (the reciprocal of the overall stabilized similarity) is obtained (see FIG. 16).

Further, as shown in FIG.
From the correspondence between the hypothetical distance z _n and the added value of the reciprocal of the stabilized similarity, the point having the highest similarity (the point at which the added value of the reciprocal of the similarity becomes the minimum value) is determined. The hypothetical distance z _nx corresponding to the point where is higher is finally estimated as the true distance (the most likely distance) to the point 50 a on the recognition target object 50. This is executed by the distance estimation unit 216. This process is performed for all pixels for each selected pixel of the reference image # 1.

As described above, the distance estimating unit 216 determines the one having the highest similarity addition value from among the addition values of the stabilized similarity obtained by sequentially changing the assumed distance z _n. Then, the assumed distance z _{nx at} which the added value of the similarity becomes the highest is estimated as the true distance and is output. Then, since such distance estimation is performed for all pixels of the reference image # 1, an image (distance image) in which distance information is added to all selected pixels of the reference image # 1 is generated.

As a paper on the processing performed by the in-window addition units 212, 213,... 214 and the final similarity calculation unit 215 in FIG. Matching ”(Transactions of the Institute of Electronics, Information and Communication Engineers, D-2VO. J75-D-2 No. 819)
92-8, Masatoshi Okutomi, Takeo Kanade) are known. In this paper, for each stereo image pair, first, the sum of squares of the differences in the brightness values within the window (hereinafter, SSD: Sum of Suqared D
ifferences), and then simply add the SSDs from all stereo image pairs to calculate the reciprocal of the overall similarity. By using this method, even if a similar pattern repeatedly exists on the object to be measured,
Good measurement results with less erroneous correspondence are obtained as compared with the twin-lens stereo.

That is, in the above article 1, a window centered on the position of the selected pixel of the reference image is cut out, a window centered on the position of the corresponding candidate point of the corresponding image is cut out, and pattern matching is performed for these windows. A technique is described in which the similarity calculated in a window for a selected pixel of a reference image is calculated. Specifically, the square of the difference between the image information of the pixel in one window and the image information of the pixel in the other window corresponding to this pixel is obtained for each pixel in the window. The sum of the squared values of the differences for all the pixels in the window is defined as the reciprocal of the stabilized similarity. Then, the reciprocal of the stabilized similarity obtained for each stereo pair is added to calculate the reciprocal of the overall stabilized similarity (see FIG. 16).

As described above, in the multi-view stereo, if a certain point on the object to be measured is observed as a common point by all image sensors, the detection accuracy of the corresponding point is not impaired. .

However, actually, as shown in FIG.
In a case where two independent objects 50F and 50R are arranged side by side in front and rear (50F is a front object and 50R is a rear object), the left and right image sensors 1 and 2 observe the object. As shown in FIGS. 20A and 20B, even if one reference image sensor 1 can observe the entire appearance of these two objects 50F and 50R, another image sensor 2 In some cases, only a part of the rear object 50R can be observed. Thus, part of the object
If a state occurs in which the object is hidden behind a foreground object and is invisible (this is called occlusion (concealment)), the detection accuracy of the corresponding point is significantly impaired.

That is, since the erroneous information which is not the information of the correct corresponding position enters the captured image # 2 of the image sensor 2 in which the occlusion has occurred, the true distance cannot be obtained accurately.

Of course, multi-view stereo has more image sensors than binocular stereo, so even if occlusion occurs in one stereo pair, the probability of normal observation in another stereo pair increases. However, the effect of occlusion is somewhat alleviated as a whole.

However, as shown in the article 1, the reciprocal of the stabilized similarity is calculated for each stereo pair, and the reciprocal of the stabilized similarity obtained for each stereo pair is added. Therefore, the effect of occlusion cannot be eliminated by a simple multi-view stereo method of calculating the reciprocal of the overall stabilized similarity.

As a paper describing a typical method that can deal with the problem of occlusion, see Paper 2
"A Study on Hidden Detection Method in High Definition Stereo SEA Using Camera Matrix" (IEICE Technical Report, 1996-03, IE95-158, PRU95-245, University of Tsukuba: Kiyohide Sato, Yuichi Ohta) is there.

In this paper, a plurality of patterns in which occlusion occurs are prepared in advance for image sensors arranged in a matrix, and the similarity when occlusion does not occur and the similarity when occlusion occurs are described. The similarity of each pattern is obtained, and based on these, the presence / absence of occlusion and the presence of occlusion are determined, and the occlusion pattern is determined. A technique for selecting a degree of similarity corresponding to. Here, the occlusion occurrence pattern is generated in consideration of the number and types of effective image sensors in which occlusion has not occurred.

FIG. 21 shows the configuration of a multi-view stereo apparatus corresponding to the above article 2. In this apparatus, it is assumed that there are m occurrence patterns (including patterns in which occlusion does not occur).

Reference image input unit 301 and image input unit 3
02, 303, and 304 are the same as the reference image input unit 201 and the image input units 202, 203, and 204 shown in FIG. 18, and the corresponding candidate point coordinate generating unit 305 includes the corresponding candidate point coordinate generating unit 205 shown in FIG. The corresponding candidate point information extracting units 306, 307, and 308 are the same as the corresponding candidate point information extracting units 206, 207, and 208 shown in FIG. The intra-window addition units 312, 313, and 314 are the same as the similarity calculation units 209, 210, and 211 shown in FIG. 18, and the intra-window addition units 212, 213, and 214 shown in FIG.
Since these are the same as those described above and are the same as those already described, their description will be omitted.

Each occlusion pattern 1, 2, 3... M
.. 318 are provided for each of them. In each similarity calculation section, based on the stabilized similarity calculated by the in-window addition sections 312 to 314, the similarity calculation sections 315, 316, 317.
The overall stabilization similarity is calculated assuming that occlusion of the corresponding pattern has occurred.

The similarity selection section 319 compares the overall stabilization similarity calculated for each occlusion pattern, determines the most probable occlusion pattern from these, and determines the determined occlusion pattern. The overall stabilized similarity corresponding to the pattern is selected from the overall stabilized similarities calculated by the similarity calculating units 315, 316, 317,.

The distance estimating unit 320 determines the highest similarity among the selected overall stabilization similarities for each assumed distance, and estimates the true distance.

By performing such processing, it is possible to accurately estimate the distance even if occlusion occurs.

However, it is necessary to calculate the overall stabilization similarity for each of the assumed patterns 1 to m of the occlusion, which increases the amount of calculation processing and the time required for calculation. . The greater the number of occlusion patterns, the greater the load of these arithmetic processes. Further, the hardware configuration becomes complicated, and it is difficult to realize the hardware.

As described above, the prior art has the following problems.

(1) In the multi-view stereo apparatus (FIG. 18) configured as shown in the article 1, distance estimation cannot be performed accurately in consideration of the case where occlusion has occurred.

(2) In the multi-view stereo apparatus (FIG. 21) configured as shown in the article 2, although the distance can be accurately estimated in consideration of the case where the occlusion occurs, the data to be handled is increased. At the same time, it is difficult to realize hardware.

According to the first invention of the present invention, by reviewing and optimizing the processing procedure of the conventional multi-view stereo apparatus, it is possible to reduce the data to be handled and to easily realize hardware which takes into account the occurrence of occlusion. Is a first solution. That is, the conventional multi-view stereo apparatus is merely an extension of the binocular stereo apparatus, and the processing efficiency is low. As described above, for each stereo pair, the difference in the brightness value within the window is first determined (the reciprocal of the stabilized similarity is determined for each stereo pair), and finally, all the stereo pairs are simply added. This is because the reciprocal of the overall stabilized similarity is obtained (the overall stabilized similarity is obtained by adding the stabilized similarities obtained for each stereo pair).
In such a conventional method of adding the stabilized similarities obtained for each stereo pair to obtain the overall stabilized similarity, hardware resources for each stereo pair are required. further,
In order to eliminate the effects of occlusion, it is necessary to obtain the overall stabilization similarity described above for each occlusion pattern, and the amount of data and processing to be handled becomes enormous, and hardware resources are further increased. It will be huge. The first invention of the present invention has been made in view of such a situation. Even in a complicated problem in which the occurrence of occlusion is taken into consideration, the image data needs to have sufficient and sufficient information (possibility of stereo processing) at an early stage. Into a feature vector that includes information such as the direction and intensity of a certain occlusion as an element, perform appropriate stabilization processing on this feature vector, and further utilize the feature vector information and the relationship between the vectors. A first object of the present invention is to obtain information in a three-dimensional space to reduce the amount of information to be held and simplify processing to reduce hardware resources.

By the way, the size and shape of the window used for stabilizing the similarity are unique. However, as shown in FIG.
Is remarkable, the edge of the object 50 may be present in the vicinity thereof. In this case, if a square window WD having a unique shape is used, the similarity of the similarity is greatly affected by the background. The accuracy of the stabilization process, and thus the accuracy of the distance measurement, may be impaired. The second invention of the present invention has been made in view of such a situation, and improves measurement accuracy by changing a region for stabilization such as a window shape according to a feature in an image. The second
Is a problem to be solved.

As described above, in the conventional stereo apparatus, the size of the window used for stabilizing the similarity is fixed regardless of the characteristics of the object to be measured, and pattern matching is performed. When the characteristic of the region to be measured is small, there is a problem that the stabilization becomes insufficient and the measurement accuracy deteriorates. Therefore, in such a case, the size of the window is increased so that more features in the image # 1 are taken in, so that sufficient stabilization is performed. Sufficient stabilization can improve the accuracy of distance measurement. The third invention of the present invention has been made in view of such a situation, and has a window size and the like according to features in an image.
A third object of the present invention is to improve measurement accuracy by changing a region for stabilization.

Further, in the conventional stereo apparatus, as shown in FIG. 14 or FIG. 16, the assumed distance z _nx when the reciprocal Qssad of the similarity becomes the smallest is uniquely estimated as the true distance. ing.

However, as shown in FIG. 11, depending on the correspondence between the assumed distance and the reciprocal Qssad of the similarity, the value is not the assumed distance z _x2 when the reciprocal Qssad of the similarity is minimized, but the value Although not small, the assumed distance z _{x1 at the} local minimum point where the rate of change is large in the vicinity may be the true distance. That is, if the assumed distance z _x2 when the reciprocal Qssad of the similarity becomes the smallest is uniquely estimated as the true distance, the accuracy of the estimation may be impaired.

The fourth invention of the present invention has been made in view of such a situation, and estimates the distance in consideration of not only the magnitude of the similarity but also the rate of change of the similarity. It is a fourth solution to improve the accuracy.

[0066]

Therefore, in order to solve the above-mentioned first problem, a first invention of the present invention arranges a plurality of image pickup means at predetermined intervals, and comprises a plurality of image pickup means. Calculate the distance to a point on the object corresponding to the selected pixel in the image captured by the one imaging unit when the target object is imaged by the one imaging unit, based on the distance obtained for each pixel In the object recognition device configured to recognize the object, corresponding candidate point information extracting means for extracting local information such as brightness information near a corresponding candidate point of each imaging means corresponding to the selected pixel for each assumed distance And a feature vector generating means for generating a feature vector for accurately characterizing a local feature by using local information in the vicinity of a corresponding candidate point of each imaging means extracted here as input, and a data of the generated feature vector. Feature vector stabilization means that performs information stabilization processing by accumulating position or time data for each element, and using the relationship between the elements of these stabilized feature vectors to the object. By providing a distance estimating means for estimating the distance of the three-dimensional space, information of a three-dimensional space is estimated.

That is, according to this configuration, as shown in FIG. 1, image data is compressed into necessary and sufficient information for stereo processing at an early stage, and an appropriate stabilizing process is performed on this feature vector. Further, information of a three-dimensional space can be obtained by utilizing information of the feature vector and a relationship between the vectors. Therefore, even in the case of a complicated problem in which the occurrence of occlusion is considered in the multi-view stereo apparatus, the amount of data to be handled and the amount of processing can be reduced, and the hardware resources can be reduced.

According to a second aspect of the present invention, in order to solve the second problem, in a similar object recognizing apparatus, a feature is extracted for each selected pixel of an image picked up by the image pickup means. The image processing apparatus includes a feature extracting unit and a unit that changes a spatial domain and a time domain for stabilization by the stabilizing unit in accordance with the feature extracted by the feature extracting unit.

That is, according to this configuration, FIG.
As shown in (b), the space region and the time region for stabilization are changed according to the extracted features (the direction of the edge 51). That is, the window W
The shape of DX is changed to a shape along the direction of the edge 51. On the other hand, conventionally, the size and shape of the window used for stabilizing the similarity have been univocal. For this reason, as shown in FIG. 9A, when the edge 51 in the image # 1 is remarkable, there is a possibility that the edge of the object 50 exists near the edge 51, and in this case, a unique shape When the square window WD is used, the accuracy of the similarity stabilization process and the accuracy of the distance measurement may be impaired due to the influence of the background. On the other hand, in the second invention of the present invention, the shape of the window WDX is changed to a shape along the direction of the edge 51, so that it is not greatly affected by the background, and the measurement accuracy is improved. Can be improved.

According to a third aspect of the present invention, in order to solve the third problem, in a similar object recognizing apparatus, a feature is extracted for each selected pixel of an image picked up by the image pickup means. The image processing apparatus further includes a feature extracting unit, and a unit that changes a size of a space area to be stabilized by the stabilizing unit according to the feature extracted by the feature extracting unit.

That is, according to this configuration, FIG.
As shown in (a), when the value obtained by adding the absolute value of the change rate of the brightness of the reference image within the window WD of a normal size is smaller than a certain threshold value, the feature vector VQ
Is changed to a window WDX having a size larger than usual, as shown in FIG. 10B. On the other hand, conventionally, the size of the window used when stabilizing the similarity is fixed. For this reason, when the feature of the object to be measured is small, the stabilization is insufficient and the measurement accuracy may be impaired. On the other hand, according to the third aspect of the present invention, when it is determined that the feature amount is small, the window size is increased to take in more features in the image # 1, and sufficient stabilization is achieved. The accuracy of the distance measurement can be improved.

According to a fourth aspect of the present invention, in order to solve the fourth problem, in a similar object recognizing apparatus, a feature is extracted for each selected pixel of an image picked up by the image pickup means. A feature extraction unit for predicting a change rate of the overall stabilization similarity at the estimated distance in accordance with the feature extracted by the feature extraction unit; The assumed distance at which the degree is changing,
Means for outputting the estimated distance as a candidate.

That is, according to this configuration, as shown in FIG. 11, the reciprocal Q ′ of the overall stabilization similarity at the estimated distance according to the extracted features (rate of change in brightness).
The rate of change of ssad is predicted. That is, when the rate of change in brightness is large, the rate of change in similarity at a true distance is predicted to be large. Therefore, when the change rate of the brightness is large, it is predicted that the change rate of the reciprocal Q'ssad of the overall stabilized similarity at the estimated distance z _x1 is large,
With this large rate of change, the inverse Q'ss of the overall stabilization similarity
The hypothetical distance z _x1 in which ad changes is set as a candidate for the estimated distance.

As a result, not the assumed distance z _{x2 in} which the reciprocal Q'ssad of the overall stabilized similarity is minimized, but the change rate of the reciprocal Q'ssad of the overall stabilized similarity more than this is assumed distance z _x1 which is larger, is determined as the estimated distance.

Here, the distance estimating section 320 (107, 216) of the conventional stereo apparatus has the configuration shown in FIG.
As shown in (1), the assumed distance z _nx when the reciprocal Qssad of the similarity becomes the smallest is uniquely estimated as the true distance.

However, as shown in FIG.
Depending on the correspondence between _n and the reciprocal Qssad of the similarity, it is not the hypothetical distance z _x2 when the reciprocal Qssad of the similarity is the smallest, but the value is not the smallest but the local minimum where the rate of change in the vicinity is large. _May be the true distance. That is, in the related art, the assumed distance z _x2 when the reciprocal Qssad of the similarity becomes the smallest is obtained.
Because it is uniquely estimated that the distance is a true distance, the assumption distance z _{x1 at the} time when the rate of change becomes a minimum is sometimes overlooked as a true distance.

In this respect, according to the fourth aspect of the present invention, since the distance is estimated in consideration of the rate of change of the similarity in addition to the magnitude of the similarity, the true distance may be overlooked. In addition, the accuracy of distance estimation can be improved.

[0078]

Embodiments of the present invention will be described below with reference to the drawings. Here, a case where the similarity is used as an example will be described.

FIG. 22 is a block diagram showing the configuration of an object recognition apparatus for multi-view stereoscopic vision according to this embodiment.

In the apparatus of the embodiment shown in FIG. 9, nine image sensors 1, 2, 3,..., N (N = 9) are arranged in a “field” around the reference image sensor 1. It is assumed that they are arranged. Note that, even when the arrangement of the image sensors and the number of image sensors are different from those in the present embodiment, the same effects can be obtained by applying the same processing as described below. The reference image input unit 401 has a parallax d
A reference image # 1 captured by the image sensor 1 serving as a reference when calculating (distance z) is captured. On the other hand, the image input unit 402 captures a captured image # 2 of the image sensor 2 which is an image having a corresponding point corresponding to a point on the reference image # 1. Also in other image input units 403,..., 404, image # 3 of image sensor 3 corresponding to reference image # 1.
,..., The image sensor N (N =
The image #N of 9) is captured, respectively.

The corresponding candidate point coordinate generator 405 calculates the position coordinates of the corresponding candidate point of the image # 2 of the image sensor 2 and the position coordinates of the corresponding image of the image sensor 3 for each pixel of the reference image # 1 at every assumed distance z _n . The position coordinates of the corresponding candidate point of the image # 3,..., The position coordinates of the corresponding candidate point of the image #N of the image sensor N are stored and stored, and by reading these, the position coordinates of each corresponding candidate point are generated. I do.

That is, the reference image # of the reference image sensor 1
1, a pixel specified by (i, j) existing at the predetermined position P ₁ is selected, and a distance z _n to the recognition target object 50 is assumed. Then, the position coordinates of the corresponding candidate point P ₂ in the image # 2 image sensor 2 corresponding to this assumption distance z _n is read. Similarly, the position coordinates of the selected pixel P ₁ (i, j) of the reference image # 1 and the corresponding candidate point P ₃ of the image # 3 of the image sensor 3 corresponding to the assumed distance z _n are read out. image # 1 of the selected pixel P ₁ (i, j), the position coordinates of the corresponding candidate point P _n of the image #N image sensors n which corresponds to the assumed distance z _n is read. And the assumed distance z _n
Are sequentially changed, and similar reading is performed. Similar reading is performed by sequentially changing the selected pixels. Thus the corresponding candidate point P ₂ position coordinates (X _2, Y
₂ ), position coordinates (X ₃ , Y ₃ ) of the corresponding candidate point P ₃ ,.
The position coordinates (X _N , Y _N ) of the corresponding candidate point P _N are output from the corresponding candidate point coordinate generator 405.

Next, in the local information extracting section 406, the position coordinates (X ₂ , Y 2) of the corresponding candidate point P ₂ of the image sensor 2 generated by the corresponding candidate point coordinate generating section 405 in this way.
Process of extracting the local information of the corresponding candidate points P ₂ based on ₂₎ is performed. Similarly, in the local information extraction unit 407,
Image sensor 3 generated by corresponding candidate point coordinate generation unit 405
Image # 3 position coordinates of the corresponding candidate point P ₃ (X _3, Y ₃₎ based on the local information of the corresponding candidate point P ₃ is, ..., the local information extracting unit 408, the corresponding candidate point coordinate generating unit 405 The local information of the corresponding candidate point P _N is obtained based on the position coordinates (X _N , Y _N ) of the corresponding candidate point P _N of the image #N of the image sensor N generated in the above.

[0084] Further, the similarity calculation unit 409, the degree of similarity between the local information extracting unit of the corresponding candidate point P ₂ obtained in 406 local information and the reference image # 1 of the image information of the selected pixel P ₁ is calculated You.

Similarly, the similarity calculating section 410 calculates the similarity between the local information of the corresponding candidate point P ₃ obtained by the local information extracting section 407 and the image information of the selected pixel P ₁ of the reference image # 1. Is done. Similarly, in the similarity calculation unit 411,
Similarity between the local information and the reference image # 1 of the image information of the selected pixel P ₁ of the corresponding candidate points P _N obtained above the local information extracting unit 408 is calculated.

The configuration described above is common to the conventional apparatus shown in FIG.

The subsequent-stage feature vector calculation unit 412 determines a local occlusion vector (which will be described later) on the basis of the similarity calculated for each stereo pair by the similarity calculation units 409 to 411. , referred to as a feature vector) VQ is calculated based on the image # each selected pixel P ₁ of 1.

The feature vector stabilizing section 413 calculates the stabilized global occlusion vector SVQ by adding each local occlusion vector VQ within a predetermined window, as described later.

The correction similarity calculating section 414 determines whether or not occlusion has occurred based on the stabilized global occlusion vector SVQ, and removes the influence of occlusion if occlusion has occurred. , The inverse of the stabilized overall similarity Qssad (i,
j, z _n ) is corrected, and the reciprocal Q'ssad (i, j, z _n ) of the corrected stabilized overall similarity is calculated.

In the distance estimating section 415, the assumed distance z _n ,
The inverse Qssad (i, j,
z _n ), a point having the highest similarity (a point at which the reciprocal of the similarity becomes the minimum value) is determined from the correspondence, and the reciprocal Qssad (i, j, z _n )
The hypothetical distance z _nx corresponding to the point where is lowest is finally estimated as the true distance (the most likely distance) to the point 50a on the recognition target object 50. Such processing is
This is performed for all pixels for each selected pixel of the reference image # 1.

As described above, the distance estimating section 415 determines the one having the highest similarity from the similarities obtained by sequentially changing the assumed distance z _n , and has the highest similarity. The hypothetical distance z _nx is estimated as the true distance and is output. Then, since such distance estimation is performed for all pixels of the reference image # 1, an image (distance image) in which distance information is added to all selected pixels of the reference image # 1 is generated.

Next, with reference to FIGS. 2 to 7, the processing performed by the feature vector calculation section 412, feature vector stabilization section 413, correction similarity calculation section 414, and distance estimation section 415 will be described in detail. Will be described.

A method of generating an occlusion vector As a method of generating an occlusion vector, a pattern in which occlusion occurs is prepared in advance, as shown in the above-mentioned paper 2, and an occlusion vector is generated in accordance with the generated pattern. There are a method of generating a vector and a method of generating an occlusion vector without preparing an occlusion occurrence pattern.

Hereinafter, each case will be described separately, but they may be implemented in combination.

(Method 1) Embodiment in which Occlusion Occurrence Patterns are Preliminarily Prepared This method uses each of the occlusion occurrence patterns assumed to be possible when the object 50 is observed by the nine image sensors 1 to 9. Are prepared in advance, and stored in a storage table (see FIG. 3).
At 11, the reciprocal Qad of the similarity obtained for each stereo pair
Based on (i, j, k, z _n ), these distributions (see FIG. 2 (a)) are obtained, and this distribution is converted into a binarized distribution pattern (see FIG. 2 (b)). The local occlusion vector VQ is obtained according to the storage pattern of the storage table that matches the binarized distribution pattern.

The feature vector calculator 412 calculates the local occlusion vector VQ as follows. Hereinafter, the stereo pair k (k = 1, 2,... 8)
A pair of the reference image sensor 1 and the other image sensors 2... That is, the stereo pair 1 is a pair of the reference image sensor 1 and the image sensor 2, and the stereo pair 2 is
A pair of the reference image sensor 1 and the image sensor 3... A stereo pair 8 means a pair of the reference image sensor 1 and the image sensor 9.

First, as shown in FIG.
A dimensional coordinate system XY is set, and the reference image sensor 1 is positioned at the origin of the coordinate system XY. The other image sensors 2 to 9 are respectively positioned at predetermined coordinate positions around the origin on the coordinate system XY.

Here, the reciprocal Qad (i, j,) of the similarity calculated for each of the image sensors 2 to 9 (each stereo pair k = 1 to 8) is provided at each predetermined coordinate position in the XY coordinate system. k, z _n ), the reciprocal Q of the similarity as shown in FIG.
ad (i, j, k, z n) distribution is obtained. That is, FIG.
In (a), the reciprocal of the similarity is large in the image sensor (stereo pair) corresponding to the coordinate position where the mountain is high, and there is a high possibility that occlusion has occurred in the stereo pair there. Means that.

Next, the similarity calculating sections 409 to 411
The reciprocal of the similarity Qad (i, j,
k, based on the z _n), these average values _{μ (i, j, z n} ) and the standard deviation _{σ (i, j, z n} ) is determined.

Then, the obtained average value μ (i, j, z
_n ) + standard deviation σ (i, j, z _n ) as a threshold (threshold), and the reciprocal Qad (i, j, k,
The magnitude of z _n ) is determined in a binary manner.

That is, when the condition of Qad (i, j, k, z _n )> μ (i, j, z _n ) + σ (i, j, z _n ) (2) is satisfied, 0 and When it is determined that the condition of the above equation (2) is not satisfied, it is determined as 1.

Here, the reciprocal Qad (i, j, k, z _n ) of the similarity is
Is determined to be 0, the reciprocal Qad (i, j, k, z _n ) of the similarity is greater than a predetermined reference level, and in this stereo pair k, occlusion occurs. Means that it is likely to be Conversely, when it is determined to be 1, the possibility of occurrence of occlusion in the stereo pair k is low.

The binary decision shown in the above equation (2) is an example, and the following implementation is also possible.

That is, instead of the above equation (2), Qad (i, j, k, z _n )> μ (i, j, z _n ) (2) ′ may be used. The reciprocal Qad (i, j, k,
z _n ), it is possible to select the top h (h is set in advance) with a large value, set these to 0, and judge those below it to be 1.

In this way, as shown in FIG. 2B, a distribution pattern of the reciprocal of the binarized similarity is generated. In other words, in the image sensor (stereo pair) corresponding to the coordinate position “0” in FIG. 2B, the reciprocal of the degree of similarity is large, and occlusion occurs in the stereo pair. Means that it is likely to be. The distribution pattern of the reciprocal of the binarized similarity is referred to as an occlusion pattern (a pattern indicating the distribution of the possibility of occurrence of occlusion).

Next, a pattern that matches the occlusion pattern thus generated is selected from the occlusion patterns stored in the storage table.

FIG. 3 shows an object 5 with nine image sensors 1 to 9.
Occlusion patterns assumed to occur when 0 is observed. These patterns are prepared in advance,
Stored in a storage table.

In FIG. 3, “×” indicates the reciprocal Q of the similarity.
Indicates the size “0” of ad (i, j, k, z _n ). Each of the stored occlusion patterns includes an X component ox1 (m) of the normalized occlusion vector V (ox1 (m), oy1 (m)),
The Y component oy1 (m) is associated.

Here, the normalized occlusion vector V is a vector in which the reciprocal Qad (i, j, k, z _n ) of the similarity goes from the origin to the direction of “0” and the scalar quantity is 1. It is. The direction of this vector indicates the stereo pair k in which occlusion is likely to occur.

Therefore, an occlusion pattern indicated by an arrow is selected from the storage table of FIG. 3 as the one corresponding to the occlusion pattern generated as shown in FIG. 2B, and the normalized occlusion pattern corresponding to this is selected. From the X component and the Y component (0, 1) of the occlusion vector V, FIG.
The normalized occlusion vector V is referred to as shown in FIG.

When there is no corresponding pattern in the storage table shown in FIG. 3, that is, when the contents of the binarized distribution pattern are all 1, it is determined that "occlusion has not occurred".

Next, the local occlusion vector VQ (QOX (QOX ( i, j, z _n ), QOY (i, j, z _n ))
It is obtained by the following equation.

[0113] _{QOX (i, j, z n} ) = Z × ox1 (m) ... (3-1) QOY (i, j, z n) = Z × oy1 (m) ... (3-2) where, The scalar quantity Z is It is. However, in Expression (3-3), k is a stereo pair in which the reciprocal of the binarized similarity is 0.

As described above, from the origin, the reciprocal Qad (i, j, k, z _n ) of the similarity is at a predetermined level (μ (i, j, z _n ) +
stereo pair k larger than σ (i, j, z _n ))
And the reciprocal of the similarity Qad (i, j, k, z
_n ) is a local occlusion vector V of a scalar quantity Z of a size (smaller as the degree of similarity increases).
Q (QOX (i, j, z n), QOY (i, j, z n)) is obtained.

That is, as shown in FIG. 4 (b), the direction of this vector VQ points to the stereo pair k which has a high possibility that occlusion has occurred, and the magnitude of this vector VQ indicates that the occlusion is The degree of occurrence (the degree of occurrence) is shown.

On the other hand, based on the reciprocal Qad (i, j, k, z _n ) of the similarity calculated for each stereo pair k, the image information of the selected pixel P ₁ of the reference image sensor 1 and all other overall similarity Qsad abutted with each image information of each candidate corresponding points P ₂ to P ₉ of the image sensor 2 to 9 (i, j, z _n) is determined by the following equation (4).

[0117] Here, Qsad (i, j, z _n ) represents the reciprocal of the overall similarity on the assumption that no occlusion has occurred.

(Method 2) Embodiment in which Occlusion Occurrence Patterns are Not Prepared In this method, a local occlusion vector is not obtained from a normalized occlusion vector, but a vector obtained for each stereo pair is synthesized. , The local occlusion vector VQ is determined.

That is, as shown in FIG. 5, the direction cosine vector V indicating the direction from the origin is provided for each stereo pair k.
_k (ox2 (k), oy2 (k)) is obtained. For example, the direction cosine vector V ₁ directed from the origin to the coordinate position of the stereo pair 1 (image sensor 2) is determined as _{V 1 (-1 / √2,1 / √2} ).

When the direction cosine vector V _k is calculated for each stereo pair k, the reciprocal Qad (i, j, k, z _n ) of the corresponding similarity is calculated for these direction cosine vectors V _k .
By There is multiplied, the similarity of the inverse of the magnitude is taken into account the vector _{V k Q (Qad (i,} j, k, z n) × ox2 (k), Qad
(i, j, k, z _n ) × oy2 (k)) for each stereo pair k,
Desired.

The vector V _k Q obtained in this way
(K = 1 to 8) is shown in FIG. FIG.
(A) shows a case where the magnitudes of the vectors V ₄ Q and V ₈ Q are zero.

Next, for each stereo pair k,
Vectors V ₁ Q and V ₂ to which the magnitude of the reciprocal of similarity is added
.. Are added as shown in the following equations (5-1) and (5-2), whereby the local occlusion vector V
Q (QOX (i, j, z n), QOY (i, j, z n)) is obtained.

[0123] That is, as shown in FIG.
Indicates the stereo pair k in which the possibility of occurrence of occlusion is high, and the magnitude of the vector VQ indicates the height (degree of occurrence) of the possibility of occurrence of occlusion.

On the other hand, based on the reciprocal Qad (i, j, k, z _n ) of the similarity calculated for each stereo pair k, the image information of the selected pixel P ₁ of the reference image sensor 1 and all other The reciprocal Qsad (i, j, z _n ) of the overall similarity that is matched with the image information of each of the corresponding candidate points P _{2 to} P ₉ of the image sensors 2 to 9
Is obtained by the following equation (4).

[0125] Here, Qsad (i, j, z _n ) represents the reciprocal of the overall similarity on the assumption that no occlusion has occurred.

Next, the feature vector stabilizing section 413 determines whether the local occlusion vector VQ obtained as described above is obtained.
(QOX (i, j, z n), QOY (i, j, z n)) temporally or spatially global occlusion vector SVQ which stabilized (QSOX (i, j, z n), QSOY ( i, j, z _n )).

In the present embodiment, it is assumed that the space is stabilized.

That is, the following formulas (6-2) and (6-3)
By performing the calculation shown in, global occlusion vector SVQ (QSOX (i, j, z n), QSOY (i, j, z n)) is obtained. In this calculation, a predetermined window area W _ij is set, and a local occlusion vector VQ in the window area is added to obtain a global occlusion vector SVQ in which the occlusion information is stabilized.

At the same time, the reciprocal of the stabilized overall similarity Qsad (i, j, z _n ) obtained by spatially stabilizing the reciprocal Qsad (i, j, z _n ) of the overall similarity by performing the operation shown in the following equation (6-1). Qssad (i, j,
z _n ) is determined.

[0130] _{Qssad (i, j, z n} ) = Σp, q W i, j Qsad (p, q, z n) ... (6-1) QSOX (i, j, z n) = Σp, q W i _{, j QOX (p, q,} z n) ... (6-2) QSOY (i, j, z n) = Σp, q W i, j QOY (p, q, z n) ... (6-3) where W _{i, j} : window area around pixel i, j of reference image sensor 1 for performing pattern matching p, q: window area W _{i, j} for performing pattern matching
In this embodiment, it is assumed that spatial stabilization is performed. However, temporal stabilization may be performed, and distance stabilization may be performed.

Calculation of Correction Value Similarity Next, processing performed by the correction similarity calculation unit 414 will be described.

In the corrected similarity calculating section 414, the reciprocal Qssad of the stabilized overall similarity obtained as described above is obtained.
(i, j, z _n ) is the global occlusion vector SVQ
(QSOX (i, j, z n), QSOY (i, j, z n)) is corrected using a corrected stabilized whole similarity reciprocal Q'ssad (i,
j, z _n ).

The following equation (7) is an equation showing an example of this correction operation.

[0134] _{Q'ssad (i, j, z n} ) = Qssad (i, j, z n) -k 1 × {QSOX (i, j, z n) 2 + QSOY (i, j, z n) 2} ^1/2 (7) where the k ₁ is a positive constant.

As shown in the equation (7), the larger the scalar amount of the global occlusion vector SVQ, the larger the correction amount.

Also, the inverse Qssad (i,
j, z _n ) and the global occlusion vector SV
Q (QSOX (i, j, z n), QSOY (i, j, z n)) by comparing the magnitude of, and determine the presence or absence of occlusion occurs, as a result, it is determined that occlusion has occurred Only when this occurs, a method of performing a correction operation is conceivable.

That is, according to the following equation (8-1),
First, it is determined whether or not occlusion has occurred.

[0138] _{Qssad (i, j, z n} ) -k 2 × {QSOX (i, j, z n) 2 + QSOY (i, j, z n) 2} 1/2 <ε ... (8-1) provided that , K ₂ and ε are positive constants.

When the above equation (8-1) is satisfied, that is, the scalar amount of the global occlusion vector SVQ is relative to the magnitude of the inverse Qssad (i, j, z _n ) of the stabilized overall similarity. Is determined to be occlusion has occurred.

If it is determined that occlusion has occurred, the following (8-
The correction calculation based on the expression 2) is performed.

[0141] _{Q'ssad (i, j, z n} ) = Qssad (i, j, z n) -k 2 × {QSOX (i, j, z n) 2 + QSOY (i, j, z n) 2} ^1/2 (8-2) On the other hand, the (8-1) when the equation is not satisfied, that is stabilized whole similarity reciprocal _{Qssad (i, j, z n} ) with respect to the size of global occlusion vector When the SVQ scalar amount is relatively small, it is determined that occlusion has not occurred.

If it is determined that occlusion has not occurred, the reciprocal Qssad (i, j, z _n ) of the stabilized overall similarity is finally used as shown in the following equation (8-3). Inverse Qssad (i, j, z _n )
Output as

Q′ssad (i, j, z _n ) = Qssad (i, j, z _n ) (8-3) Distance Estimation Finally, the distance estimating unit 415 sets the assumption as shown in FIG. The distance z _n and the inverse Q ′ of the corrected stabilized overall similarity
A correspondence relationship (shown by a broken line) with ssad (i, j, z _n ) is obtained, and from this correspondence relationship, a point having the highest similarity (a point at which the reciprocal of the similarity has the minimum value) is determined. The hypothetical distance z _nx corresponding to the point with the highest similarity is finally estimated as the true distance (the most likely distance) to the point 50 a on the recognition target object 50. Such processing is
This is performed for all pixels for each selected pixel of the reference image # 1.

In FIG. 7, the correspondence indicated by the solid line is a hypothetical distance z _n obtained when no correction is performed in consideration of the conventional occlusion, and a reciprocal Qssad (i, j, z) of the stabilized overall similarity. _n ). Comparing the dashed line with the solid line in FIG. 7, when the conventional occlusion is not corrected, the reciprocal of the similarity at the true distance z _nx is not the minimum value, and the true distance z _nx May not be accurately captured, but as in this embodiment,
When performing correction in consideration of the occlusion, the reciprocal of the similarity true distance z _nx is at the minimum value, it can be seen that it is possible to capture accurately the true distance z _nx. As described above, according to the present embodiment, by performing correction in consideration of the case where occlusion has occurred, the true distance z _nx can be accurately captured, and accurate distance estimation can be performed.

Further, according to the present embodiment, it is possible not only to perform accurate distance estimation in consideration of the case where occlusion has occurred, but also to consider the effect of occlusion as in the conventional configuration shown in FIG. It is not necessary to perform the process of calculating the overall stabilized similarity for each occlusion pattern in order to eliminate the following. Only one reciprocal Qssad (i, j, z _n ) of the stabilized overall similarity is calculated in advance. It is only necessary to keep it, and the amount of calculation is greatly reduced.

In addition, the reciprocal Qss of this stabilized overall similarity
To obtain ad (i, j, z _n ), as in the conventional method, the in-window addition is performed for each stereo pair to obtain the stabilized similarity, and finally, the stability obtained for each stereo pair is obtained. It is not necessary to perform the complicated calculation of adding the generalized similarity to obtain the overall stabilized similarity.

That is, the reciprocal Qsad (i, j, z _n ) of the overall similarity is calculated without performing in-window addition for each stereo pair.
(See the above equation (4)), and only a simple operation of obtaining the stabilized overall similarity Qssad (i, j, z _n ) by stabilizing this (see the above (6-1)) See formula).

Therefore, in the multi-view stereo apparatus, the amount of data to be handled and the amount of processing can be reduced.
Hardware resources can be reduced. This is possible because the conventional multi-view stereo apparatus has a configuration in which a plurality of binocular stereo systems are combined, whereas the basic processing flow of the present embodiment is as follows: Compress image data into information (feature vectors) necessary and sufficient for stereo processing at an early stage. 2) Perform appropriate stabilization processing on feature vectors. 3) Relationship between feature vector information and vectors. This is because there are three stages of finally obtaining the information of the target three-dimensional space by using such a method.

Image processing refers to extracting necessary information (a small amount of compressed information) from a huge amount of data that has spread two-dimensionally. For example, suppose that an apple is recognized from an image of an apple, and information that an apple is present is obtained. In this case, the resulting information is extremely compressed information compared to the original image data. In this case, if sufficient information necessary to determine whether an apple is present can be obtained at the initial stage of the image processing, the subsequent processing only needs to use this information. It can be simplified and the amount of information to be held is reduced.

On the other hand, in the case of image processing in which the same amount of information as the image information is carried out until immediately before the final determination as in the conventional apparatus, the amount of information to be processed and held is Will be large.

When performing image processing, it is important how to quickly and accurately compress the original image into essential information.

If it becomes possible to quickly and accurately compress necessary information from an original image, duplication of processing contents can be eliminated, and efficient processing can be realized. In addition, if information redundancy is eliminated at an early stage, it is not necessary to hold unnecessary information.

The same applies to multi-view stereo processing.

The conventional method of performing the same stereo processing for each stereo pair and then estimating the distance using the result again has redundancy in the processing contents.

On the other hand, if the entire processing is considered as one stereo processing and is compressed into a feature vector necessary for obtaining information of the three-dimensional space at an initial stage, the redundancy as a whole is small, In addition, stereo processing that effectively uses each image information can be executed.

In the conventional multi-view stereo apparatus, as the number of image sensors increases, the processing content and intermediate processing information increase in proportion thereto. On the other hand, in the present embodiment, the information amount of the feature vector as the compression result is always the same, only by increasing the input information for generating the feature vector. Therefore, the entire processing is simplified, and it is possible to flexibly and accurately cope with an increase in the number of image sensors. This means that the desired three-dimensional spatial information can be obtained by making full use of information obtained from a plurality of image sensors.

In the present embodiment, a feature vector (occlusion vector) is generated for the purpose of removing occlusion. However, a feature vector is generated for other purposes, and the similarity is corrected based on the generated feature vector. May be. For example, a feature vector may be generated, and the similarity may be corrected based on the feature vector in order to perform correction in consideration of “protrusion” in which an object protrudes from the image sensor screen and is shot.

That is, the information that the feature vector should have differs depending on the target processing content. In the conventional multi-view stereo apparatus, the similarity of each stereo pair is obtained. However, in the present embodiment, the individual individual similarities are not used as the elements of the feature vector, but are vector information in a form suitable for the purpose. Is different.

When stereo image processing is performed in consideration of occlusion as in the present embodiment, the intensity and direction of occlusion are elements of a feature vector.

On the other hand, in order to perform the correction in consideration of the "protrusion" that an object protrudes from the screen of the image sensor and is taken, the strength and direction of the "protrusion" are elements of the feature vector.

In this embodiment, (6-2), (6)
Although the stabilization processing is performed according to the equation -3), the present invention is not limited to this.

In the stabilizing process, the feature vector is spatially and
This is a process for smoothing temporally.

Since the feature vector is information calculated for each pixel, it may be affected by local noise. In addition, it is necessary to correct individual feature vectors according to the overall tendency or to find a feature vector expressing the tendency in a certain area. In such a case, smoothing is performed by obtaining a partial sum or the like in a certain neighborhood area to obtain a stabilized feature vector.

By performing the stabilization, the characteristic information of each pixel can be adjusted according to the overall tendency and the characteristics of the processing contents. As a result, final processing using the feature vector can be performed more smoothly and effectively.

Therefore, as a method of stabilization, in addition to in-window addition and spatial smoothing, a method of adding feature vectors on the time axis, continuity, and a constraint condition obtained from the characteristics of feature vectors are used. For example, a method of stabilizing by utilizing may be used. These stabilization processes are processes for changing the information content of the feature vector itself according to the surrounding situation.

Further, the stabilization processing is characterized in that the content of the stabilization processing is adjusted according to the purpose of the final processing, rather than simply performing smoothing. Therefore, it is also possible to adjust the strength of stabilization and the processing content of the stabilization processing by using the information of the feature vector itself and other information.

According to the present embodiment, the correction value QOX (
i, j, z _n ) and QOY (i, j, z _n ) and stabilizing them, and using the stabilized correction values QSOX (i, j, z _n ) and QSOY (i, j, z _n ) , the reciprocal of the overall stability of similarity _{Qssad (i, j, z n} ) because so as to correct the reciprocal of the overall stabilization similarity _{Qssad (i, j, z n} ) be obtained If possible, accurate distance estimation will be performed in consideration of the case where occlusion has occurred.

That is, as in the configuration shown in FIG. 22, the overall similarity is calculated by simply adding the reciprocal Qad (i, j, z _n ) of the similarity without performing in-window addition for each stereo pair. Qsad (i, j, z _n ) is obtained (see the above equation (4)), and the reciprocal Qssad (i, j,
z _n ) is simply calculated (see equation (6-1)), and is not limited to the configuration of the apparatus of the present embodiment.

As in the conventional multi-view stereo apparatus shown in FIG. 18, a stabilized similarity is obtained by performing in-window addition for each stereo pair, and finally, a stabilized similarity obtained for each stereo pair. Add the degree and the overall stabilized similarity Q
The present invention can also be applied to a configuration that requires a complicated operation of obtaining ssad (i, j, z _n ).

The final similarity calculating section 215 of the multi-view stereo apparatus shown in FIG. 18 obtains the reciprocal Qssad (i, j, z _n ) of the overall stabilized similarity without considering any occlusion. Here, the global occlusion vector SVQ
Based on the correction value _{QOX (i, j, z n} ), QOY (i, j, z n) if separately sought, this correction value _{QOX (i, j, z n} ), QOY (i, j,
z _n ), the reciprocal Qssad (i, j, z _n ) of the overall stabilized similarity calculated by the final similarity calculation unit 215 is corrected, for example, as in the above equation (7). Is possible. As a result, accurate distance estimation can be performed in consideration of the case where occlusion has occurred.

Next, an embodiment in which more accurate distance estimation can be performed based on the features of the reference image # 1 will be described.

In this embodiment, for each pixel of the reference image # 1 obtained from the reference image sensor 1, features of a two-dimensional image such as a rate of change in brightness and a direction of an edge are extracted and extracted. According to the features, a feature vector stabilizing unit 413, a correction similarity calculating unit 414, and a distance estimating unit 41 shown in FIG.
By controlling the arithmetic processing of No. 5, more accurate distance estimation is performed.

FIG. 8 shows the configuration of the multi-view stereo apparatus of this embodiment.

As shown in the figure, the reference image input unit 501
The image input units 502, 503, and 504 include a reference image input unit 401 and image input units 402, 4 shown in FIG.
03, 404, and the corresponding candidate point coordinate generation unit 50
5 is the same as the corresponding candidate point coordinate generation unit 405 shown in FIG. 1, and the local information extraction units 506, 507, 508
2 is the same as the local information extraction units 406, 407, and 408 shown in FIG.
2 are the same as the similarity calculation units 409, 410, and 411 shown in FIG. 2, the feature vector generation unit 512 is the same as the feature vector generation unit 412 shown in FIG. 22 is the same as the feature vector stabilizing unit 413 shown in FIG.
The same as the correction similarity calculating section 414 shown in FIG. 22, and the distance estimating section 515 is the same as the distance estimating section 415 shown in FIG. 22, and the description already described is omitted.

The reference image two-dimensional feature extraction unit 516 extracts a two-dimensional feature for each selected pixel of the reference image # 1 obtained by the reference image sensor 1. Note that the feature of the reference image # 1 may use information of the target pixel alone, or may use information obtained by performing spatial stabilization on information around the target pixel. Is also good.

The feature vector generation control unit 517 controls the feature vector generation processing performed by the feature vector generation unit 512 according to the two-dimensional features of the reference image # 1 extracted by the feature extraction unit 516. Things.

The feature vector stabilization control unit 518 controls the stabilization processing performed by the feature vector stabilization unit 513 according to the two-dimensional feature of the reference image # 1 extracted by the feature extraction unit 516. Is what you do. That is, the stabilization processing of the feature vector VQ (Equations (6-2) and (6-3))
Alternatively, the window area W _{i, j} used for the stabilization process (Equation (6-1)) of the reciprocal Qsad (i, j, z _n ) of the overall similarity
Processing for changing the size and shape of the window (hereinafter referred to as a window WD) is performed.

The correction similarity calculation control unit 519 performs a correction similarity calculation process performed by the correction similarity calculation unit 514 according to the two-dimensional feature of the reference image # 1 extracted by the feature extraction unit 516. Is controlled.

The control unit 520 for distance estimation includes the feature extraction unit 5
The processing of the distance estimation performed by the distance estimating unit 515 is controlled in accordance with the two-dimensional characteristics of the reference image # 1 extracted in step S16.

Next, the controller 51 for stabilizing the feature vector
8 will be described with reference to FIGS. 9 and 10. FIG.

When the direction of the edge is extracted as the two-dimensional feature of the reference image # 1 It is assumed that the feature extraction unit 516 extracts the direction of the edge 51 as the two-dimensional feature of the reference image # 1. .

Now, assuming that the “edge direction” is extracted from the information of the selected pixel P1 of the reference image # 1, the feature vector VQ or the reciprocal Qsad (i, j, z) of the overall similarity is assumed.
The shape of the window WD used to stabilize _n ) changes from a normal square shape (see FIG. 9A) to the direction of the extracted edge 51 as shown in FIG. 9B. The window WDX is changed to the window WDX having the changed shape.

Here, when the shape of the window WD used to stabilize the feature vector VQ or the reciprocal Qsad (i, j, z _n ) of the overall similarity is uniquely set to a square or the like, The following problems arise.

That is, as shown in FIG. 9A, when the edge 51 in the reference image # 1 is remarkable, there is a possibility that the edge of the object 50 exists near the edge 51. With the window WD of the unique square in this case, greatly affected by background feature vector VQ or overall similarity reciprocal _{Qsad (i, j, z n} ) accuracy of stabilization treatment, hence the distance There is a possibility that the accuracy of the measurement is impaired.

In this respect, in the present embodiment, as shown in FIG. 9B, the shape of the window WDX is changed to a shape along the direction of the edge 51, so that it is not greatly affected by the background. In addition, it is possible to improve the accuracy of the stabilization process, and thus the accuracy of the distance measurement.

In the case of extracting the change rate of lightness as a two-dimensional feature of the reference image # 1 It is assumed that the feature extractor 516 extracts the change rate of lightness as the two-dimensional feature of the reference image # 1. I do.

Now, assuming that the “brightness change rate” is extracted from the information of the selected pixel P ₁ of the reference image # 1, FIG.
As shown in FIG. 0 (a), the absolute value of the extracted change rate of the brightness is added within the window WD of the normal size.

[0188] Therefore, when the added value a is smaller than a certain threshold value is used to stabilize the feature vector VQ or overall similarity reciprocal _{Qsad (i, j, z n} ) of The size of the window WD is changed from a normal size (see FIG. 10A) to a window WDX having a size larger than normal, as shown in FIG. 10B.

This is because if the value a obtained by adding the absolute value of the change rate of the brightness within the window is not equal to or greater than a certain threshold value, the features in the image # 1 are insufficient, and the stabilization is not possible. This is because it is expected to be sufficient. Therefore, in such a case, the size of the window is increased and the image #
In order to stabilize sufficiently, the feature in 1 is taken in more. Sufficient stabilization can improve the accuracy of distance measurement.

Next, the contents performed by the control unit 520 for distance estimation will be described with reference to FIG.

It is assumed that the feature extraction unit 516 extracts a change rate of brightness as a two-dimensional feature of the reference image # 1.

[0192] Now, when the reference image # 1 information of the selected pixel P ₁ and an "brightness change rate" is extracted, the absolute value of the rate of change of the extracted brightness, are added in the window WD You.

Therefore, when the added value a is equal to or larger than a certain threshold value, the rate of change of the reciprocal Q'ssad of the overall stabilization similarity at the true distance is predicted to be large. . Conversely, if this added value a is smaller than a certain threshold value, the rate of change of the reciprocal Q'ssad of the overall stabilized similarity at the true distance is predicted to be small.

That is, in the pixel P ₁ having the large addition value a (the change rate of the lightness is large), the feature of the image # 1 is large, and the reciprocal Q′ss of the overall stabilized similarity near the true distance is obtained.
The rate of change of ad is expected to be large.

Therefore, if it is determined that the added value a is equal to or larger than a certain threshold value (the change rate of brightness is large), as shown in FIG. A minimum point at which the reciprocal Q'ssad of the generalized similarity is changed is searched for, and a hypothetical distance z _x1 corresponding to this is set as a candidate for an estimated distance (true distance).

On the other hand, the inverse Q'ss of the overall stabilization similarity
The hypothetical distance z _x2 in which ad is minimized is searched, and the rate of change of the reciprocal Q′ssad of the overall stabilized similarity near the hypothetical distance z _x2 is obtained. Here, the rate of change in the assumption distance z _x2 vicinity, and the rate of change in the assumed distance z _x1 vicinity already asked to described above are compared, assuming has become larger rate of change distance z _x1 finally Estimated to be true distance.

Conversely, if it is determined that the addition value a is smaller than a certain threshold value (the change rate of brightness is small), the reciprocal Q ′ of the overall stabilization similarity is determined. ssad
It is sufficient to search for a minimum point where the rate of change of is small, and to set a hypothetical distance corresponding to this as a candidate for the estimated distance (true distance).

Here, distance estimating section 320 (or distance estimating section 107, distance estimating section 21) of the conventional stereo apparatus is used.
In 6), as shown in FIG. 14 or FIG. 16, the assumed distance z _nx when the reciprocal Qssad of the similarity becomes the smallest is uniquely estimated as the true distance.

However, as shown in FIG.
Depending on the correspondence relationship between the reciprocal Qssad of the similarity and the reciprocal Qssad of the similarity, instead of the assumed distance z _x2 when the reciprocal Qssad of the similarity becomes the smallest, the value is not the smallest, but the local minimum where the rate of change in the vicinity is large In some cases, the assumed distance z _x1 may be a true distance. In other words, in the related art, when the reciprocal Qssad of the similarity is minimized, the assumed distance z _x2 when the reciprocal Qssad becomes the minimum is uniquely estimated as the true distance. _May be overlooked that the assumed distance z _x1 is a true distance.

In this regard, according to the present embodiment, the distance is estimated in consideration of the rate of change of the similarity other than the magnitude of the similarity, so that the true distance is not missed. , The accuracy of distance estimation can be improved.

In this embodiment, the processing for changing the shape and size of the window shown in FIGS.
Has been described as being applied to the process for stabilizing the feature vector VQ in the feature vector stabilizing unit, but the present invention is not limited to this.
The processing for changing the shape and size of the window shown in FIGS. 9 and 10 may be applied to the similarity stabilization processing in the intra-window addition unit in FIGS. 17, 18, and 21.

Further, in the embodiment, the windows are changed spatially as shown in FIGS. 9 and 10, but the windows may be changed temporally.

Further, in the present embodiment, the processing for obtaining the true distance candidates shown in FIG. 11 has been described as being applied to the true distance estimation processing in the distance estimating unit in FIG. Is not limited to this.
The processing for obtaining the true distance candidates shown in FIG.
The present invention may be applied to the processing of estimating a true distance in the distance estimating unit in FIGS.

In the embodiment, the features of the image are extracted from the reference image # 1 acquired by the reference image sensor 1, but the image # 2 acquired by the other image sensors 2 to N is extracted.
2 to #N.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of an apparatus of the present invention.

FIGS. 2A and 2B are diagrams showing how an occlusion pattern is generated.

FIG. 3 is a diagram illustrating contents of a storage table according to the embodiment;

FIGS. 4A and 4B are diagrams illustrating a process of generating a local occlusion vector from a normalized occlusion vector.

FIG. 5 is a diagram showing normalized occlusion vectors obtained for each stereo pair.

FIGS. 6A and 6B are diagrams for explaining a process of synthesizing a local occlusion vector from a normalized occlusion vector obtained for each stereo pair.

FIG. 7 is a diagram illustrating a correspondence relationship between an assumed distance, a reciprocal of a similarity, and a reciprocal of a corrected similarity;

FIG. 8 is a block diagram showing a configuration of an apparatus according to an embodiment of the present invention that uses two-dimensional features of a reference image.

FIGS. 9A and 9B are diagrams illustrating a process of changing the shape of a window for stabilization.

FIGS. 10A and 10B are diagrams illustrating a process of changing the size of a window for stabilization.

FIG. 11 is a diagram for explaining how a true distance is estimated from two candidate points.

FIG. 12 is a diagram illustrating the principle of image processing of a conventional twin-lens stereo.

FIG. 13 is a diagram for explaining a conventional distance measurement process of a twin-lens stereo.

FIG. 14 is a graph showing a correspondence relationship between an assumed distance and a reciprocal of similarity in the case of binocular stereo.

FIG. 15 is a diagram for explaining the processing content of a conventional multi-lens stereo distance measurement.

FIG. 16 is a graph showing the correspondence between the assumed distance and the reciprocal of the similarity in the case of multi-view stereo.

FIG. 17 is a block diagram showing a configuration of a conventional twin-lens stereo apparatus.

FIG. 18 is a block diagram showing a configuration of a conventional multi-view stereo apparatus.

FIG. 19 is a perspective view showing a state in which two independent objects are imaged.

FIGS. 20A and 20B are diagrams used to explain occlusion. FIGS.

FIG. 21 is a block diagram showing a configuration of a conventional multi-view stereo apparatus.

FIG. 22 is a block diagram illustrating a configuration of an apparatus according to an embodiment of the present invention.

[Explanation of symbols]

 1 to N Image sensor 412, 512 Feature vector generation unit 413, 513 Feature vector stabilization unit 414, 514 Corrected similarity calculation unit 415, 515 Distance estimation unit 516 Reference image two-dimensional feature extraction unit 517 Feature vector generation Control unit 518 control unit for feature vector stabilization 519 control unit for correction similarity calculation 520 control unit for distance estimation

Continued on the front page (72) Inventor Shigeru Kimura 9-1-403, Suugagaoka, Miyamae-ku, Kawasaki-shi, Kanagawa (72) Inventor Katsuyuki Nakano 2-2-30 Nakameguro, Meguro-ku, Tokyo (72) Inventor Hiroyoshi Yamaguchi 2597 Shinomiya, Hiratsuka-shi, Kanagawa Prefecture, Komatsu Ltd.Specialty Equipment Division, Research Department (72) Inventor Tetsuya Shinbo 2597 Shinomiya, Hiratsuka-shi, Kanagawa Prefecture, Komatsu Ltd. 2-8-24 Arima, Miyamae-ku Inside Cyvers Corporation (72) Inventor Masato Ogata 345 Kamimachiya, Kamakura City, Kanagawa Prefecture Inside Mitsubishi Precision Corporation (56) References JP-A-10-283474 (JP, A JP-A-9-231358 (JP, A) JP-A-9-196633 (JP, A) JP-A-9-27969 (JP, A) JP-A-9-9294 (JP, A) JP-A 8- 313212 (JP, A) JP-A-6-265322 (JP, A) (58) Fields investigated (Int. Cl. ⁶ , DB name) G01B 11/00 G01C 3/06 G06T 7/00

Claims (10)

(57) [Claims] A plurality of image pickup means are arranged at a predetermined interval, and correspond to a selected pixel in a picked-up image of the one image pickup means when one of the plurality of image pickup means picks up a target object. An object recognition device that calculates a distance to a point on the object and recognizes the object based on the distance obtained for each pixel; a correspondence candidate of each imaging unit corresponding to the selected pixel; The corresponding candidate point information extracting means for extracting local information such as brightness information near the point for each assumed distance, and the local information near the corresponding candidate point of each imaging means extracted here as input, and A feature vector generating means for generating a feature vector that accurately characterizes the feature, and a feature for performing information stabilization processing by integrating the generated feature vector data positionally or temporally for each element. Estimating information in three-dimensional space by providing a vector stabilizing means and a distance estimating means for estimating the distance to an object using the relationship between the elements of the stabilized feature vectors. Characteristic space recognition device. 2. A feature vector generating means for generating a feature vector including information such as a possible occlusion direction and intensity as an element by using local information of a corresponding candidate point of each imaging means as an input; Using the feature vector stabilization means that performs information stabilization processing by accumulating the data of feature vectors that have been positioned or temporally for each element, and using the relationship between those stabilized feature vector elements 2. The space recognition apparatus according to claim 1, further comprising a distance estimating means for estimating a distance to the object, thereby estimating information in a three-dimensional space from which influence of occlusion of the object has been removed. 3. The method of claim 1, wherein when a target object is imaged by one of the plurality of imaging means, an image obtained by a captured image input means of the one imaging means is used for each selected pixel. A feature extraction unit that extracts a feature of a two-dimensional image such as a change rate or an edge direction of the image, and a feature vector generation unit that changes a feature vector generation method according to the feature extracted by the feature extraction unit. 2. A space recognition apparatus according to claim 1, further comprising means for causing the space to be recognized. 4. A method for controlling the brightness of each selected pixel by using an image obtained by a captured image input unit of one of the plurality of imaging units when a target object is imaged. A feature extraction unit for extracting a feature of a two-dimensional image such as a rate of change or an edge direction, and a space for stabilization by the feature vector stabilization unit according to the feature extracted by the feature extraction unit. 2. The space recognition apparatus according to claim 1, further comprising means for changing a region or a time region. 5. The method according to claim 1, wherein when one of the plurality of imaging units captures an image of the target object, the image obtained by the captured image input unit of the one imaging unit is used to adjust the brightness of each selected pixel. A feature extraction unit for extracting a feature of a two-dimensional image such as a rate of change or an edge direction, and a unit for changing a method of estimating a distance by the distance estimation unit according to the feature extracted by the feature extraction unit. The space recognition apparatus according to claim 1, further comprising: 6. A plurality of imaging means are arranged at a predetermined interval, and correspond to a selected pixel in a captured image of the one imaging means when one of the plurality of imaging means images a target object. A distance to a point on the object to be calculated, and an object recognition device configured to recognize the object based on the distance obtained for each pixel; a correspondence candidate of each imaging unit corresponding to the selected pixel; A corresponding candidate point information extraction process for extracting local information such as brightness information near a point for each assumed distance, and a local feature based on the local information near the corresponding candidate point of each imaging unit extracted here as input. A feature vector generation process for generating a feature vector that accurately characterizes, and a feature for performing information stabilization processing by integrating the generated feature vector data positionally or temporally for each element. Estimating information in 3D space by providing a vector stabilization process and a distance estimation process that estimates the distance to an object using the relationship between elements of these stabilized feature vectors A featured space recognition method. 7. A feature vector generation process for generating a feature vector including information such as a possible occlusion direction and intensity as an element by using local information of a corresponding candidate point of each imaging unit as an input, and generating the feature vector. Using the feature vector stabilization process that performs information stabilization processing by accumulating the feature vector data for each element positionally or temporally, and using the relationship between those stabilized feature vector elements 6. The space recognition method according to claim 5, further comprising: estimating information in a three-dimensional space from which influence of occlusion of the object has been removed by providing a distance estimation step of estimating a distance to the object. 8. A brightness for each selected pixel using an image obtained by a captured image input unit of one of the plurality of imaging units when the target object is imaged by the one of the plurality of imaging units. A feature extraction step of extracting a feature of a two-dimensional image such as a rate of change or an edge direction, and a feature vector generation method is changed in the feature vector generation step according to the feature extracted in the feature extraction step. 6. The space recognition method according to claim 5, further comprising the steps of: 9. A method of controlling the brightness of each selected pixel using an image obtained by a captured image input unit of one of the plurality of imaging units when the target object is imaged. A feature extraction step for extracting a feature of a two-dimensional image such as a rate of change or an edge direction, and a space for stabilizing the feature vector in a stabilization step of the feature vector according to the feature extracted in the feature extraction step. 6. The space recognition method according to claim 5, further comprising a step of changing a region or a time region. 10. A brightness for each selected pixel using an image obtained by a captured image input unit of one of the plurality of imaging units when the target object is imaged by the one of the plurality of imaging units. A feature extraction step of extracting a two-dimensional image feature such as a change rate or an edge direction of the image; and a step of changing a distance estimation method in the distance estimation step according to the feature extracted in the feature extraction step. 6. The space recognition method according to claim 5, further comprising: