JPH11250252A - Three-dimensional object recognizing device and method therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a working visual field corresponding to a working situation without being limited by a working visual field unitarily decided according to the arrangement position or attitude of a picture sensor. SOLUTION: A virtual picture #21 obtained by a virtual picture sensor 21 at the time of image picking-up an object 5 with a desired visual field at a desired position is set instead of a picture #1 of a first image pickup means 1. Then, a position coordinate (Xk , Yk ) (k=1,...,N) of a corresponding candidate Pk in pictures #1-Ν of plural image pickup means 1-N corresponding to a selected picture element P21 specified by a position (i, j) in the set virtual picture #21 is generated for each size of a virtual distance zn from the virtual picture sensor P21 to a point 50a on the object 50 corresponding to the selected picture element P21 . Then, the similarity of the picture information of the position coordinate (Xk , Yk ) (k=1,..., N) of the generated corresponding candidate point Pk is calculated. Thus, an as umption distance znx when the calculated similarity is made the maximum is obtained as a distance from the virtual picture sensor 21 to the point 50a on the object 50 corresponding to the selected picture element P21 , and this distance znx is calculated for each selected picture element.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus and a method for recognizing an object, and more particularly to distance information to an object to be recognized by using triangulation principles from image information obtained by a plurality of image pickup means arranged at different positions. More specifically, the present invention relates to an apparatus and a method which are preferably applied to a case where three-dimensional information of a recognition target object is calculated and an object is recognized using the three-dimensional information.

[0002]

2. Description of the Related Art Hitherto, as a method of measuring a distance to a recognition target object based on an image pickup result of an image sensor as an image pickup means, a measurement method using stereo vision (stereo vision) has been widely known.

This measurement method is a useful method for obtaining three-dimensional information such as distance, depth and depth from a two-dimensional image.

That is, two image sensors are arranged, for example, on the left and right, and the distance to the object is measured by the principle of triangulation from parallax generated when the same image of the object to be recognized is picked up by these two image sensors. It is a method of doing. The pair of left and right image sensors at this time is called a stereo pair,
This is called twin-lens stereo vision because measurement is performed by two units.

FIG. 20 shows the principle of such two-lens stereo vision.

[0006] As shown in FIG.
In image # 1 (obtained on imaging surface 1a) and image # 2 (obtained on imaging surface 2a) of left and right image sensors 1 and 2,
It is necessary to measure a parallax (disparity) d, which is a difference between the positions of the corresponding points P ₁ and P ₂ . In general, the following equation is established between the parallax d and the distance z to a point 50a (a point on the recognition target object 50) in the three-dimensional space.

Z = F · B / d (1) where B is the distance (base line length) between the left and right image sensors 1 and 2, and F is the lens 31 of the image sensor 1 and the lens of the image sensor 2. 32 focal length. Usually, since the base line length B and the focal length F are known, if the parallax d is known, the distance z can be uniquely obtained. The parallax d can be calculated by searching for a point corresponding to which point between the images # 1 and # 2. Other image # 2 corresponding to the one of the image # point P ₁ on 1 at this time
To a point P ₂ of the upper and is referred hereinafter as "corresponding points", to explore the corresponding points, it will be hereinafter referred to as "corresponding point search." When assuming a distance to an object 50, that the following referred to as "candidate corresponding points" to a point on the other image # 2 corresponding to the point P ₁ on one of the image # 1 to be searched with this assumption distance To When performing measurement by binocular stereo vision, as a result of the corresponding point search, if it is possible to detect the true corresponding point P ₂ corresponding to the true distance z, it was possible to calculate the true disparity d That is, at this time, it can be said that the true distance z to the point 50a on the object 50 has been measured.

By executing such processing for all pixels of one image # 1, an image (distance image) in which distance information is added to all selected pixels of image # 1 is generated. The processing for finding the true distance by searching for the corresponding point will be described in detail with reference to FIGS. 21, 22, and 23.

FIG. 23 is a block diagram showing the configuration of a conventional distance measuring device (object recognizing device) based on binocular stereo vision.

The reference image input unit 101 receives a reference image # 1 captured by the image sensor 1 serving as a reference when calculating the parallax d (distance z). On the other hand, the image input unit 1
In 02, an image # 2 of the image sensor 2 which is an image having a corresponding point corresponding to a point on the reference image # 1 is captured.

Next, the processing in the corresponding candidate point coordinate generator 103, the local information extractor 104, the similarity calculator 105, and the distance estimator 106 will be described with reference to FIG. So for each pixel of the reference image # 1, for each distance z _n assuming the image sensor 2
The position coordinates of the corresponding candidate point of the image # 2 are stored and stored, and by reading this, the position coordinates of the corresponding candidate point are generated.

That is, the reference image # of the reference image sensor 1
1, a pixel P ₁ specified by the position (i, j) is selected, and a distance z _n to the recognition target object 50 is assumed. Then, the position coordinates (X ₂ , Y ₂ ) of the corresponding candidate point P ₂ in the image # 2 of the other image sensor 2 corresponding to the assumed distance z _n are read.

Next, the local information extracting section 104 executes a process of extracting local information based on the position coordinates of the corresponding candidate points generated by the corresponding candidate point coordinate generating section 103 in this way. Here, the local information is image information of a corresponding candidate point obtained in consideration of a pixel near the corresponding candidate point.

Further, the similarity calculating section 105 calculates the similarity between the local information F _{2 of} the corresponding candidate point P ₂ obtained by the local information extracting section 104 and the image information of the selected pixel P ₁ of the reference image. You.

More specifically, pattern matching between the area around the selected pixel of the reference image # 1 and the area around the corresponding candidate point of the image # 2 of the image sensor 2 allows the areas of both images to be compared. The comparison is performed to calculate the similarity. That is, the similarity stabilization process is performed.

That is, as shown in FIG. 21, a window WD ₁ centered on the position coordinates of the selected pixel P ₁ of the reference image # ₁ is cut out, and the image # 2 of the image sensor 2 is cut out.
Window WD ₂ around the second position coordinates of the corresponding candidate point P ₂ is cut out, by performing pattern matching for these windows WD _1, WD ₂ together, these similarities are calculated. This pattern matching is performed for each assumed distance z _n . Then, the same pattern matching is performed for all pixels for each selected pixel of the reference image # 1.

FIG. 22 is a graph showing the correspondence between the assumed distance z _n and the inverse similarity Qs. Window W in FIG.
As a result of matching between D ₁ and the window WD ′ ₂ centered on the position of the corresponding candidate point when the assumed distance is z ′ _n, a large value is obtained as the reciprocal Qs of the similarity as shown in FIG. (Similarity is smaller),
The window WD ₁ in FIG. 21, matching the results of the corresponding window WD ₂ around the position of the candidate point when assumptions distance z _nx is reciprocal Qs of similarity as shown in FIG. 22 are small (The similarity is increased).

The similarity is generally obtained as the absolute value of the difference between the selected pixel to be compared and the image information of the corresponding candidate point, or the sum of squares of the difference. In this way, from the correspondence between the hypothetical distance z _n and the reciprocal Qs of the similarity, the point having the highest similarity (the point at which the reciprocal Qs of the similarity has the minimum value) is determined. Hypothetical distance z corresponding to the point
Finally, _nx is estimated as the true distance (the most probable distance) to the point 50a on the recognition target object 50.

That is, the corresponding candidate point P ₂ corresponding to the assumed distance z _nx in FIG. 21 is determined to be the corresponding point for the selected pixel P ₁ . As described above, the distance estimating unit 106 determines the one having the highest similarity from among the similarities obtained by sequentially changing the assumed distance z _n for the selected pixel of the reference image # 1, and determines the most similar one. The assumed distance z _{nx at} which the degree becomes higher is estimated as the true distance and is output.

Although the above description has been made of the case of the binocular stereo vision, three or more image sensors may be used. Performing distance measurement (object recognition) using three or more image sensors is referred to as distance measurement (object recognition) by multi-view stereo vision.

[0021] Multi-view stereo vision has recently been often used because the ambiguity of corresponding points can be reduced and the reliability can be remarkably improved. In this multi-view stereo distance measurement device (object recognition device), a plurality of image sensors are divided into stereo pairs consisting of two image sensors, and each stereo pair is subjected to the principle of the above-described two-view stereo vision. Is applied repeatedly.

That is, a reference image sensor is selected from a plurality of image sensors, and a stereo pair is formed between the reference image sensor and another image sensor. Then, the processing in the case of binocular stereo vision is applied to each stereo pair. As a result, the distance from the reference image sensor to the recognition target present in the field of view of the reference image sensor is measured.

The relationship between stereo pairs in a conventional multi-view stereo will be described with reference to FIG. 24. In the twin-lens stereo shown in FIG. 21, the corresponding image paired with the reference image # 1 is # 2.
However, in the multi-view stereo, the pair of the reference image # 1 and the image # 2 of the image sensor 2, the pair of the reference image # 1 and the image # 3 of the image sensor 3, ..., the reference image # 1 Image sensor N
There are a plurality of stereo pairs such as the pair of image #N.

Before performing the processing based on each stereo pair of the corresponding image and the reference image, it is necessary to consider the mounting distortion of the camera serving as the image sensor, and the correction processing by calibration is usually performed in advance. I have. The process of searching for a corresponding point by multi-view stereo to obtain a true distance is the same as that of the above-described binocular stereo shown in FIGS.
24 and FIG. 25 corresponding to FIG.

FIG. 24 is a view for explaining the configuration of a conventional distance measuring device (object recognition device) based on multi-view stereo vision. The image sensors 1, 2, 3,..., N are horizontal,
It is assumed that they are arranged at predetermined intervals in the vertical or oblique direction (for convenience of explanation, FIG. 24 shows a case where they are arranged on the left and right at regular intervals).

The reference image input unit 201 shown in FIG. 26 receives a reference image # 1 captured by the image sensor 1 serving as a reference when calculating the parallax d (distance z). On the other hand, the image input unit 202 receives an image # 2 of the image sensor 2 which is an image having a corresponding point corresponding to a point on the reference image # 1. Also in the other image input units 203 and 204, the image # 3 of the image sensor 3 corresponding to the reference image # 1 and the image #N of the image sensor N corresponding to the reference image # 1 are captured.

[0027] In the corresponding candidate point coordinate generating unit 205, for each selected pixel P ₁ of the reference image # 1, for each distance z _n assuming the position coordinates of the corresponding candidate point P ₂ of the image # 2 image sensor 2 , storage location coordinates of the corresponding candidate point P ₃ of the image # 3 image sensor 3, the position coordinates of the corresponding candidate point P _N of the image #N image sensors N, respectively, are stored, each corresponding by reading these Generate the position coordinates of the candidate points.

That is, the reference image # of the reference image sensor 1
1, a pixel P ₁ specified by the position (i, j) is selected, and a distance z _n to the recognition target object 50 is assumed. The position coordinates (X ₂ , X ₂ ) of the corresponding candidate point P ₂ in the image # 2 of the image sensor 2 corresponding to the assumed distance z _n
Y ₂ ) is read.

[0029] Similarly, the selected pixel P ₁ of the reference image # 1, left position coordinates of the corresponding candidate point P ₃ of the image # 3 of the image sensor 3 corresponding to the assumed distance _{z n (X 3, Y 3} ) is read And
Selected pixels P ₁ of the reference image # 1, the position coordinates (X _N, Y _N) of the corresponding candidate points P _N of the image #N image sensors N which corresponds to the assumed distance z _n is read. Then, similar reading is performed by sequentially changing the assumed distance z _n . Similar reading is performed by sequentially changing the selected pixels. Position coordinates (X ₂ corresponding candidate point P ₂ Thus,
Y _2), the position coordinates (X _3, Y ₃ corresponding candidate points P _3), the position coordinates (X _N corresponding candidate point P _N, Y _N) is outputted from the corresponding candidate point coordinate generating unit 205.

Next, the local information extracting section 206 extracts local information based on the position coordinates (X ₂ , Y ₂ ) of the corresponding candidate point P ₂ generated by the corresponding candidate point coordinate generating section 205 in this way. The processing is executed. In the same manner, the local information extracting unit 207 performs the correspondence based on the position coordinates (X ₃ , Y ₃ ) of the corresponding candidate point P ₃ of the image # 3 of the image sensor 3 generated by the corresponding candidate point coordinate generating unit 205. local information of the candidate point P ₃ is in the local information extracting unit 208, the position coordinates of the corresponding candidate point P _N of the image #N image sensors N which is generated in the corresponding candidate point coordinate generating unit 205 (X _N, Y _N) , Local information of the corresponding candidate point P _N is obtained.

Further, the similarity calculating section 209 calculates the similarity between the local information F _{2 of} the corresponding candidate point P ₂ obtained by the local information extracting section 206 and the image information of the selected pixel P ₁ of the reference image # 1. Is calculated. Specifically, by performing pattern matching between a region around the selected pixel of the reference image # 1 and a region around the corresponding candidate point of the image # 2 of the image sensor 2, the regions of both images are compared. , The degree of similarity is calculated.

That is, as shown in FIG. 24, a window WD ₁ centered on the position coordinates of the selected pixel P ₁ of the reference image # ₁ is cut out, and the image # 2 of the image sensor 2 is cut out.
Window WD ₂ around the second position coordinates of the corresponding candidate point P ₂ is cut out, by performing pattern matching for these windows WD _1, WD ₂ together, these similarities are calculated. This pattern matching is performed for each assumed distance z _n .

FIG. 2 (1) shows the assumed distance z _n and the reciprocal Q of the similarity between the stereo pair (reference image sensor 1 and image sensor 2).
It is a graph showing a relationship between a s _1.

[0034] the window WD ₁ of FIG. 24, as a result of assumptions distance was matching with the ₂ 'window WD around the position coordinates of the corresponding candidate points when the _n' where z is 2 (1)
A large value is obtained as the reciprocal Qs of similarity as shown in (similarity is smaller) is the window WD ₁ of FIG. 24, the position coordinates of the corresponding candidate points when assuming the distance is z _nx as a result of matching with the window WD ₂ centered at the reciprocal Qs similarity is smaller as shown in FIG. 2 (1) (similarity is larger)
I understand.

[0035] In the similarity calculation unit 210 in the same manner, the reference image # the window WD ₁ around the 1 position coordinates of the selected pixel P _1, corresponding candidate point P ₃ of the image # 3 image sensors 3
Pattern matching between the window WD ₃ around the position coordinates of the runs, these similarities are calculated.
Then, since pattern matching is performed for each assumed distance z _n , the stereo pair (image sensor 1 and image sensor 3) also has the assumed distance z _n and the reciprocal Qs of the similarity as shown in FIG. Correspondence with ₂ is required.

Similarly, in the similarity calculation unit 211, the corresponding candidate point P _N between the window WD ₁ centered on the position coordinates of the selected pixel P ₁ of the reference image # ₁ and the image #N of the image sensor _N
Pattern matching between the window WD _N around the position coordinates of the runs, these similarities are calculated.
Then, since pattern matching is performed for each assumed distance z _n , the stereo pair (image sensor 1 and image sensor N) is also subjected to the assumed distance z _n and the reciprocal Qs of similarity as shown in FIG. Correspondence with _N is required.

Finally, the correspondence between the assumed distance z _n obtained for each stereo pair and the reciprocal of the similarity is added for each assumed distance. Further, as shown in FIG.
_From the correspondence between _n and the reciprocal of the similarity, the point having the highest similarity (the point at which the reciprocal of the similarity has the minimum value) is determined, and the highest similarity is determined. The hypothetical distance z _nx corresponding to the point is finally estimated as a true distance (the most likely distance) to the point 50 a on the recognition target object 50. This process is performed for all pixels for each selected pixel of the reference image # 1.

As described above, the distance estimating section 212 determines the one having the highest similarity addition value from the similarity addition values obtained by sequentially changing the assumed distance z _n . The hypothetical distance z _{nx at} which the sum of the similarities becomes the highest is estimated as the true distance and is output. Then, since such distance estimation is performed for all pixels of the reference image # 1, an image (distance image) in which distance information is added to all selected pixels of the reference image # 1 is generated.

[0039]

SUMMARY OF THE INVENTION The present invention is to solve the following problems.

(1) The work field of view is fixed and cannot be set to a desired work field of view.

As described above, according to the conventional measuring method, a reference image sensor for obtaining an image to be a reference is prepared, whether it is a binocular stereo or a multi-view stereo, and distance measurement is performed based on the reference image. Since it is performed, it is possible to generate only a distance image viewed from an observation field of view (referred to as a field of view obtained from an actual image sensor) determined by the arrangement position of the reference image sensor and the inclination at the time of attachment.

That is, once the image sensor is fixed, the observation field of view is uniquely determined, and a distance image viewed from a desired observation field cannot be generated. However, in actuality, when recognizing an object, there is a request to change the observation field of view according to the work situation and to perform the work with a desired work view (a view necessary for the work).

For example, as shown in FIG.
When the work is performed by mounting the image sensor group 11 on the vehicle 60, not only the work field of view in front of the vehicle 60 as viewed from the image sensor fixed to the vehicle 60, but also the vehicle 60 depending on the situation.
There is a request to work while observing an obstacle with a work field of view that looks down at an obstacle in front.

The first invention of the present invention has been made in view of such circumstances, and is not limited to a work field determined by the arrangement position and attitude of the image sensor, and can obtain a work field according to the work situation. This is a first solution.

(2) There are restrictions on the type of optical system of the image sensor.

The conventional measurement based on multi-view stereo vision is based on the principle of binocular stereo, and since it is necessary to compare the similarity on the same basis for all stereo pairs, the multi-view stereo is used. Are required to have exactly the same optical characteristics.

Therefore, image sensors having different lens characteristics cannot be used at the same time, and image sensors having different sensitivities cannot be used at the same time.

Here, in general, when distance measurement is performed outdoors or the like, a wide-angle lens and a telephoto lens are used at the same time to improve the measurement capability according to the distance to be measured, or according to the situation of external light. There is a demand to use two types of image sensors having different sensitivities, that is, a monochrome image sensor having a normal sensitivity and an infrared image sensor.

However, in the conventional multi-view stereo system, since the optical characteristics of the image sensors need to be exactly the same, two types of image sensors having different optical characteristics cannot be used at the same time. Was.

In this case, two types of image sensors, a monochrome image sensor group with normal sensitivity and an infrared image sensor group, are prepared, and a reference image sensor is set for each image sensor group to perform measurement. However, since the reference image sensors are disposed at different positions, it has been impossible to make a comprehensive judgment by comparing the measurement results of these two types of image sensors. Further, it is impossible to make the arrangement positions of the respective reference image sensors physically identical.

The second invention of the present invention has been made in view of such a situation. In a multi-view stereo measurement, by setting a common virtual visual field, two or more types of image sensor groups having different imaging conditions are set. The second solution is to improve the accuracy of the measurement by making it possible to judge the measurement results in total.

(3) There are restrictions on the combinations of image sensors.

Conventional measurement using multi-view stereo vision presupposes the principle of binocular stereo, and it is necessary to compare the similarity of all stereo pairs based on the same reference image sensor. Once the reference image sensor is determined, a plurality of stereo pairs are fixed.
Then, it is necessary to compare the similarities of all stereo pairs on the same basis. Therefore, depending on the situation, it is not possible to sequentially select stereo pairs having different references, correct information on similarity from a specific stereo pair, or adopt a different similarity calculation method.

Here, a specific image sensor pair (or a combination of three or more) according to the working field of view that changes every moment.
Are sequentially selected, and there is a demand to perform a detailed measurement according to the situation change, but this cannot be dealt with.

The third invention of the present invention has been made in view of such a situation. In a multi-view stereo measurement, a stereo pair is not fixed and measurement is performed according to an ever-changing work field. A third solution is to sequentially select a specific stereo pair of image sensors (or a combination of three or more image sensors) and perform fine measurement corresponding to a situation change. is there.

(4) Inefficiency caused by recognizing an object by using a distance measurement result.

The general method of the conventional object recognition is to first measure the distance to the object in the entire space, and then analyze the distance information to obtain features such as the distance to the obstacle, the shape and the size. It recognizes.

However, if the condition of the visual field is uniform as in the case where the vehicle 60 is traveling on the road 61 as shown in FIG. 26, it is inefficient to measure the distance in all the spaces. It is a target.

In other words, when the location of the road 61 is known in advance and an obstacle for the vehicle 60 can be expected to exist immediately above the road 61, the area in the air or under the road is determined. However, performing the distance measurement involves a lot of waste in the arithmetic processing, and the arithmetic processing becomes enormous. Further, a process for recognizing the object is required after the process of the distance measurement. However, this process is complicated even when recognizing an obstacle on a simple plane. As described above, according to the conventional measurement method, not only is it not possible to efficiently discriminate an object, but also there is a possibility that an error occurs in recognition.

The fourth invention of the present invention has been made in view of such a situation. When the shape of an object to be recognized is specified in advance, measurement can be performed more efficiently without erroneous recognition. The fourth solution is to make it possible.

[0061]

Means and effects for solving the problems
In the main invention of the present invention, in order to achieve the first solution, a plurality of imaging means are arranged at a predetermined interval, and one of the plurality of imaging means when one of the plurality of imaging means images a target object. The information of the corresponding candidate point in the image picked up by the other image pickup means corresponding to the selected pixel in the image picked up by the image pickup means is the assumed distance from the one image pickup means to the point on the object corresponding to the selected pixel. Is extracted for each size, and the similarity between the image information of the selected pixel and the image information of the corresponding candidate point is calculated. A distance from the means to a point on the object corresponding to the selected pixel; and an object recognition device configured to recognize the object based on the distance obtained for each selected pixel. In front of the field of view A virtual image information set by the virtual image information set by the virtual image information set by the virtual image information set by the virtual image pickup means when the object is imaged, instead of the image picked up by the one image pickup means. The information of the corresponding candidate points in the captured images of the plurality of imaging units corresponding to the selected pixels in the assumption from the viewpoint set by the virtual visual field information setting unit to the point on the object corresponding to the selected pixels Corresponding candidate point information extracting means for extracting for each magnitude of distance, similarity calculating means for calculating the similarity between the image information of the corresponding candidate points extracted by the corresponding candidate point information extracting means, and the similarity calculating The assumed distance when the similarity calculated by the means is the largest, the distance from the viewpoint set by the virtual visual field information setting means to a point on the object corresponding to the selected pixel, this distance So that comprising a distance estimation means for obtaining for each selected pixel.

According to such a configuration, as shown in FIGS. 1 and 4, the virtual visual field information setting unit 305 uses the virtual image pickup means 21 to image the object 50 at a desired position with a desired visual field. The image # 21 is set instead of the image # 1 of one imaging unit 1.

[0063] Then, in the corresponding candidate point coordinate generating unit 306, a plurality of image pickup means corresponding to the selected pixel P ₂₁ specified by the position in the virtual image # 21 is set (i, j). 1 to
Position coordinates of the corresponding candidate point P _k of the image # 1 to # in N of _{N (X k, Y k)} (k = 1, ..., N) and generated every assumption distance z _n.

Then, in the similarity calculation means 311 to 313, the position coordinates (X _k , Y _k ) of the generated corresponding candidate point P _k
k ) (k = 1,..., N) is calculated for the similarity between the pieces of image information.

Then, in the distance estimating means 314, the assumed distance z _nx at which the calculated sum of the similarities becomes the largest is determined by the point 50 a on the object 50 corresponding to the selected pixel P ₂₁ of the virtual imaging means 21. It is calculated as the distance to

As described above, according to the present invention, it is possible to arbitrarily set a virtual image pickup means 21 having a desired observation field of view instead of one image pickup means 1 serving as a reference image sensor. The work field of view according to the work situation can be acquired without being limited to the work field of view determined by the arrangement positions and postures of the sensors 1 to N. In particular, even in places where it is difficult to physically place the image sensor or where it is difficult to use the image sensor, the same field of view as that where the actual reference image sensor is placed in that place. A range image can be obtained.

Further, the following various effects can be obtained.

That is, according to the present invention, since the observation field of view, which is a reference for performing space recognition and object recognition, can be freely changed, the subsequent processing can be performed efficiently.

In this respect, in the conventional measuring method, the working visual field actually obtained is limited to the observation visual field obtained by any of the actual image sensors 1 to N. For this reason, when trying to recognize or judge an object according to the acquired observation field of view,
It was necessary to acquire a working field of view according to the purpose, and therefore, it was necessary to perform coordinate conversion and physically change the direction of the image sensor. Also, the algorithm for recognition sometimes becomes complicated.

According to the present invention, the working field of view can be freely changed in accordance with the subsequent processing such as recognition and judgment, so that complicated downstream processing such as physically changing the orientation of the image sensor can be performed. Processing such as recognition and determination at a later stage can be easily and efficiently performed without requiring processing.

According to the present invention, virtual image # 21,
That is, when local information from each image sensor is extracted based on the virtual visual field, calibration of each image sensor is performed at the same time. For this reason,
It is not necessary to add a new process only for the calibration, and the calibration can be executed efficiently.

Further, according to the present invention, the virtual image # 21, that is, the virtual visual field, can be arbitrarily changed through arithmetic processing without actually changing the arrangement position of the image sensor. Therefore, it is not necessary to provide a mechanism for mechanically changing the position and the posture of the image sensor. For this reason, it is possible to construct a highly reliable system without any mechanical failure. Therefore, it is also possible to change the virtual visual field according to the movement of the human eye by applying such “electronic swinging”. By constructing such a system, it is possible to generate an image in the case where the user is in a remote place, as if at the observation point, and turns his or her eyes freely.

In the main invention of the second invention, in order to achieve the second solution, a plurality of imaging means are arranged at predetermined intervals, and one of the plurality of imaging means is used to detect a target object. The information of the corresponding candidate point in the image picked up by the other image pickup means corresponding to the selected pixel in the image picked up by the one image pickup means when the image is picked up is displayed on the object corresponding to the selected pixel from the one image pickup means. Is extracted for each size of the assumed distance to the point, and the similarity between the image information of the selected pixel and the image information of the corresponding candidate point is calculated. The assumed distance when the calculated similarity is the largest Is the distance from the one imaging means to a point on the object corresponding to the selected pixel, in the object recognition device to recognize the object based on the distance obtained for each selected pixel,
The plurality of imaging units are classified into at least two types of imaging units having different conditions for imaging the object, and are virtual imaging units when the object is imaged at a desired position with a desired field of view. The virtual captured image according to, instead of the captured image of the one image capturing means, a virtual visual field information setting means to set, and each of the above-mentioned corresponding to a selected pixel in the virtual image set by the virtual visual information setting means Correspondence to extract information on the corresponding candidate points in the captured image of the imaging means group for each magnitude of the assumed distance from the viewpoint set by the virtual visual field information setting means to the point on the object corresponding to the selected pixel Candidate point information extracting means; similarity calculating means for calculating the similarity between the image information of the corresponding candidate points extracted by the corresponding candidate point information extracting means for each of the imaging means groups; A fusion similarity is obtained based on the similarity of each of the imaging means groups calculated by the degree calculation means, and the assumed distance when the fusion similarity is maximized is set by the virtual visual field information setting means. And a distance estimating means for obtaining a distance from the viewpoint to a point on the object corresponding to the selected pixel and obtaining the distance for each selected pixel.

According to such a configuration, for example, in the case where two groups of imaging means are described, as shown in FIG. 7B, a plurality of imaging means 11 ″ have different conditions for imaging the object 50.
Are classified into first imaging means groups 141, 142,... And second imaging means groups 151, 152,.

Then, as shown in FIGS. 1 and 9, the virtual visual field information setting means 505 converts the virtual image # 21 by the virtual imaging means 21 when the object 50 is imaged at a desired position with a desired visual field. , One imaging means 141, 1
51 are set instead of the images # 141 and # 151.

[0076] Then, in the corresponding candidate point coordinate generating unit 506, the first imaging means groups corresponding to the selected pixel P ₂₁ specified by the position in the virtual image # 21 is set (i, j) 1
, And the corresponding coordinates of the corresponding candidate points _Pk in the images # 151, # 152,... Of the second imaging means groups 151, 152,. X _k , Y _k ) are generated for each magnitude of the assumed distance z _n .

Then, the similarity calculating means 511 and 512 calculate the position coordinates (X _k , Y _k) of the generated corresponding candidate point P _k.
k ), the degree of similarity between the pieces of image information
1, 142,..., A second imaging unit group 151, 152,
.., Are calculated for each. And the distance estimating means 5
In FIG. 15, as shown in FIGS. 10A, 10 </ b> B, and 10 </ b> C, the calculated first imaging unit groups 141 and 14 are calculated.
2, the similarity (reciprocal similarity Qs ₁₎ and second image pickup means group 151 and 152 for ..., (similarity reciprocal Qs ₂₎ similarity ... for a fused fusion similarity (similarity Assumed distance z _nx when reciprocal addition value Qs ₁₂ ) is largest
Is obtained for each selected pixel.

As described above, in the measurement by the multi-view stereo, the virtual image pickup means (virtual image sensor) 21 is set, and this is set to the first image pickup means group (first image sensor group) and the second image pickup means. Since the reference image sensor is used in common for the group (second image sensor group) (since it is not necessary to separately set a reference image sensor for each imaging unit group), the bright-field image sensor group and the dark-field image sensor are used. Sensor groups, or monochrome image sensors with normal sensitivity, and even between image sensors with different optical characteristics, such as infrared image sensors,
It is possible to make a comprehensive judgment by fusing the measurement results of these two types of image sensors. As a result of comprehensively determining the measurement results of the two types of image sensors having different imaging conditions, the accuracy of the measurement can be dramatically improved.

Although the above description has been made for the case of two types of imaging means groups, the same applies to the case of three or more types of imaging means groups.

In the main invention of the third invention, in order to achieve the third solution, a plurality of imaging means are arranged at a predetermined interval, and one of the plurality of imaging means is used to detect a target object. The information of the corresponding candidate point in the image picked up by the other image pickup means corresponding to the selected pixel in the image picked up by the one image pickup means when the image is picked up is displayed on the object corresponding to the selected pixel from the one image pickup means. Is extracted for each size of the assumed distance to the point, and the similarity between the image information of the selected pixel and the image information of the corresponding candidate point is calculated. The assumed distance when the calculated similarity is the largest Is the distance from the one imaging means to a point on the object corresponding to the selected pixel, in the object recognition device to recognize the object based on the distance obtained for each selected pixel,
The plurality of image pickup units are classified into at least two types of image pickup unit groups having different conditions for imaging the object, and at least one image pickup unit from the image pickup unit group according to the image pickup condition of the object. A means group, an imaging means group selection means for selecting as an imaging means group to be actually used, and a virtual image captured by a virtual imaging means at a desired position when capturing the object with a desired field of view, Instead of the image picked up by the one image pickup means, a virtual field information setting means to be set, and an image picked up image of the selected image pickup means group corresponding to a selected pixel in the virtual image set by the virtual field information set means. Extraction of corresponding candidate point information for each magnitude of an assumed distance from the viewpoint set by the virtual field of view information setting means to a point on the object corresponding to the selected pixel And a similarity calculating means for calculating the similarity between the image information of the corresponding candidate points extracted by the corresponding candidate point information extracting means for the selected group of imaging means; The assumed distance when the similarity for the selected imaging unit group is the largest is a distance from the viewpoint set by the virtual visual field information setting unit to a point on the object corresponding to the selected pixel, Distance estimating means for obtaining a distance for each selected pixel.

According to such a configuration, as shown in FIG.
The image sensor group 1 that is a plurality of image pickup units according to the image pickup condition (the speed V of the vehicle 60, the traveling direction φ) of the work visual field Ar.
Image sensors 2 and 3, which are at least two image pickup means to be actually used, are selected from among the image sensors 2 and 3.

A virtual image # 21 obtained by the virtual image sensor 21 shown in FIG. 1 when the road 61 is imaged at a desired position according to the imaging conditions of the road 61 with a desired visual field.
Is set in place of the image # 1 of one imaging unit 1.
Then, the position (i, j) in the set virtual image # 21
In the corresponding candidate point P in the image of the at least two imaging means 2, 3 said selected corresponding to the selected pixel P ₂₁ specified
_k position coordinates (X _k, Y _k) generated for each size of the assumed distance z _n a.

Then, the similarity between the image information of the position coordinates (X _k , Y _k ) of the generated corresponding candidate point P _k is calculated.

Then, the assumed distance z _nx at which the calculated similarity becomes _maximum is set as the distance to a point on the object in the work field of view Ar and is obtained for each selected pixel.

As described above, according to the present invention, in the measurement using the multi-view stereo, instead of performing the measurement while fixing the stereo pair, the measurement is performed according to the working field Ar that changes every moment.
Since the image sensors 2 and 3 of a specific stereo pair can be sequentially selected, fine measurement corresponding to a change in the situation (a change in the speed V of the vehicle 60, a change in the traveling direction φ, etc.) can be performed.

According to the fourth aspect of the present invention, in order to achieve the fourth solution, a plurality of image pickup means arranged at predetermined intervals for picking up an object to be recognized and a model group corresponding to the object to be recognized are assumed. Along with a hypothetical model information setting means for setting the position coordinates of each point on each model in the model group, and the one when the recognition target object is imaged by one of the plurality of imaging means. For each hypothetical model selected from the model group, the information of the corresponding candidate point in the captured image of the other imaging means corresponding to the selected pixel in the captured image of the imaging means is used by using the position coordinates of the hypothetical model. Candidate corresponding point information extracting means, and similarity calculating means for calculating the similarity between the image information of the selected pixel and the image information of the corresponding candidate point extracted by the corresponding candidate point information extracting means,
A model estimating unit that sets the hypothetical model when the similarity calculated by the similarity calculating unit is the largest as a model of a point corresponding to the selected pixel, and obtains this model for each selected pixel. I have to.

According to such a configuration, as shown in FIG. 13, in the case where the road 61 is recognized using the image sensor group 11 on the vehicle 60, the processing is efficiently performed as follows. be able to.

That is, as shown in FIG. 27, the reference image input unit 801 and the other image input units 802 to 804
, A plurality of reference images and corresponding images of the road 61 to be recognized are fetched, and the model M ₁ , M ₂ , M ₃ , and M ₄ corresponding to the road 61 are determined by the hypothetical model information setting unit 805 as parameters θ.
Are generated for each of the values θ ₁ , θ ₂ , θ ₃ , θ ₄ .

Then, as shown in FIG. 28, the corresponding candidate point coordinate generator 806 outputs another image corresponding to the selected pixel P ₁ specified by the position (i, j) in the image # 1 of the reference image sensor 1. position coordinates (X _k, Y _k) of the corresponding candidate point P _k in the image #k of the sensor k and selected hypothesized model M _i (i =
1 to 4).

Further, local information extracting sections 807 to 810 extract respective local information, and calculate similarity calculating sections 811 to 811.
In step 813, each similarity is calculated by comparing with the local information of the reference image.

Then, the model estimating unit 814 generates
As shown in the example, a model in which the sum of the reciprocals of the calculated similarity is the largest (here, a model M _2.6 between M ₃ and M ₄ [θ = 3 degrees] it is also possible to obtain the finer than the width value.) is selected pixel P ₁
Is estimated as a model corresponding to This model is obtained for each selected pixel, and the most frequent model (M ₃ ) among the models obtained for each selected pixel is the road 61.
Is recognized as a model indicating

As described above, when the shape of the object to be recognized is specified in advance, the model is modeled to determine the object to be recognized. Since it is not necessary to perform a complicated process of performing distance measurement and specifying and extracting an object from the distance measurement result, measurement can be performed more efficiently without erroneous recognition.

[0093]

Embodiments of the present invention will be described below with reference to the drawings.

First Embodiment This embodiment is an embodiment in which a desired work field can be obtained.

Hereinafter, description will be made with reference to the related drawings.

FIG. 4 is a block diagram showing the configuration of an object recognition apparatus for multi-view stereo vision according to this embodiment.
Corresponding to the conventional device shown in FIG. FIG. 3 is a configuration diagram of each image sensor including the virtual image sensor of the present embodiment, and corresponds to the configuration diagram of the conventional image sensor shown in FIG. In the present embodiment, it is assumed that the present invention is applied to multi-view stereo, but the present invention may be applied to twin-view stereo.

Reference image sensor 1 constituting multi-view stereo
, N are arranged at predetermined intervals in the horizontal, vertical, or oblique directions. For convenience of explanation, FIG. 3 shows a case where the antennas are arranged on the left and right at a constant interval.

The reference image input unit 301 and the image input units 302, 303,...
1 and each of the images # 2, # 3,..., #N.

In the virtual visual field information setting section 305, as shown in FIG.
Image # by the virtual image sensor 21 when the image is taken
21 is set. That is, a virtual visual field to be a working visual field is set. This virtual image sensor 2
1 defines parameters such as the arrangement position, orientation (posture), focal length, and resolution of the image sensor, as in the actual image sensor. Further, the parameters of the virtual image sensor 21 may be freely changed according to the purpose. In FIG. 1, reference numeral 41 denotes a virtual lens constituting the virtual image sensor 21.

[0100] In the corresponding candidate point coordinate generating unit 306, for each selected pixel of the virtual image # 21, for each distance z _n assuming the position coordinates of the corresponding candidate points of the reference image # 1, the image sensor 2 image # And the position coordinates of the corresponding candidate point of image # 3 of the image sensor 3 and the position coordinates of the corresponding candidate point of image #N of the image sensor N are read out. Thus, the position coordinates of each corresponding candidate point are generated. Here, for the sake of simplicity, an example will be described in which the position coordinates are stored and stored, but the position coordinates may be obtained by calculation.

[0102] Virtual image # the selected pixel P ₂₁ of 21, each of the image sensors k at assumed distance _{z n (k = 1,2, ...} , N:
Hereinafter, the corresponding candidate points of k is used as the same meaning) P _k
FIG. 1 shows the relationship with the position coordinates (X _k , Y _k ).

The position coordinates (x, y,
z) is represented by the global coordinate system xyz, and the position coordinates (X, Y) on the image are expressed in the local coordinate system X-Y set for each image.
Let it be represented by Y. As shown in the figure, the selected pixel P21 specified by (i, j) in the virtual image # 21 of the virtual image sensor ₂₁ is selected, and the point 50a (x, y) on the recognition target object 50 is selected. , the distance z _n to z) is assumed. Then, the actual position coordinates (X _k , Y _k ) of the corresponding candidate point P _k in the image #k of each image sensor k corresponding to the assumed distance z _n are read.

Next, the local information extracting sections 307 to 310 respectively execute processing for extracting local information based on the position coordinates of the corresponding candidate point _Pk generated by the corresponding candidate point coordinate generating section 306 in this way. You.

Then, images # 1, # 2, # 3,..., #
Corresponding candidate points _{N P 1, P 2, P} 3, ···, the image information F ₁ of _{P N, F 2, F 3} , ···, the F _N is obtained respectively, corresponding reference image # 1 similarity relative to the image information F ₁ of the candidate point P ₁ is calculated by the similarity calculation unit 311 to 313.

[0105] That is, the similarity calculation unit 311, by comparing the image information F ₁ corresponding candidate point P ₁ of the image # 2 image sensor 2 of the corresponding candidate point P ₂ image information F ₂ and the reference image # 1 And the similarity of these pieces of image information, and the similarity calculation unit 312 calculates the corresponding candidate point P ₃ of the image # 3 of the image sensor _3.
Of the comparison of the image information F ₃ and the image information F ₁ corresponding candidate point P ₁ of the reference image # 1, the similarity of these image information,
In addition, the similarity calculation unit 313 calculates the image # of the image sensor N
By comparing the image information F ₁ corresponding candidate point P ₁ of the image information of the N corresponding candidate point P _N F _N and the reference image # 1, the similarity of image information are calculated.

That is, in the present embodiment, the reference image # 1
Is used as a reference image for calculating the similarity,
This is common to the conventional similarity calculation units 209 to 211 of FIG. For example, the square of the difference between the image information of the reference image # 1 and the image information of the corresponding image (for example, # 2) is obtained as the similarity. Specifically, the similarity calculating unit 311, and the surrounding region of the corresponding candidate point P ₁ of the reference image # 1, the pattern matching with the surrounding area of the corresponding candidate points of the image # 2 image sensor 2, both The regions of the image are compared with each other to calculate the similarity.

That is, as shown in FIG.
Together with the window WD ₁ is cut around the 1 position coordinates of the corresponding candidate points P _1, the image of the image sensor 2 #
Window WD ₂ around the second position coordinates of the corresponding candidate point P ₂ is cut out, by performing pattern matching for these windows WD _1, WD ₂ together, these similarities are calculated. This pattern matching is performed for each assumed distance z _n .

FIG. 2A is a graph showing the correspondence between the assumed distance z _n and the reciprocal Qs of the similarity obtained for the stereo pair (reference image sensor 1 and image sensor 2). As shown in the figure, the window W of the reference image sensor centered on the position of the corresponding candidate points when assumptions distance z _'n
D result of the matching between the _{2 '1} and the window WD of the image sensor 2' is large values obtained as the inverse Qs similarity (is smaller similarity) is the window WD _1, results assumptions distance was matching with the window WD ₂ around the position coordinates of the corresponding candidate points when the z _nx is the reciprocal Qs similarity is smaller (similarity is larger) I understand. In the similarity calculation unit 312 in the same manner, the center and the window WD ₁ around the position coordinates of the corresponding candidate point P ₁ of the reference image # 1, the position coordinates of the corresponding candidate point P ₃ of the image # 3 image sensors 3 Pattern matching with the window WD ₃
These similarities are calculated. Then, since pattern matching is performed for each assumed distance z _n , the stereo pair (image sensor 1 and image sensor 3) also has the assumed distance z _n and the reciprocal Qs of the similarity as shown in FIG.
Is required.

[0109] In the similarity calculation unit 313 in the same manner, the window WD ₁ around the position coordinates of the corresponding candidate point P ₁ of the reference image # 1, the position of the candidate corresponding points P _N of the image #N image sensor N pattern matching between the window WD _N centered coordinates is performed, these similarities are calculated. Then, by performing pattern matching for each assumed distance z _n , this stereo pair (image sensor 1
Correspondence between the assumed distance z _n and similarity reciprocal Qs shown in FIG. 2 (N) is also the image sensor N) and is obtained.

[0110] Finally, by adding the correspondence relationship between the reciprocal of the similarity assumed distance z _n obtained for each stereo pair, and FIG. 2 (a fusion between the sum of the reciprocal of the similarity assumed distance z _n result)
Is required.

In this way, from the correspondence between the assumed distance z _n and the reciprocal of the similarity, the point having the highest similarity (the point at which the reciprocal of the similarity has the minimum value) is determined. ,
The hypothetical distance z _nx corresponding to the point with the highest similarity is finally estimated as the true distance (the most likely distance) to the point 50 a on the recognition target object 50. This process is performed for all pixels for each selected pixel of the virtual image # 21.

As described above, the distance estimating unit 314 discriminates the one having the highest similarity addition value from the similarity addition values obtained by sequentially changing the assumed distance z _n . The hypothetical distance z _{nx at} which the sum of the similarities becomes the highest is estimated as the true distance and is output.

Since image information of the corresponding point of each of the images # 1 to #N is obtained for each selected pixel of the virtual image # 21, the image information of the corresponding point is added to each pixel of the virtual image # 21. By doing so, it is possible to generate a captured image when the virtual image sensor 21 captures an image of the object 50. In this case, as the image information of the corresponding points assigned to each pixel of the virtual image # 21, an average value of the brightness of the corresponding points of the images # 1 to #N, a median value, or the like can be used. In this manner, an image as if photographed by the virtual image sensor 21 can be generated.

As shown in FIG. 2 (combined result), the estimated distance z _{nx at} which the reciprocal Qs of the similarity becomes minimum is obtained for each selected pixel of the virtual image # 21. At this time, the minimum value of the reciprocal of the similarity is set as data indicating the reliability of the estimated distance z _nx (the smaller the minimum value, the higher the reliability), and is associated with each estimated distance z _nx. It is also possible to carry out such operations. In addition, as the reliability, a value obtained based on a change rate of the similarity or a change rate of the original image may be used.

As described above, according to the present embodiment, it is possible to arbitrarily set the virtual image sensor 21 having a desired observation field of view and obtain a distance image and a captured image by the virtual image sensor 21. Therefore, processing such as recognition and determination of the object 50 can be performed with a work field according to a work situation without being limited to a work field uniquely determined by the arrangement positions and postures of the image sensors 1 to N. Become like In particular, even in a place where it is difficult to physically place the image sensor or where the environment in which the use of the image sensor is difficult is not good, the virtual image sensor 21 is provided in that place.
By setting, it is possible to obtain an image equivalent to the case where an actual image sensor is arranged at that location.

Further, since the virtual image # 21 obtained by the virtual image sensor 21 can be arbitrarily changed in position and posture only by arithmetic processing without using a mechanical swing mechanism (see ""). Electronic swing "), while keeping the image sensors 1 to N fixed, an image whose viewpoint is freely changed can be obtained. By utilizing this, it is possible to construct a system for tracking an object or a system for automatically changing the field of view according to the work purpose without the need for a mechanical swing mechanism.

Next, a modification of the above embodiment will be described.

FIG. 5 is a block diagram of the image sensor corresponding to FIG. 3 described above, and FIG. 6 is a block diagram corresponding to FIG. 4 described above.

The difference from the embodiment of FIGS. 3 and 4 is that, when calculating the similarity, each stereo image is obtained based on the image information F ₁ of the corresponding candidate point P ₁ of the reference image # 1 of the reference image sensor 1. Instead of finding the similarity for each pair, the image sensor 1
The point is that the similarity of the entire multi-view stereo is calculated from the degree of variation of the local information F ₁ to F _N of the corresponding candidate points P ₁ to P _N of all the image sensors ₁ to _N.

In FIG. 6, a virtual visual field information setting unit 40
4. Corresponding candidate point coordinate generator 405, local information extractor 40
, 408 correspond to the virtual visual field information setting unit 305, the corresponding candidate point coordinate generating unit 306, and the local information extracting units 307, 308,.

In the similarity calculating section 409, the images # 1 and # 2 extracted by the local information extracting sections 406, 407,.
# 2, ..., candidate corresponding points P _1, P ₂ of # N, ..., local information F _1, F ₂ of the P _N, ..., based on F _N, these local information F _1, F _2, ..., the evaluation value indicating the "unity degree" of F _N is calculated as the similarity. The evaluation value indicating the “unity degree” is, for example, a sum of absolute values of differences from the average value of the image information, a variance value of the image information, and the like. For example, when the variance value of the image information is small, the evaluation of “the unity” is high, and the similarity is high.

As described above, the similarity is calculated for each stereo pair by taking a difference based on the image information of the reference image # 1 as shown in FIG. 4, and the similarity for each stereo pair is added. Instead of calculating the overall similarity, the overall similarity is obtained by a collective calculation based on all the image information of all the images # 1 to #N. Can be requested. All images # 1
Similarity of ~ # N is calculated for each assumed distance z _n.

In the distance estimating section 410, the one having the highest similarity is determined from the similarities obtained by sequentially changing the assumed distance z _n, and the assumed distance z _nx having the highest similarity is true. Is estimated and output. Since image information of the corresponding points of the images # 1 to #N is obtained for each selected pixel of the virtual image # 21, the image information of the corresponding points is added to each pixel of the virtual image # 21. In addition, a captured image when the object 50 is captured by the virtual image sensor 21 can be generated.

Second Embodiment In this embodiment, a plurality of types of image sensors having different arrangement positions, directions, optical characteristics, and resolutions can be used at the same time. This is an embodiment in which distance measurement and object recognition with high reliability can be performed using information.

FIG. 7B shows an image sensor group 11 ″ assumed in the present embodiment. The image sensor groups 141 to 144 for bright field and the image sensor groups 151 to 144 for dark field are shown.
54. An illuminometer 16 is disposed at the center of the image sensor group 11 ″. The bright-field image sensors 141 to 144 have higher sensitivity than the dark-field image sensors 151 to 154 in an environment with high illuminance. Show high properties,
The dark-field image sensors 151 to 154 exhibit higher sensitivity than the bright-field image sensors 141 to 144 in an environment with low illuminance.

FIG. 9 is a block diagram of an apparatus for performing distance measurement and object recognition based on a captured image of the image sensor group 11 ″, and is a block diagram corresponding to FIG. 6, and is similar to FIG. Does not set a reference image sensor as a reference when calculating the similarity.

The difference from the embodiment of FIG. 6 is that the similarity is
For each type of image sensor, that is, bright field image sensor group 1
41 to 144, and is calculated for each dark-field image sensor group 151 to 154.

The image input units 501 to 502 receive the images # 141 to # 144 of the bright-field image sensors 141 to 144, respectively. Similarly, the image input units 503 to 504 include images # 151 of the dark-field image sensors 151 to 154, respectively.
To # 154 are captured.

In FIG. 9, the virtual visual field information setting unit 50
5. The corresponding candidate point coordinate generating unit 506 corresponds to the virtual visual field information setting unit 305 and the corresponding candidate point coordinate generating unit 306 in FIG. 4, or the virtual visual field information setting unit 404 and the corresponding candidate point coordinate generating unit 405 in FIG. Things.

In the virtual visual field information setting unit 505, as shown in FIG. 1, a virtual image # 21 common to the bright field image sensors 141 to 144 and the dark field image sensors 151 to 154 is set. The selected pixel P21 specified at the position (i, j) in ₂₁ is selected.

[0131] In the corresponding candidate point coordinate generating unit 506, the set virtual image # bright field image sensors 141-144 corresponding to the selected pixel P ₂₁ in the 21 image # 141 to #
Corresponding candidate points in 144 and dark field image sensor group 15
Corresponding candidate point P in images # 151 to # 154 of Nos. 1 to 154
_k position coordinates (X _k , Y _k ) (k = 141, 142, 14)
3, 144, 151, 152, 153, 154: k below
Are used in the same sense) for each magnitude of the assumed distance z _n .

The local information extracting units 507 to 508 correspond to the local information extracting units 307, 308, 310 of FIG. 4 or 406, 407,..., 408 of FIG.
Images # 141 to # of bright field image sensors 141 to 144
144, the local information F ₁₄₁の of the corresponding candidate points P ₁₄₁ 144P ₁₄₄
F ₁₄₄ is extracted. Similarly, in the local information extraction units 509 to 510, the local information F _{151 to} F ₁₅₄ of the corresponding candidate points P _{151 to} P ₁₅₄ on the images # 151 to # 154 of the dark field image sensors ₁₅₁ to ₁₅₄ are extracted.

[0133] In the similarity calculation unit 511, image # 141 of the image sensor brightfield extracted by the local information extracting unit 507 to 508, # 142, # 143, candidate corresponding points P _{14 1} of # 144, P _{142, P143} , _P144 image information _F141 , _F142 ,
Based on F ₁₄₃ and F ₁₄₄ , these image information F ₁₄₁ , F ₁₄₂ ,
The variance values of F ₁₄₃ and F ₁₄₄ (evaluation values indicating “the degree of unity” of the image information) are calculated as the similarity. This similarity is obtained for each assumed distance z _n .

As a result, as shown in FIG. 10A, the reciprocal Qs _{1 of the} similarity corresponding to each assumed distance z _n is output from the similarity calculator 511 to the distance estimator 515. . On the other hand, the similarity calculating unit 512, the image # 151 darkfield image sensor extracted in the local information extracting unit 509 to 510, # 152, # 153, candidate corresponding points P _151, P ₁₅₂ of # 154, P ₁₅₃ , _P154 image information _F151 ,
Based on F ₁₅₂ , F ₁₅₃ , and F ₁₅₄ , these image information F ₁₅₁ ,
The variance of F ₁₅₂ , F ₁₅₃ , and F ₁₅₄ (evaluation value indicating “the degree of unity” of the image information) is calculated as the similarity.

The similarity is obtained for each assumed distance z _n . As a result, the similarity calculation unit 512 outputs FIG.
As shown in (b), the reciprocal Qs _{2 of the} similarity corresponding to each of the assumed distances z _n is output to the distance estimating unit 515.

The external information input unit 513 has the illuminance meter 1
A signal indicating the illuminance, which is the detection output of No. 6, is input, and is output to the weight coefficient generation unit 514 of the sensor group.

The weighting factor generator 514 outputs the reciprocal Qs _{1 of the} similarity calculated by the similarity calculator 511 and the similarity calculator 512 based on the magnitude of the illuminance detected by the illuminometer 16. Weight coefficients ω1 and ω2 to be multiplied by Qs ₂ are calculated and output to distance estimation section 515.

The weight coefficients ω1 and ω2 are calculated such that the larger the illuminance detected by the illuminometer 16 is, the larger the weight coefficient ω1 is and the smaller the weight coefficient ω2 is. That is, the contribution ratio of the similarity calculated based on the images of the bright-field image sensors 141 to 144 is large, and the contribution ratio of the similarity calculated based on the images of the dark-field image sensors 151 to 154 is small. Is calculated. Meanwhile, the illuminometer 16
The calculation is performed such that the smaller the illuminance detected in the above, the smaller the weight coefficient ω1 and the larger the weight coefficient ω2.
That is, the contribution ratio of the similarity calculated based on the images of the bright-field image sensors 141 to 144 is small, and the contribution ratio of the similarity calculated based on the images of the dark-field image sensors 151 to 154 is large. Is calculated.

[0139] In the distance estimation unit 515, as shown in FIG. 10 (a), the weighting coefficient ω1 is multiplied by the reciprocal Qs ₁ of calculated similarity based on the image of the image sensor 141 to 144 for bright field Rutotomoni, 10 (b), the above-mentioned weighting coefficient ω2 against the reciprocal Qs ₂ of calculated similarity based on the image of the darkfield image sensor 151 to 154
Is multiplied. That is, the assumed distance z _n and the reciprocal Qs of the similarity
The correspondence relationship with ₁ and the correspondence relationship between the assumed distance z _n and the reciprocal Qs _{2 of the} degree of similarity are corrected by the weight coefficients ω 1 and ω 2, respectively. Then, these weight coefficients .omega.1, correspondence between the reciprocal Qs ₁ similarity assumed distance z _n corrected by the .omega.2, correspondence between the assumed distance z _n the reciprocal Qs ₂ of similarity is added, FIG. 10 ( As shown in c), a correspondence between the assumed distance z _n and the sum Qs ₁₂ of the reciprocal of the similarity is generated.

In this way, from the correspondence between the assumed distance z _n and the reciprocal of the similarity Qs ₁₂ , the point having the highest similarity (the reciprocal of the similarity Qs ₁₂ becomes the minimum value) Is determined, and the assumed distance z _nx corresponding to the point with the highest similarity is finally estimated as the true distance (the most probable distance) to the point 50a on the recognition target object 50. You. This process is performed for all pixels for each selected pixel of the virtual image # 21.

[0141] As described above, the distance estimating unit 515, the determination is that among the assumed distance z sum of the similarity obtained by the sequentially changing _n Qs _12, the sum of the most similarity is high Then, the assumed distance z _{nx at} which the added value of the similarity becomes the highest is estimated as the true distance and is output.

According to the present embodiment, the following effects can be further obtained.

That is, in the multi-view stereo measurement, the virtual image sensor 21 is set and used as a virtual visual field common to the bright-field image sensor group and the dark-field image sensor group. Even if the image sensor groups differ from each other, it is possible to perform comprehensive judgment by fusing the measurement results of these two types of image sensor groups. The measurement results of each of the two image sensor groups having different imaging conditions can be comprehensively determined, and the measurement accuracy can be dramatically improved.

In this embodiment, two types of different image sensor groups are assumed, but it is also possible to apply the present invention to three or more types of different image sensor groups.

In this embodiment, the weight of the similarity by the bright-field sensor groups 141 to 144 and the weight of the similarity by the dark-field sensor groups 151 to 154 are changed in accordance with the detection output of the illuminometer 16. The distance is estimated by fusing similarities obtained from these two types of image sensors.
It is also possible to completely switch the type of image sensor to be used according to the detection output of the illuminometer 16.

For example, when a threshold value for determining the magnitude of the illuminance detected by the illuminometer 16 in a binary manner is set, and the illuminance detected by the illuminometer 16 exceeds this threshold value, , The weight coefficient ω1 is set to 1, the weight coefficient ω2 is set to 0, and the distance is estimated only by the similarity by the bright-field sensor groups 141 to 144. On the contrary, the distance is detected by the illuminometer 16. When the illuminance is smaller than the threshold value, the weighting factor ω1 is set to 0, the weighting factor ω2 is set to 1, and only the similarity by the dark-field image sensors 151 to 154 is used.
The distance may be estimated.

The type of image sensor to be used is arbitrary.

As shown in FIG. 7A, an image sensor group 11 'comprising a group of monochrome image sensors 121 to 124 of normal sensitivity and a group of infrared image sensors 131 to 134.
The present embodiment described above may be applied to the present invention.

The parameters such as the focal length, aperture, and shutter speed of the image sensor may be different for each type of image sensor, and the orientation of the image sensor may be adjusted according to conditions.

In the present embodiment, the illuminometer 16 is used, but the imaging conditions of the image sensor (surrounding environmental conditions)
Any sensor can be used as long as the sensor can detect the change in.

An embodiment in which the weight coefficient is changed depending on the time is also conceivable.

Third Embodiment In the present embodiment, when a plurality of image sensors constituting a multi-view stereo are arranged, not all of the plurality of image sensors are used for measurement. In this embodiment, by selecting at least two image sensors to be used for measurement from a plurality of image sensors, it is possible to perform fine-grained measurement according to the situation. FIG. 8 shows an image sensor group 11 composed of image sensors 1, 2, 3,... Mounted on a vehicle 60. It is assumed that the vehicle travels while recognizing and judging the working field of view Ar.

The vehicle 60 is equipped with a speed sensor 62 for detecting the speed V of the vehicle and a posture angle sensor 63 such as a gyro for detecting the traveling direction φ of the vehicle 60. Note that a sensor that detects the steering angle of the vehicle 60 may be used instead of the attitude angle sensor 63. Here, if the speed V and the traveling direction φ of the vehicle 60 are determined, the vehicle forward distance to be recognized by the vehicle 60 and the direction to be recognized are determined accordingly, so that the work visual field Ar is specified. The at least two image sensors most suitable for imaging the working field of view Ar are selected from the image sensor group 11.

Therefore, the speed sensor 62 and the posture angle sensor 6
At least two image sensors to be actually used, for example, the image sensors 2 and 3 are selected from the image sensor group 11 in accordance with the vehicle speed V and the traveling direction φ detected at 3.

As a result, as in FIG.
1, the image # 2 of the image sensor 2 is captured, and the image input unit 402 captures the image # 3 of the image sensor 3. Hereinafter, similarly to FIG. 6, the corresponding candidate point coordinate generating unit 405 selects the position coordinates (X _k , X _k) of the corresponding candidate point P _k of the selected pixel of the virtual image # 21 from the images of the two image sensors 2 and 3. Y _k ) (k = 1, 2: k is used in the same meaning) is generated for each magnitude of the assumed distance z _n .

The local information extraction units 406 and 407 extract the image information F _k of the corresponding candidate points P _k of the image sensors 2 and 3, and the similarity calculation unit 409 extracts the correspondence of the generated image sensors 2 and 3. Position coordinates (X _k , Y _k ) of candidate point P _k
Image information F _k similarity between are calculated for.

In the distance estimating section 410, the assumed distance z _nx at which the calculated similarity becomes _maximum is calculated as follows:
Is the distance to the point on the object in the work field Ar corresponding virtual image sensor 21 in the selected pixel P _21, the distance z _nx
Is obtained for each selected pixel. As a result, the distance image and the captured image indicating the working field of view Ar viewed from the virtual image sensor 21 indicate that the traveling state of the vehicle 60 changes,
It will always be generated as accurate.

As described above, according to the present embodiment, in the measurement by the multi-view stereo, the measurement is not performed by fixing the image sensor to be used, but the work field A that changes every moment.
According to r, at least two image sensors most suitable for imaging this are sequentially selected, so that
Even if the running situation of the vehicle 60 (the change in the speed V and the traveling direction φ of the vehicle 60) changes, fine measurement corresponding to the change can be performed.

In this embodiment, it is assumed that the image sensor group is mounted on the vehicle. However, the image sensor group is not limited to this, and may be mounted on, for example, a mobile robot. . In addition, by providing a posture angle adjustment mechanism that can adjust the posture angle of the image sensor group 11 in the directions indicated by arrows A and B, the speed sensor 62, An embodiment in which the attitude angle of the vehicle sensor group 11 is changed according to the detection output of the attitude angle sensor 63 is also possible.

Further, the degree of zoom of the image sensor may be adjusted according to the distance to the working visual field Ar.

Fourth Embodiment This embodiment introduces the concept of a model representing a recognition target object, instead of searching for all the virtual distances for each selected pixel. In this embodiment, the efficiency of model estimation can be improved by performing only the matching process.

First, the concept and definition of the model will be described.

The model referred to in this specification is a recognition target having a three-dimensional structure defined in a space.
The model has a three-dimensional structure defined by model information such as coordinate positions and shape data in an absolute space (global coordinate system xyz). The position of the model viewed in the observation field of view of each of the image sensors 1 to N or the virtual field of view of the virtual image sensor 21 can be calculated based on this model information. As a model, not only a simple plane and a curved surface but also a model having a complicated three-dimensional shape can be used. Since the model is ultimately useful for recognizing the three-dimensional structure of the object, the method of expressing the model information and the contents of the model information can be adapted to the purpose of recognizing the three-dimensional structure of the object. .

There is also a method of parameterizing model information according to the purpose in order to efficiently represent related models.

For example, a series of plane models arranged in parallel at a certain interval is represented by a parameter called a parallel position. Then, by changing this parameter, it is estimated whether the recognition target surface is any one of a series of plane model groups.

Hereinafter, this embodiment will be described with reference to FIGS.
8 will be described.

In this embodiment, as shown in FIG. 13, an image sensor group 11 is mounted on a vehicle 60, and the inclination θ of a road 61 ahead in the vehicle traveling direction is determined based on the image pickup result of the image sensor group 11. It is assumed that the vehicle travels while recognizing and judging.

FIG. 12 is a block diagram showing the configuration of an object recognition apparatus using multi-view stereoscopic vision according to this embodiment.
Corresponds to the block diagram of FIG.

In this embodiment, it is assumed that the present invention is applied to a multi-view stereo, but the present invention may be applied to a twin-view stereo.

In this embodiment, as in FIG. 6, it is assumed that the virtual image # 21 is set by the virtual image sensor 21, but in this embodiment, the virtual image # 21 is set. Without this, a configuration equivalent to that of FIG. 27 can be realized.

That is, the present embodiment is not only applied to a configuration for searching for the corresponding point of each of the images # 1 to #N corresponding to the selected pixel of the virtual image # 21,
The present embodiment may be applied to a configuration in which a corresponding point of each of the images # 2 to #N corresponding to one selected pixel is searched.

The image sensors 1, 2, 3,..., N constituting the image sensor group 11 which is a multi-view stereo are arranged at predetermined intervals in a horizontal, vertical or oblique direction. The image input unit 601 shown in FIG.
02, the image # 2 of the image sensor 2 is captured, and the image input unit 603 captures the image #N of the image sensor N, respectively.

As shown in FIG. 11, the virtual visual field information setting unit 604 uses the virtual image sensor 21 to capture a virtual image of the object to be recognized (in this case, the road 61) at a desired position with a desired visual field. Information about the field of view is set. That is, a virtual visual field to be a working visual field is set. In this virtual image sensor 21, the actual image sensor 1
As in the case of N, parameters such as the arrangement position, orientation (posture), focal length, and resolution of the image sensor are defined.

Further, the parameters of the virtual image sensor 21 may be freely and dynamically changed according to the purpose.

The lens 41 shown in FIG. 11 is a virtual lens constituting the virtual image sensor 21.

In the assumption model information setting section 605, model information relating to the shape of the model of the object to be recognized in the three-dimensional space is set. In the present embodiment, as shown in FIG. 13, since it is necessary to recognize and determine the inclination θ of the road 61, the models M ₁ , M ₂ , M
_3, the M _4, the value theta ₁ parameter theta, theta _2, theta _3, is set as the model information for each theta _4.

[0177] As a specific setting method of the model information, many methods can be considered.

For example, in the case where the assumed model is a road composed of a simple plane as described above, a method of specifying the plane includes the spatial coordinates of a point passing through the plane and the normal vector of the plane. May be given, or spatial coordinates of three points somewhere through the plane may be given. When the assumed model is complex with undulations, interpolation may be performed by giving a large number of coordinates of a plurality of points sufficient to sufficiently reproduce the undulations, or set by combining arbitrary geometric models. May be.

In the correspondence candidate point coordinate generation unit 606, FIG.
As shown in, the virtual virtual image # position in 21 of the image sensor 21 (i, j) each image sensor corresponding to the selected pixel P ₂₁ specified by k when capturing the road 61 in the virtual image sensor 21 Is the position coordinates (X _k , Y _k ) of the corresponding candidate point P _k in the image #k of the hypothetical model M _i (i = 1 to 1) selected from the models M ₁ , M ₂ , M ₃ , and M ₄ 4) Generated every time. The position coordinates (X _k , Y _k ) of the corresponding candidate point P _k may be obtained corresponding to the point (x, y, z) on each model as shown in FIG. It is not necessary to prepare a conversion table and actually calculate.

Next, in the local information extracting units 607 to 609, processes for extracting local information based on the position coordinates of the corresponding candidate point _Pk generated by the corresponding candidate point coordinate generating unit 606 are respectively executed. You.

[0181] That is, the local information extracting unit 607, from the position coordinates of the corresponding candidate point P ₁ of the image # 1 of the image sensor 1, which is generated by the corresponding candidate point coordinate generating unit 606, in accordance with the image information of the surrounding pixels by interpolation, extracts image information F ₁ corresponding candidate point P _1. Similarly, the local information extraction unit 60
In 8, based on the position coordinates of the corresponding candidate point P ₂ of the image # 2 image sensor 2 generated by the corresponding candidate point coordinate generating unit 606, image information F ₂ corresponding candidate point P ₂ is, the local information extracting unit in 609, based on the position coordinates of the corresponding candidate point P _N of the image #N image sensors N which is generated in the corresponding candidate point coordinate generating unit 606, image information F _N corresponding candidate point P _N are obtained, respectively. Here, the image information refers to lightness or a second-order differential value of the lightness.

[0182] Thus, image # 1, # 2, ..., the corresponding candidate points P _1, P ₂ of # N, ..., image information F _1, F ₂ of the P _N, ...,
When F _N is obtained, the similarity calculation unit 610 calculates
The variance value (evaluation value indicating “the degree of unity” of the image information) of the image information F ₁ to F _N of the corresponding candidate points P ₁ to P _N of all the image sensors ₁ to _N is calculated as the similarity. The higher the similarity indicates a larger value (the smaller the variance of the image information is), the closer the model is to the assumption model M. The similarity is obtained for each of the hypothetical models M ₁ , M ₂ , M ₃ , and M ₄ .

In this manner, as shown in FIG. 14, the correspondence between the hypothetical model M and the reciprocal Qs of the similarity is obtained.

Next, the model estimating section 611 determines a model having the highest similarity from the correspondence between the hypothetical model M and the reciprocal Qs of the similarity. This is performed for the correspondence relationship for each selected pixel, and a model is estimated for each pixel.

By performing model estimation on a partial area or the entire area of the screen, the most frequent model can be finally determined as a model representing the recognition target object. Alternatively, a model may be estimated from the correspondence between the hypothetical model M and the reciprocal Qs of the similarity with a precision finer than the search interval, and this model may be determined as a model close to the recognition target object.

Now, assuming that FIG. 14 shows the correspondence between the assumption model M and the added value of the reciprocal Qs of the similarity, based on this correspondence, the similarity (the reciprocal Q of the similarity Q
The hypothetical model when s) is the largest is the model representing the recognition target object. In this case, the polygonal line relationship of Figure 14, by applying curve fitting, can be accurately define the M _2.6 to reciprocal Qs similarity takes a minimum value. Now, the model M _2, the inclination angle theta ₂ = 0 degrees associated, in the model M _3, the inclination angle theta ₃ = 5 ° is assumed to be correlated, the M _2.6 is the inclination angle theta = It can be estimated to be three degrees. Therefore, it can be recognized and determined that the inclination angle θ of the road 61 in front of the vehicle 60 is 3 degrees.

The model group selecting section 612 selects a presumed model from the model groups based on the information on the degree of existence of each model of the model group output from the auxiliary information input section 613 and the model estimation result of the model estimating section 611. A process for selecting a model to be executed is executed. That is, in the vehicle 60,
A pitch angle sensor for detecting the pitch angle of the vehicle 60 is mounted, and the current pitch angle of the vehicle 60 is detected by the pitch angle sensor. Therefore, it is possible to predict the degree of the inclination of the road surface ahead of the vehicle 60 based on the detection output of the pitch angle sensor.

Auxiliary information input section 613 outputs the detection output of the pitch angle sensor to model group selection section 612.

The model group selecting section 612 binarizes the detected value of the pitch angle sensor with a predetermined threshold value, and if the detected value of the pitch angle sensor is equal to or larger than this threshold value,
From the models M ₁ , M ₂ , M ₃ , and M ₄ , the models M ₁ and M ₂ with gentle slopes are selected as models to be assumed.

On the other hand, when the detected value of the pitch angle sensor is smaller than the threshold value, the models M ₁ , M ₂ , M ₃ ,
From M ₄ , the steep pitches M ₃ and M ₄ are selected as models to be assumed.

Only the specific pitch selected by model group selecting section 612 in this way is assumed model information setting section 6
05, and is assumed as a model to be actually used by the corresponding candidate point coordinate generation unit 606.

When the model group indicating the road 61 is composed of a model group having a positive inclination angle θ (uphill gradient) and a model group having a negative inclination angle θ (downhill gradient), The selection unit 612 can select and output one of a model group having a positive inclination angle θ (uphill gradient) and a model group having a negative inclination angle θ (downhill gradient).

As described above, in the present embodiment, since the model group selecting section 612 is provided, not all of the prepared model groups are subjected to the arithmetic processing but the prepared model groups are selected. Since it is only necessary to narrow down to a specific model group and perform the arithmetic processing only on this specific model group, it is possible to recognize and judge the object extremely efficiently.

As described above, according to the present embodiment shown in FIG. 12, when the shape (plane) of the road 61 is specified in advance, it is modeled to determine the recognition target. Since it is not necessary to perform distance measurement for the entire space for each selected pixel or to identify and extract an object from the distance measurement result as in the past, it is not necessary to perform complex processing more efficiently and without erroneous recognition. Measurement can be performed.

Next, various modifications applied to the present embodiment will be described.

In FIG. 15, it is assumed that not only the inclination θ of the road 61 ahead of the vehicle 60 is recognized, but also an obstacle 62 ahead of the vehicle 60 is recognized.

In this case, the model estimating unit 611 in FIG. 12 shows a case where the model M ₂ is estimated to indicate the road 61 from the models M ₁ , M ₂ , M ₃ , and M ₄ . Therefore, the following calculation is performed in the model estimation unit 611.

That is, when the model M ₂ is assumed,
For each selected pixel P ₂₁ of the virtual image # 21, the inverse of similarity Q
Since s is calculated by the similarity calculation unit 610, FIG.
Each selected pixel P ₂₁ of the virtual image # 21, as shown, the value of the corresponding similarity reciprocal Qs is associated.

Therefore, based on the distribution of the reciprocal of the degree of similarity, it is determined whether or not the obstacle 62 exists on the road 61 and the location of the obstacle 62.

As shown in the figure, the distribution of the reciprocal Qs of the similarity on the virtual image # 21 is represented by a region 26 having a low numerical value (in this region, the reciprocal Qs of the similarity indicates “2”). ,
A region 25 with a high numerical value (in this region, the reciprocal Qs of the similarity is Qs)
Indicate “3” and “4”). Identification of these regions can be performed by setting a threshold value for determining the magnitude of the reciprocal Qs of the similarity in a binary manner and determining whether or not the value is equal to or smaller than the threshold value.

Therefore, it is determined that the area 26 where the numerical value of the reciprocal Qs of the similarity is low indicates the road 61, and the area 25 where the numerical value of the reciprocal Qs of the similarity is high is an obstacle existing on the road 61. 62.

Here, for each selected pixel of the virtual image # 21, the image information of the corresponding point of each of the images # 1 to #N is obtained, so that each pixel of the virtual image # 21 is added to the image information of the corresponding point. By giving the corresponding brightness, an image when the recognition target object is captured by the virtual image sensor 21 can be generated as shown in FIG. In this case, the above-mentioned road 6
It is possible to determine that the specific area 64 in the image indicates the obstacle 62 based on the result of identifying the obstacle 1 and the obstacle 62.

In order to determine whether or not an obstacle is present on the traveling path, it is not always necessary to prepare a plurality of models indicating the traveling path.

[0204] Figure 17, it is known the road 61 is flat, as a model to prepare only flat (inclination angle theta = 0 degrees) model M _2, there is an obstacle 62 on a road 61 It is assumed that it is recognized and determined whether or not to perform.

In this case, the following calculation is performed in the model estimating unit 611 of FIG.

That is, when the model M ₂ is assumed,
For each selected pixel P ₂₁ of the virtual image # 21, the inverse of similarity Q
Since s is calculated by the similarity calculation unit 610, FIG.
Each selected pixel P ₂₁ of the virtual image # 21, as shown, the value of the corresponding similarity reciprocal Qs is associated.

Therefore, based on the distribution of the reciprocal of the degree of similarity, it is determined whether or not the obstacle 62 exists on the road 61 and the location of the obstacle 62.

As shown in the figure, the distribution of the reciprocal Qs of the similarity on the virtual image # 21 is such that the magnitude of the reciprocal Qs of the similarity is 2
Area 2 where the numerical value is low according to the threshold value that is determined by value
6 ′ and a region 25 ′ having a high numerical value.

Therefore, it is determined that the area 26 'where the numerical value of the reciprocal Qs of the similarity is low indicates the road 61 (model M ₂ ), and the area 25' where the numerical value of the reciprocal Qs of the similarity is high is the road 25 '. It is determined that the obstacle 62 exists on the object 61.

In the embodiment described above, it is assumed that the model has a single shape (plane). However, the present invention is also applicable to the case of recognizing a plurality of objects having various shapes. Can be applied.

For example, in recognizing and discriminating a building installed on the ground, a model group such as a model indicating the ground, a model indicating the wall surface of the building, a model indicating the roof of the building, etc. The same process may be performed on each hypothetical model of these model groups, generated by the setting unit 605.

Here, in order to avoid complexity of processing, a hierarchical structure is provided between the models in the model group, and the processing can be executed efficiently.

That is, a model group of the whole building and the ground is
First, it is classified into a model group including a plurality of ground models, a model group including a plurality of wall models, and a model group including a plurality of roof models. Further, the ground model group is classified into a flat ground model group and an inclined ground model group. Then, parameters such as the height of the ground and the inclination angle with respect to the horizontal plane are introduced, and the parameters are given to each model of the ground as model information. Similarly, for a model of a wall surface of a building, a parameter such as an inclination angle with respect to a vertical plane is introduced, and this parameter is added as model information.

Then, the model estimating unit 611 in FIG. 12 can divide the virtual image # 21 into several regions and estimate a model for each region.

Now, a certain building is captured in the virtual image # 21. A vertical wall surface is present in front of the center of the virtual image # 21, and a sloping roof is present thereon. Assume that there is a wall inclined to the depth direction to the left of.

Therefore, by executing a model estimation process for each of the central region, the central upper region, and the region to the left of the center of the virtual image # 21, objects which may be present in each region can be reliably and efficiently detected. It becomes possible to recognize and judge.

Further, it is possible to recognize not only one building but also a plurality of buildings. In this case, a model group representing each of a plurality of buildings (for example, a single-family house and a building) is at the top, and a model group having a hierarchical structure in which models such as walls and roofs constituting the building exist at the bottom. Here, as a higher-order structure, a model of each building may be individually prepared, or an entire configuration (building group) in which a plurality of buildings exist may be used as a model.

When the model is estimated for each region of the virtual image # 21 described above, the model estimation process is performed more efficiently by giving the existence probability of each model as information in advance for each region. You may. For each model, information indicating in which region and at what probability the model is present is given as model information.

Therefore, based on the model information given to this model, it is possible to proceed with the processing for an area having a very low existence probability without using this model as a hypothetical model.

For example, since the probability that a model representing a roof exists in a region below virtual image # 21 is extremely low, the model representing a roof can be excluded from this hypothetical model in this region below. . Therefore, the processing efficiency can be further improved.

For example, specific local information, special parameters calculated from the local information (information on edge strength, direction, position, etc., texture information, etc.), information relating to the virtual visual field, and information relating to the assumed model Based on the information or information based on the detection results of other sensors, in a specific area of the virtual image # 21, the type of the model to be assumed is changed, the type of the model to be assumed is restricted, A special operation such as selecting an image to be selected from the images # 1 to #N can also be added.

In this embodiment, as described above,
Although it is not always necessary to set the virtual image # 21,
When estimating the model using virtual image # 21,
The model estimation may be performed based on the result of observing the model from all directions using the visual field flexibility of the virtual image # 21.

That is, when the three-dimensional structure of a model (for example, a spherical model) is defined not only for the normally observed front side but also for the back side, the virtual image # 21 is displayed not only on the front side of the model but also on the back side. By setting the field of view to be observed from the back side of the model, the flexibility in estimating the model can be increased. However, this requires that the back side of the model can be imaged by any of the image sensors 1 to N.

In the present embodiment, an image (virtual image)
In the description, it is assumed that any one of a predetermined model group or one predetermined model exists, but the image (virtual) It is also possible to determine whether or not the model exists in the image).

For example, as shown in FIGS.
The distribution of the reciprocal Qs of the similarity related to the estimated model is obtained, and if the reciprocal Qs of the similarity is higher than a predetermined threshold, it may be determined that the estimated model does not exist in the image. .

Fifth Embodiment In the present embodiment, by utilizing the concept of recognizing a real object described in the fourth embodiment, the actual background matches the background model on the image. For example, a pseudo image is displayed for an existing area, and a real object is displayed for a non-matching area. For example, the image display content is changed to a content different from the actual one.

That is, the present invention relates to the technical field of video broadcasting such as chroma key, and is suitable for performing the following image processing. As a model, for example, a curtain with a pattern is prepared as a background model. Then, it is assumed that a curtain with a pattern is normally captured as a background by the image sensor. In this state, if an object other than the background, for example, an object other than the background, such as a person, appears in front of the background, inconsistency with the background model occurs in an area occupied by the object other than the background in the image. Therefore, in the area where the mismatch occurs, an image of an object other than the background such as an actual person captured by the image sensor is displayed,
On the other hand, for an area that matches the background model, a pseudo background image prepared in advance, for example, another image of the Tokyo Tower is displayed.

According to the present embodiment, the processing can be performed very simply and quickly without the need to separate the background from the object in accordance with the complex contour shape of the real object. Become.

The background model is obtained by extracting only the shape of the background feature and extracting it. Even if the background pattern and brightness change, the processing is always performed with high accuracy without being affected by the change. Is possible.

Hereinafter, the image processing of this embodiment will be described with reference to the block diagram shown in FIG. 19 corresponding to FIG.

First, one reference image sensor, for example, the image sensor 1 is selected from the image sensors 1 to N. The background separation process is performed based on the reference image # 1 of the reference image sensor 1. The reference image sensor 1 captures an image of a background (for example, a curtain with a pattern) and an object (for example, a person) appearing in front of the background. Other image sensor 2
Also in ~ N, the background and an object appearing in front of the background are imaged.

The reference image input unit 701 receives a reference image # 1 captured by the reference image sensor 1. The other image input units 702 and 703 capture images # 2 to #N captured by the other image sensors 2 to N, respectively.

In the background model information setting section 704, FIG.
In the same manner as the assumed model information setting unit 605 of FIG.
Model information indicating a dimensional structure is set.

In the corresponding candidate point coordinate generator 705, the selected pixel P in the background image # 1 of the reference image sensor 1 is set in the same manner as the corresponding candidate point coordinate generator 606 in FIG. Other image sensor k corresponding to ₁
Coordinates (X _k , Y _k ) of the corresponding candidate point P _k in the image #k
(K = 2,..., N) are generated from the model information of the assumed model M.

In the local information extraction unit 706, the reference image # 1
Image information F ₁ of the selected pixel P ₁ of is extracted as the local information. In the local information extracting units 707 and 708, processes of extracting local information F _k (k = 2,..., N) based on the position coordinates of the corresponding candidate point P _k generated by the corresponding candidate point coordinate generating unit 705 are executed. Is done.

In the similarity calculating section 709, as in the similarity calculating section 610 shown in FIG. 12, the selected pixel P ₁ of all the image sensors 1 to _N , the image information F _{1 of} the corresponding candidate points P ₂ to P _N , F
The variance values of _{2 to} F _N (evaluation values indicating “the degree of unity” of the image information) are calculated as the similarity. The higher the degree of similarity is (the smaller the variance of the image information is), the higher the possibility that the area in the image matches the hypothetical model M, that is, the higher the possibility of representing the background. become.

In the background model determination section 710, the reference image #
For each selected pixel, it is determined for each region whether the selected pixel and the background have a background. Specifically, a selected pixel or area whose similarity calculated by the similarity calculation unit 709 is equal to or greater than a predetermined threshold is determined to be a background, and the calculated similarity is determined to be a predetermined value. The selected pixel or area smaller than the threshold value is determined to be an object other than the background.

When determining whether or not a background exists, auxiliary information given in advance for the background model may be used. The auxiliary information is information such as accuracy and brightness of the background model.

For example, depending on the accuracy of the background model, the accuracy of the determination is increased by taking into account this point, for example, by easing the determination or in a region where the accuracy of the local information deteriorates due to the influence of lighting or the like. be able to.

The range of searching for the corresponding candidate point may be set for each area based on the auxiliary information. The pseudo background generation unit 711 generates a desired pseudo background image, for example, “Tokyo Tower and Blue Sky”. This pseudo background is an image that is not currently captured by the image sensors 1 to N, and may be captured in advance by another image sensor. In addition, since the pseudo background is a substitute for the actual background captured in the reference image # 1, it is more realistic if captured in accordance with the movement of the field of view of the reference image # 1.
The image selection unit 712 determines whether the background of the reference image # 1 is the pseudo background (“Tokyo Tower and the blue sky”) generated by the pseudo background generation unit 711 or the actual image (“the person in front of the patterned curtain”). This is to select whether or not to display for each pixel.

The image display unit 713 determines, based on the determination result of the background model determination unit 710, of the area of the reference image # 1 which is determined to be the background.
Instead of the actual image, the background (for example, a pseudo background) selected by the image selection unit 712 is displayed, and it is determined that the area of the reference image # 1 is an object (such as a person) other than the background. As for the area, the image information of the reference image # 1 is left as it is, and an object other than the background captured by the image sensor 1 is displayed as it is. That is, the image display unit 71
In 3, the object (person) as viewed from the field of view of the image sensor 1 is displayed together with the pseudo background.

In this embodiment, the image display unit 713
Displays the image viewed from the field of view of the image sensor 1,
It is also possible to display an image viewed from the other image sensors 2,..., N.

In this embodiment, unlike FIG.
Although the case where the processing is performed without setting the virtual image # 21 by the virtual image sensor 21 has been described as an example, the virtual image # 21 by the virtual image sensor 21 is set as in FIG. It is also possible to carry out an alternative to the reference image 1. In this case, in the image display unit 713,
The image viewed as virtual image # 21 is displayed.

The background model set by the background model information setting section 704 may be generated based on a distance image capturing the actual background, or a sensor for detecting the movement of the camera may be provided. The model may be modified according to the movement of. In this case, a distance image viewed from the virtual image # 21 can be generated, and the distance image itself can be registered as a background model and model information. This makes it possible to use the auxiliary information such as the reliability of the distance image to improve the accuracy of determining whether or not the image is a background.

[Brief description of the drawings]

FIG. 1 is a diagram showing a concept of a virtual image sensor for explaining a first embodiment.

FIG. 2 is a diagram illustrating a method of merging similarities of multi-view stereo.

FIG. 3 is a diagram for explaining the first embodiment, and is a diagram for explaining processing content of distance measurement of a multi-view stereo using a virtual visual field (when a reference image sensor is set).

FIG. 4 is a diagram for explaining the first embodiment, and is a block diagram of a multi-view stereo distance measurement system using a virtual visual field (when a reference image sensor is set).

FIG. 5 is a diagram for explaining the first embodiment, and is a diagram for explaining processing content of distance measurement of a multi-view stereo using a virtual visual field (when a reference image sensor is not set).

FIG. 6 is a diagram for explaining the first embodiment, and is a block diagram of a multi-view stereo distance measuring system using a virtual visual field (when a reference image sensor is not set).

FIGS. 7A and 7B are views for explaining a second embodiment, and are perspective views showing image sensor groups in which different types of image sensors are arranged.

FIG. 8 is a diagram conceptually showing a vehicle-mounted system assumed in a third embodiment.

FIG. 9 is a diagram illustrating a second embodiment, and is a block diagram of a multi-view stereo distance measuring system that integrates different types of image sensors using a virtual field of view.

FIGS. 10A, 10B, and 10C are diagrams for explaining the contents of processing executed by a distance estimating unit shown in FIG. 9;

FIG. 11 is a diagram for explaining the fourth embodiment, and is a diagram showing where points on a model appear in an image when a virtual visual field is used.

FIG. 12 is a diagram illustrating a fourth embodiment, and is a configuration block diagram of a multi-view stereo apparatus that estimates a model using a virtual visual field.

FIG. 13 is a diagram illustrating an example of a model assumed in a fourth embodiment.

FIG. 14 is a graph showing a correspondence relationship between a hypothetical model obtained by the similarity calculation unit shown in FIG. 12 and a reciprocal of the similarity;

FIG. 15 is a diagram illustrating a modified example of the fourth embodiment, and is a diagram illustrating a case where an obstacle existing on a traveling path is recognized.

FIG. 16 is a diagram illustrating a process of identifying a traveling path and an obstacle in the embodiment shown in FIG.

FIG. 17 is a diagram illustrating a modified example of the fourth embodiment, in which it is known that a traveling path is flat and an obstacle existing on this plane is recognized. FIG.

FIG. 18 is a diagram illustrating a process of identifying a traveling path and an obstacle in the embodiment shown in FIG. 17;

FIG. 19 is a diagram illustrating a fifth embodiment, and is a block diagram illustrating a system that displays an image with a pseudo background as a background.

FIG. 20 is a diagram showing the principle of a conventional twin-lens stereo.

FIG. 21 is a diagram illustrating a conventional distance measurement process of a twin-lens stereo.

FIG. 22 is a graph showing a correspondence relationship between an assumed distance and a reciprocal of a similarity.

FIG. 23 is a block diagram showing a configuration of a conventional twin-lens stereo apparatus.

FIG. 24 is a diagram for explaining processing content of a conventional multi-lens stereo distance measurement.

FIG. 25 is a block diagram showing a configuration of a conventional multi-view stereo apparatus.

FIG. 26 is a diagram for explaining a conventional technique for measuring a distance based on an imaging result of an image sensor group mounted on a vehicle.

FIG. 27 is a diagram illustrating a fourth embodiment, and is a block diagram of a multi-view stereo apparatus that estimates a model without using a virtual visual field.

FIG. 28 is a diagram illustrating the fourth embodiment, and is a diagram illustrating where points on a model are projected on an image when a virtual visual field is not used.

[Explanation of symbols]

 Reference Signs List 21 virtual image sensor 1 to N image sensor 11 image sensor group 50 recognition target object 60 vehicle 61 road 62 obstacle 305 virtual visual field information setting unit 314 distance information estimation unit 513 external information input unit 605 assumed model information setting unit 611 model estimation unit 711 Pseudo background generation unit

Continued on the front page (72) Inventor Shigeru Kimura 9-1-403, Sugogaoka, Miyamae-ku, Kawasaki-shi, Kanagawa (72) Inventor Katsuyuki Nakano 2-2-30 Nakameguro, Meguro-ku, Tokyo (72) Inventor Hiroyoshi Yamaguchi 2597 Shinomiya, Hiratsuka-shi, Kanagawa Pref., Ltd.Tetsuya Shinbo Inspector Tetsuya Shinbo 2597 Shinoho, Hiratsuka-shi, Kanagawa Pref. 2-8-24 Arima, Miyamae-ku Inside Cyvers Corporation (72) Inventor Masato Ogata 345 Kamimachiya, Kamakura-shi, Kanagawa Prefecture Mitsubishi Precision Corporation

Claims (30)

[Claims] A plurality of image pickup means are arranged at a predetermined interval, and correspond to a selected pixel in an image picked up by the one image pickup means when one of the plurality of image pickup means picks up an image of a target object. Information of corresponding candidate points in a captured image of another imaging means to be extracted for each magnitude of an assumed distance from the one imaging means to a point on the object corresponding to the selected pixel, Calculate the similarity between the image information and the image information of the corresponding candidate point, and calculate the assumed distance when the calculated similarity is the largest, from the one imaging unit on the object corresponding to the selected pixel. A distance to a point, and in an object recognition device configured to recognize the object based on the distance obtained for each selected pixel, a virtual position when the object is imaged at a desired position with a desired field of view By imaging means A virtual visual field information setting unit that sets a virtual captured image instead of the captured image of the one imaging unit, and the plurality of captured images corresponding to selected pixels in the virtual image set by the virtual visual field information setting unit Corresponding candidate points for extracting information on corresponding candidate points in the captured image of the means for each magnitude of an assumed distance from the viewpoint set by the virtual visual field information setting means to a point on the object corresponding to the selected pixel Information extracting means, similarity calculating means for calculating the similarity between the image information of the corresponding candidate points extracted by the corresponding candidate point information extracting means, and the similarity calculated by the similarity calculating means being the largest And the distance estimating means for obtaining the distance from the viewpoint set by the virtual visual field information setting means to a point on the object corresponding to the selected pixel, and obtaining this distance for each selected pixel. Object recognition device. 2. A method for determining each of the selected pixels in the virtual image based on the magnitude of the similarity, the rate of change of the similarity, or the rate of change of the original image for each assumed distance calculated by the similarity calculating means. 2. The object recognition apparatus according to claim 1, further comprising a distance reliability calculating means for calculating the reliability of the distance to be obtained. 3. An image indicating a distance or a degree of reliability from a viewpoint set by the virtual visual field information setting means to each point of the object based on a distance obtained for each selected pixel in the virtual image. 3. The object recognition apparatus according to claim 1, further comprising an image generation unit for generating the image. 4. The method according to claim 1, further comprising the step of: setting, based on the distance determined for each selected pixel in the virtual image and image information of a corresponding point corresponding to the distance, the object at the viewpoint set by the virtual visual field information setting means. 2. The object recognition apparatus according to claim 1, further comprising an image generation unit configured to generate an image when the image is captured. 5. A plurality of imaging means are arranged at predetermined intervals, and correspond to a selected pixel in a captured image of one of the plurality of imaging means when the target object is imaged by one of the plurality of imaging means. Information of corresponding candidate points in a captured image of another imaging means to be extracted for each magnitude of an assumed distance from the one imaging means to a point on the object corresponding to the selected pixel, Calculate the similarity between the image information and the image information of the corresponding candidate point, and calculate the assumed distance when the calculated similarity is the largest, from the one imaging unit on the object corresponding to the selected pixel. In an object recognition device configured to recognize the object based on a distance determined for each selected pixel as a distance to a point, the plurality of imaging units may include at least two types of images under different conditions for imaging the object. Imaging A virtual image picked up by a virtual image pickup means when the object is picked up at a desired position and with a desired visual field is set instead of the image picked up by the one image pickup means. Virtual visual field information setting means, and information on corresponding candidate points in the captured images of the respective imaging means groups corresponding to the selected pixels in the virtual image set by the virtual visual field information setting means. Means for extracting corresponding candidate point information for each magnitude of an assumed distance from the viewpoint set by the means to the point on the object corresponding to the selected pixel; and a corresponding candidate extracted by the corresponding candidate point information extracting means. A similarity calculating unit that calculates the similarity between the pieces of image information of points for each of the imaging unit groups, and based on the similarity of each of the imaging unit groups calculated by the similarity calculating unit. Seeking fusion similarity, the assumptions distance when the fusion similarity is maximized,
An object recognizing device comprising: a distance from a viewpoint set by the virtual visual field information setting unit to a point on the object corresponding to the selected pixel; and a distance estimating unit that calculates the distance for each selected pixel. 6. A plurality of imaging means are arranged at a predetermined interval, and correspond to a selected pixel in a captured image of the one imaging means when one of the plurality of imaging means images a target object. Information of corresponding candidate points in a captured image of another imaging means to be extracted for each magnitude of an assumed distance from the one imaging means to a point on the object corresponding to the selected pixel, Calculate the similarity between the image information and the image information of the corresponding candidate point, and calculate the assumed distance when the calculated similarity is the largest, from the one imaging unit on the object corresponding to the selected pixel. In an object recognition device configured to recognize the object based on a distance determined for each selected pixel as a distance to a point, the plurality of imaging units may include at least two types of images under different conditions for imaging the object. Imaging Imaging means group selecting means for selecting at least one imaging means group from the imaging means group as an imaging means group to be actually used, according to an imaging condition of the object; A virtual field of view information setting means for setting a virtual captured image by a virtual image capturing means when capturing the object with a desired visual field at a desired position, instead of the captured image of the one image capturing means, The information of the corresponding candidate point in the captured image of the selected image capturing unit group corresponding to the selected pixel in the virtual image set by the virtual visual field information setting unit, from the viewpoint set by the virtual visual field information setting unit Corresponding candidate point information extracting means for extracting for each magnitude of the assumed distance to the point on the object corresponding to the selected pixel; and image information of corresponding candidate points extracted by the corresponding candidate point information extracting means. Similarity, similarity calculating means for calculating the selected imaging means group, and the assumed distance when the similarity for the selected imaging means group calculated by the similarity calculating means is the largest, The distance from the viewpoint set by the virtual visual field information setting means to a point on the object corresponding to the selected pixel,
An object recognizing device comprising a distance estimating means for obtaining this distance for each selected pixel. 7. A plurality of image pickup means are arranged at predetermined intervals, and correspond to a selected pixel in an image picked up by the one image pickup means when one of the plurality of image pickup means picks up an image of a target object. Information of corresponding candidate points in a captured image of another imaging means to be extracted for each magnitude of an assumed distance from the one imaging means to a point on the object corresponding to the selected pixel, Calculate the similarity between the image information and the image information of the corresponding candidate point, and calculate the assumed distance when the calculated similarity is the largest, from the one imaging unit on the object corresponding to the selected pixel. In an object recognition device configured to recognize the object based on the distance obtained for each selected pixel as a distance to a point, an actual object is selected from among the plurality of imaging units in accordance with an imaging condition of the object. Few to use for An image pickup means selecting means for selecting two image pickup means; and a virtual image picked up by the virtual image pickup means when the object is imaged at a desired position and a desired visual field according to the image pickup condition of the object. Virtual field information setting means to be set instead of the captured image of the image capturing means, and captured images of the selected at least two image capturing means corresponding to the selected pixels in the virtual image set by the virtual visual field information setting means Corresponding candidate point information extracting means for extracting information of corresponding candidate points in, for each magnitude of an assumed distance from the viewpoint set by the virtual visual field information setting means to a point on the object corresponding to the selected pixel. A similarity calculating unit that calculates the similarity between the image information of the corresponding candidate points extracted by the corresponding candidate point information extracting unit; and the similarity calculated by the similarity calculating unit is the largest. And the distance estimating means for obtaining the distance from the viewpoint set by the virtual visual field information setting means to a point on the object corresponding to the selected pixel, and obtaining this distance for each selected pixel. Object recognition device. 8. A plurality of image pickup means arranged at predetermined intervals for picking up an image of a recognition target object, and a model group corresponding to the recognition target object is assumed, and position coordinates of each point on each model of the model group are assumed. A hypothetical model information setting unit that sets the other imaging corresponding to a selected pixel in a captured image of the one imaging unit when one of the plurality of imaging units images the recognition target object. Corresponding candidate point information extracting means for extracting information on corresponding candidate points in the captured image of the means for each hypothetical model selected from the model group by using position coordinates of the hypothetical model; and A similarity calculating unit that calculates a similarity between the image information and the image information of the corresponding candidate point extracted by the corresponding candidate point information extracting unit; and a process before the similarity calculated by the similarity calculating unit becomes the largest. Assumptions model, a model of a point corresponding to the selected pixel, the object recognition device with the model estimating means for obtaining the model for each selected pixel. 9. An image pick-up of one of the plurality of image pick-up means based on a magnitude of similarity, a change rate of the similarity, or a change rate of an original image for each hypothetical model calculated by the similarity calculating means. 9. A model reliability calculating means for calculating a model reliability calculated for each selected pixel in an image picked up by the one image pick-up means when the recognition target object is picked up by the means. An object recognition device according to the above. 10. A model obtained for each selected pixel in an image picked up by the one imaging means when one of the plurality of imaging means images the recognition target object, 10. The object recognition apparatus according to claim 8, further comprising an image generation unit configured to generate an image indicating a value or reliability of the model. 11. A plurality of imaging means arranged at predetermined intervals for imaging a recognition target object in a space, a model corresponding to the recognition target object in the space is generated, and a position of each point on the model is generated. Hypothetical model information setting means for setting coordinates, and the model corresponding to a selected pixel in a captured image of the one imaging means when one of the plurality of imaging means images the recognition target object Corresponding candidate point information extracting means for extracting information on a corresponding candidate point in a captured image of another imaging means corresponding to the above point using the position coordinates of the model; image information of the selected pixel and the corresponding candidate A similarity calculating unit that calculates the similarity of the image information of the corresponding candidate points extracted by the point information extracting unit; and a similarity calculated by the similarity calculating unit is obtained for each selected pixel. Asked for It is by using the degree of similarity,
The region of the selected pixel whose similarity is equal to or larger than a predetermined threshold is determined to be the model, and the region of the selected pixel whose similarity is smaller than the predetermined threshold is an object other than the model. Object recognizing device, comprising: determining means for determining 12. A plurality of image pickup means arranged at predetermined intervals to image a recognition target object and a background in a space, a model corresponding to the background in the space is generated, and a position of each point on the model is generated. Assumption model information setting means for setting coordinates, and selection indicating the background in an image captured by the one imaging means when one of the plurality of imaging means images the recognition target object and the background. Correspondence candidate point information extraction means for extracting information on a corresponding candidate point in a captured image of another imaging means corresponding to a pixel using position coordinates of the background model; and image information of the selected pixel and the corresponding candidate A similarity calculating unit that calculates the similarity of the image information of the corresponding candidate points extracted by the point information extracting unit; and a similarity calculated by the similarity calculating unit is obtained for each selected pixel. Request Obtained by using the degree of similarity,
The area of the selected pixel whose similarity is equal to or more than a predetermined threshold is determined to be the background, and the area of the selected pixel whose similarity is smaller than the predetermined threshold is an object other than the background. Determining means, a pseudo background image generating means for generating a desired pseudo background image, and a pseudo pixel generated by the pseudo image background image generating means for an area of a selected pixel determined to be a background by the determining means. Image display means for displaying a background image and displaying the object other than the background as it is based on the image information of the selected pixel for the area of the selected pixel determined to be an object other than the background by the judgment means. Display device. 13. An appropriate one model to be assumed as a hypothetical model from the model group based on the model estimation information obtained by the model estimation means or the model reliability calculation means and other sensor information. 12. The apparatus for recognizing an object according to claim 8, further comprising model group selecting means for sequentially selecting a group of models. 14. A virtual image picked up by a virtual image pickup means when the object is picked up at a desired position with a desired field of view, is set instead of the image picked up by said one image pickup means, and Information on the corresponding candidate points in the captured images of the plurality of imaging units corresponding to the selected pixels in the virtual image, for each hypothetical model selected from the model group,
11. The object recognition device according to claim 8, wherein extraction is performed using the position coordinates of the hypothetical model, and a degree of similarity between the image information of the extracted corresponding candidate points is calculated. . 15. A virtual image picked up by a virtual image pickup means when the object is picked up at a desired position with a desired field of view, is set instead of the image picked up by the one image pickup means, and The information of the corresponding candidate points in the captured images of the plurality of imaging units corresponding to the selected pixels in the virtual image is extracted using the position coordinates of the model, and the similarity of the image information of the extracted corresponding candidate points is determined. 13. The object recognition device according to claim 11, wherein the degree is calculated. 16. A plurality of imaging means are arranged at a predetermined interval, and correspond to a selected pixel in a captured image of one of the plurality of imaging means when the target object is imaged by one of the plurality of imaging means. Information of corresponding candidate points in a captured image of another imaging means to be extracted for each magnitude of an assumed distance from the one imaging means to a point on the object corresponding to the selected pixel, Calculate the similarity between the image information and the image information of the corresponding candidate point, and calculate the assumed distance when the calculated similarity is the largest, from the one imaging unit on the object corresponding to the selected pixel. In a method for recognizing the object based on the distance obtained for each selected pixel as a distance to a point, a virtual position when the object is imaged with a desired field of view at a desired position. For imaging means A virtual captured image to be set, instead of the captured image of the one imaging unit, a virtual visual field information setting step to be set, and the plurality of virtual image information corresponding to selected pixels in the virtual image set in the virtual visual field information setting step. Correspondence candidates for extracting information on corresponding candidate points in an image picked up by an image pickup means for each magnitude of an assumed distance from a viewpoint set by the virtual visual field information setting means to a point on the object corresponding to the selected pixel. A point information extraction step, a similarity calculation step of calculating the similarity between the image information of the corresponding candidate points extracted in the corresponding candidate point information extraction step, and a similarity calculated in the similarity calculation step is the largest. The assumed distance is a distance from the viewpoint set in the virtual visual field information setting step to a point on the object corresponding to the selected pixel, and a distance estimation step of obtaining this distance for each selected pixel. Ingredient Object recognition method. 17. A method for calculating each of the selected pixels in the virtual image based on the magnitude of the similarity, the rate of change of the similarity, or the rate of change of the original image for each assumed distance calculated in the similarity calculating step. 17. The object recognition method according to claim 16, further comprising a distance reliability calculation step of calculating the reliability of the distance to be obtained. 18. An image indicating a distance or a degree of reliability from a viewpoint set in the virtual visual field information setting step to each point of the object based on a distance obtained for each selected pixel in the virtual image. 18. The method for recognizing an object according to claim 16, further comprising an image generation step for generating. 19. The object according to the viewpoint set in the virtual visual field information setting step, based on distances obtained for each selected pixel in the virtual image and image information of corresponding points corresponding to the distances. 17. The method for recognizing an object according to claim 16, further comprising an image generation step of generating an image when the image is captured. 20. A plurality of image pickup means arranged at predetermined intervals, corresponding to a selected pixel in a picked-up image of the one image pickup means when one of the plurality of image pickup means picks up an image of a target object. Information of corresponding candidate points in a captured image of another imaging means to be extracted for each magnitude of an assumed distance from the one imaging means to a point on the object corresponding to the selected pixel, Calculate the similarity between the image information and the image information of the corresponding candidate point, and calculate the assumed distance when the calculated similarity is the largest, from the one imaging unit on the object corresponding to the selected pixel. A method for recognizing an object based on a distance determined for each selected pixel as a distance to a point, wherein the plurality of imaging units are different in at least two types under different conditions for imaging the object. Shooting A virtual image picked up by the virtual image pickup means when the object is picked up at a desired position and with a desired visual field is set instead of the image picked up by the one image pickup means. A virtual visual field information setting step, and information on a corresponding candidate point in a captured image of each of the imaging means groups corresponding to a selected pixel in the virtual image set in the virtual visual field information setting step. A corresponding candidate point information extracting step of extracting for each magnitude of an assumed distance from a viewpoint set in the process to a point on the object corresponding to the selected pixel; and a corresponding candidate extracted in the corresponding candidate point information extracting process. A similarity calculation step for calculating the similarity between the image information of points for each of the imaging means groups; and a similarity calculation section for each of the imaging means groups calculated in the similarity calculation step. Seeking fusion similarity Te, the assumptions distance when the fusion similarity is maximized,
A method for recognizing an object, comprising: a distance from a viewpoint set in the virtual visual field information setting step to a point on the object corresponding to the selected pixel, and a distance estimating step of obtaining the distance for each selected pixel. 21. A plurality of image pickup means arranged at predetermined intervals, corresponding to a selected pixel in a picked-up image of the one image pickup means when one of the plurality of image pickup means picks up an image of a target object. Information of corresponding candidate points in a captured image of another imaging means to be extracted for each magnitude of an assumed distance from the one imaging means to a point on the object corresponding to the selected pixel, Calculate the similarity between the image information and the image information of the corresponding candidate point, and calculate the assumed distance when the calculated similarity is the largest, from the one imaging unit on the object corresponding to the selected pixel. A method for recognizing an object based on a distance determined for each selected pixel as a distance to a point, wherein the plurality of imaging units are different in at least two types under different conditions for imaging the object. Shooting An imaging means group selecting step of selecting at least one imaging means group from the imaging means group as an imaging means group to be actually used, according to an imaging condition of the object; At a desired position, a virtual image captured by the virtual image capturing means when capturing the object with a desired visual field, instead of the image captured by the one image capturing means, a virtual visual field information setting process, The information of the corresponding candidate point in the captured image of the selected imaging unit group corresponding to the selected pixel in the virtual image set in the virtual visual field information setting step is obtained from the viewpoint set in the virtual visual field information setting step. A corresponding candidate point information extraction step to extract for each magnitude of an assumed distance to a point on the object corresponding to the selected pixel; and image information of corresponding candidate points extracted in the corresponding candidate point information extraction step. A similarity, a similarity calculation step for calculating the selected imaging unit group, and the assumed distance when the similarity for the selected imaging unit group calculated in the similarity calculation stage is the largest, The distance from the viewpoint set in the virtual visual field information setting step to a point on the object corresponding to the selected pixel,
A method for recognizing an object, comprising: a distance estimation step of obtaining this distance for each selected pixel. 22. A plurality of image pickup means are arranged at a predetermined interval, and correspond to a selected pixel in a picked-up image of the one image pickup means when one of the plurality of image pickup means picks up an image of a target object. Information of corresponding candidate points in a captured image of another imaging means to be extracted for each magnitude of an assumed distance from the one imaging means to a point on the object corresponding to the selected pixel, Calculate the similarity between the image information and the image information of the corresponding candidate point, and calculate the assumed distance when the calculated similarity is the largest, from the one imaging unit on the object corresponding to the selected pixel. A method for recognizing the object based on the distance determined for each selected pixel as a distance to a point, the method comprising: Small to use An imaging means selection step of selecting at least two imaging means, a desired position according to the imaging conditions of the object, a virtual image captured by the virtual imaging means when imaging the object with a desired field of view, Virtual field information setting means for setting, instead of the image picked up by one image pickup means; imaging of the selected at least two image pickup means corresponding to selected pixels in the virtual image set in the virtual field information setting step A corresponding candidate point information extracting step of extracting information on a corresponding candidate point in an image from the viewpoint set in the virtual visual field information setting step for each magnitude of an assumed distance from the viewpoint on the object corresponding to the selected pixel; A similarity calculation step of calculating the similarity between the image information of the corresponding candidate points extracted in the corresponding candidate point information extraction step; and the similarity calculated in the similarity calculation step is the largest. The assumed distance at the time of setting the distance from the viewpoint set in the virtual visual field information setting step to a point on the object corresponding to the selected pixel, and a distance estimation step of obtaining this distance for each selected pixel. A method for recognizing equipped objects. 23. An imaging process of imaging a recognition target object by a plurality of imaging means arranged at predetermined intervals, and a model group corresponding to the recognition target object is assumed. A hypothetical model information setting step of setting a position coordinate of a point, and corresponding to a selected pixel in an image captured by the one imaging unit when the recognition target object is imaged by one of the plurality of imaging units. Corresponding candidate point information extraction step of extracting information of corresponding candidate points in an image captured by another imaging means for each hypothetical model selected from the model group using the position coordinates of the hypothetical model, A similarity calculation step of calculating the similarity between the image information of the selected pixel and the image information of the corresponding candidate point extracted in the corresponding candidate point information extraction step; and the similarity calculated in the similarity calculation step is the largest. The assumptions model, a model of a point corresponding to the selected pixel, the object recognition method that comprises the model estimation step of obtaining the model for each selected pixel when made. 24. One of the plurality of image pickup means based on the magnitude of similarity, the rate of change of the similarity, the rate of change of the original image, etc. for each hypothetical model calculated in the similarity calculation step. 24. The apparatus according to claim 23, further comprising: a model reliability calculation step of calculating a model reliability calculated for each selected pixel in the image picked up by the one image pickup means when the recognition target object is picked up by the means. The method of recognizing the described object. 25. Based on a model obtained for each selected pixel in an image picked up by the one imaging means when one of the plurality of imaging means images the recognition target object, An image generating step for generating an image indicating a model value or reliability is further provided.
26. The object recognition method according to claim 24 or claim 25. 26. An imaging step of imaging one recognition target object in a space by a plurality of imaging means arranged at predetermined intervals, and generating a model corresponding to the one recognition target object in the space; A hypothetical model information setting step of setting the position coordinates of each point on the model; and a captured image of the one imaging unit when the one of the plurality of imaging units captures the recognition target object. A corresponding candidate point information extracting step of extracting information of corresponding candidate points in a captured image of another imaging unit corresponding to the selected pixel indicating the model using the position coordinates of the model; and image information of the selected pixel. And a similarity calculation step of calculating the similarity of the image information of the corresponding candidate points extracted in the corresponding candidate point information extraction step, and a similarity calculated in the similarity calculation step is obtained for each selected pixel, Each selected picture Using the similarity obtained for each,
The region of the selected pixel whose similarity is equal to or greater than a predetermined threshold is determined to be the recognition target object, and the region of the selected pixel whose similarity is smaller than the predetermined threshold is determined as the recognition target object. A method for recognizing an object, comprising: a determination step of determining that the object is other than an object. 27. A recognition target object and a background in a space,
An imaging step of imaging with a plurality of imaging means arranged at predetermined intervals, and a model corresponding to a background in the space, and a hypothetical model information setting step of setting position coordinates of each point on the model, When one of the plurality of imaging units captures the recognition target object and the background, the captured image of another imaging unit corresponding to the selected pixel indicating the background in the captured image of the one imaging unit A corresponding candidate point information extracting step of extracting information of corresponding candidate points in the image using the position coordinates of the background model; and a corresponding candidate point extracted in the image information of the selected pixel and the corresponding candidate point information extracting step. A similarity calculation process for calculating the similarity of the image information of the, and the similarity calculated in the similarity calculation process is obtained for each selected pixel, using the similarity obtained for each selected pixel,
The region of the selected pixel whose similarity is equal to or greater than a predetermined threshold is determined to be the background, and the region of the selected pixel whose similarity is smaller than the predetermined threshold is determined by an object other than the background. A determination process for determining that there is a pixel; a pseudo background image generation process for generating a desired pseudo background image; and a region of a selected pixel determined to be a background in the determination process is generated in the pseudo image background image generation process. Displaying a pseudo background image, and for an area of a selected pixel determined to be an object other than the background in the determination step, an image display step of displaying the object other than the background as it is based on the image information of the selected pixel. Equipped display method. 28. An appropriate model to be a hypothetical model from the model group based on the model estimation information and other sensor information obtained in the model estimation step or the model reliability calculation step. 27. The object recognition method according to claim 23, further comprising a model group selection step of sequentially selecting a model group. 29. A virtual image picked up by a virtual image pickup means when the object is picked up at a desired position with a desired field of view, is set instead of the image picked up by the one image pickup means, and Information on the corresponding candidate points in the captured images of the plurality of imaging units corresponding to the selected pixels in the virtual image, for each hypothetical model selected from the model group,
26. The object recognition method according to claim 23, wherein the extraction is performed using the position coordinates of the hypothetical model, and the degree of similarity between the image information of the extracted corresponding candidate points is calculated. . 30. A virtual image picked up by a virtual image pickup means when the object is picked up at a desired position and with a desired visual field, is set instead of the image picked up by the one image pickup means, and The information of the corresponding candidate points in the captured images of the plurality of imaging units corresponding to the selected pixels in the virtual image is extracted using the position coordinates of the model, and the similarity of the image information of the extracted corresponding candidate points is determined. 28. The object recognition method according to claim 26, wherein the degree is calculated.