JP2001243456A - Device and method for detecting obstacle - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an obstacle detecting device which can detect an obstacle, such as a preceding vehicle and a walker, present on a road plane even if the road surface slants or a camera vibrates. SOLUTION: This device is equipped with an image input part 1 which inputs images from a right and a left camera mounted on an automobile, an image storage part 2 which stores the two images inputted from the right and left cameras, an image conversion part 3 which converts the image inputted from one camera to the viewpoint of the other camera by using plural kinds of conversion parameters previously found according to the geometric relation between the cameras and road plane by plural kinds of attitudes selected within the range of attitudes that the cameras can take, an image conversion part 3 which generates plural converted images, and a matching process part 4 which compares the converted image with the image inputted from the other camera by pixels or small areas to find how much the converted image matches the image from the other camera.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention detects an obstacle existing on a road, such as a preceding vehicle, a parked vehicle, or a pedestrian, by using an on-vehicle camera in order to realize safe driving assistance and automatic driving of an automobile. The present invention relates to an obstacle detection device and an obstacle detection method.

[0002]

2. Description of the Related Art Techniques for detecting an obstacle can be roughly classified into those using a laser or an ultrasonic wave and those using a TV camera.

[0003] Those using a laser are expensive, and those using an ultrasonic wave have a low resolution of the ultrasonic wave, so that there is a problem in the accuracy of detecting an obstacle. Further, the active lane using the laser or the ultrasonic wave alone cannot recognize the traveling lane.

On the other hand, TV cameras are relatively inexpensive and are suitable for obstacle detection in terms of resolution, measurement accuracy, and measurement range. It is also possible to recognize the traveling lane. TV
When a camera is used, there are a method using one camera and a method using a plurality of cameras (stereo cameras).

In the method using one camera, a road area and an obstacle area are separated from one image captured by the camera by using information such as luminance, color, and texture. For example, in the image, a medium-luminance area with low saturation, that is, a gray area is extracted to obtain a road area, or an area with less texture is obtained, a road area is extracted, and the other areas are determined as obstacle areas. And However, since there are many obstacles having luminance, color, or texture similar to a road, it is difficult to separate an obstacle region from a road region by this method.

On the other hand, the method using a plurality of cameras detects an obstacle using three-dimensional information as a clue.
This method is generally called stereo vision. Stereo vision refers to, for example, arranging two cameras on the left and right, associating a point that is the same point in a three-dimensional space between left and right images, and finding a three-dimensional position of the point in a triangulation manner. is there. If the position, posture, and the like of each camera with respect to the road plane are obtained in advance, the height of an arbitrary point in the image from the road plane can be obtained by stereo vision. In this way, the obstacle area and the road area can be separated depending on the presence or absence of the height. According to the stereo vision, it is possible to avoid the problem as in the case of using one camera.

However, ordinary stereo vision has a problem of searching for corresponding points. In general, stereo vision is a technique for obtaining a three-dimensional position of an arbitrary point on an image in a coordinate system fixed to a stereo camera (hereinafter, referred to as a stereo camera coordinate system). The corresponding point search means a search calculation required for associating the same point in space between the left and right images, and has a problem that the calculation cost is extremely high. The corresponding point search is a factor hindering the practical use of stereo vision.

However, if it is only necessary to separate the road area and the obstacle area on the image, it is not necessary to search for the corresponding point, and the presence or absence of the height from the road plane can be determined as follows, for example. .

If the projection points of the points on the road plane onto the left and right images are (u, v) and (u ', v'), a relational expression such as the following expression (1) is established.

[0010]

(Equation 1)

H ^→ = (h ₁₁ , h ₁₂ , h ₁₃ , h ₂₁ , h ₂₂ ,
h ₂₃ , h ₃₁ , h ₃₂ , h ₃₃ ) are parameters depending on the position and orientation of each camera with respect to the road plane, the focal length of the lens of each camera, and the image origin. h ^→ is projected points to 4 points or more of the left and right images on the advance road plane _{(u i, v i),} (u i ', v i') (i = 1,2,
..., N).

Using this relational expression, a corresponding point P ′ (u ′, v ′) on the right image when an arbitrary point P (u, v) on the left image is assumed to be on the road plane. Ask. If the point P exists on the road plane, the points P and P 'are a correct pair of corresponding points, and the two points have the same luminance. Therefore,
If the luminance of the point P is different from that of the point P ′, it can be determined that the point P belongs to an obstacle. This method is based on the equation (1)
Alone, it is possible to directly determine the presence or absence of the height of an arbitrary point on the image from the road surface.
It can be obtained only from the projection points of the feature points or more on the left and right images, and the corresponding point search between the left and right images is unnecessary.

When moving on a flat floor surface at a relatively low speed in an indoor environment, h ^→ can be regarded as fixed, and an obstacle can be correctly detected using h ^→ once obtained. However, when the car travels outdoors, the relationship between the road plane and the relative position and posture of each camera changes every moment due to vibrations of the car itself, changes in the inclination of the road, and the like. Therefore, since the parameter h ^→ also changes with the movement of the vehicle, h ^→ previously obtained, for example, when the vehicle is at rest, is used as it is, a certain camera image is converted by the equation (1), and the obstacle is caused by a simple difference from the other camera image. When an object is detected, there is a problem that the detection accuracy is significantly reduced.

[0014]

As described above, obstacle detecting devices can be roughly classified into those using a laser or an ultrasonic wave and those using a TV camera. There are problems in that they are expensive, have low measurement accuracy, and cannot recognize the traveling lane. In addition, an obstacle detection device using a TV camera has a limited use environment, requires a high-cost calculation search, and cannot cope with vibrations during driving and inclination of a road. However, there is a problem that the performance is significantly deteriorated.

[0015] The present invention has been made in consideration of the above circumstances, and when an obstacle is detected using a plurality of cameras,
It is an object of the present invention to provide an obstacle detection device and an obstacle detection method capable of detecting an obstacle existing on a plane with high accuracy even when the plane is tilted or the camera shakes.

The present invention also provides a method for detecting an obstacle existing on a road surface, such as a preceding vehicle or a pedestrian, using a vehicle-mounted stereo camera. An object of the present invention is to provide an obstacle detection device and an obstacle detection method capable of detecting an obstacle existing on a plane with high accuracy.

[0017]

According to the present invention (claim 1), an obstacle detection device comprises: an image input means for inputting an image from a plurality of image pickup means having different viewpoints mounted on an object; Image storage means for storing a plurality of images respectively input by the image pickup means, and each of a plurality of planes prepared in advance (πn (n = 1,
.., N)) and a plurality of image transformations (T) each derived for each plane from the geometric relationship between the imaging means.
n (n = 1,..., n)), converting the image input from one imaging unit to the viewpoint of another imaging unit, thereby generating a plurality of converted images; Matching for comparing the generated plurality of converted images and the image input from the other imaging means for each pixel or for each small region including a plurality of pixels to obtain the degree of coincidence of each pixel or small region. Processing means; and an obstacle detecting means for detecting an area on the image having a low degree of coincidence as an obstacle area for the object.

Preferably, the matching processing means includes:
When comparing one of the converted images with an image input from the other imaging unit, the converted image is divided into small regions, and for each of the small regions, an image of the image input from the other imaging unit is divided. A matching process may be performed in a search area including a corresponding small area and its periphery, and the maximum matching degree obtained by searching in the search area may be set as the matching degree of the small area.

Preferably, the matching processing means includes:
When comparing a certain small area of a certain conversion image among the plurality of conversion images with an image input from the other imaging means, a search is made in advance in a search area of the image input from the other imaging means. If a matching degree equal to or greater than the predetermined threshold value is obtained, at that time, all subsequent matching processes for the small area are terminated, and the matching degree is determined for all the converted images. The degree may be used.

Preferably, the size of the search area, the pitch at which pixels are shifted during the search, or the type of the image conversion used to generate the converted image, according to the position of the small area on the image At least one of the numbers may be changed.

Preferably, the object is an automobile, and the plurality of image pickup means are right and left image pickup means for obtaining a stereo image, and the image conversion is performed by a left image input from the left image pickup means. Is converted into an image of the viewpoint of the right imaging unit, or the right image input from the right imaging unit is converted into an image of the viewpoint of the left imaging unit, and a plurality of types of the image conversion Are used in advance for each of a plurality of postures selected from a range of possible postures of the vehicle, based on the geometric relationship between the imaging means and the road plane. It may be.

In the obstacle detection method according to the present invention (claim 6), an image is input from a plurality of image pickup units having different viewpoints mounted on a certain object, and the plurality of input images are respectively input by the plurality of image pickup units. A plurality of images are accumulated, and each of a plurality of planes in the space prepared in advance (πn (n = 1,
.., N)) and a plurality of image transformations (T) each derived for each plane from the geometric relationship between the imaging means.
n (n = 1,..., n)), the image input from one imaging unit is converted to the viewpoint of another imaging unit, thereby generating a plurality of converted images, and generating the plurality of converted images. The converted image and the image input from the other imaging means are compared for each pixel or for each small area composed of a plurality of pixels, to determine the degree of coincidence of each pixel or small area, A region on the image with a low degree of coincidence is detected as an obstacle region for the object.

It should be noted that the present invention relating to the apparatus is also realized as an invention relating to a method, and the present invention relating to a method is also realized as an invention relating to an apparatus.

Further, the present invention relating to an apparatus or a method is provided for causing a computer to execute a procedure corresponding to the present invention (or for causing a computer to function as means corresponding to the present invention, or for causing a computer to correspond to the present invention). The present invention is also realized as a computer-readable recording medium in which a program for realizing the function of performing the above is recorded.

According to the present invention, in an image conversion for converting a camera image derived from a geometric relationship between a plurality of TV cameras and a road surface to another camera viewpoint, an assumed camera and road surface are taken. A plurality of suitable postures are selected from a range of relative relationships (postures) to be obtained, image conversions corresponding to all the postures are prepared, converted images of images taken by a certain TV camera are generated, and all or a part thereof is generated. By comparing the converted image with other camera images, obstacles are detected based on the presence or absence of the height of the object on the road plane.Therefore, it is not affected by brightness fluctuations and shadows. An obstacle such as a pedestrian can be detected. In particular, although the process assumes that the road surface is a plane, it has a configuration that also has a mechanism that assumes a plurality of road surfaces and reduces the conversion error. Not affected. Further, it is possible to easily realize hardware and parallel processing of the processing, and particularly to easily increase the processing speed by using the SIMD operation function, which is a high-speed technique generally used in recent processor technology.

[0026]

Embodiments of the present invention will be described below with reference to the drawings.

In this embodiment, two left and right stereo cameras are mounted on a car, and a car equipped with the stereo cameras (hereinafter, referred to as a car) is provided.
Under such conditions that the vehicle equipped with the stereo camera is referred to as a vehicle) and the road surface is subject to vibrations and changes in road surface inclination, obstacles existing on the road surface, such as pedestrians and preceding vehicles, are detected. Assume the situation.

Hereinafter, an image input from the left camera is referred to as a left camera image, and an image input from the right camera is referred to as a right camera image.

In the following, a description will be given of an example of a configuration in which a left camera image is converted to a right camera viewpoint, but, of course, a configuration in which a right camera image is converted to a left camera viewpoint performs essentially the same processing. The present invention can be implemented similarly, and the same effect can be obtained.

FIG. 1 shows a schematic configuration of an obstacle detection device according to an embodiment of the present invention. As shown in FIG. 1, this obstacle detection device uses the two left and right cameras mounted on the own vehicle in front of the own vehicle (when monitoring the rear, the rear of the own vehicle).
An image input unit 1 for inputting a stereo image, an image storage unit 2 for storing input left and right camera images, an image conversion unit 3 for converting one camera image into a viewpoint of the other camera, and converting one camera image into the other camera image. A matching processing unit 4 that performs a matching process on the converted image converted into the camera viewpoint and the other camera image, and an obstacle detection unit 5 that performs an obstacle detection process based on the result of the matching process. In addition,
In the present embodiment, the conversion parameters used for image conversion are not changed / corrected while the vehicle is running (when an obstacle is detected), and the image conversion unit 3 uses a plurality of types of conversion parameters prepared in advance. As a method of obtaining a plurality of types of conversion parameters in advance, an appropriate plurality of postures are selected from a range of postures that the vehicle can take, and for each of the selected postures, the geometric relationship between the camera and the road plane (road surface) is selected. Is used to determine the conversion parameters from the relative relationship between the camera and the camera, that is, the posture. That is, as described later in detail, the image conversion unit 3 converts one camera image to the viewpoint of the other camera using each of the plurality of types of conversion parameters, and generates a plurality of types of converted images. The matching processing unit 4 performs matching processing on the plurality of types of converted images and the other camera image.

Before describing the details of the obstacle detection device of the present embodiment, an image of one camera image to the other camera viewpoint derived from the geometrical relationship between the road surface and the two cameras. The conversion and its conventional problems will be described.

First, two cameras (cars equipped with) are arranged on a plane (on a road plane) as shown in FIG. Further, there are two parallel white lines extending in the front-rear direction on this plane, and these are referred to as L and L '. It is assumed that the relationship between the positions and postures of these two cameras is unknown and only the epipolar constraint is known and does not change during traveling. The epipolar constraint is a constraint that holds for a general stereo image, and FIG. 3A shows a left image, and FIG.
(Right image) indicates that a corresponding point on the right image of a point on the left image is constrained on a straight line. This straight line is called an epipolar line. For example, when the optical axes of the cameras are arranged in parallel, the corresponding point of an arbitrary point on the left image exists on the same scanning line on the right image, so that the epipolar line coincides with the scanning line. The epipolar constraint is based on the relative position / posture relationship between stereo cameras,
Since the intrinsic parameters of each camera are dependent on the focal length of the camera lens and the image origin, the invariable epipolar constraint means that the relative positional relationship and intrinsic parameters of the stereo camera do not change during traveling. I do.

The epipolar constraint is formulated as follows. Assuming that an arbitrary point on the left image is (u, v) and a corresponding point on the right image is (u ′, v ′), the relational expression of Expression (2) holds. Here, [F] is a 3 × 3 matrix,
It is called a fundamental matrix.

[0034]

(Equation 2)

When formula (2) is expanded and arranged, formula (3) is obtained.

[0036]

(Equation 3)

Equation (3) represents the epipolar line on the right image relative to point (u, v) on the left image. Where F
_ij (i, j = 1, 2, 3) is an element at the j-th row and the i-th column of the matrix [F], and is obtained in advance from a set of a plurality of corresponding points.

Although the matrix [F] is composed of nine elements, each element is not independent, and theoretically each element can be obtained from a set of seven or more corresponding points. Since the three-dimensional position of each set of corresponding points is unnecessary, the calculation of the matrix [F], that is, the epipolar constraint, is relatively easy.

On the left and right camera images, two straight lines L and L 'as shown in FIG. 4 ((a) is a left image, (b) is a right image)
Is obtained, the straight line L and the straight line L ′ are parallel to each other in the three-dimensional space, but intersect at a point at infinity called a vanishing point on the image.

A relational expression that holds between corresponding points on the road surface is obtained. FIG. 5 ((a) is a left image, (b) is a right image)
In the left image, any two points on the straight line L are A and C, respectively, and any two points on the straight line L 'are B and
D. The corresponding points A ', B', on the right image of these four points
C ′ and D ′ can be easily obtained by using the previously obtained epipolar constraint. That is, point A
In the corresponding point A 'is the right image corresponds to the intersection of the epipolar line L _A of the straight line L and the point A. Similarly, points B ′,
C ′ and D ′ are also epipolar lines L at points B, C and D, respectively.
_B, can be determined as _L C, the intersection of the _{L D.} Points A, B, C, D and their corresponding points A ', B', C ', D'
Are (u ₁ , u ₁ ), (u ₂ , u ₂ ), (u
_{3, u 3), (u} 4, u 4), (u 1 ', u 1'),
(U ₂ ′, u ₂ ′), (u ₃ ′, u ₃ ′), (u ₄ ′,
u ₄ ′). _{(U i, u} i) and _{(u i ', u i'} )
(I = 1 to 4), the relational expression of Expression (4) holds.

[0041]

(Equation 4)

These eight equations are given by h ^→ = (h ₁₁ ,
_{h 12, h 13, h 21} , h 22, h 23, h 31, h 32, h 33) solving for. If one solution h ^→ satisfies equation (4), then k · h ^→ multiplied by a constant k also satisfies equation (4), so that even if h ₃₃ = 1, generality is not lost. Therefore, h ^→ consisting of nine elements can be obtained from eight equations.

The obtained h ^→ = (h ₁₁ , h ₁₂ , h ₁₃ , h ₂₁ ,
According to _{h 22, h 23, h 31} , h 32, h 33) the use of,
Corresponding point P '(u', v ') on the right image assuming that an arbitrary point P (u, v) on the left image exists on the road surface
Can be obtained as shown in Expression (5).

[0044]

(Equation 5)

The conversion obtained in this manner is, for example, shown in FIG.
(A) is a left image and (b) is a right image, when the left camera image of (a) is converted to the right camera viewpoint, the converted image becomes as shown in (c). .
That is, the pixels on the road surface (in the example of FIG. 6, the contact point between the tire of the preceding vehicle and the road surface) are correctly converted into corresponding points, while the object having a spatial height (FIG. (In the example, the preceding vehicle) is transformed with distortion such as falling down in the image. Therefore, the points P (u, v) and P (u ',
v ′) are represented by I _L (u, v) and I _R (u ′,
v ′), if the point P (u, v) actually exists on the road surface, the point P and the point P ′ are a correct pair of corresponding points. The brightness of P 'becomes the same. Conversely, if the luminance of point P is different from that of point P ', points P and P' are points that are not on the road surface. If the relationship between the road surface and the camera is constant, for example, the evaluation value D is obtained as in Expression (6), and it is possible to determine that the point P where D ≠ 0 belongs to the obstacle area. Alternatively, the threshold value Thr is set in consideration of an error such as a difference between the characteristics of the left and right cameras, and
It is possible to determine that the point P that becomes Thr belongs to the obstacle area.

[0046]

(Equation 6)

However, in practice, various changes occur, such as camera vibration and road surface inclination due to the movement of the vehicle, and it is difficult to determine an obstacle only by equation (6). For example, since the luminance difference between a landmark (for example, a character “stop” drawn on the road surface, a speed limit display, and a pattern on the road surface such as a white line) is large, the assumed road surface and camera geometry are assumed. (6) is not an obstacle because of the positional relationship (the relationship between the camera and the road surface when the above-mentioned image conversion parameters are obtained) and the actual geometric relationship between the road surface and the camera. (= Around the edge).

Therefore, in the present embodiment, a plurality of appropriate postures are selected from the range of possible relative relationships between the vehicle and the road surface, and a plurality of types of image conversion parameters are obtained in advance. Are converted by the plurality of types of image conversion, and a matching process is performed between the converted image and the other camera image to reduce the influence of the conversion error described above.

Hereinafter, the obstacle detecting device of the present embodiment will be described in detail.

The image input unit 1 inputs two left and right images using two left and right TV cameras. However, it is assumed that the geometric relationship between the two cameras and the configuration when the vehicle is mounted do not change / change from the above-described image conversion parameter calculation.

The image storage unit 2 stores the two images input by the image input unit 1 in an image memory.

The image converter 3 converts an image input from one (left in this example) camera into another (right in this example) camera according to each of the plurality of types of conversion parameters obtained by the above-described method. Convert to viewpoint image. Image conversion is performed for each conversion parameter, and converted images are generated by the number of assumed postures.

The matching processing section 4 performs a matching process between the pixels of all the converted images and the corresponding pixels on the other (right side in this example) camera image to obtain the maximum degree of coincidence. As the coincidence, for example, an absolute value of the difference between the pixels (in this example, the smaller the value, the higher the coincidence) can be used, as in Expression (7).

[0054]

(Equation 7)

Here, M (u, v) is a pixel coordinate (u, v).
matching score at v), I _Ti is transformed left camera image by i-th conversion, the I _R is the right camera image.

However, when the range of the assumed posture change is wide or when the conversion parameters used for image conversion are small, the edge of the above-described road surface pattern can be obtained only by using the conversion parameters prepared in advance. There is a possibility that a detection error occurs in the section. Therefore, the matching process is not performed for each pixel, but for an image small area composed of a plurality of pixels (for example, a rectangular area obtained by dividing one image vertically and horizontally) and a corresponding peripheral area (search area). The conversion error can be reduced by performing the search processing of ()).

For example, as shown in a plurality of converted images in FIG. 7A and a right image in FIG. 7B, the maximum degree of coincidence is obtained for a small area at a certain position on the image as follows. .
That is, an image of a small area at a certain position on one converted image and the small area in a search area (a peripheral area including the area corresponding to the small area and its periphery) on the right camera image corresponding to the small area. The degree of coincidence is determined by comparing the area with an area having the same shape,
This is repeated by changing the position of the region having the same shape in the search region, thereby obtaining the maximum matching degree for the small region at the certain position on the one converted image, and calculating the maximum matching degree for all the converted images. This is performed for a small area at a certain position, and finally the maximum matching degree for the small area at the certain position in all the converted images is obtained. This is performed for small areas at all positions on the image. This makes it possible to reduce the conversion error for the small area at each position on the image.

The maximum degree of coincidence for each small area of each converted image is determined by shifting the target area to be matched in the search area by one pixel or several pixels (for example, a spiral shape or the like from the center of the search area). This can be performed by repeating the process of finding the degree of coincidence (searching on the route).

It should be noted that, contrary to the above, it is of course possible to provide a search area by providing a search area in the converted image.

By performing the above-described search processing for the maximum coincidence in all the small areas in all the converted images, it is possible to generate a two-dimensional image formed by the maximum value of the coincidence in each area. For example, when a block shape is selected as the shape of a small area, and a search is performed around a small block on the converted image and a region corresponding to a position on the right image, for example, a matching degree image as shown in FIG. 8 is obtained. Can be. In FIG. 8,
In this example, the degree of coincidence M is normalized to 0 ≦ M ≦ 1 and in increments of 0.05.

On any one of the converted images, the road surface is almost correctly projected, while on the other hand, a tall object such as a car or a pedestrian ahead has a higher height from the road surface on any projected image. Accordingly, the image is projected with a large distortion. Therefore, even if there is a large conversion error in a certain converted image, the small area on the road surface has the highest matching degree with the converted image of the closest posture, so that it has a high matching degree. become. On the other hand, in a small area on a tall object, since the possible posture range of the vehicle is physically constrained, matching cannot be obtained with the converted image corresponding to any posture, resulting in a low match. Degree. In the matching process for each small area,
Normalization function or SAD (Sum of Absolut)
e Difference), SSD (Sum of S)
For example, a general matching method used in image processing such as “Quare Difference” can be used. However, while the normalized correlation has a higher value as the degree of coincidence is higher, the SAD and SSD have a higher degree of coincidence as the value is smaller.

Since the obstacle detection unit 5 detects an area having a low degree of coincidence as an obstacle as will be described later, the area around the right camera image corresponding to a small area at a certain position on the converted image (search area) ), When the maximum matching score is detected, if a matching score equal to or greater than a certain threshold is obtained, at that point, all subsequent searches for the small area at that position are terminated, and the matching score is determined. If the maximum degree of coincidence for the small area at a certain position on all the converted images is determined (or the search for the small area at that position is discontinued at that point in time and the degree of matching is converted to If the maximum degree of coincidence is as to the small area at the certain position above), the speed of the matching process can be remarkably increased. For example, even if there is an error between the image conversion parameter and the optimum image conversion parameter derived from the relationship between the actual camera and the road surface, even if an error occurs in the conversion of the left camera image, the texture (pattern) ), There is often no texture even in the corresponding peripheral region of the corresponding right camera image. Therefore, such a region does not greatly differ from the result obtained even if the entire peripheral region is searched. , The failure detection performance does not deteriorate.

In the above description, regardless of the position of the small area on the image, the size of the search area, the pitch at which pixels are shifted during the search, and the number of types of conversion images (the number of types of conversion parameters to be used) Is fixed, but all or a part of them is located at the position of the small area on the image, especially the vertical position on the image (the lower part of the image is closer to the vehicle, and the upper part of the image is far from the vehicle). Accordingly, it may be changed continuously or stepwise.

For example, the size of the search area may be made larger in the lower area of the image (the area closer to the own vehicle) and smaller in the upper area of the image (the area farther from the own car).

Further, for example, the bitch for shifting the pixels during the search is made smaller in the lower region of the image (the region closer to the own vehicle) and larger in the upper region of the image (the region farther from the own vehicle). You may. Alternatively, the search may be performed more finely in the lower region of the image (the region closer to the own vehicle), and may be searched more coarsely in the upper region of the image (the region farther from the own vehicle).

Further, for example, the number of types of the converted image may be increased in the lower region of the image (region closer to the own vehicle) and reduced in the upper region of the image (region farther from the own vehicle). Good.

In the obstacle detecting section 5, the matching processing section 4
By performing threshold processing on the image (see FIG. 8) composed of the coincidences generated in step (1), an area having a low coincidence on the image is output as an obstacle area. For example, in FIG.
When the threshold value is set to 0.25, the first hatching (20)
Area is detected as an obstacle area (0.80
The following portions are indicated by second hatching (21)).

The result of the obstacle detection can be used for automatic driving of the vehicle, danger avoidance, and the like.

As described above, according to this embodiment, an obstacle is detected based on the presence or absence of an object on a road plane.
An obstacle such as a preceding vehicle or a pedestrian can be detected from the image without being affected by fluctuations in brightness or shadows. In particular, although the process assumes that the road surface is a plane, it has a configuration that also has a mechanism that assumes a plurality of road surfaces and reduces the conversion error. Not affected.

In this embodiment, for the sake of simplicity, the matching calculated by the matching processing unit 4 is a two-dimensional matching image, but the maximum matching of each pixel or each small area is obtained. The same effect can be obtained even if the threshold value processing is performed each time and the data is output as an obstacle area.

By the way, according to the present embodiment, in any of the image conversion processing in the image conversion section 3, the processing for obtaining the degree of coincidence in the matching processing section 4, and the obstacle detection processing in the obstacle detection section 5, SIMD (Single Instrument), which can be easily implemented in hardware and parallelism, and is provided for speeding up multimedia data processing with recent processors.
Ction stream Multiple Dat
It is also easy to increase the speed by using the arithmetic function.

In the above description, the left and right 2
Although the case where the front of the own vehicle is monitored by one camera has been mainly described, it is naturally possible to monitor the rear of the own vehicle as well. In addition, one vehicle may be equipped with both a forward monitoring system and a backward monitoring system.

The present invention can be applied to the case of a running object other than an automobile, and can also be applied to the case where the object is mounted on a stationary object and other objects are monitored.

When the obstacle detecting device has a processor, the processing according to the present embodiment can be realized as software.

When the obstacle detecting device has a processor, the processing according to the present embodiment is for causing a computer to execute predetermined means (or for causing the computer to function as predetermined means). Alternatively, the program may be installed in the obstacle detecting device as a computer-readable recording medium (for example, a ROM) in which a program for causing a computer to perform a predetermined function is recorded.

The present invention is not limited to the above-described embodiments, but can be implemented with various modifications within the technical scope thereof.

[0077]

According to the present invention, since an obstacle is detected based on the presence or absence of the height of an object on a plane, it is possible to detect an obstacle from an image without being affected by variations in brightness or shadows. it can. In particular, since it has a configuration in which a plurality of planes are assumed and a mechanism for reducing the conversion error is also provided, it is not affected by the vibration of the image pickup means or the inclination of the plane.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration example of an obstacle detection device according to an embodiment of the present invention.

FIG. 2 is a diagram showing a positional relationship between a road surface and a stereo camera;

FIG. 3 is a diagram for explaining epipolar constraint;

FIG. 4 is a diagram for explaining a method of calculating an image conversion parameter.

FIG. 5 is a diagram for explaining a method of calculating an image conversion parameter.

FIG. 6 is a diagram for explaining image conversion and characteristics of the converted image;

FIG. 7 is a diagram for explaining a matching process;

FIG. 8 is a diagram illustrating an example of a matching degree image.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 image input unit 2 image storage unit 3 image conversion unit 4 matching processing unit 5 obstacle detection unit

──────────────────────────────────────────────────の Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme court ゛ (Reference) G08G 1/16 G08G 1/16 C H04N 7/18 H04N 7/18 JK (72) Inventor Kazunori Onoguchi Kanagawa 1 Tokoba, Komukai Toshiba-cho, Kawasaki-shi, Tokushima F-term in the Toshiba R & D Center (reference)

Claims (6)

[Claims] An image input means for inputting images from a plurality of image pickup means having different viewpoints mounted on an object, and an image storage for storing a plurality of images respectively input by the plurality of image pickup means. Means, using a plurality of image transformations respectively prepared for each plane from a geometric relationship between each of a plurality of planes in space and the imaging means, prepared in advance, and inputting from a certain imaging means. Image conversion means for generating a plurality of converted images by converting the obtained image into a viewpoint of another imaging means; and generating the plurality of converted images and the image input from the other imaging means. Is compared for each pixel or for each small area composed of a plurality of pixels, and a matching processing means for determining the degree of coincidence of each pixel or small area; An obstacle detection device comprising: an obstacle detection unit that detects an obstacle area with respect to the obstacle. 2. The method according to claim 1, wherein the matching processing unit divides the converted image into small regions when comparing one of the converted images with an image input from the other imaging unit. A matching process is performed within a search area including the corresponding small area of the image input from the other imaging means and the periphery thereof, and the maximum matching degree obtained by searching within the search area is determined by the matching of the small area. The obstacle detection device according to claim 1, wherein the obstacle is detected in degrees. 3. The image processing apparatus according to claim 1, wherein the matching processing unit is configured to compare the small area of the converted image of the plurality of converted images with an image input from the other imaging unit. If a degree of coincidence equal to or greater than a predetermined threshold value is obtained in the search area of the image input from, at that time, all subsequent matching processing is terminated for the small area, and the degree of coincidence is determined. 3. The obstacle detection device according to claim 2, wherein: is a degree of coincidence for the small area in all the converted images. 4. The size of the search area, the pitch at which pixels are shifted during the search, or the number of types of the image conversion used to generate the converted image, according to the position of the small area on the image. The obstacle detection device according to any one of claims 2 to 4, wherein at least one of the following is changed. 5. The image processing apparatus according to claim 1, wherein the object is an automobile, the plurality of imaging units are right and left imaging units for obtaining a stereo image, and the image conversion is performed by converting a left image input from the left imaging unit. Converting the right image input from the right imaging unit into an image of the viewpoint of the left imaging unit, or converting the right image input from the right imaging unit into an image of the viewpoint of the left imaging unit. The plurality of types of conversion parameters used are determined in advance for each of the plurality of postures selected from the range of possible postures of the vehicle based on the geometric relationship between the imaging means and the road plane. The obstacle detection device according to any one of claims 1 to 4, wherein 6. An image input from a plurality of image pickup means having different viewpoints mounted on a certain object, accumulating a plurality of images respectively input by the plurality of image pickup means, a space prepared in advance, The image input from one imaging unit is used for another imaging unit by using a plurality of image transformations respectively derived for each plane from the geometric relationship between each of the plurality of planes and the imaging unit. A plurality of converted images are generated by converting the generated plurality of converted images and the image input from the other imaging means for each pixel or for each small area including a plurality of pixels. Obtaining a degree of coincidence of each pixel or small area, and detecting an area of the obtained image with a low degree of coincidence as an obstacle area for the object. .