JP2001076128A - Device and method for detecting obstacle - Google Patents

Abstract

PROBLEM TO BE SOLVED: To fast detect an obstacle existing on a road surface with high accuracy even though stereo cameras are not corrected and even under a situation in which there are changes in vibrations during traveling and in slopes on a road surface by using the stereo cameras on a vehicle. SOLUTION: This obstacle detecting device consists of a plurality of uncorrected TV cameras inputting images, an image storing part 2 for storing a plurality of images inputted by the plurality of TV cameras, a characteristic extracting part 3 extracting a plurality of mutually parallel lines existing on a road surface, a parameter calculating part 4 calculating a relational expression established among projected positions to each image of an optional point on the road surface from the plurality of lines extracted by the part 3, and a detecting part 5 which uses the relational expression calculated by the part 4 and detects an object having height from the road surface.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an obstacle detecting device for detecting an obstacle existing on a road, such as a preceding vehicle, a pedestrian, a parked vehicle, or the like, using an on-vehicle camera to assist safe driving of an automobile.

[0002]

2. Description of the Related Art Techniques for detecting an obstacle by a sensor are roughly classified into those using a laser or an ultrasonic wave, and those using a TV camera. Those using lasers are expensive and impractical. Further, those using ultrasonic waves have a low resolution of the ultrasonic waves, and thus have a problem in the detection accuracy of obstacles.

On the other hand, a TV camera is relatively inexpensive and is suitable for detecting an obstacle in terms of resolution, measurement accuracy, and measurement range. When a TV camera is used, there are a method using one camera and a method using a plurality of cameras (stereo cameras).

In a method using one camera, a road area and an obstacle area are separated from one image captured by the camera by using information such as luminance, color, and texture. For example, in the image, a low-saturation medium luminance area, that is, a gray area is extracted to obtain a road area,
In order to find an area with less texture, extract the road area,
The other area is an obstacle area. However, since there are many obstacles having luminance, color, or texture similar to a road, it is difficult to separate an obstacle region from a road region in a general situation by this method.

On the other hand, the method using a plurality of cameras detects an obstacle using three-dimensional information as a clue. A technique for obtaining three-dimensional information of a target scene using a plurality of cameras is generally called stereo vision. Stereo vision refers to, for example, arranging two cameras on the left and right, associating a point that is the same point in a three-dimensional space between left and right images, and finding a three-dimensional position of the point in the manner of triangulation. . If the position, posture, and the like of each camera with respect to the road plane are obtained in advance, the height of an arbitrary point in the image from the road plane can be obtained by stereo vision. Therefore, the obstacle area and the road area can be separated depending on the presence or absence of the height. 1
It is difficult to detect an area having brightness, color, and texture similar to a road as an obstacle by using a single camera, but according to stereo vision, the obstacle from the height from the road surface is a clue. Since an object is detected, it is possible to detect an obstacle in a more general scene.

[0006] Ordinary stereo vision is a technique for determining the distance of an arbitrary point on an image from a stereo camera. To this end, the distance and orientation of a plurality of cameras, the focal length of a camera lens, the center of the image, and the like are determined in advance. It is necessary to find parameters related to The operation for obtaining these is called calibration. In the calibration, a number of points whose three-dimensional positions are known are prepared, the projection positions of the points on the image are obtained, and parameters relating to the position and orientation of the camera, the focal length of the camera lens, and the like are calculated. However, this operation requires a great deal of time and effort, and is one of the factors that hinders practical use of obstacle detection by stereo vision.

However, if it is sufficient to separate the road area and the obstacle area on the image, the calibration is not necessarily required. That is, the projection point to the left and right image points on the road plane _{(u l,} v _l), if _{(u r, v} r),

(Equation 1)

[Outside 1] These are parameters relating to the position and orientation of each camera with respect to the road plane, the focal length of the lens of each camera, and the image origin. h is obtained in advance from four or more projected points on the left and right images on the road plane. Using this relation,
Arbitrary point P (u _{l, v l)} on the left image corresponding point P on the right image when it is assumed that exists on the road plane '(u _r,
v _r ). If point P exists on the road plane, point P
And P 'are a correct set of corresponding points, so that the difference in luminance between the two points is small. Therefore, when the difference in luminance between the points P and P 'is large, it is determined that the point P belongs to the obstacle area. Hereinafter, Equation 1 is referred to as road plane constraint.

This method has another merit that, along with calibration, it is not necessary to search for correspondence which is a problem of stereo vision. In normal stereo vision, it is necessary to associate the same point between the left and right images, and the association is performed by search calculation, so that the calculation cost is high. However, the above-described method is a practical method that requires extremely low computational cost because no corresponding search is required.

If the stereo camera is fixed in the three-dimensional space, the geometric relationship between the road plane and each camera is invariable. An object can be detected. However, when the car is running, the relationship between the road plane and the relative position and attitude of each camera changes every moment due to vibration of the car itself, changes in the inclination of the road, and the like. That is, since the parameter h changes during traveling, the road plane constraint obtained at rest cannot be used for detecting an obstacle during traveling.

To solve such a problem, a method of calculating a road plane constraint by using a large number of feature points on a road surface (corner points of paint on a road, etc.) and detecting an obstacle is known. Used. However, it is difficult to extract a large number of feature points on a road surface, and often erroneous extraction of feature points on an obstacle. Furthermore, since it is necessary to perform a correspondence search for the extracted feature points, the calculation cost is high. Further, there is a problem that it is extremely difficult to stably obtain the road plane constraint because the number of parameters to be obtained is large.

[0011]

As described above, obstacle detecting devices can be roughly classified into those using a laser or an ultrasonic wave and those using a TV camera. There were problems that it was expensive and the measurement accuracy was low. In addition, an obstacle detection device using a TV camera has a limited use environment, requires calibration that requires a great deal of time and effort, and requires a search for correspondence between left and right images at high computational costs. And there is no practical measure to cope with the vibration and the inclination of the road while the vehicle is running. Therefore, the present invention has been made in view of the above circumstances, and requires only a small amount of labor for calibration, and the movement of the two white lines at both ends of the road on the moving image on the basis of the movement of the road plane and the geometrical shape of each camera. By obtaining a relationship, an obstacle detection device for quickly detecting an obstacle existing on a road plane even if the road or the road itself is inclined during traveling is provided.

[0012]

A plurality of TV cameras for inputting images, an image storage unit for storing a plurality of images input by the plurality of TV cameras, and a line existing on a road surface are identified. A feature extraction unit to be extracted, a parameter calculation unit that obtains a relational expression that holds a projection position of an arbitrary point on a road surface onto each image from a plurality of lines extracted by the feature extraction unit, and a parameter calculation unit. It comprises a detection unit for detecting objects having different heights from the road surface using the obtained relational expression.

[0013]

Embodiments of the present invention will be described below with reference to the drawings. This embodiment uses two left and right stereo cameras mounted on a car as shown in FIG.
It is assumed that an obstacle, such as a parked vehicle, existing on a road plane is detected. FIG. 2 shows a schematic configuration of the present apparatus in the embodiment, which comprises an image input unit 1, an image storage unit 2, a feature extraction unit 3, a parameter calculation unit 4, and a detection unit 5. This device obtains in advance a relational expression (hereinafter referred to as road plane constraint) that holds between the projected positions of points on the road plane onto the left and right images when the vehicle is stationary, and calculates the relational expression based on the vibration of the vehicle, the inclination of the road, and the like. The road plane constraint that changes every moment during traveling is calculated only from the movement of the two white lines existing on the road, and is used to identify obstacles and road areas existing on the road plane.

[0014] In the embodiment, as shown in FIG. 3, the two white lines of the road ends and _l 1, _{l 2,} straight lines _l 1, _{l 2} directions Y-axis, right and left, each X in the vertical direction, Z A stereo camera coordinate system in which a plane (reference plane) having no axis and no inclination is set as an XY plane is set. The image input unit 1 inputs two images using two left and right TV cameras. It is not necessary to determine the positions and orientations of these two cameras in advance, but here, each camera is fixed to the car and does not change during traveling.

The image storage unit 2 stores two images input by the image input unit 1 in an image memory.

The feature extracting unit 3 detects _two straight lines l ₁ and l ₂ on the two images stored by the image storing unit 2 as shown in FIG. ). This straight line detection is performed using edge extraction processing and Hough transform.

The parameter calculating unit 4 calculates the road plane constraint at the time of running from the road plane constraint on the reference plane obtained at the time of stopping and the two white lines obtained by the feature extracting unit 3 and their vanishing points. Hereinafter, this method will be described.
Assuming that a point (X, Y, Z) in a three-dimensional space is a projection point (u, v) on an image, generally,

(Equation 2)

[Outside 2] These are parameters related to the mind. Since h represents the same camera model even when multiplied by a constant, generality is not lost even if an arbitrary element of h is set to "1". Therefore, in the following, h ₃₂ = 1.

In the stereo camera coordinate system shown in FIG. 3, since the road plane (reference plane) is represented by Z = 0, the projection point of the point P (X, Y) on the road plane is represented by Z = 0 in the above equation. And assign

(Equation 3) Becomes Here, a camera model is considered under the following assumptions. (A) The target area is relatively far from the camera. (B) The positional deviation between the left and right cameras before and after is very small.

Under these assumptions,

(Equation 4) Becomes Here, β is a shift in the Y direction between the middle point of the viewpoint of the left and right cameras and the coordinate origin as shown in FIG. 5, and t ₃ = β +
Δt ₃ . Therefore, equation (4) becomes

(Equation 5) It can be simplified. Further, if Yc = Y + β,

(Equation 6)

[Outside 3] _Let u _{l and} ur be the projection points of

(Equation 7) To become _(t l, t _r is the vanishing point of the white line). Than this,

(Equation 8) Becomes Here, “l” and “r” are subscripts for the left and right images, respectively. Since the stereo camera has not been calibrated, M ₁ and M _r are unknown, but A is obtained in advance from feature points on a road plane that is not inclined when stationary.

It is assumed that the road plane changes from the reference plane Z = 0 to the inclined plane Z = pY due to a change in the inclination of the road during traveling or the vibration of the own vehicle (FIG. 4). In general, the gradient in the X direction can be ignored because it is sufficiently smaller than the gradient in the Y direction. If the intersection of the inclined surface and the reference surface is taken as the X axis, the equation of the inclined surface can be expressed as Z = pY. A method for calculating a road plane constraint for Z = pY from the movement of two white lines (FIG. 7) will be described. The projection position (u ′, v ′) of the point (X, Y, Z) on the image on the inclined plane is obtained by substituting Z = pY into the equation (2), under the above two assumptions.

(Equation 9)

[Outside 4]

(Equation 10) By setting Yc = Y + β and performing the same equation modification on v ′ from equation (3),

[Equation 11]

[Outside 5] So the above formula is

(Equation 12)

[Outside 6]

(Equation 13) And

[Equation 14]

[Outside 7]

(Equation 15) Becomes Here, β ₁ = m ₂₁ β and β ₂ = m ₂₂ β. As shown in FIG. 7, one side of the image

[Outside 8]

(Equation 16) Get. When the equation transformation is similarly performed for another white line (l ₂ → l ′ ₂ ),

[Equation 17]

[Outside 9] The matrix K in (16) can be obtained. When the above processing is performed on each of the left and right images,

[Outside 10] It is. Therefore, using equation (9),

(Equation 18) Becomes A of formula (9), A the slope _'= K r AK _l
^This means that it has changed to ^-1 . Equation (19) is the road plane constraint on the slope.

The detection unit 5 detects an obstacle using the road plane constraint obtained by the parameter calculation unit 4. Arbitrary point _{(u l, v} l) of the left image luminance _{I L (u l, v l} ) and then, the point (u, v) is the corresponding point on the right image when it is assumed that exists on the road plane _(u r, _{v r)} calculated from equation (19), and the luminance _{I R (u r, v r} ) and. The point (u,
If v) actually exists on the road plane, the points P and P 'are a correct set of corresponding points, so that the luminances of the points P and P' are basically the same. That is,

[Equation 19] In consideration of D あ る い は 0 or an error, D> Thr
(Thr is a threshold value set in advance) The point P is determined to belong to the obstacle area.

As described above, an obstacle on a road plane can be detected from a stereo camera mounted on a vehicle during traveling.

In this embodiment, the image input unit 1 inputs two images by arranging two TV cameras side by side, but these cameras may be arranged vertically. Further, three or more cameras may be arranged.

The case where the feature extracting unit 3 extracts two lines on the road plane has been described. However, the feature extracting unit 3 may extract three or more lines.

If it is sufficient to consider only the vibration of the own vehicle during traveling, β = 0 (β ₁ = β ₂ =
0), the matrix on the right side of the equation is K = I (I
Is a unit matrix), and in equation (19), A ′ = A. Thereby, the road plane constraint on the inclined surface can be obtained even faster.

Further, the detecting section 5 can further have a configuration shown in FIG. Here, it is configured by an image conversion unit 5-1 and a difference calculation unit 5-2. The image conversion unit 5-1 includes:
The right image is converted according to the following procedure. Generally, an image can be expressed as a function f (u, v) in which points (u, v) on the image are variables and a luminance value is defined for each point. Hereinafter, an image is represented in this manner.

Assuming that a stereo image as shown in FIG. 9 has been input, the right image is g (u, v), and the converted image is g '(u, v). G ′ (u, v) is obtained as follows.

(Equation 20) Here, (u ′, v ′) is obtained from Expression 19. g '
(U, v) is an image obtained by the left camera, assuming that an arbitrary point on the image g (u, v) exists on the road plane. For example, a converted image as shown in FIG. 10 is obtained from the right image in FIG. As shown in FIG. 11, the projected point of the point on the road plane is the same in the left image and the converted image, but on the point not on the road plane, that is, on the obstacle (in this case, the preceding vehicle). Are projected at different positions depending on the height from the road. Therefore, an obstacle on the road plane is detected by taking the difference between the left image and the converted image. That is, if the left image is f (u, v),

(Equation 21) D ′ ≠ 0 or, considering the error, D ′> Th
r (Thr is a preset threshold) (u,
v) is determined to belong to the obstacle area.

The detection unit 5 detects the difference between the two images by calculating the difference between the pixels. However, the detection unit 5 sets a (2w + 1) × (2w + 1) window for each point, and sets the luminance value in the window. The difference may be detected by calculating the normalized cross-correlation C of. Two images F (u, v), G (u,
C at point (u, v) in v) is

(Equation 22) Here, N = (2w + 1) × (2w + 1), a ₁ , a ₂
Is the average brightness in the window of the two images,

[Outside 11] It is determined that the point (u, v) serving as the threshold value set in advance belongs to the obstacle area.

In this embodiment, two white lines at both ends of the road are extracted as straight lines. However, when the road is curved, the white lines are curved. In this case, if the white line is extracted as a curve, an obstacle can be detected in the same manner.

Although the description has been made on the assumption that the road surface is a plane, an obstacle can be detected even in the case of a curved surface as in the case of a plane.

Although the present embodiment has been described with reference to the detection of an obstacle from an on-vehicle camera, the present invention can be applied to, for example, autonomous traveling of a mobile robot. It is not limited.

In addition, modifications can be made without departing from the spirit of the present invention.

[0033]

According to the present invention, an obstacle is detected based on the presence or absence of a height from a road plane. Therefore, it is possible to detect an obstacle such as a preceding vehicle or a pedestrian from an image without being affected by a change in brightness or a shadow. it can. In addition, since the constraint equation, which is based on the geometric relationship between the road plane and each camera, is obtained only from the two white lines at both ends of the road, even if the road is vibrating or inclined on the road plane, the road can be driven at high speed. An obstacle on a plane can be detected, and the practical effect is great.

[Brief description of the drawings]

FIG. 1 is an example of an embodiment of the present invention.

FIG. 2 is an overall configuration of the present invention.

FIG. 3 is a diagram for explaining a stereo camera coordinate system.

FIG. 4 is a diagram illustrating an inclined surface.

FIG. 5 is a diagram for explaining a parameter β.

FIG. 6 is a diagram for explaining two white lines at both ends of a road and vanishing points thereof.

FIG. 7 is a diagram for explaining changes during traveling of two white lines at both ends of a road.

FIG. 8 is a modified example of the detection unit 5.

FIG. 9 is a stereo image.

FIG. 10 is a diagram for explaining a right image and its converted image.

FIG. 11 is a view for explaining a converted image of a left image and a right image.

[Explanation of symbols]

 1, image input unit 2, image storage unit 3, feature extraction unit 4, parameter calculation unit 5, detection unit

 ──────────────────────────────────────────────────続 き Continued from the front page F term (reference)

Claims (6)

[Claims] 1. A plurality of TV cameras for inputting images,
An image storage unit for storing a plurality of images input by a plurality of TV cameras, a feature extraction unit for extracting a line existing on a certain surface in a three-dimensional space from the image, and an extraction by the feature extraction unit A parameter calculation unit that obtains a relational expression that holds between the projection positions of the arbitrary points on the surface on the plurality of images from the obtained line, and a relational expression calculated by the parameter calculation unit. An obstacle detection device, comprising: a detection unit that detects an area not existing above. 2. The obstacle according to claim 1, wherein the plurality of TV cameras according to claim 1 have unknown positions, postures, focal lengths, and image centers of the cameras. Detection device. 3. The parameter calculation unit according to claim 1,
A relational expression that holds between the projection positions of an arbitrary point on a certain surface in a three-dimensional space onto a plurality of images is represented by two-dimensional affine transformation, and the affine transformation parameter is obtained. The obstacle detection device according to claim 1. 4. The feature extraction unit according to claim 1, which is present on a certain surface in a three-dimensional space, and in a direction substantially the same as that of the plurality of TV cameras in the three-dimensional space. 4. The method according to claim 1, wherein a plurality of lines parallel to each other are extracted from the image, and vanishing points of the lines are obtained.
The obstacle detection device according to any one of the above. 5. The feature extraction unit according to claim 1, which exists on a certain surface in a three-dimensional space, and which is substantially in the same direction as the plurality of TV cameras according to claim 1, and in the three-dimensional space. The method according to claim 1, wherein a plurality of lines parallel to each other are extracted from the image, and inclinations of the lines on the image and vanishing points are obtained.
The obstacle detection device according to claim 2. 6. A method for accumulating a plurality of images input by a plurality of TV cameras, extracting a line existing on a certain surface in a three-dimensional space from these images, and extracting a line extracted by a feature extracting unit. Finding a relational expression that holds between the projection positions of the arbitrary points on the surface onto the plurality of images, and detecting an area that does not exist on the surface using the obtained relational expression. Obstacle detection method.