JP2001041741A - Stereoscopic car-outside monitoring device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a stereoscopic car-outside monitoring device capable of detecting precisely the road surface state from imaged images. SOLUTION: This stereoscopic car-outside monitoring device for detecting the road surface state of a road from a pair of imaged images has cameras 1, 2 for obtaining a pair of image data by imaging the scenery outside a car, a stereoscopic image processing part 6 for calculating distance data of an object based on a parallax of the object relative to the pair of the image data, a road recognition part 10 for recognizing a three-dimensional shape of the road based on the image data and the distance data, and a fail determination part 12. The fail determination part 12 sets a distant data monitoring region in a road region recognized by the road recognition part 10, and calculates the number of the distance data existing on the lower side than the road surface position of the road as the number of wet data, among the distance data existing in the monitoring region, and detects the road surface state based on the number of the wet data.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a stereo exterior monitoring device for detecting a road surface condition of a traveling road, particularly a wet road surface, from a pair of captured images.

[0002]

2. Description of the Related Art In recent years, attention has been paid to a stereo exterior monitoring device using a pair of in-vehicle cameras (stereo cameras) incorporating a solid-state imaging device such as a CCD. This monitoring device
The traveling environment (for example, the distance between an object outside the vehicle and the vehicle) is recognized based on the pair of images captured by the on-vehicle camera. Then, if necessary, the driver is alerted, and vehicle control such as deceleration by downshifting is performed. As a method of recognizing a traveling environment, first, a positional shift (parallax) of the same object in a captured image pair is obtained by a stereo matching method. Then, from the calculated parallax, it is possible to specify the road shape (straight or curved state) of the traveling path, the inter-vehicle distance to the preceding vehicle, and the like using the principle of triangulation.

[0003] In putting such an outside monitoring device into practical use, it is important to accurately detect the road surface condition and the road shape of the traveling road in order to guarantee the safe operation of the device. There are various road surface conditions to be detected. One of them is a condition in which the road surface is wet due to rainfall or the like. When the road surface is wet, the friction coefficient μ of the road surface is significantly reduced as compared with the case where the road surface is dry. Therefore, when a braking force such as a downshift is to be exerted in a situation where the friction coefficient μ of the road surface is low, it is necessary to perform gentler braking than in the normal control so that the vehicle does not slip.

[0004] From the viewpoint of ensuring the safety of the stereo exterior monitoring device at a high level, it is important to accurately detect a wet road surface. At the same time, it is important to perform appropriate control including fail-safe depending on the road surface condition. It should be noted that “fail-safe” in the present specification is a broad concept of performing monitoring control different from normal control. Therefore, not only the normal monitoring control is temporarily interrupted, but also the control of performing the vehicle control with a gentler braking force than in the normal control is included in the concept of the “fail safe”.

[0005]

As described above, in the conventional vehicle exterior monitoring device, a method for accurately detecting the road surface condition has not been established, and the safety of the device has to be ensured at a high level. Challenges were left.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a stereo exterior monitoring device capable of accurately detecting a road surface condition from a captured image.

Another object of the present invention is to provide a stereo type external monitoring apparatus capable of performing appropriate control as necessary according to the condition of the road surface.

[0008]

SUMMARY OF THE INVENTION In order to solve the above-mentioned problems, the present invention provides a stereo exterior monitoring device for detecting the road surface condition of a road from a pair of captured images, by capturing a scene outside the vehicle. Stereo imaging means for obtaining a pair of image data; distance calculating means for calculating distance data of the target object based on the parallax of the target object in the image data pair; and tertiary of the road based on the image data and the distance data. Recognizing means for recognizing the original shape, setting means for setting a distance data monitoring area in the road area recognized by the recognizing means, and, among distance data existing in the distance data monitoring area, recognized by the recognizing means. Calculated the number of distance data existing below the road surface position of the road as the number of wet data, and detected the road surface condition based on the number of wet data. Providing a stereoscopic vehicle surroundings monitoring apparatus having a detecting means that.

Here, it is preferable that the recognizing means calculates a predicted travel line which predicts a travel route to be followed by the vehicle from the three-dimensional shape of the road. In this case, the setting means sets the distance data monitoring area by defining a range in the vehicle width direction and a range in the vehicle length direction based on the predicted traveling line.

The detecting means calculates the number of distance data existing at the road surface position of the road recognized by the recognizing means among the distance data existing in the distance data monitoring area as the number of dry data. It is preferable to determine whether or not the road surface is wet by comparing the ratio between the number and the wet data number with a determination threshold value.

Further, the detection means may change a control parameter relating to vehicle control based on the result of the determination.

[0012]

FIG. 1 is a block diagram of a vehicle exterior monitoring apparatus according to this embodiment. A pair of cameras 1 and 2 having a built-in image sensor such as a CCD are mounted near a rearview mirror at predetermined intervals in a vehicle width direction of a vehicle such as an automobile, and capture an image of a scene in front of the vehicle. The main camera 1 captures a reference image (right image) necessary for performing stereo processing, and the sub camera 2 captures a comparison image (left image) in this processing. In a state where the images are synchronized with each other, the analog images output from the cameras 1 and 2 are converted into digital images of a predetermined luminance gradation (for example, 256 gray scales) by the A / D converters 3 and 4. Is converted. The digitized image is subjected to luminance correction, geometric conversion of the image, and the like in the image correction unit 5. Usually, since the mounting positions of the pair of cameras 1 and 2 have an error to some extent, a shift caused by the error exists in the left and right images. In order to correct this displacement, geometric transformation such as rotation or translation of the image is performed using affine transformation or the like. The reference image and the comparative image corrected in this way are stored in the original image memory 8.

On the other hand, the stereo image processing unit 6 calculates the three-dimensional position (including the distance from the vehicle to the object) of the same object in the image from the reference image and the comparison image corrected by the image correction unit 5. calculate. This distance is calculated in pixel block units of 4 × 4 pixels, and can be calculated based on the principle of triangulation from the relative displacement of the position of the same object (pixel block) in the left and right images. The distance information of the image thus calculated is stored in the distance data memory 7.

The microcomputer 9 recognizes the road shape (straight line or curve curvature) ahead of the vehicle including white line detection based on the information stored in the original image memory 8 and the distance data memory 7 (road recognition unit 10). ). In addition, a three-dimensional object (running vehicle) ahead of the vehicle is also recognized (the three-dimensional object recognition unit 1)
1). When it is determined from the information from the recognition units 10 and 11 that an alarm is required, the processing unit 13 alerts the driver by an alarm device 19 such as a monitor or a speaker, or performs various control as needed. It controls the units 14-18. For example, it instructs an AT (automatic transmission) control unit 14 to execute downshifting. Further, it may instruct the engine control unit 18 to reduce the engine output. In addition, a vehicle suitable for the anti-lock brake system (ABS) control unit 15, the traction control system (TCS) control unit 16, or the vehicle behavior control unit 17 for controlling the torque distribution and the rotation speed of each wheel. It is also possible to instruct control.

Further, the failure determination unit 12 determines the failure based on the image data stored in the original image memory 8, the distance data stored in the distance data memory 7, and the predicted travel line L calculated by the road recognition unit 10. Fail determination is performed according to a routine described later. When fail safe is instructed from the fail determination unit 12, the processing unit 13 executes fail safe. As a result, for example, the above-described vehicle control or the like is temporarily interrupted in order to prevent a malfunction of the device due to erroneous recognition of a road or the like. Further, the control units 14 to 1 are controlled so that the vehicle braking force is gentler than that in the normal control.
8 are changed.

FIG. 2 is a flowchart showing a failure determination routine according to the present embodiment. This flowchart is repeatedly executed for each predetermined control cycle.
When it is determined by this routine that the current traveling environment matches the fail condition, the fail determining unit 12 instructs the processing unit 13 to execute fail safe.

First, in step 1, the failure determining unit 12 determines whether the image data stored in the original image memory 8 is
The distance data stored in the distance data memory 7 and the predicted travel line L calculated by the road recognition unit 10 are read. The predicted travel line L is a line that predicts a travel route to be followed by the vehicle, and is calculated based on the detected white line. The predicted travel line L is calculated from the recognized road shape as shown in FIG. That is, when the parallax d of an object (for example, a road or a preceding car) projected on the image pair obtained by the stereo cameras 1 and 2 is calculated, the distance Z to the object is calculated based on the following equation. Can be calculated.
Here, r is an attachment interval between the stereo cameras 1 and 2, and f is a focal length of the stereo cameras 1 and 2.

[0018]

## EQU1 ## Z = (rf) / d

The coordinate system of the captured image is such that the origin is the lower left corner of the captured image, the horizontal direction is the i coordinate axis, and the vertical direction is j.
Coordinate axes (unit is pixel). On the other hand, the coordinate system of the real space set based on the position of the vehicle is a stereo camera 1,
With the road surface just below the center of 2 as the origin, the vehicle width direction is defined as the X axis (right direction is positive), the vehicle height direction is defined as the Y axis (upward direction is positive), and the vehicle length direction is defined as the Z axis (forward direction is positive). I do. Therefore, if the traveling road is horizontal, the XZ plane (Y = 0) matches the road surface.
Distance data (i, j,
When Z) is specified, its three-dimensional position (X, Y, Z) can be uniquely specified according to the following coordinate conversion formula. Here, in the same formula, CH is the height (constant) at which the stereo cameras 1 and 2 are attached, r is the attachment interval (constant) between the stereo cameras 1 and 2, PW is the viewing angle per pixel (constant), and , IV, and JV are the values (constants) of the i or j coordinates on the image of the point at infinity in front of the vehicle.

[0020]

X = r / 2 + Z · PW · (i-IV) Y = CH−Z · PW · (j−JV)

The three-dimensional coordinates (X, Y, X) of the real space
Is converted into the two-dimensional coordinates (i, j) of the captured image as follows.

[0022]

I = (X−r / 2) / (Z · PW) + IV j = (CH−Y) / (Z · PW) + JV

When the road and the preceding vehicle are displayed overlapping each other, the road recognizing unit 10 separates and extracts only the white line using the position information (height Y) of the preceding vehicle and the white line. Perform "recognition of road shape". In the three-dimensional space, the white line is at the road surface position (Y = 0 in an ideal state), and a three-dimensional object such as a preceding vehicle is at a position higher than the road surface (Y> 0 in an ideal state). Therefore, if the height of the road surface, that is, the value of the Y coordinate can be specified, the white line and the three-dimensional object can be distinguished. Here, “recognition of road shape” means that each parameter of a function (road model) expressing the road shape is
The setting is to match the three-dimensional road shape. The road model according to the present embodiment approximates left and right white lines (side lines) in a recognition range (for example, from the camera position to a point 84 m ahead of the vehicle) by a three-dimensional linear equation for each predetermined section, and then approximates the broken lines. It is expressed by connecting in a shape. In the example of FIG. 3, the recognition range is divided into seven sections for each predetermined distance (Z1 to Z7), and in each section, the parameters a, b, c, and d of the linear equation approximated by the following equation are used. It is derived for each side line of the road. Then, an area sandwiched between the left and right side lines is a traveling path. Since this road model has not only a horizontal (X direction) but also a vertical (Y direction) linear formula, it expresses a vertical road shape such as a gradient or undulation of the road. .

[0024]

(Left side line) X = a _L · Z + b _L Y = c _L · Z + d _L (right side line) X = a _R · Z + b _R Y = c _R · Z + d _R

The predicted running line L can be calculated from the left and right side lines thus calculated. The predicted travel line L can be obtained as the center line of the travel path, that is, the intermediate line between the left and right side lines approximated by the above equation (4). The predicted running line L calculated in this way is output to the failure determination unit 12.

In step 2 following step 1, a distance data monitoring region R is set. This monitoring area R is
This is a target range for counting the number of output distance data. FIG. 4 is a diagram for explaining the distance data monitoring region R. This monitoring area R is used to suppress an increase in the amount of calculation.
Based on the predicted travel line L read in step 1,
It is set as 40 m (0 ≦ Z ≦ 40) in the forward direction of the own vehicle, and 2 m (−2 ≦ X ≦ 2) in both left and right widths. FIG. 4 shows a predicted travel line L extending linearly in the Z direction.
, The predicted travel line L has a displacement amount in the X direction in the curve.

Next, in step 3, effective distance data is specified from the distance data existing in the distance data monitoring region R. The effective distance data refers to distance data relating to a pixel block having a predetermined number or more of luminance edges in the horizontal direction (horizontal direction) of an image. FIG. 6 is a diagram for explaining a method of calculating a luminance edge in the horizontal direction for a pixel block that is a unit for calculating distance data. As described above, one distance data is calculated for each 4 × 4 pixel block. In this pixel block, first, the luminance change amount (absolute value) ΔP of a pair of pixels adjacent in the horizontal direction is calculated. However, for the leftmost pixel column _{(P 11, P 12, P} 13, P 14), calculates the luminance variation ΔP from the rightmost pixel column in the pixel blocks adjacent to the left side. Therefore, for one pixel block, 16 luminance change amounts Δ
P is calculated. Next, these 16 luminance change amounts Δ
Among the P, the number of Ps that are equal to or more than a predetermined threshold (hereinafter, referred to as DCDX threshold) is counted. This DCDX threshold is a value appropriately set in the range of 3 to 7. Then, distance data relating to a pixel block in which the number of luminance change amounts ΔP equal to or greater than the DCDX threshold value is four or more is defined as effective distance data. The evaluation of the road surface condition is not performed based on all the distance data existing in the monitoring area R, but is performed only on the effective distance data related to the pixel block having the characteristic regarding the luminance change in the horizontal direction.
Accordingly, noise-like distance data calculated due to unevenness of the road surface or the like can be excluded, so that the accuracy of the evaluation of the road surface condition can be improved.

In subsequent steps 4 and 5, the number of dry data DRY and the number of wet data WET are counted from the effective distance data in the distance data monitoring area R calculated in step 3 above. FIG.
FIG. 6 is a diagram for explaining distribution of distance data in a distance data monitoring region R. In the figure, the displacement of the predicted traveling line L in the height direction (Y direction) is shown normalized on the Z axis. From the above-described conversion formula of Expression 2, the distance data (i, j, Z) is represented by the three-dimensional coordinates (X, Y, Z) of the real space.
And both are in a one-to-one relationship. Therefore, the distance data is classified into the following three based on the height (the value of Y).

(1) Three-dimensional object data Three-dimensional object data is distance data calculated based on a three-dimensional object such as a preceding vehicle. In an ideal state, when the distance data relating to the three-dimensional object is converted by the above equation 2, Y> 0 (that is, above the road surface). However, in an actual driving environment, due to pitching of the vehicle and undulation of the road surface,
Distance data such as a white line (ideally, Y = 0) may satisfy Y> 0. Therefore, considering such a point, Y>
The distance data of 0.3 (the unit is m) is set as three-dimensional object data.

(2) Dry data Dry data is obtained from the road surface (white line, road rut,
This is distance data calculated due to gravel or the like. In an ideal state, if the distance data relating to the white line and the like is converted by Expression 2, Y = 0 (that is, the road surface). However, as described above, in an actual driving environment, such distance data may have a positive or negative value in the height direction. And

(3) Wet Data On a road surface that is wet due to rain or the like, a three-dimensional object may be reflected on the road surface, and distance data resulting therefrom is calculated. The false distance data caused by the reflection is Y <0 in an ideal state, but in consideration of the actual traveling environment, the distance data of Y <−0.4 is used as the wet data.

Therefore, the dry data number DRY calculated in step 4 is the number of effective distance data that satisfies the following condition based on the predicted travel line L.

(Dry Data Conditions) -2≤X≤2-0.4≤Y≤0.30≤Z≤40

The wet data number WET calculated in step 5 is the number of effective distance data that satisfies the following condition based on the predicted travel line L.

(Condition of Wet Data) -2≤X≤2 Y≤-0.40≤Z≤40

Then, from the number of dry data DRY and the number of wet data WET calculated in steps 4 and 5,
The wet data rate RT is calculated according to the following equation (step 6). The wet data rate RT indicates the degree to which a three-dimensional object is reflected on a road surface. The more three-dimensional objects reflected on a wet road surface, the higher the wet data rate RT.

[0037]

## EQU5 ## RT = WET / DRY

Then, it is determined whether or not the wet data rate RT calculated in step 6 is larger than an appropriately set determination threshold value RTth (step 7). If an affirmative determination is made in step 7, that is, if a lot of three-dimensional objects are reflected on the road surface, it is determined that the road surface is wet, and the fail flag NG is set to 1 (step 8). During a period in which the fail flag NG is set to 1, the fail determining unit 12 instructs the processing unit 13 to execute fail safe. As a result, the normal monitoring control is stopped, and various control parameters related to vehicle control are changed as a fail-safe so that, for example, a braking force that is gentler than that during the normal control is exerted. When the wiper is operating,
It may be determined that it is rainfall, and vehicle control (for example, setting of a gentler braking force) taking this into consideration may be performed. On the other hand, if a negative determination is made in step 7, that is, if there are few three-dimensional objects reflected on the road surface, it is determined that the road is dry and the fail flag NG is set to 0 (step 9). While the fail flag NG is set to 0, normal monitoring control is continued.

As is clear from the above description, the stereo type external monitoring apparatus according to the present embodiment sets the distance data monitoring area R in front of the own vehicle where a road is predicted to exist, and monitors this area. With respect to the distance data existing in the region R, the distribution in the height direction is evaluated. That is, the height (value in the Y direction) of the calculated distance data in the three-dimensional space is obtained, and the number of distance data due to the three-dimensional object being reflected on the road surface is counted as the number of wet data. On a wet road surface, compared to a dry road surface, there is a characteristic that a larger amount of distance data is calculated below the road surface position. In consideration of such characteristics, when a large amount of wet data exists in the distance data monitoring region R, it is determined that the vehicle is in a wet road surface condition. This allows
The reliability of outside monitoring and the safety of control can be ensured at a high level. As a result of the inventor actually repeating the running test, it was confirmed that a good judgment result can be obtained by the above judgment procedure.

From the viewpoint of ensuring both the reliability of monitoring outside the vehicle and the stability of control, the wet data rate R
If the situation where T becomes larger than the determination threshold value RTth continues over a plurality of control cycles (for example, equivalent to 10 seconds),
It is preferable to set the fail flag NG to 1.
From a similar viewpoint, if the situation where the wet data rate RT becomes equal to or less than the determination threshold value RTth continues over a plurality of control cycles (for example, corresponding to 20 seconds), the fail flag NG is set to 0.

Further, in order to improve the detection accuracy of the failure situation, the above-described failure determination routine may be executed according to the traveling situation. For example, the failure determination is performed only when the inter-vehicle distance between the own vehicle and the preceding vehicle is 13 m or more, the vehicle speed is 20 km / h or more, and the steering angle is ± 23 ° or less. If the inter-vehicle distance with the preceding vehicle is short, even if the reflection on the wet road surface occurs, most of it is masked by the preceding vehicle, making it difficult to accurately detect such a situation. . Further, in particular, when a method of calculating the wet data rate RT for each frame and performing the evaluation at a predetermined frame interval is used, if the vehicle speed is low, there is a possibility that the fail determination accuracy may be reduced. When the vehicle speed is low, the moving amount of the own vehicle is small, so that only a local road surface is to be monitored. If the local road surface is in an unpredicted state that accidentally matches the failure situation, an inappropriate failure determination may be made. If the high vehicle speed is used as a condition for executing the failure determination, it is possible to avoid erroneously determining a failure even if such a road surface exists locally. Further, the reason for taking the steering angle into consideration is to improve the determination accuracy by limiting the pattern of the scenery in the field of view of the camera to some extent.

Further, the distance data for judging the road surface condition targets only distance data relating to a pixel block having a large horizontal luminance edge. Therefore, erroneous distance data calculated due to noise or the like can be excluded, and a wet road surface condition can be detected with higher accuracy.

In the above embodiment, the condition of the road surface is determined based on the ratio of the number of dry data to the number of wet data. However, the present invention is not limited to this, and can be widely applied to a method of determining a road surface condition based on the number of wet data. For example, a method of comparing the number of wet data with a predetermined threshold value without counting the number of dry data may be used.

[0044]

As described above, according to the present invention, the road surface is determined based on the number of distance data (the number of wet data) existing below the road surface position among the distance data existing in the distance data monitoring area. It is determined whether or not it is wet. Thereby, the condition of the road surface can be accurately detected from the captured image, and appropriate vehicle control can be performed according to the condition of the road surface. As a result, it is possible to ensure the safety of the vehicle exterior monitoring device at a higher level.

[Brief description of the drawings]

FIG. 1 is a block diagram of a vehicle exterior monitoring device according to an embodiment;

FIG. 2 is a flowchart illustrating a fail determination routine according to the embodiment;

FIG. 3 is a diagram for explaining a predicted traveling line;

FIG. 4 is a diagram for explaining a distance data monitoring area;

FIG. 5 is a diagram for explaining distribution of distance data in a distance data monitoring area.

FIG. 6 is a diagram for explaining a method of calculating a luminance edge in the horizontal direction.

[Explanation of symbols]

1 main camera, 2 sub camera, 3,
4 A / D converter, 5 image correction unit, 6 stereo image processing unit, 7 distance data memory, 8 source image memory, 9 microcomputer,
10 road recognition unit, 11 three-dimensional object recognition unit,
12 failure judgment unit, 13 processing unit, 14
AT control unit, 15 ABS control unit, 16
TCS control unit, 17 Vehicle behavior control, 18
Engine control unit, 19 alarm device

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 2F065 AA04 AA06 BB05 BB15 DD03 FF05 FF24 JJ03 JJ26 QQ03 QQ21 QQ24 QQ25 QQ32 SS09 SS13 2F112 AC03 AC06 BA06 CA05 CA12 FA03 FA09 FA21 5B057 AA16 AA19 CC03 DA07 DC03 DB03

Claims (4)

[Claims] 1. A stereo-type outside monitoring device for detecting a road surface condition of a road from a pair of captured images, a stereo imaging unit for obtaining a pair of image data by capturing a scene outside the vehicle, A distance calculating unit that calculates distance data of the target object based on the parallax of the target object; a recognition unit that recognizes a three-dimensional shape of the road based on the image data and the distance data; Setting means for setting a distance data monitoring area in the road area recognized by the means; and a part of the distance data present in the distance data monitoring area below the road surface position of the road recognized by the recognition means. Detecting means for calculating the number of the distance data existing in the area as the number of wet data and detecting a road surface condition based on the number of wet data. Stereo type vehicle surroundings monitoring apparatus according to claim Rukoto. 2. The method according to claim 1, wherein the recognizing means calculates a predicted travel line that predicts a travel route to be followed by the vehicle based on a three-dimensional shape of the road, and the setting means determines a vehicle width based on the predicted travel line. The stereo type outside monitoring apparatus according to claim 1, wherein the distance data monitoring area is set by defining a range in a direction and a range in a vehicle length direction. 3. The method according to claim 1, wherein the detecting means calculates the number of the distance data existing at the road surface position of the road recognized by the recognizing means as the number of dry data among the distance data existing in the distance data monitoring area. And the ratio between the number of dry data and the number of wet data is
The stereo type outside monitoring apparatus according to claim 1, wherein it is determined whether or not the road surface is wet by comparing with a determination threshold value. 4. A stereo outside monitoring apparatus according to claim 3, wherein said detecting means changes control parameters relating to vehicle control based on the determination result.