JP3649327B2 - Stereo exterior monitoring system with fail-safe function - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a stereo-type vehicle exterior monitoring device that performs fail-safe when an abnormality occurs in an image pair captured by a stereo camera.
[0002]
[Prior art]
2. Description of the Related Art In recent years, a stereo-type outside vehicle monitoring apparatus using an in-vehicle camera incorporating a solid-state imaging device such as a CCD has been attracting attention. This apparatus obtains a positional shift, that is, a parallax, of a target object from a pair of images captured by left and right cameras (stereo cameras), and uses the principle of triangulation to measure the distance to the target object (distance information). ) Is calculated. Based on the calculated distance, the vehicle recognizes the driving environment (for example, the distance between the object outside the vehicle and the vehicle), and alerts the driver or downshifts as necessary. Vehicle control is performed.
[0003]
In putting such a stereo-type vehicle exterior monitoring device into practical use, it is necessary to provide a fail-safe function in order to ensure safe operation of the device. One of the failures to be detected by this type of apparatus is a situation in which an imbalance due to an external factor has occurred with respect to the brightness of an image pair (stereo image) captured by a stereo camera. In other words, in order to ensure accuracy related to the calculation of the distance by the stereo method and the recognition of the front three-dimensional object, the brightness of the left and right images is originally balanced, that is, the overall brightness of the image pair is substantially the same It is set to such a state. For this reason, if the brightness balance of the image pair is shifted due to some external cause, normal outside monitoring cannot be performed. As an external cause that causes such an abnormal state, for example, the field of view of one of the cameras constituting the stereo camera may be blinded. It can also occur when the optical system (especially the lens) of one camera is dirty or cloudy. Further, when a front scene is imaged through the windshield, the windshield may be partially soiled or clouded, or may be caused by irregular reflection due to raindrops. From the viewpoint of ensuring the safety of stereo-type vehicle exterior monitoring devices at a high level, fail-safe that temporarily interrupts monitoring control in situations where brightness imbalance occurs with respect to the left and right image pairs. It is necessary to make the function work.
[0004]
[Problems to be solved by the invention]
Conventionally, the fail-safe function that is indispensable for the practical application of stereo-type vehicle exterior monitoring devices has not been established, and there remains a problem to ensure the safety of the device at a high level.
[0005]
Accordingly, an object of the present invention is to reliably detect fail-safe situations by accurately detecting stereo situations where brightness imbalance occurs and appropriate stereo processing cannot be performed. It is to provide a stereo type outside vehicle monitoring device capable of performing the above.
[0006]
[Means for Solving the Problems]
In order to solve such a problem, the present invention provides a stereo imaging unit that obtains a pair of captured images by imaging a scene outside the vehicle in a stereo-type vehicle exterior monitoring device that performs fail-safe when it is determined to be a failure. The overall luminance level of the first monitoring area set in one image area imaged by the stereo imaging means and the second monitoring set in the other image area imaged by the stereo imaging means. Calculating means for determining the overall brightness level of the area, and determination means for determining a failure when the brightness level in the first monitoring area is different from the brightness level in the second monitoring area. ing. Then, the second monitoring area in the other image area is set in correspondence with the first monitoring area in one image area.
[0007]
Here, it is preferable that the position of the second monitoring area in the other image area is offset in the stereo matching direction with respect to the position of the first monitoring area in the one image area. As a result, since the scenes projected on the first and second monitoring areas are substantially the same, the overall luminance of both in a normal state is approximately the same.
[0008]
Further, each of the first monitoring area and the second monitoring area may be composed of a plurality of monitoring areas. That is, the first monitoring area is set at a different position in one image area. In addition, the second monitoring area is set at a different position in the other image area, and each of the second monitoring areas includes a plurality of monitoring areas corresponding to the monitoring areas constituting the first monitoring area. In this case, it is preferable that the offset amount of each monitoring area is different. As a result, the offset amount of each monitoring area can be determined according to the parallax of the three-dimensional object that is highly likely to be projected in each monitoring area, so that the same regarding the magnitude of the overall luminance in the normal state can be obtained. Can be increased.
[0009]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 is a block diagram of a stereo type vehicle exterior monitoring apparatus according to the present embodiment. A pair of cameras 1 and 2 incorporating an image sensor such as a CCD are mounted in the vicinity of the rearview mirror at predetermined intervals in the vehicle width direction of a vehicle such as an automobile. Take an image. The main camera 1 arranged on the right side in the traveling direction of the vehicle picks up a reference image (right image) necessary for performing stereo processing, and the sub camera 2 arranged on the left side compares images in this processing. (Left image) is captured. In a state of being synchronized with each other, each analog image output from the cameras 1 and 2 is converted into a digital image of a predetermined luminance gradation (for example, 256 gradation gray scale) by the A / D converters 3 and 4. Converted. The digitized image is subjected to brightness correction, image geometric conversion, and the like in the image correction unit 5. Usually, there is an error in the mounting position of the pair of cameras 1 and 2 to some extent, and a shift caused by the difference exists in the left and right images. In order to correct this deviation, geometrical transformation such as image rotation or translation is performed using affine transformation or the like. The reference image and the comparison image corrected in this way are stored in the original image memory 8.
[0010]
On the other hand, the stereo image processing unit 6 determines the three-dimensional position (including the distance from the host vehicle to the target object) of a certain object displayed in the image from the reference image and the comparison image corrected by the image correction unit 5. calculate. This distance can be calculated based on the principle of triangulation from the relative shift (parallax) regarding the position of the same object in the left and right images. The distance information of the image calculated in this way is stored in the distance data memory 7.
[0011]
The microcomputer 9 recognizes a road condition or the like in front of the vehicle based on each information stored in the original image memory 8 and the distance data memory 7 (road recognition unit 10), or a three-dimensional object (running vehicle) in front of the vehicle. Recognize (three-dimensional object recognition unit 11). Then, when it is determined from the information from the recognition units 10 and 11 that an alarm is necessary, the processing unit 13 alerts the driver with an alarm device 19 such as a monitor or a speaker, or as necessary. Then, the various control units 14 to 18 are controlled. For example, an AT (automatic transmission) control unit 14 is instructed to execute a downshift. Further, the engine control unit 18 may be instructed to reduce the engine output. In addition, an appropriate vehicle for the anti-lock brake system (ABS) control unit 15, the traction control system (TCS) control unit 16, or the vehicle behavior control unit 17 that controls the torque distribution and the rotation speed of each wheel. It is also possible to instruct control.
[0012]
Further, the fail determination unit 12 performs fail determination according to a routine described later based on the original image information stored in the original image memory 8. The vehicle control described above is performed in order to prevent malfunction of the device due to erroneous recognition of roads and three-dimensional objects during a period in which a failure is determined, that is, a period in which the failure flag NG is set to 1 in the failure determination routine described later. Etc. are temporarily interrupted.
[0013]
FIG. 2 is a flowchart showing a failure determination routine based on brightness imbalance. This flowchart is repeatedly executed every predetermined control cycle (for example, 100 ms). When the fail flag NG is set to 1 by this routine, the fail determination unit 12 instructs the processing unit 13 to do so, and the processing unit 13 executes fail-safe of the vehicle exterior monitoring device.
[0014]
First, in step 1, in order to examine the luminance state of the stereo image pair (reference image and comparative image), the luminance value of each pixel in the monitoring area is read (step 1). FIG. 3 is a diagram for explaining a monitoring area set in an image area of a stereo image pair. The right image, which is the reference image, has a size of 200 × 512 pixels (vertical and horizontal) as an example, and monitoring areas R1, R2, and R3 are set in the upper area, the middle area, and the lower area, respectively. Yes. On the other hand, the left image as the comparative image has a size of 200 × 640 pixels (vertical and horizontal) as an example, and monitoring regions R4, R5, and R6 are set in the upper region, the middle region, and the lower region, respectively. Has been. Here, the reason why the left image has a larger number of pixels than the right image in the horizontal direction of the image is that stereo matching is taken into consideration. In the stereo method, a comparison image is searched for an object that matches an object in a reference image, and the three-dimensional distance of the object from the parallax (offset amount in the stereo matching direction) of the object in both images. This is a method for obtaining. The smaller the distance from the camera to the object, the greater the offset amount in the stereo matching direction. Therefore, in order to calculate the distance of the object located at the right end of the right image, it is necessary to extend the right end of the left image in the stereo matching direction.
[0015]
The monitoring areas R1, R2, and R3 “positionally correspond” with the monitoring areas R4, R5, and R6 in the left image. In this specification, “corresponding to position” is set not only when the left and right monitoring areas are set to the same position, but also with a slight offset in consideration of stereo matching. It is used in a broad sense to include cases. This offset amount is preferably determined in consideration of the tendency of the parallax of the object that would generally exist in each monitoring area. For example, in the outside monitoring for monitoring the front of the vehicle, the monitoring areas R1 and R4 set on the upper side of the image area usually display the sky (infinity) or a relatively distant three-dimensional object (such as a building). Tend. Accordingly, the parallax calculated in this region tends to be relatively small. Therefore, in consideration of the tendency of the distance of the three-dimensional object or the like displayed on the upper part of the image, the offset amount regarding the monitoring areas R1 and R4 is set to be small. For example, as shown in FIG. 3, the monitoring region R4 is offset by 15 pixels in the stereo matching direction with respect to the monitoring region R1. In addition, in the monitoring areas R2 and R5 set in the middle of the image area, normally, there is a tendency for a car or the like traveling ahead to be displayed, so that the parallax in this area tends to be medium. Therefore, in consideration of the tendency of the landscape displayed in the middle of the image, the offset amount for the monitoring areas R2 and R5 is set to a medium level. From the experimental results regarding this, it is preferable to assume a distance of about 30 to 40 m. As an example, as shown in FIG. 3, the monitoring region R5 is offset by 25 pixels in the stereo matching direction with respect to the monitoring region R2. Furthermore, since the monitoring areas R3 and R6 set below the image area usually tend to display the ground surface such as a road, the parallax in this area tends to be relatively large. Therefore, in consideration of such a tendency of the scenery displayed in the lower part of the image, the offset amount regarding the monitoring areas R3 and R6 is set to be relatively large. For example, as shown in FIG. 3, the monitoring region R6 is offset by 30 pixels in the stereo matching direction with respect to the monitoring region R3. In this way, each monitoring area is offset in the stereo matching direction in consideration of the distance tendency of the scenery projected in each area. As a result, the left and right monitoring area pairs corresponding to each other can project the same object, and therefore, in a normal imaging situation, each monitoring area pair has substantially the same luminance state.
[0016]
In step 2 following step 1, the brightness level of the first monitoring area and the brightness level of the second monitoring area are calculated. That is, the average value A1 of the luminance values of the respective pixels existing in the first monitoring areas R1, R2, and R3 in the right image area is obtained. Similarly, the average value A2 of the luminance values of the respective pixels existing in the second monitoring areas R4, R5, R6 in the left image area is obtained. Although the average value of the luminance values of all the pixels existing in the monitoring area may be obtained, in order to avoid an increase in the amount of calculation associated therewith, in this embodiment, the average value is uniformly distributed in the area. Luminance average values A1 and A2 are calculated using pixels as samples. As an example, samples are extracted every 4 pixels in the vertical direction and every 8 pixels in the horizontal direction, and the average value thereof is set as the luminance level in the entire monitoring area.
[0017]
In step 3, it is determined whether or not the difference (absolute value) between the average luminance value A1 of the first monitoring area and the average luminance value A2 of the second monitoring area is equal to or greater than a predetermined determination value. (Step 3). As described above, almost the same scenery is displayed in the left and right monitoring areas. Therefore, in a normal situation where the brightness of the left and right images is balanced, the average luminance values A1 and A2 of the left and right monitoring areas are substantially equal. Therefore, in a normal situation, a negative determination is made at this step, so the fail flag NG is set to 0 (step 4). As a result, normal control related to outside vehicle monitoring is continued. On the other hand, in an abnormal situation where the brightness is unbalanced due to disturbance, an affirmative determination is made in this step, and the fail flag NG is set to 1 (step 5). As a result, fail-safe for temporarily stopping normal monitoring control is executed.
[0018]
In actual control, it is preferable to perform the switching of the NG flag when a predetermined cycle is continued in order to ensure the reliability of the outside monitoring. For example, when the fail flag NG is set from 0 to 1, it is changed when the affirmative determination in step 3 continues for 5 cycles (0.5 seconds). Further, when the fail flag NG is set from 1 to 0, it is changed when the negative determination in step 3 continues for 20 cycles (2.0 seconds).
[0019]
As described above, in the fail determination according to the present embodiment, whether or not the brightness of the left and right images is balanced is monitored in real time. And when this balance is not taken, it determines with a failure. Thereby, for example, even if a situation occurs in which only one camera is blinded and a normal image cannot be obtained, it is possible to accurately detect it by monitoring the brightness balance. The same applies to a situation where a normal image cannot be obtained due to partial dirt or cloudiness of the lens or windshield of one camera, or irregular reflection due to raindrops. As a result, even when an abnormal image is generated due to these external causes, fail-safe corresponding to it can be accurately performed, so that the safety of the stereo-type vehicle exterior monitoring device can be ensured at a high level.
[0020]
Further, in this embodiment, the failure determination is performed based on a part of the image data instead of referring to the entire image. Therefore, the fail determination can be performed with a small amount of calculation so that the fail detection can be performed in real time.
[0021]
Further, regarding the setting of each monitoring area, the offset amount of the left and right monitoring areas is determined in consideration of the tendency of parallax in the area, so that the accuracy of fail determination can be further improved. .
[0022]
【The invention's effect】
As described above, in the present invention, when a situation occurs in which the brightness of the left and right images cannot be balanced and is not suitable for stereo processing, it is possible to perform fail-safe in response to the situation appropriately. It becomes possible to ensure the safety of the apparatus at a higher level.
[Brief description of the drawings]
FIG. 1 is a block diagram of a stereo-type vehicle exterior monitoring apparatus according to an embodiment. FIG. 2 is a flowchart showing a failure determination routine based on brightness imbalance. FIG. 3 is a diagram for explaining a monitoring area set in a stereo image. [Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 Main camera 2 Sub camera 3, 4 A / D converter 5 Image correction part 6 Stereo image processing part 7 Distance image memory 8 Original image memory 9 Microcomputer 10 Road recognition part 11 Three-dimensional object recognition part 12 Fail judgment part 13 Processing part 14 AT Control unit 15 ABS control unit 16 TCS control unit 17 Torque balance control unit 18 Engine control unit

Claims (4)

In the stereo-type vehicle exterior monitoring device that monitors the front of the vehicle and performs fail-safe when it is determined to fail,
Stereo imaging means for obtaining a pair of captured images by imaging a scene outside the vehicle in front of the vehicle;
An original image memory for storing image information of the pair of captured images;
A stereo image processing unit that calculates distance information by calculating a relative shift related to the position of the same object in the pair of captured images;
A distance data memory for storing the distance information;
A recognition unit for recognizing an object based on image information stored in the original image memory and distance information stored in the distance data memory;
Based on information from the recognition unit, a processing unit that performs monitoring control;
Based on the image information stored in the original image memory, when performing a fail determination, and determining a failure, a failure determination unit that instructs the processing unit to temporarily suspend monitoring control,
The fail determination unit
The overall luminance level of the first monitoring area set in one image area imaged by the stereo imaging means and the second luminance value set in the other image area imaged by the stereo imaging means. Calculating means for obtaining the overall luminance magnitude of the monitoring area;
Determining means for determining a failure when the luminance magnitude in the first monitoring area is different from the luminance magnitude in the second monitoring area;
The first monitoring area is set in advance at a predetermined position so that when monitoring the front of the vehicle, a specific distance tendency appears in the scenery displayed in the first monitoring area,
The second monitoring area is a predetermined offset amount that is a fixed value set in consideration of the specific distance tendency and is offset in the stereo matching direction with respect to the position of the first monitoring area. A stereo-type vehicle exterior monitoring device having a fail-safe function, characterized in that it is preset at the position . The first monitoring area is composed of a plurality of monitoring areas set at different positions in the vertical direction in the one image area,
The second monitoring area is set at different positions in the vertical direction in the other image area, and each of the plurality of monitoring areas corresponds to the monitoring area constituting the first monitoring area. And consisting of
The stereo-type vehicle exterior monitoring apparatus having a fail-safe function according to claim 1 , wherein offset amounts of the monitoring areas are different. Offset of the upper monitoring region, a stereoscopic vehicle surroundings monitoring apparatus having a fail-safe function as described in claim 2, towards the upper monitoring region being less than the offset amount of the lower side of the monitoring area . The stereo system having a fail-safe function according to any one of claims 1 to 3 , wherein the overall luminance level of the monitoring area is an average luminance of luminance existing in the monitoring area. Outside monitoring device.

Images