JP4066869B2 - Image processing apparatus for vehicle - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a vehicle image processing apparatus for recognizing a lane in which a host vehicle travels.
[0002]
[Prior art]
Development of an automatic travel device that automatically travels in the set lane, a technology that warns inadvertent departure from the lane of the vehicle, and activates a system that protects passengers in anticipation of a collision after the departure Is underway. In such a technique, it is necessary to accurately recognize the lane in which the host vehicle is to travel. As a technique for that purpose, a boundary line that divides the lane by image recognition such as edge extraction is obtained by acquiring an image of the vehicle traveling direction. An apparatus for detecting a white line is known (for example, see Patent Document 1).
[0003]
[Patent Document 1]
Japanese Patent Laid-Open No. 7-239996 (paragraphs 0013 to 0029, FIGS. 1 to 5)
[0004]
[Problems to be solved by the invention]
In this type of lane recognition technology, a white line that divides a road is recognized by image recognition based on image data acquired by an imaging device. However, the imaging conditions change significantly as the vehicle moves. As a result, the obtained image data may not be suitable for white line recognition due to backlight, overexposure, underexposure, and the like. Also, as a result of edge extraction, there are many lane marking candidates, and it may be difficult to select a correct lane marking. When image data inappropriate for recognition is a part of a series of acquired images, it is sufficient to exclude such inappropriate image data from the recognition processing target. In addition, a method of selecting candidates based on past recognition data may be used. However, when it is difficult to use such past recognition data or image data, for example, when the white line is detected for the first time after the device that performs lane recognition is activated, or when the detection is continued after the undetectable state continues. When there are many lane marking candidates at the time of detection or the like, it is difficult to determine which candidate is to be adopted as the lane marking.
[0005]
Accordingly, an object of the present invention is to provide a vehicle image processing apparatus capable of selecting an accurate lane marking at the time of starting detection or returning from an undetectable state.
[0006]
[Means for Solving the Problems]
In order to solve the above-described problem, an image processing apparatus for a vehicle according to the present invention detects a road lane line of a lane in which the vehicle travels by edge extraction from an image acquired by an imaging unit mounted on the vehicle. The processing device further includes means for detecting the traveling direction of the vehicle, and compares the detected traveling direction of the vehicle with the direction of the road lane marking when starting detection of the road lane marking or returning from the undetectable state. Thus, the road lane marking detection result is output when both agree with each other within a predetermined range.
[0007]
At the start of detection or when returning from the undetectable state, the vehicle is likely to travel along the lane. Therefore, the traveling direction of the vehicle is compared with the extending direction of the road lane line candidate determined by image recognition, and the road lane line candidate that matches when the vehicle lane is within a predetermined range is If it is determined as a road lane marking, an accurate lane marking can be selected.
[0008]
Furthermore, it is provided with instruction means for giving a predetermined instruction to the driver, and at the time of starting detection of the road lane marking or returning from the undetectable state, while comparing the traveling direction of the vehicle and the direction of the road lane marking, It is preferable to instruct the driver to maintain the traveling lane by this instruction means.
[0009]
DETAILED DESCRIPTION OF THE INVENTION
DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described in detail with reference to the accompanying drawings. In order to facilitate the understanding of the description, the same reference numerals are given to the same components in the drawings as much as possible, and duplicate descriptions are omitted.
[0010]
FIG. 1 is a block diagram of a vehicle control apparatus including a vehicle image processing apparatus according to the present invention, and FIG. 2 is a perspective view showing a vehicle equipped with the same. The vehicle control apparatus 100 includes a front camera 11 that acquires an image ahead of the vehicle, and a white line recognition ECU 1 that recognizes a white line on the forward path of the vehicle by image processing from the image acquired by the front camera 11 (the vehicle according to the present invention). An image processing apparatus for use). The white line recognition ECU 1 also receives output signals from the yaw rate sensor 12, the steering angle sensor 13, and the vehicle speed sensor 14, and the recognition result of the white line recognition ECU 1 is a PCS (Pre-Crash Safety) system via the in-vehicle LAN 2. 3. It is sent to the departure warning system 4, the lane keeping system 5, etc. A monitor device 42 and a speaker 43 are connected to the white line recognition ECU 1. Details of the PCS system 3, the departure warning system 4, and the lane keeping system 5 will be described later.
[0011]
As shown in FIG. 2, the front camera 11 is arranged at the upper part of the front window of the vehicle 200 (for example, the back side of the rearview mirror), and is an image in front of the vehicle 200, that is, an image of the travel lane 300 in front of the vehicle Line 301 is included).
[0012]
Hereinafter, a process for recognizing the traveling lane 300 by an image process from an image acquired by the front camera 11 will be specifically described. FIG. 3 is a processing flow of this image processing, and FIGS. 4 to 8 are diagrams for explaining the processing contents. The processing described below is performed by the white line recognition ECU 1 unless otherwise specified.
[0013]
First, an image as shown in FIG. 4 is acquired by the front camera 11 (step S1). On the right side of the traveling lane 300 with the lane markings 301R and 301L as a boundary, there is an opposing lane 310 across the lane marking 301R, and there is a lane marking 301X on the right side thereof. An oncoming vehicle 210 exists on the opposite lane 310. In addition to being a continuous line, this partition line may be a discontinuous line such as a dotted line or a broken line. Since these lane markings 301 are usually painted white, the painted portion is hereinafter referred to as a white line. It is included in the white line. Furthermore, an auxiliary line may be installed in parallel with the lane marking 301.
[0014]
The acquired image is subjected to AD conversion to obtain a monochrome gradation (for example, x × y is 320 × 240 pixels, where the horizontal direction in FIG. 4 is the x direction and the vertical direction is the y direction). It is converted into digital image data of 8 gradations and sent to the white line recognition ECU 1 (step S2). Of course, a digital camera capable of directly outputting digital data having a predetermined number of pixels and gradations may be used for the front camera 11, and the analog output of the front camera 11 may be AD converted inside the white line recognition ECU 1. The converted digital image is substantially the same as the analog image shown in FIG.
[0015]
Next, the white line recognition ECU 1 performs edge extraction processing on the received digital image to extract white line candidates in the image (step S3). For example, the average luminance in the image is used as a threshold value, and the pixels whose left pixel luminance is lower than the threshold value and whose right pixel luminance is higher than the threshold value are extracted as edges. Thereby, the pixel position of the position corresponding to the left boundary line of the white line (partition lines 301L, 301R, 301X, etc.) is extracted. The edge extraction is not limited to this. For example, a pixel whose luminance difference with the left pixel is equal to or greater than a threshold value may be extracted as an edge. In these, the pixel position at the position corresponding to the left boundary line of the white line is extracted by extracting the edge that changes from dark to light, but the white line is extracted by extracting the edge that changes from light to dark. Alternatively, the pixel position at the position corresponding to the right boundary line may be extracted.
[0016]
FIG. 5 shows the extracted edge position superimposed on the original image. Black circles are extracted edge positions, and broken lines indicate pixel positions in the Y direction. Here, the acquired image of the traveling lane 300 is obtained by projecting the traveling lane 300 onto the image plane of the front camera 11. In contrast, the positions of the lane markings 301 </ b> L and 301 </ b> R that are actually desired are positions on the travel lane 300 with respect to the vehicle. Therefore, by geometric transformation, the vehicle is set to the origin (0, 0), the lateral direction in the X-axis direction (the right direction is positive when viewed from the vehicle), and the distance from the vehicle in the Y-axis direction (the larger the numerical value, the larger the value). A position corresponding to the edge pixel position on the travel plane expressed by taking the front, that is, away from the vehicle is obtained (step S4). This geometric transformation can be performed based on the attachment position, the imaging direction, and the imaging area of the front camera 11. In addition, you may correct | amend an imaging direction etc. by detecting the inclination of a vehicle body, etc. with several vehicle height sensors. It is also preferable to correct image distortion caused by optical aberrations or the like of the imaging system of the front camera 11. FIG. 6 shows the positions of the extracted edge pixels corresponding to the lane markings 301L thus obtained by geometric transformation with respect to the vehicle.
[0017]
Next, line / curve extraction is performed by performing Hough transform on the result of geometric transformation (step S5). Taking the case of straight line extraction as an example, when the position of the extracted edge point on the travel plane is represented by (x, y), the inclination θ is set to a predetermined angle Δθ from 0 to 180 degrees with respect to each edge point. And ρ corresponding to each θ is obtained as ρ = x cos θ + ysin θ. Then, it is searched in which area (ρ, θ) obtained in this way is located in an area obtained by dividing the Hough space at predetermined intervals, and the searched area is voted. That is, the number of votes in each region indicates the number of edge point Hough transform results in this region. This voting is preferably performed separately on the left side and the right side of the traveling lane 300. The center line of the driving lane 300 required to divide the driving lane 300 from left to right can be grasped by referring to the lane recognition results so far, the traveling direction of the vehicle, the yaw rate, the steering angle, and the like. it can. FIG. 7 shows the voting result in the Hough transform area on the left side of the traveling lane 300 obtained in this way.
[0018]
A region (ρ, θ) having a plurality of votes is a straight line extracted from an edge point (also extracted as a single line even when being interrupted halfway). This straight line corresponds to y = ρ / sin θ−x × cos θ / sin θ on the (x, y) plane. Here, the more voting results are (ρ, θ), the more straight line the edge points are closer to. Considering the influence of noise or the like, a straight line having the number of votes equal to or greater than a threshold (3, preferably) is taken as the extraction result. The area where the number of votes is 2 in FIG. 7 indicates this noise.
[0019]
Here, the case of a straight line has been described as an example, but in the Hough transform, a curve can be similarly extracted. Then, a straight line or curve that has the largest number of votes and is considered to be the most approximate is determined as a lane line candidate as an extraction result.
[0020]
Next, it is determined whether or not past white line recognition results are accumulated in the white line recognition ECU 1 (step S6). Here, the past recognition result is determined based on whether or not the lane marking information recognition result up to the past, which is a predetermined time later than the present, is stored. At this time, it is not necessary that all lane line information recognition results up to the past that have gone back a predetermined time are stored, but if a predetermined number or more of data has been saved up to a past that has gone back a predetermined time, Even if erroneous recognition occurs, control based on past recognition information is possible, which is preferable.
[0021]
In step S6, if the lane line information recognition result up to the past, which is traced back for a predetermined time, is stored, it indicates that the lane recognition has succeeded continuously for a predetermined time or more after starting the lane recognition. The process proceeds to step S7, where it is checked whether or not the information on the lane line candidate recognized this time is consistent with the past recognition result. If it is determined that the difference between the lane line candidate and the past recognition result is within a predetermined range and is consistent, the information of the recognized lane line candidate is output as white line information, and this information is stored in the white line recognition ECU 1. (Step S10).
[0022]
On the other hand, if it is determined that the deviation exceeds the predetermined range and is not matched, the information on the recognized lane line candidate is discarded and the lane line information estimated from the past recognition result is output (step) S8).
[0023]
If it is determined in step S6 that the lane marking information recognition result up to the past that is a predetermined time has not been saved, the process proceeds to step S9. In the case where the lane line information recognition result up to the past that has been back for a predetermined time is not saved, if the predetermined time has not elapsed since the start of lane recognition (in addition to the time when the lane recognition means is activated, the parking lot・ When there is a road entry from the parking area, etc.), or when the recognition result stored due to a momentary interruption or some kind of trouble is lost, or when returning from a state where lane recognition cannot be performed for a predetermined time or longer (for example, , When passing through a construction section or a shadowed part). In such a case, the consistency between the past recognition result and the current recognition result cannot be checked. Therefore, in such a case, the expected course of the vehicle 200 is estimated from the vehicle state such as the yaw rate, the steering angle, and the vehicle speed output from the yaw rate sensor 12, the steering angle sensor 13, and the vehicle speed sensor 14, and this expected course and this time The travel lane 300 determined from the recognition result by the image processing is compared (step S9).
[0024]
When the deviation between the expected course and the travel lane 300 obtained from the recognition result is within a predetermined range, the lane line recognition is regarded as successful, and the recognized lane line candidate information is output as white line information. It is stored in the white line recognition ECU 1 (step S10). Whether or not the deviation between the expected route and the travel lane determined by the image recognition process is within a predetermined range can be verified as follows, for example. Judgment is made based on whether or not the predicted route deviates from the driving lane for a preset distance or for a preset time. Specifically, the determination may be made based on whether or not the angle formed between the expected course and the travel lane is within a predetermined angle. It is also possible to make a determination based on whether or not the curvature of the expected traveling route and the curvature of the traveling lane are deviated by a predetermined value or more.
[0025]
On the other hand, when the deviation between the predicted course and the travel lane 300 obtained from the recognition result exceeds a predetermined range (when the lane marking recognition results 302R and 302L are as shown in FIG. 8), the lane marking is erroneously recognized. Considering it as a thing, the recognized lane marking candidate is discarded, and the process is terminated (step S11).
[0026]
Thus, in order to determine the success or failure of the detection of the lane marking information based on the vehicle state, the vehicle 200 needs to travel correctly in the traveling lane 300, and when the driver changes the lane, etc. Even if the recognition is successful, it is determined to be a misrecognition. Therefore, when past white line information is not stored, it is desirable to notify the driver that the route and lane line information are being compared by the monitor device 42 and the speaker 43. By this notification, the driver recognizes that the white line recognition ECU 1 is comparing road and lane marking information, and for comparison of these information, it is detected that the lane in which the vehicle is currently traveling is maintained. Be careful to maintain the driving lane, as it will lead to improved accuracy. Furthermore, the correctness of the past lane information may be verified by collating the trajectory on which the vehicle 200 actually traveled with the travel lane predicted from the past recognition result.
[0027]
According to the present invention, it is possible to test whether or not the lane line information obtained by image recognition is correct lane line information without requiring a special operation by the driver even immediately after the start of lane recognition. For this reason, since vehicle behavior control using erroneously detected lane marking information can be suppressed, the driver is not disturbed and the operability is improved.
[0028]
The image recognition means for the lane markings (white lines) is not limited to the recognition method described above, and various recognition methods can be used. The recognized lane marking information may be verified based on map information or road information. For example, when a curve having an extremely small radius of curvature is detected while traveling on an expressway, it can be determined that this is a false detection.
[0029]
Next, vehicle control in the vehicle control apparatus according to the present invention using the white line information thus obtained will be described. 9 to 11 are block configuration diagrams of the PCS system 3, the departure warning system 4, and the lane keeping system 5 shown in FIG.
[0030]
First, the PCS system 3 shown in FIG. 9 will be described. This PCS system includes a braking system 32 for controlling a braking force, a steering assist motor 33 for applying an assisting force to a driver's steering force, a seat belt pretensioner 34 for winding up a seat belt, and an air bag 35 for protecting an occupant. The PCS / ECU 31 includes a laser radar 16 for detecting a preceding vehicle and an obstacle, a yaw rate sensor 12, a rudder angle sensor 13, a vehicle speed sensor 14, and a brake pedal. The output of the brake pedal switch 15 for detecting an operation / non-operation state is input.
[0031]
The PCS / ECU 31 detects preceding vehicles and obstacles on the course of the own vehicle based on the lane information (white line information) sent from the lane recognition ECU 1 and the outputs of the sensors 12 to 15 and the laser radar 16. , Predict the possibility of collision. Then, if it is determined that the collision is unavoidable, the braking force is instructed to the braking system 32 and the assist force applied at the time of the brake operation is switched to a setting that increases in advance. The braking force should be obtained.
[0032]
Further, the assist steering force by the steering assist motor 33 is increased so that the driver can perform steering with a weaker force. If necessary, steer automatically to avoid collisions.
[0033]
Further, the seat belt pretensioner 34 winds up the extra length of the seat belt in advance before the collision, so that the occupant is strongly restrained by the seat and the damage at the time of the collision is reduced. Moreover, the damage to the passenger at the time of the collision is reduced by operating the airbag at an optimal timing before and after the collision, not after detecting the collision. By these controls, even when a collision is unavoidable, serious damage to the vehicle and the occupant can be suppressed and the shock of the collision can be reduced.
[0034]
Next, the departure warning system 4 shown in FIG. 10 will be described. The departure warning system 4 includes a departure warning ECU 41, a display monitor device 42, and a speaker 43 for voice output. The departure warning ECU 41 includes lane information (white line information), vehicle speed information, and yaw rate. Information is entered.
[0035]
The departure warning ECU 41 predicts the course of the host vehicle from the vehicle speed / yaw rate information, and compares this with the lane information. When a lane departure is expected, the monitor device 42 displays that the departure from the lane is expected, and the speaker 43 issues an alarm with an alarm sound or a sound output, so that the driver is informed. Notify you.
[0036]
Next, the lane keeping system 5 shown in FIG. 11 will be described. The lane keeping system 5 includes a lane keeping ECU 51, a braking system 32 that controls a braking force, a steering assist motor 33 that applies an assisting force to a driver's steering force, and a transmission unit 52 that changes a transmission state of the transmission. The lane keep ECU 51 includes a laser radar 16 for detecting a preceding vehicle and an obstacle, a yaw rate sensor 12, a rudder angle sensor 13, a vehicle speed sensor 14, and an operation of a brake pedal. The outputs of the brake pedal switch 15 for detecting the non-operation state and the accelerator opening sensor 17 for detecting the operation amount of the accelerator pedal are input.
[0037]
The lane keeping system 5 travels while maintaining the position of the vehicle in the traveling lane 300 that is currently traveling. Specifically, by predicting the course of the host vehicle from the outputs of the yaw rate sensor 12, the rudder angle sensor 13, and the vehicle speed sensor 14, and comparing this with the lane information, whether or not the traveling in the traveling lane 300 can be maintained. Determine whether. Then, based on the forward lane information, the steering assist motor 33 is controlled so that the own vehicle travels in the white line from the position of the own vehicle and the white line, thereby maintaining the travel in the travel lane 300.
[0038]
As described above, by detecting the road lane marking of the traveling lane of the host vehicle using the vehicle image processing apparatus according to the present invention, it becomes possible to recognize the lane marking and thus the traveling lane at an early stage. It is possible to improve the accuracy of behavior control of various vehicles using this.
[0039]
【The invention's effect】
As described above, according to the present invention, the course of the vehicle is predicted based on the vehicle state, and the correctness of the lane line information is verified by comparing this with the lane line information obtained by image recognition. The lane line information can be verified without any special operation by the driver. At this time, if the driver is instructed to maintain the lane, accurate lane line information can be obtained.
[Brief description of the drawings]
FIG. 1 is a block diagram of a vehicle control apparatus including a vehicle image processing apparatus according to the present invention.
FIG. 2 is a perspective view showing a vehicle on which the vehicle control device of FIG. 1 is mounted.
3 is a processing flow of image processing in the vehicular image processing apparatus of FIG. 1;
FIG. 4 is a diagram illustrating an example of an image acquired by a front camera.
FIG. 5 is a diagram illustrating a result of performing edge extraction processing from the image of FIG. 4;
6 is a diagram illustrating a result of the geometric transformation process in FIG. 5;
7 is a diagram illustrating a result of the Hough transform process of FIG. 6. FIG.
FIG. 8 is a diagram illustrating an example when a lane marking is erroneously recognized from the image of FIG. 4;
9 is a block diagram of the PCS system shown in FIG.
10 is a block configuration diagram of the departure warning system shown in FIG. 1. FIG.
FIG. 11 is a block configuration diagram of the lane keeping system shown in FIG. 1;
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 ... White line recognition ECU, 2 ... In-vehicle LAN, 3 ... PCS (Pre-Crash Safety) system, 4 ... Deviation warning system, 5 ... Lane keeping system, 11 ... Front camera, 12 ... Yaw rate sensor, 13 ... Steering angle sensor, DESCRIPTION OF SYMBOLS 14 ... Vehicle speed sensor, 15 ... Brake pedal switch, 16 ... Laser radar, 17 ... Accelerator opening sensor, 31 ... PCS / ECU, 32 ... Braking system, 33 ... Steering assist motor, 34 ... Seat belt pretensioner, 35 ... Air Bag, 41 ... Deviation warning ECU, 42 ... Monitor device, 43 ... Speaker, 51 ... Lane keep ECU, 52 ... Transmission means, 53 ... Electronically controlled throttle, 100 ... Vehicle control device, 200 ... Vehicle, 300 ... Driving lane, 301 ... marking line, 302 ... marking line recognition result (misrecognition line).

Claims (1)

In an image processing apparatus for a vehicle that detects a road lane marking of a lane in which the vehicle travels by image recognition from an image acquired by an imaging unit mounted on the vehicle,
The vehicle further comprises means for detecting the traveling direction of the vehicle, and when the detection of the road lane marking is started or when returning from the undetectable state, the detected traveling direction of the vehicle is compared with the direction of the road lane marking, If it matches within the range of, output the road lane marking detection result ,
Further comprising an instruction means for giving a predetermined instruction to the driver, at the time of starting detection of the road lane marking or when returning from the undetectable state, while comparing the traveling direction of the vehicle and the direction of the road lane marking, A vehicular image processing apparatus that instructs the driver to maintain a traveling lane by the instruction means .

Images