JP5868586B2 - Road characteristic analysis based on video image, lane detection, and lane departure prevention method and apparatus - Google Patents

Description

  The present invention relates to a roadway scene image processing method and apparatus used in an automobile.

  In order to detect streak or solid line type lane markings, some conventional approaches use images from a video camera mounted on an automobile. However, such an approach cannot detect sparsely arranged dome-shaped white disks called botsdots. Other conventional approaches are limited to determining the camera distance and yaw angle relative to the lane marking and the curvature of the lane marking. Such an approach cannot detect figures and objects on the road other than lane markings, and can only find a pair of lane markings (one on the left side and one on the right side of the vehicle). It is. In such an approach, the location of road boundaries is not considered and no further characterization of the road is done. In yet another approach, pre-stored lane data is required for later use as a reference standard.

Dickmanns, “An Integrated Spatio-Temporal Approach to Automatic Visual Guidance of Autonomous Vehicles”, 1990

  The object of the present invention is to provide a video image based lane departure warning method and apparatus for vehicles on the road that can solve the above-mentioned problems.

A method for warning a lane departure based on a video image for a vehicle on a road according to the present invention uses an apparatus configured to perform at least the following steps:
Receiving an image of a road ahead of the vehicle from a video imager;
A road marking detector detecting in the image one or more road markings corresponding to the road marking on the road;
A characteristic analysis module analyzing the characteristics of the image area to determine an evaluation of road drivability corresponding to the image area outside the detected road marking;
An offset detecting means detecting a lateral offset on the road of the vehicle relative to the road surface sign based on the detected road surface sign;
A lane departure detector generates a warning signal as a function of the lateral offset and the evaluation;
The step of detecting one or more road markings in the image further comprises the road marking detector determining a region of interest in the image and detecting the one or more road markings in the region of interest. And a road surface sign attribute detecting means specifying the attribute of the road surface sign,
The step of identifying the attribute of the road surface sign further includes the step of the road surface sign attribute detection means identifying the type and number of the detected road surface sign,
In the step of analyzing the characteristics of the image area outside the detected road sign, the characteristic analysis module includes a characteristic vector composed of a combination of a plurality of image characteristics inside the road sign and the same outside the road sign. If there is a similarity between the comprising a combination characteristic vectors of a plurality of image characteristics, it is determined that it is possible to the vehicle enters the region outside the safe the road marking, further, on the outside of the road markings A plurality of image characteristics are compared with the image characteristics in the database of the image area, and based on this database, including a level that allows safe driving and a level that does not allow safe driving outside the road surface sign, We have at least two rows of the evaluation of,
The step of generating the warning signal further includes the step of the lane departure detector generating a warning signal as a function of the attribute in addition to the lateral offset and the driving ease evaluation. And
The step of detecting the road surface sign is further characterized in that the symbol detecting means detects an image feature representing the symbol on the road, identifies such an image feature, and the image feature is a symbol representing a preset warning. Alternatively, in the case of a character sign, it includes a step of notifying the driver of the presence of the sign by voice.
An apparatus for warning a lane departure based on a video image for a vehicle on a road according to the present invention is also provided.
Image processing means configured to receive an image of a road ahead of the vehicle from a video imager;
A road marking detector configured to detect in the image one or more road markings corresponding to the road markings on the road;
A characteristic analysis module configured to analyze the characteristics of the image area in order to determine an evaluation of the road drivability corresponding to the image area outside the detected road surface sign;
Offset detecting means configured to detect a lateral offset on the road of the vehicle relative to the road surface sign based on the detected road surface sign;
A lane departure detector configured to generate a warning signal as a function of the lateral offset and the evaluation,
The road marking detector is further configured to determine a region of interest in the image and to detect one or more road markings in the region of interest;
When the characteristic analysis module has an approximation between a characteristic vector composed of a combination of a plurality of image characteristics inside the road surface sign and a characteristic vector composed of a combination of the same plurality of image characteristics outside the road sign Determining that the vehicle can safely enter the area outside the road sign, and comparing the plurality of image characteristics outside the road sign with image characteristics in the image area database; Is configured to perform at least two classifications including a level at which safe driving is possible and a level at which safe driving is impossible outside the road surface sign,
Detecting an image feature representing a symbol on the road, identifying such an image feature, and if the image feature is a symbol or character sign representing a preset warning, the presence of the sign by voice Is further provided with a symbol detection means for informing the driver of this.

  These and other features, aspects, and advantages of the present invention will become better understood with reference to the following description, claims, and accompanying drawings.

FIG. 2 is a functional block diagram of a road image analysis based on video image, lane detection and lane departure prevention device according to an embodiment of the present invention coupled to a vehicle. 4 is an overall flowchart of steps of a road image analysis based on video image, lane detection and lane departure prevention process, according to one embodiment of the present invention. It is a top view which shows an example of the position of the video camera represented relatively with the road surface sign on a roadway. It is a figure which shows an example of the relationship between the pixel in an image surface, and the corresponding point on the road surface projected on the pixel in an image with a video camera. FIG. 6 shows a trapezoidal region of interest (ROI) superimposed on a video image, where gray level gradients and mean absolute deviations are extracted from the pixels of the image. It is a figure which shows the parallelogram on a road surface. It is a figure which shows an example of a rough region of interest represented by a dimension on a road surface when a lane boundary is predicted in advance. It is a figure which shows an example of the side surface of the video camera represented relatively to the road in order to demonstrate the relationship between an image surface and the front distance of a road. FIG. 4 is a diagram illustrating an example of an array of pixels p having a vertical pixel size Vp and a horizontal pixel size Hp and arranged in rows r and columns w in a road image. It is a figure which shows an example of 2 points | pieces of the set of the road position on one road surface sign ahead of a video camera. It is a graph which shows the Dickmann (Dickmann) type road marking detection method. It is a figure which shows the measuring method used for the detection of the tar zone on a road. It is a figure which shows the region of interest for detecting the symbol on the road, and the proximal distance and the distal distance representing the position of the symbol. FIG. 6 is a diagram showing the temporal behavior of the proximal distance and the distal distance for detecting symbols on the road. It is a figure which shows an example of the application to a road surface marker fragment detection of a broken-line type road surface marker model. It is a figure which shows an example which expands the size of the horizontal direction of ROI according to the increase in the uncertainty of detection. It is a figure which shows the region of interest used for the clutter detection in a road image. It is a figure which shows an example of the snowfall detection in a road image. It is a figure which shows an example of the reflected light image detection in a road image. It is a figure which shows an example of a road surface sign tracking. It is a figure which shows another example of road surface tracking. It is a figure which shows an example of the road surface sign length detection method. It is a figure which shows an example of the road marking length measurement process in a single image frame. It is a figure which shows the Dickmann (Dickmann) type sign finder applied to the counting of road markings. It is a figure which shows an example of arrangement | positioning of a sign finder for pinpointing the type of a road surface sign. It is a figure which shows an example of the characteristic analysis of an image area | region. It is a figure which shows an example of the identification of the detected object which is not a road surface sign in a road image. It is a graph which shows an example of identification of the detected object which is not a road surface marker in a road image. It is a figure which shows an example of a mean absolute deviation (MAD) calculation used for the comparison of the area | region of the both sides of a road boundary. It is a figure which shows an example of the calculation of the approximation of the image characteristic of two groups. It is a graph which shows an example of four image characteristics. FIG. 6 is a functional block diagram of a road image analysis based on video image, lane detection, and lane departure prevention device according to another embodiment of the present invention coupled to a vehicle. FIG. 25 is a flowchart of steps of a road image analysis based on video image, lane detection, and lane departure prevention process performed by the apparatus shown in FIG. 24. It is a figure which shows an example of ROI which fits the road surface marker on the curved road.

  The present invention provides a method and apparatus for road characteristic analysis, lane detection, and lane departure prevention based on video images. In one embodiment of this apparatus, an image of a road ahead of a video camera mounted on an automobile is used to find a lane marking that includes a straight line portion and a dash portion on the road surface. In order to reproduce the position of the video camera (vehicle) on the road, measured relative to the lane marking, the position of the lane marking found in the image is the geometric information of the video camera and Combined. The reproduced position information of the video camera is used to assist the driver of the car.

  By using a series of substantially real-time image frames from the video camera, the device further detects other graphics and objects on the road in addition to the lane markings. Road boundary position specifying information and road characteristic analysis are used to detect a lateral position (offset) relative to the road boundary. This device derives substantially real-time information about the vehicle position and road characteristics on the roadway, for example to help drive the car in a lane departure warning approach and guide the car traveling on the road.

  This device performs detection of reflected light from the road surface and suppresses the influence of such reflected light in road boundary detection (such reflected light or other such as tar strips). Objects may prevent proper detection of road boundaries). Further, this apparatus may perform detection of road boundaries having a plurality of different types of road markings, road marking classification, various light / dark adjustments, identification of various weathers, elimination of false road markings, and the like. When there are multiple lines in multiple lanes, the apparatus may recognize the multiple lines and select an appropriate lane marking (generally closest to the vehicle in the lateral direction). In addition, this device can detect road edges using road edge information instead of lane line information to help guide the car traveling on the road when no lane line exists. Good.

  In addition, the device may identify the location of intermittent road markings (such as botsdots and reflectors) on the road and the location of the vehicle on the road from the information. Once road surface sign candidate (road surface sign fragment candidate) information is collected from the image, this device makes various attempts through such information to detect the vehicle position on the road as needed.

  FIG. 1 shows an apparatus 15 for road characteristic analysis based on video images, lane detection and lane departure prevention according to an embodiment of the invention, in which a video camera 1 is connected to a vehicle 2 mounted to face forward. It is a functional block diagram. The video camera 1 is mounted on the vehicle 2 so as to face forward. The video camera 1 acquires a [two-dimensional (2-D)] image 3 of a road (lane / roadway) 4 in front of the vehicle 2, and uses these images for road characteristic analysis, lane detection, and lane departure prevention. Therefore, it is transmitted to the controller 5. A sampler 17 that samples the digitized video input signal may be used to create a two-dimensional image as an array of pixels arranged in rows and columns.

  In this example, the controller 5 includes a lane boundary detection device 6, a lane departure detection device 9, a camera calibration module 10, a warning module 11, and a curvature measurement module 12. The lane boundary detection device 6 processes the image 3 by road characteristic analysis, and tracks the positions of road marking image features 7L and 7R representing the actual road markings 8L and 8R on the road 4 in the image 3, respectively. The lane boundary detection device 6 includes a reflected light image detection module 6a that detects a reflected light image from the road surface in the image 3 and reduces (suppresses) the influence on the detection of the road sign image features 7L and 7R. . Relative to the video camera 1, the road surface sign 8L represents the left side of the road, and the road surface sign 8R represents the right side of the road. The data of the position and yaw angle (relative to the position of the video camera 1) of the road marking image features 7L and 7R are supplied from the lane boundary detection device 6 to the lane departure detection device 9. The curvature measurement module 12 measures the curvature of the road using the information about the lane markings or road boundaries detected by the lane boundary detection device 6 and supplies the information to the lane departure detection device 9. The lane departure detection device 9 further receives speed information from the vehicle 2 and also receives calibration data (which can be supplied from the camera calibration module 10). The lane departure detection device 9 may periodically receive vehicle speed, lane line position, curvature information, and yaw angle information in order to identify the position of the vehicle 2 relative to the road markings 8L, 8R. preferable. Road markings 8L, 8R can include lane markings, road boundaries, and the like.

  Based on the vehicle speed information, camera calibration information, and tracked image feature positions of the road marking image features 7L and 7R, the controller 5 uses the lane departure detector 9 to move the vehicle 2 from the appropriate travel lane on the road 4. Specify whether they are off. When the lane departure detection device 9 detects that a lane departure or road departure is imminent, the warning module 11 is activated. Specifically, the lane boundary detection device 6 determines a position where the road boundary or lane marking is to be searched for in the image 3, and the road surface sign image features 7L and 7R such as the road boundary or lane marking are determined for each image. 3 to detect. The lane departure detection device 9 associates such detected road marking image features 7L, 7R with a position on the road surface by a geometric transformation based on the video camera's geometric arrangement information, thereby The position of the video camera (vehicle) on the road is reproduced relative to the actual road markings 8L and 8R which are lane markings or road boundaries. The lane departure detection device 9 further detects when the video camera (vehicle) gets too close to the road boundary or lane marking and notifies the warning module 11 of it.

  The controller 5 may further function to detect lane markings or road boundaries using a variety of attempts with image features that may be fragments of lane markings or road boundaries in difficult situations. Good. The controller 5 further eliminates image features in the image that are not road boundaries or road markings, predicts regions of interest for detection of road boundaries or lane markings in the next image, new and more appropriate road boundaries or lanes. It may function to detect lane markings and to determine that deviations from roads or lanes are natural. These supplementary functions can be used to reduce the occurrence of false alarms by the controller 5 by enhancing the ability to properly detect road markings or boundaries.

  Using information from the camera calibration module 10, the lane boundary detection device 6 predicts a basic region of interest for rough detection of lane markings or boundaries in the image in the initial state. Next, the lane boundary detection device 6 receives an image frame (pixels arranged in rows and columns) from the video camera 1 and expresses road surface signs (lane markings or roads) represented by the positions of the rows and columns in the image. Boundary) Check the region of interest in the image to detect the position of the image features 7L, 7R. Next, the lane departure detection device 9 identifies the corresponding position on the road surface using the row and column positions of the road surface sign image features detected in the image 3. Each such corresponding position is represented using the lateral offset and the forward distance of the road markings 8L, 8R relative to the video camera 1. The lane departure detection device 9 specifies the lateral position of the video camera 1 (and the vehicle 2) relative to the position of the road surface sign. If the lateral position is too close to the position of the road surface sign, the lane departure detection device 9 generates a warning accordingly. The controller 5 then uses the position of the road marking image feature detected in the current image as the region of interest for detecting the road marking image feature in the next image frame, in the next image frame from the video camera. The road sign detection (by the lane boundary detection device 6) and the vehicle position detection (by the lane departure detection device 9) are repeated.

FIG. 2 is a flowchart of the steps of the all lane line detection and lane departure warning process 20 performed by the controller 5 according to one embodiment of the present invention. Process 20 includes the following steps.
Step 21: Receive an image frame.
Step 22: A region of interest (ROI) for detecting road marking image features 7L, 7R corresponding to road markings 8L, 8R (eg, lane markings, road boundaries, etc.) is determined in the image.
Step 23: Detect road marking image features from pixels in the region of interest.
Step 24: A road projection map is created by executing a geometric transformation for projecting the image position of the road surface sign to the corresponding position on the road.
Step 25: Find the position of the video camera from the road marking projected on the road.
Step 26: Determine whether the video camera / vehicle is too close to the road marking. If it is too close, go to step 27. Otherwise, return to step 21 to receive and process the next image frame from the video camera.
Step 27: Issue a warning or give assistance and return to Step 21.

One embodiment of the process 20 performed by the controller 5 includes the following steps.
・ Receive an image of the road ahead.
Use previous image information, if available, to determine one or more regions of interest (ROI) in the image where road signs or road boundaries are expected to be present.
Process the image row by row in each ROI to identify and eliminate any unwanted image artifacts (eg images of reflected light, sunlight streaks, tar bands, etc.).
Process the ROI line by line to detect road markings and road boundary fragments.
-In broken line type road markings (road markings formed by dashes and gaps), the marker fragments detected in the gap between the dashes are eliminated.
Filter the sign / boundary fragment (eg, detect and eliminate image features representing snowfall, etc.).
・ Collect the remaining sign fragments in each row into lane line candidates and road boundary candidates.
Eliminate false signs (eg, detect and eliminate tar bands, arrows, and symbols) to get final road markings and road marking fragments with (row, column) coordinates in the image.
• Detect objects outside road markings as they may be important objects.
Project points (rows, columns) of road markings (road marking image features) in the image to corresponding road marking positions on the road surface (forward distance and lateral offset) relative to the video camera.
Filter the corresponding road surface marker position data for smoothing and noise removal.
• Associate the smoothed road marking position data with the video camera position.
• Identify if the video camera (vehicle) is too close to the road boundary or lane marking.
• If the video camera (vehicle) is too close to the lane line or road boundary, the extent of the danger posed by the object that is outside the lane line or road boundary (eg off the lane or road) ), And as a function of the type and number of road markings, a signal is generated that selects between warning and assistance.
Predict the image position, i.e., ROI, for searching for the next road boundary / road surface sign in the road surface and determine the degree of uncertainty in that prediction.
The device has two modes called search mode and tracking mode. In the search mode, the device has not yet determined the approximate location (ie, ROI) in the image where the road marking will be, and therefore looks for the road marking in a large image area. In the tracking mode, the device has already found a road sign in front of the current image frame, and therefore expects a road sign to be again in the same image area or ROI as before, within that ROI. Track road markings. One tracking is a group of position measurements. A new tracking begins when one group of position measurements ends and another group of position measurements begins. When tracking is aborted (for example, because no road markings were detected in the number of images while the vehicle is traveling a certain distance), the device returns to search mode, and a new road marking is To search, the window or ROI for finding the road boundary / road marking is returned to its maximum size and default position. Thus, the position of the road marking need not be predicted based on (relied upon) the road marking detected in the previous image.
If the next image frame is expected to be an image of difficult road conditions (eg rain, snow), reduce the size of the region of interest in the next one or more image frames, thereby Address the road condition, including the step of keeping track of the current road sign (ie, not tracking the new road sign).
Check for clutter in the image (a variety of sources that generate false positives). If clutter is present, the clutter is dealt with by not accepting the detection in the clutter until the clutter is reduced again.
・ Check whether it is daytime or nighttime. If necessary, the device parameter settings are switched to meet general lighting conditions before the next image frame is processed.
The curvature measurement module 12 measures the road curvature using the detected lane marking or road boundary information, and stores the curvature value.
Identify the type of detected road sign by checking how far and how long the road sign is detected in the last image frame.
• If necessary, the detected information (eg vehicle position relative to road markings / road boundaries, road markings, lanes, number and types of objects detected outside road markings / road boundaries, road curvature) Etc.) and output.
• Perform real-time camera calibration and update the initial yaw and pitch angles. The updated yaw and pitch angles are used to more accurately project the position of the road marking image feature detected in the image to the corresponding position on the road. This makes the warning more accurate.
Check any characters and symbols on the road ahead and identify where they will appear in the next image frame.
Check the reflected and sunlight streak images and remember their position and their predicted position for the next image frame (this will cause the image of the lane markings in the next image frame Improved frame detection).
If there is a dashed type road marking, learn the dash-gap pattern and store the predicted dash-gap position for the next image frame.
From the predicted next dimension (in the road surface), determine the ROI for the road marking / road boundary in the next image frame in the image plane.
Repeat the above steps for the next image frame received from the video camera 1.

FIG. 3A shows a plan view of the position of the video camera 1 relative to the road markings 8L and 8R and an example of the dimensions of the road markings 8L and 8R. Those skilled in the art will recognize that other video camera placements and road marking dimensions are possible. For a better understanding of the road marking / road boundary detection process, in FIG. 3B, the pixel p in the image plane and the corresponding point P on the road surface projected as a pixel p via the video camera 1 are shown. An example for explaining the relationship between them is shown. The image 3 of the road 4 is superimposed on the road surface 40 of the road 4. The first pixel p _{1 at} the first row-column position in the image 3 corresponds to the first point P ₁ on the road surface. Similarly, the second pixel p _{2 at} the second row-column position in the image 3 corresponds to the second point P ₂ on the road surface. The step of geometrically converting (projecting) the pixel p _{1 at the} image position (r, w) to the point P ₁ at the position (x, y) on the road surface is based on the vertical coordinate r. Finding the directional coordinate x and finding the lateral coordinate y based on the column coordinate w is included. A similar transformation is applied to point P ₂ and pixel p ₂ .

Therefore, given the points P ₁ and P ₂ on the road markings 8L and 8R, the distance from the video camera 1 to the respective points P ₁ and P ₂ (ie, the longitudinal distance d on the road ahead of the video camera). = X and the lateral offset distance LO = y) from the road centerline, the row-column positions (r, w) of the corresponding pixels p ₁ and p ₂ on the road marking image features 7L, 7R in image 3 ). FIG. 3C shows a trapezoidal region of interest (ROI) superimposed on image 3. Within the ROI, a gray level gradient and optionally an average absolute deviation are extracted from each pixel. Both slope value and mean absolute deviation are characteristics of a group of pixels, the former representing the contrast between adjacent pixels, and the latter representing the variation from the median in one group. ing. Each trapezoid (region of interest ROI-L, ROI-R) is generally placed at a position in image 3 where road marking image features 7L, 7R are expected to be present. The road image 3 and the vehicle speed are inputs to the lane boundary detection device. Also, the geometry for camera calibration (focal length, mounting height, etc.) is known.

  As shown in FIG. 3C, a trapezoidal region of interest (ROI) or window is defined in the image 3. Within the ROI, the pixel gray level gradient and mean absolute deviation are identified. Each trapezoidal ROI in image 3 corresponds to a parallelogram on road 4 (e.g., regions of interest ROI-L and ROI-R are transverse to road 4 as shown in FIG. 3D). Respectively corresponding to two parallelograms 160L and 160R spaced 3-4 meters apart, each parallelogram having a width of about 0.7-2.3 meters and about 14 meters, for example And their lower end 162 and upper end 164 are at a distance of about 6 meters and about 20 meters from the video camera 1, respectively). The size of each ROI in the image, given by the width Wt and the height Ht, is large enough to accommodate the error in the expected position of the road marking image features 7L, 7R within it. If the approximate location of the road marking image features 7L, 7R is not known, the width of the region of interest (ie, the window) is maximized. FIG. 3E shows another example of a rough region of interest when a lane boundary is predicted in advance (dimensions are indicated by dimensions on the corresponding road surface).

  In another example, each ROI corresponds to a parallelogram on the road having a width that is at least four times the minimum road marking width (but up to 2.3 meters wide). The upper limit of the width is limited by the calculation capacity. Each ROI covers a distance on the road 4 starting from a position of about 0.7 meters on the left (or right) side of the video camera and reaching a position of about 3 meters. That is, most of the road lane width and vehicle width are covered. The widths of the regions of interest ROI-L and ROI-R are maximum in the road marking search mode and smaller in the road marking tracking mode. The width of the road marking in the tracking mode depends on the degree of uncertainty in predicting the lateral position of the road marking in the image. The region of interest is arranged in the image symmetrically with respect to the vanishing point VP of the road in the initial state. In the tracking mode, road markings or boundaries are detected in the ROI of the current image frame. Accordingly, the ROI of the next image frame is placed in an area where a road sign or boundary is expected to be present in the next image frame based on the position of the road sign or boundary of the current image frame.

The actual size and periodicity of the road markings 8L, 8R is based on the focal length FL of the video camera and the height h of the video camera relative to the road 4, and the corresponding road marking image features 7L, 7R in the image plane 201. Converted to size and periodicity (FIG. 4A). Specifically, the vertical distance d = x on the road ahead of the video camera to the pitch angle (tilt angle) α of the video camera 1, the height h of the video camera, and the point 203 (P) on the road 4. The relationship between the vertical pixel size Vp, the focal length FL, and the row coordinate r (measured relative to the optical axis 205) in the image plane is described by equation (1).

Furthermore, the pixel p (horizontal pixel size Hp and vertical pixel size Vp (FIG. 4B)) of the row coordinate r and the column coordinate w is a point of the lateral coordinate y (FIG. 3B) on the road ahead as follows: Projected on.
w = y * (FL ² + r ² * Vp ² ) ^1/2 / Hp * (h ² + d ² ) ^−1/2 (2)
Rewriting this equation gives the following:
y = w * Hp * (h ² + d ² ) ^1/2 / (FL ² + r ² * Vp ² ) ^1/2 (3)

  The column coordinate w and the horizontal coordinate y are measured relative to the optical axis of the video camera 1 or a straight line obtained by vertically projecting the optical axis of the video camera 1 on the road ahead. When the video camera is mounted on the vehicle at a constant yaw angle, the column coordinate w and the lateral coordinate y are measured with reference to the lateral position of the vanishing point of the straight road.

  If the horizontal coordinate y is taken as the minimum road marking width, if the distance to the road marking point is known, the vertical distance d on the road ahead of the video camera is determined, and then exists within the minimum road marking width. The column coordinate w corresponding to the number of pixels to be determined is determined. The number of pixels changes according to the vertical distance d on the road ahead of the video camera, and accordingly on the image plane, according to the corresponding row coordinate r. The vertical pixel size and the horizontal pixel size may be different from each other.

  Once the size of the minimum width indicator, expressed in pixels, is determined, smaller (less pixels) image features are filtered from consideration by filtering with the appropriate size in pixels. Can be removed. Note that the width of the road marking, expressed in pixels, varies with the distance from the video camera to the road marking point. When changing the perspective distance (PD), the farther the same size road surface sign 8L or 8R is from the video camera, the less it occupies on the image, and the closer it is to the video camera, Occupies relatively more pixels. In other words, for a certain minimum road marking width, a shorter filter length is used in rows near the top of the image than in rows near the bottom of the image. Therefore, the filter length calculated from the minimum road marking width changes according to the position of the row in the image. Even if the road marking width is the same, a shorter filter length is used in the upper row (distant on the road) than in the lower row (near on the road). A road marking image feature typically consists of one or more pieces (eg, one or more pixels in each row) in each row. The road marking fragments from each row are grouped into overall road markings and road marking image features 7L, 7R in each image.

The video camera creates an image of a road sign in front of it. Curve fitting is used to identify the lateral position of the video camera relative to the road markings at a forward distance of 0 (and thus just to the right and left of the video camera position). The above steps give the road surface two sets of points P (one on the road marking 8L and the other on the road marking 8R). Each set is represented by (x _i , y _i ). FIG. 5 shows an example of two points A and B in one set. The point A is at a position on the road 18 meters ahead of the video camera 1, and the point B is at a position on the road 10 meters ahead of the video camera 1. A fitting line segment 50 connecting the two points A and B is extended from the video camera located at the origin x = 0, y = 0 (zero distance) to a position at a distance of 0 meter in the vertical direction. That is, the points A and B on the road sign ahead of the video camera are connected by the matching line segment 50, and the road sign is backprojected to the same position in the vertical direction as the video camera (forward distance 0, ie, on the y axis). Has been.

The example of FIG. 5 is more general in that more than one point P can be used in each set, and thus more curved (higher order) curves can be used besides straight lines (primary curves). This is one special case of backprojection. There are many possibilities for this curve, but the parabola (conic curve) works effectively as a fitting curve. Specifically, a least-squares parabola fit is applied to the set of points P (x _i , y _i ) and its curve (parabola) is estimated towards the direction of the zero distance video camera. Thereby, a lateral position 52 (on the y-axis) of the road marking relative to the video camera is provided. Different estimations may be performed on the set of points P (x _i , y _i ) on each of the road surface signs 8L and 8R. In order to make a more accurate estimation, it is advantageous to select the closest point in the set (eg point B in FIG. 5) as close as possible to the video camera 1.

  Thus, the ROI in image 3 (used to detect the pixel p of the road marking image feature, which is geometrically transformed to a point P on the road marking) is preferably as close as possible to the bottom of image 3. . A set of pixels p of the road marking image feature detected on the image plane is geometrically transformed into points (positions) on the road surface, and one curve is fitted to those points P to obtain a video camera. To the same vertical position, and the lateral distance of the video camera relative to the road marking on the road is identified. For example, as shown in FIG. 5, the lateral distance of the video camera 1 relative to the road sign is considerably shorter at the same longitudinal position as the video camera 1 than at a farther longitudinal position. (In some cases, it is too short).

Next, in one example, the lateral distance calculated relative to the detected road marking is compared to a distance threshold to identify whether the video camera is too close to the road marking. The distance threshold is usually set to allow time for the driver or assistance device to react, i.e., to allow the vehicle back toward the center of the lane and away from the road sign. Taking into account the reaction rate, a warning may be issued when the following conditions are satisfied:
Time remaining before crossing with road sign <Time threshold

  Typical human reaction time is on the order of 1 second, which is also generally a good value for the time threshold.

In order to identify the time left before the intersection with the road marking or lane boundary (TLC), the distance from the video camera to the road marking in the image frame f is numerically calculated in a series of consecutive image frames. Differentiated. Thereby, the speed of the video camera toward the road surface sign in the image frame f is obtained as follows.
Speed (f) = [distance (f) -distance (f-1)] / image frame period toward the road surface marker

In other words, the speed toward the road marking is equal to the distance in the current image frame minus the distance in the previous image frame divided by the time spent between the two measurements. In order to remove noise, low pass filter processing may be performed as follows.
Smoothing speed (f) = F * Smoothing speed (f-1) + (1.0-F) * Speed toward road surface marker (f)

As an example, the value of F may be about 0.8 or 0.9. The TLC (f) for the road marking in the image frame f is specified by the following equation.
TLC (f) = distance to road marking (f) / smoothing speed (f)

Then, the warning / support rule based on time can be expressed as follows.
A warning is issued when TLC (f) <time threshold.

  Even when the vehicle is already crossing a road sign, there is usually room for returning to the lane, and the warning is still valid, so in such cases, Another warning activation rule may be used. Generally speaking, the purpose in this case is to issue a warning to assist the driver in returning in time to the lane or road. This warning may be acoustic, but may be tactile (sensory) transmitted through the vibration of the handle, for example. The support is mechanical and either actively steers to return into the lane or encourages the driver to steer. Other examples are possible.

  A more natural sensory device is formed when using curvature information from the curvature measurement module 12 to determine where to give warnings or assistance. Many drivers “cut” the inside of the curve with their vehicle, allowing the vehicle to drive more linearly. The controller 5 associates the “cut” amount with the curvature and increases the “cut” amount as the curve becomes tighter.

  The type of road markings 8R, 8L affects the nature of the warning / support activated. For example, a stronger (eg, larger, longer, more elaborate) warning / assistance may be given when crossing a double solid line type road sign than when crossing a single dashed type road sign . A set of rules associates the type of road marking with the nature of the warning / assistance that is activated. The type of road marking is identified by checking how often (over a period of time) and how long (in a single image frame) the road marking image features are detected.

  The lane boundary detection device 6 attempts to detect image features of a plurality of road marking fragments from information in a single image in order to provide better warning / support, and a plurality of image features detected in the same image. (E.g., first using the image feature of the road sign fragment with the highest contrast and then using the image feature of the brightest road sign fragment). Even in low-contrast cases (eg during cloudy days), further improvements may be achieved by detecting other road markings in addition to lane markings. In the case of low contrast, the lane boundary detection device 6 may detect a difference in texture (measured using the average absolute deviation) in the image. Checks the likelihood of detecting road markings in the current image frame based on the expected degree of deviation and image position relative to the image position of the road marking detected in one or more previous image frames Is done. When a lane division line or boundary assembled from individual lane division lines / boundary fragments is detected in the image, the lane boundary detection device 6 outputs the position of the lane division line / boundary.

  An improved road marking detection scheme may be performed in parallel with simpler detection schemes (eg, Non-Patent Document 1) and may in principle take precedence over such simpler detection schemes. In the improved road marking detection scheme, if a road marking is found, the output of the simpler detection scheme may not be used.

  One example of a process according to the present invention begins with the prediction of the position of the road marking in the image plane and the sum of the pixels p (r, w) for each column in the expected direction of the road marking image feature 7L. (FIG. 6) and enhancing the signals representing the points P (x, y) on the road surface corresponding to those pixel locations. This process involves finding a road marking edge candidate that transitions from dark to light and light to dark in this enhanced signal, and edge pairs (paired from dark to bright and light in the same enhanced signal). And checking the distance between edge pairs (the road marking is brighter than the background and constrained in width). The number of edge pairs with appropriate spacing (eg, within a range) is counted. The number of such edge pairs is a measure of the number of road markings. When a plurality of road markings are detected, the closest road marking is taken as an appropriate road marking for measuring the distance between the road marking and the video camera.

  The lane boundary detection device 6 may further perform a process of eliminating image features that do not properly represent road markings. Referring to the road 4 in FIG. 7A, the most common of such image features is that a normally dark tar band 8T is brightened by low-angle sunlight that enters from the front of the video camera 1 ("lights up" )). The second cause is a symbol 7C (for example, a character, an arrow, a pedestrian crossing) on the road (FIG. 7B). These are often parallel to the road markings 8L, 8R and may look the same. The lane boundary detection device 6 detects both the shining tar band 8T and the symbol 7C and removes them from consideration. Based on the standard that the tar band 8T tends to have an irregular width than the road markings 8L and 8R that are lane markings, the lane boundary detection device 6 excludes the tar band as a road marking. . Furthermore, the tar band 8T can only shine when bright sunlight from a bright background is in front. It can be detected.

  By checking whether the change in the width of the tar band 8T is larger than the threshold and measuring the brightness of the entire background, the probability of erroneously identifying the tar band 8T as the road markings 8L, 8R is reduced. The lane boundary detection device 6 performs processing of lane marking line candidate image features (pixels), measurement of the width and length thereof, and processing of image luminance. Based on the above criteria, the lane boundary detection device 6 determines whether the lane marking candidate image features (pixels) represent the road markings 8L and 8R or the tar band 8T. The lane marking candidate image features that appear to be tar bands are not subsequently used. The tar band 8T usually has a larger width variation than the lane marking, and therefore the tar band 8T can be identified according to this characteristic.

  The symbol 7C (FIG. 7B) is usually located between the lane markings, and therefore the symbol region of interest ROI-C is arranged between the lane markings. The presence of the symbol 7C is specified using the symbol region of interest ROI-C arranged in the vicinity of the upper end of the road. As the vehicle moves forward, when a symbol moving downward from near the top of the image toward the bottom enters the symbol region of interest ROI-C, the symbol region of interest ROI-C or window detects the symbol. Symbol 7C is tracked at the proximal and distal distances. All labeled fragments detected in the row between this proximal distance and the distal distance are ignored.

  As described below, when a symbol 7C is detected within a symbol region of interest, the symbol 7C is tracked. Referring to FIG. 8, tracking is done via two quantities. One is a quantity representing the position where the symbol begins in the symbol region of interest (proximal end close to the video camera), ie the proximal distance, and the other is the position where the symbol ends in the symbol region of interest (video A quantity representing the distal end from the camera, ie the distal distance. These two quantities are required because the symbol may have a considerable length. The symbol detected in the next image frame approaches the vehicle (video camera) by a distance equal to one image frame time multiplied by the vehicle speed. The symbol extends far beyond the vehicle (e.g., a long arrow) and may therefore continue to be detected in the symbol region of interest. As long as the symbol continues to be detected at the top of the symbol region of interest, the top of the symbol within the symbol region of interest (distal distance) remains the distance from the video camera to the top of the symbol region of interest, while the proximal distance is 1 For each image frame, it is reduced by a distance equal to the amount of one image frame time multiplied by the vehicle speed. When a symbol is no longer detected at the top of the symbol region of interest ROI-C, the distal distance also decreases by a distance equal to the amount of one image frame time multiplied by the vehicle speed for each image frame.

In FIG. 8, the symbol is detected for the first time at time t ₀ . Symbol, as it enters the symbol region of interest in the ROI-C is the analysis area, Genwashi a longer length, for the first time at time t _1, reveal its entire length (Ln = L2-L1). The proximal end always approaches the vehicle. The distal end approaches the vehicle only when the top of the symbol is finally visible at time t ₁ . The slope of the straight line 54 matches the vehicle speed. All detection of road marking fragment candidates in rows between the proximal and distal distances is eliminated.

  The predictive function of the Kalman filter generally makes it possible to eliminate “falsely suitable” signs without compromising the ability to alert the driver when they are too close to the road sign. Because Kalman prediction can fill in the blanks and produce a useful output signal, even if actual measurements are not available in some image frames (possibly by eliminating road marking fragment candidates). is there.

  The lane boundary detection device 6 may further exclude detection in a gap between dashes of a broken line type road surface sign. In order for a road marking fragment candidate found in an image to be accepted as a true road marking fragment, it must match the dashed road marking model. The uncertainty of the dash position is taken into account. Dashed type road markings include short road marking segments (dashes) with a gap in between. The regularity of the dash-gap pattern can be learned and used to eliminate spurious road marking fragments that are present in the gap. The process includes superimposing dash-gap patterns from multiple image frames on the basis of vehicle speed information, thereby enhancing the dash-gap pattern learning cycle. Based on vehicle speed and expected dash position, detection of image features within the gap in the image is abandoned.

  The abandonment process predicts where dashes and gaps will be in the next image frame, and is in a position where road marking fragments are expected (while allowing some uncertainty) This is accomplished by using only road marking fragments. As a result, a shaped set of broken-line type road marking fragment candidates is formed and assembled more reliably into the true pattern of broken-line type road markings. The elimination of candidate road marking fragments in the gap is applied only after a dash-type marking pattern identified by periodic disappearance and reappearance of dashes over a period of time is detected.

  An example of application of dash-gap prediction (model) is shown in FIG. The road marking structure 169 (shown at the left end) coming from the image is passed through a filter 171 (shown in the center), resulting in a filtered dashed type road marking 173 (shown at the right end). Is generated). Block 177 represents a fragment of the actual dashed type road sign found on the road. Some of them may be noise. By using the filtering process, objects similar to road markings in the image that are in both the gap between dashes and the uncertainty region 175 are not used to construct road markings from the image (for curve fitting). Not used). Tolerance of road markings, while allowing a certain amount of uncertainty, exclude road marking candidates within the gap between dashes [eg, the degree of such uncertainty is the position of each expected dash A road marking fragment candidate found within the area (not in the gap) is acceptable).

The lane boundary detection device 6 may further predict the ROI in the next image frame. This is accomplished by using a large default ROI in the image (detects road markings within it). By using a large area, it is possible to detect objects that look like lane markings but are not. By reducing the size of the ROI and placing the ROI at a location where a road marking is expected to be present based on previous detections, the occurrence of false road marking detection can be reduced. In order to predict the position of the ROI in the next image frame, the measurement position of the road sign in the next image frame is predicted using several previous road sign detections. This can be done using Kalman filtering, but simpler schemes such as alpha-beta filtering may be used. The state variables of the Kalman filter are the 0th, 1st, and 2nd order coefficients of the parabola that best fits the road surface data, and the lateral coordinate y, that is, the lateral distance is expressed as follows.
Horizontal coordinate y = 0th-order term + 1st-order term * forward distance + second-order term * forward distance * forward distance

  The Kalman filtering process smoothes a large number of measured values, predicts the position of the road marking in the next image, and calculates the degree of uncertainty in the prediction. The width of the ROI is set to a size proportional to the degree of uncertainty, but is always maintained at least four times the minimum road marking width. If the ROI is reduced to (1/10) times its original width, the probability of false detection is significantly reduced, and at the same time the computation time is shortened.

  If there are no measurements available for a lane marking or boundary location in a given image frame, Kalman prediction is used. If the measurement remains unavailable, the prediction becomes more uncertain, and so the device widens the ROI, as shown in the example of FIG. When tracking is discontinued, the width of the region of interest is increased, representing increased uncertainty in predicting the location of the road marking. Due to the limited calculation time and because the lane cannot be wider than approximately 6 meters, the width of the region of interest is limited to a certain maximum value. The length of the region of interest is not adjusted.

  Since the state variable of the Kalman filter is in the road surface, the prediction is made in the road surface. The road-to-image geometric transformation described above is used to convert the predicted road marking position on the road into an image position. The region of interest is arranged symmetrically with respect to the predicted boundary position of the image 3. If the predicted value of the road marking position is used in place of the actual road marking position measurement at the critical number of image frames (the device has been operated "in the sky measurement state"), the device Again, return the width of the region of interest to the maximum. This empty state is monitored for two empty state quantities. The first is the number of image frames that have elapsed since the last actual measurement was available, and the second is the distance that has passed since the last measurement was available. (Product of vehicle speed and accumulated image frame time). When either of these two empty state quantities reaches a critical value, the region of interest is reset to its maximum width. Assuming typical highway operating speeds, a typical value for the critical distance for air travel is approximately 20 meters.

  The Kalman filter is only activated when there is sufficient certainty that the device is continuously detecting the same road marking. This is done by checking a series of consecutive image frames and confirming that road markings have been detected in at least a certain percentage of those image frames (usually over 60%). Data at three distances (eg, in front of the video camera, approximately 6, 9, 12 meters) is sent to the Kalman filter. These distances are also the positions where Dickmann type sign finders (mainly used to count the number of signs) are placed. If an acceptable road sign is found, the position of the road sign at three distances is calculated, producing up to three (row, column) pixel positions in the image. These pixels are the input to the Kalman filter. If the detected road sign is too short to cover all three distances, only less than three pixel locations are generated.

  In order to determine the start of tracking (coherent road markings or boundaries that can be traced), the road markings must be detected frequently enough at approximately the expected position in the image. If the detection at at least one position is confirmed, the device starts tracking at that position and activates the Kalman filter. The first stage of tracking is usually tracking in a confirmation zone, for example over 15 meters. This tracking is not used for warning or control until it is confirmed as a valid tracking at the end of this verification interval. In this first stage, the device examines the road area in the image in the trajectory predicted from the first detection. The device stores further detection at each of the three detected pixel locations. During this verification interval, the device stores the minimum number of detections at each of the three detected pixel positions. This minimum number of detections is usually the number of detections observed when passing a 3 meter long road marking at the current vehicle speed.

  While passing through the confirmation zone, the road marking position must remain outside the outer edge of the vehicle. At the end of the tracking interval, both of the above conditions are met (ie, the road marking is detected frequently enough and at the expected position in successive image frames) and is empty If the quantity (distance and time) does not exceed the critical value, the device continues to track the road marking and the warning / assistance activation for that road marking is enabled. If one of the empty state quantities is too large, the device stops tracking the road marking and returns to the search mode for that road marking.

  During tracking of road markings, the device stores a history of detection at each position in the latest travel section. The length of the travel section in which the history is stored is usually 18 meters. At each image frame time, when there is no detection in the image frame, or when the sum of the detections in all three pixel positions within the history interval is less than a certain threshold value, the road markings in the tracking Is not found and the empty interval is incremented. If the empty section exceeds a certain threshold, the device stops tracking this road marking and returns to the search mode for that road marking.

  Therefore, by using the Kalman filter, it is possible to predict the position in the image where the road surface marker should be searched, calculate the size of the required image evaluation area, and provide a smoothed output signal. Probably the optimal mechanism is provided with sufficient characteristics to make the best use of the machining input signal.

  There are many situations that can make it difficult to find lane markings or road boundaries. The lane boundary detection device 6 senses such a difficult situation and improves the device performance in such a situation. Clutter in the road image results in the detection of too many image feature signals. In particular, if two of those signals are very similar (eg, the intensity is approximately equal, the places where they are expected to be close are close, the characteristics are similar, etc.) It may be difficult to select one signal. A second difficulty may arise when the signal suddenly becomes too weak to follow. A special third difficulty is snowfall that appears to be road marking fragments and should be removed from consideration when detecting road markings. Since the reflected light from the road looks like long white streaks on the road, the fourth difficulty is the reflected light from the road that looks like road markings. Therefore, it is necessary to identify such reflected light and exclude the reflected light from consideration when detecting a road surface sign.

  In order to process an image containing clutter, the clutter detection process is performed at approximately the middle position between the proximal detection position and the distal detection position (bottom and top of the image, respectively), as shown in FIG. It includes the step of placing a special region of interest covering almost the entire width, ie the clutter detection region of interest ROI-CD. Data from the clutter detection region of interest ROI-CD is fed into a Dickmann type sign finder in order to obtain the position of the current road sign in each image frame and the position of further road signs that may be present. Such outputs are used to detect clutter (confusing signals) and persistent tracks (marked fragments) that are more closely interleaved to be switched. Such additional road markings (clutter) can cause confusion to the device. Most of them are unable to form a clear signal, i.e. a track to follow, when they change position (especially intermittently). Examples are labeled fragments that are rarely present (not long enough to give a credible detection signal), or randomly present (close enough to each other to give a consistent detection signal). No) is a labeled fragment.

For this purpose, the lane boundary detection device 6 checks the detection signal from the clutter detection region of interest ROI-CD, and during the detection of the clutter, the following three clutter quantities, that is, are presently being tracked. No, the number of beacon detection signals that may disrupt tracking, the timing frequency of the detection signals (e.g., sometimes present but too intermittent or infrequent to form a valid track, etc.), And at least one of the image feature position variations that generate the detection signal (large variations interfere with the formation of a valid track). The clutter detection function of the lane boundary detection device 6 looks for the presence of such signals that can disrupt the device, and eliminates all detections that could cause confusion and thus mistracking. Based on the clutter amount calculation described above, the lane boundary detection device 6 executes at least one of the following types of processing of the sign detection signal.
Detection of a signal to be tracked when no clutter detection signal is present.
Detection of a signal to be tracked when there is a clutter detection signal that does not give a valid track.
There are other detection signals that form a valid track, but the valid track is not as close to the camcorder as the current track (the alternate track on which the device must not switch on the warning Detection of the signal to be tracked, if any).
There is another detection signal that forms a valid track, which is closer to the video camera laterally than the current track (the device is an alternative track on which the warning should be switched on) Detection of signals to be tracked.

  In addition, the tracking function of the lane boundary detection device 6 can detect situations where the signal strength of the image may drop sharply, such as dark shadows, and another, non-authentic signal can be used instead. . For example, the lane boundary detection device 6 checks a particularly dark image area (dark section) using a luminance histogram of such an image area, and such a dark section approaches at a vehicle speed in a plurality of consecutive image frames. And that it reveals that the dark section represents a static area on the road. Before the vehicle enters such a dark stationary area (eg, a stationary shadow), the lane boundary detection device 6 compensates for the temporary disappearance or degradation of road marking image features (detection signals). When the vehicle enters such a dark still area, the lane boundary detection device 6 predicts when the detection signal to be compensated suddenly disappears, so that the size of the corresponding dark image section is small. Check if it becomes. Thereby, the lane departure detection device 9 can avoid an erroneous activation of warning / support accompanying entering / exiting a dark place.

  To reduce the possibility of false detection when in a dark place, the tracking window (region of interest) is made relatively narrow and a new tracking cannot be started. The processing of dark image sections is further dependent on the type of video camera (imager) with respect to image resolution. In a high dynamic range video camera capable of resolving contrast in dark areas, the lane boundary detection device 6 can process dark shadows with relatively high resolution. In order to pinpoint exactly what the dark areas are, lens-imager-vehicle lighting parameterization may be used.

  When falling objects such as snow are present in the scene, image features corresponding to such falling objects must be excluded from consideration by the lane boundary detection device 6 and the lane departure detection device 9. For example, if snowfall occurs only at a sufficiently low temperature, it can be measured. Snowfall, which is white flakes that look like road marking fragments, may generate intermittent sign fragment detection signals. It can be determined by connectivity analysis. If low temperatures are detected, especially if intermittentness persists, the device can eliminate these false “road marking fragments” that do not connect to each other.

  In FIG. 12, the image feature of the true road marking fragment among the image features 7S (“+” sign) of the road marking fragment candidate is in the vicinity of the road marking (broken line type or solid line type) image features 7L and 7R. An example is shown where the image features that are dense and not are separated from the road marking image features 7L, 7R. The lane boundary detection device 6 functions to eliminate an isolated pixel that is identified as a road marking fragment candidate from the road marking fragment detection signal, but is actually likely to be a snowfall piece. Therefore, in order to improve the detection accuracy, it is preferable to appropriately identify the broken line type road surface sign and the solid line type road surface sign.

  In the embodiment of the controller 5 in FIG. 13, the lane boundary detection device 6 displays an image feature representing a reflected light image from the road surface (reference numeral 19 indicates a pixel luminance pattern of the reflected light image candidate) as a road sign. The apparatus includes a reflected light image detection device 6b that is excluded from consideration in detection (FIG. 13 further shows a street lamp 14 and another vehicle 7 on the road 4 in front of the vehicle 2 on which the video camera 1 is mounted. ) As described above, a reflected light image that looks bright and looks like a vertical streak similar to a road marking may confuse the lane marking detection device. The reflected light image detection device 6b checks, as a reflected light image candidate, a pixel luminance pattern 19 extending vertically in the image 3 that transitions from dark to bright to dark in the horizontal direction with high contrast. The reflected light image detection device 6b further checks the length and width of these high-contrast pixel luminance patterns 19 and eliminates pixel luminance patterns that are too wide or too short. The reflected light image detection device 6b identifies an image portion (for example, the pixel luminance pattern 19) that should not be used for road marking detection analysis. Since the reflected light image is nearly vertical, it may be convenient to store these image portions only at their column positions, but it is possible that a single pixel that is a candidate for the reflected light image is ignored. is there. Alternatively, when this reflected light image detection device 6b is applied, an image feature representing a symbol on the road is detected and image analysis is performed, and a specific symbol or character representing a preset warning is recognized. The driver may be configured to notify the driver of the presence of the sign by voice.

  The controller 5 (the lane boundary detection device 6 and the lane departure detection device 9) is further adapted to a new or more appropriate road surface when two or more road surface signs are detected in one image frame. Determine if the sign is being tracked (followed). For example, if a road sign is detected more frequently than the currently tracked road sign, the controller 5 determines which road sign to track. Generally speaking, the innermost (closest) road marking to the video camera is the road marking that defines the lane in which it is traveling. If the controller 5 detects a road sign closer to the currently tracked road sign, the controller 5 can switch to tracking the road sign closer to it. For example, when a new road sign is being added to switch from an older and disappearing road sign that is currently being tracked, it is necessary to switch to tracking a different road sign.

  In one example, the following conditions: controller 5 is currently following a road sign, detection signal containing clutter is minimal, and a new road sign has been detected in a sufficiently large number of image frames. Whether the ROI does not include multiple road markings or multiple road markings, the new road marking is closer to the video camera and the new road marking is at least constant That it is detected at a mileage (for example 15 meters) or that an existing road sign is not detected at that constant mileage, that a new road sign is present over the constant mileage More frequently detected, new road signs are not too close to the video camera (to avoid false warnings) It, when all conditions are present in that the width of the lane that corresponds to the new road markings is reasonable, the controller 5, the track road marking existing (current), switch to tracking a new road marking. Other tracking switching examples are possible, including some combination of these conditions (or other conditions).

  Furthermore, what appears to be a new road sign may actually be the current noisy road sign. As described above, when a road sign is noisy or missing, a large uncertainty may occur at the predicted position of the road sign in the next image frame. This uncertainty region is called the acceptance region or gate. 14A and 14B show two examples of tracking road markings detected in successive image frames. The first example of FIG. 14A represents a detection signal (corresponding to a road marking candidate pixel p) in a track with little noise. The size of the gate 60 (one gate for one image frame) is relatively small, and the measured value is close to the predicted value. The second example of FIG. 14B represents a detection signal in a noisy track. The size of the gate is relatively large and the measured value is only in the gate 60 in the first two image frames (the measured values in the three image frames are outside the gate 60a centered on the predicted value). is there). If the measurements are consistently outside the gate, a new track may have started and the above switching may occur. The unsatisfactory situation where the measured value exists outside the gate or the road marking (measured value) does not exist is handled using the predicted value instead of the measured value.

  The lane departure detection device 9 further defines when crossing the road marking (or crossing) will yield certain serious consequences. The result is more severe in some cases than others. In one example, three parameters are used to determine if crossing the road marking is critical. Those parameters detected from the road image include the number of road markings, the type of road markings, and the classification of what appears outside the road markings. A road sign (for example, double, solid, perhaps yellow) that means you must not cross should be stipulated to give a stronger warning than crossing a single dashed type sign. Also good. A stronger warning may be issued for entry into an area that includes a plurality of vertical edges outside the road marking in the image. Depending on the three parameters, the warning or assistance given to the driver is changed.

  The type of road marking may be identified by detecting the road marking length in a single image frame. The road marking length is a basic quantity that determines the type of road marking. Solid line type road markings are generally long. On the other hand, the dash of the broken line type road marking has a shorter length. Furthermore, botsdots are very short. An assembly process (connecting road marking fragments with matching straight lines or curves) can be used to find road markings. Road markings have fragments that are almost connected, which can be pinched by gaps (eg where there are no road marking fragments). By detecting the almost connected fragments and describing the gap with the smallest length, the closest and farthest distance from the video camera of such an assembly can be determined.

  The gap between the dashes of the broken line type road marking has a length of at least 2 meters, for example. The gap (missing part) of solid line type road markings is usually shorter and is at most about 0.5 meters, for example. A gap consists of one or more rows in which there are no road marking fragments that contribute to the formation of road markings. Each row of image 3 represents a certain length or section of road 4 that can be calculated from the video camera's geometry information (installation height, pitch angle, etc.) using the geometric transformation described above. Covering. As an example, as shown in FIG. 15, the length of the road marking can be found by examining a list of rows containing road marking fragments used to form the road marking. The road marking length is measured by summing the individual pixel zones forming the road marking. For example, gaps up to 0.5 meters are overlooked. Any longer gaps are gaps between road markings.

  FIG. 15 shows a series of rows labeled A, B, C, E, F, H, I, J, K, and O. The directly adjacent rows A, B, C have road marking fragments inside them. The next row, row D, has no road marking fragment (a gap). This is followed by rows E and F with road marking fragments. Then, in line G, another gap follows. Finally, rows H, I, J, and K have road marking fragments, and in row O, the next road marking fragment continues.

  Rows D, G, L, M, N lacking road marking fragments are examined for the length of the road covered by (corresponding to) each pixel of those rows. For example, row D covers 0.2 meters, row G covers 0.4 meters, and rows L, M, and N together cover 1.9 meters. The length of row D and row G is less than the critical maximum gap value of 0.5 meters, but the combined length of rows L, M, and N exceeds the critical maximum gap value of 0.5 meters. Therefore, the road surface sign extends from the lower end of the row A to the upper end of the row K, and the length is given by the sum of the respective divided areas on the road covered by the rows A to K. Rows D and G that are shorter than the critical maximum gap value of 0.5 meters are connected to their neighboring rows and incorporated into the road marking. In order to detect the presence of road marking fragments, all rows of the region of interest are processed and the above steps are applied to find the road marking length (and start and end points).

Once the road marking length is measured, the type of road marking just measured is specified by the classifier function of the lane boundary detection device 6. This classifier follows a rule that takes a road sign length as its input and generates a road sign type as an output. For example, there can be four road marking types: type 0 = no road marking, type 1 = bottsdots, type 2 = dashed type road marking, type 3 = solid type road marking. An example of a road marking type classification rule may include the following.
If road marking length = 0 meters, road marking type = type 0.
If 0 meter <road marking length ≦ 0.5 meters, road marking type = type 1.
If 0.5 meters <road marking length ≤ 3 meters, road marking type = type 2.
If 3 meters <road marking length, road marking type = type 3.

These classifications of road marking types may be ambiguous because road markings may not appear completely within the ROI and may appear shorter than their actual length. For this purpose, a filtering process is applied to the classification of the road marking type obtained at the end of the run, for example during 20 meters, the maximum value of the road marking type is selected and the final road marking in a single image frame. A classification of the type is given. For example, even if a solid line type road marking classification (ie road marking type = type 3) is detected only once in the last 20 meters, the road marking type is classified as a solid line type road marking. FIG. 16 shows an example of a process 70 for measuring road marking length in a single image frame. Process 70 includes the following steps.
Step 71: Find all road marking fragments in the ROI.
Step 72: Create a list of all rows where road marking fragments are present.
Step 73: Overlook gaps with a length of less than 0.5 meters, for example, and find the longest connected section.
Step 74: Obtain the proximal and distal ends of the longest section.
Step 75: Calculate the road marking length as the difference between the distal end position and the proximal end position.
Step 76: Using the road marking length, for example, four types, ie, type 0 = no road marking, type 1 = botsdots, type 2 = dashed type road marking, type 3 = solid type (continuous) road marking, road surface Classify the indicator type.
Step 77: To obtain the final road marking type, the largest of the road marking types (0, 1, 2, 3) found during the last 20 meters of the run is selected. .

  An alternative identification method of the road marking type is a method of identifying how often a road marking fragment is detected within a certain time. One of the regions of interest ROI-L and ROI-R having a normal size is divided into smaller regions of interest, and road marking fragment detection is performed in the divided regions of interest. In one example, infrequent detection (˜1%) of road marking fragments is associated with botsdots, and more frequent detection (15%) is associated with dashed type road markings, even more frequent detection. (60%) is associated with the solid line type road marking.

  The plurality of road markings is generally a combination of a broken line type road marking and a solid line type road marking. Such a combination, as shown in the example of FIG. 17, sums pixel values from multiple rows and reduces the noise by summing and using the enhanced detection signal to It can be detected by a Dickmann type marker finder. The summed pixel values are processed to identify a luminance gradient having a magnitude that exceeds a threshold (eg, in the range of 1 to 255, preferably in the range of 6 to 24). Next, gradient peaks (maximum gradient) and valleys (minimum gradient) are identified, and the location of the peak and its adjacent valley is examined to determine the spacing between them. The spacing is between the minimum and maximum spacing values allowed for a row [eg between 1 and 100 cm, measured in centimeters, preferably between 7 and 70 cm (this centimeter Is converted to the number of pixels before use)), the peaks and valleys are paired as representing road markings (peak-valley pairs or PV pairs). FIG. 17). Therefore, the number of paired gradient peak-valleys is the number of road markings present in the ROI of image 3.

  By using multiple Dickmann type sign finders and road sign detection at several distances on the same side with respect to the video camera, multiple measurements for the number of road sign presents can be made. As an example, referring to FIG. 18, three sign finders 80 are used per side (one ROI) of the video camera. Each of the sign finder 80 covers the ground by about 1 meter in the vertical direction. Either sign finder 80 detects gaps between sign fragments (eg gaps between dashes of dashed-type road markings) at one side of the video camera (eg region of interest ROI-R), and other sign finders. You may count fewer road markings. The largest of the number of road markings found is chosen as representing the number of road markings present on the measured side of the given image 3.

  Locally scattered gradients can be detected as road markings. For this purpose, the detected number of road markings is subjected to filter processing (time average). As an example, referring again to FIG. 17, the maximum number of road markings (P-V pairs) detected in a given image frame is filtered. For example, the number of image frames corresponding to about 20 meters of vehicle travel is used as the filter length. By this filtering process, the number of detected road surface signs is averaged, and the final number of detected road surface signs is specified.

  In addition to the detection of the road marking described above, as shown in the example of FIG. 19, the image features on the inside and outside of the road marking image features 7L and 7R may be compared for feature analysis. Various quantities used to find road markings are used as inputs or characteristic values for comparison and characteristic analysis. FIG. 19 shows two image regions NR and BBR. The image area NR is in the vicinity of the video camera, and the image area BBR is outside the identified road marking image feature 7L.

  These image areas can be characterized by several characteristics. These characteristics can include average brightness, size, presence of edges, and the like. As an example, by measuring image texture in image regions NR and BBR and identifying image features in those image regions, the presence of edges in those image regions can be revealed. The sign finder computes the luminance gradient of each row of pixels, among other quantities. Some slopes may have a magnitude that exceeds a threshold (eg, in the range of 1 to 255, preferably in the range of 6 to 24). Depending on the extent of the spacing between these sufficiently large slopes (peaks and valleys), the area between them is accepted or rejected as a road marking fragment candidate. The density and median spacing of these sufficiently large gradients in the image areas NR and BBR (otherwise not identified as lane segment line candidates or boundary fragment candidates) are determined by the road marking image feature 7L. It can serve as a property for classifying what is outside / inside. In one example, the density is given as the number of gradients of sufficient magnitude per meter in the lateral direction. The median interval is given by the number of columns.

  One other image characteristic in the image areas NR and BBR that can be used to classify what is outside / inside the road marking image feature 7L is a gradient having a sufficiently large magnitude as the row position changes. It is a way of fluctuation of the median interval between. As shown in the example of image 3 in FIG. 20A, as a row in the image approaches the horizon, road markings and other objects in the road surface typically appear to shrink in length and width. . The median spacing in such objects also appears to narrow as the rows get closer to the horizon. However, an object that is not on the road surface, such as the tree 82, does not appear to have a narrower width or spacing even when the row approaches the horizon.

  As shown in the example of FIG. 20B, to describe the characteristics of the image features corresponding to such an object on the road, the median interval of gradients with sufficient magnitude and the rows of the image The associated least square line is calculated and the slope is stored. Rows between the top and bottom edges of the ROI for label detection (eg, image regions NR and BBR) are used for such calculations. For example, even when a row approaches the horizon, a tree trunk that is substantially parallel to the image plane is kept constant in width, while an object in the road surface is generally not kept constant in width. The fact that the slope of the least square fitting straight line is almost 0 indicates that there is an object having a certain width (in the image plane) such as a tree. The lane departure detection device 9 can use this to issue an appropriate warning / support.

ここで、ｉは、ｉ＝１、２、…、ｍであり、average(p)は、ｍ個のピクセルの輝度ｐ_iの平均値であり、和は、全てのｉにわたって行われる。 Another image characteristic defined by the mean absolute deviation (MAD) of the rows or regions may be used for the detection of objects in the image regions NR and BBR. Smooth areas such as roads tend to have low MAD values, while areas with larger textures such as trees have higher MAD values. MAD is the mean absolute deviation from the mean value. The luminance MAD for a group of luminance p _i of m pixels can be expressed as:

Here, i, i = 1,2, ..., a m, average (p) is the average value of the luminance p _i of the m pixels, sums are performed over all i.

  As an example, as shown in FIG. 21, the MAD values in the vicinity of the video camera in each row are specified, and then the MAD value at a position further away laterally outward from the video camera is detected in the same row. Beyond the sign, it is identified in an image area that extends outward by approximately one lane width (eg, one lane width may be approximately 3 meters). Two MAD values are generated for each row of the image. FIG. 21 shows a comparison of MAD values inside and outside the road marking or boundary. A ratio δ between the inner and outer MAD values in each row is obtained and the ratio δ is averaged. If the value of the ratio δ is approximately 1, it means that the texture of the area outside the road sign is the same as the texture of the area inside the road sign. This property is unique because it requires two regions as input to generate one value.

Therefore, in this example, four image characteristics are considered. The first image characteristic is the gradient density per meter, the second image characteristic is the median spacing between sufficient gradients, expressed in number of columns, and the third image characteristic is sufficient The row change of the median interval between gradients having magnitude, and the fourth image characteristic is the average MAD ratio. Such image region characteristics can be processed in various ways. Two examples are described below.
1. The first three image characteristics in the area near the video camera (image area NR) are compared with the same three image characteristics in the image area BBR outside the road surface sign (to the outside by about one lane width). As an image characteristic comparison of 4, an average MAD ratio is incorporated. These image characteristics are compared for closeness. The first three image characteristics are compared using the Manhattan distance.
2. The database is compared with the first three image characteristics and the average MAD ratio in the area outside the road marking (image area BBR) to identify the category with the most approximate image characteristics in the database.

  In the first comparison process described above, the image characteristic results in the image area inside the road sign evaluate the ease of driving on the road corresponding to the image area inside the road sign, and then based on the evaluation Used by the lane departure detection device 9 to activate the appropriate warning / assistance function. As the vehicle moves forward, the combination of image characteristics described above is monitored so that the vehicle moves forward safely in a passable zone (eg, according to road markings on the road). If the same characteristic vector as this combination of image characteristics (characteristic vector) is found in the zone of the image area BBR outside the road surface sign, the vehicle must enter the zone of the image area BBR safely. It is reasonable to expect that By storing safe characteristic vectors collected as the vehicle moves forward, the device learns cumulatively about safe zones.

  In the second comparison process described above, the image characteristics that are currently measured in the image area BBR are compared with representative and typical image characteristics in the database. This database defines at least two categories (or evaluations): a level at which safe driving is possible and a level at which safe driving is not possible. The database may include other categories, such as a level that may be runnable, ie an intermediate safety level. The database contains all combinations of these image characteristics and assigns a safety measure to each of those combinations.

  The lane departure detection device 9 can use two comparisons as follows. The first comparison shows that there is a considerable difference between the image characteristics measured near the video camera (image area NR) and the image characteristics measured outside the road sign (image area BBR). If so, a second comparison is made. As a final result, it is determined how safe it is to enter the area outside the road marking (image area BBR). Depending on this determination, given as a value between 0.0 and 1.0, the level / type of warning or assistance provided to the driver during the lane assistance situation is adjusted.

The closeness of one characteristic vector to another characteristic vector is measured by the Manhattan distance between them as follows.
Manhattan distance = | | f1_a − f1_b | + | f2_a − f2_b | + | f3_a − f3_b |
Here, f1_a is the first image characteristic of group a, f2_b is the second image characteristic of group b, and f3_b is the third image characteristic of group b. For example, group a may include three image characteristic measurements from a straight road ahead where the vehicle can enter safely, and group b includes three image characteristic measurements from the area outside the road marking. May be included. If the first three of the four image characteristics of group a are sufficiently close to the corresponding three image characteristics of group b, entry into the area outside the road marking is safe. Expected to be. The Manhattan distance can be represented by a two-dimensional graph as shown in FIG. Alternatively, group a may be the average of the image characteristics measured in the area outside the road marking (image area BBR), and group b is a reference point in the database, representing for example grass or guardrail. May be. Group a is assigned the same category in the graph 85 as the most ordered reference feature vector. Symbols (Equation 3) and (Equation 4) represent the first image characteristic difference and the second image characteristic difference between group a and group b, respectively.

  A large Manhattan distance represents a large difference in image characteristics between image regions (groups). Prior to calculating the Manhattan distance, the image characteristics are normalized. This distance measurement is used in the first comparison. Also, a threshold for the maximum difference allowed for the image characteristics of the two image regions that can be considered substantially the same (e.g., in the range of 0-1.4, in the range of 0.2-0.6). Is preferable). In order to be able to determine that the two image regions are approximate, the fourth characteristic needs to be in the range of 0.8 to 1.2.

  If the first comparison indicates that the two image regions are significantly different from each other, a second comparison is activated. In the simplest form, a set of rules is applied to the image characteristic values. The set of rules assigns an identification category and a safety measure to each combination of feature vectors. One example of a rule is as follows: The spacing between gradients of sufficient magnitude in one image area does not change sufficiently for changes in the rows of the image (checked by finding the slope of the fitted line for row-to-median interval data ), This image area is not a travelable area and has a high risk level of 1.0. In a single image area, if the number of sufficiently large gradients per meter in the lateral direction is small and the median interval is small, the risk of this image area is considered to be considerably low. When the MAD value in the image area BBR is large, the degree of risk is high. The above steps receive the above four image characteristics of one image area and in response to the risk (or safety), 0.0 (no danger at all) and 1.0 (very dangerous) Can be implemented as a four-dimensional classifier that gives values between

  FIG. 23 is a graph showing an example of four image characteristics. The first image characteristic (# 1) is 8 gradients with sufficient magnitude over a length of 5 meters, ie 8/5 or 1.6 enough magnitude per meter Is a gradient density consisting of gradients with The second image characteristic (# 2) is a median value of the distances between all adjacent gradient pairs, which is a characteristic value of the distance between adjacent gradients. The third image characteristic (# 3) is how to calculate MAD. The fourth image characteristic (# 4) is the way the approximate change in median spacing (shown via the least squares fit line) as a function of image row. Thereby, it is determined whether what is outside the road surface marker is in the road surface or whether it is upright. As an additional image characteristic, an average gradient amount in the image area may be included.

  FIG. 24 is a functional block diagram of an example of an apparatus 100 that implements another embodiment of the controller 5 according to the present invention. The lane boundary detection device 101 detects the lane marking as described above. The symbol detector 102 checks for signs that may interfere with the lane boundary detector 101 and identifies both boundaries (inside and outside) of each of those signs. The lane boundary detection device 101 ignores image features detected between these two boundaries. The clutter detection device 103 checks for irregular signs, excessive signs, and inconsistent signs that may disrupt the lane boundary detection device 101 and eliminates all detection if necessary. The new sign detector 104 switches the tracking module 106 to check for sufficiently persistent, consistent and closer road signs that the vehicle can follow.

  The camera calibration module 105 analyzes the internal and external geometric characteristics of the video camera. The internal geometry includes quantities such as focal length, pixel size, etc., while the external geometry includes installation height, pitch angle, and the like. The tracking module 106 receives the road marking position measurements, one identified at each side of the video camera, smooths them, predicts the position of the road marking in the next image frame, and still makes a prediction. Is possible (prediction is appropriate within a certain range of uncertainty before reducing its reliability too much). The lane departure detection device 107 determines whether the vehicle is too close to the road surface sign. The input for that is a smoothed position measurement from the tracking module 106. The output is a determination as to whether or not to warn.

  The image area characteristic analysis and comparison device 108 first compares the area in front of the vehicle with the area of approximately one lane width outside the road marking / boundary, and both areas have image characteristics that are similar to each other. Judge whether or not. If both areas have similar image characteristics to each other, it is safe to enter the area outside the road marking / boundary and it is not necessary to issue an excessively strong warning. If the two areas are different from each other, the image characteristics of the area outside the road marking / boundary are compared with the database. Thereby, the area outside the road marking / boundary is classified as a safe area, possibly a safe area, or an unsafe area. This classification thus affects the alerts that are triggered. The warning that is triggered is also affected by the attributes of the road markings, such as the number and type of road markings present. A double solid line type lane marking causes a stronger warning than a single broken line lane marking. The rules combine image area classification and road marking type, and this combination affects the strength or nature of the triggered alert. The road sign type / number / length measuring device 109 determines whether the road sign itself indicates that a departure from the lane is dangerous. The warning module 110 warns or assists the driver to maintain an appropriate lane position or road position. The inputs are warnings and risk decisions, induced or suggested by road markings and areas outside road markings.

FIG. 25 is a flowchart of a process 200 performed by the apparatus 100 including the following steps.
Step 201: Obtain a road image frame.
Step 202: Define the limit of the expected road marking position as an ROI in the image.
Step 203: Eliminate image artifacts (reflected light, sunlight streaks).
Step 204: Find a road marking fragment in the ROI in the image.
Step 205: Eliminate labeled fragments in the gap.
Step 206: Eliminate snowfall sign fragments.
Step 207: Eliminate symbols from detection.
Step 208: Assemble the remaining road marking fragments into road markings.
Step 209: Exclude the tar band from detection.
Step 210: Project image points on a road surface by geometric transformation.
Step 211: Detect clutter.
Step 212: Kalman filter processing is performed on the road surface data.
Step 213: Switch tracks as necessary.
Step 214: Locate the video camera / vehicle relative to the road marking.
Step 215: Is the road sign too close? If yes, go to step 216, otherwise return to step 201.
Step 216: Give warning / support according to the situation.
Step 217: Predict the next position of the lane marking in the next image frame.
Step 218: Predict difficult situation in next image frame.
Step 219: Perform curvature compensation for the detected road surface marker.
Step 220: Analyze the characteristics of the roadway and road markings. Then, the process returns to step 201.

  The roadway scenes look different at night and during the day. Specifically, the different appearances appear as a relatively strong contrast at night, a relatively low background noise level at night, and the like. Such information can be used for analysis of roadway images. Bright scenes require a short exposure time and / or low gain of the imager. Dark scenes require a long exposure time and / or high gain of the imager. By storing exposure time and gain history (eg, 1 minute long), it is possible to identify which of the settings is relatively low / high and short / long. It can deal with “don't know” zones (ie if the history applies to daytime imager settings that exceed a certain number, it is daytime and exceeds a certain number, nighttime If it's the imager's settings, it's nighttime).

  In order to obtain sufficient performance, it is desirable to fully calibrate the video camera. The quantities associated with the video camera used in the geometric transformation equation, i.e. the height, focal length, pitch angle, etc. of the video camera play an important role. Calibration allows internal parameters (such as focal length, optical axis position, parameters related to the video camera, determined by the video camera itself), and external parameters (installation height, pitch angle, distance from vehicle centerline, etc.) The value of the parameter (related to the position on the vehicle) on which the camera is mounted is determined.

  All measurement information regarding the location / type / number of road markings, objects that would be outside the road markings, etc. can be combined for road delineation and characterization.

  A sharp curve does not fit in either a straight region of interest or a straight road marking. Road signs are usually better detected when in the vicinity of a video camera. Therefore, if possible, it is advantageous to select the position and / or shape of the region of interest so that the region of interest follows the curve better and it is easier to detect road marking fragments near the video camera. As shown in the example of FIG. 26, even with road markings having a considerably small radius of curvature, by appropriately positioning the region of interest (eg, region of interest ROI-R) to fit well with the curve, It can be detected better.

  In addition, real-time calibration can be used to learn the pitch angle and yaw angle of an operating video camera. This learning uses the position of a vanishing point (a point where both sides of a straight road appear to be together in a distant place) as an input. On average, a camcorder that is consistently facing sideways will see a vanishing point that is also located on one side. A video camera pointing upwards or downwards will see, on average, vanishing points located relatively upwards or downwards in the image. In any given image frame, it is only necessary to see one road marking or boundary of the road. The learning that takes place while driving affects the location where warnings and assistance are activated, making it more accurate.

The above steps / processes can be performed in real time or can be performed on stored images. This process can be performed on a single CPU having a pre-processing stage, or a plurality of CPUs, and both can be performed in parallel if necessary. Further, as is well known to those skilled in the art, the foregoing example architecture according to the present invention may be implemented in many ways, including program instructions, software modules, logic circuits, application specific integrated circuits, firmware, etc., for execution by a processor. can do.
The present invention is described in considerable detail herein with respect to several preferred versions thereof. However, other versions are possible. Therefore, the spirit and scope of the claims should not be limited to the preferred version description contained herein.

DESCRIPTION OF SYMBOLS 1 Video camera 2 Vehicle 3 Image 4 Road 5 Controller 6 Lane boundary detection apparatus 6a Reflected light image detection module 6b Reflected light image detection apparatus 7 Another vehicle 7C Symbol 7L, 7R Road sign image feature 7S Image feature 8L of a road sign fragment candidate 8R road surface sign 8T tar zone 9 lane departure detection device 10 camera calibration module 11 warning module 12 curvature measurement module 14 streetlight 15 device 17 sampler 19 pixel luminance pattern 20 process 40 road surface 50 fit line segment 52 relative to the video camera, Road sign lateral position 54 Straight line 60, 60a Gate 70 Process 80 Sign finder 82 Tree 85 Graph 100 Device 101 Vehicle boundary detection system 102 Symbol detection system 103 Clutter detection system 104 New sign detector 105 Camera calibration model 107 Vehicle deviation detection system 106 Tracking module 108 Image area characteristic analysis and comparison device 109 Road marking type / number / length measuring device 110 Warning module 160L, 160R Parallelogram 162 Lower end 164 Upper end 169 Road marking structure 171 Filter 173 Filtered broken line type road marking 175 Uncertain area 177 Block 200 Process 201 Image plane 203 Point 205 Optical axis BBR, NR Image area d Longitudinal distance FL on road ahead of video camera Focal length h Video camera height Hp Horizontal pixel size Ht Height LO Horizontal offset distances p, p ₁ , p ₂ pixels P, P ₁ , P ₂ points from road center line Row coordinates ROI-C Symbol region of interest ROI-CD Clutter detection interest Region ROI-L, ROI-R Region of interest t ₀ , t ₁ Time Vp Vertical pixel size VP Vanishing point w Column coordinate Wt Width x Vertical coordinate y Horizontal coordinate α Pitch angle

Claims (9)

A method for warning a lane departure based on a video image for a vehicle on a road using an apparatus configured to perform at least the following steps:
Receiving an image of a road ahead of the vehicle from a video imager;
A road marking detector detecting in the image one or more road markings corresponding to the road marking on the road;
A characteristic analysis module analyzing the characteristics of the image area to determine an evaluation of road drivability corresponding to the image area outside the detected road marking;
Offset detecting means detects a lateral offset on the road of the vehicle relative to the road surface sign based on the detected road surface sign;
A lane departure detector generates a warning signal as a function of the lateral offset and the evaluation;
The step of detecting one or more road markings in the image further comprises the road marking detector determining a region of interest in the image and detecting the one or more road markings in the region of interest. A road surface sign attribute detecting means for identifying an attribute of the road surface sign,
The step of identifying the attribute of the road surface sign further includes the step of the road surface sign attribute detection means identifying the type and number of the detected road surface sign,
In the step of analyzing the characteristics of the image area outside the detected road sign, the characteristic analysis module includes a characteristic vector composed of a combination of a plurality of image characteristics inside the road sign and the same outside the road sign. If there is a similarity between the feature vector comprising a combination of a plurality of image characteristics, it is determined that the vehicle is able to enter the area outside the secure the road marking, further, on the outside of the road markings A plurality of image characteristics are compared with the image characteristics in the database of the image area, and based on this database, including a level that allows safe driving and a level that does not allow safe driving outside the road surface sign, We have at least two rows of the evaluation of,
The step of generating the warning signal further includes the step of causing the lane departure detector to generate a warning signal as a function of the attribute in addition to the lateral offset and the ease of driving evaluation. And
The step of detecting the road surface sign is further characterized in that a symbol detection means detects an image feature representing the symbol on the road, identifies such an image feature, and the image feature is a symbol representing a preset warning. or if the character is a label, and wherein the free Mukoto the step of informing the driver of the presence of said labeled by speech. The method of claim 1, wherein the road marking detector further comprises tracking the detected road marking. The road sign detector further comprises selecting a set of road signs from the detected road signs based on the validity of the detected road signs. the method of. The method of claim 3, wherein the road marking detector further comprises tracking the selected road marking. The step of detecting the road marking further comprises the step of the road marking detector detecting clutter in the image and removing such clutter from consideration of road marking detection. 5. The method according to any one of 4 above. Image processing means configured to receive an image of a road ahead of the vehicle from a video imager;
A road marking detector configured to detect in the image one or more road markings corresponding to road markings on the road;
A characteristic analysis module configured to analyze characteristics of the image area to determine an evaluation of road drivability corresponding to the image area outside the detected road surface sign;
Offset detecting means configured to detect a lateral offset on the road of the vehicle relative to the road surface sign based on the detected road sign;
A lane departure detector configured to generate a warning signal as a function of the lateral offset and the evaluation;
The road marking detector is further configured to determine a region of interest in the image and to detect one or more road markings in the region of interest;
The characteristic analysis module has an approximation between a characteristic vector composed of a combination of a plurality of image characteristics inside the road surface sign and a characteristic vector composed of a combination of the same plurality of image characteristics outside the road sign Determining that the vehicle can safely enter the area outside the road sign, and comparing a plurality of image characteristics outside the road sign with image characteristics in the image area database; Is configured to perform at least two classifications including a level at which safe driving is possible and a level at which safe driving is impossible outside the road surface sign,
Detecting an image feature representing a symbol on the road, identifying such an image feature, and if the image feature is a symbol or character sign representing a preset warning, the presence of the sign by voice A lane departure warning device based on a video image for a vehicle on a road, further comprising symbol detection means for informing the driver of the vehicle. The apparatus further comprises road marking attribute detection means configured to identify an attribute of the road marking,
The apparatus of claim 6, wherein the lane departure detector is further configured to generate a warning signal as a function of the attribute in addition to the lateral offset and the evaluation.   The apparatus according to claim 7, wherein the road marking attribute detection means is further configured to identify a type and number of the detected road markings.   9. The apparatus of any one of claims 6-8, further comprising a tracking module configured to track the detected road marking in one or more road image frames.

Images