JP2007299414A - Lane marker recognition method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an algorithm capable of stably recognizing a lane marker composed of white lines or road studs without distinguishing the type of the lane marker. <P>SOLUTION: In a lane marker recognition method comprising an image input process for inputting an image including a lane marker, an edge extracting process for extracting a luminance change point in the input image input in the image input process, and a lane marker estimating method for estimating the position of the lane marker by using edge points extracted in the edge extracting process, the edge extracting process includes a means for calculating the angle of orientation of each of the edges, and the lane marker position estimating process includes a means for extracting edge points that are oriented toward a vanishing point of a road from among the edge points extracted in the edge extracting process. Consequently, since the lane marker can stably be recognized without distinguishing the type of the lane marker, there is no need of replacing an algorithm and even the unknown shape of a lane marker can be made to correspond to. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a lane mark recognition method for recognizing a vehicle traveling lane mark by image processing from road surface information obtained by an input means such as a camera, and an apparatus using the same.

  Conventionally, lane marks such as white lines laid on the road are recognized by image processing, and a warning is given to the driver when the vehicle deviates from the driving lane, and vehicle steering control is performed using the lane mark recognition results. A method for performing the above has been proposed.

  In order to recognize a white line by image processing, in a conventional general method, a threshold value is set for the luminance of the image and the image is binarized to recognize a portion with high luminance as a white line. This utilizes the fact that white lines have higher brightness than surrounding roads, and thus have higher brightness than road areas on the image. If a threshold is set between the brightness of the road area and the white line area, only the white line area can be detected by binarization.

  However, in the case of an image with a shadow on the white line, the brightness of the shadowed white line area is lower than that of the non-shadowed area, and the shadowed area cannot be detected well. There was an inconvenience. Further, if a luminance value that can detect a shaded white line region is set as a threshold, a portion other than the white line is also detected with this threshold.

  Japanese Patent Laid-Open No. 4-152406 discloses a method for setting a threshold based on the average value and the maximum value of the luminance of the entire image. With this method, the threshold value can be changed depending on the state of the image, so that more stable white line recognition is possible.

  However, this method has a disadvantage that the white line cannot be detected satisfactorily when the state of the image changes due to weather or shadow.

  Further, there is a method of recognizing a white line by performing edge extraction processing instead of binarization by luminance and detecting a white line outline. In this edge extraction process, a portion where the luminance of the image is changed is extracted. That is, in this method, the white line on the road is recognized as the end of the white line by utilizing the fact that the surrounding road is dark and there is a change in brightness at the end of the white line. Since this edge extraction process is a process for detecting a change in brightness, even if the brightness of a wide area of an image changes due to a change in weather, a white line can be detected if there is a change in brightness at the end of the white line. There are advantages. However, the method of performing the edge extraction process can stably extract the edge of the end portion of the white line, but has a high possibility of extracting an extra edge component in the road such as a preceding vehicle or a fence.

  Therefore, a method for determining whether or not the edge is a white line is important.

  According to Japanese Patent Laid-Open No. 11-195127, a method is disclosed in which edge angle information obtained when an edge point is obtained is used to determine a region surrounded by an edge point forming an angle pair as a white line. In the case of the white line, the edge is extracted in pairs at the left end and the right end of the white line, and the angle difference is about 180 degrees. When there is no edge point to be an angle pair, The edge point can be excluded as not being a white line.

JP-A-4-152406 JP 11-195127 A

  Among the lane marks, when a white line whose edge point distribution is linearly connected is a recognition target, the white line portion can be estimated from the position angle information of the edge point as disclosed in JP-A-11-195127. However, in the above-mentioned conventional example, the route where the road fence is laid as a lane mark is not taken into consideration, and therefore it is impossible to recognize the road fence.

  When recognizing a roadway, there is a method of registering a roadway pattern as a template and detecting the position of the roadway by template matching. However, in general, the size of the road fence on the image is small and easily affected by noise, so it is difficult to stabilize the recognition rate.

  In addition, when the white line algorithm and the road fence algorithm are different, it is necessary to correctly determine the type of the lane mark in order to recognize the lane mark of the road in which the white line and the road fence are mixed. As a result, even if each algorithm is excellent, it cannot be recognized if the lane mark type is wrongly determined, so the overall recognition rate decreases.

  An object of the present invention is to provide an algorithm capable of stably recognizing a lane mark composed of a white line or a roadside without distinguishing between lane mark types.

  In order to achieve the above object, in the lane mark recognition method of the present invention, an image input step for inputting an image including a lane mark, and an edge point that is a luminance change portion in the input image input in the image input step are extracted. An edge extraction step for calculating an angle representing the direction of each extracted edge point, and among the edge points extracted in the edge extraction step, the edge point angle is within a predetermined angle or a predetermined angle range. A lane mark position estimation step for extracting and estimating the lane mark position is provided.

  The edge extraction step includes a step of extracting edge points that are separated from the vanishing point.

  The lane mark position estimation process creates a histogram for each angle from the edge points that are at an angle toward the vanishing point among the edge points that are far from the vanishing point. And the angle.

  In the lane mark position estimation step, the lane mark position is linearly estimated from the distribution of edge points having the lane mark angle.

  In the lane mark recognition method of the present invention, even if the lane mark is not only a white line but also a lane mark, it can be detected if there is an edge at the lane mark position, so it is stable without being distinguished by the type of lane mark. Can provide a recognizable algorithm.

  FIG. 1 is a processing flow of a lane mark recognition method according to the present invention.

  In one embodiment of the present invention, a lane mark recognition method includes a vehicle front image acquisition step (step S1), which is an image input step for inputting an image including a lane mark, and an input input in the vehicle front image acquisition step S1. An edge extraction step with angle information (step S2) for calculating the direction angle of the edge using the luminance change portion of the image as an edge, and an edge point from the position and direction angle of the edge point toward the vanishing point of the road is selected. Edge selection step (step S3), lane mark position estimation step (steps S4 to S5) for determining the edge point of the lane mark by analyzing the position distribution of the edge points lined up toward the vanishing point, the vanishing point, etc. The process of updating the information (step S6) and the result output process (step S7) of outputting the recognition result.

  Here, the vanishing point is an infinite point where lane marks on the left and right of the host vehicle intersect.

  Hereinafter, after describing the configuration of the entire application system of the lane mark recognition method in the present embodiment, details of each step in the lane mark recognition method of the present embodiment will be described.

  The system of the present embodiment is shown in FIG. First, the lane recognition unit 101 transfers a vehicle front image obtained from an input unit such as the camera 104 directed to the front of the vehicle to the image processing unit 105. The image processing means 105 recognizes the position of the lane mark on the image by the lane mark recognition method of the present invention, and calculates how much the host vehicle is deviated from the lane. The amount of deviation will be described with reference to FIG. The deviation amount S is 0 when the host vehicle 71 is at the center of the lane, a positive value when the vehicle 71 is on the right side, a negative value when the vehicle 71 is on the left side, and a value from −1 to 1. . The calculated shift amount S is transferred to the lane keep control unit 102. The lane keep control unit 102 controls the steering actuator 106 so as to return the vehicle to the center of the lane, if necessary, or sounds the lane departure alarm 107 from the transferred deviation amount S. That is, in the present lane mark recognition method, the video of the camera is analyzed, the position of the lane mark composed of a white line or a roadside is detected, and the deviation amount of the own vehicle with respect to the lane is calculated.

  Hereinafter, the lane mark recognition method will be described in detail with reference to FIG.

  First, in step S1, an image ahead of the vehicle obtained by a camera as input means is captured and stored in an image memory in the image processing apparatus. The absolute condition is that the image includes a lane mark composed of a white line or a road fence as shown in FIG. FIG. 2A shows an example of a white line, and FIG. 2B shows an example of a roadway.

  In step S2, the brightness change portion of the input image stored in the image memory is extracted as an edge, and is calculated as data to which angle information of the edge is added. This data includes position information x, y and angle information θ for each edge point.

  Here, the position information is a value on the screen, the unit is a pixel, and the angle information θ represents the direction of the edge on the screen, and takes a value from 0 degrees to 360 degrees. In this embodiment, the direction of the edge is defined to be inclined 90 degrees to the left with respect to the direction with high luminance. The reason for tilting 90 degrees with respect to the direction of high brightness is to match the direction of the contour line by the edge, and the tilting direction may be right. In general, since the brightness of both the white line 1 and the roadside 2 is higher than that of the surrounding road surface 3, the edge direction is as shown in FIG. 3 according to the definition of this embodiment.

  FIG. 3 is an enlarged view of the white line 1 and the road fence 2. The white circle in FIG. 3 is the obtained edge point 4 and the arrow is the edge point direction 5. Since it is inclined 90 degrees to the left with respect to the direction of high luminance, the luminance on the right side increases toward the edge, and the luminance on the left side decreases. Although FIG. 3 shows the case where the road fence 2 is circular, the same applies to other shapes such as a square. Details of the edge point extraction method will be described later. In fact, in addition to the lane mark, there are many elements that appear as edges, such as preceding cars, dirt such as slip marks, patterns, and shadows of street trees. All of these edges become unnecessary noise for lane mark recognition. Therefore, noise separation is performed after step S3.

  In step S3, edge points that cause noise are excluded while leaving as many edge points as possible. For this purpose, using the fact that the lane mark is toward the vanishing point, only the edge point toward the vanishing point is selected. That is, an edge point that is not directed to the vanishing point is excluded. In the selection method, first, an equation of a straight line passing through x and y and having an angle of θ is obtained from the edge point information x, y, and θ. A distance between the straight line and the vanishing point is obtained, and if the distance is smaller than a predetermined threshold set in advance, it is determined that the vehicle is moving toward the vanishing point and is left as a lane mark candidate. By providing a predetermined threshold value, the angle of the extracted edge point has a width, and there is an effect of reducing the influence of the angle shift due to noise. Next, edge points that are originally close to the vanishing point are removed regardless of θ. Since the edge point close to the vanishing point is an edge point of a distant object in real space, the lane mark is projected very small in the vicinity of the vanishing point on the screen. As a result, the edge points of the lane mark are few in the vicinity of the vanishing point, and the ratio of edge points other than the lane mark that causes noise increases.

  As a means for determining whether or not it is in the vicinity of the vanishing point, there is a method of determining by simply calculating the distance between the vanishing point and the edge point on the screen. That is, all edge points within a circle or ellipse with a constant radius centered on the vanishing point are removed. In addition, there is a method of determining from the difference of the y coordinate in consideration of the calculation cost. In this method, the difference between the y-coordinate of the vanishing point and the y-coordinate of the edge point is obtained. The lane mark on the road plane can obtain the depression angle from the camera based on the y-coordinate position on the screen. Therefore, the actual distance can be obtained together with the installation height of the camera. This is the reason for using only the y coordinate.

  Removing the edge points near the vanishing point in advance has the effect of reducing noise. By selecting the edge points as described above, only the edge points toward the vanishing point are selected from the near edge points away from the vanishing point. FIG. 4 is an edge point selection result when the vanishing point is on the upper right in the white line 1 and the roadway 2. The edge point 4 of the white line 1 remains almost as it is, but most of the edge points 4 of the road fence 2 are excluded. However, since the left end and right end edge points having the same direction component as the white line 1 remain, processing similar to the white line can be performed in the subsequent processing. Most of the edge points that become noise other than the lane mark are removed by this selection process, but the edge points parallel to the lane mark remain, so the determination is made in the analysis after step S4. Further, the vanishing point position is not constant and must be calculated every time because it changes depending on the inclination of the vehicle with respect to the lane, but in this step, the vanishing point position obtained in the previous frame is used. This utilizes the property that the position of the vanishing point does not change extremely in a short time. At the start, a default vanishing point position is determined in advance from the camera installation position and the value is used. How to find the vanishing point will be described in step S6.

In step S4, the angles of the left and right lane marks of the host vehicle are estimated from the angle information of the edge points extracted in step S3. For analysis, an edge angle histogram is used in this embodiment. That is, a histogram for each angle is created for all the pixels constituting the edge. In this edge angle histogram, if there are many pixels having an edge angle in a certain direction, this appears as a peak. For example, assuming that there is only one straight white line in the image, a peak appears for a particular edge angle. Therefore, the angles θ _l and θ _r of the left and right lane marks are estimated from the angles at which peaks are formed by creating a histogram as shown in FIG. In the example shown in FIGS. 5A and 5B, a 180 ° shift is ignored, and a histogram is created with a value obtained by subtracting 180 ° from an angle of 180 ° or more. This is because the white line 1 and the road ridge 2 appear with a set of edges that differ by about 180 degrees at the left end and the right end as shown in FIGS. 4 (a) and 4 (b). effective. However, in practice, the noise component often forms a peak, and in order to distinguish it from this, estimation is performed using lane width information. Since the lane width is substantially constant, the combination of the angles θ _l and θ _r of the left and right lane marks can be limited. When the number of lanes is large as in the example of FIG. 5A, it is possible to estimate θ _l and θ _r by using the angles θ _nl and θ _nr of the adjacent lane mark. In this case, the fact that the lane width is almost constant is used.

In step S5, the straight line of the left and right lane marks is estimated from the distribution of edge points having the angles θ _l and θ _r . Here, it is conceivable that the lane mark is bent due to a curve or the like, but the influence of this bend can be almost ignored. In step S3, all the edge points near the vanishing point are excluded, and only the edge point of the lane mark in front of the own vehicle is used, so that even a curve can be partially linearly approximated. This is because the property is available. The Hough transform is used for the straight line estimation of the lane mark. The Hough transform is a technique for deriving a straight line that passes through the most points from the arrangement of several points, and is not described because it is a general technique in image processing. Hereinafter, description will be given to the straight line estimation of the lane mark with reference to FIG.

FIG. 6 shows an example of a white line on the left side of the host vehicle. Since edges appear at both the left and right edges of the white line, the Hough transform is performed separately for the left edge point and the right edge point. If the angle of the white line on the left that Motoma' in step S4 is to theta _l, direction angles of the right end of the edge points can be estimated theta _l, direction angles of the left end of the edge point in the opposite direction of so theta _l +180. Therefore, Hough transform is performed from the distribution of edge points having an angle close to θ _l , and the obtained straight line is defined as the rightmost line 62 of the white line. Similarly, the Hough transform is performed from the distribution of edge points having an angle close to θ _l +180, and the obtained straight line is defined as the leftmost line 61 of the white line. The center line 63 is obtained from the two straight lines obtained in this way, and is set as the left white line. The white lines have been described above, but the same applies to roadsides. The Hough transform used for the straight line estimation can be substituted by a least square approximation method.

  In step S6, the vanishing point is calculated. The vanishing point can be calculated as the intersection of the straight lines of the left and right lane marks calculated in step S5. In general, when the vanishing point is a curve or the like and the lane mark is bent, the vanishing point is in a bent direction. However, in this embodiment, distant edge points are ignored, and the intersection point obtained by linearly approximating the lane mark in front is used as the vanishing point. The vanishing point obtained in this way has a small variation due to the curve, so the variation range can be limited, and the vanishing point position can be easily predicted and analyzed.

  In step S7, the recognition result is output. The information to be output is the positional deviation amount S of the own vehicle with respect to the lane (FIG. 7). The displacement S is 0 when the host vehicle 71 is traveling in the center of the lane (FIG. 7A), 1 when the vehicle 71 is on the right lane mark (FIG. 7B), and the left lane. When it is on the mark, it is set to -1 (FIG. 7C), and it is between -1 and 1. S is obtained from the two angles of the straight line approximation line 81 in FIG.

  here,

FIG. 9 illustrates the process of each step described above. The processing of this lane mark recognition method will be described again using FIG. First, FIG. 9A shows a result of extracting edge points in step S2 from an image obtained from a camera. In FIG. 9, the edge points are indicated by line segments so that the direction of each edge point can be easily understood. FIG. 9B is a diagram in which all the edge points included in the vicinity region 91 of the vanishing point 10 calculated in the previous frame are removed in the middle of step S3. In FIG. 9C, an edge point that is not directed toward the vanishing point 10 is further removed. FIG. 9D shows a result of analyzing the angle histogram in step S4 and extracting only edges having an angle with a high appearance frequency and satisfying the lane width condition. In this example, θ _nl and θ _nr that are the angles of the adjacent lane marks are left, but only θ _l and θ _r may be used. FIG. 9E shows a line approximation line 81 obtained by performing Hough transform for each angle. The vanishing point 10 is updated to the intersection of the straight line approximation line 81. As described with reference to FIG. 9, in the present lane recognition method, the recognition process is performed without particularly distinguishing the white line and the road fence, and therefore, it is effective for a route in which the white line and the road fence are mixed. However, since it is a basic premise that the edge point of the lane mark can be detected whether it is a white line or a road fence, acquire an image so that a luminance difference appears even on the road fence, and reduce the edge detection threshold sufficiently. is important.

  Finally, an edge detection method will be described. There are various methods for detecting only the position of the edge. Examples of the calculation method including the edge direction component include a Sobel filter method and a two-dimensional zero-cross method. In this embodiment, a method using a 3 × 3 Sobel filter will be described.

  The 3 × 3 Sobel filter multiplies nine pixel values (up, down, left, and right) centered on a certain target pixel by coefficients as shown in FIG. 11 and sums the results to calculate the edge strength. To do. This processing is performed using two coefficient matrices in the vertical direction and the horizontal direction.

If the total value in the vertical direction is G _x and the total value in the horizontal direction is G _y , the pixel value G of the target pixel is expressed by Equation (3).

  This pixel value G represents the intensity of the edge, and the larger the value, the greater the difference in brightness near that point. If the value of G is larger than a preset edge extraction threshold, the pixel in that portion is extracted as an edge point. In the case of the white line 1, since the luminance difference with the road surface 3 is generally large, there is no problem even if the edge extraction threshold is set appropriately. However, in the case of the road fence 2, the brightness difference with the road surface 3 is often small. In order to prevent extraction omission, it is necessary to sufficiently reduce the edge extraction threshold.

The angle θ of the edge can be obtained by the following formula (4) from the directional components G _x and G _y of the edge strength.

  As described above, the angle θ represents the direction of the edge on the screen, and takes a value of 0 degree or more and less than 360 degree. The direction of the edge is a direction shifted to the left by 90 degrees with respect to the high luminance direction.

  Finally, a description will be given of preprocessing applied to the input image in order to improve the recognition rate. It is desirable to install a camera as a means for inputting an image at a position as high as possible so that the surroundings can be easily seen. However, it may be possible that the vehicle cannot be installed at a high position, such as a vehicle with a low vehicle height. In this case, since the installation height of the camera is lowered, the angle formed by the edge components 121 of the left and right lane marks of the traveling lane on the screen is widened as shown in FIG. 12A, and the edge component 121 of the lane mark is horizontal. It becomes a state close to. At this time, the edge points of the noise edge component 122 of the adjacent lane mark, guardrail, and other vehicles traveling on the adjacent lane also approach the horizontal simultaneously, so the edge angle of the noise edge component 122 and the edge of the lane mark The difference in edge angle of the component 121 becomes difficult to appear. In this state, according to the lane mark recognition method of the present invention, it is difficult to remove noise using the angle information of the edge points, and the recognition rate may be reduced.

  Therefore, as a means for improving the recognition rate in such a case, preprocessing for changing the aspect ratio of the input image is performed. Specifically, by compressing the input image in the horizontal direction or expanding it in the vertical direction, an image having an aspect ratio that is longer than that of the original input image is generated as shown in FIG. This process can be realized by affine transformation. By this processing, the difference between the edge angle of the edge component 121 of the lane mark and the edge angle of the other edge component 122 of noise becomes large.

  For this reason, the noise removal accuracy using the angle information of the edge point is improved, and the recognition rate in the lane mark recognition method of the present invention is improved. In order to achieve the same effect as the pre-processing, a vertically long image may be obtained in advance by changing the shape of the optical system, that is, the light receiving unit such as a camera lens or CCD.

It is the figure which showed the processing flow of the lane mark recognition method concerning a present Example. It is the figure which showed the vehicle front containing the lane mark obtained at the image input process of a present Example. It is the figure which showed the direction of the edge point of the lane mark of the lane mark recognition method concerning a present Example. It is explanatory drawing of the edge point which is approaching a vanishing point when there exists a vanishing point in the upper right in the lane mark recognition method concerning a present Example. It is a figure explaining the edge angle histogram of the lane mark recognition method concerning a present Example. It is a figure explaining the straight line estimation of the lane mark recognition method concerning a present Example. It is a figure explaining the deviation | shift amount S with respect to the lane of the own vehicle in the lane mark recognition method concerning a present Example. It is a figure explaining the method of calculating | requiring the deviation | shift amount S with respect to the lane of the own vehicle in the lane mark recognition method concerning a present Example. It is the figure which showed the process of the process flow of the lane mark recognition method concerning a present Example. It is a system configuration block diagram using the lane mark recognition method according to the present embodiment. It is the figure which showed the coefficient matrix of the 3x3 Sobel filter method which is one method of detecting an edge position. It is a figure explaining the pre-processing with respect to the input image in the lane mark recognition method concerning a present Example.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... White line, 2 ... Road fence, 3 ... Road surface, 4 ... Edge point, 5 ... Edge point direction, 10 ... Vanishing point, 61 ... White line left end, 62 ... White line right end, 63 ... White line centerline, 71 DESCRIPTION OF SYMBOLS: Self-vehicle, 81 ... Linear approximation line, 91 ... Neighborhood area, 101 ... Lane recognition part, 102 ... Lane keep control part, 103 ... Lane keep execution part, 104 ... Camera, 105 ... Image processing means, 106 ... Steering actuator, 107 ... Lane departure warning, 121 ... Lane mark edge component, 122 ... Noise edge component.

Claims (8)

An image input means for inputting an image including a lane mark;
Edge extraction means for extracting an edge point that is a luminance change portion in the input image input in the image input step, and calculating an angle representing the direction of each extracted edge point;
Lane mark position estimating means for extracting the edge point whose angle is within a predetermined angle or a predetermined angle range from the edge points extracted in the edge extracting step and estimating a lane mark position. Lane mark recognition device. The lane mark recognition apparatus according to claim 1,
The lane mark recognition apparatus according to claim 1, wherein a predetermined angle or a predetermined angle range of the edge point extracted by the lane mark position estimating means changes according to the position of each edge point. In the lane mark recognition apparatus according to claim 1 or 2,
The predetermined angle or predetermined angle range of the edge point extracted by the lane mark position estimating means is a straight line connecting the edge point and the vanishing point of the road or a range shifted by 180 degrees from the straight line angle. A lane mark recognition apparatus characterized by In the lane mark recognition apparatus according to claim 3,
The lane mark recognition apparatus according to claim 1, wherein the lane mark position estimating means excludes edge points that are within a predetermined range from the vanishing point. In the lane mark recognition apparatus according to claim 3,
The lane mark position estimating means creates a histogram for each angle from an edge point having an angle facing the vanishing point, and estimates a lane mark from an edge point having an angle of appearance frequency equal to or greater than a predetermined value. Lane mark recognition device. In the lane mark recognition apparatus according to claim 4,
The lane mark position estimating means creates a histogram for each angle from edge points having an angle facing the vanishing point among edge points outside the predetermined range from the vanishing point, and the appearance frequency is a predetermined value or more. A lane mark recognition apparatus, wherein a lane mark is estimated from edge points having an angle of. In the lane mark recognition device according to claim 5 or 6,
The lane mark position estimation means limits the angle of the left and right lane marks from the lane width among the high-frequency angles obtained from the histogram. In the lane mark recognition device according to claim 7,
The lane mark position estimation means linearly estimates a lane mark position from a distribution of edge points having an angle of the lane mark.

Images