JP5189556B2 - Lane detection device - Google Patents

Description

  The present invention relates to a lane detection device, and more particularly, to a lane detection device that detects a lane marked on a road surface of a traveling path of a host vehicle from captured images.

  Conventionally, for example, the front of the vehicle is imaged with a camera equipped with an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and is displayed on the road surface of the traveling path of the own vehicle from the captured image. Various techniques for detecting lanes have been developed. In this specification, continuous lines and broken lines marked on road surfaces such as overtaking prohibited lines and lane markings that divide roadside zones and roadways are called lanes, and the road surface on which the vehicle between the lanes travels. Is called the traveling path of the vehicle.

  For example, in the lane recognition device described in Patent Document 1, a luminance difference (that is, edge strength) between adjacent pixels when searching on a horizontal line having a width of one pixel of a captured image in the left-right direction is greater than or equal to a predetermined threshold value. A large pixel is detected as a lane candidate point that is a pixel corresponding to the lane, and this search is performed for each horizontal line to obtain a plurality of lane candidate points.

  Then, while excluding lane candidate points corresponding to points above the road surface in real space and lane candidate points corresponding to road surface markings, shadows, etc., that is, lane candidate points not corresponding to the lane, A straight line corresponding to the lane is detected by performing Hough transform on the lane candidate point on the side close to the image, that is, on the lower side of the image.

  In addition, the lane candidate points farther from the host vehicle, that is, the lane candidate points on the upper side of the image, are tracked in order from the lower side to the upper side of the image, and are consistent with the lane candidate points on the lower side of the image. Connect certain lane candidate points. In this way, for example, when a lane is marked linearly, it is connected linearly, and when the lane is curved forward, it is connected while detecting lane candidate points along the curve. The vehicle is configured to detect a lane marked on the traveling path of the vehicle.

JP 2006-331389 A

  By the way, when detecting a lane on an image as described above, if the process of searching for a lane candidate point by searching on the horizontal line of the image is executed for the entire area of the image, the execution time of the detection process is long. Therefore, real-time execution becomes difficult, and false detection of lane candidate points increases.

  Therefore, as described in Patent Document 1, usually, for example, the position of the lane on the image in the current sampling period from the information on the position of the lane detected on the image in the past and the behavior of the host vehicle thereafter. Is set, a search region including the same is set, and only the search region is searched. And by comprising in this way, it becomes possible to detect a lane exactly on an image in most cases.

  However, as shown in FIG. 26, for example, when a branching lane such as a right turn lane is captured in the image T, a straight line is determined on the near side, that is, on the side closer to the own vehicle as described above, and on the far side, If the lane candidate points are tracked and connected in order from the lower side to the upper side of the image regardless of the straight line, the lane candidate points corresponding to the lane branching to the right turn lane (as shown in the figure) In some cases, a lane that branches to a right turn lane is detected based on the detected black circle.

  Then, in the sampling period after the next time, the position of the lane on the image is estimated from the information on the position of the lane branching to the right turn lane and the behavior of the host vehicle thereafter, and the search region including the estimated position (two in the figure) Thus, only the lane candidate points corresponding to the lane branching to the right turn lane are continuously detected. Therefore, there is a possibility that the lane candidate point (see the white circle in the figure) corresponding to the main line that the host vehicle is originally going to drive, that is, the lane on the left side of the right turn lane cannot be detected.

  In this case, for example, if searching within a predetermined distance range in the right direction from the lane detected on the left side of the host vehicle, the lane candidate points corresponding to the lane on the main line side (see the white circle in the figure) It can also be detected. However, as shown in FIG. 26, if the left lane cannot be detected because it is covered with snow or the like, or the left lane is not originally marked, a lane candidate corresponding to the main lane The point cannot be detected, and eventually the main lane cannot be detected.

  This problem is difficult to solve as long as a search area is set and a method for detecting lane candidate points is adopted. However, if lane candidate points are detected for the entire area of the image, the detection process is performed as described above. Causes a problem such as a longer execution time and a loss of real-time performance.

  Further, as one of the conditions for searching the horizontal line of the image T and detecting the lane candidate point corresponding to the lane, the lane candidate point is equal to or greater than a predetermined threshold value from the average value of the luminance of the pixels corresponding to the road surface. Therefore, in the lane recognition device described in Patent Document 1, a histogram of the luminance of pixels is created on a horizontal line in the above search region or the like. The average value and the luminance that gives the maximum frequency are calculated.

  However, if this is applied to search the entire area of the image without setting the search area, a huge amount of histograms must be created and the execution time of the detection process becomes very long. The method of creating a histogram can be applied only when setting a search area.

  The present invention has been made in view of such circumstances, and provides a lane detection device capable of accurately detecting a lane marked on the traveling path of the host vehicle from a captured image. Objective. It is another object of the present invention to provide a lane detection device that can execute accurate lane detection in real time.

In order to solve the above problem, the first invention is a lane detection device,
Imaging means for capturing an image of the surroundings of the host vehicle;
Integration means for integrating pixels in which the luminance difference between adjacent pixels in the image is within a predetermined threshold as a group;
Road surface detection means for detecting the group as a road surface group, when the average brightness and area of the group are each within a predetermined threshold range;
Search the image portion other than the road surface group in the image, and find a pixel whose luminance difference with the road surface group is equal to or greater than a predetermined threshold and whose luminance difference with an adjacent pixel is equal to or greater than a predetermined threshold. Lane candidate point detecting means for detecting lane candidate points;
Lane detection means for detecting a lane marked on the traveling path of the host vehicle in the image based on the detected lane candidate point;
It is characterized by providing.

  According to a second aspect of the present invention, in the lane detection device according to the first aspect, the lane candidate point detection means detects an end point of the lane corresponding to the lane candidate point, and the lane candidate point and the end point of the lane The group existing between and is detected and registered as a lane group, and if the vertical length of the lane group in the image is equal to or less than a predetermined threshold, the registration of the lane group is deleted. In addition, at least the lane candidate point is deleted from the detected lane candidate point.

  According to a third aspect of the present invention, in the lane detection device of the second aspect, the lane candidate point detection means registers the lane group having the largest area among the detected lane groups, and the lane group of the registered lane group When the area or the length in the horizontal direction is equal to or greater than a predetermined threshold value, the registration of the lane group is deleted, and at least the lane candidate point is deleted from the detected lane candidate point.

  According to a fourth aspect of the present invention, in the lane detection device of the second or third aspect, the lane candidate point detection means does not search the lane candidate point for each pixel belonging to the lane group, and Is a pixel that no longer belongs, and a pixel whose luminance difference with the lane group is equal to or smaller than a predetermined threshold and whose luminance difference with an adjacent pixel is equal to or smaller than a predetermined threshold is detected as an end point of the lane It is characterized by doing.

  According to a fifth aspect of the present invention, in the lane detection device according to any one of the second to fourth aspects, the lane candidate point detection means does not search the lane candidate point for each pixel belonging to the lane group, When a pixel that does not belong to a lane group belongs to the road surface group, the pixel is detected as an end point of the lane.

  According to a sixth aspect of the present invention, in the lane detection device according to any one of the first to fifth aspects, the lane detection means extracts one or more straight lines as lane candidates and exists in the vicinity of the straight lines. The type of each straight line is determined based on the continuity of the lane candidate points, and based on the reliability calculated based on at least the type of the straight line and the number of lane candidate points existing in the vicinity of each straight line. A straight line representing the lane is selected from each straight line.

7th invention is the lane detection apparatus of 6th invention,
The type of the straight line is determined based on the continuity of the lane candidate points existing in the vicinity of each straight line, whether the straight line corresponds to at least a continuous line shape, a broken line shape, or a combination thereof,
The continuity of the lane candidate points is determined based on the distance between the lane candidate points on the image or the distance in real space, and the lane candidate points belonging to the same group are continuous regardless of the determination. It is judged that it is.

  According to the first invention, the lane candidate point detecting means does not set the search area for searching for the lane candidate point, and other than the image portion other than the road surface group, that is, the ground portion of the road surface such as the asphalt surface. Search for road surface parts to detect lane candidate points. Therefore, for example, not only the lane candidate point corresponding to the lane that branches to the right turn lane (see FIG. 26, etc.), the lane candidate point corresponding to the inner lane in the double lane (see FIG. 24, etc. described later), etc. It is possible to accurately detect lane candidate points corresponding to the main line that the host vehicle is about to travel, lane candidate points corresponding to the outer lane in the double lane, and the like.

  Therefore, even if the lane is branched, the lane corresponding to the main line that the host vehicle is about to travel is accurately detected based on the lane candidate points, and the lane of the host vehicle is detected from the captured image. It becomes possible to accurately detect the lane marked on the road. In addition, when the lane is a double lane, it is possible to accurately and stably detect not the inner lane that is disappearing or the outer lane together with it, from the captured image. It becomes possible to accurately detect the lane marked on the traveling path of the host vehicle.

  The lane candidate point detection means searches for and detects lane candidate points in image portions other than the road surface group detected by the road surface detection means (for example, see road surface groups G1 to G4 in FIG. 17 described later). At that time, the lane candidate point corresponding to the high-brightness lane does not belong to the road surface group corresponding to the dark road surface area, and it is not necessary to perform a search for the pixels belonging to the road surface group. In the face group, it is possible to skip without searching. In addition, since the average brightness of the road surface group used when detecting the lane candidate points has already been calculated by the integration means, a huge amount of histogram is created and the road surface is newly created as in the conventional lane detection. It is not necessary to calculate the average value of the luminance of the pixels corresponding to.

  Therefore, it becomes possible to detect lane candidate points at high speed, and it becomes possible to detect lanes at high speed based on those lane candidate points by the lane detection means. For this reason, for example, even when images of several frames to several tens of frames are captured and data is input per second, it is possible to detect lanes quickly and to execute lane detection in real time. It becomes possible.

  According to the second invention, in addition to the effects of the above invention, the number of pixels in each group and the length in the vertical direction in the image are grouped by grouping each pixel of the image by the integrating means. To do. Therefore, when the size of the lane group corresponding to the lane candidate point detected by the lane candidate point detecting means is small, the lane candidate is assumed to correspond to high-intensity noise or the like that is reflected in the image. The registration of points can be deleted. As a result, erroneous detection of lane candidate points can be prevented accurately, and the reliability of detection of the finally detected lane can be improved.

  According to the third invention, contrary to the case of the second invention, when the size of the lane group corresponding to the lane candidate point detected by the lane candidate point detecting means is abnormally large, the lane candidate point Is considered to correspond to snow or the like deposited on the side of the road imaged in the image. Therefore, even in such a case, by deleting the registration of the lane candidate points, it becomes possible to prevent erroneous detection of the lane candidate points accurately, and it is possible to exert the effects of the above inventions accurately. .

  According to the fourth invention, the detection of the lane candidate point is further performed by skipping each pixel belonging to the lane group existing between the lane candidate point and the lane end point without searching for the lane candidate point. It becomes possible to perform at high speed, and it becomes possible to exhibit the effect of each said invention exactly.

  According to the fifth invention, the detection of the lane candidate point is further performed by skipping each pixel belonging to the lane group existing between the lane candidate point and the lane end point without searching for the lane candidate point. It can be performed at high speed. At the same time, when the search pixel moves from the lane group to another group, if the group is a road surface group, the pixel that switches from the lane group to the road surface group can be detected as the end point of the lane. Therefore, in such a case, by detecting the pixel as the end point of the lane, it is not necessary to perform a determination process as to whether or not the pixel is the end point of the lane. Therefore, it becomes possible to detect the lane candidate points at higher speed, and it is possible to accurately exhibit the effects of the respective inventions.

  According to the sixth aspect of the present invention, a plurality of straight lines that are lane candidates may be extracted based on the detected plurality of lane candidate points, for example, by a technique such as Hough transform. In such a case, the reliability of each straight line is calculated based on the type of straight line and the number of lane candidate points existing in the vicinity of each straight line, and the lane is represented from each straight line based on the calculated reliability. By selecting a straight line, it becomes possible to accurately select and detect a straight line suitable for approximating the lane marked on the traveling path of the host vehicle from the captured image, and more accurately achieve the effects of the above inventions. Can be demonstrated.

  According to the seventh invention, as the type of straight line, by determining whether the straight line is a continuous line or a broken line based on the continuity of lane candidate points existing in the vicinity of each straight line, It becomes possible to accurately calculate the reliability of a straight line, to further improve the reliability of a straight line that approximates a detected lane, and to exhibit the effects of the respective inventions more accurately. .

It is a block diagram which shows the structure of the lane detection apparatus which concerns on this embodiment. It is a figure which shows the example of the image imaged with an imaging means. It is a flowchart which shows the process sequence in an integration means. It is a flowchart which shows the process sequence in an integration means. It is a flowchart which shows the process sequence in an integration means. It is a flowchart which shows the process sequence in a road surface detection means. It is a figure explaining an estimation locus and a course. It is a photograph explaining the check area set in the image. It is a figure explaining the input pixel and the pixel adjacent to the left already input. It is a figure explaining the example of the group to which an input pixel and the pixel adjacent on the left belong. It is a figure explaining the input pixel and the pixel adjacent under the already input. It is a figure explaining the example of the group to which an input pixel and the adjacent pixel below belong. (A) It is a figure explaining the example of the group which two pixels adjoin to the left and the bottom of an input pixel, (B) The input pixel is grouped with the pixel which adjoins to the left, and the pixel which adjoins below is also grouped It is a figure explaining the example to be done. (A) It is a figure explaining the example of the group to which the lower adjacent pixel belongs, and the group to which the left adjacent pixel belongs, (B) The example which two groups are grouped with an input pixel and become one group It is a figure explaining. It is a photograph showing the some group integrated based on the image of FIG. It is a figure which shows the example of the stop line marked continuously on the lane. It is a photograph showing the road surface group extracted from the image of FIG. It is a figure showing the presumed locus | trajectory from which the position on the image T was calculated. It is a figure showing the transition of the brightness | luminance of a search pixel, a lane candidate point, etc. when searching on a horizontal line. It is a photograph showing the lane candidate point detected on the image of FIG. It is a figure explaining the example of a Hough plane, and a cell. It is a photograph showing each straight line extracted based on each lane candidate point of FIG. FIG. 21 is a photograph showing a lane marked on the travel path of the host vehicle detected on the image of FIG. 20. It is a photograph showing the double lane detected with the lane detector concerning this embodiment. It is a photograph showing the double lane detected by the conventional detection method. It is a photograph showing that a lane candidate point corresponding to a lane branching to a right turn lane at a lane branch is detected in a conventional detection method, and a lane candidate point corresponding to a main line is not detected.

  Embodiments of a lane detection device according to the present invention will be described below with reference to the drawings.

  As shown in FIG. 1, the lane detection device 1 according to the present embodiment mainly includes an imaging unit 2 and a processing unit 5. In the present embodiment, the imaging unit 2 is a monocular imaging unit such as a camera with a built-in image sensor such as a CCD or CMOS sensor. However, a single or a plurality of imaging units 2 are provided. It is also possible to configure so that the processing according to the present invention is performed on one or a plurality of images picked up by the image pickup means.

  Further, in the present embodiment, for example, when an image T as shown in FIG. 2 is captured, the imaging unit 2 scans in the right direction sequentially from the leftmost imaging element of each horizontal line j of the image T. The horizontal line j to be picked up is picked up while switching upward from the lowermost line in order, and each data of the luminance D is sequentially transmitted to the conversion means 3 in the order picked up by each image pickup device.

  The conversion means 3 is composed of an A / D converter, and when the data of the luminance D imaged for each imaging element (each pixel p) by the imaging means 2 is sequentially transmitted, the luminance of each pixel p. The D data is converted into luminance D data of a gray scale digital value such as 256 gradations and output to the image correction unit 4. The image correction unit 4 sequentially performs image correction such as displacement and noise removal, luminance correction, and the like on the transmitted data of the luminance D, and sequentially transmits the image corrected luminance D data to the processing unit 5. It is supposed to be.

  In this embodiment, the processing unit 5 is configured by a computer in which a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), an input / output interface, and the like (not shown) are connected to a bus. The processing unit 5 includes integration means 6, road surface detection means 7, lane candidate point detection means 8, and lane detection means 9.

  Sensors Q such as a vehicle speed sensor, a yaw rate sensor, and a steering angle sensor for measuring the steering angle of the steering wheel are connected to the processing unit 5, and each measurement value is input from them. Note that the processing unit 5 may be configured to perform other processing.

  In the following description, for example, with respect to the pixels in the image T shown in FIG. 2, the coordinates of the pixels (i, i, i) and (i, j) is used to represent a pixel pi, j. Further, the luminance D of the pixel pi, j is expressed as Di, j.

  In the present embodiment, when the data of the luminance D of each pixel p of the image T is sequentially input from the imaging unit 2, the integration unit 6 refers to the input pixel pi, j (hereinafter referred to as input pixel pi, j). ) And the luminance D of the pixel p adjacent to the input pixel pi, j are compared with the luminance Di, j of the input pixel pi, j and the group g to which the adjacent pixel p belongs. The average value of each luminance D of all the pixels p to which it belongs is compared, and the input pixel pi, j and the adjacent pixel p are integrated into one group g when the conditions described later are met.

  This integration processing is performed every time the luminance D data of each pixel p of the image T is sequentially input as described above, and is finally performed for all the pixels p of one image transmitted. Each pixel p of the image T is divided into a plurality of groups g.

  The road surface detection means 7 detects the group g as the road surface group G when the average luminance and area of the group g integrated by the integration means 6 are within a predetermined threshold range. ing.

  Hereinafter, the processing in the integration unit 6 and the road surface detection unit 7 will be described with reference to the flowcharts of FIGS.

  When the imaging unit 2 starts imaging in the current sampling cycle (step S1 in FIG. 3), the integrating unit 6 firstly has the vehicle speed V and yaw rate γ of the host vehicle input from the sensors Q at that time. The turning curvature Cua of the host vehicle is calculated based on information such as the steering angle δ of the steering wheel, and the check region R is set (step S2).

  As will be described later, the road surface detection means 7 detects the road surface group G based on the average luminance and area of the group g. At this time, the group g corresponding to the sky (see group gsky in FIG. 15 described later), etc. May be detected as the road surface group G. Therefore, in the present embodiment, the integration unit 6 calculates a traveling path area that is estimated to proceed in the future based on the behavior information of the host vehicle, and sets a group g having a certain number of pixels in the traveling path area as a road surface group. Detect as. Specifically, first, based on information such as the vehicle speed V and yaw rate γ of the host vehicle, the steering angle δ of the steering wheel, the turning curvature Cua of the host vehicle is expressed by the following formula (1) or the following formulas (2) and (3): Calculate according to

Cua = γ / V (1)
Re = (1 + Asf · V ² ) · (Lwb / δ) (2)
Cua = 1 / Re (3)
Here, in each of the above equations, Re is a turning radius, Asf is a vehicle stability factor, and Lwb is a wheelbase.

  Then, as shown in FIG. 7, based on the calculated turning curvature Cua of the host vehicle MC, an estimated trajectory Lest that the host vehicle is estimated to travel in the future is calculated, and a predetermined distance apart in the left-right direction around the estimated trajectory Lest. The range up to the determined position is calculated as the traveling path of the host vehicle MC. Then, for example, as shown in FIG. 8, by assigning the above range, that is, the traveling path of the host vehicle MC, on the image T, a check region R is set in the image T, and at least some pixels are included in the check region R. The group g is detected as the road surface group G.

  The check region R is estimated by, for example, estimating the position of the lane in the current sampling period from the information on the position of the lane detected in the past sampling period and the subsequent behavior of the host vehicle, and changing the position of the estimated lane to the left and right. It is also possible to set the area as an edge. Further, in FIG. 8 and FIG. 15 to be described later, in order to indicate that the groups gsky and gtree corresponding to the sky and the roadside tree may be grouped, the image pickup means 2 such as a CCD camera is intentionally turned upward. The figure which was installed and imaged is shown.

  Subsequently, the integration unit 6 first receives the data of the brightness Di, j of the pixel pi, j from the image pickup unit 2, and the pixel number threshold value Nth ( 0) is calculated and set (step S3 in FIG. 3), and 0 is set as the values of i and j (step S4).

  Here, the pixel number threshold Nth is a threshold for preventing the group g corresponding to each white line block of the pedestrian crossing marked on the road surface from being detected as the road surface group G. For example, in the real space The horizontal width is set to 60 cm.

  The horizontal width in the real space can be calculated by a technique such as stereo matching based on a pair of images captured by a pair of CCD cameras, for example. In this embodiment, the horizontal line j of the image T is previously stored. For each horizontal line, the horizontal width in the real space is calculated for each pixel, and the integration unit 6 calculates the number of pixels corresponding to the horizontal width of 60 cm in the real space for each horizontal line j to calculate the horizontal line j Is assigned as the pixel number threshold Nth.

  As described above, when the input of the luminance D0,0 data of the leftmost pixel p0,0 (that is, the origin pixel) on the horizontal line 0 imaged by the imaging means 2 is started (step S5), the processing unit .., Data of luminance D1,0, D2,0, D3,0,... Of pixels p1,0, p2,0, p3,0,. Then, if the integration process has not been completed up to the rightmost pixel of the horizontal line j (step S6; NO), the integration unit 6 continues the integration process while incrementing the i coordinate by one (step S7).

  When the integration process is completed up to the rightmost pixel of the horizontal line j (step S6; YES), the integration unit 6 does not complete the integration process up to the uppermost horizontal line of the image T (step S8; NO). , J is incremented, i is set to 0 (step S9), the horizontal line to be integrated is shifted to the horizontal line j one row above, and the integration process is continued. At that time, the integration unit 6 calculates and sets a pixel number threshold Nth (j) in the shifted horizontal line j (step S10).

  Next, the process after step S11 of FIG. 4 is demonstrated. First, the integration means 6 inputs an input pixel pi, j and a pixel pi-1, which is input before the input pixel pi, j and is adjacent to the left of the input pixel pi, j as shown in FIG. For j, it is determined whether or not the following condition 1 and condition 2 are satisfied (step S11).

[Condition 1] The difference ΔDleft (i, j) between the luminance Di, j of the input pixel pi, j and the luminance Di-1, j of the pixel pi-1, j adjacent to the left, that is,
ΔDleft (i, j) = | Di, j−Di−1, j | (4)
Is less than a preset first luminance threshold value ΔDth. Hereinafter, the difference ΔD in luminance D between adjacent pixels as described above is referred to as edge strength.

[Condition 2] As shown in FIG. 10, the average value Dave-left of the luminance D i, j of the input pixel p i, j and the luminance D of all the pixels belonging to the group g to which the adjacent pixel p i-1, j on the left belongs. Difference δDleft (i, j), that is,
δDleft (i, j) = | Di, j−Dave-left | (5)
Is less than the preset second luminance threshold value δDth. Hereinafter, as described above, the difference δD between the luminance Di, j of the input pixel pi, j and the average value Dave of the luminance D of the group g to which the adjacent pixel belongs is referred to as an average value difference.

  Note that the average value Dave of the luminance D of the group g to which the adjacent pixel belongs is calculated by the calculation process in step S25 of FIG. In some cases, the group g to which the pixel pi-1, j adjacent to the left belongs is composed of only the pixel pi-1, j adjacent to the left, and in this case, the luminance D of all the pixels belonging to the group g. The average value Dave-left is equal to the luminance Di-1, j of the pixel pi-1, j adjacent to the left. Furthermore, in the present embodiment, the first luminance threshold value ΔDth and the second luminance threshold value δDth are set to the same value, but may be set to different values and set as appropriate. Further, the values of the first luminance threshold value ΔDth and the second luminance threshold value δDth can be set to be variable according to the situation.

  If the integration unit 6 determines that both the condition 1 and the condition 2 are satisfied (step S11; YES), the integration unit 6 proceeds to the determination process in step S12, and determines that at least one of the condition 1 and the condition 2 is not satisfied. (Step S11; NO), the process proceeds to the determination process of step S15.

  If the integration unit 6 determines in the determination process in step S11 that both condition 1 and condition 2 are satisfied (step S11; YES), then the integration unit 6 continues to input pixel pi, j and input pixel pi, j as shown in FIG. In the same manner as described above, it is determined whether or not the following condition 3 or condition 4 is satisfied for the pixel pi, j-1 that is input before j is input and is adjacent to the input pixel pi, j. It performs (step S12).

[Condition 3] Edge intensity ΔDlower (i, j) between the luminance Di, j of the input pixel pi, j and the luminance Di, j-1 of the pixel pi, j-1 adjacent below, that is,
ΔDlower (i, j) = | Di, j−Di, j−1 | (6)
Is less than the preset first luminance threshold value ΔDth.

[Condition 4] As shown in FIG. 12, the average value Dave-lower of the luminance D i, j of the input pixel p i, j and the luminance D of all the pixels belonging to the group g to which the adjacent pixel p i, j−1 belongs. Mean value difference δDlower (i, j), that is,
δDlower (i, j) = | Di, j−Dave-lower | (7)
Is less than the preset second luminance threshold value δDth.

  When the integration unit 6 determines that at least one of the condition 3 and the condition 4 is not satisfied (step S12; NO), the integration unit 6 defines the input pixel pi, j as the pixel pi, j-1 adjacent thereto below. Although not integrated, since it is determined in the determination process in step S11 that the above conditions 1 and 2 are satisfied, the input pixel pi, j and the pixel pi-1, j adjacent to the left are integrated into one group. (Step S13).

  At this time, as shown in FIG. 9, for example, if the pixel pi-1, j adjacent to the left is not grouped with other pixels, the pixel pi-1, j adjacent to the input pixel pi, j is left. Are integrated into one group, and a group of two adjacent pixels on the left and right is newly formed. For example, as shown in FIG. 10, if the pixel pi-1, j adjacent to the left is integrated with other pixels and belongs to the group g, the input pixel pi, j is added to the group g. The group g is enlarged by one pixel by the input pixel pi, j.

  When the group g to which the pixel pi-1, j adjacent to the left belongs has a shape as shown in FIG. 13A, for example, the determination process in step S12 satisfies at least one of the condition 3 and the condition 4. Even if the input pixel pi, j is not integrated with the lower adjacent pixel pi, j-1 as shown in FIG. 13B, it is determined that the input pixel pi, j is not integrated (step S12; NO). By integrating with the pixels pi-1, j adjacent to the left (step S13), the input pixel pi, j is integrated with the adjacent pixels pi, j-1 and one group g as a result. In some cases.

  Next, in the determination process in step S12, the integration unit 6 determines that both the condition 3 and the condition 4 are satisfied (step S12; YES), and the input pixel pi, j and the adjacent pixel pi, j-1 and the pixel pi-1, j adjacent to the left are integrated into one group (step S14).

  At this time, for example, as shown in FIG. 11, if the pixel pi, j-1 adjacent below and the pixel pi-1, j adjacent to the left are not grouped with other pixels, the input pixel pi, j and the pixel pi, j-1 adjacent to the lower side and the pixel pi-1, j adjacent to the left are integrated into one group to form a new group of three pixels. Further, as shown in FIGS. 10 and 12, for example, either the pixel pi, j-1 adjacent to the lower side or the pixel pi-1, j adjacent to the left side is integrated with the other pixels, and the group g , The input pixel pi, j and the pixel that does not belong to the group g are integrated so as to be added to the group g, and the group g is expanded by two pixels.

  For example, as shown in FIG. 14A, when the pixel pi-1, j adjacent to the left belongs to the group g1, and the pixel pi, j-1 adjacent to the lower belongs to another group g2, When the input pixel pi, j is integrated with the pixel pi, j-1 adjacent to the lower side and the pixel pi-1, j adjacent to the left side (step S14), as shown in FIG. The group g1 and the group g2 are integrated through j to form one group g.

  On the other hand, when the integration unit 6 determines that at least one of the condition 1 and the condition 2 is not satisfied in the determination process of step S11 (step S11; NO), the integration unit 6 proceeds to the determination process of step S15, and similarly to the above. Then, it is determined whether or not the conditions 3 and 4 are satisfied (step S15).

  If the integration unit 6 determines that both the condition 3 and the condition 4 are satisfied (step S13; YES), the integration unit 6 determines that at least one of the condition 1 and the condition 2 is not satisfied in the determination process of step S9. Therefore (step S9; NO), the input pixel pi, j and the pixel pi-1, j adjacent to the left are not integrated, but only the pixel pi, j-1 adjacent below is integrated into one group ( Step S14). At this time, as a result of integrating the input pixel pi, j and the lower adjacent pixel pi, j-1 into one group (step S16), the input pixel pi, j may be integrated with the left adjacent pixel pi-1, j. This can be easily inferred from the cases shown in FIGS.

  If the integration unit 6 determines in step S15 that at least one of condition 3 and condition 4 is not satisfied (step S15; NO), the integration unit 6 sets the input pixel pi, j to the pixel adjacent to the left. Neither pi-1, j nor the adjacent pixels pi, j-1 are integrated and registered as a new group g to which only the input pixel pi, j belongs (step S17). In the integration process, the number of pixels of the integrated group g is a very small group that can be ignored as noise and the like. When the number of pixels does not increase any more, such a group g is registered as a group. It is also possible to configure so that it is not included in the determination of whether or not it is the road surface group G.

  When the integration unit 6 integrates the input pixels pi, j into the group g (steps S13, S14, S16), for example, as shown in FIG. 14B, a plurality of groups are integrated into one group. For example, the group number of the group g integrated into one is updated as necessary, for example, by selecting the smallest number among the group numbers of the plurality of groups to be integrated. When the input pixel pi, j is registered as a new group g (step S17), a new group number is assigned to the new group g (step S18).

  Further, the integrating means 6 integrates the input pixel pi, j into the group g (steps S13, S14, S16) or registers the input pixel pi, j as a new group g (step S17). If there is a change in the coordinates of the left and right pixels in the horizontal line j, the coordinates of the left and right and upper and lower pixels of the entire group g, the coordinates of the center point of the group g, etc., it is calculated and updated. The information is stored in a storage means (not shown).

  Subsequently, when the input unit pi, j is registered as a new group g (step S17), the integration unit 6 sets a later-described condition satisfaction flag F associated with the group g to 0 (step in FIG. 5). S19).

  Further, when the input pixels pi, j are integrated into the group g (step S14), and the plurality of groups are integrated into one group (see, for example, FIG. 14B), the integration unit 6 If there is a group whose condition satisfaction flag F is 1 among a plurality of groups targeted for integration, the condition satisfaction flag F of the group g integrated into one is set to 1 and becomes a target of integration. If all the condition satisfaction flags F of the plurality of groups are all 0, the condition satisfaction flag F of the group g integrated into one is set to 0 (step S19).

  Subsequently, if the condition satisfaction flag F of the group g that has undergone the integration process is currently 0 (step S20; YES), the integration unit 6 performs pixel processing on the horizontal line j (pixel row j) of the group g. The number Nj is counted (step S21). Then, referring to the pixel number threshold value Nth (j) in the horizontal line j set in the setting process in step S3 or step S10 in FIG. 3, if the counted pixel number Nj is equal to or larger than the pixel number threshold value Nth (j), (Step S22; YES), the condition satisfaction flag F of the group g is switched to 1 (Step S23). If the counted pixel number Nj is not greater than or equal to the pixel number threshold Nth (j) (step S22; NO), the condition satisfaction flag F of the group g remains at 0.

  As can be seen from the fact that the condition establishment flag F of the group g is switched to 1 when the number of pixels Nj in the horizontal line j of a group g is greater than or equal to the pixel number threshold Nth (j) (step S23), the condition establishment flag is 1 means that there is a pixel row j in each pixel row j of the group g that has a pixel number threshold Nth (j) or more, that is, a lateral width of 60 cm or more in real space.

  And if the imaging object on the road surface for the group g has a portion having a width of 60 cm or more in real space, it is not the white line block of the pedestrian crossing marked on the road surface, but the road surface. This is considered to be the road surface area. Therefore, as described later, if the condition satisfaction flag F is 1, the group g may be detected as the road surface group G. However, if the condition satisfaction flag F is 0, the group g is the road surface group. Excluded from G detection.

  For this reason, if the condition satisfaction flag F of the group g currently subjected to the integration process is already 1 (step S20; NO), the group g has a pixel row that is equal to or greater than the pixel number threshold Nth (j). Since j already exists, it is not necessary to perform each processing of steps S21 to S23 again. Therefore, if the condition establishment flag F of the group g for which the integration process is currently performed is 1 (step S20; NO), the process is skipped without performing each process of steps S21 to S23.

  The integration unit 6 calculates the number N of pixels of the group g subjected to the integration process (step S24). That is, when the input pixel pi, j is registered as a new group g, the number of pixels N of the group g is 1, and when the input pixel pi, j is integrated into the group g, the number of pixels N of the group g. Is increased by 1, and when the other pixel p is also integrated into the group g by integrating the input pixels pi, j into the group g, the number of pixels N of the group g is increased by 2, When the two groups g1 and g2 are integrated through the input pixel pi, j to form one group g, the number of pixels N of the group g integrated into the one is equal to the number of pixels in the groups g1 and g2. The number of pixels is calculated by adding one pixel of the input pixel pi, j to the sum.

  Subsequently, the integration unit 6 calculates the sum of the luminance D of each pixel p belonging to the group g, divides it by the number N of pixels of the group g, and calculates the luminance D of each pixel p belonging to the group g. The average value Dave is calculated and updated (step S25). In this case, when the input pixel pi, j is registered as a new group g, the number of pixels N of the group g is 1, so that the luminance Di, j of the input pixel pi, j belongs to the group g. The average value Dave of the luminance D of the pixel p is obtained.

  Further, the integration unit 6 determines whether or not the input pixel pi, j is in the check region R based on the check region R set in the setting process in step S2 of FIG. 3 (step S26). If pi, j is in the check region R (step S26; YES), the number of pixels An that the group g occupies in the check region R is calculated as an area (step S27).

  That is, when the input pixel pi, j is in the check region R (step S26; YES), when the input pixel pi, j is registered as a new group g, the number of pixels in the check region R of the group g When An is set to 1 and only the input pixels pi, j are integrated into the group g, the number of pixels An in the check region R of the group g is increased by 1.

  Further, when the other pixel p is also integrated into the group g by integrating the input pixel p i, j into the group g, if the other pixel p is within the check region R, the group g When the number An of pixels in the check region R is increased by 2 and the other one pixel p is not in the check region R and only the input pixel pi, j is in the check region R, the group g The number of pixels An in the check region R is increased by 1.

  Further, when two groups g1 and g2 are integrated into one group g via the input pixel pi, j, the number of pixels An in the check region R of the group g integrated into the one group is expressed as a group g. The pixel number is calculated by adding one pixel of the input pixel pi, j to the sum of the pixel number An of each area A of g1 and g2.

  Then, the integration unit 6 repeats each of the above processes for the input pixel pi, j until the integration process of the uppermost horizontal line of the image T is completed (step S8 in FIG. 3).

  When the integration process is completed up to the uppermost horizontal line of the image T by the integration means 6 (step S8; YES), each pixel p is integrated into each group g, and finally each pixel p of the image T is a plurality of groups. divided into g. For example, each pixel p of the image T shown in FIG. 8 is integrated into a plurality of groups g as shown in FIG.

  Next, the road surface detection means 7 divides the total number of pixels N belonging to the group g calculated in the calculation process of step S24 of FIG. 5 by the number of vertical pixel rows j of the group g to calculate the number of pixels. The average value Nave is calculated, and the average value Wave of the horizontal width of each group g is calculated as the average value Nave of the number of pixels (step S28 in FIG. 6). The average value Wave of the width is information used to prevent the group g corresponding to the marking such as the lane marked more narrowly than each white line block of the pedestrian crossing described above from being detected as the road surface group G. is there.

  Using only the pixel number threshold value Nth (j), a group g in which a pixel row j having a horizontal width of 60 cm or more as a horizontal width in the real space exists in each pixel row j of the group g is designated as a road surface group G. For example, as shown in FIG. 16, when a stop line SL or the like that is continuously displayed in the image T is captured in the image T, the lanes LL and LR are supported. The number of pixels Nj in the horizontal line j of the group g to be increased at the portion of the stop line SL and becomes equal to or greater than the pixel number threshold Nth (j). Therefore, there is a possibility that the group g clears the determination process (step S22 in FIG. 5) based on the pixel number threshold Nth (j), the condition establishment flag F is set to 1, and is detected as the road surface group G. .

  Therefore, in order to avoid this, in the present embodiment, the road surface detection means 7 calculates the average width Wave of the group g as described above (step S28 in FIG. 6). If it is less than the average threshold Nave_th set to the number of pixels corresponding to a slightly larger width than the lane marked with a width of about 15 cm on the road surface (step S31; NO described later), the group g is the lane LL. , LR and the like may be a group g. Therefore, the group g is not detected as the road surface group G.

  Note that the lateral width of the lanes LL, LR and the like captured in the image T is very small and the number of pixels is small compared to the lateral width of the road surface. Therefore, in the present embodiment, in the calculation process for calculating the average value Wave of the horizontal width, for the purpose of simplifying the process, instead of calculating the horizontal width in the real space, all of the groups belonging to the group g as described above are used. The average width Nave is calculated as the number of pixels obtained by dividing the number of pixels N by the number of rows of pixel rows j in the vertical direction of group g.

  Subsequently, the road surface detection means 7 determines for each group g whether or not the group g is the road surface group G, and if the condition is satisfied, detects the group g as the road surface group G. To register.

  Specifically, if there is a group g that has not yet been determined (step S29; YES), the road surface detection means 7 first determines whether or not the condition satisfaction flag F of the group g is 1. (Step S30) If the condition satisfaction flag F is 0 (Step S30; NO), the group g does not have a width portion of 60 cm or more in the real space, and corresponds to each white line block of the pedestrian crossing. Therefore, the group g is excluded from the detection target of the road surface group G, and the determination process for the group g is ended.

  Subsequently, the road surface detection means 7 determines whether or not the average value Nave of the width as the number of pixels calculated for the group g in the calculation process of step S28 is equal to or greater than a set average value threshold value Nave_th ( Step S31), if the average width Nave is not equal to or greater than the average threshold Nave_th (Step S31; NO), the group g is excluded from the detection target of the road surface group G as described above, and the group g The determination process ends.

  Subsequently, the road surface detection means 7 determines that the average value Dave of the luminance D of each pixel p of the group g calculated in the calculation process of step S25 in FIG. 5 is equal to or less than a predetermined luminance average threshold value Dave_th (step S32; NO), the group g is excluded from detection targets of the road surface group G, and the determination process for the group g is ended.

  Usually, the road surface area of the road surface is imaged with dark luminance. Therefore, the luminance average threshold value Dave_th is set to a luminance that is low enough to be detected as the luminance D of the pixel p in the road surface area of the road surface. Then, the group g corresponding to the marking on the road surface such as a pedestrian crossing, a stop line, a lane, a number, a character, and an arrow is excluded from the detection target of the road surface group G in the determination processing of the above step S30 and step S31. Even if not, it is excluded from the determination target in the determination process of step S32.

  Further, as shown in FIG. 15, when the group gsky corresponding to the sky is captured in the image T, the group gsky corresponding to the sky has high brightness, so that the group gsky also performs the determination process in step S32. Thus, the road surface group G is excluded from detection targets. However, in the case of cloudy weather or rainy weather, the group gsky corresponding to the sky may be imaged with low luminance. In addition, as shown in FIG. 15, the group gtree corresponding to the roadside roadside tree may also clear the determination processes in the above steps S30 to S32.

  Therefore, the road surface detection means 7 subsequently has the number of pixels An as the area occupied in the check region R of the group g calculated in the calculation process of step S27 in FIG. 5 equal to or greater than a preset area threshold Ath. It is determined whether or not there is (Step S33), and if the pixel number An is not equal to or larger than the area threshold Ath (Step S33; NO), the group g is excluded from the detection target of the road surface group G, and the group g This determination process ends.

  With this configuration, the number of pixels An in the check region R of the groups gsky and gtree corresponding to the sky and the roadside tree shown in FIG. 15 becomes 0. Therefore, these groups gsky and gtree are set in step S33. It becomes possible to exclude from the detection target of the road surface group G by the determination process.

  When the group g clears all the determination processes in steps S30 to S33, the road surface detection means 7 detects the group g as the road surface group G and registers it in the storage means (step S34). If there is a group g that has not been determined (step S29; YES), the processing of steps S30 to S34 is repeated.

  In addition, when the integration unit 6 and the road surface detection unit 7 complete the processing for all the groups g (step S29; NO), the processing for the image T for one image input from the imaging unit 2 in the current sampling cycle. When the imaging unit 2 starts imaging in the next sampling cycle (step S1 in FIG. 3), each process from step S2 is executed again.

  For example, among the groups g shown in FIG. 15, groups G1 to G4 are detected as road surface groups G as shown in FIG. Then, by connecting these groups G1 to G4 on the image T, it is possible to extract the road surface area of the road surface from the image T. Each information of the groups G1 to G4 as the road surface group G is output to an external device as necessary.

  Next, the lane candidate point detection means 8 searches for an image portion other than the road surface group G (G1 to G4) in the image T (see FIGS. 15 and 17) divided into the groups g as described above. The lane candidate points cl and cr corresponding to the edge portion of the lane are detected.

  In the present embodiment, the lane candidate point detection means 8 includes the luminance D of the road surface group G (that is, each pixel of the road surface group G described above) among the pixels p of the image portion other than the road surface group G (G1 to G4). Pixels cl and cr whose difference from the average value Dave) of the luminance D of p is equal to or greater than a predetermined threshold and whose luminance difference with the adjacent pixel p changes to be equal to or greater than the predetermined threshold are defined as lane candidate points cl and cr. It comes to detect. The average value Dave of the luminance D of the road surface group G is calculated by the integration unit 6 as described above (see step S25 in FIG. 5).

  As in the case of the determination process based on condition 2 in the integration unit 6 described above, here the luminance Di, j of the pixel (pi), which is a predetermined target (hereinafter referred to as a search pixel), and the road surface group The difference from the average value Dave of the luminance D of G is called the average value difference, and is represented by δD. Unlike the processing in the integration unit 6, the lane candidate point detection unit 8 uses the search pixel pi, j and the road. The face group G is not necessarily adjacent. Further, the threshold value for the average value difference δD is set to an appropriate value regardless of the second luminance threshold value δDth in the determination process based on the condition 2 in the integration unit 6, and is represented here as δDth1.

  Further, the luminance difference between the search pixel pi, j and the adjacent pixel p is referred to as edge strength ΔD as in the case of the determination process based on condition 1 in the integration unit 6 described above, but the threshold for the edge strength ΔD is the integration unit. 6 is set to an appropriate value regardless of the first luminance threshold value ΔDth in the determination process based on the condition 1 in FIG.

  Hereinafter, the detection process of the lane candidate points cl and cr in the lane candidate point detection means 8 of the present embodiment will be specifically described.

  The lane candidate point detection means 8 first calculates the turning curvature Cua using the above formula (1) or (2) and (3) for setting the check region R by the integration means 6 as described above. If the estimated trajectory Lest of the host vehicle is calculated based on that, it is used. If not, the estimated trajectory Lest is calculated in the same manner as described above, and as shown in FIG. The position of the estimated locus Lest is calculated.

  FIG. 18 shows a case where the lane detection device 1 includes a three-dimensional object detection unit, and three-dimensional objects O and S existing above the road surface are detected in the image T by the three-dimensional object detection unit. Yes. The lane is marked on the road surface and is not above the road surface. Therefore, if the three-dimensional object detection means is provided in the lane detection device 1, it is not necessary to search for an image area in which the three-dimensional object is detected in the image T in the lane detection, so that the lane search range can be narrowed. Become.

  Subsequently, the lane candidate point detecting means 8 reads out the data of the luminance D of each pixel p of the horizontal line j for one line of the image T from the storage means, and horizontally from the position of the estimated locus Lest of the image T in the horizontal direction. The line j is searched, and lane candidate points cl and cr are detected. This process is started from the horizontal line j at the lower end of the image T, that is, the horizontal line 0, and the horizontal line j is moved upward by one pixel at a time. The search in the horizontal direction on the horizontal line j may be performed from the position of the estimated trajectory Lest, or may be configured to be performed from the center position of the image T or the like.

  Further, the lane candidate point detecting means 8 reads the information on the road surface group G detected by the road surface detecting means 7 as described above, that is, the information on the road surface groups G1 to G4 in the above case from the storage means, and this horizontal line In the search on j, pixels belonging to the road surface groups G1 to G4 on the horizontal line j are skipped without searching.

  This is because the pixels belonging to the road surface group G are not pixels corresponding to at least the lane. Also, with this configuration, it is possible to avoid performing a useless search, and to significantly reduce the processing time. Since the pixel p at the position of the estimated trajectory Lest often belongs to the road surface group G, in most cases, the search is skipped immediately after the search is started. Hereinafter, a case where the horizontal line j is searched for in the right direction will be described.

  The lane candidate point detecting means 8 obtains the average value Dave of the luminance D of the road surface group G from the search pixel pi, j adjacent to the right of the pixel p belonging to the road surface group G on the horizontal line j as described above. Or the average value difference δD is greater than or equal to the threshold value δDth1, and the luminance difference from the pixel p adjacent to the left, that is, the edge strength ΔD, is determined to be greater than or equal to the threshold value ΔDth1. When there are a plurality of road surface groups G1 to G4 on the horizontal line j, the average value Dave of the luminance D of the road surface group G adjacent to the left of the search pixel pi, j that has started the determination is used.

  When the luminance Di, j of the search pixel pi, j is searched, for example, the transition is made as shown in FIG. 19, and the lane candidate point detecting means 8 sets the search pixel pi, j satisfying the above condition as the lane candidate point cr. It comes to detect. The group g to which the lane candidate point cr belongs and the group g existing on the right side thereof are detected as the lane group Gline and registered in the storage means.

  In the present embodiment, the lane candidate point detection means 8 detects a lane end point (hereinafter referred to as a lane end point) cre corresponding to the lane candidate point cr detected on the horizontal line j. Yes.

  In this embodiment, the lane candidate point detection means 8 skips the search for the lane end point cre for the group g (lane group Gline) to which the lane candidate point cr belongs and searches on the horizontal line j. When the pixel belongs to another group g, the following determination is performed. If the pixel is not the lane end point cre, the other group g is registered as a lane group Gline, and the group g (lane group For Gline), the lane end point cre is searched by skipping without searching for the lane end point cre.

  In addition, by skipping the search for the lane end point cre in the lane group Gline in this way, it is possible to avoid performing a useless search and further reduce the processing time.

  In this embodiment, the difference in luminance between the search pixel pi, j and the lane group Gline, that is, the average value difference δD is equal to or less than a predetermined threshold value δDth2 set to a negative value, and the pixel p adjacent to the left When the luminance difference ΔD is equal to or less than a predetermined threshold value ΔDth2 set to a negative value, the search pixel pi, j is detected as the lane end point cre.

  Moreover, as shown in FIG. 17 etc., the road surface group G may be detected on the right side of the lane group Gline. In such a case, after searching the lane group Gline on the horizontal line j, if the road surface group G is found again, the pixel that switches from the lane group Gline to the road surface group G is set as the lane end point. It can be detected as cre.

  Therefore, in this embodiment, the lane candidate point detection means 8 searches on the horizontal line j of the image T, and when a pixel that does not belong to the lane group Gline belongs to the road surface group G, the pixel is The pixel is detected as the lane end point cre regardless of whether the detection condition of the lane end point cre is satisfied.

  However, the single or plural high-intensity groups g registered as the lane group Gline existing between the lane candidate point cr and the lane end point cre may not actually be the lane group Gline corresponding to the lane.

  Therefore, for example, when the lane end point cre is detected, the number of pixels between the lane candidate point cr and the lane end point cre is counted, and the horizontal direction in the real space per pixel for each horizontal line j described above. The distance in the real space between the lane candidate point cr and the lane end point cre is estimated by multiplying the width of the lane, and the lane candidate point cr and the lane end point are found when the distance is too wide or too narrow. It is also possible to delete the cre registration and delete it. In addition, it is also possible to configure to detect an interval in the real space between the lane candidate point cr and the lane end point cre by a method such as stereo matching.

  In the present embodiment, the lane candidate point detection means 8 includes the coordinates of the upper and lower pixels in the image T of the lane group Gline calculated by the integration means 6 in the image T of the lane group Gline. When the maximum length in the vertical direction is calculated and the maximum length is equal to or smaller than a predetermined threshold, the registration of the lane group Gline is canceled and the registration of the lane candidate point cr and the lane end point cre is performed. It is supposed to be deleted.

  Even if the lane candidate point cr and the lane end point cre that satisfy the above detection conditions are detected on the horizontal line j, the maximum length in the vertical direction in the image T of the group g existing between them is shortened. For example, the group g is considered to be high-intensity noise reflected in the image T. In this case, the registration of the group g as the lane group Gline and the registration of the lane candidate point cr and the lane end point cre are deleted. This is because it is possible to prevent erroneous detection of the lane candidate points cr and the like.

  Further, in this embodiment, the lane candidate point detection means 8 determines the real space of the lane group Gline from the coordinates of the left and right end pixels in each horizontal line j of the lane group Gline calculated by the integration means 6. The upper area and the horizontal length are calculated, and among the lane group Gline existing between the lane candidate point cr and the lane end point cre, only the group g having the largest area is registered as the lane group Gline. When the area and the lateral length of the registered lane group Gline are equal to or greater than a predetermined threshold, the registration of the lane group Gline is deleted and the registration of the lane candidate point cr and the lane end point cre is deleted. It is like that.

  Even if a lane candidate point cr and a lane end point cre satisfying the above detection condition on the horizontal line j are detected, the group g having the largest area between them satisfies the above condition. This is because the group g is not the lane group Gline corresponding to the lane but is considered to be, for example, snow accumulated on the side of the road. In such a case, the registration of the group g as the lane group Gline and the registration of the lane candidate point cr and the lane end point cre are deleted and deleted, thereby preventing erroneous detection of the lane candidate point cr and the like. This is because it becomes possible.

  Therefore, in this case, the threshold values relating to the area and the lateral length accurately exclude such accumulated snow and the like, while the group g corresponding to the lane marked in a continuous line is exactly the lane group Gline. Is set to a value that can be detected as.

  In the present embodiment, specific information indicating that it is snow or the like is registered in the storage unit in association with the group g that has been deregistered as the lane group Gline satisfying this condition. Yes. Even if the lane candidate point cr is detected in the subsequent processing, if this specific information is associated with the corresponding group g, the registration of the lane candidate point cr is immediately deleted and deleted. It is supposed to be.

  In this embodiment, even if the lane candidate point detection means 8 detects the lane candidate point cr or the lane end point cre on the horizontal line j, the lane candidate point detection means 8 continues the search on the horizontal line j and a plurality of lane candidate points on the horizontal line j. If lane candidate points cr are detected, all of them are detected and registered.

  Further, the lane candidate point detecting means 8 is configured to continue the search for at least one width of the traveling road after the road surface group G is finally detected on the horizontal line j. The width of the travel path may be set based on the distance between the left and right lanes of the travel path of the host vehicle detected in the past sampling cycle, or a value corresponding to the width of the travel path may be set in advance. Good.

  As described above, when the lane candidate point detection unit 8 finishes the search on the horizontal line j for one line of the image T, the lane candidate point detection unit 8 moves the horizontal line j upward by one pixel and again estimates the image T, for example. A search is made on the horizontal line j in the left-right direction from the position of the locus Lest, and the lane candidate point cr and the lane end point cre are detected. At that time, the search is performed in the same manner as described above. For example, pixels belonging to the road surface group G and the lane group Gline are skipped without being searched.

  Further, as described above, the lane candidate point detection means 8 determines the search pixel pi, j when the search pixel pi, j belonging to the group g other than the road surface group G on the horizontal line j satisfies the above detection condition. Detected as a lane candidate point cr, but when the lane group G is detected by searching on the horizontal line j, the lane group G is shifted from another group g without performing the determination based on the above detection condition. The search pixel pi, j at the time may be configured to be immediately detected as the lane candidate point cr. If comprised in this way, it will become possible to aim at the further speed-up of a process.

  In addition, the lane candidate point detection means 8 searches the horizontal line j and detects the group g associated with the specific information indicating that it is snow or the like, the determination based on the detection condition described above Without performing the search, the pixels belonging to the group g are skipped without being searched.

  If the lane candidate point cr is detected by the lane candidate point detection means 8 using the image T shown in FIG. 26, for example, as described above, it corresponds to the lane branching to the right turn lane as shown in FIG. Not only the lane candidate point cr to be detected, but also the lane candidate point cr corresponding to the main line that the host vehicle is going to travel, that is, the lane marked on the travel path of the host vehicle is detected. In the scene shown in FIG. 20, as described above, the left lane is covered with snow, and the left lane candidate point cl is not detected.

  The lane detection means 9 detects the lanes LL and LR marked on the travel path of the host vehicle in the image T based on the lane candidate points cl and cr detected by the lane candidate point detection means 8. .

Specifically, the lane detection means 9 is based on the lane candidate points cl and cr each time the lane candidate point detection means 8 searches the horizontal line j of the image T and detects the lane candidate points cl and cr. Hough conversion is performed. That is, when the lane candidate point detection unit 8 detects the lane candidate point cr at, for example, the position (Ij, Jj) on the horizontal line j, the lane detection unit 9 indicates that the lane candidate point cr is ρ = isinθ + jcosθ (8)
Assuming that it exists above, substitute (Ij, Jj).
ρ = Ijsinθ + Jjcosθ (9)

  Then, the above equation (9) is regarded as a function of ρ and θ, and as illustrated in FIG. 21, the count number of each cell on the Hough plane through which the function curve represented by the above equation (9) passes is increased by 1. . Thus, the lane detecting means 9 performs Hough transform based on the lane candidate point cr every time the lane candidate point cr is detected, and increases the count number of each cell on the Hough plane through which the function curve passes. I will let you. In addition, a Hough plane is created for the lane candidate point cl on the left side of the image T separately from the lane candidate point cr and similarly counted for each cell.

  A cell with a large number of counts is a cell through which many function curves pass, and a straight line obtained by substituting ρ and θ corresponding to the cell into the above equation (8) passes through many lane candidate points cl and cr. It becomes a straight line. Therefore, the lane detection unit 9 checks the count number of each cell on the Hough plane when the detection of the lane candidate points cl and cr by the lane candidate point detection unit 8 is completed, and the ρ of the cell having the peak count number And θ are detected.

  When a straight line is calculated by substituting ρ and θ of the square having the count number into the above equation (8), a large number of lane candidate points cl and cr are gathered in the vicinity of the straight line. When straight lines are calculated in this way, for example, as shown in FIG. 22, straight lines r1 to r3 as lane candidates are extracted from the result of the Hough transform relating to each lane candidate point cr described above.

  In the present embodiment, the lane detecting means 9 determines the type of each straight line r1 to r3 based on the continuity of the lane candidate points cr existing in the vicinity of each straight line r1 to r3. The straight lines r1 to r3 are calculated based on the type of the straight lines r1 to r3, the number of lane candidate points cr existing in the vicinity of the straight lines r1 to r3, and the straight lines based on the reliability. A straight line representing a lane is selected from r1 to r3.

  In the present embodiment, the type of straight line is determined based on the continuity of the lane candidate points cr gathered in the vicinity of the straight lines r1 to r3, whether the straight line is a continuous line, a broken line, or any other state ( For example, it is determined as a lane branch state) or a combination thereof. For example, it is determined that the straight line r1 is a broken line, the straight line r2 is a continuous line, and the straight line r3 is a lane branching state.

  In addition, the reliability based on the type of straight line is highest when the straight line is a continuous line, and when the straight line is a broken line, the reliability is medium, and other states (for example, lane branching state). Is assigned to each of the straight lines r1 to r3. Therefore, when viewed only from the reliability based on the type of straight line, the reliability of the straight line r2 is the highest among the straight lines r1 to r3.

  However, in the case of FIG. 22, since the number of lane candidate points cr existing in the vicinity of each straight line r1 is much larger than the number of lane candidate points cr existing in the vicinity of the straight lines r2 and r3, the reliability of the straight line r1. The degree is the highest. Therefore, the lane detecting means 9 selects the straight line r1 as a straight line suitable as a lane from the straight lines r1 to r3, and detects it as a lane LR marked on the traveling path of the host vehicle as shown in FIG. ing.

  In FIG. 22 and the like, it is expressed that only a few lane candidate points cr are detected, but in reality, a large number of lane candidate points cl and cr are detected. In FIG. 22 and the like, the lane candidate point cl is not detected on the left side of the image T. However, when the lane candidate point cl is detected, the lane detection unit 9 similarly performs the lane candidate point cl. A straight line is extracted from the result of the Hough transform on the vehicle, a straight line having the highest reliability is selected, and detected as a lane LL marked on the travel path of the host vehicle.

  Further, when selecting the straight line r1 etc. from the straight lines r1 to r3 etc., a straight line that is not suitable as a lane is excluded from selection based on the position, behavior, etc. of the own vehicle and the parallelism of the straight lines on both sides It is also possible to configure to perform processing. Further, in the present embodiment, a case has been described in which one straight line is selected on each of the left and right sides of the host vehicle as a straight line suitable as a lane as described above. However, the host vehicle is associated with information such as reliability and continuity. It is also possible to configure such that a plurality of straight lines are detected as lanes on the right and left sides of the vehicle.

  Further, the continuity of the lane candidate points cl and cr serving as a reference for determining the type of straight continuous line, broken line or the like is the distance between the lane candidate points cl or the lane candidate points cr on the image T, that is, pixels. For example, when the number of pixels between the lane candidate points cl and cr and the distance in the real space are continuous within a predetermined threshold, for example, based on the number and the distance in the real space calculated by a method such as stereo matching. The straight line is determined to be a continuous line, and when there is a portion where the number of pixels or the distance is more than a predetermined threshold, it is determined to be a broken line.

  Further, when the lane candidate points cl or the lane candidate points cr belong to the same group g integrated by the integration unit 6, it is determined to be continuous regardless of the determination based on the number of pixels and the distance. It can be configured to do so. As described above, when the lane candidate point cl and the lane candidate point cr belong to the same group g, it is possible to easily and accurately determine the type of straight line by determining that the lane candidate point cl and the lane candidate point cr belong to the same group g.

  Next, the operation of the lane detector 1 according to this embodiment will be described.

  In the conventional lane detection, in order to shorten the execution time of the detection process to ensure real-time performance and to suppress erroneous detection of the lane candidate points cl and cr, the search areas of the lane candidate points cl and cr are set to, for example, the past. It was limited to the search area estimated from the position of the lane detected in the sampling period. Therefore, as shown in FIG. 26, only the lane that branches to the right turn lane is detected, and the lane candidate point (see the white circle in the figure) corresponding to the main line that the host vehicle is originally trying to travel can be detected. There was a possibility that it could not be done.

  On the other hand, in the lane detection device 1 according to the present embodiment, first, when the data of the luminance D of each pixel p is input from the imaging unit 2 by the integration unit 6, the pixel p having the same luminance D is simultaneously acquired. The integration process is grouped, and the integration process is completed at almost the same timing when the data of the luminance D of all the pixels p for one image of the image T is input.

  Then, the road surface detection unit 7 detects the road surface group G (for example, road surface groups G1 to G4 in FIG. 17) according to the average luminance Dave and area of each group g already calculated by the integration unit 6, and the lane The candidate point detection means 8 searches and detects the lane candidate points cl and cr in the image portion other than the road surface group G.

  At that time, since the lane candidate points cl and cr corresponding to the high-brightness lane do not belong to the road surface group G corresponding to the dark-bright road surface area, it is necessary to search for the pixels belonging to the road surface group G. There is no. Therefore, it is possible to skip the road surface group G without performing a search. Further, since the average luminance Dave of the road surface group G used when detecting the lane candidate points cl and cr has already been calculated by the integration unit 6, it is not necessary to calculate again.

  Therefore, the lane candidate points cl and cr can be detected at a very high speed, and the lane detection means 9 can detect the lanes LL and LR at a high speed based on the lane candidate points cl and cr. . Therefore, for example, even when images of several frames to several tens of frames are captured per second and data is input, lanes LL and LR can be detected quickly, and lane detection is performed in real time. It becomes possible to do.

  Further, the lane candidate point detection means 8 does not limit the search area of the lane candidate points cl and cr, and the road surface (that is, the road surface other than the road surface ground portion such as the image portion other than the road surface group G, that is, the asphalt surface). The lane candidate points cl and cr are detected by searching for a lane marking portion marked on the surface).

  Therefore, for example, as shown in FIG. 20 which is the same image T as FIG. 26, not only the lane candidate point cr corresponding to the lane branching to the right turn lane but also the lane candidate point corresponding to the main line on which the host vehicle is going to travel. cr can also be accurately detected. And even if it is a case where a lane branches as shown in FIG. 23, it is possible to accurately detect the lane LR corresponding to the main line on which the host vehicle is going to travel based on the lane candidate points cr. It becomes.

  Moreover, if comprised like this embodiment, as shown, for example in FIG. 24, even when a lane is a double lane, it becomes possible to detect the lane LR accurately and stably.

  That is, in the conventional lane detection, the search area for the lane candidate points cl and cr is limited as described above. Therefore, as shown in FIG. 25, for example, the edge portion of the inner lane of the double lane is detected. However, the inner lane is often thinned by the tires of the vehicle and may have disappeared.

  In such a case, if the search area is limited to the inner lane of the double lane, the inner lane becomes thinner or disappears, so the lane candidate points cl and cr are not detected or detected. The position of the detected lanes LL and LR also becomes unstable due to the position becoming unstable, and the reliability of the detected lanes LL and LR may be lowered.

  On the other hand, in the lane detection device 1 according to the present embodiment, the lane candidate point cl is searched by searching for an image portion other than the road surface group G without limiting the search area for the lane candidate points cl and cr as described above. , Cr are detected. Therefore, as shown in FIG. 24, not only the lane candidate points cl and cr corresponding to the inner lane of the double lane, but also the lane candidate points cl and cr corresponding to the outer lane not thinned are detected. .

  Even if the inner lane is stepped on and thinned or disappeared, the lane candidate points cl and cr corresponding to the inner lane are not detected, or the detected position becomes unstable. Since the lane candidate points cl and cr corresponding to this lane are detected accurately and stably, the outer lanes LL and LR can be accurately and stably detected as shown in FIG. For this reason, the reliability of the detected lanes LL and LR can be improved.

  In a situation where the inner lane of the double lane is not thin and is clearly marked in white, the lane detection device 1 according to this embodiment also uses the lane candidate points cl and cr corresponding to the inner lane. In some cases, the inner lane may be detected as lanes LL, LR. In such a case, based on the number of pixels between the lane candidate points cl or between the lane candidate points cr, the distance in real space, or the size or shape of the group g to which the lane candidate point cl or the lane candidate point cr belongs. Since the types of the lanes LL and LR are classified as “other states (lanes inside the double lanes)”, it is clear that the lanes inside the double lanes are detected.

  In addition, if the lane detection device 1 according to the present embodiment is configured to detect a plurality of straight lines on the left and right sides of the own vehicle as lanes, the inner lane and the outer lane of the double lane are clearly defined as described above. Thus, it is possible to detect both lanes on both sides of the double lane.

  Also in this case, as the lane type, the lane inside the double lane is classified as “other state (lane inside the double lane)”, and the outside lane is the right lane in FIG. By classifying as “broken line” and “continuous line” in the left lane, it is possible to detect a double lane in a state where the type of the detected lane is clarified. In addition, it is also possible to classify | categorize as a "other state (lane outside the double lane)" etc. as a classification of the lane outside the double lane.

  As described above, according to the lane detection device 1 according to the present embodiment, the lane candidate point detection unit 8 does not set a search area for searching for the lane candidate points cl and cr, and other than the road surface group G. The road surface portion other than the ground portion of the road surface such as the image portion, that is, the asphalt surface is searched to detect the lane candidate points cl and cr.

  Therefore, for example, not only the lane candidate points cl and cr corresponding to the lane branching to the right turn lane, the lane candidate points cl and cr corresponding to the inner lane in the double lane, and the main line on which the host vehicle is about to travel The lane candidate points cl and cr corresponding to, the lane candidate points cl and cr corresponding to the outer lane in the double lane, and the like can be accurately detected.

  Therefore, even when the lane is branched, the lane LR corresponding to the main line that the host vehicle is about to travel is accurately detected based on the lane candidate points cl and cr, and the captured image T It becomes possible to accurately detect the lane marked on the traveling path of the host vehicle from the inside. Further, when the lane is a double lane, it is possible to accurately and stably detect not the inner lane that is disappearing, or the outer lane together with it, and in the captured image T Thus, it is possible to accurately detect the lane marked on the traveling path of the host vehicle.

  Further, the lane candidate point detection means 8 searches for and detects the lane candidate points cl and cr in image portions other than the road surface group G detected by the road surface detection means 7 (for example, the road surface groups G1 to G4 in FIG. 17). To do. At that time, since the lane candidate points cl and cr corresponding to the high-brightness lane do not belong to the road surface group G corresponding to the dark-bright road surface area, it is necessary to search for the pixels belonging to the road surface group G. There is no. Therefore, it is possible to skip the road surface group G without performing a search.

  Further, since the average luminance Dave of the road surface group G used when detecting the lane candidate points cl and cr has already been calculated by the integration means 6, a huge amount of histogram is created as in the conventional lane detection. Thus, there is no need to newly calculate the average value of the luminance of the pixels corresponding to the road surface.

  Therefore, the lane candidate points cl and cr can be detected at high speed, and the lane detection means 9 can detect the lanes LL and LR at high speed based on the lane candidate points cl and cr. Therefore, for example, even when images of several frames to several tens of frames are captured per second and data is input, lanes LL and LR can be detected quickly, and lane detection is performed in real time. It becomes possible to do.

  Furthermore, when the pixels p of the image T are grouped by the integration unit 6, when data of the luminance D of each pixel p is input from the imaging unit 2, the pixels p having the same luminance D are integrated in parallel therewith. If the integration processing is completed at almost the same timing when data of luminance D of all pixels p for one image of image T is input, the lane candidate points cl and cr are Further, it becomes possible to detect quickly and lane detection can be executed in real time.

  In addition, by grouping the pixels p of the image T by the integration unit 6, the number of pixels of each group g, the maximum length in the vertical direction in the image T, and the like are found when they are grouped. Even if the lane candidate points cl and cr are detected by the lane candidate point detection means 8, if the size of the corresponding lane group Gline is small, the lane candidate points cl and cr are reflected in the image T. The registration can be deleted as a response to the noise.

  Therefore, erroneous detection of the lane candidate points cl and cr can be accurately prevented, and the reliability of detection of the finally detected lanes LL and LR can be improved.

DESCRIPTION OF SYMBOLS 1 Lane detection apparatus 2 Image pickup means 6 Integration means 7 Road surface detection means 8 Lane candidate point detection means 9 Lane detection means cl, cr Lane candidate point cre Lane end point (lane end point)
Average brightness of Dave group Dave_th Brightness average threshold (predetermined threshold)
G road surface group Gline lane group g group LL, LR lane MC own vehicle p pixel r1 to r3 straight line T image ΔD edge strength (luminance difference)
ΔDth first luminance threshold (predetermined threshold)
ΔDth1 threshold value ΔDth2 threshold value δD Average value difference (luminance difference from road surface group)
δDth1 threshold δDth2 threshold

Claims (7)

Imaging means for capturing an image of the surroundings of the host vehicle;
Integration means for integrating pixels in which the luminance difference between adjacent pixels in the image is within a predetermined threshold as a group;
Road surface detection means for detecting the group as a road surface group, when the average brightness and area of the group are each within a predetermined threshold range;
Search the image portion other than the road surface group in the image, and find a pixel whose luminance difference with the road surface group is equal to or greater than a predetermined threshold and whose luminance difference with an adjacent pixel is equal to or greater than a predetermined threshold. Lane candidate point detecting means for detecting lane candidate points;
Lane detection means for detecting a lane marked on the traveling path of the host vehicle in the image based on the detected lane candidate point;
A lane detection device comprising:   The lane candidate point detecting means detects an end point of the lane corresponding to the lane candidate point, and detects and registers the group existing between the lane candidate point and the end point of the lane as a lane group. When the vertical length of the lane group in the image is equal to or smaller than a predetermined threshold, the lane group registration is canceled and at least the lane candidate point is detected from the detected lane candidate point. The lane detection device according to claim 1, wherein the lane detection device is deleted.   The lane candidate point detection means registers the lane group having the largest area among the detected lane groups, and the area or the lateral length of the registered lane group is equal to or greater than a predetermined threshold. The lane detecting device according to claim 2, wherein the lane group registration is deleted and at least the lane candidate point is deleted from the detected lane candidate point.   The lane candidate point detecting means does not search the lane candidate point for each pixel belonging to the lane group, and is a pixel that does not belong to the lane group, and a luminance difference from the lane group is predetermined. 4. The lane detection device according to claim 2, wherein a pixel that is equal to or less than a threshold value and has a luminance difference with an adjacent pixel that is equal to or less than a predetermined threshold value is detected as an end point of the lane. .   The lane candidate point detection means does not search for the lane candidate point for each pixel belonging to the lane group, and if a pixel that no longer belongs to the lane group belongs to the road surface group, The lane detection device according to any one of claims 2 to 4, wherein the lane detection device detects the lane end point.   The lane detecting means extracts one or more straight lines as lane candidates, determines the type of each straight line based on continuity of the lane candidate points existing in the vicinity of each straight line, and at least the straight line The straight line representing the lane is selected from the straight lines based on the reliability calculated based on the type of the lane and the number of lane candidate points existing in the vicinity of the straight lines. The lane detection device according to claim 5. The type of the straight line is determined based on the continuity of the lane candidate points existing in the vicinity of each straight line, whether the straight line corresponds to at least a continuous line shape, a broken line shape, or a combination thereof,
The continuity of the lane candidate points is determined based on the distance between the lane candidate points on the image or the distance in real space, and the lane candidate points belonging to the same group are continuous regardless of the determination. The lane detection device according to claim 6, wherein the lane detection device is determined to be.