JP4797846B2 - Lane marking device and lane detection device - Google Patents

Description

  The present invention relates to a lane line detection device and a lane detection device that detect a lane line (lane marker) on a road.

In order to use for lane keeping, automatic steering, etc., a device for detecting a lane marker (white line, yellow line, etc.) that is a lane marking on a road and a lane composed of the lane marker has been developed. As an apparatus for detecting a lane marker, for example, in the lane recognition image processing apparatus described in Patent Document 1, a lane marker candidate point is extracted by detecting an edge in a horizontal direction from a captured image, and the inclination between the extracted candidate points is determined. The main line lane marker is correctly recognized by comparing the above.
JP 2003-252149 A

  In the apparatus described above, the lane marker is determined by a single edge direction in the horizontal direction. However, since the direction of the lane marker changes on a curved road or the like, the edge cannot be detected as the direction becomes horizontal. As a result, the lane marker cannot be partially detected, and the detection accuracy decreases.

  Then, this invention makes it a subject to provide the lane line detection apparatus and lane detection apparatus which detect the lane which consists of a lane line and two parallel lane lines with high precision.

A lane marking detection device according to the present invention is a lane marking detection device that detects a lane marking on a road, and calculates an imaging unit that images a road and a luminance gradient vector of each pixel of an image captured by the imaging unit. A brightness gradient vector calculating means; an evaluation means for sequentially extracting a set of brightness gradient vectors from the brightness gradient vectors calculated by the brightness gradient vector calculating means; and evaluating the extracted brightness gradient vectors for each set; And a lane line detecting means for detecting a lane line from the extracted set of pixels , the luminance gradient vector calculating means comprising: For each pixel, a primary differential value of the luminance in the horizontal direction of the image and a primary differential value of the luminance in the vertical direction of the image are calculated, and the primary differential value of the luminance in the horizontal direction and the luminance in the vertical direction are calculated. And calculating the intensity gradient vectors from the primary differential value of.

  In this lane marking detection apparatus, a road is imaged by an imaging unit, and a luminance gradient vector of each pixel of the captured image is calculated by a luminance gradient vector calculation unit. The luminance gradient vector is a vector obtained by taking luminance gradients in a plurality of different directions and synthesizing the luminance gradients in the plurality of directions. In the lane marking detection device, the evaluation unit sequentially extracts a set of luminance gradient vectors composed of the luminance gradient vectors of two pixels from the luminance gradient vectors of each pixel, and the luminance gradient vector of each set is considered to be a lane line. To evaluate. When the road is marked with a lane marking (white line, yellow line, etc.), the reflectance of the road differs from that of the lane marking. So, of the two contour lines (boundary line with the road) that are parallel to the lane marking, The brightness of one contour line changes from dark to bright (from road to lane), and the other contour changes from light to dark (from lane to road). Therefore, the direction of the luminance gradient vector is opposite between the corresponding points on the two contour lines of the partition line. Therefore, if one set of luminance gradient vectors is the luminance gradient vector of corresponding points (pixels) on the two contour lines of the partition line, the direction of the vector of the one set of luminance gradient vectors is opposite. Therefore, in the lane marking detection device, the lane marking detection means extracts a set of pixels in which the direction of the luminance gradient vector is opposite based on the evaluation result for each set of luminance gradient vectors, and the extracted set of pixels. In other words, the partition line is detected from the pixels forming the two contour lines of the partition line. As described above, this lane marking detection apparatus uses a luminance gradient vector composed of luminance gradients in a plurality of directions, and is used to reverse the direction of the luminance gradient vector at each point on the two contour lines of the lane marking. By doing so, it is possible to detect a marking line in an arbitrary direction. As a result, not only straight roads but also lane markings of various curved roads can be detected accurately, and lane markings can be detected with high accuracy. In addition, by calculating the luminance gradient vector for each pixel and evaluating each set of luminance gradient vectors, it is possible to detect the lane markings in the image, so the processing load is relatively low and the lane markings are detected efficiently. it can.

A lane detection device according to the present invention is a lane detection device that detects a lane composed of two parallel lane markings on a road, and includes an imaging unit that images a road, and each pixel of an image captured by the imaging unit. A luminance gradient vector calculating unit that calculates a luminance gradient vector, and a set of luminance gradient vectors are sequentially extracted from the luminance gradient vectors calculated by the luminance gradient vector calculating unit, and the extracted luminance gradient vectors are evaluated. An evaluation unit, a lane line detection unit that extracts a set of pixels in which the direction of the luminance gradient vector is opposite based on an evaluation result by the evaluation unit, and detects a lane line from the extracted set of pixels, and a lane line detection and a lane shape estimating means for estimating a lane shape from division line detected by means brightness gradient vector calculation means, for each pixel, calculate the first-order derivative in the horizontal direction of the luminance of the image Calculating a first-order differential value of the vertical luminance of the image as well as, and calculates the brightness gradient vector from the primary differential value in the horizontal direction of the primary differential value and the vertical luminance brightness.

  This lane detection device is provided with each means similar to the lane marking detection device described above, and detects the lane marking by the same method as the lane marking detection device. In the lane detection device, the lane shape estimation means estimates the lane shape from the detected lane marking. In this lane detection device, not only straight roads but also lane shapes of various curved roads can be accurately detected from the lane lines detected with high accuracy, and lanes can be detected with high accuracy.

  In the lane marking detection apparatus or the lane detection apparatus according to the present invention, the evaluation means is an inner product of a set of luminance gradient vectors consisting of two point-symmetric pixels existing within a certain distance around an arbitrary pixel in the image. The evaluation value may be obtained by calculating.

  The evaluation means sequentially extracts a set of luminance gradient vectors composed of two pixels that are point-symmetric and exist within a certain distance from an arbitrary pixel in the image from the calculated luminance gradient vector of each pixel. An evaluation value is obtained by calculating the inner product of the brightness gradient vectors of the extracted set. Two corresponding pixels on the two contour lines of the lane marking are in a point-symmetrical position around the pixel on the center line of the lane marking. Also, the width of the lane marking is determined, and the width of the lane marking in the image is within a predetermined range. Therefore, in order to search for a pair of two corresponding pixels on the two contour lines of the partition line, it is not necessary to evaluate all the combinations of all the pixels in the image, and within a certain distance centering on an arbitrary pixel. What is necessary is just to evaluate about the combination of 2 pixels which become point symmetry existing in. The fixed distance is a distance set in consideration of the range of the width of the lane marking in the image. Thus, in this lane marking detection device or lane detection device, the processing load can be reduced by narrowing down the combinations to be evaluated.

A lane detection device according to the present invention is a lane detection device that detects a lane composed of two parallel lane markings on a road, and includes an imaging unit that images a road, and each pixel of an image captured by the imaging unit. Luminance gradient vector calculation means for calculating a luminance gradient vector, lane shape candidate generation means for generating lane shape candidates based on a shape model or past detection results, lane shape candidates and luminance gradients generated by the lane shape candidate generation means A lane shape estimation unit that collates the luminance gradient vector calculated by the vector calculation unit and estimates a lane shape based on the comparison result, and the luminance gradient vector calculation unit includes a horizontal gradient of the image for each pixel. A first derivative value of luminance and a first derivative value of luminance in the vertical direction of the image are calculated, and brightness is obtained from the first derivative value of luminance in the horizontal direction and the first derivative value of luminance in the vertical direction. And calculates the gradient vector.

  In this lane detection device, the road is imaged by the imaging unit, and the luminance gradient vector of each pixel of the captured image is calculated by the luminance gradient vector calculating unit. In the lane detection device, lane shape candidates are generated by the lane shape candidate generation means from the past detection results of the shape model or lane markings and lanes. The shape model is a model for representing various shapes of lane markings and lanes, and is a shape model that can represent straight lines and various curves. Lane shape candidates are various candidates representing the shape of the lane. In the lane detection device, the lane shape estimation means collates the generated lane shape candidate with the calculated luminance gradient vector, and estimates the lane shape based on the collation result. The closer the shape of a lane shape candidate is to the lane (actual lane) in the captured image, the difference between the direction of the luminance gradient of each pixel in the lane shape candidate and the direction of the luminance gradient vector of the pixel corresponding to each pixel And the degree of coincidence between the lane shape candidate and the brightness gradient vector increases. Therefore, the lane (division line) can be estimated from the lane shape candidate having the highest degree of coincidence. As described above, in this lane detection device, a lane in an arbitrary direction can be detected by using a luminance gradient vector composed of luminance gradients in a plurality of directions and collating the luminance gradient vector with lane shape candidates. . As a result, not only a straight road but also lanes of various curved roads can be detected accurately, and the lane can be detected with high accuracy. In addition, it can detect the lane in the image by calculating the brightness gradient vector for each pixel and comparing the lane shape candidate with the brightness gradient vector, so the processing load is relatively low and the lane can be detected efficiently. it can. Furthermore, the shape and width of the lane marking and the lane width can be detected simultaneously.

  The present invention uses a brightness gradient vector and uses the fact that the direction of the brightness gradient vector of each pixel in the two contour lines of the partition line is in the opposite direction, thereby allowing a partition line (and thus a lane) in any direction. Can be detected with high accuracy.

  Embodiments of a lane marking detection device and a lane detection device according to the present invention will be described below with reference to the drawings.

  In the present embodiment, the present invention is applied to the lane marker detection device according to the first embodiment and the lane detection devices according to the second and third embodiments. The lane marker detection device according to the first embodiment detects a lane marker (partition line) by calculating a luminance gradient vector of each pixel and evaluating each set of luminance gradient vectors. The lane detection device according to the second embodiment detects a lane marker by a method similar to that of the first embodiment, and estimates the lane using the detected lane marker. The lane detecting apparatus according to the third embodiment calculates a luminance gradient vector of each pixel, generates a lane shape candidate from the shape model, and detects the lane by collating the luminance gradient vector with the lane shape candidate.

  With reference to FIG.1 and FIG.2, the lane marker detection apparatus 1 which concerns on 1st Embodiment is demonstrated. FIG. 1 is a configuration diagram of a lane marker detection device according to the first embodiment. FIG. 2 is an example of a primary differential filter used in the luminance gradient vector calculation unit of the ECU of FIG. 1, where (a) is a horizontal filter and (b) is a vertical filter.

  The lane marker detection device 1 detects a lane marker (white line constituting the lane) of the lane in which the host vehicle is traveling, and provides the detected lane marker information to other devices such as a lane keeping device and an automatic steering device. To do. In the lane marker detection device 1, in order to improve the detection accuracy, the lane marker is detected by using the fact that the direction of the luminance gradient vector of the corresponding points in the two contour lines of the lane marker is opposite. For this purpose, the lane marker detection device 1 includes a camera 2 and an ECU [Electronic Control Unit] 3, and a luminance gradient vector calculation unit 3 a and a lane marker detection unit 3 b are configured in the ECU 3.

  In the first embodiment, the camera 2 corresponds to the imaging unit described in the claims, the luminance gradient vector calculation unit 3a corresponds to the luminance gradient vector calculation unit described in the claims, and the lane marker detection unit 3b corresponds to the evaluation means and the lane marking detection means described in the claims.

  The camera 2 is a camera including an image sensor such as a CCD [Charge coupled device] or a CMOS [Complementary Metal Oxide Semiconductor]. The camera 2 is attached to the front of the vehicle, and is attached to the vehicle at a pitch angle or attachment height such that the road ahead of the vehicle falls within the range of 1/2 to 2/3 below the captured image. The camera 2 captures the front of the vehicle and transmits the captured image data to the ECU 3 as an image signal. FIG. 3 shows an example of the captured image. Since the ECU 3 can perform processing if there is at least luminance information, the camera 2 may be a color camera or a monochrome camera. Although the pitch angle of the camera 2 with respect to the vehicle is constant from the time of attachment, the pitch angle θ of the camera 2 with respect to the road plane varies according to the variation in the pitch direction of the vehicle.

  The ECU 3 is an ECU for image processing, and includes a CPU [Central Processing Unit], a ROM [Read Only Memory], a RAM [Random Access Memory], and the like. In the ECU 3, when the lane marker detection device 1 is activated, a dedicated application program stored in the ROM is loaded into the RAM, and each processing unit 3a, 3b is executed by the CPU executing each process described in the program. Is configured. The ECU 3 takes captured image data from the camera 2 at regular intervals. Then, the ECU 3 calculates a luminance gradient vector for each pixel of the captured image. Further, the ECU 3 detects a lane marker (a pixel row constituting the center line) based on the luminance gradient vector of each pixel.

The luminance gradient vector calculation unit 3a calculates a luminance gradient vector from the luminance of each pixel of the captured image. The luminance gradient vector is a vector in which luminance gradients (edges) in a plurality of directions different for each pixel of the captured image are obtained and the luminance gradients in the plurality of directions are synthesized. The luminance gradient vector calculation unit 3a uses a horizontal first-order differential filter shown in FIG. 2A, and uses the first-order differential value (in the horizontal direction for the pixels _{Pu and v} in the image (1) (1). Luminance gradient) dI ₁ (u, v) is calculated. Further, the luminance gradient vector calculation unit 3a uses the vertical first-order differential filter shown in FIG. 2B, and uses the first-order differential filter of the vertical luminance for the pixels _{Pu and v} in the image according to the equation (2). The value dI ₂ (u, v) is calculated.

  I in the formulas (1) and (2) is a luminance value of the pixel. With the horizontal first-order differential filter (by equation (1)), it is possible to extract a portion (edge) in which the luminance changes in the horizontal direction and extract a contour in the vertical direction. In addition, by the vertical first-order differential filter (by the equation (2)), the portion (edge) where the luminance changes in the vertical direction can be extracted, and the contour in the horizontal direction can be extracted.

  FIG. 4 shows the output result of the horizontal first-order differential filter for the captured image of FIG. As can be seen from FIG. 4, the horizontal first-order differential filter can extract most of the outline of the lane marker on the left side of the lane, but sufficiently extracts the outline of the lane marker on the right side and the lane marker of the adjacent lane. I can't. FIG. 5 shows the output result of the vertical first-order differential filter for the captured image of FIG. As can be seen from FIG. 5, the vertical first-order differential filter can extract most of the contours of the lane marker on the right side of the lane and the lane marker of the adjacent lane, but can sufficiently extract the lane marker on the left side. It is not possible (for example, there is a defect in a contour portion composed only of vertical edges). In this way, a defect and sufficient edge strength cannot be obtained in only one direction. However, by extracting an edge in two directions, a sufficient edge strength can be obtained in the other direction even in a portion where a defect or a sufficient edge strength cannot be obtained in one direction. Therefore, the contour of the lane marker in various directions can be extracted with high accuracy by using the luminance gradient vector composed of the edge components in two directions.

In the luminance gradient vector calculation unit 3a, a luminance gradient vector is set from the outputs of the first-order differential filters in the two directions. When the output of the horizontal primary differential filter is dI ₁ (u, v) and the output of the vertical primary differential filter is dI ₂ (u, v), the luminance gradient vector dI (u, v) = (dI ₁ (u, v), dI ₂ (u, v)) ^T. Note that u indicates the horizontal position of the pixel on the image, and v indicates the vertical position of the pixel on the image.

  Note that the luminance gradient vector may be calculated for all pixels of the captured image. However, in consideration of the pitch angle of the camera 2 and the like, if the range in which the road is captured in the captured image can be limited, the pixels within that range A luminance gradient vector may be calculated only for. Moreover, since the output result of each primary differential filter may change greatly locally due to the influence of noise or the like, the influence of noise may be reduced by using an average filter.

  The lane marker has two parallel contour lines (boundary between the road and the lane marker). Since the reflectance of the lane marker (white line) is larger than the reflectance of the road, the brightness changes from dark to bright (road to lane marker) on one contour, and bright to dark (lane marker to road) on the other contour. To do. Therefore, the luminance gradient vector of an arbitrary pixel on one contour line and the luminance gradient vector of the corresponding pixel on the other contour line are in the opposite direction in the case of first-order differentiation, and the intensity is approximately the same. It becomes. By using this characteristic, a lane marker is detected from a luminance gradient vector.

  The lane marker detection unit 3b evaluates the likelihood of a lane marker from a luminance gradient vector in which two pixels satisfying a predetermined relationship are set as one set, and detects the center line of the lane marker from the evaluation result. Since the lane marker has a certain width, the two contour lines of the lane marker (that is, the shape of the lane marker) are specified by specifying the center line of the lane marker.

The lane marker detection unit 3b sequentially extracts arbitrary pixels _{Pu, v} on the image. As the pixels _{Pu, v} , pixels within the range in which the luminance gradient vector is calculated are extracted. Then, the lane-mark detection unit 3b, extracted pixels _{P u, v} around the pixel _{P u, v} from the constant distance d (dmin or more and dmax less) of 2 points which are symmetrical points existing in a pixel (u ', V') and (u ", v") are extracted. In the case of the lane marker, since each point corresponding to the right and left contour lines is point-symmetric within a certain range from a certain point on the center line of the lane marker, it exists within a certain distance d from the pixel P _{u, v} and the pixel P _{u. , V} are extracted and two pixels having a point-symmetrical relationship are extracted. dmin is a value that is 1/2 of the minimum value of the width of the lane marker in the image, and dmax is a value that is 1/2 of the maximum value of the width of the lane marker in the image. Further, in the lane marker detection unit 3b, the luminance gradient vectors dI (u ′, v ′) and dI (2) of the two extracted pixels (u ′, v ′) and (u ″, v ″) are obtained by Expression (3). u ″, v ″) is used to calculate an evaluation value LB (u, v) indicating the likelihood of a lane marker.

In Expression (3), dI ₁ (u ′, v ′) is the output of the horizontal first-order differential filter of the pixel (u ′, v ′), and dI ₂ (u ′, v ′) is the pixel (u ′). , V ′) is the output of the first-order differential filter in the vertical direction, dI ₁ (u ″, v ″) is the output of the first-order differential filter in the horizontal direction of the pixel (u ″, v ″), and dI ₂ (u ", V") is the output of the vertical first-order differential filter of the pixel (u ", v"), and f () is a predetermined evaluation function. In the case of the luminance gradient vector by the primary differential filter, the direction of the luminance gradient vector of each pixel on the two contour lines of the lane marker is in the reverse direction, so the inner product is used as the evaluation function. In the lane marker detection unit 3b, the evaluation value LB (u, v) is calculated by Expression (4).

  If the direction of the luminance gradient vector of each pixel on the two contour lines of the lane marker is the same direction by using the luminance gradient vector obtained by a filter such as a second derivative, the sum represented by equation (5) or The product shown in equation (6) is used.

  These evaluation values LB (u, v) increase as the direction of each luminance gradient vector of the two pixels (u ′, v ′), (u ″, v ″) is reversed. The evaluation value LB (u, v) increases as the intensity of each luminance gradient vector of the two pixels (u ′, v ′) and (u ″, v ″) increases.

In the lane marker detection unit 3b, for each pixel P _{u, v} in the image, all of the two point-symmetrical pixels (u ′, v ′), (u ″, v ″) existing within a certain distance d. The evaluation value LB (u, v) is calculated for the combination, and the maximum evaluation value LB (u, v) is selected from the evaluation values LB (u, v) of all the combinations. Then, the lane marker detection unit 3b determines whether or not the maximum evaluation value LB (u, v) is larger than the threshold value Th1 for each pixel _{Pu, v} in the image. The threshold value Th1 is sufficient when the two pixels (u ′, v ′) and (u ″, v ″) are the corresponding pixels on the two contour lines of the lane marker as the magnitude of the evaluation value LB (u, v). This is a large value that can be determined. The threshold value Th1 may be set in consideration of the reflectance of the road and the reflectance of the white line, or may be determined from the experimental results obtained by actually obtaining evaluation values on various roads.

The lane marker detecting section 3b, the threshold Th1 is greater than the evaluation value LB (u, v) is a pixel _{P u, v} and the center line of pixels of the lane marker, and outputs the pixel _{P u, v} columns as the shape of the lane marker. Note that two pixel columns forming two contour lines of the lane marker are calculated from the pixels Pu _{and v} forming the center line of the lane marker, and the two pixel columns are output as the shape of the lane marker. Good.

  The operation of the lane marker detection device 1 will be described with reference to FIG. In particular, the process of the luminance gradient vector calculation unit 3a in the ECU 3 will be described along the flowchart of FIG. 6, and the process of the lane marker detection unit 3b will be described along the flowchart of FIG. FIG. 6 is a flowchart showing a process flow in the luminance gradient vector calculation unit of the ECU of FIG. FIG. 7 is a flowchart showing the flow of processing in the lane marker detection unit of the ECU of FIG.

  The camera 2 images the front of the vehicle and transmits an image signal to the ECU 3. The ECU 3 receives an image signal every predetermined time (for example, 1/30 second) and temporarily stores data of the captured image. Then, the ECU 3 starts processing for the captured image.

First, the ECU 3 extracts the pixels _{Pu, v} from the image (S10). Then, the ECU 3 calculates a luminance gradient dI ₁ (u, v) in the horizontal direction of the pixels P _{u, v} using a horizontal first-order differential filter (S11). Further, the ECU 3 calculates the luminance gradient dI ₂ (u, v) in the vertical direction of the pixels _{Pu, v} using a vertical first-order differential filter (S12). Then, the ECU 3 sets the luminance gradient vector dI (u, v) of the pixels _{Pu, v} from the horizontal luminance gradient dI ₁ (u, v) and the vertical luminance gradient dI ₂ (u, v) ( S13).

Then, the ECU 3 determines whether or not the luminance gradient vector calculation process has been completed for all the pixels _{Pu and v} (S14). If it is determined in S14 that the process has not been completed for all the pixels P _{u, v} , the process returns to S10 and the process for the next pixel P _{u, v} is performed. On the other hand, when it is determined in S14 that the processing for all the pixels _{Pu, v} is finished, the luminance gradient vector calculation processing is finished.

When the luminance gradient vector is calculated, the ECU 3 extracts the pixels _{Pu, v} from the image (S20). Then, in the ECU 3, a set of pixels (u ′, v ′), (u ″, v ″) that are point-symmetric and exist at a fixed distance d (range from dmin to dmax) with the pixel P _{u, v} as the center. ) Are sequentially extracted and evaluated from the respective luminance gradient vectors dI (u ′, v ′) and dI (u ″, v ″) of the set of pixels (u ′, v ′) and (u ″, v ″). A value LB (u, v) is calculated (S21). This evaluation value LB (u, v) is calculated for all combinations that are point-symmetric and exist within a certain distance d. Then, the ECU 3 selects the maximum evaluation value LB (u, v) from the evaluation values LB (u, v) of all the combinations (S21). Further, the ECU 3 determines whether or not the maximum evaluation value LB (u, v) is larger than the threshold value Th1 (S22). When it is determined in S22 that the threshold value Th1 is greater (that is, the luminance gradient vectors dI (u ′, v ′) and dI (u ″, v ″) having the maximum evaluation value LB (u, v) have the opposite directions. The ECU 3 sets the pixels Pu _{and v} as pixels on the center line of the lane marker (S23). On the other hand, when it determines with it being below threshold value Th1 in S22, it transfers to the process of S24.

Then, the ECU 3 determines whether or not the lane marker detection process has been completed for all the pixels Pu _{and v} (S24). If it is determined in S24 that the processing has not been completed for all the pixels P _{u, v} , the process returns to S20 and the next pixel P _{u, v} is processed. On the other hand, if it is determined in S24 that the processing for all the pixels _{Pu, v} has been completed, the lane marker detection processing is terminated. Here, the pixel _{Pu, v} column that forms the center line of the lane marker in the captured image at the current time t is extracted. The ECU 3 outputs the data of the pixel _{Pu, v} column.

The lane marker detection device 1 repeats the above operation at regular time intervals (each time a captured image is acquired) to obtain and output the pixels _{Pu and v} that form the center line of the lane marker.

  According to this lane marker detection device 1, a luminance gradient vector composed of luminance gradients in two directions is used, and this is used when the direction of the luminance gradient vector at each corresponding point on the left and right contour lines of the lane marker is reversed. Thus, a lane marker in an arbitrary direction can be detected with high accuracy. In addition, a lane marker in an image can be detected by calculating a luminance gradient vector for each pixel and evaluating the luminance gradient vector between two pixels having a predetermined positional relationship, so that the processing load is relatively small and efficiency is improved. The lane marker can be detected.

  Furthermore, the lane marker detection device 1 can narrow down the combinations to be evaluated by evaluating only two point-symmetric pixels existing within a certain distance from an arbitrary pixel, thereby further reducing the processing load. it can.

  A lane detection device 11 according to the second embodiment will be described with reference to FIG. FIG. 8 is a configuration diagram of a lane detector according to the second embodiment. In the lane detection device 11, the same components as those in the lane marker detection device 1 according to the first embodiment are denoted by the same reference numerals, and the description thereof is omitted.

  The lane detection device 11 detects the lane in which the host vehicle is traveling, and provides information on the detected lane to other devices such as a lane keeping device and an automatic steering device. The lane detection device 11 detects a lane marker by the same process as the lane marker detection device 1, and estimates the lane from the lane marker. For this purpose, the lane detection device 11 includes a camera 2 and an ECU 13, and a luminance gradient vector calculation unit 3a, a lane marker detection unit 3b, and a lane shape estimation unit 13c are configured in the ECU 13.

  In the second embodiment, the camera 2 corresponds to the imaging unit described in the claims, the luminance gradient vector calculation unit 3a corresponds to the luminance gradient vector calculation unit described in the claims, and the lane marker detection unit 3b corresponds to the evaluation means and lane marking detection means described in the claims, and the lane shape estimation unit 13c corresponds to the lane shape estimation means described in the claims.

  The ECU 13 is an ECU for image processing, and includes a CPU, a ROM, a RAM, and the like. In the ECU 13, when the lane detecting device 11 is activated, a dedicated application program stored in the ROM is loaded into the RAM, and each processing unit 3a, 3b is executed by the CPU executing each process described in the program. , 13c are configured. The ECU 13 takes captured image data from the camera 2 at regular intervals. And ECU13 detects a lane marker by the process similar to ECU3 which concerns on 1st Embodiment. Further, the ECU 13 estimates the lane using the detected lane marker.

In the lane shape estimation unit 13c, the shape and position of the lane are estimated using the shape model using the pixels _{Pu, v} columns forming the center line of the lane marker. Here, the shape model of Expression (7) disclosed in the literature [B. Southall CJTaylor, “Stochastic roadshape estimation”, ICCV2001] is used as the shape model.

The shape model shown in Expression (7) is a model obtained by projecting a three-dimensional shape model onto a plane, where the x coordinate is the traveling direction of the vehicle and the y coordinate is the direction orthogonal to the traveling direction of the vehicle. In equation (7), y ₀ is a parameter corresponding to the offset from the center line of the lane, ε is a parameter corresponding to the direction (inclination) of the lane marker with respect to the imaging direction of the camera 2, and C ₀ is the curvature of the lane marker. C ₁ is a parameter corresponding to a change in the curvature of the lane marker. y ₀ is the left offset and right offset is represented by a positive value and a negative value with respect to the center line of the lane, the difference between the left offset and right offset becomes the width W of the lane.

The shape and position of the lane can be defined by the parameters y ₀ , ε (tan (ε)), C ₀ , C ₁ and the width W of the lane (the distance between a pair of lane markers). Therefore, a model is considered in which the pitch angle θ of the camera 2 necessary for conversion between the coordinate system of the camera image and the coordinate system of the shape model is added to these parameters. In the lane shape estimation unit 13c, each parameter y ₀ , tan (ε), C ₀ , C is applied by applying the shape model of Expression (7) to the pixel _{Pu, v} columns forming the center line of the lane marker. ₁ and W are estimated. In the lane shape estimation unit 13c, as an estimation method using a Kalman filter to estimate the estimation parameters phi _t shown in equation (8).

The subscript t indicates the current time t, and the subscript t-1 indicates the time before the predetermined time Δt of the current time t. First, the lane shape estimation unit 13c calculates the estimated value φ _t ′ of the estimated parameter at the current time t using Equation (9) using the estimated parameter φ _t−1 estimated at the previous time t−1.

The matrix A in Equation (9) is expressed by Equation (10), and ω _t is disturbance noise. Δx in Expression (10) is expressed by Expression (11). V in Formula (11) is a vehicle speed, and is detected by a vehicle speed sensor (not shown). In Expression (11), Δt is an interval at which captured images are input, and is a constant time interval (for example, 1/30 seconds).

Next, the lane shape estimation unit 13c uses the a posteriori error covariance matrix M _t−1 calculated at the previous time t−1 to calculate the predicted value M of the a posteriori error covariance matrix at the current time t according to the equation (12). _t ′ is calculated. In the lane shape estimation unit 13c, the Kalman gain K at the current time t is calculated by Equation (13) using the predicted value M _t ′ of the posterior error covariance matrix. In the lane shape estimation unit 13c, the estimated parameter φ at the current time t is calculated by Expression (14) using the pixel P _{u, v} column of the detected center line of the lane marker, the estimated value φ _t ′ of the estimated parameter, and the Kalman gain K. _t is calculated. In addition, the lane shape estimation unit 13c calculates the posterior error covariance matrix M _t at the current time t using Equation (15) using the predicted value M _t ′ of the posterior error covariance matrix and the Kalman gain K. This posterior error covariance matrix M _t is used at the next time t−1.

The matrix Q in Equation (12) is a system error covariance matrix. H _t in Equation (13) is expressed by Equation (16), and the matrix R is an observation error covariance matrix. H (φ _t ′) in Expression (14) is an observation equation and is expressed by Expression (17).

X _{1, t} ,..., X _{n, t} in equation (17) form the center line of the detected n lane markers based on the initial mounting height l of the camera 2 and the pitch angle θ of the camera 2. The coordinates P _{x, y} = (x, y) ^T in which the pixels P _{u, v} to be projected on the road plane are projected (X coordinates) in the traveling direction of the vehicle. In addition, y _t (y _{1, t} ,..., Y _{n, t} ) in the equation (14) is the vehicle progression among the coordinates P _{x, y} = (x, y) ^T projected onto the road plane. It is the coordinate (y coordinate) of the orthogonal direction of a direction. The position from the position of the camera 2 (position on the captured image) to the road plane can be roughly determined by the initial mounting height l (known) of the camera 2 and the pitch angle θ of the camera 2 with respect to the road plane. The pitch angle θ may be detected by an angular velocity sensor (not shown) or may be estimated by a predetermined method.

By estimating parameters phi _t at the current time t is calculated, lane markers of the shape model (equation (7)) and will be the lane width W is determined, obtained by projecting the three-dimensional on (road plane lane ) Is estimated. The lane shape estimation unit 13c outputs the shape model (lane marker shape) and the lane width W in which these parameters are determined as lane shape and position information.

  The operation of the lane detection device 11 will be described with reference to FIG. In particular, the processing of the lane shape estimation unit 13c in the ECU 13 will be described along the flowchart of FIG. FIG. 9 is a flowchart showing a flow of processing in the lane shape estimation unit of the ECU of FIG. Since the operation until the lane marker is detected is the same as that in the first embodiment, the description thereof will be omitted, and the operation after the detection of the lane marker will be described.

When detecting the pixel _{Pu, v} row that constitutes the center line of the lane marker, the ECU 13 estimates at the current time t using the estimated parameter φ _t-1 obtained at the previous time t-1 according to the equation (9). The parameter predicted value φ _t ′ is calculated (S30). Next, the ECU 13 obtains the predicted value M _t ′ of the posterior error covariance matrix at the current time t using the posterior error covariance matrix M _t−1 obtained at the previous time t−1 according to the equation (12). Calculate (S31). Next, the ECU 13 calculates the Kalman gain K at the current time t by using the predicted value M _t ′ of the posterior error covariance matrix according to the equation (13) (S32). Next, in the ECU 13, based on the pitch angle θ of the camera 2 and the initial mounting position l of the camera 2, the coordinates P _x, which are obtained by projecting the pixels P _{u, v} constituting the center line of the detected lane marker on the road plane _. Each is converted to _y (S33). Then, the ECU 13 calculates the estimated parameter φ _t at the current time t by using the converted coordinates P _{x, y} , the estimated value φ _t ′ of the estimated parameter, and the Kalman gain K according to the equation (14) (S33). ). Further, the ECU 13 calculates the posterior error covariance matrix M _t at the current time t by using the predicted value M _t ′ of the posterior error covariance matrix and the Kalman gain K according to the equation (15) (S34). Further, the ECU 13, the parameters of the calculated estimated parameter phi _t to set the shape and position of the lane, and outputs the information of the shape and position of the lane (S35).

  The lane detection device 11 repeats the above operation at regular time intervals (each time a captured image is acquired) to obtain and output information on the shape and position of the lane.

  According to this lane detection device 11, in addition to the same effects as the lane keep detection device 1 according to the first embodiment with respect to the detection of the lane marker, not only the straight road from the lane marker detected with high accuracy, but also various The width and shape of the lane of the curved road can be accurately detected, and the lane can be detected with high accuracy.

  Further, the lane detection device 11 uses the shape model and the Kalman filter to estimate the shape of the lane (lane marker) by looking at the entire image. Can be suppressed. In order to further improve the detection accuracy of the lane marker, the lane marker may be recalculated using the shape of the lane marker estimated from the entire image.

  A lane detection device 21 according to a third embodiment will be described with reference to FIG. FIG. 10 is a configuration diagram of a lane detector according to the third embodiment. In the lane detection device 21, the same components as those in the lane marker detection device 1 according to the first embodiment are denoted by the same reference numerals, and the description thereof is omitted.

  The lane detection device 21 detects the lane in which the host vehicle is traveling, and provides information on the detected lane to other devices such as a lane keeping device and an automatic steering device. The lane detection device 21 calculates a luminance gradient vector by the same process as the lane marker detection device 1, generates a lane shape candidate, and estimates the lane using the luminance gradient vector and the lane shape candidate. For this purpose, the lane detection device 21 includes a camera 2 and an ECU 23, and a luminance gradient vector calculation unit 3a, a lane shape candidate generation unit 23b, and a lane shape estimation unit 23c are configured in the ECU 23.

  In the third embodiment, the camera 2 corresponds to the imaging unit described in the claims, the luminance gradient vector calculation unit 3a corresponds to the luminance gradient vector calculation unit described in the claims, and the lane The shape candidate generation unit 23b corresponds to the lane shape candidate generation unit described in the claims, and the lane shape estimation unit 23c corresponds to the lane shape estimation unit described in the claims.

  The ECU 23 is an ECU for image processing, and includes a CPU, a ROM, a RAM, and the like. In the ECU 23, when the lane detection device 21 is activated, a dedicated application program stored in the ROM is loaded into the RAM, and each processing unit 3a, 23b is executed by the CPU executing each process described in the program. , 23c are configured. The ECU 23 takes captured image data from the camera 2 at regular time intervals. Then, the ECU 23 calculates a luminance gradient vector by the same process as that of the ECU 3 according to the first embodiment. In addition, the ECU 23 generates a number of lane shape candidates from the shape model. Further, the ECU 23 collates the brightness gradient vector with each lane shape candidate, and estimates the lane from the collation result.

The lane shape candidate generation unit 23b generates many lane candidates using the shape model. Here, the shape model shown in Formula (7) used in the second embodiment is used. Further, a model in which the lane mark width lw is added to the shape model parameters y ₀ , tan (ε), C ₀ , C ₁ , the lane width W, and the camera pitch angle θ will be considered. By estimating the width lw of the lane marker at the same time, an appropriate lane shape candidate can be generated even when the width of the lane marker is different. Any N lane shape candidates are generated.

Since there are white lines of various shapes such as pedestrian crossings in addition to the white lines that serve as lane markers on the road, it is necessary to extract only the lane markers that constitute the lane. Therefore, using a shape model, only shapes that can be regarded as lane markers are generated as lane shape candidates. As for the shape of the lane marker (lane), a curve from a straight line (curve radius infinite) to a curve radius of about several tens, a lane width, a lane mark width, and the like are determined according to the road structure. Also, the pitch angle θ of the camera 2 can be estimated from the pitch angle of the camera with respect to the vehicle and the limit of the pitch angle of the vehicle. Therefore, the ranges of the parameters y ₀ , ε, C ₀ , C ₁ , W, θ, and lw can be defined, and a lane marker is obtained by configuring the model with combinations of the values of the parameters within the ranges. Various possible lane shape candidates can be generated. Therefore, for each of the parameters y ₀ , ε, C ₀ , C ₁ , W, θ, and lw, each density distribution Pt indicating a possible range and a possible value distribution is set in advance. The distribution is not particularly limited such as uniform distribution and binomial distribution. Therefore, the number N of lane shape candidates is the number of combinations of values that each parameter y ₀ , ε, C ₀ , C ₁ , W, θ, and lw can take from each density distribution.

In lane shape candidate generating unit 23b, when generating the n-th lane shape candidate, the parameters _{y 0, ε, C 0,} C 1, W, θ, based on the density distribution of Pt lw, n-th parameters The values of y ₀ , ε, C ₀ , C ₁ , W, θ, and lw are determined. Then, the lane shape candidate generation unit 23b uses the determined values of the parameters y ₀ , ε, C ₀ , C ₁ , W, θ, and lw and the shape model of Expression (7) to determine the camera of the nth lane shape candidate. The lane shape on the image (the contour line of each lane marker) is calculated. At this time, when projecting the shape model of Expression (7) onto the two-dimensional image, conversion is performed using the pitch angle θ of the camera 2 and the initial mounting height l of the camera 2. In the lane shape candidate generation unit 23b, the above processing is performed for N lane shape candidates. FIG. 11 shows an example in which the generated lane shape candidate LC is projected on the image and does not match the actual lane shape.

  The lane shape estimation unit 23c collates N lane shape candidates with the luminance gradient vector, and estimates the lane shape and position from the collation result. The closer the shape of a lane shape candidate is to the actual lane in the captured image, the difference between the direction of the luminance gradient of each pixel in the lane shape candidate projected on that image and the direction of the luminance gradient vector corresponding to that pixel. Becomes smaller, and the degree of coincidence between the lane shape candidate and the luminance gradient vector increases. In particular, when the lane shape candidate projected on the image overlaps the lane in the captured image, the direction of the luminance gradient vector of the corresponding pixel of the two contour lines of the lane marker is reversed, and the lane shape candidate has the same direction as this. The direction of the brightness gradient corresponding to the two contour lines of each lane marker is also opposite, and the width of the lane marker and the lane width are the same width, so the degree of coincidence becomes very large. Therefore, the lane (lane marker) can be estimated from the lane shape candidate having the highest degree of coincidence.

In the lane shape estimation unit 23c, for the nth lane shape candidate, a unit vector n in the luminance gradient direction for each pixel (i, j) constituting each contour line of the lane shape candidate (candidate projected on the image). _c (i, j) is calculated. This n _c (i, j) is a unit vector in the luminance gradient direction of the contour line of the lane shape candidate at each coordinate point (i, j) projected on the image, as in the first embodiment. A unit vector of luminance gradients in two directions, the horizontal direction and the vertical direction. In the lane shape estimation unit 23c, the unit vector n _c (i) of the luminance gradient direction of each coordinate point is obtained for all coordinate points (i, j) constituting each contour line of the lane shape candidate by Expression (18). , J) and an evaluation value V (n) representing the degree of coincidence between the luminance gradient vector dI (i, j) of the corresponding pixel.

  The function Cor (i, j) in the equation (18) is a function represented by the equation (19). The function of Expression (19) is a function that increases the degree of matching if the vectors are in the same direction (and thus the contour line), and decreases the degree of matching if the vectors are in different directions.

  The lane shape estimation unit 23c performs the above process on N lane shape candidates. Then, the lane shape estimation unit 23c extracts the lane shape candidate having the maximum evaluation value V (n) from the evaluation values V (n) of the N lane shape candidates, and extracts each lane shape candidate from each parameter of the lane shape candidates. The resulting shape model (lane marker shape), lane width W, and lane marker width lw are output as lane shape and position information.

  The operation of the lane detector 21 will be described with reference to FIG. In particular, the processing of the lane shape candidate generation unit 23b in the ECU 23 will be described along the flowchart of FIG. 12, and the processing of the lane shape estimation unit 23c will be described along the flowchart of FIG. FIG. 12 is a flowchart showing a process flow in the lane shape candidate generation unit of the ECU of FIG. FIG. 13 is a flowchart showing the flow of processing in the lane shape estimation unit of the ECU of FIG. Since the operation until the luminance gradient vector is calculated is the same as that in the first embodiment, the description thereof is omitted, and the operation after the luminance gradient vector is detected will be described.

When the brightness gradient vector is calculated, the ECU 23 extracts a lane shape candidate n to be generated (S40). Then, the ECU 23 determines the values of the parameters y ₀ , ε, C ₀ , C ₁ , W, θ, and lw for the lane shape candidate n based on the density distribution Pt of each parameter (S41). Further, the ECU 23 calculates the lane shape on the image of the lane shape candidate n based on the determined parameters y ₀ , ε, C ₀ , C ₁ , W, θ, lw and the shape model (Equation (7)). (S42).

  Then, the ECU 23 determines whether or not the generation processing for all (N) lane shape candidates has been completed (S43). If it is determined in S43 that the process has not been completed for all lane shape candidates, the process returns to S40, and the generation process for the next lane shape candidate is performed. On the other hand, when it determines with the process about all the lane shape candidates having been complete | finished in S43, a production | generation process is complete | finished.

When the lane shape candidate is generated, the ECU 23 extracts a lane shape candidate n from the generated lane shape candidates (S50). Then, the ECU 23 calculates a unit vector n _c (i, j) in the luminance gradient direction for each coordinate (i, j) constituting the contour line of the lane shape candidate n (S51). Further, the ECU 23 corresponds to the unit vector n _c (i, j) in the luminance gradient direction and all the coordinates for all the coordinates (i, j) constituting the contour line of the lane shape candidate n by Expression (18). An evaluation value V (n) with the luminance gradient vector dI (i, j) of the pixel is calculated (S52).

  Then, the ECU 23 determines whether or not the collation processing for all (N) lane shape candidates has been completed (S53). If it is determined in S53 that the processing has not been completed for all lane shape candidates, the process returns to S50, and the next lane shape candidate is collated. On the other hand, when it is determined in S53 that the processing for all lane shape candidates has been completed, the ECU 23 determines the actual lane by using the lane shape candidate having the highest evaluation value V (n) among all lane shape candidates. The shape and position are set, and information on the shape and position of the lane is output (S54).

  The lane detection device 21 repeats the above operation at regular time intervals (each time a captured image is acquired) to obtain and output information on the shape and position of the lane.

  According to this lane detection device 21, a lane in any direction is detected with high accuracy by using a luminance gradient vector composed of luminance gradients in a plurality of directions and collating the luminance gradient vector with a number of lane shape candidates. can do. In addition, it can detect the lane in the image by calculating the brightness gradient vector for each pixel and comparing the lane shape candidate with the brightness gradient vector, so the processing load is relatively low and the lane can be detected efficiently. it can. Furthermore, the shape, width and lane width of the lane marker can be detected simultaneously.

  As mentioned above, although embodiment which concerns on this invention was described, this invention is implemented in various forms, without being limited to the said embodiment.

  For example, in the present embodiment, the first-order differential filter in the horizontal direction and the vertical direction is used to obtain the luminance gradient vector. However, a filter in another direction such as a 45 ° direction may be used, and the second-order differential may be used. Other dimension differential filters may be used.

Further, in the present embodiment, the configuration is such that the evaluation of the luminance gradient vector of two point-symmetric pixels within a certain distance of any pixel P _{u, v} is performed, but on any line passing through the pixels P _{u, v} (for example, The luminance gradient vector may be evaluated by a combination of other methods such as a combination of pixels existing on the (horizontal line).

  Further, in the present embodiment, the configuration is such that the center line of the lane keep is detected, but the configuration may be such that two contour lines of the lane keep are detected.

  In the second embodiment, the lane shape is estimated using the shape model. However, the lane shape may be estimated by another method. Further, although the configuration is such that the parameters of the shape model and the lane width are estimated using the Kalman filter, other filters such as a particle filter may be used.

  In the third embodiment, the lane shape candidate is generated using the shape model. However, if there is information on the lane marker or lane detected in the past, the lane shape candidate is detected using the detection result. It is good also as a structure which produces | generates. In particular, the lane shape candidate at the next time can be determined probabilistically based on the lane detection result at the previous time, like a particle filter. In this case, the number of candidates is reduced, the processing load is reduced, and it is easy to detect lane shape candidates that match the luminance gradient vector even with a small number of candidates.

It is a block diagram of the lane marker detection apparatus which concerns on 1st Embodiment. It is an example of the primary differential filter used in the brightness | luminance gradient vector calculation part of ECU of FIG. 1, (a) is a filter of a horizontal direction, (b) is a filter of a vertical direction. It is an example of the captured image of the camera of FIG. It is an output result of the horizontal direction primary differential filter with respect to the captured image of FIG. FIG. 4 is an output result of a vertical first-order differential filter for the captured image of FIG. 3. FIG. It is a flowchart which shows the flow of a process in the brightness | luminance gradient vector calculation part of ECU of FIG. It is a flowchart which shows the flow of a process in the lane marker detection part of ECU of FIG. It is a block diagram of the lane detection apparatus which concerns on 2nd Embodiment. It is a flowchart which shows the flow of a process in the lane shape estimation part of ECU of FIG. It is a block diagram of the lane detection apparatus which concerns on 3rd Embodiment. It is an example of the lane shape candidate produced | generated by the lane shape candidate production | generation part of ECU of FIG. It is a flowchart which shows the flow of a process in the lane shape candidate production | generation part of ECU of FIG. It is a flowchart which shows the flow of a process in the lane shape estimation part of ECU of FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Lane marker detection apparatus, 11, 21 ... Lane detection apparatus, 2 ... Camera, 3, 13, 23 ... ECU, 3a ... Luminance gradient vector calculation part, 3b ... Lane marker detection part, 13c, 23c ... Lane shape estimation part, 23b ... lane shape candidate generator

Claims (5)

A lane marking detection device for detecting a lane marking on a road,
Imaging means for imaging a road;
A luminance gradient vector calculating means for calculating a luminance gradient vector of each pixel of the image captured by the imaging means;
Evaluation means for sequentially extracting a set of luminance gradient vectors from the luminance gradient vectors calculated by the luminance gradient vector calculation means, and evaluating the extracted luminance gradient vectors of each set;
A lane marking detecting means for extracting a set of pixels in which the direction of the luminance gradient vector is opposite based on the evaluation result by the evaluation means, and detecting a lane marking from the extracted set of pixels ;
The luminance gradient vector calculating means calculates, for each pixel, a primary differential value of the luminance in the horizontal direction of the image and a primary differential value of the luminance in the vertical direction of the image. A lane marking detection apparatus that calculates a luminance gradient vector from a value and a first-order differential value of luminance in a vertical direction .   The evaluation means obtains an evaluation value by calculating an inner product of a set of luminance gradient vectors composed of two point-symmetric pixels existing within a certain distance with an arbitrary pixel in the image as a center. The lane marking detection apparatus according to claim 1. A lane detection device for detecting a lane composed of two parallel lane markings on a road,
Imaging means for imaging a road;
A luminance gradient vector calculating means for calculating a luminance gradient vector of each pixel of the image captured by the imaging means;
Evaluation means for sequentially extracting a set of luminance gradient vectors from the luminance gradient vectors calculated by the luminance gradient vector calculation means, and evaluating the extracted luminance gradient vectors of each set;
A lane line detection unit that extracts a set of pixels in which the direction of the luminance gradient vector is opposite based on the evaluation result by the evaluation unit, and detects a lane line from the extracted set of pixels;
Lane shape estimation means for estimating the lane shape from the lane line detected by the lane line detection means ,
The luminance gradient vector calculating means calculates, for each pixel, a primary differential value of the luminance in the horizontal direction of the image and a primary differential value of the luminance in the vertical direction of the image. A lane detection device that calculates a luminance gradient vector from a value and a first-order differential value of luminance in a vertical direction .   The evaluation means obtains an evaluation value by calculating an inner product of a set of luminance gradient vectors composed of two point-symmetric pixels existing within a certain distance with an arbitrary pixel in the image as a center. The lane detection device according to claim 3. A lane detection device for detecting a lane composed of two parallel lane markings on a road,
Imaging means for imaging a road;
A luminance gradient vector calculating means for calculating a luminance gradient vector of each pixel of the image captured by the imaging means;
Lane shape candidate generating means for generating lane shape candidates based on a shape model or past detection results;
A lane shape estimation unit that collates a lane shape candidate generated by the lane shape candidate generation unit with a luminance gradient vector calculated by the luminance gradient vector calculation unit, and estimates a lane shape based on the comparison result ;
The luminance gradient vector calculating means calculates, for each pixel, a primary differential value of the luminance in the horizontal direction of the image and a primary differential value of the luminance in the vertical direction of the image. A lane detection device that calculates a luminance gradient vector from a value and a first-order differential value of luminance in a vertical direction .

Images