JP2016126613A - Road boundary line recognizing device, and road boundary line recognizing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce the possibility of mistaking boundary line candidates markedly deviating from a real boundary line as boundary lines even when the real boundary line cannot be extracted as a boundary line candidate.SOLUTION: A virtual boundary line candidate V2 lying along the real boundary line candidate R2 of an n-th picked-up image Fnwithout duplication is set by a virtual boundary line candidate setting unit. Where a position predicting unit supposes that the virtual boundary line candidate V2 and the like to be boundary lines, an n-th picked-up image of an n-th marker M (Tn) recognized by a predicted position m(Tn) of the n-th picked-up image Fnand the like are predicted, and a boundary line recognizing unit recognizes some of the virtual boundary line candidate V2 and the like as the boundary line B on the basis of comparison of a predicted position m(Tn) of the n-th marker M (Tn) and an extracted position M(Tn) of the n-th marker M(Tn) extracted from the n-th picked-up image.SELECTED DRAWING: Figure 8

Description

  One aspect of the present invention relates to a road boundary line recognition apparatus and a road boundary line recognition method.

  For the purpose of application to a system or the like that prevents the vehicle from deviating from the traveling lane, a three-dimensional structure such as a guard rail that forms the road edge of the road on which the vehicle travels and the road surface A technique for recognizing the boundary line is proposed. For example, in the apparatus of Patent Document 1, a plurality of line segments extending toward the vanishing point are detected in the captured image at the imaging time t1 in front of the vehicle, and these line segments are set as boundary line candidates between the guardrail and the road surface. . In the apparatus of Patent Document 1, in each case where these boundary line candidates are assumed to be boundary lines between the guard rail and the road surface, the predicted position of the guard rail post in the captured image at the imaging time t2 after the imaging time t1 The boundary line is recognized based on the comparison with the actual position of the post in the captured image at the imaging time t2.

US Patent Application Publication No. 2012/0069185

  By the way, in the apparatus of the above-mentioned patent document 1, since the boundary line between the three-dimensional structure such as the guard rail and the road surface of the road is shielded by the shielding objects such as the planted trees, the native plants, and the soil, In some cases, the boundary line candidate cannot be extracted. In such a case, there is a possibility that a boundary line candidate greatly deviating from the actual boundary line is erroneously recognized as a boundary line.

  Therefore, the present invention reduces the possibility of erroneously recognizing a boundary line candidate that deviates significantly from the actual boundary line as a boundary line even when the actual boundary line cannot be extracted as a boundary line candidate. An object of the present invention is to provide a road boundary line recognition device and a road boundary line recognition method capable of performing the above.

One aspect of the present invention is a road boundary line recognition device for recognizing a boundary line between a three-dimensional structure forming a road edge of a road on which a vehicle travels and a road surface of the road, based on an image captured by an in-vehicle camera, A real boundary line candidate extraction unit that extracts a real boundary line candidate that is a candidate for the boundary line for the n _1st captured image at the n _1st time point in front of the vehicle traveling on the road; and an actual boundary line of the n _1st captured image In each case where the virtual boundary line candidate setting unit that sets virtual boundary line candidates that do not overlap with the candidates and the actual boundary line candidates and the virtual boundary line candidates of the n _1st captured image are assumed to be boundary lines , for the n ₁ marker recognized by the n ₁ captured image, based on the movement information of the vehicle, the position prediction for predicting a predicted position in the n ₂ captured image of the n ₂ time points later than the n ₁ time And the position prediction unit The predicted position of the n ₁ marker, based on a comparison of the extraction position of the n ₁ markers extracted from the n ₂ captured image, recognizes one of the actual boundary line candidate and the imaginary boundary line candidate as a boundary line A road boundary line recognition apparatus including a boundary line recognition unit.

According to this configuration, by the virtual boundary line candidate setting unit, a virtual boundary line candidate along without overlapping actual boundary line candidate of the n ₁ captured image is set. Moreover, the position prediction section, in each case where the actual boundary line candidate and the imaginary boundary line candidate of the n ₁ captured image is assumed to be a boundary line, the first n ₁ marker recognized by the n ₁ captured image, based on the movement information of the vehicle, the predicted position in the n ₂ captured image is predicted, by the boundary line recognizing portion, and a predicted position of the predicted second n ₁ marker, the n ₁ extracted from the n ₂ captured image Based on the comparison with the marker extraction position, either the real boundary line candidate or the virtual boundary line candidate is recognized as the boundary line. Therefore, since the actual boundary line between the three-dimensional structure and the road surface is blocked by the shielding object, the actual boundary line itself cannot be directly recognized from the captured image, and the actual boundary line cannot be recognized on the actual boundary line candidate. Even if the line is not included, the virtual boundary line candidate can supplement the actual boundary line candidate, so the boundary line candidate that is greatly deviated from the actual boundary line may be erroneously recognized as a boundary line Can be reduced.

In this case, the actual boundary line candidate extraction unit, for the n _-1 captured image of the n _-1 time earlier than the n ₁ time to extract the actual boundary line candidate which is a candidate for the boundary line, position prediction section , in the case where the actual boundary line candidate of the n _-1 captured image is assumed to be a boundary line, the first n _-1 marker recognized by the n _-1 captured image, based on the movement information of the vehicle, the n A predicted position in the n _0th captured image at the n ₀ time point between the ₋₁ time point and the n ₁ time point is predicted, and the virtual boundary line candidate setting unit determines the n ₋₁ marker predicted by the position prediction unit. and predicted position, if the shift amount between the extraction position of the n _-1 markers extracted from the n ₀ captured image exceeds the first threshold value, it does not overlap the actual boundary line candidate of the n ₁ captured image The virtual boundary line candidate along can be set.

According to this configuration, if the shift amount between the extraction position of the n _-1 and the predicted position of the marker, the n _-1 markers extracted from the n ₀ captured image exceeds the first threshold value, the imaginary boundary The line candidate setting unit sets virtual boundary line candidates that do not overlap with the actual boundary line candidates of the n _1st captured image. For this reason, compared with the case where a virtual boundary line candidate is always set, the calculation load of a road boundary line recognition apparatus can be reduced.

Further, in each case where the real boundary candidate and the virtual boundary line candidate of the n _1st captured image are assumed to be boundary lines, the predicted position of the n ₁ marker predicted by the position prediction unit and the n _2nd imaging When there is no real boundary line candidate or virtual boundary line candidate whose deviation from the extracted position of the n _1st marker extracted from the image is equal to or smaller than the second threshold, the virtual boundary line candidate setting unit is more than the n ₂ time point. The n _3rd captured image at the later n ₃ time point does not overlap the actual boundary line candidate extracted by the real boundary line candidate extraction unit, and the number is larger than the virtual boundary line candidates set in the n _1st captured image. Many virtual boundary line candidates can be set.

According to this configuration, when the actual boundary line candidate or the virtual boundary line candidate that is sufficiently close to the actual boundary line whose deviation amount between the predicted position of the n _1st marker and the extraction position is equal to or smaller than the second threshold value cannot be recognized. since increasing the number of the virtual boundary line candidate set, it increases the possibility of being able to recognize a sufficiently close imaginary boundary line candidate to the real border in the first n ₃ time points after the first n ₂ time be able to.

Further, in each case where the real boundary candidate and the virtual boundary line candidate of the n _1st captured image are assumed to be boundary lines, the predicted position of the n ₁ marker predicted by the position prediction unit and the n _2nd imaging When there is no actual boundary line candidate or virtual boundary line candidate whose deviation from the extracted position of the n _1st marker extracted from the image is equal to or smaller than the third threshold, the virtual boundary line candidate setting unit is more than the n ₂ time point. the n for the n ₃ captured image of a _three point along without overlapping actual boundary line candidate extracted by the actual boundary line candidate extraction unit, and the imaginary boundary line candidate set to the n ₁ captured image position after the Different virtual boundary line candidates can be set.

According to this configuration, when the real boundary line candidate or the virtual boundary line candidate that is sufficiently close to the actual boundary line whose deviation amount between the predicted position and the extraction position of the n _1st marker is equal to or smaller than the third threshold value cannot be recognized. since changing the position of the imaginary boundary line candidate set, it increases the possibility of being able to recognize a sufficiently close imaginary boundary line candidate to the real border in the first n ₃ time points after the first n ₂ time be able to.

Another aspect of the present invention is a road boundary line recognition method for recognizing a boundary line between a three-dimensional structure forming a road edge of a road on which a vehicle travels and a road surface based on an image captured by an in-vehicle camera. An actual boundary line candidate extracting step of extracting an actual boundary line candidate that is a boundary line candidate for the n _1st captured image at the n _1st time point in front of the vehicle traveling on the road, and the n _1st captured image a virtual boundary line candidate setting step of setting a virtual boundary line candidate to be adjacent along without overlapping actual boundary line candidate, and recognizing the marker recognition step the first n ₁ marker at the n ₁ captured image, the n _In each case where the real boundary line candidate and the virtual boundary line candidate of _one captured image are assumed to be boundary lines, the n _2nd imaging at the n ₂ time point after the n ₁ time point is based on the movement information of the vehicle. Predicted position of the n _1st marker in the image Based on the comparison between the predicted position and the extracted position, the position extracting process of extracting the n _1st marker from the n _2nd captured image, and the predicted position and the extracted position, A road boundary line recognition method including a boundary line recognition step of recognizing any one of candidates as a boundary line.

  According to one aspect and another aspect of the present invention, even if a real boundary line cannot be extracted as a boundary line candidate, a boundary line candidate that is greatly deviated from the actual boundary line is erroneously detected as a boundary line. Can be reduced.

It is a block diagram which shows the road boundary line recognition apparatus of 1st Embodiment. It is a flowchart which shows operation | movement of the road boundary line recognition apparatus of FIG. It is a diagram showing a first _{n 1} captured image Fn ₁ of the _{n 1} time Tn _1. Is a diagram showing a state obtained by extracting the actual boundary line candidate R1~R4 by edge processing to the n ₁ captured image Fn ₁ of FIG. It is a diagram showing a state of setting a virtual boundary line candidate V1~V3 to the _{n 1} captured image Fn ₁ of FIG. It is a diagram showing a first _{n 1} marker M, which is extracted (Tn ₁₎ About _{n 1} captured image Fn ₁ of FIG. When it is assumed that the actual boundary line candidate R2 is the boundary line B, the predicted position m _R2 of the n ₁ marker M (Tn ₁ ) in FIG. 6 at the n ₂ time point Tn ₂ in the n ₂ captured image Fn ₂ (Tn ₂₎ is a diagram showing a state in which predicted. When it is assumed that the virtual boundary line candidate V2 is the boundary line B, the predicted position m _{V2 in} the n ₂ captured image Fn ₂ at the n ₂ time point Tn ₂ of the n ₁ marker M (Tn ₁ ) in FIG. (Tn ₂₎ is a diagram showing a state in which predicted. In the case where the actual boundary line candidate R3 was assumed to be the boundary line B, predicted position _{m R3} in the _{n 2} captured image Fn ₂ of the _{n 2} time Tn ₂ of the _{n 1} marker M in FIG. 6 (Tn ₁₎ (Tn ₂₎ is a diagram showing a state in which predicted. It is a diagram illustrating a second _{n 2} captured image Fn ₂ of the _{n 2} time Tn _2. Is a diagram showing an extracted position M (Tn ₂₎ of the _{n 1} marker M (Tn ₁₎ in the _{n 2} captured image Fn ₂ in FIG. It shows a comparison of the predicted position _m R2 (Tn ₂₎ and the extraction position M (Tn _2). It shows a comparison of the predicted position _m V2 (Tn ₂₎ and the extraction position M (Tn _2). It shows a comparison of the predicted position _m R3 (Tn ₂₎ and the extraction position M (Tn _2). It is a flowchart which shows the main flow of the road boundary line recognition apparatus of 2nd Embodiment. FIG. 16 is a flowchart illustrating an operation when a deviation amount between a predicted position and an extraction position in FIG. 15 exceeds a first threshold value. It is a flowchart which shows the main flow of the road boundary line recognition apparatus of 3rd Embodiment. FIG. 18 is a flowchart illustrating an operation when a deviation amount between a predicted position and an extraction position in FIG. 17 exceeds a second threshold value. It is a flowchart which shows the main flow of the road boundary line recognition apparatus of 4th Embodiment. It is a flowchart which shows operation | movement when the deviation | shift amount of an estimated position and the extraction position in FIG. 19 exceeds the 3rd threshold value.

  Hereinafter, a road boundary line recognition apparatus and a road boundary line recognition method according to an embodiment of the present invention will be described with reference to the drawings.

(First embodiment)
As shown in FIG. 1, the road boundary line recognition device 1 according to the first embodiment includes an in-vehicle camera 11, a vehicle speed sensor 12, a yaw rate sensor 13, an ECU 20, a display 31, and a speaker 32. The road boundary line recognition apparatus 1 is a boundary line between a three-dimensional structure such as a guard rail, a curbstone, a median strip, and a wall that forms the end of a road on which a vehicle travels based on an image captured by an in-vehicle camera 11 and the road surface of the road. Recognize

  Hereinafter, the boundary line means a line that divides a three-dimensional structure such as a guard rail, a curb, a median strip, and a wall that form the road edge of the road on which the vehicle travels and a road surface of the road. The boundary line is not necessarily a painted line. The boundary line is a shape that appears linearly or substantially linearly in the captured image due to the intersection of the surface of the three-dimensional structure and the road surface of the road. The boundary line may be a three-dimensional structure such as a recess, a road surface of a road, and a boundary part, instead of a strict straight line part. For example, in the case where the three-dimensional structure is a guard rail supported by a plurality of posts, the boundary line is a line connecting positions where the posts are installed.

  For example, the road boundary line recognition apparatus 1 detects the distance between the boundary line and the vehicle and displays the distance to the driver of the vehicle in order to prevent the vehicle from departing from the traveling lane. Moreover, the road boundary line recognition apparatus 1 notifies a necessary warning to the driver of the vehicle when the distance between the recognized boundary line and the vehicle is less than a preset threshold value. In the present embodiment, the detection of the white line that divides the road lane on the road surface is not the main purpose, but the white line may be detected using the detected boundary line after the detection of the boundary line. Alternatively, in the present embodiment, a virtual lane line that simulates a white line that is close to the vehicle side by a predetermined amount from the detected boundary line is set, and a distance between the lane line and the vehicle is set in advance. When it becomes less than, a warning necessary for the vehicle driver may be notified.

The in-vehicle camera 11 is, for example, a monocular camera having one image pickup unit provided on the back surface of a vehicle windshield. The imaging unit images the front of the vehicle. The in-vehicle camera 11 captures the front of the vehicle as an n _1st captured image at an arbitrary n ₁ time point, and captures the front of the vehicle as an n ₂ captured image at an arbitrary n ₂ time point after the n ₁ time point. To do. The _{n 1} point and the distance between the _{n 2} times, for example, be 50 msec.

Vehicle camera 11, similarly, the first _{n -1} captured image of the _{n -1} time before the first _{n 1} time captured, the _{n 0} time point between the first _{n -1} time and the _{n 1} time The n _0th captured image is captured. Vehicle camera 11, similarly, to image the second n ₃ captured image of the n ₃ time points later than the n ₂ times. In this way, the vehicle-mounted camera 11, for any natural number k, the _{n -k} time, the _{n -k + 1} point, ... the _{n -1} time, the _{n 0} time point, the _{n 1} time, the _{n 2} time, the _{n 3} times, ..., the _{n k-1} times, the _{n -k} captured image in the _{n k} point respectively, the _{n -k + 1} captured image, ... a _{n -1} captured image, the _{n 0} captured image, the n ₁ captured image, the _{n 2} captured image, the _{n 3} captured image, ..., the _{n k-1} captured image, a first _{n k} captured image for example, for imaging at a period of 50 msec. The in-vehicle camera 11 transmits information related to the captured image ahead of the vehicle to the ECU 20. The in-vehicle camera 11 may be either a monochrome camera or a color camera. The in-vehicle camera 11 may be a stereo camera.

  The vehicle speed sensor 12 is a sensor for detecting the speed of the vehicle. As the vehicle speed sensor 12, for example, a wheel speed sensor that is provided for a vehicle wheel or an axle that rotates integrally with the wheel and detects the rotation speed of the wheel as a signal is used. The vehicle speed sensor 12 transmits a signal corresponding to the rotational speed of the wheel to the ECU 20.

  The yaw rate sensor 13 is a sensor for detecting the direction of the vehicle by detecting the yaw rate (rotational angular velocity) around the vertical axis of the center of gravity of the vehicle. As the yaw rate sensor 13, for example, a gyro sensor can be used. The yaw rate sensor 13 outputs a signal corresponding to the detected yaw rate of the vehicle to the driving support ECU 2.

  The ECU 20 is an electronic control unit having a CPU [Central Processing Unit], a ROM [Read Only Memory], a RAM [Random Access Memory], and the like. In the ECU 20, the program stored in the ROM is loaded into the RAM and executed by the CPU, so that the hardware of the ECU 20 causes the real boundary line candidate extraction unit 21, the virtual boundary line candidate setting unit 22, the position prediction unit 23, and the boundary It functions as the line recognition unit 24. ECU20 may be comprised from several ECU.

The actual boundary line candidate extraction unit 21 extracts an actual boundary line candidate that is a boundary line candidate for an n ₁ -th captured image at an arbitrary n ₁ time point in front of a vehicle traveling on a road. The actual boundary line candidate, for the n ₁ captured image, by performing image processing, a candidate of the n ₁ boundary is directly extracted from the captured image. As described above, the n ₁ captured image of the n ₁ point is an image captured by the onboard camera 11. The actual boundary line candidate extraction unit 21 extracts an actual boundary line candidate by performing image processing such as edge processing on the n _1st captured image. In addition, the actual boundary line candidate extraction unit 21 may extract actual boundary line candidates by other methods.

Imaginary boundary line candidate setting unit 22 sets a virtual boundary line candidate along without overlapping actual boundary line candidate of the n ₁ captured image. The virtual boundary line candidate is located at a distance from the actual boundary line candidate, for example, for every one to several pixels set in advance along the actual boundary line candidate extracted from the n _1st captured image. This is a candidate for the boundary line set as follows. In the present embodiment, unlike the second embodiment described later, always imaginary boundary line candidate along without overlapping actual boundary line candidate of the n ₁ captured image is set. In the present embodiment, virtual boundary line candidates do not have to be set for all real boundary line candidates. Further, a plurality of virtual boundary line candidates may be set for one real boundary line candidate.

Position prediction unit 23, in each case where the actual boundary line candidate and the imaginary boundary line candidate of the n ₁ captured image is assumed to be a boundary line, the first n ₁ marker recognized by the n ₁ captured image, vehicle based on the movement information, it predicts the predicted position in the n ₂ captured image of the n ₂ time points later than the n ₁ time. The movement information is information related to the movement of the vehicle and means, for example, the speed, direction, yaw rate, and the like of the vehicle. In the present embodiment, the vehicle speed is detected by the vehicle speed sensor 12, and the vehicle yaw rate is detected by the yaw rate sensor 13. Assuming that the real boundary line candidate and the virtual boundary line candidate are boundary lines is, for example, within the area of the n _1st captured image divided by the real boundary line candidate or the virtual boundary line candidate assumed to be a boundary line. Thus, it is assumed that the region closer to the vehicle is the road surface of the road, and the region far from the vehicle is assumed to be a surface that is perpendicular or substantially perpendicular to the road surface of the road.

The n _1st marker means, for example, an arbitrary pixel group for specifying an arbitrary position in the n _1st captured image. As the n _1st marker, for example, a pixel group at the intersection of a line segment extending in the vertical direction of the n _1st captured image recognized from a guard rail post and the actual boundary line candidate can be applied. Alternatively, any pixel group that can be easily recognized in the n _1st captured image may be applied to the n _1st marker. For example, a pixel group of 4 pixels × 4 pixels, 8 pixels × 8 pixels, 16 pixels × 16 pixels, and the like can be applied to the pixel group of the n _1st marker. For specifying the pixel group, the luminance, brightness, saturation, hue, and the like of the pixels included in the pixel group can be used. The predicted position, the first n ₁ marker recognized by the n ₁ captured image of the n ₁ point at a position that is expected in the first n ₂ captured in the image of the n ₂ time points later than the n ₁ time is there.

When the position predicting unit 23 predicts the predicted position of the n _1st marker, the position predicting unit 23 determines the actual boundary line of the n _1st captured image from parameters such as the position and angle of the in-vehicle camera 11 with respect to the vehicle, for example. in the case of the candidate and each of the imaginary boundary line candidate assumed to be a boundary line, and calculates the position for the vehicle on a three-dimensional space of the n ₁ first n ₁ marker in the captured image of the n ₁ time. Next, the position prediction unit 23 calculates the position of the n ₁ marker in the three-dimensional space at the n ₂ time point after the n ₁ time point based on the vehicle movement information. Finally, the position predicting unit 23 calculates the n ₂ time point from the position of the n ₁ marker at the n ₂ time point with respect to the vehicle in the three-dimensional space based on parameters such as the position and angle of the in-vehicle camera 11 with respect to the vehicle. The predicted position of the n _1st marker in the n _2nd captured image is predicted. Thereby, the position prediction unit 23 can predict the predicted position of the marker.

As will be described later, the boundary line recognition unit 24 is based on a comparison between the predicted position of the n _1st marker predicted by the position prediction unit 23 and the extracted position of the n _1st marker extracted from the n _2nd captured image. Then, either the real boundary line candidate or the virtual boundary line candidate is recognized as a boundary line. The extraction position means the position of the n ₁ marker extracted as a pixel group corresponding to the n ₁ marker recognized in the n ₁ captured image in the n ₂ captured image.

Recognizing either a real boundary line candidate or a virtual boundary line candidate as a boundary line based on the comparison between the predicted position and the extracted position is, for example, that the amount of deviation between the predicted position and the extracted position of the n _1st marker is the smallest. This means that either a certain real boundary line candidate or a virtual boundary line candidate is recognized as a boundary line. Alternatively, recognizing either the actual boundary line candidate or the virtual boundary line candidate as a boundary line based on the comparison between the predicted position and the extracted position means, for example, that the amount of deviation between the predicted position of the n _1st marker and the extracted position is This means that either a real boundary line candidate or a virtual boundary line candidate equal to or less than a threshold value for recognizing a preset boundary line is recognized as a boundary line. The deviation amount means, for example, the shortest distance between the predicted position and the extraction position, the horizontal distance, the vertical distance, and the like. The threshold value for recognizing the boundary line may be a fixed value or a fluctuation value.

  The boundary line recognition unit 24 displays information such as the position of the recognized boundary line and the distance between the boundary line and the vehicle on the driver of the vehicle on the display 31. In addition, the boundary line recognition unit 24 notifies the vehicle driver of an alarm through the speaker 32 when the distance between the boundary line and the vehicle is less than a preset threshold value.

In the present embodiment, the real boundary line candidate extraction unit 21, the virtual boundary line candidate setting unit 22, the position prediction unit 23, and the boundary line recognition unit 24 perform the n- _k time point and the n- _{k + 1} time point for an arbitrary natural number k. , ..., a _{n -1} time, the _{n 0} time point, the _{n 1} time, the _{n 2} time, the _{n 3} times, ..., the _{n k-1} times, the _{n -k} captured image in the _{n k} point, respectively , the _{n -k + 1} captured image, ..., the _{n -1} captured image, the _{n 0} captured image, the _{n 1} captured image, the _{n 2} captured image, the _{n 3} captured image, ..., the _{n k-1} captured image , The process for each of the _nk captured images is repeatedly executed.

The actual boundary line candidate extraction unit 21, a virtual boundary line candidate setting unit 22, the position prediction unit 23, for any natural number j, the _{n -2j + 1} point, ..., the _{n -1} time, the _{n 1} times, the n ₃ point, ..., the _{n -2j + 1} captured image in the _{n 2j + 1} point each, ..., the _{n -1} captured image, the _{n 1} captured image, the _{n 3} captured image, ..., for the first _{n 2j + 1} captured image, respectively may be carried out a process, necessarily, the _{n -2J} point, ..., the _{n 0} time point, the _{n 2} times, ..., the _{n -2J} captured image in the _{n 2j} point respectively, ..., the _{n 0} captured image, the It is not necessary to perform processing for each of the n ₂ captured images,..., the n _2j captured image.

  The display 31 displays information such as the position of the boundary line recognized by the boundary line recognition unit 24 and the distance between the boundary line and the vehicle to the driver of the vehicle in response to a command signal from the boundary line recognition unit 24 of the ECU 20. When the distance between the boundary line and the vehicle is less than a preset threshold, the speaker 32 notifies the vehicle driver of an alarm by a command signal from the boundary line recognition unit 24.

Hereinafter, the operation of the road boundary line recognition apparatus 1 of the present embodiment will be described. As shown in FIG. 2, as the n _1st captured image acquisition step, an in-vehicle camera 11 of the road boundary line recognition apparatus 1 determines an n _1st captured image at an arbitrary n ₁ point in front of a vehicle traveling on the road. Obtained (S101).

In the n _1st captured image acquisition step, for example, an n _1st captured image Fn ₁ as illustrated in FIG. 3 is acquired. The n _1st captured image Fn ₁ shows the road surface 101 of the road 100 on which the vehicle travels and the guard rail 104 that is a three-dimensional structure that forms the road edge of the road 100. The road surface 101 is provided with a white line 102 that divides the traveling lane of the road 100. The guard rail 104 is supported by a plurality of posts 106 standing upright from the road surface 101 in the vertical direction.

The boundary line B between the guard rail 104 and the road surface 101 is shown in the n _1st captured image Fn ₁ . A part of the boundary line B is shielded by a shield 120 such as a plant, a native plant, and soil. In the n _1st captured image Fn ₁ , the boundary line B, the white line 102, and the contour line along the traveling direction of the road 100 of the guard rail 104 extend toward the vanishing point VP. The vanishing point VP, in the n ₁ captured image Fn _1, the perspective is the not parallel lines with a line which is parallel to line the n ₁ captured image Fn within ₁ in outline or the like of the real objects This is the point where the lines intersect.

As shown in FIG. 2, as the n _1st actual boundary line candidate extraction step, the actual boundary line candidate extraction unit 21 of the road boundary line recognition apparatus 1 performs the n _1st time point in front of the vehicle traveling on the road 100. For the n ₁ captured image Fn ₁ , actual boundary line candidates that are candidates for the boundary line B are extracted (S102).

In the n _1st actual boundary line candidate extraction step, for example, as shown in FIG. 4, the boundary line B, the guard rail 104, and the white line 102 are contour lines along the traveling direction of the road 100 toward the vanishing point VP. The extending line segment is extracted as a plurality of real boundary line candidates R1 to R4 by edge processing. In the edge processing, a plurality of edge points e are detected for each line segment, and line segments passing through the edge point e and extending toward the vanishing point VP are extracted as actual boundary line candidates R1 to R4. As a method for extracting the vanishing point VP, various known methods such as an optical flow can be applied.

  In the example of FIG. 4, the boundary line B is shielded by the shielding object 120, so that there are few edge points e detected. Therefore, for boundary line B, no actual boundary line candidate is extracted. Therefore, in consideration of the case where the boundary line B is shielded by the shielding object 120 in this way, the road boundary line recognition device 1 of the present embodiment always performs the following processing.

As shown in FIG. 2, the virtual boundary line candidate setting unit 22 of the road boundary line recognition apparatus 1 sets the real boundary line candidates R1 to R4 of the n ₁ captured image Fn ₁ as the n _1st virtual boundary line candidate setting step. Virtual boundary line candidates that are not overlapped are set (S103).

In the first n ₁ imaginary boundary line candidate setting step, for example, as shown in FIG. 5, the actual boundary line candidate R1~R4 at a distance by one to several pixels in the n ₁ periphery of the captured image Fn ₁ overlap allowed each other A plurality of virtual boundary line candidates V1 to V3 that are adjacent to each other and extend toward the vanishing point VP are set. The distances of the virtual boundary line candidates V1 to V3 from the real boundary line candidates R1 to R4 can be arbitrarily changed. In the example of FIG. 3, the virtual boundary line candidate V2 is set at a position that substantially matches the boundary line B.

As shown in FIG. 2, as a marker recognition step, the position prediction unit 23 of the road boundary line recognition system 1, the _{n 1} marker is recognized by the _{n 1} captured image Fn ₁ (S104). The marker recognizing step comprises for example, as shown in FIG. 6, as the _{n 1} marker M in the _{n 1} captured image Fn ₁ of the _{n 1} time Tn ₁ (Tn _1), the vertical contour of the posts 106 A pixel group of a square region whose center is located at the intersection of the line segment and the actual boundary line candidate R1 is recognized.

Thus, the position prediction unit 23, easily recognized in the _{n 1} captured image Fn _1, any pixel group is easy to identify the position in the _{n 1} captured image Fn ₁ as the _{n 1} marker M (Tn ₁₎ Can be recognized. In the n _1st captured image Fn ₁ , the n _1st marker M (Tn ₁ ) may be recognized at the positions of the real boundary line candidates R1 to R4 or the virtual boundary line candidates V1 to V3, and the real boundary line candidates R1 to R1. You may recognize in positions other than R4 or virtual boundary line candidates V1-V3.

As described later, when the first _{n 1} captured image Fn ₁ second _{n 1} marker M (Tn ₁₎ is recognized at the position of the real road surface 101, the actual boundary line candidate R1~R4 and imaginary boundary line candidate V1 Even if the assumption that ~ V3 is a boundary line is not actually correct, the predicted position of the n _1st marker M (Tn ₁ ) matches the extracted position, and it may not be possible to determine whether the assumption is correct. is there. Therefore, for example, the position prediction unit 23 can recognize the n _1st marker M (Tn ₁ ) at a position that is considered as far as possible from the vehicle (road surface 101) in the n _1st captured image Fn ₁ . . Alternatively, for example, as shown in the example of FIG. 6, the position prediction unit 23, on the actual boundary line candidate R1 in the position where the distance from the vehicle (road 101) In the n ₁ captured image Fn ₁ appears to farthest The n _1st marker M (Tn ₁ ) can be recognized.

As illustrated in FIG. 2, as the position prediction step, the position prediction unit 23 of the road boundary line recognition apparatus 1 uses the boundary candidates of the real boundary line candidates R1 to R4 and the virtual boundary line candidates V1 to V3 of the n _1st captured image Fn ₁ as boundaries. In each case assumed to be a line, based on vehicle movement information, the n _1st marker M (Tn ₁ ) in the n ₂ captured image at the n ₂ time point Tn ₂ after the n ₁ time point Tn _1. Is predicted (S105).

In the position prediction step, for example, as shown in FIG. 7, parameters such as the position and angle of the in-vehicle camera 11 with respect to the vehicle are assumed when the real boundary line candidate R2 of the n _1st captured image Fn ₁ is assumed to be a boundary line. To the position of the n ₁ marker M (Tn ₁ ) in the n ₁ captured image Fn ₁ at the n ₁ time point Tn ₁ relative to the vehicle in the three-dimensional space.

In this case, for example, it is assumed that the area closer to the vehicle in the n _1st captured image Fn ₁ divided by the actual boundary line candidate R2 is the road surface 101. On the other hand, for example, an area far from the vehicle in the n _1st captured image Fn ₁ divided by the actual boundary line candidate R2 is assumed to be a plane that intersects the road surface 101 perpendicularly with the actual boundary line candidate R2 as an intersection line. Is done. In the example of FIG. 7, it is assumed that the n _1st marker M (Tn ₁ ) exists on a plane that intersects the road surface 101 perpendicularly with the actual boundary line candidate R2 as an intersection line. From a parameter such as the position and angle of the in-vehicle camera 11 with respect to the vehicle, a plane that intersects the road surface 101 perpendicularly with the actual boundary line candidate R2 as an intersection, and a three-dimensional space of the n _1st marker M (Tn ₁ ) on the plane The position relative to the upper vehicle can be calculated.

Next, based on the vehicle movement information detected by the vehicle speed sensor 12 and the yaw rate sensor 13, the position prediction unit 23 performs the n _1st marker M ((n) at the n ₂ time point Tn ₂ after the n ₁ time point Tn _1. The position of Tn ₁ ) relative to the vehicle in the three-dimensional space is calculated. Finally, the position predicting unit 23 determines the position of the n ₁ marker M (Tn ₁ ) in the three-dimensional space at the n ₂ time point Tn ₂ based on parameters such as the position and angle of the in-vehicle camera 11 with respect to the vehicle. Thus, the predicted position m _R2 (Tn ₂ ) of the n ₁ marker M (Tn ₁ ) in the n ₂ captured image at the n ₂ time point Tn ₂ is predicted. Thereby, the position prediction unit 23 assumes that the real boundary line candidate R2 of the n _1st captured image Fn ₁ is a boundary line, and the predicted position m _R2 (Tn) of the n _1st marker M _R2 (Tn ₁ ). ₂ ) can be predicted.

Similarly, as shown in FIG. 8, in the case where the imaginary boundary line candidate V2 of the n ₁ captured image Fn ₁ assumed to be a boundary line, the n in the n ₂ time Tn ₂ of the n ₂ captured image A predicted position m _V2 (Tn ₂ ) of _one marker M (Tn ₁ ) is predicted. Similarly, as illustrated in FIG. 9, when it is assumed that the actual boundary line candidate R3 of the n _1st captured image Fn ₁ is a boundary line, in the n _2nd captured image at the n ₂ time point Tn ₂ . A predicted position m _R3 (Tn ₂ ) of the n _1st marker M (Tn ₁ ) is predicted.

As shown in FIG. 7, in the n _1st captured image Fn ₁ , assuming that the actual boundary line candidate R2 farther from the vehicle than the actual boundary line B is the boundary line between the guardrail 104 and the road surface 101, The moving distance of the n _1st marker M (Tn ₁ ) to the predicted position m _R2 (Tn ₂ ) is longer than the actual moving distance. Further, as shown in FIG. 8, when the first n ₁ captured image Fn _1, the imaginary boundary line candidate substantially coincide with the real border B V2 is assumed to be a boundary line between the guardrail 104 and the road surface 101 The moving distance of the n _1st marker M (Tn ₁ ) to the predicted position m _V2 (Tn ₂ ) is substantially the same as the actual moving distance. As shown in FIG. 9, when it is assumed that in the n _1st captured image Fn ₁ , the actual boundary line candidate R3 closer to the vehicle than the actual boundary line B is the boundary line between the guardrail 104 and the road surface 101. The moving distance of the n _1st marker M (Tn ₁ ) to the predicted position m _R3 (Tn ₂ ) is shorter than the actual moving distance.

As shown in FIG. 2, as the n _2nd captured image acquisition step, an arbitrary nth time point after the n ₁ time point Tn ₁ ahead of the vehicle traveling on the road is detected by the in-vehicle camera 11 of the road boundary recognition device 1. The n _2nd captured image at the _second time point Tn ₂ is acquired (S106). In the n _2nd captured image acquisition step, for example, an n ₂ captured image Fn ₂ as illustrated in FIG. 10 is acquired. As shown in FIG. 10, in the n ₂ captured image Fn ₂ at the n ₂ time point Tn ₂ , the post 106 and the shielding object 120 are compared to the n ₁ captured image Fn ₁ at the n ₁ time point Tn ₁ . It is reflected in the position on the near side.

As shown in FIGS. 2 and 11, as the n _2nd real boundary line candidate extraction step, the real boundary line candidate extraction unit 21 of the road boundary line recognition apparatus 1 performs the n _{2nd in} front of the vehicle traveling on the road 100. For the n ₂ captured image Fn ₂ at the time point Tn ₂ , actual boundary line candidates R1 to R4 that are candidates for the boundary line B are extracted in the same manner as the above-described n ₁ actual boundary line candidate extraction step (S107). . As the n _2nd virtual boundary line candidate setting step, the virtual boundary line candidate V1 along the real boundary line candidates R1 to R4 of the n ₂ captured image Fn ₂ by the virtual boundary line candidate setting unit 22 of the road boundary line recognition apparatus 1. To V3 are set in the same manner as in the above-described nth _first virtual boundary line candidate setting step (S108). Note that, as described above, the n _2nd real boundary line candidate extraction step and the n _2nd virtual boundary line candidate setting step are not essential and can be omitted.

As shown in FIG. 2 and FIG. 11, as the position extraction step, the boundary line recognition unit 24 of the road boundary line recognition system 1, the extraction position from the _{n 2} captured image Fn ₂ first _{n 1} marker M (Tn ₁₎ M (Tn ₂ ) is extracted (S109). Extraction extraction position M (Tn ₂₎ is carried out by extracting the pixel group corresponding to the _{n 1} pixel group marker M (Tn ₁₎ from the _{n 2} captured image Fn _2.

Since the n _1st marker M (Tn ₁ ) is recognized on the real boundary line candidate R1 of the n _1st captured image Fn ₁ , the extraction position M (Tn ₂ ) on the n _2nd captured image Fn ₂ is also actual. It is considered to be on the boundary line candidate R1. Therefore, in order to extract the pixel group corresponding to the pixel group of the n _1st marker M (Tn ₁ ) from the n _2nd captured image Fn ₂ , for example, the actual boundary line candidate R 1 is placed on the n ₂ captured image Fn _2. Along the area is set. In the region by the matching processing, while shifting the position of the pixel group corresponding to the n ₁ pixel group marker M (Tn _1), corresponding to the pixel group of the at the position first n ₁ marker M (Tn ₁₎ The correlation value between the pixel group to be processed and the pixel group of the n _2nd captured image Fn ₂ is calculated. The position on the n ₂ -th captured image Fn ₂ where the correlation value is maximized is determined as the extraction position M (Tn ₂ ) of the n ₁ marker M (Tn ₁ ).

As shown in FIG. 2, as the boundary line recognition process, the boundary line recognition unit 24 of the road boundary line recognition apparatus 1 performs prediction positions m _R2 (Tn ₂ ), m _V2 (Tn ₂ ), m _R3 (Tn ₂ ), and the like. Based on the comparison with the extraction position M (Tn ₂ ), any of the real boundary line candidates R1 to R4 and the virtual boundary line candidates V1 to V3 is recognized as the boundary line B (S110).

In the boundary line recognition step, for example, as shown in FIG. 12, the predicted position m _R2 (Tn ₂ ) of the n _1st marker M (Tn ₁ ) when the actual boundary line candidate R2 is assumed to be a boundary line is extracted. The position M (Tn ₂ ) is compared. As described above, when it is assumed that the actual boundary line candidate R2 farther from the vehicle than the actual boundary line B is the boundary line between the guardrail 104 and the road surface 101, the prediction of the n _1st marker M (Tn ₁ ) is performed. The moving distance to the position m _R2 (Tn ₂ ) becomes longer than the actual distance. Therefore, the predicted position m _R2 (Tn ₂ ) is closer to the vehicle than the extracted position M (Tn ₂ ) by the amount of deviation g, and the predicted position m _R2 (Tn ₂ ) and the extracted position M (Tn ₂ ) match. do not do.

In the boundary line recognition step, for example, as shown in FIG. 13, the predicted position m _V2 (Tn ₂ ) of the n _1st marker M (Tn ₁ ) when the virtual boundary line candidate V2 is assumed to be a boundary line. And the extraction position M (Tn ₂ ) are compared. As described above, when it is assumed that the virtual boundary line candidate V2 that substantially matches the actual boundary line B is the boundary line between the guardrail 104 and the road surface 101, the predicted position of the n _1st marker M (Tn ₁ ) The moving distance to m _V2 (Tn ₂ ) is almost the same as the actual distance. Therefore, the predicted position m _V2 (Tn ₂ ) substantially matches the extraction position M (Tn ₂ ).

In the boundary line recognition step, for example, as shown in FIG. 14, the predicted position m _R3 (Tn ₂ ) of the n _1st marker M (Tn ₁ ) when the real boundary line candidate R3 is assumed to be a boundary line. And the extraction position M (Tn ₂ ) are compared. As described above, when it is assumed that the actual boundary line candidate R3 closer to the vehicle than the actual boundary line B is the boundary line between the guardrail 104 and the road surface 101, the prediction of the n _1st marker M (Tn ₁ ) is performed. The moving distance to the position m _R3 (Tn ₂ ) is shorter than the actual distance. Therefore, the predicted position m _R3 (Tn ₂ ) is located behind the vehicle by the shift amount g from the extraction position M (Tn ₂ ), and the predicted position m _R3 (Tn ₂ ) matches the extraction position M (Tn ₂ ). do not do.

In the boundary line recognition step, the boundary line recognition unit 24, for example, in each case where the real boundary line candidates R1 to R4 and the virtual boundary line candidates V1 to _V3 are assumed to be boundary lines, the predicted position m _V2 (Tn ₂ ) And the extraction position M (Tn ₂ ), the virtual boundary line candidate V2 having the smallest deviation amount g is recognized as the boundary line B.

In the case where the _{n 1} marker M (Tn ₁₎ is recognized in the real road surface 101, the _{n 1} marker M in the actual boundary line candidate R1~R4 and imaginary boundary line candidate V1 to V3 (Tn ₁ For each of the real boundary line candidate and the virtual boundary line candidate located on the far side from the vehicle, whether or not the assumption that the real boundary line candidate and the virtual boundary line candidate are boundary lines is correct. The predicted position of the n _1st marker M (Tn ₁ ) matches the extraction position M (Tn ₂ ), the deviation g is 0, and it cannot be determined whether or not the assumption is correct.

  In such a case, the boundary line recognition unit 24, for example, assumes that each of the real boundary line candidate and the virtual boundary line candidate is a boundary line. The real boundary line candidate or virtual boundary line candidate closest to the vehicle among any of the boundary line candidates can be recognized as the boundary line B.

  In addition, as described above, when it is assumed that each of the real boundary line candidate and the virtual boundary line candidate is a boundary line, when there are a plurality of real boundary line candidates or virtual boundary line candidates whose deviation amount g is 0, If the amount g exceeds a preset threshold value for recognizing the boundary line, the boundary line recognition unit 24 suspends recognition that the boundary line B cannot be recognized or the boundary line B is recognized. May be output.

The processes of S101 to S110 are repeatedly executed at a predetermined cycle by the road boundary line recognition apparatus 1. Therefore, for any natural number k, the _{n -k} time, the _{n -k + 1} point, ..., the _{n -1} time, the _{n 0} time point, the _{n 1} time, the _{n 2} time, the _{n 3} times, ..., the _{n k-1} times, the _{n k} time the _{n -k} captured images in each, the _{n -k + 1} captured image, ..., the _{n -1} captured image, the _{n 0} captured image, the _{n 1} captured image, the n ₂ captured image, the _{n 3} captured image, ..., the _{n k-1} captured image, processing for the _{n k} captured image, respectively, are repeatedly performed.

For example, for any natural number j, the _{n -2j + 1} point, ..., the _{n -1} time, the _{n 1} time, the _{n 3} times, ..., the _{n -2j + 1} captured image in the _{n 2j + 1} point each, ..., a n _-1 captured image, the _{n 1} captured image, the _{n 3} captured image, ... for the _{n 2j + 1} captured image, respectively, the _{n 1} captured image acquisition process described above, the _{n 1} actual boundary line candidate extraction step, the n ₁ imaginary boundary line candidate extraction process, the same process as the marker recognition step and the position prediction step is performed, the n _-2J point, ..., the n ₀ time point, the n ₂ times, ..., the in the n _2j point respectively n _-2J captured image, ..., the _{n 0} captured image, the _{n 2} captured image, ..., for the _{n 2j} captured image, respectively, the _{n 2} captured image acquisition process described above, the _{n 2} actual boundary line candidate extraction process, the n ₂ imaginary boundary line candidate extraction Degree, the same processing as the position extraction step and boundary recognition process may be performed.

Also, for any natural number k, the _{n -k} time, the _{n -k + 1} point, ..., the _{n -1} time, the _{n 0} time point, the _{n 1} time, the _{n 2} time, the _{n 3} times, ..., the _{n k-1} times, the _{n k} time the _{n -k} captured images in each, the _{n -k + 1} captured image, ..., the _{n -1} captured image, the _{n 0} captured image, the _{n 1} captured image, the n ₂ captured image, the _{n 3} captured image, ..., the _{n k-1} captured image, for the _{n k} captured image, respectively, the _{n 1} captured image acquisition process described above, the _{n 1} actual boundary line candidate extraction step, the n _{The same processing as the one} virtual boundary line candidate extraction step, the marker recognition step, and the position prediction step is executed, and the n _−k−1 time point, the n _−k time point _{,. -2} point, the _{n -1} time, the _{n 0} time point, the _{n 1} time, the _{n 2} times, ..., the _{n k-} Point, the _{n -k-1} captured image in the _{n k-1} times, respectively, the _{n -k} captured image, ..., _{n-th -2} captured image, the _{n -1} captured image, the _{n 0} captured image, the n ₁ captured image, the n ₂ captured image, ..., the n _k-2 captured image, based on the results of the n _k-1 marker recognized is executed for the captured image each step and the position prediction step, the n ₂ described above Processes similar to the captured image acquisition process, the n _2nd real boundary line candidate extraction process, the n _2nd virtual boundary line candidate extraction process, the position extraction process, and the boundary line recognition process may be executed.

In the present embodiment, the virtual boundary line candidate setting unit 22 sets the virtual boundary line candidate V2 and the like that do not overlap the actual boundary line candidate R2 and the like of the n _1st captured image Fn ₁ . Moreover, the position prediction unit 23, in each case where such first n ₁ actual boundary line candidate of the captured image Fn ₁ R2 and imaginary boundary line candidate V2, etc. were assumed to be the boundary line, in the n ₁ captured image Fn ₁ With respect to the recognized _first marker M (Tn ₁ ), the predicted position m _V2 (Tn ₂ ) in the n _2nd captured image Fn ₂ is predicted based on the movement information of the vehicle, and is predicted by the boundary line recognition unit 24. and the _{n 1} predicted position _m V2 of the marker _{M (Tn 1) (Tn 2} ) or the like which is, the _{n 1} extraction position of the marker M (Tn ₁₎ M extracted from the _{n 2} captured image and (Tn ₂₎ Based on the comparison, either the real boundary line candidate R2 or the like or the virtual boundary line candidate V2 or the like is recognized as the boundary line B. Therefore, since the actual boundary line between the three-dimensional structure and the road surface is blocked by the shielding object, the actual boundary line itself cannot be directly recognized from the captured image, and the actual boundary line cannot be recognized on the actual boundary line candidate. Even if the line is not included, the virtual boundary line candidate can supplement the actual boundary line candidate, so the boundary line candidate that is greatly deviated from the actual boundary line may be erroneously recognized as a boundary line Can be reduced.

(Second Embodiment)
Hereinafter, a second embodiment of the present invention will be described. In the present embodiment, virtual boundary line candidates are not always set as in the first embodiment, but conditions for setting virtual boundary line candidates are defined. As shown in FIG. 15, as in the n- ₁ captured image acquisition process of the _first embodiment, the vehicle-mounted camera 11 of the road boundary line recognition device 1 is running on the road as the n- ₁ captured image acquisition process. The (n− ₁ ) th captured image at the (n− ₁ ) th time point before the _n1th time point in front of the vehicle is acquired (S201).

As with the n ₁ actual boundary line candidate extraction process in the first embodiment, as the n _-1 actual boundary line candidate extraction step, the actual boundary line candidate extraction unit 21 of the road boundary line recognition system 1, the road 100 About n _-1 captured image of the n _-1 time earlier than the n ₁ point in front of the traveling vehicle, actual boundary line candidate which is a candidate of the border line B is extracted (S202).

Similar to the first embodiment, as a marker recognition step, the position prediction unit 23 of the road boundary line recognition system 1, the _{n -1} marker is recognized by the _{n -1} captured image (S203). As in the first embodiment, as the position prediction step, the position prediction unit 23 of the road boundary line recognition apparatus 1 assumes that the actual boundary line candidate of the (n− ₁₎ -th captured image is a boundary line. based on the movement information, the predicted position in the n ₀ captured image of the n ₀ time point between the first n _-1 time and the n ₁ time of the n _-1 marker recognized by the n _-1 captured image Is predicted (S204). In this embodiment, since no virtual boundary line candidate has been set yet at this time, the n _- th marker of the (n- ₁₎ -th marker when it is assumed that the virtual boundary line candidate of the (n- ₁₎ -th captured image is a boundary line. The prediction position in the ₀ captured image is not predicted.

As with the n ₂ captured image acquisition step of the first embodiment, as the n ₀ captured image acquisition process, the vehicle-mounted camera 11 of the road boundary line recognition system 1, the n ₀ in the front of a vehicle traveling on a road The n _0th captured image at the time is acquired (S205). Similar to the n _2nd real boundary line candidate extraction step of the first embodiment, the actual boundary line candidate extraction unit 21 of the road boundary line recognition apparatus 1 travels on the road 100 as the n _0th real boundary line candidate extraction step. A real boundary line candidate that is a candidate for the boundary line B is extracted from the n _0th captured image at the n ₀ time point in front of the middle vehicle (S206). Incidentally, as in the first embodiment, the n ₀ actual boundary line candidate extraction process is not essential and can be omitted.

Similar to the first embodiment, as the position extraction step, the boundary line recognition unit 24 of the road boundary line recognition system 1, the extraction position of the n _-1 marker of the n ₀ captured image is extracted (S207). In the present embodiment, when the deviation amount g of the extraction position of the n _-1 markers extracted from the predicted position and the n ₀ captured image of the n _-1 marker is equal to or less than the first threshold value (S208), the boundary In the line recognition process, when the boundary line recognition unit 24 of the road boundary line recognition apparatus 1 assumes that the actual boundary line candidate is a boundary line, among the actual boundary line candidates whose deviation amount g is equal to or smaller than the first threshold value, Thus, the actual boundary line candidate having the smallest deviation amount g is recognized as the boundary line (S209). The first threshold value may be a fixed value or a variable value. Further, the first threshold value may be the same value as or different from the threshold value for recognizing the boundary line described above.

On the other hand, in this embodiment, when the shift amount g of the extraction position of the n _-1 markers predicted position of the n _-1 marker and extracted from the n ₀ captured image exceeds the first threshold value (S208 ), As shown in FIG. 16, the n _1st captured image acquisition step (S210), the n _1st real boundary line candidate extraction step (S211), and the n th ₁ virtual boundary line candidate extraction step (S212), marker recognition step (S213), position prediction step (S214), n _2nd captured image acquisition step (S215), n _2nd real boundary line candidate extraction step (S216), An n ₂ virtual boundary line candidate extraction step (S217), a position extraction step (S218), and a boundary line recognition step (S219) are executed. That is, the virtual boundary line candidate setting unit 22, the deviation amount g of the extraction position of the n _-1 markers extracted from the predicted position and the n ₀ captured image of the n _-1 marker exceeds a first threshold value In this case, a virtual boundary line candidate that does not overlap with the actual boundary line candidate of the n _1st captured image is set.

According to this embodiment, when the predicted position of the n _-1 marker, shift amount g of the extraction position of the n _-1 markers extracted from the n ₀ captured image exceeds the first threshold value, The virtual boundary line candidate setting unit 22 sets virtual boundary line candidates that do not overlap the actual boundary line candidates of the n-th _first captured image. For this reason, compared with the case where a virtual boundary line candidate is always set, the calculation load of the road boundary line recognition apparatus 1 can be reduced.

(Third embodiment)
Hereinafter, a third embodiment of the present invention will be described. In the present embodiment, when the deviation amount g exceeds a preset second threshold value, the number of virtual boundary line candidates to be set is increased. As shown in FIG. 17, the first embodiment of the same first _{n 1} captured image acquisition step and S101~S109 of FIG. 2 (S301), the _{n 1} actual boundary line candidate extraction step (S302), the _{n 1} virtual boundary line candidate extraction step (S303), a marker recognition step (S304), the position prediction step (S305), the _{n 2} captured image acquisition step (S306), the _{n 2} actual boundary line candidate extraction step (S307), the _{n 2} A virtual boundary line candidate extraction step (S308) and a position extraction step (S309) are executed.

In each case where the real boundary line candidate and the virtual boundary line candidate are assumed to be boundary lines, the extraction position of the n _1st marker predicted by the position prediction unit 23 and the nth extracted from the n _2nd captured image _When there is a real boundary line candidate or a virtual boundary line candidate whose deviation amount g from the extraction position of _one marker is equal to or smaller than the second threshold (S310), the boundary line recognition of the road boundary line recognition apparatus 1 is performed as a boundary line recognition step. the parts 24, the predicted position of the n ₁ marker, the n ₁ marker following actual boundary line candidate and the threshold for the deviation amount g recognizes the boundaries of the extraction position of the extracted from the n ₂ captured image Any of the virtual boundary line candidates is recognized as a boundary line (S311). In the present embodiment, the second threshold value is set to the same value as the threshold value for recognizing the boundary line, but the second threshold value may be different from the threshold value for recognizing the boundary line. The second threshold value may be a fixed value or a variable value.

On the other hand, in each case where the real boundary line candidate and the virtual boundary line candidate are assumed to be boundary lines, the predicted position of the n _1st marker predicted by the position prediction unit 23 and the n _2nd captured image are extracted. When there is no real boundary candidate or virtual boundary line candidate whose deviation amount g with respect to the extraction position of the n _1st marker is equal to or smaller than the second threshold (S310), as shown in FIG. 18, it is after the n ₂ time point. for the second _{n 3} captured image of the _{n 3} time, the first 1 S101 of FIG. 2 embodiment and S102 similar of the _{n 3} captured image acquisition step (S312) and the _{n 3} actual boundary line candidate extraction process (S313 ) Is executed.

As the n _3rd virtual boundary line candidate setting step, the virtual boundary line candidate setting unit 22 extracts the n ₃ captured image at the n ₃ time point after the n ₂ time point by the real boundary line candidate extracting unit 21. Along with not overlapping with the actual boundary line candidates, virtual boundary line candidates having a larger number than the virtual boundary line candidates set in the n _1st captured image in the n _1st virtual boundary line candidate extracting step (S303) are set ( S314).

Then, for the _{n 3} captured image, the same marker recognizing step and S104 and S105 of FIG. 2 in the first embodiment (S315) and the position prediction step (S316) is executed. Further, for the _{n 4} captured image of the _{n 4} time points later than the _{n 3} point, the first embodiment of S106~S110 the same first _{n 4} captured image acquisition step of FIG. 2 (S317), the n _{A four} real boundary line candidate extraction step (S318), an nth _fourth virtual boundary line candidate extraction step (S319), a position extraction step (S320), and a boundary line recognition step (S321) are executed.

In the present embodiment, the virtual boundary line candidate setting unit 22 performs the n ₃ time point, the n ₇ time point, the n ₁₀ time point,..., The n _{3 (m−1) +4} time point for an arbitrary natural number m. the _{n 3m + 4} time of the _{n 3} captured image, the _{n 7} captured image, the _{n 10} captured images, ..., the _{n 3 (m-1) +4} captured image, for the _{n 3m + 4} captured image, said first _{n 3} virtual The number of virtual boundary line candidates to be set may be increased in stages until the number of virtual boundary line candidates to be set reaches an arbitrary upper limit number by performing the same process as the boundary line candidate setting step.

According to the present embodiment, a real boundary line candidate or a virtual boundary line candidate that is sufficiently close to an actual boundary line whose deviation amount g between the predicted position of the n _1st marker and the extraction position is not more than the second threshold value cannot be recognized. the so increasing the number of the virtual boundary line candidate set, the possibility of being able to recognize a sufficiently close imaginary boundary line candidate to the real border in the first n ₃ time points after the first n ₂ time Can be increased.

(Fourth embodiment)
The fourth embodiment of the present invention will be described below. In the present embodiment, when the deviation amount g exceeds a preset third threshold value, the position of the set virtual boundary line candidate is changed. As shown in FIG. 19, the n _{1 first} captured image acquisition step (S 401), the n ₁ real boundary line candidate extraction step (S 402), and the n ₁ virtual boundary line candidate extraction step (S403), a marker recognition step (S404), the position prediction step (S405), the _{n 2} captured image acquisition step (S406), the _{n 2} actual boundary line candidate extraction step (S407), the _{n 2} A virtual boundary line candidate extraction step (S408) and a position extraction step (S409) are executed.

In each case where the real boundary line candidate and the virtual boundary line candidate are assumed to be boundary lines, the extraction position of the n _1st marker predicted by the position prediction unit 23 and the nth extracted from the n _2nd captured image _When there is a real boundary line candidate or a virtual boundary line candidate whose deviation amount g from the extraction position of _one marker is equal to or less than the third threshold (S410), the boundary line recognition of the road boundary line recognition apparatus 1 is performed as a boundary line recognition step. the parts 24, the predicted position of the n ₁ marker, the n ₁ of the extraction position of the marker shift amount g is the third threshold value or less of the actual boundary line candidate and the imaginary boundary line candidates extracted from the n ₂ captured image One of them is recognized as a boundary line (S411). In the present embodiment, the third threshold value is set to the same value as the threshold value for recognizing the boundary line described above, but the third threshold value may be different from the threshold value for recognizing the boundary line. The third threshold value may be a fixed value or a variable value.

On the other hand, in each case where the real boundary line candidate and the virtual boundary line candidate are assumed to be boundary lines, the predicted position of the n _1st marker predicted by the position prediction unit 23 and the n _2nd captured image are extracted. When there is no real boundary candidate or virtual boundary line candidate whose deviation g from the n _1st marker extraction position is equal to or smaller than the third threshold value (S410), as shown in FIG. 20, after the n ₂ time point. for the second _{n 3} captured image of the _{n 3} time, the first 1 S101 of FIG. 2 embodiment and S102 similar of the _{n 3} captured image acquisition step (S412) and the _{n 3} actual boundary line candidate extraction process (S413 ) Is executed.

As the n _3rd virtual boundary line candidate setting step, the virtual boundary line candidate setting unit 22 extracts the n ₃ captured image at the n ₃ time point after the n ₂ time point by the real boundary line candidate extracting unit 21. along without overlapping actual boundary line candidate, imaginary boundary line candidate positions are different are set with the virtual boundary line candidate set to the n ₁ captured image at the n ₁ imaginary boundary line candidate extraction step (S403) ( S414).

Then, for the _{n 3} captured image, the same marker recognizing step and S104 and S105 of FIG. 2 in the first embodiment (S415) and the position prediction step (S416) is executed. Further, for the _{n 4} captured image of the _{n 4} time points later than the _{n 3} point, the first embodiment of S106~S110 the same first _{n 4} captured image acquisition step of FIG. 2 (S417), the n ₄ actual boundary line candidate extraction step (S418), the _{n 4} imaginary boundary line candidate extraction step (S419), position extraction step (S420) and the boundary line recognizing step (S421) is executed.

According to the present embodiment, a real boundary line candidate or a virtual boundary line candidate that is sufficiently close to an actual boundary line whose deviation amount g between the predicted position of the n _1st marker and the extraction position is not more than the third threshold value cannot be recognized. the so it changes the position of the imaginary boundary line candidate set, the possibility of being able to recognize a sufficiently close imaginary boundary line candidate to the real border in the first n ₃ time points after the first n ₂ time Can be increased.

  The road boundary line recognition apparatus and the road boundary line recognition method according to the embodiment of the present invention are not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the embodiment of the present invention. Of course, it can be added. For example, two or three of the second to fourth embodiments described above may be implemented in a manner combined with each other. In this case, the first threshold value to the third threshold value may be the same value or different values.

DESCRIPTION OF SYMBOLS 1 ... Road boundary line recognition apparatus, 11 ... Car-mounted camera, 12 ... Vehicle speed sensor, 13 ... Yaw rate sensor, 20 ... ECU, 21 ... Real boundary line candidate extraction part, 22 ... Virtual boundary line candidate setting part, 23 ... Position prediction part , 24 ... boundary line recognition unit, 31 ... display, 32 ... speaker, 100 ... road, 101 ... road, 102 ... white line, 104 ... guardrail, 106 ... post, 120 ... shield, Fn 1 _... the _{n 1} captured image, Fn ₂ ... n _2nd captured image, VP ... vanishing point, B ... boundary line, e ... edge point, R1, R2, R3, R4 ... real boundary line candidate, V1, V2, V3 ... virtual boundary line candidate, M ( Tn ₁ ) ... n _1st marker, m _R2 (Tn ₂ ), m _V2 (Tn ₂ ), m _R3 (Tn ₂ ) ... predicted position, M (Tn ₂ ) ... extracted position, g ... shift amount.

Claims (5)

A road boundary line recognition device for recognizing a boundary line between a three-dimensional structure forming a road edge of a road on which a vehicle travels and a road surface of the road, based on an image captured by an in-vehicle camera,
An actual boundary line candidate extraction unit that extracts an actual boundary line candidate that is a candidate for the boundary line with respect to the n _1st captured image at the n ₁ time point in front of the vehicle traveling on the road;
A virtual boundary line candidate setting unit that sets a virtual boundary line candidate that does not overlap the actual boundary line candidate of the n _1st captured image;
In each case where the actual boundary line candidate and the imaginary boundary line candidate of the first n ₁ captured image is assumed to be the boundary line, the first n ₁ marker recognized by the first n ₁ captured image, the vehicle A position prediction unit that predicts a predicted position in the n ₂ captured image at the n ₂ time point after the n ₁ time point based on the movement information of
Based on the comparison between the predicted position of the n ₁ marker predicted by the position prediction unit and the extracted position of the n ₁ marker extracted from the n ₂ captured image, the actual boundary line candidate and the A boundary line recognition unit for recognizing any of the virtual boundary line candidates as the boundary line;
Road boundary line recognition device equipped with. The actual boundary line candidate extraction unit extracts the actual boundary line candidate which is a candidate of the boundary line for the first n _-1 captured image of the n _-1 time earlier than the n ₁ time,
The position prediction unit, in a case where the actual boundary line candidate of the first n _-1 captured image is assumed to be the boundary line, the first n _-1 marker recognized by the first n _-1 captured image, wherein Based on the movement information of the vehicle, predicting a predicted position in the n _0th captured image at the n ₀ time point between the n ₋₁ time point and the n ₁ time point;
Deviation of the imaginary boundary line candidate setting unit, a predicted position of the first n _-1 markers predicted by the position prediction unit, an extraction position of the first n ₀ the first n _-1 markers extracted from the captured image 2. The road boundary line recognition according to claim 1, wherein, when the amount exceeds a first threshold value, the virtual boundary line candidate is set along the actual boundary line candidate of the n _1st captured image without overlapping. apparatus. In each case where the real boundary line candidate and the virtual boundary line candidate of the n ₁ captured image are assumed to be the boundary line, the predicted position of the n ₁ marker predicted by the position prediction unit; When there is no real boundary line candidate or virtual boundary line candidate whose deviation from the extraction position of the n _1st marker extracted from the n _2nd captured image is equal to or less than a second threshold value,
The imaginary boundary line candidate setting unit, without overlapping the first n of the n ₃ time points later than the _second time point the n ₃ captured images for said actual boundary line candidate extracted by the actual boundary line candidate extraction unit The road boundary line recognition apparatus according to claim 1, wherein the virtual boundary line candidates are set in a larger number than the virtual boundary line candidates set in the n-th _first captured image. In each case where the real boundary line candidate and the virtual boundary line candidate of the n ₁ captured image are assumed to be the boundary line, the predicted position of the n ₁ marker predicted by the position prediction unit; When there is no real boundary line candidate or virtual boundary line candidate whose deviation from the extraction position of the n _1st marker extracted from the n _2nd captured image is a third threshold value or less,
The imaginary boundary line candidate setting unit, without overlapping the first n of the n ₃ time points later than the _second time point the n ₃ captured images for said actual boundary line candidate extracted by the actual boundary line candidate extraction unit 4. The road boundary line recognition apparatus according to claim 1, wherein the virtual boundary line candidate having a position different from the virtual boundary line candidate set in the n-th _first captured image is set. A road boundary line recognition method for recognizing a boundary line between a three-dimensional structure forming a road edge of a road on which a vehicle travels and a road surface of the road, based on an image captured by an in-vehicle camera,
An actual boundary line candidate extracting step of extracting an actual boundary line candidate that is a candidate for the boundary line for an n ₁ th captured image at the n ₁ time point in front of the vehicle traveling on the road;
A virtual boundary line candidate setting step of setting virtual boundary line candidates so as to be adjacent to each other without overlapping the actual boundary line candidates of the n _1st captured image;
Recognizing marker recognizing step to the n ₁ marker by the first n ₁ captured image,
In each case where the actual boundary line candidate and the imaginary boundary line candidate of the first n ₁ captured image is assumed to be the boundary line, on the basis of the movement information of the vehicle, the post than the first n ₁ time a position prediction step of predicting the predicted position of the n ₁ marker in the n ₂ captured image at the time point n ₂ ;
A position extracting step of extracting an extraction position of the n _1st marker from the n _2nd captured image;
Based on the comparison between the predicted position and the extracted position, a boundary line recognition step for recognizing either the real boundary line candidate or the virtual boundary line candidate as the boundary line;
Road boundary line recognition method.

Images