JP6017180B2 - In-vehicle environment recognition system - Google Patents

Description

  The present invention relates to an in-vehicle environment recognition apparatus that recognizes an environment outside a vehicle based on an image captured by an in-vehicle camera.

  2. Description of the Related Art In recent years, multi-application devices that recognize an environment outside a vehicle based on an image captured by a vehicle-mounted camera and simultaneously execute a plurality of applications according to the recognized environment outside the vehicle are becoming mainstream. In Patent Document 1, it is determined whether the adjacent lane of the host vehicle traveling lane is outside the road, and if it is outside the road, vehicle detection on the shoulder side is stopped to prevent erroneous detection as another vehicle. Technology is shown.

JP 2002-104112 A

  In order to realize a highly convenient and safe multi-application device within the limited capability range of the CPU, it is necessary to appropriately select and execute a plurality of applications according to the environment from a plurality of applications. . The technique of Patent Document 1 determines whether the left and right areas of the own vehicle traveling path are in the adjacent traveling road or outside the road, and only stops the application when it is outside the road. In order to switch to, to change the execution timing of the lane departure warning, or to change from the lane departure warning to the lane departure control, the information on the outside environment is insufficient.

  The present invention has been made in view of the above points, and an object of the present invention is to provide an in-vehicle environment recognition device that can appropriately recognize an environment outside the vehicle and execute a plurality of applications according to the environment. Is to provide.

  The in-vehicle environment recognition device of the present invention that solves the above problems recognizes a lane mark that divides a lane from an image, and sets an out-of-road determination area on at least one of the left and right sides of the vehicle lane based on the lane mark. It is characterized in that it is determined whether the road outside determination area is an adjacent lane or the road outside, and the road outside type is analyzed when it is determined that the road outside determination area is outside the road.

  According to the present invention, information on the environment outside the vehicle necessary for executing a plurality of applications can be acquired, the priority, processing time, and application change timing of the plurality of applications are controlled, and the application application for each area is controlled. The execution state can be changed. Therefore, it is possible to implement an appropriate application according to the surrounding environment of the vehicle that changes according to the traveling of the vehicle, and to provide a highly convenient and safe in-vehicle environment recognition device within the limited capability range of the CPU. Can be realized.

The block diagram of a vehicle-mounted environment recognition apparatus. The block diagram of a lane mark recognition part. The block diagram of a road exterior state determination part. The block diagram of an application control part. The block diagram of an application execution part. The flowchart which shows the content of the process in a vehicle-mounted environment recognition apparatus. The figure which shows the setting state of a lane recognition process area. The figure which shows the state which calculated the white line feature-value from the lane recognition process area. The figure which shows the state which extracted the white line from the lane recognition process area. The figure explaining the road classification determination method. The figure explaining the application change based on the analysis result of a roadside analysis part. The figure explaining the application change (vehicle detection and pedestrian detection) at the time of the left shoulder. The figure which shows the application change table at the time of the left shoulder. The figure explaining the structure of a road classification determination part. The figure explaining each structure of a convex road shoulder determination part, a concave road shoulder determination part, and a lane reduction determination part. The figure explaining each structure of a structure determination part, a periodic-object determination part, a non-artificial object determination part, and a driving | running | working out-of-road determination part. The figure which shows the specific example which searches the linearity of the lower end of a solid object.

Next, the present embodiment will be described in detail with reference to the drawings.
FIG. 1 is a configuration diagram of an in-vehicle environment recognition device.
The in-vehicle environment recognition device 10 is realized by, for example, dedicated hardware and software installed in a vehicle. As shown in FIG. 1, the in-vehicle environment recognition device 10 has an imaging unit 100, a lane mark recognition unit 200, an off-road condition, and the like. The determination unit 300, the application control unit 400, and the application execution unit 500 are included.

  The in-vehicle environment recognition device 10 recognizes a lane mark using an image photographed by the imaging unit 100 (lane mark recognition unit 200). Then, using the recognition result of the lane mark, a processing area is set, and an out-of-road condition is determined for each area (out-of-road condition determining unit 300).

  Change at least one of the priority, processing time, and setting of each application so that the appropriate application can be executed from the multiple applications at the appropriate timing according to the road conditions by area (application control) Part 400). The application is executed based on the results of the application control unit 400 (application execution unit 500). As a result, it is possible to improve the safety and convenience of the user by switching the appropriate application, changing the priority, switching from alarm to control, etc. according to the situation outside the road. it can.

FIG. 2 is a configuration diagram of the lane mark recognition unit.
The lane mark recognizing unit 200 recognizes the lane mark that partitions the road surface using the image captured by the image capturing unit 100. The lane mark recognition unit 200 recognizes, for example, the left lane mark 22 and the right lane mark 23 on both the left and right sides of the host vehicle lane 21 in which the host vehicle 1 is traveling as shown in FIG. The adjacent left lane mark 32 of the left adjacent lane 31 and the adjacent right lane mark 42 of the right adjacent lane 41 are also recognized. The recognition result is used by the out-of-road condition determination unit 300 to determine the out-of-road condition. FIG. 7A is a diagram schematically illustrating an image obtained by capturing the front from the host vehicle 1, and FIG. 7B is a plan view illustrating a traveling state of the host vehicle 1.

  The lane mark recognition area setting unit 210 sets a lane mark recognition area for recognizing a lane mark. For example, in FIG. 7, a lane mark recognition processing region L1 for recognizing the left lane mark 22 and a lane mark recognition processing region R1 for recognizing the right lane mark 23 are set on both the left and right sides of the vehicle lane 21. A lane mark recognition processing area L2 for recognizing the adjacent left lane mark 32 in the left adjacent lane and a lane mark recognition processing area R2 for recognizing the adjacent right lane mark 42 in the right adjacent lane are set.

  The white line feature amount calculation unit 220 scans the lane mark recognition processing regions L1, L2, R1, and R2 in the horizontal direction of the screen and calculates the feature amount of the white line that is the lane mark displayed on the road surface (FIG. 8). See).

  The white line extraction unit 230 performs a process of extracting the locus of the lane mark by curve approximation based on the white line feature amount calculated by the white line feature amount calculation unit 220 to obtain an extracted white line. In the example shown in FIG. 9, the extracted white lines WL1 and WL2 are extracted along the left lane mark 22 and the adjacent left lane mark 32, and the extracted white lines WR1 and WR2 are extracted along the right lane mark 23 and the adjacent right lane mark 42. The

  The three-dimensional position analysis unit 240 calculates the lateral position and yaw angle of the host vehicle 1 in the host vehicle travel lane 21 and the curvature of the host vehicle travel lane 21 based on the extracted white line extracted by the white line extraction unit 230. The lateral position is not only the distance in the road width direction from the own vehicle 1 to the left and right extracted white lines WR1 and WL1, but also the distance in the road width direction to the adjacent left lane mark 32 of the left adjacent lane 31 and the right adjacent lane 41. The distance in the road width direction to the adjacent right lane mark 42 is also estimated.

FIG. 3 is a configuration diagram of the out-of-road state determination unit.
The out-of-road condition determination unit 300 sets an out-of-road determination area for determining an out-of-road condition based on the recognition result of the lane mark (out-of-road determination area setting unit 310). The out-of-road determination area is set on at least one of the left and right sides of the host vehicle lane based on the lane mark.

  In the example shown in FIG. 7, in order to perform a road surface analysis between each lane mark as an out-of-road state, based on the positions of the lane marks 22, 23, 32, and 42 recognized by the lane mark recognition unit 200, Road determination areas RoL, RoC, and RoR are set. The out-of-road determination area Roc is set in the host vehicle lane 21 between the left lane mark 22 and the right lane mark 23, and the out-of-road determination area RoL is between the left lane mark 22 and the adjacent left lane mark 32. The left adjacent lane 31 is set, and the out-of-road determination area RoR is set to the right adjacent lane 41 between the right lane mark 23 and the adjacent right lane mark 42.

  By using these out-of-road determination areas RoL, RoC, and RoR, it is possible to stably determine the out-of-road state without being affected by the type and position of the lane mark. If there is an adjacent left lane mark 32 further to the left of the left adjacent lane 31 and an adjacent right lane mark 42 further to the right of the right adjacent lane 41, the lane on which the host vehicle 1 is traveling is Therefore, the lane recognition result can be used to determine the off-road condition. As described above, by using the result of recognizing the lane mark in advance for the processing area setting and the outside road determination, it is possible to realize a more stable and highly accurate outside road determination.

  When only the host vehicle lane 21 exists and the left adjacent lane 31 and the right adjacent lane 41 do not exist, the adjacent left lane mark 32 of the left adjacent lane 31 and the adjacent right lane mark 42 of the right adjacent lane 41 exist. Therefore, the left and right out-of-road determination areas RoL and RoR are set wide so that a three-dimensional object with a slightly distant lateral position can be detected as an obstruction outside the road, based on the lane width of the own vehicle lane 21 Is done.

  The out-of-road state determination unit 300 determines the out-of-road type for the out-of-road determination regions RoL, RoC, and RoR set by the out-of-road determination region setting unit 310 (out-of-road type determination unit 320). In addition to determining the type of road, such as whether it is a road that can be traveled by area or whether it is out of the road with obstacles, grass, etc., what kind of out-of-road state is when the vehicle cannot travel Is also analyzed (lateral position analysis unit 330). A method for analyzing the road condition will be described later.

  Next, in addition to this type, we will analyze the lateral position in order to more accurately grasp the situation of obstacles outside the road and reflect it in the timing of application switching, alarms, and control .

  As shown in FIGS. 11 to 14, the detailed lateral position of the obstacle outside the road is also analyzed, and information on the lateral position is used for timing change such as application switching, warning, and control. Using the result of analyzing the situation outside the road in this way, changing the recognition app, alarm, and control makes it possible to change the app according to the user's needs that change according to the scene, and change the timing of the alarm and control Is possible.

  FIG. 4 is a configuration diagram of the application control unit. The application control unit 400 uses the determination result of the out-of-road state determination unit 300 to change application settings, change priority of a plurality of applications, stop, return, and the like. First, in response to the result of the out-of-road situation, the priority of the execution application is determined, and depending on the low priority, determination is made such as stopping or starting the stopped application (priority determination unit 410).

  Next, for an application that is not stopped as a result of this priority, processing time is predicted (processing time prediction unit 420). Then, based on the predicted processing time, priority, out-of-road conditions, current CPU performance, and vehicle behavior information, an appropriate application is selected from a plurality of applications and alarm / control is set (application setting) Part 430).

  FIG. 5 is a configuration diagram of the application execution unit. The application execution unit 500 executes each application based on the setting contents set by the application setting unit 430. The application execution unit 500 exists in a state where a plurality of applications can be executed, and an application that is set to be executed by the application setting unit 430 is executed from among the plurality of applications. In the present embodiment, as examples of a plurality of applications, a lane departure warning unit 520, a lane departure control unit 530, a lane change support unit 540, a pedestrian detection unit 550, a vehicle detection unit 560, a sign detection unit 570, and a light distribution control 580 and road surface sign detection unit 590, and executes at least one of these applications.

  FIG. 6 shows an outline of the processing flow. First, in step S1, the outside of the vehicle is imaged from the in-vehicle camera (imaging unit 100). In step S2, the image picked up in step S1 is subjected to image processing to recognize a lane mark (lane mark recognition unit 200). In the lane mark recognition, as shown in FIG. 7, not only the left lane mark 22 and the right lane mark 23 on both the left and right sides of the vehicle lane 21, but also the left lane mark 32 of the left adjacent lane 31 on the outer side, In order to recognize the right lane mark 42 of the adjacent lane 41, a total of four lane mark recognition processing areas L1, L2, R1, and R2 are set.

  As shown in the upper part of FIG. 7, by setting four lane mark recognition processing regions L1, L2, R1, and R2, the lanes of adjacent lanes 31 and 41 adjacent to the host vehicle 1 as shown in the lower part of FIG. The marks 32 and 42 are also recognized. The recognition results of the four lane marks 22, 23, 32, and 42 are used in the following three ways.

  First, the recognition result of the two lane marks in the own vehicle lane 21, that is, the two lane marks 22 and 23 in the lane mark recognition processing areas L1 and R1 in FIG. Or, it is used as it is for lane departure control (first use).

In step S3, an out-of-road determination processing area is set using the recognition results of all four lane marks 22, 23, 32, and 42 (second use). By providing the road outside determination area RoL between the lane mark 22 and the lane mark 32, the road outside determination can be appropriately performed.
However, if the left lane 31 of the vehicle lane 21 is originally out of the road, the lane mark 22 exists but the lane mark 32 does not exist further to the left. For this reason, in order to set an area for judging the road outside, if the lane mark 32 does not exist in the first place, it is highly possible that the left side of the own vehicle traveling lane 21 is outside the road. The processing area for confirmation is set using the default lane width. Based on the left lane mark 22 of the own vehicle lane, a processing area having a total width of 5 m, that is, a basic lane width of 4.0 m and a margin width of 1 m is set on the left side, and it is determined whether or not this processing area is out of the road. In some countries, the basic lane width may be changed. The margin width is set so that a structure indicating that the vehicle is out of the road, a road shoulder, or the like can easily enter.

The same applies to the right lane.
In addition, if the vehicle lane 21 itself is such as lane marks 22 and 23 that cannot be detected well, it is assumed that the vehicle is traveling in the center of the vehicle lane, and the vehicle of the basic lane width is assumed. The road lane 21 region and the left and right traveling lanes 31 and 41 are subjected to out-of-road determination in a region having a basic lane width and a margin width.

  Finally, the lane recognition results of the lane mark recognition processing areas L2 and R2 are used for out-of-road determination (third use). For example, when the lane mark 32 is found with high accuracy in the lane mark recognition processing area L2, there is a very high possibility that the road outside determination area RoL is the left adjacent lane 31 through which the vehicle can pass.

  In addition, by providing an out-of-road determination area RoL between the lane mark recognition processing areas L1 and L2, the out-of-road determination area is not affected by obstacles or the like existing further to the left of the out-of-road determination area RoL. You can check whether RoL is really a road (left adjacent lane).

  If the lane mark is not found with high accuracy in the lane mark recognition processing area L2, the area on the left side of the vehicle lane 21 is likely to be outside the travel road. In this case, based on the lane mark recognition result in the lane recognition L1 processing area, the road outside determination area RoL is set outside and it is determined whether the road outside determination area RoL is outside the road. If the out-of-road determination area RoL is predicted to be out of the road, the process proceeds to steps S4 and S5, and the out-of-road type and lateral position are analyzed.

  In step S4, a process for determining the type outside the road is performed. When the lane marks 32 and 42 are recognized in either one of the lane mark recognition processing areas L2 and R2 in step S2, the left or right area of the own vehicle travel lane 21 is an adjacent lane in which the vehicle can travel. Since the possibility increases, an out-of-road determination area RoL is set between the lane marks 22 and 32, and an out-of-road determination area RoR is set between the lane marks 23 and 42. Then, image processing in each of the out-of-road determination areas RoL and RoR is performed, and confirmation processing is performed to determine whether the road is a traveling road (adjacent lane). If the road is not likely to be a road, the out-of-road type determination is performed as in the case where no lane mark is found in the lane mark recognition processing areas L2 and R2. Details will be described with reference to FIG.

  In step S5, a lateral position analysis outside the road is performed. Based on the out-of-road type analyzed in step S4, the distance from the travel lane of the host vehicle 1 in the lateral direction (road width direction) is analyzed in detail from which position it enters the out-of-road type area. For example, when it is found that a guard rail exists, the lateral distance of the guard rail from the travel lane of the host vehicle 1 is analyzed. Furthermore, if the yaw angle between the road type and the host vehicle is calculated, it is possible to appropriately execute switching timing such as application change.

  In step S6, the multi-app priority, which is the priority of a plurality of applications, is determined using the out-of-road type determined in step S4 and the out-of-road lateral position analyzed in step S5. Here, it is determined whether or not the priority order of applications or the application is stopped without considering the time condition such as whether the application is terminated at the processing time within the specified period.

  In step S7, based on the multi-application priority determined in step S6, processing for calculating and predicting the processing time when each application is executed is performed. In addition, the processing time of the function-limited version is also predicted for an application that has a low priority and may implement a function-limited version that is limited to a specific accuracy and range.

  In step S8, if the total processing time does not end within the preset specified processing cycle, stop the application with a lower priority or consider implementing a function-limited version, and process each application. Make the final setting so that the time ends within the specified period. As a result, the contents of the multi-application control table such as the selected application, priority, processing area, and processing time are determined.

  In step S9, using the multi-application control table determined in step S8, each application is executed within a specified period, and the process returns to step S1.

FIG. 10 is a diagram for explaining an out-of-road type determination method.
In the out-of-road type determination process of step S4, image processing is performed within the processing area set for out-of-road determination, and the type determination is performed. In the determination of the type of outside road, it is determined as one of (A1) traveling lane, (A2) leftmost lane, (A3) rightmost lane, and (A4) oncoming lane, as indicating the traveling road. In left-hand traffic areas such as Japan, the leftmost lane is outside the road on the left side, and if it is an expressway, the average lane is the slowest lane, and the rightmost lane is the overtaking lane. The area on the right side of the rightmost lane is the opposite lane, the center separation band, or the guardrail. However, in the right-hand traffic area such as the United States, the leftmost lane and the rightmost lane are handled in reverse.

  If it can be detected that the vehicle is driving the right lane on the Japanese road, it is clear that there is no further lane on the right side. Control, light distribution control, and the like are included, and the necessity of lane change support on the right side and the need for vehicle detection for tracking control are reduced. If this is the United States, this is an adaptive application for the left-side processing area in the left lane that is traveling in the left lane.

  If there is an adjacent lane and the own vehicle 1 can change lanes, information on whether or not another vehicle is running on the lane is also important information for determining which application should be executed. Become. In addition, information on whether or not the outside of the adjacent lane is outside the road also affects whether or not the sign detection or the like is further performed outside, so it is also possible to determine whether or not the outside of the adjacent lane is outside the road. carry out.

  Below, the usage method and its determination method are shown in detail for the lane type. Here, the description is based on left-hand traffic, such as Japan, but if it is right-hand traffic, such as the United States, it can be handled by handling it symmetrically.

(A1) "Travel lane"
(A1-1) <Concept of reflecting on priority> When there are adjacent lanes on the left and right of the host vehicle 1, it is considered that other vehicles travel on the adjacent lane, vehicle detection is performed, and light distribution is performed at night. Increase control priority. And the possibility that there is a pedestrian in the side of the own vehicle run lane is considered low, and the priority of pedestrian detection is reduced. Also, when the host vehicle 1 tries to change the lane from the host vehicle lane to the adjacent lane, the host vehicle 1 is prevented from traveling or the host vehicle 1 is expected to prevent other vehicles from traveling. The host vehicle 1 may perform lane departure control. Use to select apps to ensure greater safety.

(A1-2) <Determination method> Whether or not the own vehicle travel lane in which the host vehicle 1 is traveling is a travel lane is a lane in which the left and right ends of the own vehicle travel lane 21 are surrounded by lane marks, It is determined using the information on the type of lane mark (solid line, yellow line, multiple line) that the lane is not the leftmost or rightmost lane in the same direction. The accuracy may be improved by using information of car navigation, or recognition information such as other signs, three-dimensional objects, and other vehicles. Further, the presence of the left lane mark is confirmed in the lane mark recognition processing area L2, and if there is a broken lane mark on the left outer side, it can be confirmed that the left lane is a traveling lane. The same applies to the right side.

(A2) “Leftmost lane”
(A2-1) <Concept of reflection on priority> When it is determined that the host vehicle 1 is traveling in the leftmost lane (leftmost lane), the roadside determination area RoL adjacent to the left of the host vehicle 1 Since it is off-road, the priority of the application of a pedestrian jumping out and a sign detection from a plurality of applications is raised. Further, if the road outside determination area RoL is outside the road, it is clear that there is no other vehicle, and therefore, the priority of the vehicle detection application is reduced in the road outside determination area RoL. When the adjacent lane on the left side of the host vehicle 1 is the leftmost lane, the host vehicle 1 is traveling in the normal driving lane, so basically the same application priority as the driving lane is given. No problem with degree.

(A2-2) <Determination method> When there is a solid line on the left side of the traveling lane, it is determined to be the leftmost lane. Similar to the traveling lane determination method, the accuracy may be improved by using recognition information such as car navigation, other signs, three-dimensional objects, and other vehicles.

(A3) “Rightmost lane”
(A3-1) <Concept of reflecting on priority> When it is determined that the host vehicle 1 is traveling in the rightmost lane (the rightmost lane), is the right side of the host vehicle 1 an opposite lane? Or, there is a high possibility that it is out of the road where guardrails or separation zones exist. When the out-of-road determination area RoR is not an oncoming lane but outside the road, the priority of the sign detection application is increased and the priority of the vehicle detection application is decreased.

(A3-2) <Determination method> The right lane mark of the host vehicle lane is a solid line, and there is no lane mark next to it that seems to be an oncoming lane. When there is a structure, it is determined to be out of the road.

(A4) “Oncoming lane”
(A4-1) <Concept of reflecting on priority> When the right side of the vehicle lane is the oncoming lane, deviation from the right lane is likely to lead to a serious accident such as a frontal collision with the oncoming vehicle. . For this reason, lane departure control is used to ensure greater safety when the lane departs to the right side, and conversely, if there is a lane on the left side of the vehicle lane, a lane departure warning is used. Try not to disturb your intention as much as possible. In addition, when there is a risk of lane change in combination with vehicle detection or the like, safety may be improved by changing to lane departure control.

(A4-2) <Determination method> The right lane mark is a solid line, and there is a lane mark next to the lane mark that seems to be an oncoming lane, and another vehicle that reverses the traveling method from the host vehicle 1 passes. In this case, it is determined as an oncoming lane. Instead of determining from the traveling direction of the other vehicle, car navigation information or sensor information may be used.

  The type determination contents are classified into the following seven types (321) to (327) according to types related to application switching as types indicating roadside.

(321) Convex shoulder judgment part “Sidewalk, Convex shoulder”
(321-1) <Concept of reflection in priority> There is a high possibility that a pedestrian is walking, and the pedestrian is likely to jump out on the road. There is a high possibility that a sign is present, the vehicle does not travel, and there is no road sign.

(321-2) <Determination method> The type determination method on the left side will be described. If a lane mark cannot be detected from the lane mark recognition processing area L2 in FIG. And check if there is a convex step that separates the sidewalk.
First, although there is no standard that the step feature amount extraction unit 321A is white compared to the road surface like the white line, the step portion has the same edge as the road surface from the same height as the road surface and the edge from the rising portion. Two two edge portions that change in a flat direction are extracted while searching in the horizontal direction on the image. If a sidewalk step appears on the image, it is assumed that a step with the same height will continue from the vicinity of the vehicle to the distance. A straight line having a width of about 5 to 30 cm is extracted by the step straight line extraction unit 321B. In the first place, if two edge pairs with different widths on the image are found, or if they do not become straight lines, it is determined that the shoulder is not a convex road, and those with a certain width on the image gather, and in real space When the width is equal and linearity is recognized, it is determined as a convex shoulder.

(322) Concave shoulder determination section “Concave shoulder, gutter, field, valley, river”
(322-1) <Concept of reflection in priority> Pedestrians are not visible. The vehicle does not run and there are no road markings. When the vehicle approaches, there is a danger such as derailment.

(322-2) <Determination method> It is determined whether or not there is a linear edge that becomes darker than the traveling road. The concave feature amount extraction unit 322A searches in the horizontal direction on the screen, searches in the direction away from the own vehicle side, and searches for the edge of the concave portion that becomes dark. Thereafter, in the concave line extraction unit 322B, it is determined whether or not the edges are arranged in a straight line, thereby performing the concave shoulder determination.
Judgment is made based on whether a ditch such as a gutter or a field continues along the running direction. The height of the road surface may be recognized by a stereo camera.

(323) Structure determination unit "Buildings, fences and walls"
(323-1) <Concept of reflection in priority> Pedestrians are not visible. There is a high possibility that a sign is present in the foreground, the vehicle does not travel, and there is no road sign. There is a danger of collision when the vehicle approaches.

(323-2) <Determination method> If it is an artificial structure, there is a high possibility of standing perpendicular to the road surface, and the structure is determined by finding a group of vertically upward straight lines.
In the three-dimensional object feature extraction unit 323A, in order to find the edge of the artifact (three-dimensional object) that stands vertically upward with respect to the road surface, the three-dimensional object feature extraction unit 323A searches the edge extending vertically upward in the horizontal direction. To implement. Next, in the three-dimensional object lower end straight line extraction unit 323B, first, an edge group extending in the vertical direction is found and its length is set. Those that can only be regarded as straight lines that are too short are eliminated as noise factors, and those that can be confirmed with a certain degree of edge in the vertical direction even if they are halfway are determined to be edge groups of a three-dimensional object. In order to determine only the edge group lined up in the vertical direction from which position on the road surface, the lower end of the three-dimensional object is relative to the traveling direction of the host vehicle as shown in FIG. A straight line extraction is performed to determine how they are lined up.
When the lower end is obtained, the three-dimensional structure is analyzed in the fluidity determination unit 323C on the assumption that the artificial structures may be arranged along the road. It is checked whether the position of the result of the three-dimensional object lower end straight line extraction unit 323B and the result of now frm are substantially equal over the past several frames. Furthermore, when the position of the edge that is linearly aligned in the vertical direction of 1 frm and the position on the screen 1 frm ahead are predicted using the own vehicle speed movement distance from the current frame to 1 frm, the vertical linear position is almost the same The three-dimensional material fluidity determination unit 323C analyzes whether or not to do so.

(324) Periodic object judgment section "Guard rail, fence, wall"
(324-1) <Concept of reflecting on priority> The possibility that a pedestrian is walking is high, but the possibility of jumping out directly is low. There is a high possibility that a sign is present, the vehicle does not travel, and there is no road sign. There is a risk of collision when the vehicle approaches.

(324-2) <Determination method> A demographic structure such as a guardrail that has periodicity is determined. The periodicity is analyzed by tracking the movement of the artificial structure on the image using an image whose period is shifted to determine the periodic object.
Since the result of the three-dimensional object feature amount extraction unit 323A and the result of the three-dimensional object lower end straight line extraction unit 323B are also used in the structure determination unit 323, 323A is also 324A in FIG. Are attached. Actually, there is only one process, and the result data is also used by the structure determination unit 324, the non-artificial object determination unit 325, and the travelable road non-determining unit 326. Here, as preprocessing in the structure determination unit 323, the results of the three-dimensional object extraction unit 324A (323A), the three-dimensional object lower end straight line extraction unit 324B (323B), and the three-dimensional object fluidity determination unit 324C (323C) are used.
Using these results, the periodicity determination unit 324D determines whether or not the interval between the vertical straight line positions is the same in terms of distance and space, and is a structure and has periodicity. Is determined as a periodic object, and if it is a structure but has no periodicity, it is determined only as a structure.

(325) Non-artifact determination part “planting, grass, snow wall, rock, non-artifact”
(325-1) <Concept of reflection on priority> When there is a non-artificial structure, there is a possibility that a pedestrian walks outside the white line without a shoulder in places where the vehicle speed is low. The vehicle does not run. There may be road markings and there are no road markings outside the road.

(325-2) <Determination method> Although it is determined that there is no periodicity in the periodic object determination, and in the structure determination, it is determined that there is no three-dimensional object, but the three-dimensional object feature amount extraction unit 323A (325A) ) And the solid object bottom straight line extraction unit 323B (325B) determines that most of the noise is a cause of noise. The random texture determination unit 325C determines whether the texture has a higher randomness by texture analysis and determines whether the texture is a non-artifact.

(326) Non-travelable road determination section “Outside of driveable road”
(326-1) <Concept of Reflecting to Priority> Although there is no travel lane, a sufficient space for the vehicle to travel to the shoulder is prepared. In this case, even if the vehicle travels slightly beyond the vehicle lane, the possibility that the vehicle will collide with something is low.

(326-2) <Determination Method> There are almost no textures on the region, and there are no edge components that are arranged in a straight line. Further, the determination is made by utilizing the fact that the road surface luminance distribution is substantially the same as that of the vehicle lane.
The determination is performed only when it is determined from the results of the structure determination unit 323, the periodic object determination unit 324, and the out-of-travel determination unit 326 that there is no structure. There is an edge component in the three-dimensional object feature extraction unit, but in the case where the linearity in the vertical direction is not recognized in the three-dimensional object bottom straight line extraction unit, there is a high possibility that the edge component exists but is not a three-dimensional object. As described above, the road surface fluidity analysis unit 326C determines whether the texture on the road surface moves away at the vehicle speed, and determines whether the road surface is out of the road.

(327) Lane Decrease Determination Section “Before Lane Decrease”
(327-1) <Concept of reflecting on priority> The lanes in the traveling direction are integrated into one lane due to an accident or construction.

(327-2) <Determination method> When the type determination method on the left side is described, a lane mark can be detected from the lane mark recognition processing area L2 shown in FIG. Integrated with.

  As described above, the out-of-road type determination is used for reference information for adjusting the priority of the application.

Next, the lateral position analysis unit 330 in the out-of-road condition determination unit 300 will be described.
Components of road type determination unit 320, convex shoulder determination unit 321, concave shoulder determination unit 322, periodic object determination unit 323, structure determination unit 324, non-artifact determination unit 325, travelable road outside determination unit 326, lane The reduction determination unit 327 has already performed straight line extraction on the screen in each determination unit, and the convex shoulder determination unit or the like obtains a straight line in three-dimensional world coordinates in advance. In the lateral position analysis unit 330, when the classification outside the road is performed, the vertical position of the outdoor thing in the current frame in the current frame while using the result from the past, in the example of FIG. If indicated, the distance d is estimated. In some cases, not only the horizontal position but also the yaw angle may be obtained as in the lane recognition.

An example using the result of road type and its lateral position will be described below.
FIG. 11 is a diagram for explaining an application change (lane departure warning and lane departure control) by roadside lateral position analysis. As shown in FIG. 11A, for example, when the guard rail 51 exists on the left side of the host vehicle 1, it is analyzed how far the guard rail 51 exists from the host vehicle travel lane 21. Specifically, the distance d in the road width direction from the vehicle lane 21 to the guardrail 51 is detected.

  Further, as shown in FIG. 11B, even if this is a sidewalk / shoulder 52, the lateral position is similarly analyzed. In addition, when the guardrail 51 and the sidewalk / shoulder shoulder 52 are parallel to the host vehicle travel lane 21 or when they are not parallel, the inclination thereof may be analyzed.

Using these pieces of information, the priority and setting of each of the plurality of applications are changed.
For example, the embodiment of application switching in FIG. 11 will be described. In FIG. 11A, the right adjacent lane 41 of the host vehicle 1 is a normal traveling lane. The left side of the host vehicle 1 is outside the road (guard rail), and a guard rail 51 exists at a lateral position 20 cm away from the left lane mark of the host vehicle lane on the left side in the road width direction. In this way, when an obstacle such as an off-road obstacle exists at a short distance from the lane, there is a high possibility that the own vehicle 1 will come into contact with the guard rail 51 as soon as the vehicle 1 departs from the lane, which may lead to an accident.

  Therefore, if the guardrail 51 is in a lateral position about 3 meters away from the left lane mark 22 on the left lane of the own vehicle lane (d is large), safety should be maintained even if it is a lane departure warning. However, for example, when the guardrail 51 is present at a lateral position of 0.5 meters from the left lane mark 22 (d is small), it is possible to maintain safety by performing the lane departure control. I think it is highly possible.

  The lane departure warning and control timing are also changed in consideration of the lateral position of the guardrail 51 and the lateral speed of the vehicle behavior. In this way, even if the guardrail 51 is the same, by analyzing the lateral position, if there is a high possibility that safety will be maintained, consideration should be given so that the driver can freely drive in the lane without interfering with the driver more than necessary. Preventive safety can be implemented.

  In a lane position where the possibility of contact is high, such as when the guardrail 51 is very close to the lane position, the application to be implemented is changed, and in this embodiment, the lane departure warning is switched to the lane departure control. Thus, the safety of the host vehicle 1 is ensured.

  In addition, the information for determining the change of the application may include not only information on the road but also information on the vehicle behavior. If the vehicle speed of the host vehicle 1 is fast and the yaw angle is generated, if the vehicle behavior does not change as it is, it is estimated that it will take L seconds to depart from the lane, and then contact the guardrail after G seconds. Based on this, the lane departure warning and the lane departure control may be changed, or the warning and control execution timing may be changed.

  As for the same effect, the timing of the lane departure warning may be changed according to the distance of the lateral position. If the distance to the detected obstacle is far, it is estimated that the alarm was issued immediately before the departure from the lane 22; if the distance to the obstacle is relatively close, it is estimated that the departure from the lane 22 is 1 second. Even if it is used to adjust the alarm timing, such as sounding an alarm before, it can be expected to increase safety.

  Considering the same for lane departure control alone, the start timing of lane departure control is usually normal when the distance is large according to the distance of the lateral position, and safety is improved even if the control timing is advanced when the distance is short. I can expect that. In addition, the strength of the braking force pushed back into the lane of the lane departure control may be normal when the distance of the lateral position is long, and the control force may be increased when the lateral position is a short distance.

Next, an embodiment for changing the priority of the vehicle detection and pedestrian detection applications will be described.
FIG. 12 is a diagram illustrating application change (vehicle detection and pedestrian detection) when the left road is shouldered. As shown in FIG. 12A, the left lane mark 22 and the right lane mark 23 are recognized on both the left and right sides of the host vehicle lane 21, and the adjacent left lane mark 32 and the adjacent right lane mark 42 are further recognized on the outer side. In this case, there is a high possibility that both the left and right sides of the own vehicle travel lane 21 are travel lanes (adjacent lanes). For this reason, vehicle detection is carried out in each of the left and right lanes of the vehicle lane. On the other hand, when it is recognized that the left side is a road shoulder as shown in FIG. 12B, vehicle detection is not necessary. However, there are exceptions such as how to handle the road outside the driveable road, and detecting an approaching vehicle that is merging when running the merging before the lane decrease. If the reliability of being outside the road is high, the vehicle detection itself may be completely stopped, and the side may perform pedestrian detection or the like. In particular, since there is a high possibility that a person is walking on a sidewalk or the like on a convex shoulder or the like, the priority of pedestrian detection is increased and the priority of vehicle detection is decreased. On the other hand, if the vehicle speed is low and it is a non-artificial road shoulder, there is no possibility that there is a step for the sidewalk and the outside of the white line is a road for pedestrians as it is. The priority of person detection needs to be increased to some extent. On the other hand, when the vehicle speed is high and there are periodic objects or structures, there is a high possibility that the vehicle is traveling on a highway or the like, and it is not always necessary to increase the priority of pedestrian detection. In North America, etc., there is a high possibility that the shoulders on the concave road are FreeWay, and pedestrian detection does not always need to be given a high priority.

Moreover, the Example of the false alarm suppression of a vehicle detection application is shown.
In detecting a vehicle approaching to the rear side of an adjacent lane, it is used to suppress erroneous detection by determining whether the adjacent lane is a shoulder. In the vehicle detection method, in the method using an optical flow that captures the movement of a feature point, there is a high possibility of erroneous detection in the case of out-of-road determination, particularly in the case of periodic object determination and non-artifact determination. For this reason, when it is determined that the vehicle is out of the road, in the case of the determination of the two periodic objects and the non-artifact, the vehicle detection process itself is not stopped, but if it is erroneously detected, it is presented to the user. The information is the same as the result in the case of no detection. That is, a false alarm is suppressed in a system that sounds an alarm when there is an approaching vehicle. In addition, in the case of out-of-road determination, since the possibility of erroneous detection of vehicle detection is low, the vehicle detection threshold for determining that the vehicle has been detected in response to the out-of-road determination result is increased, and the accuracy is high. A response method that considers only erroneous detection of the out-of-road judgment itself, with only the vehicle as the detection target. Regarding vehicle detection, when there is an obstacle outside the road in the adjacent lane, the probability of erroneous detection of a three-dimensional object as a vehicle increases. For this reason, when it can be recognized that the vehicle is outside the road, the vehicle detection processing outside the road suppresses an alarm used for the purpose of lane change support or the like.

  According to the on-vehicle environment recognition device having the above-described configuration, the lane mark that divides the lane is recognized from the image, and the out-of-road determination area is set on at least one of the left and right sides of the own vehicle lane based on the lane mark. It is determined whether the outside determination area is an adjacent lane or outside the road, and when it is determined that the outside determination area is outside the road, a process of analyzing the type outside the road is performed.

  Therefore, it is possible to acquire information on the environment outside the vehicle necessary to execute multiple applications, control the priority, processing time, and application change timing of multiple applications, and change the application execution state of applications for each area can do. Therefore, it is possible to implement an appropriate application according to the surrounding environment of the vehicle that changes according to the traveling of the vehicle, and to provide a highly convenient and safe in-vehicle environment recognition device within the limited capability range of the CPU. Can be realized.

10 Automotive environment recognition system
100 Imaging unit
200 Lane mark recognition unit
210 Lane mark recognition area setting section
220 White line feature calculation unit
230 White line extraction unit
240 3D position analysis
300 Road condition judgment part
310 Road determination area setting section
320 Road type classification part
330 Lateral position analysis unit
400 Application control unit
410 Priority determination unit
420 Processing time prediction unit
430 Application setting section
500 app execution part
520 Lane departure warning section
530 Lane departure control unit
540 Lane Change Support Department
550 Pedestrian detection unit
560 Vehicle detector
570 Sign detector
580 Light distribution controller
590 Road surface sign detector

Claims (3)

An on-vehicle environment recognition device that recognizes the environment outside the vehicle based on an image captured by an in-vehicle camera,
A lane mark recognition unit for recognizing a lane mark that divides the road surface from the image;
An out-of-road determination area setting unit that sets out-of-road determination areas on the left and right sides of the travel lane based on the lane mark;
A road that determines whether the type of the outside road determination area is an adjacent lane or outside the road, and determines the type outside the road when the type of the road outside determination area is determined to be outside the road. An outside state determination unit;
Application setting for selecting at least one or more applications adapted to the type of road determination area and the type of road outside of the plurality of applications for each road determination area, and setting the priority of the selected application And
An application execution unit that executes a plurality of applications selected by the application setting unit based on the priority;
An on-vehicle environment recognition device comprising:   The application setting unit, when the out-of-road condition determination unit determines that the type of the left and right out-of-road determination area of the host vehicle is an adjacent lane, the lane for the left and right lane marks from among a plurality of applications A departure warning and a vehicle detection application for the traveling lane of the own vehicle and the left and right adjacent lanes are selected, and the priority of each application is set in the order of lane departure warning and vehicle detection from the highest priority. The in-vehicle environment recognition device according to claim 1. The application setting unit determines that the type of the road outside determination area on the left side of the host vehicle is outside the road by the road outside state determination unit , and determines that the type outside the road is a road shoulder. Lane departure control for the left lane mark, lane departure warning for the right lane mark, vehicle detection for the vehicle's driving lane and the right adjacent lane, and walking on the shoulder from among a plurality of applications. The vehicle-mounted vehicle according to claim 1, wherein each of the applications for detecting the pedestrian is selected, and the priority of each application is set in the order of lane departure control, lane departure warning, vehicle detection, and pedestrian detection from the highest priority. Environment recognition device.

Images