JP2022041437A - Image processing device - Google Patents

Abstract

To provide an image processing device which outputs information required for control of an own vehicle or an alarm by sensing an ambient environment of the own vehicle utilizing a sensor in order to more safely and comfortably control the vehicle.SOLUTION: An image processing device of the present invention is configured to detect a road end of a travel road on the basis of a picked-up image of an on-vehicle camera, recognizes the type of an object constituting the road end from the picked-up image and changes a parameter for detecting the road end in accordance with the type of the object.SELECTED DRAWING: Figure 24

Description

The present invention relates to an image processing device that recognizes the surrounding environment of a vehicle by utilizing a sensor mounted on the vehicle.

Patent Document 1 describes a technique for detecting a road shoulder using a sensor and using it for preventive safety and automatic driving.

Japanese Unexamined Patent Publication No. 2009-053818

In recent years, warnings and control functions related to preventive safety have entered a period of widespread use. In order to realize this preventive safety, the development of sensing functions for predicting accidents is accelerating. Currently, many of the functions that are widely used are preventive safety functions on expressways, driving assistance, and emergency braking to prevent collisions on general roads. Among them, the roadside deviation prevention function by roadside detection, which is one of the preventive safety functions, has not yet reached the popularization period due to the high difficulty of roadside sensing.

Unlike lanes that are artificially painted to indicate a section of the road, the roadside does not necessarily have artificial walls or sidewalk steps. For example, in the case of a highway, most of the roadsides are provided with artificially maintained walls, and the curves are often continuous and clean, if not as much as lanes. Recognition is relatively easy. On the other hand, in the case of general roads, obstacles such as utility poles, rocks, and grass may have jumped out onto the road, making it difficult to model in advance a curvature shape that makes it easy for vehicles to travel. The difficulty of sensing to detect the roadside is high. Further, on a general road, not only the shape of the roadside but also the break of the roadside and the type of the roadside are often mixed, and there are many scenes in which it is difficult to recognize the roadside.

On general roads, structures (objects) such as fences, guardrails, and sidewalks are often connected continuously even if the shape in the direction along the travel path is not a slightly smooth curve. However, even with these structures, there is a discontinuity at the entrance to the store or home, such as interruptions or obstruction by other obstacles, and the type of roadside is immediate. May be changed to.

Also, on the side of the road, there are lawns and gravel that are very short and there is no difference in height from the road surface, areas of unpaved soil, or from the road surface such as gutters, rice fields, fields, etc. There are also low-height areas, and the boundary between these areas and the road is also the roadside. Not only are there various types of areas in this way, but there are also various things such as utility poles, obstacles, grass and gutter lids, and trees on the roadside of the road, which are more discontinuous. Proper recognition is an important issue.

The present invention has been made in view of the above problems, and an object of the present invention is to sense the surrounding environment of the own vehicle by using a sensor in order to control the vehicle more safely and comfortably, and to control the own vehicle. An object of the present invention is to provide an image processing device that outputs information necessary for an alarm.

The image processing device of the present invention is an image processing device that detects the roadside of a traveling road based on an image captured by an in-vehicle camera, recognizes the type of an object constituting the roadside from the captured image, and recognizes the object. It is characterized in that the parameter for detecting the roadside is changed according to the type of the roadside.

According to the present invention, even in an environment where a plurality of types of roadsides exist discontinuously on a general road or the like, the characteristics of the same type of roadsides can be appropriately connected and recognized appropriately. It is possible to detect the roadside stably and accurately, and to improve the accuracy of the road deviation prevention function.

Further features relating to the present invention will be apparent from the description herein and the accompanying drawings. In addition, problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

The figure explaining the whole structure of the image processing apparatus which concerns on this embodiment. The figure explaining the internal structure of a sensor part. The figure explaining the internal structure of the roadside type characteristic part. The figure explaining the internal structure of the roadside feature amount extraction part. The figure which shows the specific example of a roadside type schematically. The figure which shows the captured image schematically and the figure which shows the image which extracted the roadside cumulative feature amount. The figure explaining the feature amount extraction method of a low step roadside and a minus roadside. The figure explaining the feature amount extraction method of the roadside without a step and the roadside division roadside. The figure explaining the internal structure of the own vehicle behavior estimation part. The figure explaining the example of own vehicle behavior estimation. The figure explaining the example of self-position estimation. The figure explaining the internal structure of the roadside feature map generation part. The figure which shows an example of the roadside characteristic map. The figure which shows the map which applied the inner feature priority processing to the roadside feature map of FIG. The figure explaining the internal structure of the traveling path determination part according to the type. The figure which shows the connection roadside candidate generation condition. The figure which shows the map which generated the connection roadside candidate from the map which performed the inner feature priority processing shown in FIG. The figure which shows the roadside candidate connection condition. The figure which shows the connection example (1) of a plurality of kinds of roadsides. The figure which shows the connection example (2) of the roadside including a vehicle. The figure which shows the connection example (3) of the roadside including the travel road division. The figure explaining the internal structure of an alarm control part. The figure which shows the information of the warning and the control margin for the roadside type. A flowchart illustrating the processing executed by the image processing apparatus.

Hereinafter, examples of the present invention will be described with reference to the drawings.

<Image processing device>
FIG. 1 is a diagram showing an overall configuration of an image processing apparatus according to the present embodiment.
The image processing device 1 includes a sensor unit 100, a roadside type characteristic unit 200, a travel path determination unit 300, and an alarm control unit 400.

The sensor unit 100 has a sensor that detects the environment around the vehicle such as obstacles and the shape around the traveling path. The sensor unit 100 has an in-vehicle front sensor. In this embodiment, as shown in the upper part (1) of FIG. 2, the left and right camera units (stereo cameras) 110 are described as an example of the in-vehicle front sensor, but the sensor itself is as shown in the middle part (2) of FIG. The camera unit 150 may be a single unit, or may be a fusion sensor of the camera unit 170 and the lidar unit 180 as shown in the lower part (3) of FIG. Further, the sensor unit 100 does not necessarily have to have a sensor, and may be configured to acquire detection signals from various sensors provided in the vehicle.

The roadside type feature unit 200 extracts the feature amount (roadside feature amount) according to the roadside type. The roadside is formed between the road and the object, and the type of the roadside differs depending on the object. The roadside type feature unit 200 extracts a roadside feature amount for each type of object, and determines the roadside type using the extracted roadside feature amount. Since there are various types of boundaries that divide the travelable area that indicates whether or not the vehicle can travel, that is, the roadside of the road, it is necessary to extract the features of different types of roadside by the same search method. It was difficult in itself. For example, a tall obstacle compared to the road surface of the road, a region lower than the road surface of the road, or a road surface having different properties from the road surface of the road (for example, the road surface has no difference in three-dimensional height). For asphalt, it was difficult to extract the boundary between an object such as turf, gravel, soil, etc. and the roadway using the same image processing algorithm as the roadside.

On the other hand, in the present embodiment, the roadside type feature unit 200 employs a method in which a plurality of types of extraction means are used in combination and a plurality of roadside feature amount extraction algorithms are used in combination as shown in FIG. The roadside type feature unit 200 performs a process of integrating each feature amount of the extracted plurality of types of roadsides in the same space (on the map), and also performs noise reduction. The roadside type feature unit 200 creates a roadside feature map based on the extracted roadside feature amount.

The travel path determination unit 300 uses the roadside feature map created by the roadside type feature unit 200 to perform travel path determination according to the roadside type. Since the roadside feature unit 300 extracts the roadside feature amount for each roadside type in the roadside type feature unit 200, the roadside feature map is created using the roadside feature amount. Then, the roadside candidates are extracted from the roadside feature map, the roadside candidates are connected for each type, and the connected roadside candidates are used to further select roadside candidates having different types. Determine whether to concatenate.

Basically, it is highly possible that the roadside features of the same type are lined up, but the frequency of occurrence of the continuity differs depending on the type of roadside features. For example, in the case of a roadside pole indicating a roadside or a triangular pole indicating a traveling road, they are not connected to each other and are not lined up continuously in the first place. For this reason, the markers themselves indicating the track divisions are rarely arranged continuously, and in these cases, they are arranged on the track with a large distance from each other, so that the characteristics of these types are characteristic. , It is necessary to judge the driving route division by connecting up to a considerably long distance.

The alarm control unit 400 gives an alarm to the driver under a predetermined situation, and controls the vehicle when the situation is not improved. The warning control unit 400 first emits an alarm sound or displays an alarm when the vehicle approaches the roadside, which is the boundary of the travel path, and is likely to deviate from the travel path, according to the determination result of the travel path determination unit 300. It was determined that the driver would be alerted, and if the situation that was likely to deviate continued, or if the distance to the road boundary was extremely short, or if the vehicle was not steered as it was, the vehicle would deviate from the road after a certain period of time. In that case, vehicle control of steering and braking is performed to prevent the vehicle from deviating from the driving path.

<Sensor unit 100>
FIG. 2 is a diagram illustrating the configuration of the sensor unit.
In this embodiment, as shown above, an embodiment assuming a stereo camera is basically described. However, as shown above, as shown in the middle row (2) of FIG. 2 and the lower row (3) of FIG. 2, the sensor unit 100 may be a monocular camera having a camera unit 150, and the camera may be a camera. It may be a Fusion sensor in which the unit 170 and the Lidar unit 180 are combined, or it may be another TOF sensor, a Fusion sensor of a millimeter wave and a camera, or the like.

In this embodiment assuming that the sensor unit 100 in the upper part (1) of FIG. 2 is a stereo camera, the left and right camera units 110 acquire images captured by the left and right cameras, correct the geometry and sensitivity, and then perform left and right. Parallelization, which is a geometric correction necessary for stereo-matching the captured image of the camera unit 110, is performed, and sensitivity calibration is performed so that the hue, brightness, and the like of the left and right cameras are the same. After calibrating the sensitivity and geometry, the stereo matching unit 120 performs stereo matching of the left and right captured images to generate a parallax image. The 3D point cloud generation unit 130 generates a three-dimensional point cloud from this disparity image by using the internal parameters of the camera.

<Roadside type feature unit 200>
FIG. 3 is a diagram illustrating the configuration of a roadside type feature section.
The roadside type feature unit 200 includes a roadside feature amount extraction unit 210, a vehicle behavior estimation unit 220, a roadside feature map generation unit 230, an inner feature priority unit 240, and an inner feature type determination unit 250. is doing.

The roadside type feature unit 200 acquires the feature amount of the roadside to be used for determining the area in which the own vehicle can travel. In particular, in the roadside feature amount extraction unit 210, three-dimensional information of the stereo camera and an image captured by the camera are obtained from the position where objects such as obstacles, gutters, and road shoulder blocks that are boundaries with the road, which is a travelable area, exist. The roadside features are extracted using the information in. The extracted roadside features include roadside type information and location information.

In the roadside feature amount extraction unit 210, these roadside feature amounts are extracted for each result of a single frame imaged by the sensor unit 100, that is, for each frame. However, it is possible to extract a stable and smooth boundary by determining the dividing line of the traveling road together with the roadside feature amount extracted in the past, rather than determining by only a single frame. That is, from the viewpoint of a vehicle moving forward, it is better to determine the boundary of the travel path division including the road edge behind the own vehicle.

Therefore, in order to arrange the extracted roadside features in chronological order to obtain the roadside features on the map, the own vehicle behavior estimation unit 220 estimates the behavior of the own vehicle, and the roadside feature map generation unit 230. , Based on the estimated result, the roadside features obtained for each frame are arranged on the map and voted. As a result, the roadside features obtained in each frame are arranged on the bird's-eye view seen from above the own vehicle, and the roadside features are arranged as if they were a map showing the boundary with the traveling road. At this time, vote on the map including information on the types of features. That is, a process is performed in which information on the type of the roadside feature amount is also associated and stored at the point where the roadside is located on the map.

However, at this point, the roadside features obtained as noise are also arranged on the map. For this reason, on the map, first, the feature amount of the object closest to the lateral direction (road width direction of the travel path) from the position where the own vehicle travels, that is, the innermost object is preferentially held, and the feature amount is held outside it. The inner feature priority unit 240 performs a process of reducing the feature amount of other objects overlapping (outside in the road width direction) from the map. In the roadside region, this is an object composed of taller walls, trees, buildings, etc. that exist outside the road width direction than objects such as road shoulder blocks or gutters, walls that exist inside the road width direction. By eliminating the roadside feature, the roadside composed of the innermost object in the road width direction, that is, the object closest to the vehicle, can be determined more accurately and without being affected by noise. ..

Next, regarding the type information of the roadside feature amount voted above the threshold value on the map, the right roadside and the left roadside of the own vehicle are processed separately, and the type of the roadside feature amount arranged on the left and right of the own vehicle is processed separately. Is specified by the inner feature type determination unit 250. At this time, there may be a case where a plurality of types of features are mixed according to the depth. Basically, the main feature types are specified, or multiple feature types to be mixed are specified, and the types with extremely small features are regarded as noise.

<Roadside feature amount extraction unit 210>
FIG. 4 is a diagram illustrating the configuration of the roadside feature amount extraction unit. The roadside feature amount extraction unit 210 includes a three-dimensional object accumulating portion 211, a moving body portion 212, a low step portion 213, a minus step portion 214, a stepless roadside portion 215, and a travel path division three-dimensional object portion 216.

The roadside feature amount extraction unit 210 changes the parameters for detecting the roadside according to the type of the object. Here, various types of roadside features are extracted according to the type of object. FIG. 5 is a diagram schematically showing an example of a roadside type in a cross section. FIG. 5 (1) shows a three-dimensional object in which a gutter 502 exists on the right side of the roadway (carriageway) 501 in the road width direction, and a tall fence is provided at the roadside on the left side in the road width direction from the center of the roadway 501. An example is shown in which the cumulative area 503 and the sidewalk 504 are further on the left side in the road width direction of the three-dimensional object cumulative area 503.

In the example shown in FIG. 5 (1), the boundary between the traveling path 501 and the gutter 502 and the boundary between the traveling path 501 and the three-dimensional object cumulative region 503 are the roadsides, respectively. Further, in FIG. 5 (2), a sidewalk 514 is provided on the left side of the travel path 511 in the road width direction via a relatively low step, and a convex road shoulder block 512 is arranged on the right side of the travel path 511 in the road width direction. Further, an example is shown in which the sidewalk 513 is arranged on the right side thereof. In the example shown in FIG. 5 (2), the boundary between the runway 511 and the sidewalk 514 and the boundary between the runway 511 and the road shoulder block 512 are the roadsides, respectively.

Then, in FIG. 5 (3), a stepless region 522 in which grass, gravel, soil, etc. are continuous at the same height as the road surface of the road 521 is arranged on the left side of the road 521 in the road width direction, and the road 521 has a stepless region 522. On the right side in the road width direction, an example is shown in which a minus step region 523, such as a rice field, a field, or a lowland, is present between the road and the road 521 via a step that is lower than the road surface of the road 521. In the example shown in FIG. 5 (3), the boundary between the traveling path 521 and the stepless area 522 and the boundary between the traveling path 521 and the minus stepped area 523 are the roadsides, respectively.

For example, as shown in FIG. 5 (1), the three-dimensional object accumulation unit 211 extracts a feature amount for a three-dimensional object that is taller than the road surface of the traveling path 501. The three-dimensional object accumulating portion 211 can extract a stable feature amount by accumulating more features as the height of the three-dimensional object is higher, that is, the higher the road surface height is. FIG. 6 (1) is an original image captured by a stereo camera, and FIG. 6 (2) is a cumulative feature extraction image created by using a parallax image. The three-dimensional object accumulation unit 211 generates a parallax image using a pair of left and right original images, and performs a cumulative roadside feature amount extraction process for the parallax image of the traveling road. In the original image 601 shown in FIG. 6 (1), the preceding vehicle 602 is imaged, and the road shoulder blocks 604 and 605 and the walls 606 and 607 are arranged on the left and right sides of the traveling path 603.

In the roadside feature amount extraction unit 210, as shown in FIG. 6 (2), the parallax value indicates the image horizontal coordinates in the horizontal direction and the depth in the vertical direction. Generates an image with a small parallax value at a long distance. Then, the voting process is performed for each row of abscissa of the parallax image. For example, considering the center of the image in which the own vehicle exists, the lower side of the parallax image is the parallax value on the road surface, so if one parallax value is read from the lower side of the parallax image, it gradually becomes a long-distance parallax value and the voting place Is the result of extracting the cumulative roadside feature amount that gradually moves upward.

However, when the parallax image reaches the foot position of the preceding vehicle, the parallax value from the foot position of the preceding vehicle to the roof position has a constant depth distance. Therefore, in the cumulative roadside feature amount extraction image, votes for the cumulative roadside feature amount are gathered at the same parallax value and the same image abscissa, resulting in a high voting value. This also has similar results for shoulder blocks and walls.

The road surface on the parallax image becomes deeper toward the upper side of the parallax image, so that the feature quantities are not accumulated and are voted for dispersed places. However, when approaching the position of a tall object such as a road shoulder block or a wall, the same parallax value is continuously obtained in the upward direction on the parallax image, so that the parallax value and the image horizontal coordinates are on the voting space. As a result, the parallax values are accumulated. Therefore, in this embodiment, it is assumed that the more the parallax values are accumulated in this voting space, the more obstacles such as walls and shoulder blocks are higher than the road surface. In the example shown in FIG. 6 (2), more parallax values are accumulated at the positions of the left and right walls 616 and 617, and are clearly shown in white. The parallax values are also accumulated and shown in white at the positions of the road shoulder blocks 614 and 615 and the foot positions of the preceding vehicle.

Among the feature quantities obtained in the three-dimensional object accumulating portion 211, there are also feature quantities relating to the side surface of the moving body. For example, when there is a vehicle traveling in an adjacent lane as a moving body, the feature amount is accumulated like a wall when considering the processing of a single frame, so that the feature amount can be extracted for this moving body as well. be. By using this extracted feature amount together with the result of vehicle detection, the moving body unit 212 identifies that it is a feature amount for a moving body, and as a feature amount of a type different from that of the three-dimensional object cumulative part 211. deal. That is, the moving body unit 212 performs a process of extracting the feature amount of the moving body such as the preceding vehicle and the oncoming vehicle.

In the low step portion 213, the three-dimensional object accumulation portion 211 mainly performs a process of extracting the feature amount of a low step of about 5 cm, which is difficult to extract the feature amount. As shown on the left side of FIG. 5 (2), the low step sidewalk 514 has a small difference in height as compared with the road surface of the traveling path 511, so that the feature amount extracted by the three-dimensional object accumulating portion 211 is cumulative. The amount is small. Therefore, the low step portion 213 exclusively performs a process of extracting the feature amount of the low step.

FIG. 7 is a diagram illustrating a method for extracting features of a low step road edge and a minus road edge. For example, as shown by the arrow 711 on the left side of FIG. 7, the parallax image 701 is searched to the left from the center position 702 of the traveling path 703. The three-dimensional position is calculated from the parallax value and the image position, and it is compared whether a high step is generated compared with the height of the road surface of the traveling road 703. The extracted portion is extracted as a roadside feature amount between the traveling road 703 and the traveling road 703.

Similarly, in the minus step portion 214, a step lower than the road surface of the traveling road 703, for which the feature amount could not be extracted by the conventional method, is extracted. The stereo camera of the sensor unit 100 can detect a step lower than the road surface of the traveling road 703 such as a gutter 705. The minus step portion 214 extracts the feature amount of the minus step region 523 lower than the traveling path, for example, the gutter 502 shown on the right side of FIG. 5 (1) and the rice field or the field shown on the right side of FIG. 5 (3). .. For example, as shown by the arrow 712 on the right side of FIG. 7, the parallax image is searched to the right from the center position 702 of the traveling path 703. The three-dimensional position is calculated from the parallax value and the image position, compared with the height of the road surface to determine whether a step lower than the road surface is generated, and the feature amount lower than the road surface of the traveling road 703 is continuously lower than the threshold value. The extracted portion is extracted as a roadside feature amount between the traveling road 703 and the traveling road 703. In the example shown in FIG. 7, the feature amount of the sidewalk 704 that is higher than the road surface but lower than the road surface can be extracted by the low step portion 213, and the feature amount of the gutter 705 lower than the road surface can be extracted by the minus step portion 214. can.

FIG. 8 is an image diagram illustrating a method for extracting features of a roadside without a step and a roadside with a travel path division. As for the roadside without steps, as shown on the left side of FIG. 8, substances different from the roadway 803, such as grass, gravel, and soil, although the height is the same as the roadway 803A on the left side of the paved road. The boundary with the stepless region 805, which is not suitable for traveling, is extracted as the roadside feature amount of the stepless roadside. In order to identify the stepless region 805, basically, machine learning such as Deep Learning is used to extract a travelable road surface region (travel road 803A) and a non-travelable stepless region 805, and this The boundary is extracted at the roadside portion 215 without a step as a roadside feature amount.

Finally, with respect to the traveling path division three-dimensional object, as shown on the right side of FIG. 8, there is a case where triangular poles are temporarily arranged for the purpose of restricting the traveling of the vehicle due to reasons such as during construction. In addition, there are cases where poles are permanently installed, such as in parking lots and intersections before entering stores, for the purpose of restricting the direction of vehicle movement. In particular, since the triangular poles for construction work cannot be provisionally implemented, it is difficult to divide the area where the triangular poles can be traveled and the area where the construction is to be disabled due to the fact that the triangular poles are far apart from each other and the number of installations is small.

However, with the stereo camera, the triangular poles arranged on the travel path 803B can be extracted as a three-dimensional object. For this reason, if the size is within the range of a general triangular pole after being detected as a three-dimensional object, this is a triangular pole that is used to restrict the movement of the vehicle by the discriminator. Perform identification. When it is identified as a traveling road division three-dimensional object for controlling the movement of the vehicle, it is extracted by the traveling road division three-dimensional object unit 216 as a roadside feature amount.

The roadside feature amount extraction unit 210 adds type information to each of the various feature amounts extracted by the extraction units 211 to 216 for each extracted process. This is used to determine which type of feature quantity is lined up.

<Vehicle behavior estimation unit 220>
FIG. 9 is a diagram illustrating the configuration of the own vehicle behavior estimation unit.
The own vehicle behavior estimation unit 220 utilizes the result of the vehicle speed sensor which is CAN information and the vehicle information such as the yaw rate or the steering angle, and the own vehicle by the four-wheel model as shown in FIG. 10 (1). The behavior estimation calculation unit 221 is used to predict the behavior of the above.

Based on this vehicle behavior estimation result, as shown in FIGS. 13 and 14, by arranging the roadside features in a two-dimensional coordinate system that looks like a bird's-eye view of the vehicle from above, the area in which the vehicle traveled. Generate a short-term relative map of the vehicle in. However, in the method of estimating the own vehicle behavior from the CAN information as shown in FIG. 10 (1), there are many errors such as tire pressure and slippage. Therefore, in order to make predictions with higher accuracy as well as predictions from CAN information, as shown in FIG. 10 (2), the behavior between two frames of the own vehicle is utilized by utilizing the corresponding points in the time series. The estimation is performed by the relative position / orientation estimation unit 222. That is, in the present embodiment, the behavior of the own vehicle is recognized by using the vehicle information of the own vehicle and the captured image of the stereo camera. FIG. 10 (2) shows that the images captured by the right camera of the own vehicle at the timings T [frame] and T + 1 [frame] correspond to each other in the images.

In the case of this method, it is possible to estimate the behavior of the own vehicle without being affected by the slippage of the tires of the vehicle, the air pressure of the tires, the size, and the like. When it is desired to use the own vehicle behavior with higher accuracy, the result of the behavior estimation calculation unit 221 is used. However, the relative position / orientation estimation unit 222 may not always be able to stably extract the corresponding points depending on the captured image. Therefore, in reality, basically, when the corresponding points between the frames can be obtained and the relative position can be estimated, the result of the relative position / attitude estimation unit 222 is used, and on the contrary, the corresponding points cannot be obtained. Alternatively, even if it is obtained, if there are many moving objects and stable vehicle behavior cannot be obtained, a combined method using the result of the behavior estimation calculation unit 221 is adopted.

Further, although the calculation load is high, the result of the self-map position estimation unit 223 such as using SLAM may be utilized. The result of the behavior estimation calculation unit 221 may be used only when the result of this SLAM cannot be used. FIG. 11 (1) shows a state in which the vehicle V ₀ passes through the corner of the road 1101 surrounded by the walls 1102 and 1103, and the previous sensing result (T [frm) embedded when the vehicle V ₀ passes through the corner. ]) And the present sensing result (T + 1 [frm]). As shown in FIG. 11 (1), when the corresponding point information is embedded in the map generated by oneself and the same route is taken again, the corresponding point information saved in the self-generated map and the current state. A method may be used in which the self-position and posture on the map are estimated by matching the corresponding points extracted from the frame, and the amount of movement in time series is also obtained.

FIG. 11 (2) is an image diagram of a high-precision map.
In this embodiment, the result of the high-precision map position estimation unit 224 may be utilized. The high-precision map 1110 is, for example, a traveling lane 1111, a branch lane 1112 branching from the traveling lane 1111, a roadside 1113 of the branch lane 1112, a branch 1114, a central separation zone 1115, an opposite lane 1116, and an opposite lane. It contains information about the merging lane 1117 merging into 1116 and the roadside 1118 of the merging lane 1117. When a high-precision map is originally prepared for automatic driving as shown in FIG. 11 (2), a method of estimating the position and posture of the own vehicle with this high-precision map may be used. Alternatively, it may be a method of grasping the position on such a high-precision map by utilizing high-precision GNSS. The own vehicle behavior estimation unit 220 estimates the own vehicle behavior in parallel by these plurality of methods, and adopts the one with high accuracy from among them.

<Roadside feature map generator>
FIG. 12 is a diagram illustrating the configuration of the roadside feature map generation unit.
In the roadside feature map generation unit 230, the roadside feature amount obtained by the roadside feature amount extraction unit 210 extracted every frame is obtained by the own vehicle behavior estimation unit 220, and the result obtained in FIG. 13 is used. As described in, arrange them in chronological order on the roadside feature map, which is a bird's-eye view of the vehicle from directly above.

Further, in the self-position estimation map generation unit 232, not only the roadside map but also the position of the corresponding point used for extracting the movement amount between frames, the feature amount of the corresponding point at that time, and the like are obtained. Save it in a three-dimensional position on the map. Once the map for self-position estimation is completed, the next time you drive on the same road, the position of the corresponding point saved in the map and the current frame will be detected in order to restore the accurate position. By extracting the correspondence relationship with the corresponding points, it is possible to specify the position on the map by obtaining the correspondence points not only between the frames but also on the map.

<Roadside feature map>
FIG. 13 is a diagram showing an example of a roadside feature map.
In the scene shown in FIG. 13, in the travel path 1311 shown on the upper side of FIG. 13, the vehicle V0 travels on the travel path _1311A on the median strip side, and the detection range 1320 is detected by the stereo camera. Then, the roadside feature amount on the left side in the traveling direction of the vehicle _V0 is extracted by the roadside feature amount extraction unit 210, and based on the estimation information of the own vehicle behavior estimation unit 220, the roadside feature map generation unit 230 detects the time series. The results are shown on a bird's-eye view map. In this scene, the gutter 1313 is on the outside (side) in the road width direction from the white line of the traveling path 1311B, and a plurality of buildings 1315 are arranged side by side on the outside. In such a case, both the feature amount of the gutter 1313 and the three-dimensional object accumulation (roadside feature amount) 1331 for the building 1315 behind it can be extracted. Although the gutters 1313 are continuously provided along the traveling path 1311B, there is a portion where the lid 1314 is partially covered and the side groove 1313 is partially interrupted, which is a gap of the feature amount in the traveling direction.

Similarly, for the travel path 1312 shown on the lower side of FIG. 13, the vehicle V1 travels on the travel path 1312B on the shoulder block _side , the roadside feature amount on the left _side in the traveling direction of the vehicle V1 is extracted, and the time series thereof. The result of is shown on the map of the bird's-eye view. A short road shoulder block 1316 having a height of about 15 cm and a wall 1318 via a sidewalk 1317 are arranged outside the white line of the road 1312B in the road width direction. It is possible to extract the feature amount of. Although the road shoulder block 1316 is continuously provided along the traveling path 1312B, there is a partially interrupted portion, which is a gap of the feature amount in the traveling direction.

In this way, when a plurality of feature amounts are doubled on one side and the other side in the road width direction of the traveling path and there is a gap between the feature amounts in the traveling direction, the longer the gap is, the longer the gap is. , It becomes difficult how to connect the features and judge it as a roadside.

The high-precision map information addition update unit 233 saves the corresponding points for saving the relationship between the high-precision map and the self-position for the high-precision map as shown in FIG. 11 (2), and updates the corresponding points as appropriate. I do.

<Inner feature priority part>
FIG. 14 is a diagram showing an example of a roadside feature map after executing the inner feature priority process by the inner feature priority section.
When a plurality of feature amounts are duplicated on the side of the traveling road, the inner feature priority unit 240 of the roadside type feature unit 200 performs a process of erasing the overlapping outer feature amounts. In this embodiment, as a process for giving priority to the feature amount close to the inside of the traveling region, that is, the roadside feature amount closer to the own vehicle in the road width direction, the feature amount inside when viewed from the own vehicle traveling direction. Is prioritized as a roadside feature amount, and a process of erasing the feature amount existing outside the roadside feature amount is implemented. As shown in FIG. 14, only the roadside feature amounts 1331, 1333, and 1334 which are the innermost roadside features when viewed from the own vehicle are left, and the overlapping feature amounts 1401 to 1406 outside the roadside feature amounts 1401 to 1406 are deleted from the bird's-eye view map. This makes it possible to prioritize and select the feature amount of the roadside existing inside.

However, when the roadside features on the inside exist almost continuously, it does not matter so much, but in the actual driving scene, even on the roadside on the inside, the car is on the sidewalk 1317. There are sections that are interrupted in consideration of the entrance and exit of the road and sections where the roadside disappears near the intersection. As for the gutter 1313, the feature amount is often interrupted, such as a section paved so that cars and pedestrians can pass, or a section blocked by the lid 1314, so the roadside feature amount is simply set along the traveling direction. There are many scenes where problems occur when they are connected together. Therefore, it is necessary to consider connecting roadside candidates appropriately according to the type.

<Inner feature type determination unit>
In the present embodiment, the inner feature type determination unit 250 first determines what kind of roadside type feature amount currently exists, and further determines the mode of the lateral position of the feature amount for each of these types. calculate. For example, in the case of the example shown on the upper side in FIG. 14, the feature amount of the minus step (feature amount of the gutter 1313) exists at the position (horizontal position) ₅ m outside in the road width direction when viewed from the vehicle V0. Although it is about half of this feature amount (length along the traveling direction), it is a cumulative three-dimensional object (roadside feature) extracted from the building 1315 at a position (horizontal position) of 8 m outside the road width direction. Amount) 1331 is present. In this way, the remaining features in FIG. 14 are accumulated so as to be projected in the traveling direction of the vehicle. That is, it is investigated what kind of features are arranged in the traveling direction according to the horizontal position. By accumulating the cumulative results of a certain type, the type of the inner roadside candidate is specified. By extracting the type and the rough horizontal position in this way, the information will be used as the judgment material for appropriately connecting the roadside candidates after this.

<Runway judgment unit>
FIG. 15 is a diagram for explaining the configuration of the travel path determination unit, and FIG. 16 is a diagram showing conditions for generating connected roadside candidates.
The travel path determination unit 300 determines the travel path according to the type of the roadside feature amount. In the travel road determination unit 300 according to this type, a plurality of types of roadside feature quantities extracted by the roadside type feature unit 200 are voted on the map, and the roadside feature amount on the map is noise-reduced with priority given to the inside. The final travel route judgment is carried out by utilizing the type information. The travel path determination unit 300 has a connection roadside candidate generation unit 310, a roadside candidate connection unit 320, and a travel path classification determination unit 330.

As shown in FIG. 16 (1), the connected roadside candidate generation unit 310 uses the roadside feature amount on the map to select and connect roadsides of the same feature amount type as roadside candidates. Judgment processing is performed. However, as shown in FIG. 16 (2), the threshold value that is the condition for generating the connected roadside candidate, which should be connected or should be the roadside candidate, is changed according to the type of the roadside. This is done by the travel path classification determination unit 330 by considering the reliability of the feature amount and the condition of whether or not the object is likely to exist continuously in the road structure according to the type of the roadside composed of the objects. It is possible to determine the travel path classification more appropriately.

The roadside candidate connecting unit 320 preferentially connects roadside candidates having the same type of features in consideration of the type of roadside, so that the roadside candidate connecting portion 320 is an accurate road existing inside the road width direction with respect to the own vehicle. Allows you to trace the edges correctly. For example, if the feature amount existing inside in the road width direction is simply prioritized and connected without judging the type, the roadside boundary line is drawn on the outer wall in the different road width direction for each interruption of the feature amount. Therefore, it is difficult to draw a stable roadside boundary line. By preferentially connecting roadside features of the same type and drawing a roadside boundary line, it is possible to more stably connect roadside candidates inside in the road width direction.

<Generation of connected roadside candidates>
From the perspective of the table shown in FIG. 16 (2), the road surface height means the voting amount of the feature amount at one point on the map, and the taller the object, the greater the depth on the parallax image of the stereo camera. Since information can be obtained, the amount of votes is large and the reliability is high. The road surface height is the amount of votes on the map, and if there is a vote amount above this threshold, it will be connected as a roadside candidate.

As shown in FIG. 16 (3), the three-dimensional object accumulation unit 211 of the roadside type feature unit 200 extracts the roadside feature amount of the object 1316 existing on the outer side 1612 in the road width direction of the travel path, and the travel path determination unit 300. As a result of the connection roadside candidate generation unit 310 of the above connecting the continuous three-dimensional object cumulative 1333 as the roadside feature amount, the minimum traveling direction length L0 is described as the minimum length in the table of FIG. 16 (2). If it is less than the threshold value, it is too short as a roadside candidate and is not recognized as a roadside candidate. Further, even when there is a gap in the traveling direction in this feature amount, if the maximum interval L1 is less than the threshold value shown in the table of FIG. Treat candidates in a concatenated manner.

For example, if it is a wall or an obstacle, since it is basically a tall three-dimensional object, the voting amount is high, and although the continuity of the wall is high when considering the continuity, for example, obstacles such as trees and utility poles are also considered. Then, there is a good possibility that the length in the traveling direction is short. Therefore, if the voting amount in the height direction is large, it is considered that the candidate is recognized as a roadside candidate even in the depth direction of 50 cm or more, which is a little short. Furthermore, the maximum spacing is the threshold value of the maximum depth width that is judged to be the same in consideration of the case where there is a gap in the travel path, and the maximum is 500 cm in consideration of the fact that there are often gaps in walls and obstacles. It is treated as a connection condition of the same roadside candidate and connected.

Next, in the case of a moving body (vehicle), since it is specified that the vehicle height is to some extent, the height is set to 100 cm or more, and the length is 50 cm or more, and two wheels or the like are also targeted. Even in a two-wheeled vehicle or a four-wheeled vehicle, there is no interruption in the middle, but the maximum distance between the gaps is up to 100 cm in consideration of the fact that there is a region where stereo parallax is difficult to appear because there are few image features in the middle of the car body. Is allowed, and the same features of the moving body are concatenated. That is, if the maximum distance is less than 100 cm, the roadside candidates are connected as if they are the same moving body.

Next, in the feature amount of the road shoulder block 1316, the height of the block separating the sidewalk and the roadway is 10 cm or more. Since the height of the sidewalk is lower than that of a wall or a moving body, the amount of votes for each point, which is a characteristic of a roadside, is small. Therefore, the minimum length is set to 100 cm or more in order to be recognized as a candidate for the road shoulder block 1316 only when there is a certain length along the traveling direction. In the case of the road shoulder block 1316, it is often interrupted, including the place where cars enter and exit, and the case where trees are planted. Therefore, roadside candidates are connected up to a maximum distance of 500 cm.

Regarding the road shoulder block 1316, there may be a gap further than that, but if it is extended further, the road shoulder block 1316 that was interrupted at an intersection etc. will also be connected, and it is judged that it is not possible to turn left or right to an orthogonal intersection. I'm in trouble when it comes to. Therefore, basically, the road shoulder blocks 1316 having a maximum distance of up to 500 cm are connected. If the gap of the road shoulder block 1316 is wider than this, it is treated as another road shoulder block.

With respect to the minus road end, the feature amount is extracted for the step that is lower than the traveling road surface in the minus step portion 214. Features are extracted at the boundaries of fields, rice fields, gutters, etc. that are lower than the road surface. However, the feature amount in a place lower than the traveling road surface is often difficult to obtain the feature amount due to the influence of shadows and the like, and the reliability is low. In addition, the reliability is low because the features of the same depth are not accumulated in the height direction. Therefore, if the condition of a certain length of 200 cm or more is not satisfied, the reliability is low, and the roadside candidate connecting portion 320 does not use such a negative step as a roadside candidate. There are many cases where the minus step is interrupted, but if you connect too long a distance, the effect of false detection is also a concern, so here, gaps up to 300 cm are connected as the same roadside candidate.

Similarly, for the stepless roadside, the roadside feature amount extraction unit 210 has the stepless roadside portion 215, which distinguishes between the traveling road and the lateral region by using the texture information on the road surface of the captured image. However, the feature amount is voted on the boundary region where it is determined that the texture is different between the traveling path and the lateral region. Using the votes on this map, roadside candidates as roadsides without steps are connected. Since there are cases where the texture is unstable, the roadside candidate is selected only when the texture is continuously 200 cm or more, and the gap is limited to 300 cm.

Finally, for a three-dimensional object having a traveling path such as a triangular pole, the height is 30 cm or more, and the minimum width in the traveling direction is 10 cm or more. With respect to the travel path division three-dimensional object, an acceptor is used for this feature quantity in order to recognize it as a roadside by connecting a considerably short object or connecting a considerably large gap. Therefore, not only is it specified as a roadside candidate with a width of 10 cm or more, but also a traveling road division three-dimensional object having a width of up to 800 cm is connected to be a roadside candidate.

The travel road classification determination unit 330 performs a process of determining the travel road classification of the own vehicle using the information of the roadside candidates connected by the roadside candidate connection unit 320.

<Generation of connected roadside candidates>
FIG. 17 shows an example in which roadside candidates of the same feature amount type are connected by the connecting roadside candidate generation unit 310 based on the conditions of FIG.
In a situation where the outer roadside feature amount is deleted due to the inner priority, the connection will be described for the left side roadside of the vehicle V0 traveling on the traveling road _1311A shown on the upper side of FIG.

First, as for the feature amount of the three-dimensional object cumulative 1331, those having a road surface height and a minimum length of 50 cm or more are selected as roadside candidates. Therefore, in the example shown in FIG. 17, two roadside candidates 1702 and 1703 are generated for the building 1315. Since the maximum distance between the two roadside candidates 1702 and 1703 is 500 cm or more, they are not connected to each other and are generated as separate roadside candidates. In FIG. 17, it is expressed that the black circles indicating the feature quantities of the three-dimensional object cumulative 1331 are connected by connecting them with a straight line.

Next, a plurality of negative step feature amounts 1334 adjacent to each other are connected. In the example shown in FIG. 17, the minus step of the gutter 1313 is partially interrupted by the lid 1314 that closes the gutter 1313. However, if the minimum length of the minus step is 200 cm or more as a roadside candidate in the connected state and the maximum interval of the minus step is less than 300 cm, the connection of the minus step feature amount 1334 is continued and the maximum interval of the minus step is 300 cm or more. If it is far from the roadside candidate, it is treated as another roadside candidate. Therefore, as the roadside on the left side of the vehicle _V0 shown on the upper side of FIG. 17, two roadside candidates 1701 and 1704 having a minus step are formed inside in the road width direction.

Since the length of one part interrupted by the lid 1314 is interrupted by 300 cm or more, the minus step is divided into two, one becomes a roadside candidate 1701 with a minus step, and the other is interrupted by the lid 1314. Since the length of the interrupted portion is less than 300 cm, one connected roadside candidate 1704 is used.

Next, considering the roadside candidate on the left _side in the traveling direction for the vehicle V1 traveling on the traveling road 1312B shown on the lower side of FIG. 17, the wall 1318 on the outer side in the road width direction is noise-reduced by the inner priority processing. Therefore, two relatively short roadside candidates 1712 and 1713 can be created. Further, a road shoulder block 1316 having a low step is connected to the inside as a roadside candidate, but there are two interrupted portions of the road shoulder block 1316. Since one interrupted portion has a length of less than 500 cm of the threshold value and the other interrupted portion has a length of 500 cm or more of the threshold value, the roadside candidate 1714 in which one low step 1333 is connected to each other and the other are low. The roadside candidate 1711 in which the step 1333 is divided is generated.

<Condition conditions for roadside candidate connection>
Next, the conditions for connecting the generated roadside candidates are shown in FIG. In particular, connection is carried out for a plurality of types of roadside candidates, but the conditions are different when a moving body is included in the plurality of types and when a traveling road division three-dimensional object is included.

In particular, when a moving body is included in a plurality of types of roadside candidates, connection is performed even when the deviation of the roadside lateral position is large. This is because in the case of a moving body, the lateral position is expected to change significantly as compared with the road shoulder block or the wall, and in the present embodiment, the lateral position (distance in the road width direction from other roadside candidates). ) Is ± 600 cm away, but it is accepted as a connection destination of roadside candidates. On the other hand, in other roadside candidates, the lateral position does not change so much, so up to about ± 200 cm is accepted as the connection destination of the roadside candidates. Of course, in the case of a running road that spreads smoothly, connect as it is. In addition, in the case of a three-dimensional object that divides the driving road, it is often used as a dividing line so that vehicles cannot enter by arranging it at a horizontal position away from the road shoulder block, wall, gutter, etc., so the distribution of the horizontal position is used. Up to ± 300 cm shall be accepted as a connection destination of roadside candidates.

<Example of roadside candidate connection (1)-(3)>
19 to 21 show an example of connecting roadside candidates in the roadside candidate connecting unit 320 that further connects the above-mentioned connecting roadside candidates. Using this roadside connection result, the travel road classification determination unit 330 determines the final roadside position and type. In FIGS. 19 to 21, the final result after the roadside candidates are connected is shown by the thick black dashed line.

FIG. 19 is a diagram illustrating a plurality of types of roadside connection example (1).
Considering the left side roadside for the vehicle _V0 shown on the upper side of FIG. 19, this roadside is at a place where the roadside of the gutter 1313 is interrupted, such as between the roadside candidates 1701 and 1704. The gutter 1313 appears again at a position away from the traveling direction by 500 cm or more, and the lid 1314 that closes the gutter 1313 continues for 500 cm or more. Therefore, it is examined whether it is possible to connect in different directions in the horizontal position, but the roadside candidate 1702 of the building 1315 is located at the position shown on the upper side in the figure in which the horizontal position with respect to the traveling direction is more than 200 cm.

However, since it is far from the connection condition, it is not the target of this connection, and the final result is that there are two roadsides of the single gutter 1313. Therefore, there are two connected results of the roadside candidates, the black broken line 1901 and 1902.

On the contrary, on the left side roadside for the vehicle V1 shown on the lower side of FIG. 19, the wall 1318 exists at _first , and the road shoulder block 1316 exists ahead of the wall 1318. Initially, the roadside of the three-dimensional object cumulative 1331 on the wall side is used, but beyond that, there is a low step 1333 which is a road shoulder block. Since the distance between the wall 1318 and the road shoulder block 1316 is less than 200 cm, the wall and the road shoulder block are connected here. Therefore, the connection result of the roadside candidates is one black dashed line 1903.

<Example of roadside connection including vehicles>
FIG. 20 is a diagram illustrating an example of roadside connection including a vehicle.
FIG. 20 shows the connection result of roadside candidates including a vehicle that is a moving body. Since the roadside of the traveling path 1311 shown on the upper side of FIG. 20 is the same as that of FIG. 19, only the roadside of the traveling path 1312 shown on the lower side of FIG. 20 will be described. The roadside connection result for the rear vehicle traveling on the travel path 1312A shown on the lower side of FIG. 20 is shown.

In the case of the example shown in FIG. 20, most of the low steps (roadside feature amount) 1333 of the road shoulder block 1316 are lined up on the outer side of the traveling road 1312 in the road width direction, and the vehicle is on the traveling road 1312A which is the inner lane. Vehicle V ₂ which is a preceding moving body exists in the traveling path 1312B in which V ₁ exists and is in the outer adjacent lane. It is known that the feature amount is a moving body because of the speed and the shape of the moving body _V2 . In this state, in the vehicle V ₁ , there are two types of feature quantities, the feature quantity 1333 of the road shoulder block 1316 and the feature quantity 1332 of the moving body V ₂ , and the connection candidate lines 1714 and 2001 for each feature quantity type. Is pulling.

Whether or not the mobile body V ₂ and the road shoulder block 1316 are to be further connected to each other, respectively, is determined by comparing with the connection conditions shown in FIG. In the case of the example shown in FIG. 20, since the lateral position (separation distance in the road width direction) between the connection candidate line 1714 and 2001 is less _than 600 cm, the connection candidate line 1714 of the moving body V2 and the road shoulder block 1316 are connected. The black dashed line connecting the candidate line 2001 is detected as the final result roadside 2002.

<Example of roadside connection including travel road classification>
FIG. 21 is a diagram illustrating an example of roadside connection including a travel road division.
An example of roadside connection for connecting the left side roadside of the vehicle _V0 shown on the upper side of FIG. 21 will be described. Unlike in the past, a plurality of triangular poles 2101 are arranged on the travel path 1311B so that general vehicles do not enter the construction site. Therefore, since two of the roadside candidate 1701 which is a part of the gutter 1313 and the roadside candidate 2102 which connects the triangular pole 2101 which is a traveling road division exist inside in the road width direction, the two roadside candidates 1701 and the roadside candidate 1701 It is connected to 2102 and detected as the final result roadside 2111. Roadside candidates created inside the connected component of the travel road division are not treated as connected targets in the first place.

Next, with respect to the vehicle V1 traveling on the travel path 1312 shown on the lower side of FIG. 21, there are _two connection candidates 1711 and 1712 on the road shoulder block 1316 and the wall 1318. In this case, since the lateral positions (separation distance in the road width direction) from the two connection candidates 1711 and 1712 are separated by 200 cm or more, they are not separated in the depth direction (vehicle traveling direction), but are not subject to connection. It is assumed that two different roadsides are detected as they are.

<Driving road classification judgment>
In the case where a plurality of the above-mentioned roadside connection results exist on one side, the traveling road determination unit 300 preferentially adopts the inner roadside and determines the type of the roadside according to the depth. By doing this on the left and right, the horizontal position for each depth is determined. Further, using this result, curve fitting is performed so as to have a spatially smooth travel path division. By curve fitting using time-series information, the position of the roadside that is spatially smooth and stable in time is calculated.

<Alarm control unit>
FIG. 22 is a diagram for explaining the configuration of the alarm control unit, and FIG. 23 is a diagram showing information on the alarm and the control margin for the roadside type.
The alarm control unit 400 uses the result of the travel path determination according to the type to determine whether or not to execute the alarm control, and implements the final warning to the driver and the vehicle control. The warning control unit 400 includes a vehicle behavior prediction unit 410, a roadside type contact deviation determination unit 420, an alarm unit 430, and a control unit 440.

The own vehicle behavior prediction unit 410 predicts the direction and position in which the own vehicle travels according to the steering angle and the vehicle speed of the vehicle. The roadside type contact deviation determination unit 420 determines whether or not there is a possibility of contact with the roadside on the vehicle behavior predicted by the own vehicle behavior prediction unit 410. The alarm unit 430 and the control unit 440 perform more appropriate control by changing the handling method of the alarm and the control according to the roadside type. For example, if the roadside type is a wall or an obstacle, contact or collision will have a large effect on vehicles and occupants. Therefore, early warning and control will be implemented for roadside types that are likely to have such a large impact.

For example, as shown in FIG. 23, for an object whose roadside type is likely to have a large effect when a vehicle comes into contact with or deviates, such as a wall, an obstacle, or a minus step (side groove), the travel path determination unit 300 When the position between the roadside and the vehicle recognized in is less than 100 cm, an alarm is given to the driver. Then, when the vehicle is further approached to the target, vehicle control is performed so that the vehicle does not come into contact with the roadside or deviate from the traveling road when the separation distance is less than 50 cm.

Furthermore, for moving objects (such as vehicles) that are more affected by contact than obstacles, an alarm is issued to the driver when the distance is less than 150 cm, and vehicle control is performed when the distance is less than 75 cm. However, these variables can be a dynamic method such as reducing the width according to the vehicle speed, and may be adjusted so as to allow driving intentionally approaching at low speed, for example. In particular, at low speeds of less than 20 km / h, the numerical distance is gradually reduced, and in the case of approximately 0 km, control is performed so that the value is about half of the value shown in the table of FIG. 23. It may be set to give priority to the intention.

When the obstacle is the road shoulder block 1316, if the contact is light, only the tires will be in contact. There are many. In addition, considering that the height of the mirror extending from the vehicle body is outside the contact target, the warning is set to less than 80 cm and the vehicle control is set to less than 40 cm.

With regard to roadsides without steps, if the road surface is made of different materials and deviates at high speeds, it may happen that the vehicle loses control, but it is unlikely that it will be a big problem at low speeds. Therefore, the warning is set to less than 50 cm and the vehicle control is set to less than 20 cm. The damage caused by contact with the triangular pole, which is a three-dimensional object, is relatively small. Therefore, the warning is set to less than 80 cm, and the vehicle control is set to less than 20 cm.

In this way, by changing the amount of warning and vehicle control margin according to the roadside type, warning and vehicle control that are more natural for the driver and safe without disturbing the driver's driving intention are implemented. .. Warnings and controls are implemented according to the lateral position distance set for each of these roadside types. In addition, when one side of the road is composed of multiple types of roadsides, safer warnings and vehicle control can be achieved by performing warnings and controls using values of the type with a larger margin. Realize.

<Process flow chart>
FIG. 24 is a flowchart illustrating the content of the recognition process executed by the image processing device. In this flow, a stereo camera is used to detect the roadside, and further, alarm control is performed.

First, the left and right images are captured by a stereo camera (S01). Then, after performing parallelization of the left and right images and sensitivity correction, stereo matching is performed to generate a parallax image (S02). Using the result of stereo matching and the camera geometry and baseline length, a 3D point cloud, which is a collection of points on the three-dimensional coordinates of the structure around the vehicle, is acquired. Then, a plurality of types of roadside features are acquired while using the parallax image and the 3D point cloud together (S03). With the steering angle, yaw rate, and vehicle speed as inputs, the vehicle behavior is estimated using the four-wheel model, and the vehicle behavior is estimated (S04). At that time, the behavior of the own vehicle may be estimated with higher accuracy by acquiring the relative position of the camera using the feature points obtained from the camera. Using this estimated own vehicle behavior and the acquired roadside feature amount, a map of the roadside feature amount is generated (S05).

Next, the feature quantities are concatenated on the roadside feature map. The feature quantities are connected to the traveling direction of the own vehicle. Then, the connection conditions are changed for each type of roadside, and roadside candidates are generated (S06). Next, it is determined whether or not to further connect the connected and linear roadside candidates, and the roadside to be the final travel path is recognized (S07). Based on the above-mentioned recognition result of the roadside, it is determined whether or not the own vehicle is likely to deviate from the traveling road (S08). An alarm or control is performed based on the above determination result (S09).

According to the image processing device of the present embodiment, a map of the roadside feature amount is created from the roadside feature amount and the vehicle behavior, the connection condition is changed for each roadside type, the roadside candidate is generated, and the roadside candidate is generated. The end candidates are connected based on the connection condition and detected as the road end of the travel path.

The conventional recognition logic that analyzes a travel path basically recognizes the shape of the travel path without specifying the type of the road edge. For this reason, it is difficult to properly detect the roadside shape in an environment where multiple types of roadsides coexist, and it is appropriate to determine whether or not to connect when the roadsides are scattered. There is a problem that it cannot be detected and it is not detected.

According to the present invention, in a complicated roadside environment in which a plurality of types of objects such as road shoulder blocks, walls, and gutters, which have been difficult in the past, are mixed, the same type of roadside features are preferentially connected and connected according to the type. By appropriately determining whether or not the roadside should be, it is possible to stably detect a more appropriate roadside.

Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various designs are designed without departing from the spirit of the present invention described in the claims. You can make changes. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the described configurations. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

100 Sensor unit 110 Left and right camera unit 120 Stereo matching unit 130 3D point cloud generation unit 150 Camera unit 170 Camera unit 180 Lidar unit 200 Roadside type feature unit 210 Roadside feature amount extraction unit 211 Three-dimensional object accumulation unit (wall, obstacle)
212 Mobile parts (cars, etc.)
213 Low step (road shoulder block)
214 Minus stepped part 215 Stepless roadside part 216 Driveway division three-dimensional object part 220 Own vehicle behavior estimation part 221 Behavior estimation calculation part 222 Relative position attitude estimation part 223 Self-map position estimation part 224 High-precision map position estimation part 230 Roadside Feature map generation unit 231 Time-series map generation unit for roadside recognition 232 Map generation unit for self-position estimation 233 High-precision map information addition update unit 240 Inner feature priority unit 250 Inner feature type determination unit 300 Travel road determination unit 310 Connected roadside Candidate generation unit 320 Roadside candidate connection unit 330 Roadway classification determination unit 400 Alarm control unit 410 Own vehicle behavior prediction unit 420 Roadside type contact deviation determination unit 430 Alarm unit 440 Control unit

Claims (14)

An image processing device that detects the roadside of a road based on an image captured by an in-vehicle camera.
Recognizing the types of objects that make up the roadside from the captured image,
An image processing device characterized in that parameters for detecting the roadside are changed according to the type of the object. The in-vehicle camera is a stereo camera.
The feature amount of the object obtained from the stereo camera is voted on the map according to the behavior of the own vehicle, and the feature amount is voted on the map.
The image processing apparatus according to claim 1, wherein the roadside is detected using the voting result on the map. Recognize multiple types of the object and
The image processing apparatus according to claim 1, wherein the roadside is detected by using a plurality of types of the objects at the same time. The object of the same type is extracted from the plurality of objects recognized from the captured image, and the object is extracted.
The image processing apparatus according to claim 3, wherein the feature amounts of the objects of the same type are connected to detect the roadside. The image processing apparatus according to claim 1, wherein a step between a region lower than the road surface of the traveling path and the traveling path is recognized as one type of the object from the captured image. .. A claim characterized in that the boundary between a region at the same height as the road surface of the travel path and the travel path is recognized as one type of the object by using the texture information of the captured image. Item 1. The image processing apparatus according to item 1. A three-dimensional object indicating a travel path classification is recognized as one type of the object from the captured image, and the object is recognized.
The image processing apparatus according to claim 4, wherein the distance between the features of the objects in the traveling direction to which the features can be connected is longer than that of other types of objects. A three-dimensional object that is a moving object is recognized as one type of the object from the captured image, and the object is recognized as one type.
The image processing apparatus according to claim 4, wherein the distance in the road width direction to which the feature amounts of the objects can be connected is large as compared with other types of objects. The image processing device according to claim 2, wherein the behavior of the own vehicle is recognized by using the vehicle information of the own vehicle and the image captured by the stereo camera. The image processing apparatus according to claim 1, wherein the threshold value of the road surface height at the time of determining the connectability of the feature amount of the object is changed according to the type of the object. The image processing apparatus according to claim 1, wherein the threshold value of the minimum length for selecting the feature amount of the object as a roadside candidate is changed according to the type of the object. The image processing apparatus according to claim 1, wherein the threshold value of the maximum interval between objects when determining the connectivity of the feature amounts of the objects is changed according to the type of the objects. The image processing apparatus according to claim 1, wherein the control margin of the alarm and the vehicle control is changed according to the type of the object. When it is recognized that there are a plurality of types of objects constituting the roadside, the control margin for the warning and the vehicle control is characterized in that the largest control margin among the types of the plurality of objects is used. 13. The image processing apparatus according to claim 13.

Images