WO2022264492A1

WO2022264492A1 - External recognition system

Info

Publication number: WO2022264492A1
Application number: PCT/JP2022/005011
Authority: WO
Inventors: 健永崎; 健志磨; 春樹的野; 孝一照井; 吉高新; 浩昭星加
Original assignee: 日立Astemo株式会社
Priority date: 2021-06-18
Filing date: 2022-02-09
Publication date: 2022-12-22
Also published as: CN117396933A; JPWO2022264492A1; DE112022001520T5

Abstract

The problem of the present invention is to provide an external recognition system capable of accurately detecting unevenness of a road surface in front of a vehicle. The external recognition system according to the present invention recognizes the three-dimensional shape of a road surface on which a subject vehicle C2 is traveling, and the subject vehicle C2 comprises: an image processing unit that executes image processing of detecting a road surface structure from an image captured by an in-vehicle camera; a vehicle communication unit that receives position information of the road surface structure ST from an external server NS; and an image processing method determination unit that changes the image processing method of the image processing unit on the basis of the position information of the road surface structure ST received by the vehicle communication unit.

Description

External recognition system

The present invention relates to an external recognition system that recognizes the unevenness of the road surface on which a vehicle travels.

　Conventionally, there is an external recognition technology that uses an in-vehicle camera to image the front of the vehicle and recognizes structures on the road surface such as speed bumps. A system has been proposed in which external world recognition information acquired by another vehicle is used for driving control of the own vehicle (Patent Document 1).

JP 2010-287044 A

When performing detection with an external recognition sensor such as an in-vehicle camera, it may not be possible to accurately recognize the unevenness of the road surface, such as structures on the road surface, depending on environmental conditions such as nighttime, backlight, and rainy weather. In addition, when the external world recognition information acquired by other vehicles is used for driving control of the own vehicle, accurate position information of structures on the road surface is required. detection accuracy is unstable, and detection accuracy is limited.

The present invention has been made in view of the above points, and its object is to provide an external world recognition system that can detect road surface unevenness in front of the vehicle with high accuracy.

The external world recognition system of the present invention for solving the above problems,
An external world recognition system that recognizes the three-dimensional shape of the road surface on which the vehicle travels,
The vehicle is
an image processing unit that performs image processing for detecting structures on the road surface from images captured by an in-vehicle camera;
a vehicle communication unit that receives position information of the structure on the road surface from an external server;
an image processing method determination unit that changes the image processing method of the image processing unit based on the position information of the structure on the road surface received by the vehicle communication unit;
characterized by comprising

According to the present invention, road surface unevenness in front of the vehicle can be detected with high accuracy. Further features related to the present invention will become apparent from the description of the specification and the accompanying drawings. Further, problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

The figure which shows the whole external world recognition system outline|summary in 1st Embodiment. The figure which shows the example of the scene which detects a structure on a road surface. The functional block diagram of an external world recognition system. 4 is a flowchart for explaining the contents of information processing in the information providing vehicle and the server; 4 is a flowchart for explaining an example of processing performed by the stereo camera of the information providing vehicle C1. 4 is a flowchart for explaining the contents of information processing in a server and an information-using vehicle; 4 is a flowchart for explaining processing by a stereo camera of information using vehicle C2. FIG. 4 is a diagram showing an example of setting an image processing region in a parallax image; 4A and 4B are diagrams for explaining an image processing method using position information; FIG. The figure explaining the method to search in a horizontal direction and to determine the parallax of unevenness|corrugation of a road surface. The figure which shows the result of the peak detection of road surface unevenness. The figure explaining the stability-analysis method of the peak movement by time progress. 9 is a flowchart for explaining the processing contents of an external world recognition system according to the second embodiment; FIG. 14 is a diagram showing an example of a situation in which the processing shown in FIG. 13 is performed; 11 is a flowchart for explaining the processing contents of an external world recognition system according to the third embodiment;

Next, an embodiment of the present invention will be described with reference to the drawings.

First, an outline of the external world recognition system of this embodiment will be explained. FIG. 1 is a diagram showing an overall overview of the external world recognition system. The external world recognition system in this embodiment aggregates information on road conditions detected by at least one or more information providing vehicles C1 in a server NS, which is an external server, and collects information on road conditions around the information using vehicle C2. In this system, the server NS distributes the information to the information-using vehicle C2, and the information-using vehicle C2 detects the road surface conditions in more detail using the road surface condition information distributed from the server NS, and controls the vehicle based on the detection results. be.

The external world recognition system uses a cloud-type map platform 101 that accumulates, manages, and analyzes the experience information necessary for automatic driving of vehicles on a map. The map platform 101 is called traffic experience information on a map, and information such as roads, lanes, driving routes called road-related maps, and facilities and equipment such as signs and traffic lights installed on roads. and past driving experience-related information such as driving history including driving experience information are managed in a two-layered state in which they are associated with each other. Road-related maps contain high-precision map information used for automatic driving of vehicles. The driving experience information includes, for example, information based on actual driving experience, such as the frequency of occurrence of traffic jams on a predetermined road, time zones, average traveling speed, and the like.

The map platform 101 receives vehicle information such as lane selection and appropriate speed from a plurality of information providing vehicles C1 (C1a-C1n), and also receives traffic information such as traffic lights and congestion from the infrastructure equipment 111. The map platform 101 analyzes big data including vehicle information and traffic information using a server NS connected to an information communication network and a server database DBs that stores various types of information in a readable manner to generate driving experience information. is generated and mapped on the road-related map.

A plurality of information-providing vehicles C1 and information-using vehicles C2 each use a self-positioning device such as GNSS (Global Navigation Satellite System) to obtain self-location information in the world coordinate system (latitude-longitude coordinate system). to get The information providing vehicle C1 recognizes the external world by means of a three-dimensional measurement device, which is an external world recognition device mounted on the vehicle, and transmits road surface condition information obtained by the external world recognition to the server NS.

The information-using vehicle C2 performs more accurate automatic driving by receiving distribution of traffic experience information on the map from the server NS of the map platform 101 based on the self-location information. The information-using vehicle C2 performs vehicle control such as automatic driving control using the three-dimensional measuring device 124, the three-dimensional external world information acquired by the three-dimensional measuring device 124, and the information acquired from the server NS of the map platform 101. AD_ECU 122 , MPU 123 that performs various kinds of arithmetic processing, steering device 125 , brake device 126 and drive device 127 that are controlled by control signals from AD_ECU 122 .

FIG. 2 is a diagram showing an example of a scene in which structures on the road surface are detected using the external world recognition system according to this embodiment.

The external world recognition system performs information aggregation processing and information distribution processing. In the information aggregation process, when a structure on the road surface is detected while the information providing vehicle C1 is traveling, the position information of the structure ST on the road surface is transmitted to the server NS, and the information is aggregated by the server NS. FIG. 2(a) is a diagram showing an example of information aggregation, and shows a scene in which a structure ST on the road surface is detected while driving in the daytime. If the three-dimensional measurement device of the information providing vehicle C1 is, for example, a stereo camera, the structure ST on the road surface can be easily and accurately detected during travel in bright daytime hours.

In the information providing vehicle C1, the three-dimensional shape of the road surface unevenness is calculated by the stereo camera, the road surface structure ST is detected, and the relative position and height (shape) of the road surface structure ST with respect to the information providing vehicle C1 are calculated. is estimated. Then, the self-position information of the information providing vehicle C1 measured by the self-position measuring device 315 and the position information of the structure ST on the road surface are transmitted to the server NS by the communication device and stored in the server database DBs as driving experience information. . The position information of the structure ST on the road surface is stored separately for each structure ST on the road surface.

The server NS receives the position information of the road surface structure ST from a plurality of information providing vehicles C1. The server NS aggregates the positional information of the structures ST on the road surface received from the plurality of information providing vehicles C1, and recalculates the positional information for each structure ST on the road surface. In this recalculation, statistical processing is performed to reduce errors in the position information and improve the accuracy of the position information.

In the information distribution process, by transmitting the self-location information from the information-using vehicle C2 to the server NS, the server NS distributes information about road conditions around the information-using vehicle C2 to the information-using vehicle C2. FIG. 2(b) is a diagram showing an example of information distribution, showing a scene in which a structure on the road surface is detected while driving at night. The stereo camera of the information-using vehicle C2 may have a low detection success rate and detection accuracy for the structure ST on the road surface, which is the object to be detected, when the vehicle is traveling in the dark at night.

On the other hand, in this embodiment, when the information-using vehicle C2 receives the distribution of the positional information of the structures on the road surface existing around its own position from the server NS, the stereo camera detects the structures on the road surface based on the positional information. The image processing method of the image processing unit for detecting objects is determined, and the structure ST on the road surface is detected by the determined image processing method. For example, when more accurate position information of the structure ST on the road surface is obtained from the server NS, it is possible to change the detection algorithm, such as performing highly accurate image recognition by narrowing the area where image processing is performed by the stereo camera. , the detection parameters are changed such as by narrowing the range for detecting the peak height position of road unevenness and increasing the threshold value. Further, when the stereo camera cannot detect the structure ST on the road surface due to the weather or outside light conditions, or when the detection accuracy is low, the information-using vehicle C2 does not use the recognition result of the image recognition. , vehicle control such as warning to passengers, suspension pressure, deceleration, steering, etc. is performed using the position information of the road surface structure ST distributed from the server NS.

FIG. 3 is a functional block diagram of the information providing vehicle C1, the information using vehicle C2, and the server NS that constitute the external world recognition system of this embodiment.
The information providing vehicle C1 provides information by passing over the stereo camera 312, which is an example of a three-dimensional measuring device that measures the three-dimensional shape of the road surface and detects structures on the road surface, and the structures ST on the road surface. Information is exchanged between a G sensor 313 that detects vehicle vibration applied to the vehicle C1, a self-position measuring device (position information acquisition means) 315 that measures the self coordinate position in the world coordinate system such as a GPS navigation device, and the server NS. It has a road-to-vehicle communication device 317 for transmitting and receiving.

The information-using vehicle C2 has a stereo camera 322, a self-position measuring device (self-position measuring unit) 325, a road-to-vehicle communication device 324, and a G sensor 323. Further, based on the detection result of the stereo camera 322, A vehicle control device 326 that performs automatic brake control and inter-vehicle distance control ACC, a human-machine interface 327 that warns vehicle occupants, a vehicle route generation device 328 that generates a route to a destination, and information-using vehicle C2. It has a vehicle database DBc that stores information such as the coordinate position and the position and size of the structure ST on the road surface. The stereo camera 322 has a pair of left and right in-vehicle cameras that capture images in front of the vehicle, and a camera control unit that performs image processing of the in-vehicle cameras and captured images. The camera control section has hardware such as a CPU and a memory, and a software program executed by the hardware, and implements the control functions of the following sections through cooperation between the hardware and software. The camera control unit of the stereo camera 322 includes an image processing unit that performs image processing for detecting structures on the road surface from images captured by the vehicle-mounted camera, and based on the position information of the structures on the road surface ST received from the server NS. and an image processing method determination unit that changes the image processing method of the image processing unit.

The server NS includes a communication device 302 that transmits and receives information to and from the road-to-vehicle communication device 317 of the information providing vehicle C1 and to and from the road-to-vehicle communication device 324 of the information-using vehicle C2. Statistical analysis of the server database DBs that collects and stores the position information of the provided road surface structure ST in a readable manner and the big data that includes vehicle information and traffic information stored in the server database DBs, It has a data analysis device 303 that generates driving experience information and maps it on a road-related map.

FIG. 4 is a flowchart for explaining the content of information processing in the information providing vehicle and the server, and shows the content of processing for providing information from the information providing vehicle C1 to the server NS.

In the information providing vehicle C1, the image pickup unit of the stereo camera 312 picks up an image in front of the vehicle to obtain a picked-up image (S401), and the image processing unit of the stereo camera 312 performs image processing on the picked-up image to generate a parallax image. The unevenness of the road surface, which is a three-dimensional shape in front of the vehicle, is measured, and the structure ST on the road surface is detected based on the measurement results (S402). The stereo camera 312 acquires three-dimensional shapes such as speed bumps and road edge shapes on the road surface, road unevenness data, etc., and information on the positions of the structures ST on the road surface through image processing by the image processing unit.

The image processing unit of the stereo camera 312 coordinates-transforms the position of the structure ST on the road surface obtained by image processing from the position relative to the own vehicle to the coordinate position of the world coordinate system. Specifically, position information in the world coordinate system corresponding to the imaging position of the own vehicle measured by the self-position measuring device 315 is acquired (S403), and position information in the world coordinate system of the structure ST on the road surface is generated. (S404). Then, the information providing vehicle C1 uses the road-vehicle communication device 317 to transmit the position information of the structure ST on the road surface after the coordinate conversion to the server NS.

The server NS stores in the server database DBs the positional information of the plurality of road surface structures ST transmitted from the plurality of information providing vehicles C1 or from the same information providing vehicle C1 (S405). When the server NS stores the position information of the structure ST on the road surface in the server database DBs, the server NS recalculates the traffic experience information on the map at a predetermined timing and updates it to the latest information. When the server NS receives the positional information of a plurality of structures ST on the road surface from the information providing vehicle C1, the server NS organizes the information by adding or deleting the information, collects the information on the same position as the same information, and stores it in the server database. Store in DBs. The server NS statistically analyzes the positional information of the structures ST on the road surface transmitted from the plurality of information providing vehicles C1 and stored in the server database DBs as big data by the data analysis device 303, and creates and updates the driving experience information. , is mapped onto a road-related map, and stored in the server database DBs as traffic experience information on the map (S406).

FIG. 5 is a flowchart for explaining a normal image processing method in a stereo camera, and shows an example of processing performed by the stereo camera of the information providing vehicle C1.

First, a pair of left and right captured images are captured by the imaging unit of the stereo camera 312 (S501), and a parallax image is generated from the pair of captured images (S502). Then, an image processing area for image processing is set in the parallax image (S503). In this embodiment, in order to acquire the road surface unevenness data, an area on the tire traveling path through which the left and right tires of the information providing vehicle C1 pass is set as an image processing area.

FIG. 8 is a diagram showing an example of setting an image processing area in a parallax image, and is a diagram explaining a method of setting the image processing area. A parallax image 801 generated by imaging the front of the vehicle with a stereo camera shows a traveling road surface 811 of a road R0 and white lines 812 on both left and right sides thereof. Processing

areas

821 and 822 are set along the vehicle traveling path. The

processing areas

821 and 822 are set so as to extend in the depth direction (advance direction) with a predetermined width on the tire travel path through which the left and right tires pass as the vehicle travel path.

Next, detailed parallax analysis is performed for each lateral search (S504). Here, the

processing areas

821 and 822 of the parallax image 801 are searched in the lateral direction of the image as indicated by

arrows

831 and 832 in FIG. Then, as shown in FIG. 10, the parallax value with the most frequent number of votes is adopted. FIG. 10 is a diagram for explaining a method of searching in the lateral direction to determine the parallax of unevenness of the road surface. In the example shown in FIG. 8, the lateral search is performed over the entire depth direction of the

processing areas

821 and 822 .

Next, the parallax values for each lateral search calculated in S504 are arranged in the depth direction of the processing area, and the peak position of the unevenness height of the road surface is calculated (S505). For example, a peak position whose height from the road surface reference plane has a value greater than a predetermined determination threshold value can be determined as the position of the structure on the road surface ST.

Then, a process of analyzing the stability of peak position determination is performed (S506). In this stability analysis processing, it is determined whether or not the peak position moves according to the movement of the own vehicle. Then, the distance between the own vehicle and the structure ST on the road surface is calculated (S507), and the final output is produced (S508). In the final output of S508, a process of outputting a warning to the occupants and an emergency braking signal to the vehicle control device is performed, and the vehicle control device controls the warning and emergency braking, but is not limited to this. For example, only the information on the distance between the own vehicle and the structure ST on the road surface may be output to the vehicle control device, and the vehicle control device may issue an alarm or an emergency brake.

FIG. 6 is a flow chart explaining the contents of information processing in the server NS and the information-using vehicle C2.

The information-using vehicle C2 measures its own position with the self-position measuring device 325, and transmits the self-position information to the server NS (S601). When receiving the self-position information from the information-using vehicle C2, the server NS searches the server database DBs and extracts the position information of structures on the road existing around the self-position of the information-using vehicle C2 (S602). . Then, the position information is distributed to the information-using vehicle C2 that has provided the self-position information.

When the information-using vehicle C2 receives the position information of the structures ST existing on the road surface around its own position from the server NS, it stores it in the vehicle database DBc (S603). Then, the stereo camera 322 picks up an image in front of the vehicle, acquires the picked-up image (S604), and performs image processing using the position information of the structure on the road surface ST received from the server NS stored in the vehicle database DBc. A structure ST on the road surface is detected (S605).

In the image processing of S605, the image processing method is determined based on the position information of the structure ST on the road surface received from the server NS. For example, first, a process of searching the vehicle database DBc using position information of structures on the road surface around the self-position measured by the self-position measuring device 325 is performed. When the road surface structure ST, which is the object to be detected, exists in the vicinity of the self position, the image processing method determination unit of the stereo camera 322 changes at least one of the image processing algorithm and parameters. done.

The stereo camera 322 performs image processing using the changed algorithm and parameters, and detects the structure ST on the road surface. The stereo camera 322 changes the algorithm or parameters based on the position information of the structure ST on the road surface received from the server NS, analyzes the image processing area corresponding to the position in detail, and adjusts the detection accuracy. .

The information-using vehicle C2 performs vehicle control using the detection result of the structure ST on the road surface by the stereo camera 322 (S606). In the vehicle control process of S606, for example, at least one of adjustment of the vehicle speed passing the road surface structure ST, route setting, and warning to the occupant is performed. For example, the vehicle control device 326 searches the vehicle database DBc for information on the road surface structure ST around the self-position measured by the self-position measuring device 325, and searches the vehicle database DBc for information on the road surface structure ST that is stored corresponding to the road surface structure ST. When it is determined that the speed of the own vehicle exceeds the speed information, deceleration control is performed. Further, if the information of the road surface structure ST stored in the vehicle database DBc is a speed bump, the speed is decelerated to a speed at which the speed bump can be passed safely or comfortably, or the upper limit speed of the inter-vehicle distance control ACC is set. Change to a speed that you can safely or comfortably pass. Then, the vehicle route generation device 328 sets a route so that the information-using vehicle C2 passes through a road or lane on which it can travel safely or comfortably. When the human-machine interface 327 determines that the speed of the information-using vehicle C2 is excessive, it sounds a warning alarm or displays a warning on the monitor to warn the occupants. If the information-using vehicle C2 has an active suspension capable of controlling the damping value of the suspension, the damping value may be adjusted to suppress the behavior of the vehicle when passing through a speed bump.

FIG. 7 is a flowchart for explaining the processing by the stereo camera of the information-using vehicle C2. The description of the configuration similar to that of the flowchart shown in FIG. 5 will be omitted.

The stereo camera 322 changes the image processing method according to whether or not the structure ST on the road surface exists around its own position. The stereo camera 322 performs normal image processing using the same algorithm and parameters as those of the information providing vehicle C1 when the structure ST on the road surface does not exist around its own position. On the other hand, when the road surface structure ST exists around the self position, image processing is performed by changing at least one of the image processing algorithm and parameters as compared with the case where the road surface structure ST does not exist.

When the structure ST on the road surface exists around the self position, in the processing area setting process of S503A and the detailed parallax analysis process for each lateral search of S504A, based on the position information of the structure ST on the road surface received from the server NS, A process of narrowing down the image processing area of the road unevenness and a process of limiting the peak position are performed.
FIG. 9 is a diagram for explaining an image processing method using position information, and shows an example of narrowing down the image processing area.

For example, as shown in FIG. 9, when it is determined that the information-using vehicle C2 approaches the position of the structure ST on the road surface received from the server NS and is within a predetermined range, the image processing method determining unit of the stereo camera 322 performs a process of narrowing down the image processing area of road unevenness based on the positional information of the road surface structure ST received from the server NS. In the example shown in FIG. 9, from the first

image processing areas

821 and 822 set on the tire traveling path in the depth direction, a second image processing area 911 corresponding to the position of the structure ST on the road surface on the tire traveling path , 912 is performed.

In addition, the self-position of the vehicle estimated by receiving radio waves from GPS or base stations, the update cycle of GPS information, the vehicle speed at that time, the position information of the road surface structure ST received from the server NS, the position information of the road surface structure ST received from the server NS, map estimation error (depending on radio wave conditions and the number of detected base stations), information transmission in the in-vehicle communication environment, and time-world coordinate transformation (imaging and Captured by a camera by converting positional information of an object into a three-dimensional to two-dimensional coordinate system that considers time-varying information such as position measurement and vehicle speed, and the coordinate system of the map and the imaging coordinate system of the camera. It is possible to further narrow down the region where the road unevenness should be analyzed from the image to a part of the tire traveling path. As a result, the density of the parallax image calculation S503 and the road unevenness calculation S504 and S505 (calculation target points and granularity) is partially increased, thereby improving the processing accuracy while suppressing the increase in the calculation load as much as possible. can be done. In addition, it is possible to improve the accuracy by increasing the probability of correct determination by relaxing the threshold for road surface unevenness determination (for example, the threshold for height determination in peak calculation S505).

Then, the image processing in the second

image processing regions

911 and 912 is changed to image processing with higher accuracy than the image processing in the first image processing region, and the structure ST on the road surface is detected. The second

image processing areas

911 and 912 are set shorter in length in the depth direction than the first

image processing areas

821 and 822, and have a size and shape that cover the road surface structure ST in the parallax image. there is In this way, by narrowing down the image processing area to make it smaller, the image processing load can be reduced. Therefore, more detailed image processing can be performed in a short time with limited hardware resources, and the structure ST on the road surface, which is the object to be detected, can be detected with high accuracy.

FIG. 11 is a diagram showing the peak detection results of road surface irregularities, showing the detected values of the road surface height in the depth direction of the tire traveling path. In the graph of FIG. 11 , the horizontal axis is the distance (m) in the depth direction, and the vertical axis is the road surface height (cm) detected by the stereo camera 322 .

The image processing method determination unit of the stereo camera 322 determines the peak position of the road unevenness in the first detection range set along the depth direction of the tire travel path when the structure ST on the road surface does not exist around the self position. However, when the structure ST on the road surface exists around the self position, the image processing algorithm or parameters of the image processing unit are changed, and a second detection range corresponding to the position of the structure ST on the road surface on the tire traveling path is detected. , the peak position of road unevenness is determined (S505A).

Further, the image processing method determination unit of the stereo camera 322 sets the determination threshold for determining the peak position of road surface unevenness in the second detection range to be higher than the determination threshold for determining the peak position of road surface unevenness in the first detection range. A large value is set (S505A). As a result, the noise before and after the peak position can be removed, the calculation load for detecting the local peak of the unevenness of the road surface can be reduced, and the determination accuracy of the structure ST on the road surface can be improved.

In the final output of S508A, similarly to S508, a process of outputting an alarm or emergency braking signal to the vehicle control device, or a process of outputting only information on the distance between the own vehicle and the structure ST on the road surface is performed. Further, it is determined whether or not the reliability of the captured image captured by the stereo camera 322 is equal to or higher than the threshold. For example, when the reliability is lower than the threshold value due to the weather or external light conditions, the processing in S501 to S507 is skipped, and the final output is made based on the position information of the structure ST on the road surface distributed from the server NS. Based on the output, vehicle control such as warning to passengers, suspension pressure, deceleration, and steering is performed.

In addition, in this embodiment, after step S508A, a step S509A for calculating verification information of the structure ST on the road surface is provided.

In the verification information calculation process of S509A, the stereo camera 322 converts the position of the structure ST on the road surface detected by the image processing into the coordinate position of the world coordinate system using the self-position information obtained by the self-position measuring device 325. , acquires information on the position where the behavior of the vehicle is detected when the vehicle passes over the structure ST on the road surface.

The process of converting the position of the structure ST on the road surface into the world coordinates is based on the distance to the structure ST on the road surface detected by image processing, (1) the vehicle speed at the time of imaging, and (2) the imaging unit of the stereo camera 322. (3) self-location information by the self-location measurement device 325; (4) acquisition timing of self-location information by the self-location measurement device 325; (5) communication delay speed of in-vehicle equipment; Time-world coordinate conversion processing converts position information into a three-dimensional to two-dimensional coordinate system, taking into consideration time-varying information such as imaging, position measurement, and vehicle speed, as well as the coordinate system of the map and the coordinate system captured by the camera. is. In addition, the behavior of the vehicle when the vehicle passes over the structure ST on the road surface can be detected by the stereo camera 322 by detecting the vertical movement of the captured image, and by the vehicle-mounted sensors such as the G sensor, pitching, vertical acceleration, A change in suspension pressure may be detected.

In the verification information calculation of S509A, the position of the road surface structure ST detected by the stereo camera 322 of the information using vehicle C2 is verified from the own vehicle behavior, and the verification information is transmitted to the server NS. The server NS can improve the estimation accuracy of the position of the road surface structure ST using the verification information.

<Second embodiment>
Next, an external world recognition system according to the second embodiment will be described with reference to the drawings.
FIG. 13 is a flow chart for explaining the processing contents of the external world recognition system in the second embodiment, and FIG. 14 is a diagram showing an example of a situation in which the processing shown in FIG. 13 is performed.

A characteristic feature of this embodiment is that although the information providing vehicle C1 cannot detect the structure ST on the road surface with the stereo camera 312, it detects vehicle vibration in the vertical direction due to passing over the structure ST on the road surface. By detecting the vibration, the server NS performs image processing using an image obtained by picking up the vibration detection position.

For example, as shown in FIG. 14, the stereo camera 312 of the information providing vehicle C1 acquires captured images captured during travel and stores them in the vehicle database (S1301). Then, for example, when the G sensor 313 detects vibration equal to or greater than a preset threshold for detecting vertical vehicle vibration due to passing over the road surface structure ST (S1302), the self-position measuring device detects The position information of the vibration detection point Pn is acquired based on the self-position information (S1303). The method of detecting vehicle vibration is not limited to the G sensor 313. For example, vertical movement of images captured by the stereo camera 312, pitching by an in-vehicle sensor such as a gyro sensor, vertical acceleration, and changes in suspension pressure are used. good too.

The stereo camera 312 searches the inside of the vehicle database and extracts the captured image of the vibration detection point Pn from the captured images stored in the vehicle database (S1304). Here, as shown in FIG. 14, an image captured when t time goes back from the time when the vehicle vibration in the vertical direction is detected to the position Pn-1 where the stereo camera 312 captures the image of the vibration detection point Pn. , is extracted as an image in which the vibration detection point Pn is captured. The road-to-vehicle communication device 317 receives from the stereo camera 312 the information of the vibration detection point Pn and the imaged image of the point, and transmits the information to the server NS.

When the server NS acquires the information of the vibration detection point Pn and the captured image of the vibration detection point Pn from the information providing vehicle C1, the data analysis device 303 detects the structure ST on the road surface from the captured image of the vibration detection point Pn. image processing is performed (S1305). The image processing performed by the server NS differs from the image processing performed by the stereo camera 312 in terms of processing content, and is more detailed and advanced than the image processing performed in the stereo camera 312 . Unlike the stereo camera 312 , the server NS does not have processing time restrictions and has higher specifications than the stereo camera 312 and abundant hardware and software resources. Image processing of content can be performed. Therefore, it is possible to detect the road surface structure ST that could not be detected by the stereo camera 312, and to verify the reason why the stereo camera 312 could not detect it.

The server NS stores the detection result of the road surface structure ST by the image processing of the data analysis device 303 and the position information of the vibration detection point Pn in the server database DBs (S1306). The position information is analyzed and recalculated (S1307).

According to the external world recognition system of this embodiment, the structure ST on the road surface, which could not be detected by the stereo camera 312, can be detected by the image processing in the server NS, and the road surface stored in the server database DBs can be detected. The accuracy of the position information of the upper structure ST can be made higher.

In this embodiment, when the stereo camera 312 of the information providing vehicle C1 cannot detect the structure ST on the road surface, the image processing for verification is performed in the server NS. Image processing for verification may be performed in the server NS also when the structure ST on the road surface is detected by . In the present embodiment, the case of the information providing vehicle C1 has been described as an example. may be used to perform image processing in the server NS.

<Third Embodiment>
Next, an external world recognition system according to the third embodiment will be described with reference to the drawings.
FIG. 15 is a flowchart for explaining the processing contents of the external world recognition system in the third embodiment, and shows an analysis method based on the information on the position of structures on the road surface and vehicle vibration.

A characteristic feature of this embodiment is that, in addition to the positional information of the structure ST on the road surface, information about the degree of influence on vehicle vibration when the vehicle actually passes is stored in the server, and the information-using vehicle C2 detects the road surface. It is used for vehicle control when passing over the upper structure ST.

The information providing vehicle C1 acquires a captured image (S1501), and performs image processing to detect the structure ST on the road surface, determine the distance to the structure ST on the road surface, and the height of the structure ST from the road surface reference plane. is measured (S1502). Note that the road surface reference plane can be obtained, for example, from the average position of road unevenness in the depth direction. Then, the information providing vehicle C1 uses the G sensor 323 to detect vibration when it passes over the structure ST on the road surface (S1503), and acquires position information of the vibration detection point (S1504).

Then, the degree of influence on the vehicle vibration is calculated based on the magnitude of the vibration detected by the G sensor 323, and the position information of the structure ST on the road surface, the position of the vibration detection point, and the information on the degree of influence on the vehicle vibration are obtained. The information is sent to the server NS, and the server NS stores this information in the server database DBs (S1505). The server NS aggregates the position information of the road surface structure ST stored in the server database DBs, the position of the vibration detection point, and the degree of influence on the vehicle vibration, and the data analysis device 303 statistically analyzes the big data. , update the driving experience information, map it on the road-related map, and create the traffic experience information on the map. After storing the position information of the road surface structure ST in the server database DBs, the server NS recalculates the traffic experience information on the map at a predetermined timing in order to update the information to the latest information (S1506).

When the server NS receives the self-position information from the information providing vehicle C1, the server NS uses the position information of the structure ST existing on the road surface around the self-position of the information-using vehicle C2 and the information on the degree of influence on the vehicle vibration. Send to vehicle C2.

The information-using vehicle C2 slows down the running speed and adjusts the suspension based on the position information of the road surface structures ST existing around its own position and the information on the degree of influence on the vehicle vibration so as to reduce the vibration of the vehicle. It performs vehicle control such as damping force adjustment.

According to this embodiment, the information on the degree of influence of the vehicle vibration by the structure ST on the road surface can be provided to the information-using vehicle C2. can be performed, and the passenger's riding comfort can be improved.

Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various designs can be made without departing from the spirit of the invention described in the claims. Changes can be made. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. Also, part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with another configuration.

C1 Information providing vehicle C2 Information using vehicle (own vehicle)
NS server (external server)
DBs Server database DBc Vehicle database ST On-road structure 322 Stereo camera 323 G sensor 324 Road-to-vehicle communication device (vehicle communication unit)
325 self-

position measuring devices

821, 822 first

image processing areas

911, 912 second image processing areas

Claims

An external world recognition system that recognizes the three-dimensional shape of the road surface on which the vehicle travels,
The vehicle is
an image processing unit that performs image processing for detecting structures on the road surface from images captured by an in-vehicle camera;
a vehicle communication unit that receives position information of the structure on the road surface from an external server;
an image processing method determination unit that changes the image processing method of the image processing unit based on the position information of the structure on the road surface received by the vehicle communication unit;
An external world recognition system comprising:
The external world recognition system according to claim 1, wherein the image processing method determination unit changes an algorithm or parameter of image processing by the image processing unit.
The image processing method determining unit selects an image processing area for image processing by the image processing unit from a first image processing area set along a vehicle traveling path of the host vehicle to an image processing area on the vehicle traveling path and on the road surface. 3. The external world recognition system according to claim 2, wherein the second image processing area is narrowed down to correspond to the position of the upper structure.
4. The external world according to claim 3, wherein the image processing method determining unit changes the image processing of the second image processing area to image processing with higher precision than the image processing of the first image processing area. recognition system.
The image processing unit detects a peak position of road unevenness by the image processing,
The image processing method determining unit determines a range for detecting a peak position of road unevenness by image processing of the image processing unit from a first detection range set along a vehicle traveling path of the own vehicle. 3. The external world recognition system according to claim 2, wherein the detection range is limited to a second detection range corresponding to the position of the road surface structure on the road.
The image processing method determining unit sets a determination threshold value for determining the peak position of the road surface unevenness in the second detection range to a value larger than a determination threshold value for determining the peak position of the road surface unevenness in the first detection range. 6. The external world recognition system according to claim 5, characterized in that it is set to .
The vehicle is
a self-localization unit that measures the position information of the own vehicle indicated by the coordinate position of the world coordinate system;
a coordinate conversion unit that converts the position information of the structure on the road surface received by the vehicle communication unit from the coordinate position of the world coordinate system to the coordinate position of the own vehicle coordinate system;
The image processing method determination unit changes the image processing method of the image processing unit based on the positional information of the structure on the road surface converted to the coordinate position of the vehicle coordinate system by the coordinate conversion unit. The external world recognition system according to claim 1.
The coordinate conversion unit uses any of the vehicle speed of the own vehicle, the imaging timing by the on-vehicle camera, the acquisition timing of the self-position information by the self-position measurement unit, and the communication delay speed between internal devices of the own vehicle, 8. The external world recognition system according to claim 7, wherein the coordinate position is converted.
An external server that is communicatively connected to the vehicle communication unit of the own vehicle,
The external server is
a server database that stores position information of the structure on the road surface;
a server communication unit that extracts position information of structures on the road surface existing around the vehicle from the server database and distributes the position information to the vehicle;
The external world recognition system according to claim 1, characterized by comprising:
At least one or more information providing vehicles that are communicatively connected to the external server and transmit position information of the structure on the road surface;
The information providing vehicle is
a road surface structure detection unit that detects structures on the road surface by measuring the three-dimensional shape of the road surface;
a position information obtaining means for obtaining position information of the structure on the road surface detected by the structure detection unit on the road surface;
The external world recognition system according to claim 9, characterized by having:
The external server is
A data analysis unit that statistically analyzes the position information of the plurality of structures on the road surface received from the information providing vehicle, creates driving experience information, and maps the created driving experience information to a road-related map. The external world recognition system according to claim 10, characterized by:
The information providing vehicle includes a vehicle vibration detection unit that detects vehicle vibration in a vertical direction, and a stereo camera as the road surface structure detection unit,
When the vehicle vibration detection unit detects vehicle vibration equal to or greater than a threshold value, the stereo camera extracts a captured image in which a vibration detection point where the vehicle vibration is detected is captured, and position information of the vibration detection point. transmitting a captured image in which the vibration detection point is captured to the external server;
The external server performs image processing for detecting the structure on the road surface from the captured image of the vibration detection point acquired from the own vehicle, and when the structure on the road surface is detected by the image processing, 11. The external world recognition system according to claim 10, wherein the position of the vibration detection point is stored in the server database as the position of the structure on the road surface.
The information providing vehicle calculates the degree of influence on the vehicle when it passes over the structure on the road surface based on the magnitude of the vehicle vibration, and calculates the position information of the vibration detection point and the degree of influence on the vehicle. and the position information of the structure on the road surface to the external server;
The external server stores the position information of the structure on the road surface and the information of the degree of influence on the vehicle in the server database,
When receiving position information of the vibration detection point and information on the degree of influence on the vehicle from the external server, the own vehicle performs vehicle control based on the information on the degree of influence on the vehicle. The external world recognition system according to claim 12.