JP2021076962A - Method and system for generating dynamic map information capable of providing environmental information - Google Patents

Abstract

To provide a method and system for generating dynamic map information by environmental information.SOLUTION: A system includes: a cloud server 30; a plurality of repeating hosts 20 distributed in a peripheral environment; and a plurality of vehicle devices 10 arranged in accordance with respectively different vehicles. The vehicle device includes a LiDAR sensor 11 and a camera 12 for sensing an environment in order to respectively generate point group data and image data. When point group data from different vehicles are transmitted to the adjacent repeating host, the repeating host merges point group data of data of the plurality of vehicles, and obtains 3D coordinate information of an object in an environment in response to the merged data. The cloud server generates dynamic map information with respect to the vehicles based on 3D coordinate information of an object, and transmits the information. Sensing areas of the respective vehicles are extended so as to reduce a dark zone or a dead zone by sharing sensing data of the different vehicles.SELECTED DRAWING: Figure 2

Description

The present invention relates to a method for automatically identifying environmental information, and more particularly to a method for identifying an obstacle by using LiDAR information and image information.

With ever-changing and rapidly evolving artificial intelligence (AI) and machine learning technology, many international companies are adapting to this new machine learning in the research and development of autonomous driving. The in-vehicle autonomous driving system can detect the state of the environment around the vehicle by sensor fusion, and makes an appropriate judgment for controlling the vehicle based on the detection result.

When all the sensing results are obtained by sensors installed in one vehicle to sense the environment, depending on the viewing angle and direction of movement of the vehicle, there is a dark zone for the vehicle when multiple obstacles are adjacent to each other. And dead zones occur.
See, for example, FIG. When the first obstacle O1 and the second obstacle O2 are in front of the vehicle A, the vehicle A can recognize only the first obstacle O1. The second obstacle O2 after the first obstacle O1 is in the dark zone Z with respect to the vehicle A and is undetectable.

Photodetection distance measurement (LiDAR) devices are widely used in autonomous driving systems. This device can quickly sense the surrounding environment and generate point cloud data representing the surrounding environment. According to the point cloud data, three-dimensional (3D) geometric information of the surrounding environment can be obtained.

However, LiDAR still has the above-mentioned problems, and when there are a plurality of objects adjacent to each other, it is impossible to completely obtain the geometric information of the objects. Thus, it is not desirable to later determine if an object is an obstacle.

An object of the present invention is to provide a system for generating dynamic map information capable of providing environmental information. The system includes multiple vehicle devices, multiple relay hosts, and cloud servers.

Each of the plurality of vehicle devices is installed corresponding to a different vehicle. Each of the vehicle devices has a light detection distance measurement (LiDAR) sensor that senses the environment around the vehicle to generate point cloud data, a camera that captures the environment around the vehicle to generate image data, and a LiDAR sensor. It comprises a vehicle controller connected to a camera and a data transmission unit to control point cloud data and image data transmitted through the data transmission unit, and a human-machine interface connected to the vehicle controller.

Each relay host receives point cloud data and image data from vehicles around the relay host itself, and merges the point cloud data received from the vehicle by executing the multiple vehicle data integration mode to create an object in the environment. A processing result including the 3D coordinate information of is generated.

The cloud server communicates with multiple relay hosts and multiple vehicle devices, receives the processing results of the multiple vehicle data integration mode, and integrates the 3D coordinate information of the object with the basemap to generate dynamic map information. The cloud server sends dynamic map information to the human-machine interface of each vehicle device.

Another object of the present invention is to provide a method for generating dynamic map information that can provide environmental information.

The methods are (a) a step of sending a confirmation request from a vehicle device installed in the vehicle to a cloud server, and (b) a relay host among a plurality of relay hosts, and a relay host near the vehicle. The step of notifying the vehicle device of the relay host specified by the cloud server that refers to the location of the vehicle, and (c) transmitting point cloud data and image data from the vehicle to the relay host specified by the vehicle device. The point cloud data is provided by the LiDAR sensor that senses the environment around the vehicle, and the image data is provided by the camera that captures the environment around the vehicle. When receiving point cloud data and image data from a vehicle, the relay host executes the multi-vehicle data integration mode to merge the point cloud data received from multiple vehicles and obtain the 3D coordinate information of the object in the environment. The step of generating the processing result to be included, and (e) the cloud server receives the processing result of the multiple vehicle data integration mode from the relay host, and the 3D coordinate information of the object is used as the base map in order to generate the dynamic map information. The integration step and (f) the cloud server transmitting dynamic map information to the human-machine interface of each vehicle device, where the dynamic map information contains and transmits 3D coordinate information of the object in the environment. Including steps.

By integrating the point cloud data sensed by the LiDAR sensors of different vehicles, the sensing area of each vehicle can be extended to identify more objects around the vehicle. The 3D coordinate information of the object can be combined with the basemap to create dynamic map information. Dynamic map information is transmitted to each vehicle in order to realize the purpose of sharing environmental information.

When the present invention is applied to self-driving vehicles, the preferred dynamic map information is a high definition map (HD MAP). The HD MAP can be used by a control system in an autonomous vehicle to automatically program a safe driving path and avoid collisions with identified obstacles.

It is the schematic of the system by this invention. It is a block diagram of the system by this invention. It is a flowchart of the linkage between a vehicle device, a relay host, and a cloud server according to the present invention. It is a figure which shows two groups of the point cloud data PA and PB obtained by two LiDAR respectively. It is a flowchart of the iteration close point (ICP) algorithm used in this invention. It is a figure which shows the two clusters of the point cloud data PA and PB in FIG. 4 which are aligned with each other by the calculation of the ICP algorithm. It is a figure which shows the point cloud data of the environment obtained by LiDAR by this invention. It is a figure which shows the image data of the environment obtained by the camera by this invention. It is a flowchart of data processing performed by one vehicle. It is a schematic diagram which shows the obstacle identified by a plurality of vehicles according to this invention. It is a schematic diagram which shows the dark zone or the dead zone with respect to a vehicle.

See FIGS. 1 and 2. The system for generating dynamic map information of the present invention includes a vehicle device 10 each of which is installed corresponding to a vehicle, a plurality of relay hosts 20 provided in an environment, preferably near a road, and a cloud server. 30 and.

Each vehicle device 10 comprises a photodetection range (LiDAR) sensor 11, a camera 12, a vehicle controller 13, a data transmission unit 14, and a human-machine interface (HCI) 15.
The LiDAR sensor 11 is connected to the vehicle controller 13 and senses the environment around the vehicle to generate point cloud data. The camera 12 is connected to the vehicle controller 13 and captures an image of the surroundings of the vehicle in order to generate image data. The vehicle controller 13 transmits point cloud data and image data through the data transmission unit 14, and a vehicle identification code unique to the vehicle is pre-stored in the vehicle controller 13 to identify the vehicle among the plurality of vehicles. .. The data transmission unit 14 is a wireless communication unit for data transmission between the vehicle device 10, the relay host 20, and the cloud server 30.

The relay host 20 is constructed and distributed in the surrounding environment to establish vehicle-to-infrastructure (V2I) communication with the vehicle device 10. Each of the relay hosts 20 has a unique host identification code. The relay host 20 also communicates with the cloud server 30.

The cloud server 30 can access the map database 31 in which the basic environment information used as the base map is stored. The cloud server 30 integrates the data processed by the relay host 20 with the base map in the map database 31 to generate dynamic map information, and then transmits the dynamic map information to the human-machine interface 15 of each vehicle device 10. ..

See FIG. The linkage between the vehicle device 10, the relay host 20, and the cloud server 30 includes the following steps.

S31: The vehicle device 10 of the vehicle sends a confirmation request to the cloud server 30 in order to inquire whether or not there is a relay host 20 around the vehicle itself. Regarding the cloud server 30, the cloud server 30 may simultaneously receive a plurality of confirmation requests from different vehicle devices 10.

S32: The cloud server 30 designates a relay host 20 that can be used by the vehicle according to the location of the vehicle that sends the confirmation request, and provides the host identification code of the designated relay host 20 to the vehicle that sends the confirmation request to the cloud. The server 30 can obtain the location of the vehicle to which the confirmation request is sent according to the GPS information.

S33: After the vehicle device 10 that sends the confirmation request receives the reply information from the cloud server 30, the vehicle device 10 transmits the detection data including the point group data and the image data, and the vehicle identification code of the vehicle by the cloud server 30. Upload to the designated relay host 20. If only one vehicle is sending a confirmation request near the relay host 20, the relay host 20 receives only the sensed data from that vehicle. When a plurality of vehicles are sending such confirmation requests near the relay host 20, the relay host 20 receives detection data from the plurality of vehicles.

S34: Based on the sensed data received from each of the vehicle devices 10, the designated relay host 20 should execute and process a "single vehicle data processing mode" or a "multiple vehicle data integration mode". It determines whether or not there is, and transmits the processing result of either the single vehicle data processing mode or the multiple vehicle data integration mode, and the host identification code to the cloud server 30.
If there is only one vehicle around the relay host 20, the relay host 20 receives the sensed data from only a single vehicle device 10 and executes a "single vehicle data processing mode". On the other hand, when there are two or more different vehicles around the relay host 20, the relay host 20 receives a plurality of sensed data from the different vehicles and executes the "multiple vehicle data integration mode". The single vehicle data processing mode and the multiple vehicle data integration mode will be described in detail later.

S35: The cloud server 30 receives the processing result from the relay host 20 and combines the processing data with the base map to generate dynamic map information. Since the processing result includes the three-dimensional (3D) coordinate information of the obstacle near the vehicle, the dynamic map information will include the coordinate information of the object near the vehicle accordingly.

S36: The cloud server 30 transmits the dynamic map information to the human-machine interface 15 of the vehicle again. When transmitting the dynamic map information, the cloud server 30 determines the receiving vehicle to which the dynamic map information should be transmitted according to the vehicle identification code.

Regarding the multi-vehicle data integration mode in step S34, when the relay host 20 receives data from two or more different vehicles, the points cloud for the same object sensed by the LiDAR sensor 11 on the vehicle because the vehicles are in different locations. The data may be different from each other.
FIG. 4 schematically shows a first group PA of point cloud data and a second group PB of point cloud data for the same object. The first group PA of the point cloud data and the second group PB of the point cloud data are generated by two LiDAR sensors 11 located at different positions, and each group PA and PB of the point cloud data is composed of a plurality of data points. Will be done. The present invention utilizes an iterative nearest point (ICP) algorithm to reduce the distance between each data point in two groups PA, PB of point cloud data. In other words, the data points in the two groups PA and PB of the point cloud data can be aligned as close to each other as possible.

See FIG. The iterative nearest point (ICP) algorithm is not a major feature of the present invention. The process of the ICP algorithm mainly includes:
S51: Step of obtaining the first group PA of the point group data and the second group PB of the point group data S52: Correlation between the first group PA of the point group data and the second group PB of the point group data Step S53: Determining data points that match each other in two groups PA and PB of point group data in order to obtain sex S53: Step S54 of calculating a conversion matrix according to the correlation between two groups PA and PB of point group data. : Performing a transformation to transform the coordinates of each data point in the first group PA of the point group data by the transformation matrix S55: The current squared mean square root value in the current transformation, and the previous transformation Step of calculating the squared mean square root value to obtain the error between the two squared mean square root values S56: Determine if the calculated error is less than the default threshold, and if it is smaller, the first group of point group data The PA is aligned with the second group PB of the point group data to finish the calculation, and if it is not small, the step of repeating step S52.

See FIG. The two groups PA and PB of the point cloud data are aligned with each other after performing the ICP calculation. In other words, when the sensed data measured by the LiDAR sensors 11 on the two vehicles have a common sensed area, the sensed data can be merged with each other and the coordinate error between the sensed data can be reduced. Since the sensing areas of different vehicles are different from each other, it is possible to obtain a wider sensing area when the vehicle sensing areas are combined, and the dead zone or dark zone for each vehicle can be reduced.
When the relay host 20 finishes the integration of the plurality of vehicle data, the integrated point cloud data will be transmitted to the cloud server 30 for the calculation of the map information. Since each data point in the point cloud data represents 3D information, the integrated point cloud data may represent the 3D coordinate information of the object.

Regarding the single vehicle data processing mode in step S34, when the relay host 20 receives only the sensed data from one vehicle, the relay host 20 processes the point cloud data and the image data provided by the vehicle.
As shown in FIGS. 7A and 7B, the LiDAR sensor 11 and the camera 12 can provide time point group data and image data to detect the same environment, respectively. According to the image data, it can be seen that there is a pedestrian behind the car. However, since the pedestrian is partially hidden by the car, it is difficult to determine whether or not the pedestrian exists according to the point cloud data. In order to accurately recognize the hidden object, as shown in FIG. 8, the relay host 20 executes a single vehicle data processing mode having the following steps.

S81: For example, the first join box for the object in the image data is identified. The object may be a vehicle object, a pedestrian object, a pseudo-pedestrian object such as a motorcycle rider or a bicycle rider, and the coupling box can be recognized by known image recognition techniques.

S82: Objects in the point cloud data are identified to obtain their second combined box, and the second combined box is mapped to the image data in order to compare the image data with the point cloud data.

S83: Determines whether the object identified by the second join box in the point cloud data exists in the image data, and if the same object exists, the object belongs to the bright region object and each data in the point cloud data. Since the points represent 3D information, the 3D coordinate information of the bright region object can be obtained based on the point cloud data. For example, a vehicle is a type of bright region object because it can be identified in image data and point cloud data. If the identified vehicle in the image data is the same in the point cloud data, the 3D coordinate information of the vehicle can be obtained based on the point cloud data.

S84: It is determined whether or not the image data is an object that is not identified in the point cloud data. If there is an object that is not identified in the point cloud data but exists in the image data, that object belongs to the dark area object. For example, the pedestrian behind the vehicle in the point cloud data may belong to the dark area object. When a pedestrian is identified in the image data, the pedestrian is identified as a dark area object.

S85: The first combination box of the dark area object is mapped to the point cloud data to obtain the 3D coordinate information of the dark area object. Since the pedestrian and its first combined box can be confirmed in step S84 described above, the coordinates of the first combined box identified in the image data are such that they are located at the positions of the dark area objects in the point cloud data. It will be mapped to the corresponding coordinates in the point cloud data.

After the relay host 20 exits the single vehicle data processing mode, it is possible to obtain 3D coordinate information of the bright area object and the dark area object from the point cloud data. The 3D coordinate information of the object will be transmitted to the cloud server 30 for the calculation of the dynamic map information.

When the cloud server 30 receives the 3D coordinate information of the object calculated by the relay host 20, the cloud server 30 combines the 3D coordinate information with the base map to generate dynamic map information, and then generates the dynamic map information, respectively. It transmits to the human-machine interface 15 of the vehicle.
In another embodiment, the dynamic map information is a high definition map (HD MAP) for automatic control of an autonomous vehicle. In yet another embodiment, the dynamic map information is a map visible to the vehicle driver.

In short, by integrating the sensed data from different vehicles or one vehicle, the present invention can achieve the following effects.

1. 1. Sense optimizations for dark or dead zones.
See FIG. 9A. The vehicle A itself can detect only the first object O1, but cannot recognize the second object O2 in the dark zone Z. However, the sensing area for the vehicle A can be effectively expanded by integrating the point cloud data from different vehicles according to the present invention, for example, to integrate the sensing data of the vehicle B capable of detecting the second object O2. After providing the integrated point cloud data to each of the vehicles, the vehicle A can successfully recognize the second object O2 in the dark zone. For self-driving vehicles, their sensing capabilities can be improved to enhance the programming of the driving path.

2. Increase the dynamic information of high-definition maps.
Through vehicle-to-infrastructure (V2I) communication, each relay host 20 can receive various types of dynamic information from various vehicles, such as dark zones, obstacle identification, and travelable intervals. The integrated dynamic map information includes real-time information as a reference for autonomous vehicles to determine preferred driving routes.

3. 3. Distributed data computing architecture.
Each of the relay hosts 20 performs front-end data processing in order to reduce the computational load of the cloud server 30, so that each vehicle can obtain environmental information in real time and quickly.

Each of the plurality of vehicle devices is installed corresponding to a different vehicle. Each of the vehicle devices has a light detection distance measurement (LiDAR) sensor that senses the environment around the vehicle to generate point cloud data, a camera that captures the environment around the vehicle to generate image data, and a LiDAR sensor. and a mechanical interface - camera, and is connected to the data transmission unit, and the vehicle controller for controlling the point cloud data and the image data is transmitted through the data transmission unit, between human.

Claims (14)

A system for generating dynamic map information that can provide environmental information.
A plurality of vehicle devices, each of which is installed corresponding to a different vehicle, each of which is a light detection distance measuring (LiDAR) sensor that senses the environment around the vehicle in order to generate point cloud data. , The point cloud data and the image data connected to the camera, the LiDAR sensor, the camera, and the data transmission unit that capture the environment around the vehicle to generate image data, and transmitted through the data transmission unit. A vehicle controller that controls the vehicle and a plurality of vehicle devices having a human-machine interface connected to the vehicle controller.
A plurality of relay hosts, each of which receives the point cloud data and the image data from the vehicle around the relay host itself, and executes the multiple vehicle data integration mode from the vehicle. A plurality of relay hosts that merge the received point cloud data to generate a processing result containing 3D coordinate information of the object in the environment.
A cloud that communicates with the plurality of relay hosts and the plurality of vehicle devices, receives the processing results of the plurality of vehicle data integration modes, and integrates the 3D coordinate information of the object with the base map in order to generate dynamic map information. A server comprising a cloud server that transmits the dynamic map information to the human-machine interface of each vehicle device.
system. When the relay host receives the point cloud data and the image data from only a single vehicle, the relay host executes a single vehicle data processing mode to generate the 3D coordinate information of the object in the environment. The system according to claim 1, wherein the 3D coordinate information is provided to the cloud server. The system according to claim 1 or 2, wherein the dynamic map information generated by the cloud server is high-definition (HD) map information. The system according to claim 1 or 2, wherein the relay host merges the point cloud data using the iterative Closet Point (ICP) algorithm in the multi-vehicle data integration mode. The point group data received by the relay host includes a first group of point group data including a plurality of data points sensed by the first vehicle, and a plurality of data points sensed by the second vehicle. The ICP algorithm for the multi-vehicle data integration mode comprises a second group of point group data comprising (a) a step of obtaining a first group of the point group data and a second group of the point group data. And (b) the first group of the point group data and the first group of the point group data in order to obtain the correlation between the first group of the point group data and the second group of the point group data. The conversion matrix is calculated according to the step of determining the data points that match each other in the two groups and (c) the correlation between the first group of the point group data and the second group of the point group data. Steps to: (d) perform transformations to transform the coordinates of each data point in the first group of point group data by the transformation matrix, and (e) the current square of the current transformation. The step of calculating the mean square root value and the previous squared mean square root value of the previous transformation to obtain the error between the current squared mean square root value and the previous squared mean square root value, and (f) calculation. It is determined whether the error is smaller than the default threshold, and if it is smaller, the first group of the point group data is aligned with the second group of the point group data and the multiple vehicle data integration mode is terminated. However, the system according to claim 4, further comprising a step of repeating the step (b) if not small. The system according to claim 1 or 2, wherein the cloud server accesses a map database in which the basemap is stored. A method for generating dynamic map information that can provide environmental information.
(A) The step of sending a confirmation request from the vehicle device installed in the vehicle to the cloud server,
(B) The said cloud server designated by the cloud server that refers to the location of the vehicle to specify a relay host among a plurality of relay hosts and to specify the relay host near the vehicle to which the confirmation request is sent. The step of notifying the relay host to the vehicle device sending the confirmation request, and
(C) A step of transmitting point cloud data and image data from the vehicle to the relay host designated by the vehicle device, wherein the point cloud data is provided by a LiDAR sensor that senses the environment around the vehicle. The image data is provided by a camera that captures the environment around the vehicle, with steps to transmit.
(D) When the relay host receives the point cloud data and the image data from a plurality of vehicles, the relay host executes the multiple vehicle data integration mode to receive the points received from the plurality of vehicles. A step of merging the point cloud data to generate a processing result containing the 3D coordinate information of the object in the environment, and
(E) A step of receiving the processing result of the plurality of vehicle data integration mode from the relay host by the cloud server and integrating the 3D coordinate information of the object with the base map in order to generate dynamic map information.
(F) A step of transmitting the dynamic map information to the human-machine interface of the vehicle device by the cloud server, wherein the dynamic map information includes 3D coordinate information of the object in the environment. And, including
Method. When the relay host receives the point cloud data and the image data from only a single vehicle, the relay host executes a single vehicle data processing mode to generate the 3D coordinate information of the object in the environment. The method according to claim 7, wherein the 3D coordinate information is provided to the cloud server. When the cloud server notifies the vehicle device of the designated relay host, the cloud server transmits the host identification code of the designated relay host to the vehicle device, and the vehicle device identifies the host. 7. The point group data and the image data are transmitted to the designated relay host according to the code, and the vehicle identification code of the vehicle device is further transmitted to the designated relay host. 8. The method according to 8. The method of claim 7, wherein the relay host merges the point cloud data using the iterative Closet Point (ICP) algorithm in the multi-vehicle data integration mode. The point group data received by the relay host includes a first group of point group data including a plurality of data points sensed by the first vehicle, and a plurality of data points sensed by the second vehicle. The ICP algorithm for the multi-vehicle data integration mode comprises a second group of point group data comprising (a) a step of obtaining a first group of the point group data and a second group of the point group data. And (b) the first group of the point group data and the first group of the point group data in order to obtain the correlation between the first group of the point group data and the second group of the point group data. The conversion matrix is calculated according to the step of determining the data points that match each other in the two groups and (c) the correlation between the first group of the point group data and the second group of the point group data. Steps to: (d) perform transformations to transform the coordinates of each data point in the first group of point group data by the transformation matrix, and (e) the current square of the current transformation. The step of calculating the mean square root value and the previous squared mean square root value of the previous transformation to obtain the error between the current squared mean square root value and the previous squared mean square root value, and (f) calculation. It is determined whether the error is smaller than the default threshold, and if it is smaller, the first group of the point group data is aligned with the second group of the point group data and the multi-vehicle data integration mode is terminated. However, the method according to claim 10, further comprising a step of repeating the step (b) if not small. The single vehicle data processing mode executed by the relay host identifies the first join box for the object in the image data to obtain the position of the object in the image data and the object in the point group data. A step of identifying to obtain a second join box of the object in the point group data and determining whether the object identified by the second join box in the point group data exists in the image data. In the step, when the same object exists in the point group data and the image data, it is determined that the same object belongs to the bright area object and the 3D coordinate information of the bright area object is obtained according to the point group data. A step and a step of determining whether or not the image data contains an object that is not identified in the point group data, and if there is an object that is not identified in the point group data but exists in the image data, the object is It is characterized by including a step of determining belonging to the dark area object and a step of mapping the first connection box of the dark area object to the point group data to obtain 3D coordinate information of the dark area object. The method according to claim 8. 12. The method of claim 12, wherein the object identifiable in the image data includes a vehicle object, a pedestrian object, and a pseudo-pedestrian object. The method according to claim 7 or 8, wherein the dynamic map information generated by the cloud server is high-definition (HD) map information.

Images