EP4457776A1 - Method and apparatus for generating ground truth for driving boundaries - Google Patents

Method and apparatus for generating ground truth for driving boundaries

Info

Publication number
EP4457776A1
EP4457776A1 EP22814111.5A EP22814111A EP4457776A1 EP 4457776 A1 EP4457776 A1 EP 4457776A1 EP 22814111 A EP22814111 A EP 22814111A EP 4457776 A1 EP4457776 A1 EP 4457776A1
Authority
EP
European Patent Office
Prior art keywords
image information
vehicle
ground truth
boundaries
generating
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP22814111.5A
Other languages
German (de)
French (fr)
Inventor
Andreas Wimmer
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Robert Bosch GmbH
Original Assignee
Robert Bosch GmbH
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Robert Bosch GmbH filed Critical Robert Bosch GmbH
Publication of EP4457776A1 publication Critical patent/EP4457776A1/en
Pending legal-status Critical Current

Links

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/02—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to ambient conditions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2420/00—Indexing codes relating to the type of sensors based on the principle of their operation
    • B60W2420/40—Photo, light or radio wave sensitive means, e.g. infrared sensors
    • B60W2420/403—Image sensing, e.g. optical camera

Definitions

  • the present invention relates to the field of vehicles, in particular to a method and an apparatus for generating a ground truth for driving boundaries, a computer storage medium, and a computer program product vehicle.
  • a manual labeling method is used to generate the ground truth, it will lead to high labor costs and take a long time. If information collected by a lidar sensor is used to generate the ground truth, the precision of the ground truth cannot be guaranteed in some scenarios such as rain, snow, foggy weather. If the same sensor (e.g., a front facing camera), which is used for the sensed value to be validated, is used to generate the ground truth, the precision of the validation result is also difficult to be guaranteed. This is because errors or omissions caused by the inherent defects of the sensor would exist in both the ground truth and the sensed value generated by the same sensor. Therefore, these errors or omissions cannot be detected by comparing the two.
  • a lidar sensor e.g., a front facing camera
  • a method for generating a ground truth for driving boundaries comprises: receiving first image information about surroundings of a vehicle from first-type cameras mounted on the vehicle; extracting second image information about driving boundaries from the first image information; and generating the ground truth for the driving boundaries based at least on the second image information, wherein the ground truth is used to validate a sensed value about the driving boundaries collected by a second-type camera mounted on the vehicle.
  • the ground truth is further used to validate a sensed value about the driving boundaries collected by other types of sensors mounted on the vehicle.
  • the foregoing method may further comprise performing a coordinate system conversion on the second image information; and generating the ground truth based at least on the converted second image information.
  • the second image information is converted from a two-dimensional image coordinate system to a three-dimensional vehicle coordinate system.
  • the first-type cameras mounted on the vehicle comprise at least two first-type cameras located at different positions of the vehicle.
  • the method further comprises: extracting second images of the at least two first-type cameras from first image information from the at least two first-type cameras, respectively; fusing the second images of the at least two first-type cameras; and generating the ground truth based at least on fused second image information.
  • the foregoing method further comprises: fusing, based on positioning information of the vehicle at different times, the second image information about the driving boundaries extracted at the different times; and generating the ground truth based at least on the fused second image information.
  • the ground truth is generated based on the second image information and third image information about the driving boundaries from other types of sensors.
  • an apparatus for generating a ground truth for driving boundaries comprises: a receiving device configured to receive first image information about surroundings of a vehicle from first-type cameras mounted on the vehicle; an extracting device configured to extract second image information about driving boundaries from the first image information; and a generating device configured to generate the ground truth for the driving boundaries based at least on the second image information, wherein the ground truth is used to validate a sensed value about the driving boundaries collected by a second-type camera mounted on the vehicle.
  • the ground truth is further used to validate a sensed value about the driving boundaries collected by other types of sensors mounted on the vehicle.
  • the foregoing apparatus further comprises a conversion device.
  • the conversion device is configured to perform a coordinate system conversion on the second image information.
  • the generating device is further configured to generate the ground truth based at least on the converted second image information.
  • the conversion device is further configured to convert the second image information from a two-dimensional image coordinate system to a three-dimensional vehicle coordinate system.
  • the foregoing apparatus further comprises a first fusion device.
  • the first-type cameras mounted on the vehicle comprise at least two first-type cameras located at different positions of the vehicle.
  • the extracting device is further configured to extract second images of the at least two first-type cameras from the first image information from the at least two first-type cameras, respectively.
  • the first fusion device is configured to fuse the second images of the at least two first-type cameras.
  • the generating device is further configured to generate the ground truth based at least on the fused second image information.
  • the foregoing apparatus further comprises a second fusion device.
  • the second fusion device is configured to fuse, based on positioning information of the vehicle at different times, the second image information about the driving boundaries extracted at the different times.
  • the generating device is further configured to generate the ground truth based at least on the fused second image information.
  • the generating device is further configured to generate the ground truth based on the second image information and third image information about the driving boundaries from other types of sensors.
  • a computer storage medium comprises instructions for implementing the forgoing method when being executed.
  • a computer program product which comprises a computer program.
  • the computer program when executed by a processor, implements the foregoing method.
  • a vehicle comprising the foregoing apparatus is provided.
  • the solution for generating driving boundaries uses first-type cameras to collect image information for driving boundaries, and processes the collected image information to generate a ground truth.
  • the solution for generating driving boundaries is precise, saves time and labor costs, and can flexibly fuse image information collected by other types of sensors.
  • FIG. 1 shows a schematic flowchart of a method 1000 for generating a ground truth for driving boundaries according to an embodiment of the present invention.
  • FIGS. 2(a)-(d) show first image information received from four vehicle-mounted first-type cameras, respectively.
  • FIG. 3 shows a schematic structural diagram of an apparatus 3000 for generating a ground truth for driving boundaries according to an embodiment of the present invention.
  • FIG. 1 shows a schematic flowchart of a method 1000 for generating a ground truth for driving boundaries according to an embodiment of the present invention.
  • the method 1000 for generating a ground truth for driving boundaries comprises the following steps.
  • step SI 10 first image information about surroundings of a vehicle is received from vehicle-mounted first-type cameras.
  • driving boundaries refers to road boundaries within which a vehicle can travel, for example, lane markings, botts dots, curbs, guard rails, concrete or steel walls, the border between asphalt and grass, reflectors, barriers, beacons, pylons or other structures used to delineate lanes or to delineate different kind of pavement.
  • first-type camera refers to a camera that is different from a second-type camera to be validated.
  • a second-type camera to be validated may be a front facing camera used in an Advanced Driving Assistant System (ADAS).
  • ADAS Advanced Driving Assistant System
  • a first-type camera may be a fisheye camera mounted on the vehicle, or a wing camera mounted on the vehicle.
  • a fisheye camera may be a camera mounted on the vehicle originally for the reversing function.
  • a fisheye camera can have a higher resolution for a sensing object in a short distance, so that a reference value with a higher precision is generated in the subsequent step S130.
  • wing cameras may be cameras mounted on both sides of the vehicle (for example, on the rearview mirrors on both sides) for sensing images on both sides of the vehicle.
  • First image information may be received from one or more first-type cameras mounted on the vehicle.
  • FIGS. 2(a)-(d) show first image information received from four vehicle-mounted first-type cameras (fish-eye cameras are specifically provided in this embodiment), respectively.
  • the four vehicle-mounted first-type cameras corresponding to FIGS. 2(a)-(d) are mounted on the front, rear, left, and right sides of the vehicle, respectively.
  • the first image information collected by the four vehicle-mounted first-type cameras each comprises lane markings, and both the first-type camera mounted on the front side (corresponding to FIG. 2(a)) and the first-type camera mounted on the right side (corresponding to FIG. 2(d)) collect the image information of lane markings 210.
  • a fisheye camera can collect image information with a large field of view.
  • an image with such a large field of view has a certain degree of distortion, and therefore, the image information can be corrected and compensated accordingly.
  • step S120 second image information about driving boundaries is extracted from the first image information received in step SI 10.
  • a ground truth for driving boundaries is generated based at least on the second image information extracted in step SI 20.
  • the generated ground truth can be used to compare with a sensed value of the same driving boundaries collected by other vehicle-mounted sensors to test and calculate the precision of other vehicle-mounted sensors' perception of the driving boundaries, and their perception performance in error triggering events.
  • error triggering event is intended to mean typical events in which other vehicle-mounted sensors are prone to perception errors, such as rain, snow, and fog scenarios.
  • the ground truth generated in step S130 may also be used to validate the sensed value about the driving boundaries collected by other types of sensors mounted on the vehicle.
  • Other types of sensors may be, for example, lidar sensors, millimeter-wave radar sensors, other types of cameras other than the first and second types, and any suitable types of sensors mounted on the vehicle.
  • the ground truth for driving boundaries is generated based on the image information of the driving boundaries collected by the first-type cameras, thereby providing a benchmark for validating the precision of the vehicle-mounted cameras or other types of sensors.
  • common cause error means that the same error caused by the same factor exists in multiple sensing results by the same sensor or the same type of sensors. The same factor may be originated from the location of this sensor, or from the inherent defects of this type of sensors, for example.
  • different types of sensors are used, which often have different focal lengths, the image information collected by which is often processed by different algorithms, and which are often mounted in different locations, thus having different lighting conditions. Therefore, the first-type cameras, which are different from the sensors to be validated, can be used to avoid common cause errors from being overlooked in validation.
  • a camera capable of collecting high-quality image information of the driving boundaries near the vehicle such as a fisheye camera, can be used as the first-type camera, so as to obtain a ground truth for driving boundaries with high precision. This is especially apparent in scenes such as rain, snow, and fog.
  • the method 1000 for generating a ground truth for driving boundaries may further comprise a coordinate system conversion of the image information.
  • the second image information extracted in step 120 is converted from a two-dimensional image coordinate system to a three-dimensional vehicle coordinate system, so as to facilitate further processing of the second image information (for example, to compare and fuse with three-dimensional image information collected by other sensors).
  • the ground truth for driving boundaries is generated based at least on the converted second image information.
  • coordinate system conversion of image information is not limited to the conversion of the second image information, and conversion may also be performed on image information generated in other steps (e.g., the first image information).
  • coordinate system conversion may first be performed on the first image information before the extraction step S120. In some cases, this is beneficial to outlier detection and plausibilisation of image information.
  • coordinate system conversion of image information is not limited to the conversion from image coordinate system to vehicle coordinate system, but may also cover mutual conversions among the various coordinate systems of camera coordinate system, image coordinate system, world coordinate system, vehicle coordinate system, and the like. It depends on the specific image processing requirements.
  • first image information about surroundings of a vehicle can be received from multiple first-type cameras mounted at different positions of the vehicle.
  • second image information can be extracted from the first image information from each first-type camera.
  • second image information can be extracted in FIGS. 2(a)-(d), respectively.
  • the method 1000 may further comprise fusing the second image information of each first-type camera (not shown in FIG. 1).
  • a ground truth for driving boundaries can be generated based at least on the fused second image information.
  • the first image information from multiple first-type cameras can also be fused before the second image information is extracted. Then, in step S120, the second image information is extracted from the fused first image information, and in step S130, a ground truth for driving boundaries is generated based on the second image information.
  • the precision of the generated ground truth for driving boundaries can be further improved. This is especially apparent for the overlapping parts of the fields of view of multiple first-type cameras.
  • the image information collected by all of the vehicle-mounted first-type cameras can be fused, or the image information collected by a subset of the first-type cameras can be fused.
  • the image information in FIGS. 2(a) and (d) can be fused to generate the ground truth of the lane markings 210. That is, the image information collected by a subset of the four first-type cameras (the front one and the right one) is fused to generate the ground truth of the lane markings 210.
  • the method 1000 for generating a ground truth for driving boundaries may also comprise fusing, based on positioning information of a vehicle at different times, second image information about the same driving boundaries extracted at the different times. Accordingly, in step S130, a ground truth for driving boundaries is generated based at least on the fused second image information.
  • the processing of the foregoing steps is usually carried out in a time frame manner. Even when the vehicle is traveling, the image information of driving boundaries at the same location around the vehicle generally does not exist in a single time frame alone, but in multiple time frames before and after. Thus, fusing, based on the positioning information of a vehicle at multiple times, the image information of the same driving boundaries collected at the multiple times, can compensate for errors and omissions in single-frame image information and effectively improve the precision of ground truth for driving boundaries that is ultimately generated.
  • positioning information of a vehicle at different times can be determined by means of global positioning, such as global navigation satellite system (GNSS), global positioning system (GPS); it can also be determined by the positioning methods of the vehicle itself, such as onboard odometry sensor, which determines the position change of the vehicle by determining the change in the distance between the vehicle and a reference object; it can also be determined by a combination of any of the above methods.
  • GNSS global navigation satellite system
  • GPS global positioning system
  • fusion of the image information of the same driving boundaries collected at different times is not limited to following the extraction of the second image information, while it can also occur before the extraction of the second image information.
  • the first image information of the same driving boundaries collected at different times can first be fused.
  • second image information is extracted from the fused first image information, and in step SI 30, a ground truth for driving boundaries is generated based on the second image information.
  • the method 1000 for generating a ground truth for driving boundaries may further comprise generating a ground truth for the driving boundaries based on second image information from first-type cameras and third image information about the same driving boundaries from other types of sensors.
  • other types of sensors can be any types of sensors that can collect the image information of the same driving boundaries, such as lidar sensors, millimeter-wave radar sensors, other types of cameras, and the like, or a combination of the foregoing various types of sensors.
  • Generating a ground truth using image information collected by different types of sensors can prevent the inherent defects of a single type of sensor from affecting the generated ground truth, and further improve the precision of the ground truth for driving boundaries that is ultimately generated.
  • FIG. 1 and the above description describe the various operations (or steps) as sequential processing, many of these operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of various operations can be rearranged. Moreover, the embodiments of the present invention may also include additional steps not included in FIG. 1 and the above description.
  • the method of generating a ground truth for driving boundaries can be implemented by a computer program.
  • the computer program is included in a computer program product.
  • the method for generating a ground truth for driving boundaries according to one or more embodiments of the present invention is implemented.
  • a computer storage medium such as a USB flash drive
  • the method for generating a ground truth for driving boundaries according to one or more embodiments of the present invention can be implemented by executing the computer program .
  • FIG. 3 shows a schematic structural diagram of an apparatus 3000 for generating a ground truth for driving boundaries according to an embodiment of the present invention.
  • the apparatus 3000 for generating a ground truth for driving boundaries comprises: a receiving device 310, an extracting device 320 and a generating device 330.
  • the receiving device 310 is configured to receive first image information about surroundings of a vehicle from vehicle-mounted first-type cameras.
  • the receiving device 310 may be configured to receive the first image information respectively from four vehicle-mounted first-type cameras described above in conjunction with FIG. 2.
  • the extracting device 320 is configured to extract second image information about driving boundaries from the first image information.
  • the generating device 330 is configured to generate a ground truth for driving boundaries based at least on the second image information.
  • driving boundaries refers to road boundaries within which a vehicle can travel, such as lane markings, botts dots, curbs, guard rails, concrete or steel walls, the border between asphalt and grass, reflectors, barriers, beacons, pylons or other structures used to delineate lanes or to delineate different kind of pavement.
  • first-type camera refers to a camera that is different from a second-type camera to be validated.
  • a second-type camera to be validated may be a front facing camera used in an Advanced Driving Assistant System (ADAS).
  • ADAS Advanced Driving Assistant System
  • a first-type camera may be a fisheye camera mounted on the vehicle, or a wing camera mounted on the vehicle.
  • a fisheye camera may be a camera mounted on the vehicle originally for the reversing function.
  • a fisheye camera can have a higher resolution for a sensing object in a short distance, so that a reference value with a higher precision is generated in the subsequent step SI 30.
  • wing cameras may be cameras mounted on both sides of the vehicle (for example, on the rearview mirrors on both sides) for sensing images on both sides of the vehicle.
  • the apparatus 3000 for generating a ground truth for driving boundaries may further comprise a correction device.
  • the correction device may be configured to perform correction and compensation on the first image information received by the receiving device 310.
  • a correction device can be used to correct the distortion in the first image information collected by a fisheye camera.
  • the extracting device 320 conventional image processing methods can be utilized to extract the second image information, for example, the edge filter algorithm, the Canny edge detection algorithm, the Sobel Operator edge detection algorithm, and the like.
  • Machine learning, artificial intelligence algorithms can also be utilized to extract the second image information, for example, neural networks, deep learning, and the like. The present invention does not make any limitations in this regard.
  • the apparatus 3000 for generating a ground truth for driving boundaries may further comprise a conversion device.
  • the conversion device is configured to perform a coordinate system conversion on the second image information.
  • the generating device 330 is configured to generate a ground truth for driving boundaries based at least on the converted second image information.
  • the second image information extracted by the extraction device 320 is converted from a two-dimensional image coordinate system to a three-dimensional vehicle coordinate system, so as to facilitate further processing of the second image information (for example, to be fused with three-dimensional image information collected by other sensors).
  • a ground truth for driving boundaries is generated based at least on the converted second image information.
  • coordinate system conversion of image information is not limited to the conversion of the second image information, and conversion may also be performed on the image information generated in other steps (e.g., the first image information).
  • coordinate system conversion of image information is also not limited to the conversion from image coordinate system to vehicle coordinate system, but may also cover mutual conversions among the various coordinate systems of camera coordinate system, image coordinate system, world coordinate system, vehicle coordinate system, and the like. It depends on the specific image processing requirements.
  • the apparatus 3000 for generating a ground truth for driving boundaries may further comprise a first fusion device.
  • the receiving device 310 may receive first image information about surroundings of a vehicle from multiple first-type cameras mounted at different positions of the vehicle.
  • second image information can be respectively extracted from the first image information of each first-type camera.
  • second image information can be respectively extracted from FIGS. 2(a)-(d).
  • the first fusion device may be configured to fuse the second image information of each first-type camera.
  • the generating device 330 may be configured to generate a ground truth for driving boundaries based at least on the fused second image information.
  • the first fusion device may also be configured to fuse first image information from multiple first-type cameras.
  • the extracting device 320 extracts second image information from the first image information fused by the first fusion device.
  • the generating device may be configured to generate a ground truth for driving boundaries based on the second image information.
  • the first fusion device can fuse the image information collected by all of the vehicle-mounted first-type cameras, or can fuse the image information collected by a subset of the first-type cameras.
  • the image information in FIGS. 2(a) and (d) can be fused to generate the ground truth of the lane markings 210. That is, the image information collected by a subset of the first-type range cameras (the front one and the right one) is fused to generate the ground truth of the lane markings 210.
  • the apparatus 3000 for generating a ground truth for driving boundaries may further comprise a second fusion device.
  • the second fusion device is configured to fuse, based on positioning information of a vehicle at different times, second image information about the same driving boundaries extracted at the different times.
  • the generating device 330 is configured to generate a ground truth for driving boundaries based at least on the fused second image information.
  • each device in the apparatus 3000 is usually performed in a time frame manner. Even when the vehicle is traveling, the image information of driving boundaries at the same position around a vehicle generally does not exist in a single time frame alone, but in multiple time frames before and after. Thus, fusing image information of the same driving boundaries collected at multiple times using the second fusion device can compensate for errors and omissions in single-frame image information, and effectively improve the precision of the ground truth of driving boundaries that is ultimately generated.
  • the fusion of the image information of the same driving boundaries collected at different times by the second fusion device is not limited to following the extraction of the second image information by the extraction device 320, while it can also occur before the extraction of the second image information.
  • the second fusion device can fuse the first image information of the same driving boundaries collected at different times. Then, in the extracting device 320, the second image information is extracted from the first image information fused by the second fusion device, and in the generating device 330, a ground truth for driving boundaries is generated based on the second image information.
  • the positioning information at different times of the vehicle, which is used by the second fusion device can be determined by means of global positioning, such as Global Navigation Satellite System (GNSS), Global Positioning System (GPS); it can also be determined by the positioning methods of the vehicle itself, such as onboard odometry sensor, which determines the position change of the vehicle by determining the change in the distance between the vehicle and a reference object; it can also be determined by a combination of any of the above methods.
  • GNSS Global Navigation Satellite System
  • GPS Global Positioning System
  • the positioning information at different times of the vehicle can be determined by means of global positioning, such as Global Navigation Satellite System (GNSS), Global Positioning System (GPS); it can also be determined by the positioning methods of the vehicle itself, such as onboard odometry sensor, which determines the position change of the vehicle by determining the change in the distance between the vehicle and a reference object; it can also be determined by a combination of any of the above methods.
  • the present invention does not make any limitations in this regard.
  • the generating device 330 may be further configured to generate a ground truth for driving boundaries based on second image information about driving boundaries from first-type cameras and third image information about the same driving boundaries from other types of sensors.
  • other types of sensors can be any types of sensors that can collect the image information of the same driving boundaries, such as lidar sensors, millimeter-wave radar sensors, other types of cameras, and the like, or a combination of the foregoing various types of sensors. Generating a ground truth using image information collected by different types of sensors can prevent the inherent defects of a single type of sensor from affecting the generated ground truth, and further improve the precision of the ground truth for driving boundaries that is ultimately generated.
  • the apparatus 3000 for generating a ground truth for driving boundaries may be integrated into a vehicle.
  • the apparatus 3000 for generating a ground truth for driving boundaries may be an apparatus separately used for generating ground truth for driving boundaries in the vehicle, or it may be combined into an electronic control unit (ECU), a domain control unit (DCU) or other processing apparatus in the vehicle.
  • ECU electronice control unit
  • DCU domain control unit
  • vehicle or other similar terms used herein include general motor vehicles, such as passenger vehicles (including sports utility vehicles, buses, trucks, etc.), various commercial vehicles, etc., and also include hybrid vehicles, electric vehicles, etc.
  • a hybrid vehicle is a vehicle with two or more power sources, such as gasoline and electric powered vehicles.
  • the apparatus 3000 for generating a ground truth for driving boundaries may be integrated into the advanced driving assistance system (ADAS) or into other L0-L5 automatic drive functions of a vehicle.
  • ADAS advanced driving assistance system
  • the ground truth for driving boundaries generated according to the aforementioned embodiments of the present invention can be used as a benchmark to compare with the sensed value for driving boundaries sensed by other vehicle-mounted sensors. Such comparison can be used to validate and calculate the precision, average availability (e.g., true positive rate, true negative rate), and average unavailability rate (e.g., false positive rate, false negative rate) of the sensed results for driving boundaries sensed by other vehicle-mounted sensors. Such comparison can also be used for error triggering events to validate the performance of other vehicle-mounted sensors in error triggering events.
  • the solution for generating driving boundaries uses first-type cameras to collect image information of the driving boundaries, and processes the collected image information to generate a highly precise ground truth for driving boundaries.
  • Generating a ground truth using first-type cameras instead of other sensors, which are to be validated can increase the redundancy of the system, and prevent common cause errors.
  • generating a ground truth using image information collected by first-type cameras can greatly reduce time and labor costs.
  • the solution for generating driving boundaries according to the embodiments of the present invention can also generate a ground truth by combining vehicle-mounted first-type cameras with other types of sensors, which can further improve the precision of the ground truth.
  • a camera capable of collecting high-quality image information of the driving boundaries near the vehicle such as a fisheye camera, can be used as the first-type camera, so as to obtain a ground truth for driving boundaries with high precision. This is especially apparent in scenes such as rain, snow, and fog.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Automation & Control Theory (AREA)
  • Mathematical Physics (AREA)
  • Transportation (AREA)
  • Mechanical Engineering (AREA)
  • Traffic Control Systems (AREA)
  • Image Processing (AREA)

Abstract

The present invention relates to a method for generating a ground truth for driving boundaries. The method comprises: receiving first image information about surroundings of a vehicle from first-type cameras mounted on the vehicle; extracting second image information about driving boundaries from the first image information; and generating a ground truth for the driving boundaries based at least on the second image information. Wherein, the ground truth is used to validate a sensed value about the driving boundaries collected by a second-type camera mounted on the vehicle. In addition, the present invention relates to an apparatus for generating a ground truth for driving boundaries, a computer storage medium, and a computer program product vehicle.

Description

Method and Apparatus for Generating Ground Truth for Driving Boundaries
Technical field
The present invention relates to the field of vehicles, in particular to a method and an apparatus for generating a ground truth for driving boundaries, a computer storage medium, and a computer program product vehicle.
Background
In vehicle automation related functions (for example, L0-L5 automatic drive functions), the precision of vehicle-mounted sensors' perception of information about surroundings of a vehicle (especially information about driving boundaries) is very important. In the development and validation of these functions, the collected information is often compared with the ground truth of the vehicle's surroundings. Such ground truth is also called reference value.
If a manual labeling method is used to generate the ground truth, it will lead to high labor costs and take a long time. If information collected by a lidar sensor is used to generate the ground truth, the precision of the ground truth cannot be guaranteed in some scenarios such as rain, snow, foggy weather. If the same sensor (e.g., a front facing camera), which is used for the sensed value to be validated, is used to generate the ground truth, the precision of the validation result is also difficult to be guaranteed. This is because errors or omissions caused by the inherent defects of the sensor would exist in both the ground truth and the sensed value generated by the same sensor. Therefore, these errors or omissions cannot be detected by comparing the two.
Summary
According to an aspect of the present invention, a method for generating a ground truth for driving boundaries is provided. The method comprises: receiving first image information about surroundings of a vehicle from first-type cameras mounted on the vehicle; extracting second image information about driving boundaries from the first image information; and generating the ground truth for the driving boundaries based at least on the second image information, wherein the ground truth is used to validate a sensed value about the driving boundaries collected by a second-type camera mounted on the vehicle.
Additionally and alternatively, in the foregoing method, the ground truth is further used to validate a sensed value about the driving boundaries collected by other types of sensors mounted on the vehicle.
Additionally and alternatively, the foregoing method may further comprise performing a coordinate system conversion on the second image information; and generating the ground truth based at least on the converted second image information.
Additionally and alternatively, in the foregoing method, during the coordinate system conversion, the second image information is converted from a two-dimensional image coordinate system to a three-dimensional vehicle coordinate system.
Additionally and alternatively, in the foregoing method, the first-type cameras mounted on the vehicle comprise at least two first-type cameras located at different positions of the vehicle. The method further comprises: extracting second images of the at least two first-type cameras from first image information from the at least two first-type cameras, respectively; fusing the second images of the at least two first-type cameras; and generating the ground truth based at least on fused second image information.
Additionally and alternatively, the foregoing method further comprises: fusing, based on positioning information of the vehicle at different times, the second image information about the driving boundaries extracted at the different times; and generating the ground truth based at least on the fused second image information.
Additionally and alternatively, in the foregoing method, the ground truth is generated based on the second image information and third image information about the driving boundaries from other types of sensors.
According to another aspect of the present invention, an apparatus for generating a ground truth for driving boundaries is provided. The apparatus comprises: a receiving device configured to receive first image information about surroundings of a vehicle from first-type cameras mounted on the vehicle; an extracting device configured to extract second image information about driving boundaries from the first image information; and a generating device configured to generate the ground truth for the driving boundaries based at least on the second image information, wherein the ground truth is used to validate a sensed value about the driving boundaries collected by a second-type camera mounted on the vehicle.
Additionally and alternatively, in the foregoing apparatus, the ground truth is further used to validate a sensed value about the driving boundaries collected by other types of sensors mounted on the vehicle.
Additionally and alternatively, the foregoing apparatus further comprises a conversion device. The conversion device is configured to perform a coordinate system conversion on the second image information. The generating device is further configured to generate the ground truth based at least on the converted second image information.
Additionally and alternatively, in the foregoing apparatus, the conversion device is further configured to convert the second image information from a two-dimensional image coordinate system to a three-dimensional vehicle coordinate system.
Additionally and alternatively, the foregoing apparatus further comprises a first fusion device. Wherein, the first-type cameras mounted on the vehicle comprise at least two first-type cameras located at different positions of the vehicle. The extracting device is further configured to extract second images of the at least two first-type cameras from the first image information from the at least two first-type cameras, respectively. The first fusion device is configured to fuse the second images of the at least two first-type cameras. The generating device is further configured to generate the ground truth based at least on the fused second image information.
Additionally and alternatively, the foregoing apparatus further comprises a second fusion device. The second fusion device is configured to fuse, based on positioning information of the vehicle at different times, the second image information about the driving boundaries extracted at the different times. The generating device is further configured to generate the ground truth based at least on the fused second image information.
Additionally and alternatively, in the foregoing apparatus, the generating device is further configured to generate the ground truth based on the second image information and third image information about the driving boundaries from other types of sensors.
According to another aspect of the present invention, a computer storage medium is provided. The medium comprises instructions for implementing the forgoing method when being executed.
According to still another aspect of the present invention, a computer program product is provided, which comprises a computer program. The computer program, when executed by a processor, implements the foregoing method.
According to yet another aspect of the present invention, a vehicle comprising the foregoing apparatus is provided.
The solution for generating driving boundaries according to the embodiments of the present invention uses first-type cameras to collect image information for driving boundaries, and processes the collected image information to generate a ground truth. The solution for generating driving boundaries is precise, saves time and labor costs, and can flexibly fuse image information collected by other types of sensors. Brief description of the drawings
In conjunction with the following detailed description of the accompanying drawings, the above and other objectives and advantages of the present invention will become more complete and clear, wherein the same reference numerals are used to denote the same or similar elements. The drawings are not necessarily drawn to scale.
FIG. 1 shows a schematic flowchart of a method 1000 for generating a ground truth for driving boundaries according to an embodiment of the present invention.
FIGS. 2(a)-(d) show first image information received from four vehicle-mounted first-type cameras, respectively.
FIG. 3 shows a schematic structural diagram of an apparatus 3000 for generating a ground truth for driving boundaries according to an embodiment of the present invention.
Detailed description
The solution for generating a ground truth for driving boundaries according to various exemplary embodiments of the present invention will be described in detail hereinafter with reference to the accompanying drawings.
It should be noted that, in the context of the present invention, the terms "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. In addition, unless specifically indicated otherwise, in the context of the present invention, the terms "comprising", "having" and similar expressions are intended to mean non-exclusive inclusion.
FIG. 1 shows a schematic flowchart of a method 1000 for generating a ground truth for driving boundaries according to an embodiment of the present invention. As shown in FIG. 1, the method 1000 for generating a ground truth for driving boundaries comprises the following steps.
In step SI 10, first image information about surroundings of a vehicle is received from vehicle-mounted first-type cameras.
In the context of the present invention, the term "driving boundaries" refers to road boundaries within which a vehicle can travel, for example, lane markings, botts dots, curbs, guard rails, concrete or steel walls, the border between asphalt and grass, reflectors, barriers, beacons, pylons or other structures used to delineate lanes or to delineate different kind of pavement.
In the context of the present invention, the term "first-type camera" refers to a camera that is different from a second-type camera to be validated. For example, a second-type camera to be validated may be a front facing camera used in an Advanced Driving Assistant System (ADAS). Accordingly, a first-type camera may be a fisheye camera mounted on the vehicle, or a wing camera mounted on the vehicle. Wherein, a fisheye camera may be a camera mounted on the vehicle originally for the reversing function. Generally, a fisheye camera can have a higher resolution for a sensing object in a short distance, so that a reference value with a higher precision is generated in the subsequent step S130. Wherein, wing cameras may be cameras mounted on both sides of the vehicle (for example, on the rearview mirrors on both sides) for sensing images on both sides of the vehicle.
First image information may be received from one or more first-type cameras mounted on the vehicle. FIGS. 2(a)-(d) show first image information received from four vehicle-mounted first-type cameras (fish-eye cameras are specifically provided in this embodiment), respectively. Among them, the four vehicle-mounted first-type cameras corresponding to FIGS. 2(a)-(d) are mounted on the front, rear, left, and right sides of the vehicle, respectively. As shown in FIGS. 2(a)-(d), the first image information collected by the four vehicle-mounted first-type cameras each comprises lane markings, and both the first-type camera mounted on the front side (corresponding to FIG. 2(a)) and the first-type camera mounted on the right side (corresponding to FIG. 2(d)) collect the image information of lane markings 210.
As shown in FIG. 2, as a wide-angle camera, a fisheye camera can collect image information with a large field of view. However, it can also be seen from FIG. 2 that an image with such a large field of view has a certain degree of distortion, and therefore, the image information can be corrected and compensated accordingly.
In step S120, second image information about driving boundaries is extracted from the first image information received in step SI 10.
Conventional image processing methods can be utilized to extract the second image information, for example, the edge filter algorithm, the Canny edge detection algorithm, the Sobel operator edge detection algorithm, and the like. Machine learning and artificial intelligence algorithms can also be utilized to extract the second image information, for example, neural networks, deep learning, and the like. The present invention does not make any limitations in this regard.
In step S130, a ground truth for driving boundaries is generated based at least on the second image information extracted in step SI 20. The generated ground truth can be used to compare with a sensed value of the same driving boundaries collected by other vehicle-mounted sensors to test and calculate the precision of other vehicle-mounted sensors' perception of the driving boundaries, and their perception performance in error triggering events. Here, "error triggering event" is intended to mean typical events in which other vehicle-mounted sensors are prone to perception errors, such as rain, snow, and fog scenarios.
In one embodiment, the ground truth generated in step S130 may also be used to validate the sensed value about the driving boundaries collected by other types of sensors mounted on the vehicle. Other types of sensors may be, for example, lidar sensors, millimeter-wave radar sensors, other types of cameras other than the first and second types, and any suitable types of sensors mounted on the vehicle.
As such, the ground truth for driving boundaries is generated based on the image information of the driving boundaries collected by the first-type cameras, thereby providing a benchmark for validating the precision of the vehicle-mounted cameras or other types of sensors.
Using the ground truth generated by the first-type cameras to validate the sensed value generated by a second-type camera or other sensors can prevent common cause error from being overlooked in the validation. In the context of the present invention, "common cause error" means that the same error caused by the same factor exists in multiple sensing results by the same sensor or the same type of sensors. The same factor may be originated from the location of this sensor, or from the inherent defects of this type of sensors, for example. In the embodiment according to the present invention, different types of sensors are used, which often have different focal lengths, the image information collected by which is often processed by different algorithms, and which are often mounted in different locations, thus having different lighting conditions. Therefore, the first-type cameras, which are different from the sensors to be validated, can be used to avoid common cause errors from being overlooked in validation.
In addition, according to the embodiments of the present invention, a camera capable of collecting high-quality image information of the driving boundaries near the vehicle, such as a fisheye camera, can be used as the first-type camera, so as to obtain a ground truth for driving boundaries with high precision. This is especially apparent in scenes such as rain, snow, and fog.
Although not shown in FIG. 1, the method 1000 for generating a ground truth for driving boundaries may further comprise a coordinate system conversion of the image information. For example, the second image information extracted in step 120 is converted from a two-dimensional image coordinate system to a three-dimensional vehicle coordinate system, so as to facilitate further processing of the second image information (for example, to compare and fuse with three-dimensional image information collected by other sensors). Accordingly, in step S130, the ground truth for driving boundaries is generated based at least on the converted second image information.
It should be noted that in the context of the present invention, coordinate system conversion of image information is not limited to the conversion of the second image information, and conversion may also be performed on image information generated in other steps (e.g., the first image information). For example, coordinate system conversion may first be performed on the first image information before the extraction step S120. In some cases, this is beneficial to outlier detection and plausibilisation of image information.
In addition, in the context of the present invention, coordinate system conversion of image information is not limited to the conversion from image coordinate system to vehicle coordinate system, but may also cover mutual conversions among the various coordinate systems of camera coordinate system, image coordinate system, world coordinate system, vehicle coordinate system, and the like. It depends on the specific image processing requirements.
As described above in conjunction with FIG. 2, first image information about surroundings of a vehicle can be received from multiple first-type cameras mounted at different positions of the vehicle. Accordingly, in step SI 20, second image information can be extracted from the first image information from each first-type camera. For example, second image information can be extracted in FIGS. 2(a)-(d), respectively. The method 1000 may further comprise fusing the second image information of each first-type camera (not shown in FIG. 1). Accordingly, in step S130, a ground truth for driving boundaries can be generated based at least on the fused second image information.
Alternatively, the first image information from multiple first-type cameras can also be fused before the second image information is extracted. Then, in step S120, the second image information is extracted from the fused first image information, and in step S130, a ground truth for driving boundaries is generated based on the second image information.
Therefore, by fusing the image information collected by multiple first-type cameras, the precision of the generated ground truth for driving boundaries can be further improved. This is especially apparent for the overlapping parts of the fields of view of multiple first-type cameras.
It will be appreciated that, the image information collected by all of the vehicle-mounted first-type cameras can be fused, or the image information collected by a subset of the first-type cameras can be fused. For example, in the embodiment shown in FIGS. 2(a)-(d), the image information in FIGS. 2(a) and (d) can be fused to generate the ground truth of the lane markings 210. That is, the image information collected by a subset of the four first-type cameras (the front one and the right one) is fused to generate the ground truth of the lane markings 210.
In addition, although not shown in FIG. 1, the method 1000 for generating a ground truth for driving boundaries may also comprise fusing, based on positioning information of a vehicle at different times, second image information about the same driving boundaries extracted at the different times. Accordingly, in step S130, a ground truth for driving boundaries is generated based at least on the fused second image information.
The processing of the foregoing steps is usually carried out in a time frame manner. Even when the vehicle is traveling, the image information of driving boundaries at the same location around the vehicle generally does not exist in a single time frame alone, but in multiple time frames before and after. Thus, fusing, based on the positioning information of a vehicle at multiple times, the image information of the same driving boundaries collected at the multiple times, can compensate for errors and omissions in single-frame image information and effectively improve the precision of ground truth for driving boundaries that is ultimately generated.
It should be noted that positioning information of a vehicle at different times can be determined by means of global positioning, such as global navigation satellite system (GNSS), global positioning system (GPS); it can also be determined by the positioning methods of the vehicle itself, such as onboard odometry sensor, which determines the position change of the vehicle by determining the change in the distance between the vehicle and a reference object; it can also be determined by a combination of any of the above methods. The present invention does not make any limitations in this regard.
Similar to the above, fusion of the image information of the same driving boundaries collected at different times is not limited to following the extraction of the second image information, while it can also occur before the extraction of the second image information. In other words, the first image information of the same driving boundaries collected at different times can first be fused. Then, in step S120, second image information is extracted from the fused first image information, and in step SI 30, a ground truth for driving boundaries is generated based on the second image information.
Although not shown in FIG. 1, the method 1000 for generating a ground truth for driving boundaries may further comprise generating a ground truth for the driving boundaries based on second image information from first-type cameras and third image information about the same driving boundaries from other types of sensors. Here, other types of sensors can be any types of sensors that can collect the image information of the same driving boundaries, such as lidar sensors, millimeter-wave radar sensors, other types of cameras, and the like, or a combination of the foregoing various types of sensors. Generating a ground truth using image information collected by different types of sensors can prevent the inherent defects of a single type of sensor from affecting the generated ground truth, and further improve the precision of the ground truth for driving boundaries that is ultimately generated.
It should be noted that different types of sensors often collect image information at different times. Therefore, it is necessary to fuse image information from different types of sensors based on the positioning information of a vehicle at different times. The positioning information of the vehicle at different times can be determined in the manner described above, and will not be repeated here. It will be appreciated that, although FIG. 1 and the above description describe the various operations (or steps) as sequential processing, many of these operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of various operations can be rearranged. Moreover, the embodiments of the present invention may also include additional steps not included in FIG. 1 and the above description.
Those skilled in the art appreciate that the method of generating a ground truth for driving boundaries provided by one or more of the above embodiments can be implemented by a computer program. For example, the computer program is included in a computer program product. When the computer program is executed by a processor, the method for generating a ground truth for driving boundaries according to one or more embodiments of the present invention is implemented. For another example, when a computer storage medium (such as a USB flash drive) storing the computer program is connected to a computer, the method for generating a ground truth for driving boundaries according to one or more embodiments of the present invention can be implemented by executing the computer program .
FIG. 3 shows a schematic structural diagram of an apparatus 3000 for generating a ground truth for driving boundaries according to an embodiment of the present invention. As shown in FIG. 3, the apparatus 3000 for generating a ground truth for driving boundaries comprises: a receiving device 310, an extracting device 320 and a generating device 330. Wherein, the receiving device 310 is configured to receive first image information about surroundings of a vehicle from vehicle-mounted first-type cameras. Optionally, the receiving device 310 may be configured to receive the first image information respectively from four vehicle-mounted first-type cameras described above in conjunction with FIG. 2. The extracting device 320 is configured to extract second image information about driving boundaries from the first image information. The generating device 330 is configured to generate a ground truth for driving boundaries based at least on the second image information.
In the context of the present invention, the term "driving boundaries" refers to road boundaries within which a vehicle can travel, such as lane markings, botts dots, curbs, guard rails, concrete or steel walls, the border between asphalt and grass, reflectors, barriers, beacons, pylons or other structures used to delineate lanes or to delineate different kind of pavement.
In addition, in the context of the present invention, the term "first-type camera" refers to a camera that is different from a second-type camera to be validated. For example, a second-type camera to be validated may be a front facing camera used in an Advanced Driving Assistant System (ADAS). Accordingly, a first-type camera may be a fisheye camera mounted on the vehicle, or a wing camera mounted on the vehicle. Wherein, a fisheye camera may be a camera mounted on the vehicle originally for the reversing function. Generally, a fisheye camera can have a higher resolution for a sensing object in a short distance, so that a reference value with a higher precision is generated in the subsequent step SI 30. Wherein, wing cameras may be cameras mounted on both sides of the vehicle (for example, on the rearview mirrors on both sides) for sensing images on both sides of the vehicle.
Although not shown in FIG. 3, the apparatus 3000 for generating a ground truth for driving boundaries may further comprise a correction device. The correction device may be configured to perform correction and compensation on the first image information received by the receiving device 310. Taking FIG. 2 as an example, a correction device can be used to correct the distortion in the first image information collected by a fisheye camera.
In the extracting device 320, conventional image processing methods can be utilized to extract the second image information, for example, the edge filter algorithm, the Canny edge detection algorithm, the Sobel Operator edge detection algorithm, and the like. Machine learning, artificial intelligence algorithms can also be utilized to extract the second image information, for example, neural networks, deep learning, and the like. The present invention does not make any limitations in this regard.
Although not shown in FIG. 3, the apparatus 3000 for generating a ground truth for driving boundaries may further comprise a conversion device. The conversion device is configured to perform a coordinate system conversion on the second image information. Accordingly, the generating device 330 is configured to generate a ground truth for driving boundaries based at least on the converted second image information. For example, the second image information extracted by the extraction device 320 is converted from a two-dimensional image coordinate system to a three-dimensional vehicle coordinate system, so as to facilitate further processing of the second image information (for example, to be fused with three-dimensional image information collected by other sensors). Accordingly, in the generating device 330, a ground truth for driving boundaries is generated based at least on the converted second image information.
It should be noted that in the context of the present invention, coordinate system conversion of image information is not limited to the conversion of the second image information, and conversion may also be performed on the image information generated in other steps (e.g., the first image information). In addition, in the context of the present invention, coordinate system conversion of image information is also not limited to the conversion from image coordinate system to vehicle coordinate system, but may also cover mutual conversions among the various coordinate systems of camera coordinate system, image coordinate system, world coordinate system, vehicle coordinate system, and the like. It depends on the specific image processing requirements.
Although not shown in FIG. 3, the apparatus 3000 for generating a ground truth for driving boundaries may further comprise a first fusion device. As described above in conjunction with FIG. 2, the receiving device 310 may receive first image information about surroundings of a vehicle from multiple first-type cameras mounted at different positions of the vehicle. Accordingly, in the extracting device 320, second image information can be respectively extracted from the first image information of each first-type camera. For example, second image information can be respectively extracted from FIGS. 2(a)-(d). The first fusion device may be configured to fuse the second image information of each first-type camera. The generating device 330 may be configured to generate a ground truth for driving boundaries based at least on the fused second image information.
Alternatively, the first fusion device may also be configured to fuse first image information from multiple first-type cameras. Accordingly, the extracting device 320 extracts second image information from the first image information fused by the first fusion device. And, the generating device may be configured to generate a ground truth for driving boundaries based on the second image information. Thus, by fusing the image information collected by multiple first-type cameras, the precision of the generated ground truth for driving boundaries can be improved. This is especially apparent for the overlapping parts of the fields of view of multiple first-type cameras.
It will be appreciated that, the first fusion device can fuse the image information collected by all of the vehicle-mounted first-type cameras, or can fuse the image information collected by a subset of the first-type cameras. For example, in the embodiment shown in FIGS. 2(a)-(d), the image information in FIGS. 2(a) and (d) can be fused to generate the ground truth of the lane markings 210. That is, the image information collected by a subset of the first-type range cameras (the front one and the right one) is fused to generate the ground truth of the lane markings 210.
In addition, although not shown in FIG. 3, the apparatus 3000 for generating a ground truth for driving boundaries may further comprise a second fusion device. The second fusion device is configured to fuse, based on positioning information of a vehicle at different times, second image information about the same driving boundaries extracted at the different times. Accordingly, the generating device 330 is configured to generate a ground truth for driving boundaries based at least on the fused second image information.
The operation of each device in the apparatus 3000 is usually performed in a time frame manner. Even when the vehicle is traveling, the image information of driving boundaries at the same position around a vehicle generally does not exist in a single time frame alone, but in multiple time frames before and after. Thus, fusing image information of the same driving boundaries collected at multiple times using the second fusion device can compensate for errors and omissions in single-frame image information, and effectively improve the precision of the ground truth of driving boundaries that is ultimately generated.
Similar to the first fusion device, the fusion of the image information of the same driving boundaries collected at different times by the second fusion device is not limited to following the extraction of the second image information by the extraction device 320, while it can also occur before the extraction of the second image information. In other words, the second fusion device can fuse the first image information of the same driving boundaries collected at different times. Then, in the extracting device 320, the second image information is extracted from the first image information fused by the second fusion device, and in the generating device 330, a ground truth for driving boundaries is generated based on the second image information.
The positioning information at different times of the vehicle, which is used by the second fusion device, can be determined by means of global positioning, such as Global Navigation Satellite System (GNSS), Global Positioning System (GPS); it can also be determined by the positioning methods of the vehicle itself, such as onboard odometry sensor, which determines the position change of the vehicle by determining the change in the distance between the vehicle and a reference object; it can also be determined by a combination of any of the above methods. The present invention does not make any limitations in this regard.
In addition, the generating device 330 may be further configured to generate a ground truth for driving boundaries based on second image information about driving boundaries from first-type cameras and third image information about the same driving boundaries from other types of sensors. Here, other types of sensors can be any types of sensors that can collect the image information of the same driving boundaries, such as lidar sensors, millimeter-wave radar sensors, other types of cameras, and the like, or a combination of the foregoing various types of sensors. Generating a ground truth using image information collected by different types of sensors can prevent the inherent defects of a single type of sensor from affecting the generated ground truth, and further improve the precision of the ground truth for driving boundaries that is ultimately generated.
It should be noted that different types of sensors often collect image information at different times. Therefore, it is necessary to fuse image information from different types of sensors according to the positioning information of the vehicle at different times. The positioning information of the vehicle at different times can be determined in the manner described above, and will not be repeated here.
In one or more embodiments, the apparatus 3000 for generating a ground truth for driving boundaries may be integrated into a vehicle. Wherein, the apparatus 3000 for generating a ground truth for driving boundaries may be an apparatus separately used for generating ground truth for driving boundaries in the vehicle, or it may be combined into an electronic control unit (ECU), a domain control unit (DCU) or other processing apparatus in the vehicle. It should be understood that the term "vehicle" or other similar terms used herein include general motor vehicles, such as passenger vehicles (including sports utility vehicles, buses, trucks, etc.), various commercial vehicles, etc., and also include hybrid vehicles, electric vehicles, etc. A hybrid vehicle is a vehicle with two or more power sources, such as gasoline and electric powered vehicles.
In one or more embodiments, the apparatus 3000 for generating a ground truth for driving boundaries may be integrated into the advanced driving assistance system (ADAS) or into other L0-L5 automatic drive functions of a vehicle.
The ground truth for driving boundaries generated according to the aforementioned embodiments of the present invention can be used as a benchmark to compare with the sensed value for driving boundaries sensed by other vehicle-mounted sensors. Such comparison can be used to validate and calculate the precision, average availability (e.g., true positive rate, true negative rate), and average unavailability rate (e.g., false positive rate, false negative rate) of the sensed results for driving boundaries sensed by other vehicle-mounted sensors. Such comparison can also be used for error triggering events to validate the performance of other vehicle-mounted sensors in error triggering events.
In view of the foregoing, the solution for generating driving boundaries according to the embodiments of the present invention uses first-type cameras to collect image information of the driving boundaries, and processes the collected image information to generate a highly precise ground truth for driving boundaries. Generating a ground truth using first-type cameras instead of other sensors, which are to be validated, can increase the redundancy of the system, and prevent common cause errors. In addition, compared with manually labeling a ground truth, generating a ground truth using image information collected by first-type cameras can greatly reduce time and labor costs. Furthermore, the solution for generating driving boundaries according to the embodiments of the present invention can also generate a ground truth by combining vehicle-mounted first-type cameras with other types of sensors, which can further improve the precision of the ground truth.
In addition, according to the embodiments of the present invention, a camera capable of collecting high-quality image information of the driving boundaries near the vehicle, such as a fisheye camera, can be used as the first-type camera, so as to obtain a ground truth for driving boundaries with high precision. This is especially apparent in scenes such as rain, snow, and fog.
It will be appreciated that, although only some of the embodiments of the present invention are described in the above specification, the present invention may, without departing from its spirit and scope, be implemented in many other forms. Therefore, the exemplified embodiments are illustrative but not restrictive. The present invention may, without departing from the spirit and scope of the present invention as defined by the appended claims, cover various modifications and substitutions.

Claims

What is claimed is:
1. A method of generating a ground truth for driving boundaries, comprising: receiving first image information about surroundings of a vehicle from first-type cameras mounted on the vehicle; extracting second image information about driving boundaries from the first image information; and generating the ground truth for the driving boundaries based at least on the second image information, wherein the ground truth is used to validate a sensed value about the driving boundaries collected by a second-type camera mounted on the vehicle.
2. The method according to claim 1, wherein the ground truth is further used to validate a sensed value about the driving boundaries collected by other types of sensors mounted on the vehicle.
3. The method according to claim 1, further comprising performing a coordinate system conversion on the second image information, and generating the ground truth based at least on the converted second image information.
4. The method according to claim 3, wherein, in the coordinate system conversion, the second image information is converted from a two-dimensional image coordinate system to a three-dimensional vehicle coordinate system.
5. The method according to claim 1, wherein the first-type cameras mounted on the vehicle comprise at least two first-type cameras located at different positions of the vehicle, the method further comprising: extracting second images of the at least two first-type cameras from the first image information of the at least two first-type cameras, respectively; fusing the second images of the at least two first-type cameras; and generating the ground truth based at least on the fused second image information.
6. The method according to claim 1, further comprising: fusing, based on positioning information of the vehicle at different times, the second image information about the driving boundaries extracted at the different times; and generating the ground truth based at least on the fused second image information.
7. The method according to claim 1, wherein the ground truth is generated based on the second image information and third image information about the driving boundaries from other types of sensors.
8. An apparatus for generating a ground truth for driving boundaries, comprising: a receiving device configured to receive first image information about surroundings of a vehicle from first-type cameras mounted on the vehicle; an extracting device configured to extract second image information about driving boundaries from the first image information; and a generating device configured to generate the ground truth for the driving boundaries based at least on the second image information, wherein the ground truth is used to validate a sensed value about the driving boundaries collected by a second-type camera mounted on the vehicle.
9. The apparatus according to claim 8, wherein the ground truth is used to validate a sensed value about the driving boundaries collected by other types of sensors mounted on the vehicle.
10. The apparatus according to claim 8, further comprising a conversion device, wherein: the conversion device is configured to perform a coordinate system conversion on the second image information, and the generating device is further configured to generate the ground truth based at least on the converted second image information.
11. The apparatus according to claim 10, wherein: the conversion device is further configured to convert the second image information from a two-dimensional image coordinate system to a three-dimensional vehicle coordinate system.
12. The apparatus according to claim 8, further comprising a first fusion device, wherein: the first-type cameras mounted on the vehicle comprise at least two first-type cameras located at different positions of the vehicle, the extracting device is further configured to extract second images of the at least two first-type cameras from the first image information of the at least two first-type cameras, respectively, the first fusion device is configured to fuse the second images of the at least two first-type cameras, and the generating device is further configured to generate the ground truth based at least on the fused second image information.
13. The apparatus according to claim 8, further comprising a second fusion device, wherein: the second fusion device is configured to fuse, based on positioning information of the vehicle at different times, the second image information about the driving boundaries extracted at the different times, and the generating device is further configured to generate the ground truth based at least on the fused second image information.
14. The apparatus according to claim 8, wherein the generating device is further configured to generate the ground truth based on the second image information and third image information about the driving boundaries from other types of sensors.
15. A computer storage medium comprising instructions, wherein the method according to any of claims 1 to 7 is implemented when the instructions is being executed.
16. A computer program product comprising a computer program, wherein the method according to any of claims 1 to 7 is implemented when the computer program is being executed by a processor.
17. A vehicle comprising the apparatus according to any of claims 8 to 14.
EP22814111.5A 2021-12-31 2022-11-10 Method and apparatus for generating ground truth for driving boundaries Pending EP4457776A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111662570.3A CN116416591A (en) 2021-12-31 2021-12-31 Method and device for generating reference value for driving boundary
PCT/EP2022/081493 WO2023126097A1 (en) 2021-12-31 2022-11-10 Method and apparatus for generating ground truth for driving boundaries

Publications (1)

Publication Number Publication Date
EP4457776A1 true EP4457776A1 (en) 2024-11-06

Family

ID=84365686

Family Applications (1)

Application Number Title Priority Date Filing Date
EP22814111.5A Pending EP4457776A1 (en) 2021-12-31 2022-11-10 Method and apparatus for generating ground truth for driving boundaries

Country Status (3)

Country Link
EP (1) EP4457776A1 (en)
CN (1) CN116416591A (en)
WO (1) WO2023126097A1 (en)

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE112020001897T5 (en) * 2019-04-12 2021-12-30 Nvidia Corporation Training of neural networks using basic truth data supplemented with map information for autonomous machine applications

Also Published As

Publication number Publication date
CN116416591A (en) 2023-07-11
WO2023126097A1 (en) 2023-07-06

Similar Documents

Publication Publication Date Title
AU2020203284B2 (en) Image fusion for autonomous vehicle operation
CN111442776B (en) Methods and equipment for sequential ground scene image projection synthesis and complex scene reconstruction
US12043283B2 (en) Detection of near-range and far-range small objects for autonomous vehicles
CN111081064B (en) Automatic parking system and automatic passenger-replacing parking method of vehicle-mounted Ethernet
CN110689761B (en) A method of automatic parking
US10628690B2 (en) Systems and methods for automated detection of trailer properties
US10599931B2 (en) Automated driving system that merges heterogenous sensor data
CN106441319B (en) A system and method for generating a lane-level navigation map of an unmanned vehicle
EP3306267B1 (en) Arithmetic logic device, camera device, vehicle and calibration method
EP3007099B1 (en) Image recognition system for a vehicle and corresponding method
CN111028534B (en) Parking space detection method and device
US10108866B2 (en) Method and system for robust curb and bump detection from front or rear monocular cameras
CN102398598A (en) Lane fusion system using forward and rear view cameras
CN102700548A (en) Robust vehicular lateral control with front and rear cameras
CN111674389A (en) Parking assistance device and parking assistance method
EP3771620A1 (en) Parking assistance device and control method of parking assistance device
Moras et al. Drivable space characterization using automotive lidar and georeferenced map information
JP2017117105A (en) Line-of-sight judgment device
CN115147787A (en) Image processing device and image processing method, vehicle control device, and storage medium
CN107220632A (en) A kind of pavement image dividing method based on normal direction feature
JP7323356B2 (en) PARKING ASSIST DEVICE AND PARKING ASSIST METHOD
EP4600609A1 (en) Lane-level positioning method and apparatus, smart device, and storage medium
US20240249435A1 (en) System and method for using dynamic objects to estimate camera pose
EP4457776A1 (en) Method and apparatus for generating ground truth for driving boundaries
US20250166392A1 (en) Method and Apparatus for Generating Ground Truth for Other Road Participant

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20240731

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC ME MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20260318