CN117456318A - Method and system for evaluating ground truth value device based on fusion data - Google Patents

Abstract

The application discloses a method and a system for evaluating ground truth GT device for lane detection. The method includes an acquisition step in which a respective set of GT data is acquired with each of a plurality of GT devices mounted on the same vehicle; a fusion step in which fusion GT data is generated based on each group of GT data; an evaluation step in which each GT device is evaluated based on the fused GT data.

Description

Method and system for evaluating ground truth value device based on fusion data

Technical Field

The present application relates to methods and systems for evaluating ground truth devices based on fused data, and in particular to methods and systems for evaluating detection devices that provide ground truth for lane detection.

Background

Advanced driving support systems (ADASs) generate an environmental model around a vehicle from signals received from various sensors such as cameras, ultrasonic sensors, radar, lidar, inertial sensors, and the like. The advanced driving assistance system may perform lane detection/lane line detection according to the environmental model.

There are a variety of methods based on visual detection that can generate a lane detection model as an environmental model. When an environmental model needs to be checked or improved, reference data capable of representing a real environment needs to be referred to. Such baseline data is referred to as ground truth/ground truth data (ground truth data). Detection devices that provide ground truth/ground truth data (herein referred to simply as GT devices) typically include high-precision, high-sensitivity, expensive sensors. In evaluating an environmental model (e.g., a lane detection model), a developer provides a "correct answer" to the lane line detection model using ground truth data (herein simply referred to as GT data), and further improves the environmental model according to the ground truth data.

In order to evaluate various performances of an ADAS, such as security, in some cases, the operation of an environmental model of the ADAS needs to be tested and monitored in the field. In this case, various sensors of the detection device that provide ground truth data will generate a large amount of data representing the real environment, and it is challenging to evaluate how multiple detection devices that provide ground truth data.

Disclosure of Invention

The invention aims to provide a method and a system for evaluating ground truth GT devices. According to one or more embodiments, a ground truth test apparatus (GT apparatus) that provides reliable ground truth data (GT data) can be efficiently selected at a low cost.

According to one aspect of the present invention, a method of evaluating ground truth GT data for lane detection is disclosed. The method comprises the following steps: an acquisition step in which a corresponding set of GT data is acquired with each of a plurality of GT devices mounted on the same vehicle, respectively; a fusion step in which fusion GT data is generated based on each group of GT data; an evaluation step in which each GT device is evaluated based on the fused GT data.

According to one example, in the acquiring step of the above method, respective sets of GT data are acquired from a plurality of GT devices configured to detect lanes according to detection conditions, wherein the sets of GT data are sets of sample point data based on lane boundaries of the same time sequence. The detection conditions may include at least one of: a specified time period, a specified weather condition, and a specified road segment.

According to one example, each set of GT data may be composed of a plurality of sets of sample point data, one set of sample point data corresponding to one time stamp of the time series, and one set of sample point data being composed of a plurality of sample point values having a bit number, each sample point value indicating a distance from a lane boundary detected under a vehicle coordinate system to a vertical axis.

According to one example, in the merging step, a plurality of sampling point values with the same bit number of all GT devices at the same time stamp are merged to obtain a merged sampling point value with each bit number of the same time stamp. Alternatively, the fused sample point values of the same rank are calculated from the positional relationship between the plurality of sample points having the same rank in the plurality of sets of sample point data of the same time stamp. Alternatively, when the fused sampling point value is calculated, abnormal sampling points are excluded according to the positional relationship between the plurality of sampling points.

According to one example, in the evaluating step, for each of the plurality of GT devices, a distance between a sampling point of the GT device at a same level of each timestamp and a corresponding fused sampling point is calculated, resulting in a set of distance data of the GT device at the level, and a standard deviation based on the set of distance data is evaluated.

In accordance with one aspect of the present invention, an evaluation ground truth GT system is disclosed. The system includes a processor and a memory, wherein the memory stores computer program instructions that, when executed by the processor, are capable of performing the methods of the various embodiments.

According to one aspect of the invention, a computer-readable storage medium having stored thereon computer instructions which, when executed by a processor, perform the method of the various embodiments is disclosed.

According to one aspect of the present invention, a computer program product is disclosed, the computer program product comprising computer instructions stored in a computer-readable storage medium, the computer instructions being adapted to be read and executed by a processor to cause a computer device having the processor to perform the method in the various embodiments.

Drawings

The principles, features and advantages of the present invention may be more fully understood with reference to the following detailed description of the embodiments in connection with the accompanying drawings.

Fig. 1 is a schematic diagram of an exemplary detection device providing ground truth data on the same vehicle according to one embodiment of the present application.

Fig. 2 is a schematic diagram of steps of a method of evaluating ground truth GT device according to an embodiment of the present application.

FIG. 3 is a schematic diagram of sample points of exemplary ground truth values according to one embodiment of the present application.

Fig. 4 is a schematic diagram of the composition of exemplary GT data according to an embodiment of the present application.

FIG. 5 is a schematic diagram of an exemplary exception sampling point according to one embodiment of the present application.

FIG. 6 is a schematic diagram of an exemplary fused sample point calculation from weights according to one embodiment of the present application.

Fig. 7 is a schematic diagram of a system for evaluating ground truth GT devices according to an embodiment of the present application.

Detailed Description

In order to make the technical problems, technical solutions and advantageous technical effects to be solved by the present invention more apparent, the present invention will be further described in detail with reference to the accompanying drawings and a plurality of exemplary embodiments. It should be understood that the embodiments described herein are merely illustrative of the present invention and are not intended to limit the scope of the present invention.

Fig. 1 is a schematic diagram of an exemplary detection device providing ground truth data on the same vehicle according to one embodiment of the present application. The detection device that provides ground truth data is referred to herein as the ground truth device, GT (ground truth) device. Fig. 1 shows a vehicle V1 and a plurality of GT devices GT-D1, GT-D2, GT-D3 mounted on the vehicle V1. Those skilled in the art will appreciate that each GT device may include a set of sensors for providing ground truth values. For example, GT-D1 may include one or more of a high sensitivity lidar, ultra-high definition camera, high precision map, high precision inertial sensor, GPS navigation system to detect real world conditions, providing an accurate representation of the surrounding environment for vehicle V1. Similarly, GT-D2, GT-D3 also independently provide information about the surrounding environment for vehicle V1.

In this embodiment, three GT devices are each used to detect lane boundaries to generate ground truth data (GT data) of the lane. Three GT devices for acquiring ground truth data operate in the same environment for the same period of time as the perception system of the lane detection model of the vehicle (not shown in fig. 1). The perception system with which the vehicle V1 itself is equipped is adapted to estimate the surroundings of the vehicle and is configured to interpret the sensed information to identify obstacles, lanes, etc. Those skilled in the art will appreciate that the sensed information may be derived from one or more sensors configured on the vehicle, which may refer to one or more of radar, lidar, cameras, GPS navigation systems, inertial measurement units.

The data detected by the GT device may be processed and analyzed locally or by cloud computing to obtain GT data. The GT data of the lanes may be used to verify the accuracy of the lane detection model of the vehicle V1 and evaluate the overall performance of the various sensors of the vehicle body with the lane detection algorithm. Although three GT devices are shown in this embodiment, it will be appreciated by those skilled in the art that the individual GT devices mounted on the vehicle V1 are not limited to three.

Fig. 2 is a schematic diagram of the steps of a method of evaluating ground truth GT devices according to an embodiment of the present application. As shown in fig. 2, the method of evaluating ground truth GT devices includes steps 210, 220, and 230. In step 210, a respective set of GT data is acquired with each of a plurality of GT devices installed on the same vehicle, and in step 220 fused GT data is generated based on the sets of GT data, from which the respective GT devices are evaluated in step 230. By determining a plurality of groups of GT data from a large number of GT data records and fusing the plurality of groups of GT data to obtain an evaluation standard, the calculation amount in the evaluation process can be effectively reduced, and the evaluation efficiency is improved.

In the step 210, respective sets of GT data are acquired from the plurality of GT devices configured to detect lanes according to detection conditions. The multiple sets of GT data are sets of sample point data based on lane boundaries of the same time sequence. Here, the detection condition includes at least one of: a specified time period, a specified weather condition, and a specified road segment. By specifying different conditions to be detected, it is advantageous to evaluate the reliability of GT data records obtained by each GT device under different conditions. For example, each GT device may be designated to acquire GT data records during the day and night (continuous or discrete time periods designated during the day), respectively. Or the lane detection is carried out under different weather conditions such as sunny days, foggy days or rainy days to evaluate the performance of the GT device for coping with various weather conditions. The specified road segments may also be used to evaluate the performance of the GT device on different road segments. The data quality of the GT device may be assessed from the part of the acquired GT data record and the GT device to be eventually employed is determined.

The three GT devices are respectively used for detecting lane boundaries to generate ground truth data (GT data) of the lanes. Since each GT device will generate a large number of data records after lane detection, a series of sampling points may be selected from the data records with a specific rule to form a plurality of sets of GT data, one GT device for each set of GT data. The specific rule may be to determine the sampling points from the ground truth data record according to a specific time sequence and to determine the number and location of the sampling points on one time stamp for each time stamp of the time sequence. The amount of data analyzed and processed can be reduced, and the evaluation efficiency can be improved without reducing the evaluation accuracy.

The acquired set of GT data will optionally include a vehicle id, and a time sequence including a plurality of time stamps. The time sequence comprising a plurality of time stamps may be a UTC time sequence based on a global navigation satellite system GNSS. FIG. 3 is a schematic diagram of sample points of ground truth according to one embodiment of the present application. Step 210 of fig. 2 is illustrated in connection with fig. 3, where respective sets of GT data are acquired from a plurality of GT devices (e.g., GT devices GT-D1, GT-D2, GT-D3 as illustrated in fig. 1) configured to detect lanes according to detection conditions. In fig. 3, a vehicle V1 locates lane boundaries on both sides of the vehicle in a vehicle coordinate system. The vertical axis of the vehicle coordinate system is the x-axis and the horizontal axis is the y-axis. Three sets of sample points are shown at time t1, schematically illustrated by triangles, circles and squares, respectively. For example, a set of 6 sample points listed by triangles are points on the lane boundary determined by GT device GT-D1, while two sets of sample points shown by circles and squares are points on the lane line boundary determined by GT devices GT-D2, GT-D3. The sampling points may be determined from the ground truth data record according to a specific time sequence (t 1, t2, t3 …). The number and location of sampling points on one timestamp is determined for each timestamp (e.g., t 1) of the time series. The amount of data analyzed and processed can be reduced, and the evaluation efficiency can be improved without reducing the evaluation accuracy. The interval of each time stamp t may be, for example, 100 milliseconds. A set of sampling points corresponding to each timestamp is determined at a specified bit order to meet data interface requirements for evaluating ground truth data. For example, a set of samples at a time stamp includes 101 samples arranged in bit order within 200 meters in front of the host, each sample being spaced 2 meters apart. Those skilled in the art will appreciate that the sampling points may also be determined in other ways. Those skilled in the art will appreciate that only one lane line is shown in fig. 3, and that the GT device may determine the point on the lane line on the other side in the same manner.

As described above, the GT data will optionally include a vehicle id, a time series including a plurality of time stamps, a set of sampling points corresponding to each time stamp. Fig. 3 shows the sampling points at time t1 in the time sequence, and at the next time t2, a GT device will determine another set of sampling points of the same specification for the lane line. In the embodiment of fig. 3, each time, three GT devices will determine three sets of 6 sampling points as ground truth basis for lane boundaries. Since the sets of GT data are provided according to the same specification, the three sets of sampling points at time t1 are all composed of a set of 3 sampling points at the same level, as shown by the dashed ellipses in the figure. As described above, a predetermined distance in front of the vehicle may be specified, and the arrangement of the sampling points may be determined at predetermined intervals. For a set of samples at the same time t1, their values in the x-axis direction are equal, but have different values in the y-axis, due to the same bit times. Those skilled in the art will appreciate that the two-dimensional vehicle coordinate system herein is merely an example, and that a three-dimensional coordinate system may be employed in other situations. Thereafter, the sets of sample point data will be fused in step 220. As will be described below, the fused sample point data will be used to evaluate the quality of GT data for the GT device.

In step 220, fused GT data is generated based on the sets of GT data. Fig. 4 is a schematic diagram of the composition of exemplary GT data according to an embodiment of the present application. Fig. 4 shows in tabular form three sets of GT data determined by GT devices GT-D1, GT-D2, GT-D3, each set of GT data comprising a plurality of sets of sampling point SP data. Each set of sample point data of the plurality of sets of sample point data corresponds to each time stamp of the time series, e.g. at time t1 to sample points sp1 to sp4. Wherein a set of sample points sp1 to sp4 consists of a plurality of sample point values with bit times, each sample point value indicating the distance of the detected lane boundary to the longitudinal axis in the vehicle coordinate system. For example, for the GT device GT-D1, it is installed on the vehicle V1, which detects the generation of GT data records from which one or more sets of GT data are determined. The set of GT data corresponds to each time stamp T1 to T5 of the time series T with a set of sampling points sp1 to sp4. It will be appreciated by those skilled in the art that the time stamps T1 to T5 and the sampling points sp1 to sp4 of the time series T are only examples herein. Accordingly, GT-D2 and GT-D3 also determine the sampling points of the same bit-order at the same time stamp. The data of each sampling point is the value of the abscissa of the detected lane boundary in the vehicle coordinate system.

In step 220, a plurality of sampling point values with the same bit number of all GT devices in the same time stamp are fused to obtain fused sampling point values with the same time stamp in each bit number. The right side in fig. 4 shows a fused sample point data table Fu, the fused sample point set FuSP consisting of fused sample points FuSP1 to FuSP4 corresponding to the sample points sp1 to sp4 of the GT device.

In one example, the fused sample point values of the same rank are calculated from positional relationships between multiple sample points of the same rank in multiple sets of sample point data of the same timestamp. In fig. 4, for example, three sample point data of sp1 bit times at time t1 of GT devices GT-D1, GT-D2, and GT-D3 are fused to obtain fused sample point data fusp1, as indicated by a broken line. Thereby obtaining a fusion value of all sampling point data on the time sequence.

The calculation of the fusion value is described below. FIG. 5 is a schematic diagram of an exemplary exception sampling point according to one embodiment of the present application. In this embodiment, the fused sample point value is calculated for the positional relationship between a plurality of sample points having the same bit times at the same time stamp. In step 220, when calculating the fused sampling point value, abnormal sampling points are excluded according to the positional relationship among the plurality of sampling points. As shown in fig. 5, three sampling points A, B, C of the same bit times of the same time stamp are shown. The position of the sample point A, B, C in the figure corresponds to the position of the horizontal axis of the sample point vehicle coordinate system. The distance between sampling points a and B is d1 and the distance between sampling points B and C is d2. In the first case at the top of the figure, when the sampling points B and C are closer and the ratio of d1 to d2 is greater than a first threshold (e.g., greater than 2), the sampling point a is regarded as an abnormal sampling point. At this time, the sampling point a is excluded when the fused sampling point is calculated, and only the point B and the point C are considered. The value yF of the fusion sampling point F on the abscissa of the vehicle coordinate system is the average of the values of point B and point C, i.e., (yB+yC)/2. In the second case at the bottom of the figure, when the sampling points a and B are closer and the ratio of d1 to d2 is smaller than the second threshold (for example, smaller than 0.5), the sampling point C is regarded as an abnormal sampling point. At this time, the sampling point C is excluded when the fused sampling point is calculated, and only the points a and B are considered. The value yF of the fusion sampling point F on the abscissa of the vehicle coordinate system is the average of the values of points A and B, i.e., (yA+yB)/2. In this example, for three samples A, B, C of the same order of the same timestamp, regarding a closer sample as a more accurate sample and excluding a "far" sample may better evaluate the GT device providing ground truth data. This example shows only the case of three GT devices, and one skilled in the art will appreciate that other ways of excluding outlier samples may be used depending on the positional relationship between the plurality of samples. For example, an average value of values of a plurality of sampling points is obtained, and sampling points deviating from the average value by more than a threshold value are excluded.

FIG. 6 is a schematic diagram of an exemplary fused sample point based on weight calculation according to an embodiment of the present application. Similar to fig. 5, fig. 6 also shows three sampling points A, B, C of the same bit times for the same time stamp. The position of the sample point A, B, C in the figure corresponds to the position of the horizontal axis of the sample point vehicle coordinate system. The distance between sampling points a and B is d1 and the distance between sampling points B and C is d2. If the ratio of d1 to d2 is now less than the first threshold (e.g., less than 1.2) and greater than the second threshold (e.g., greater than 0.8), then three sampling points are considered simultaneously in calculating the fused sampling points. In this case, there are no "remote" sampling points. The value yF of the fusion sampling point F on the abscissa of the vehicle coordinate system is 2/3 (d 2/(d1+d2) yA+d1/(d1+d2) yC) +1/3yB. As shown, more weight is considered to be given to sample a that is closer to sample B when calculating the value of the fused sample. This example shows only the case of three GT devices, and one skilled in the art will appreciate that other means may be used to weight each sample point depending on the positional relationship between the multiple sample points. For example, an average value of values of a plurality of sampling points is obtained, and sampling points closer to the average value are given a larger weight.

The process of acquiring GT data of step 210 and the process of fusing GT data of step 220 of fig. 2 are described above. After calculating the values of the fused sampling points, the GT device may be evaluated from these values. Step 230 of fig. 2, the process of evaluating GT data, will now be described. In step 230, for each of the plurality of GT devices, a distance between a sampling point of the GT device at a same level of each timestamp and a corresponding fused sampling point is calculated, resulting in a set of distance data of the GT device at the level, the evaluation being based on standard deviations of the set of distance data. Referring back to fig. 4, taking GT-D1 as an example, calculating the distance between the sampling point of the same bit of each time stamp and the corresponding fusion sampling point of a GT device refers to calculating the distance between sp1 of GT-D1 at time t1-t4 and corresponding FuSP1 of fusion sampling point set FuSP at time t1-t4, which may be the absolute value of the difference between the two. The same operation is performed on spn of other bit times to obtain the set of absolute values of the differences between the sampling points of other bit times and the fusion sampling point. The stability and authenticity of the GT device's provided ground truth data at a particular level is evaluated by calculating the standard deviation of a set of distance data (the set of absolute values of the differences) at that level. Similarly, the same calculation is performed for other GT devices to obtain the standard deviation of a set of distance data (the set of absolute values of the differences) for that GT device at a particular level. The GT device corresponding to the group of distance data with lower standard deviation has higher stability and authenticity of the sampling point data on the bit corresponding to the group of distance data. Thus, an efficient method of screening GT devices based on less data is provided herein. The selected GT device will provide more reliable ground truth data.

Fig. 7 is a schematic diagram of a system 700 for evaluating ground truth GT data according to an embodiment of the present application. The system 700 is used to implement, for example, the method step operations in fig. 2, and is applied to the processes shown in fig. 3-6. The system 700 may include a processor 710 and a memory 720. Processor 710 and memory 720 communicate via a bus, or may communicate via other means such as wireless transmission. The memory 720 is used to store instructions and the processor 710 is used to execute the instructions stored by the memory 520. Memory 720 may include Read Only Memory (ROM), random Access Memory (RAM). Processor 710 may invoke program modules stored in memory 720 to perform the various steps of the methods described above. The system 700 also includes memory and communication interfaces not shown in fig. 7. The system 700 obtains multiple sets of GT data via input/output (I/O) interfaces. Computer program code stored in memory 720 is loaded into memory for execution by processor 720. In the example of fig. 7, the memory 720 includes three program modules, namely an acquisition module 721, a fusion module 722, and an evaluation module 723.

The acquisition module 721 acquires GT data, which is a corresponding set of GT data acquired respectively with each of a plurality of GT devices installed on the same vehicle. The acquisition module 721 may receive GT data through various communication modes such as wired, wireless, and the like. The acquisition module 721 acquires respective sets of GT data from the plurality of GT devices configured to detect lanes according to detection conditions, wherein the sets of GT data may be sets of sample point data based on lane boundaries of the same time sequence.

The fusion module 722 generates fused GT data based on the sets of GT data. The fusion module 722 may fuse multiple sampling point values with the same bit times of all GT devices in the same time stamp to obtain fused sampling point values with the same time stamp in each bit time. The fused sample point values of the same bit order may be calculated according to a positional relationship between a plurality of sample points having the same bit order among a plurality of sets of sample point data of the same time stamp. In calculating the fused sample point value, the fusion module 722 excludes abnormal sample points according to the positional relationship among the plurality of sample points.

The evaluation module 723 evaluates the individual GT devices based on the fused GT data. And calculating the distance between the sampling point of the GT device at the same bit of each time stamp and the corresponding fusion sampling point aiming at each of the plurality of GT devices to obtain a group of distance data of the GT device at the bit, and evaluating the GT device according to the standard deviation of the group of distance data.

One or more of the embodiments, various examples described above, may be implemented in whole or in part by software, hardware, firmware, or a combination thereof. When implemented in software, embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from a network server to another computer by wire or wirelessly.

In various embodiments of the present application, the program modules and apparatus described are illustrative only. The division of the functional units is a logic function division, and other division modes can be adopted when the division is realized. Multiple units and devices may be physically separated, or may be distributed over a network unit, which may be combined or may be integrated into another system. The foregoing is merely a specific embodiment of the present application. Variations or alternatives will occur to those skilled in the art from the detailed description provided herein and are intended to be within the scope of the present application.

Claims (10)

1. A method of evaluating ground truth GT apparatus, comprising:

an acquisition step of acquiring a corresponding set of GT data with each of a plurality of GT devices installed on the same vehicle,

a fusion step of generating fusion GT data based on each group of GT data,

and an evaluation step of evaluating each GT device according to the fused GT data.

2. The method of claim 1, wherein,

in the acquiring step, respective sets of GT data are acquired from the plurality of GT devices configured to detect lanes according to detection conditions, wherein the sets of GT data are sets of sampling point data based on lane boundaries of the same time sequence.

3. The method of claim 2, wherein,

the detection conditions include at least one of: a specified time period, a specified weather condition, and a specified road segment.

4. The method of claim 2, wherein,

each set of GT data is composed of a plurality of sets of sample point data, one set of sample point data corresponding to one time stamp of the time series, and one set of sample point data is composed of a plurality of sample point values having a bit order, each sample point value indicating a distance from a lane boundary detected under a vehicle coordinate system to a vertical axis.

5. The method of claim 4, wherein,

in the fusing step, a plurality of sampling point values with the same bit number of all GT devices in the same time stamp are fused to obtain fused sampling point values with the same time stamp in each bit number.

6. The method of claim 5, wherein,

in the fusing step, the fused sample point values of the same bit number are calculated according to the positional relationship among the plurality of sample points with the same bit number in the plurality of groups of sample point data of the same time stamp.

7. The method of claim 6, wherein,

in the fusing step, when the fused sampling point value is calculated, abnormal sampling points are eliminated according to the position relation among a plurality of sampling points.

8. The method of claim 7, wherein,

in the step of evaluating, for each of the plurality of GT devices, a distance between a sampling point of the GT device at a same level of each timestamp and a corresponding fusion sampling point is calculated, resulting in a set of distance data of the GT device at the level, the evaluating being based on standard deviations of the set of distance data.

9. A system for evaluating ground truth GT devices, characterized in that the system comprises a processor and a memory, wherein the memory stores computer program instructions that, when executed by the processor, are capable of performing the method according to any of claims 1-8.

10. A computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the method of any of claims 1 to 8.