CN117475612A - Method and system for evaluating ground truth value device based on synchronous information - Google Patents

Abstract

Methods and systems for evaluating ground truth GT devices are disclosed. The method includes determining delay information between a plurality of GT devices mounted on the same vehicle; respectively acquiring a corresponding group of GT data by utilizing each of the plurality of GT devices according to the delay information, wherein each group of GT data is synchronized according to the delay information; the plurality of GT devices is evaluated based on the sets of GT data.

Description

Method and system for evaluating ground truth value device based on synchronous information

Technical Field

The present application relates to methods and systems for evaluating ground truth devices based on synchronization information, and in particular to methods and systems for evaluating detection devices that provide ground truth for lane detection.

Background

Advanced driving support systems (ADASs) generate an environmental model around a vehicle from signals received from various sensors such as cameras, ultrasonic sensors, radar, lidar, inertial sensors, and the like. The advanced driving assistance system may perform lane detection/lane line detection according to the environmental model.

There are a variety of methods based on visual detection that can generate a lane detection model as an environmental model. When an environmental model needs to be checked or improved, reference data capable of representing a real environment needs to be referred to. Such baseline data is referred to as ground truth/ground truth data (ground truth data). Detection devices that provide ground truth/ground truth data (referred to herein simply as GT devices) typically include high-precision, high-sensitivity, expensive sensors. In evaluating an environmental model (e.g., a lane detection model), a developer provides a "correct answer" to the environmental model using ground truth data (herein simply referred to as GT data), and further improves the environmental model according to the ground truth data.

In order to evaluate various performances of an ADAS, such as security, in some cases, the operation of an environmental model of the ADAS needs to be tested and monitored in the field. In this case, various sensors of the detection device that provide ground truth data will generate a large amount of data representing the real environment, and it is challenging to evaluate how multiple detection devices that provide ground truth data.

Disclosure of Invention

The invention aims to provide a method and a system for evaluating ground truth GT devices. According to one or more embodiments, a ground truth test apparatus (GT apparatus) that provides reliable ground truth data (GT data) can be efficiently selected at a low cost.

In accordance with one aspect of the present invention, a method of evaluating ground truth GT devices is disclosed. The method includes determining delay information between a plurality of GT devices mounted on the same vehicle; respectively acquiring a corresponding group of GT data by utilizing each of a plurality of GT devices according to the delay information; multiple GT devices are evaluated based on each set of GT data. Here, the sets of GT data are synchronized according to delay information. The synchronized sets of GT data improve the efficiency and accuracy of the evaluation process.

According to one example, a reference GT device is selected from a plurality of GT devices and delay information is calculated for each other GT device than the reference GT device described above with respect to the reference GT device. The delay information between two GT devices may be determined by performing an alignment process on the respective inertial measurement unit signals of the two GT devices. The inertial measurement unit signal may be an angular velocity signal or a lateral acceleration signal.

According to one example, a reference time sequence of the reference GT device is also determined, and the reference time sequence is compensated according to the delay information to obtain time sequences corresponding to other GT devices.

According to one example, respective sets of GT data are acquired from a plurality of GT devices configured to detect lanes according to detection conditions, wherein the detection conditions include at least one of: a specified time period, a specified weather condition, and a specified road segment. The set of GT data of the reference GT device is a set of sample point data of lane boundaries according to the reference time series, and each set of GT data of the other GT devices is a set of sample point data of lane boundaries according to the respective compensated time series.

According to one example, each set of GT data is composed of a plurality of sets of sample point data, one set of sample point data corresponding to one time stamp of the time series, and one set of sample point data is composed of a plurality of sample point values having a bit number, each sample point value indicating a distance from a lane boundary detected under a vehicle coordinate system to a longitudinal axis, wherein the plurality of GT devices are evaluated according to a positional relationship between the plurality of GT devices at the plurality of sample points having the same bit number of the corresponding time stamp.

According to one aspect of the present invention, a system for evaluating ground truth GT devices is disclosed, the system comprising a processor and a memory, wherein the memory stores computer program instructions that, when executed by the processor, are capable of performing the methods of the various embodiments.

According to one aspect of the invention, a computer-readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the method of the various embodiments is disclosed.

According to one aspect of the present invention, a computer program product is disclosed, the computer program product comprising computer instructions stored in a computer-readable storage medium, the computer instructions being adapted to be read and executed by a processor to cause a computer device having the processor to perform the method in the various embodiments.

Drawings

The principles, features and advantages of the present invention may be more fully understood with reference to the following detailed description of the embodiments in connection with the accompanying drawings.

Fig. 1 is a schematic diagram of an exemplary detection device providing ground truth data on the same vehicle according to one embodiment of the present application.

Fig. 2 is a schematic diagram of steps of a method of evaluating ground truth GT device according to an embodiment of the present application.

FIG. 3 is a schematic diagram of sample points of exemplary ground truth values according to one embodiment of the present application.

Fig. 4 is a schematic diagram of the composition of exemplary GT data according to an embodiment of the present application.

FIG. 5 is a schematic diagram of an exemplary exception sampling point according to one embodiment of the present application.

FIG. 6 is a schematic diagram of an exemplary fused sample point calculation from weights according to one embodiment of the present application.

Fig. 7 is a schematic diagram of a system for evaluating ground truth GT devices according to an embodiment of the present application.

Detailed Description

In order to make the technical problems, technical solutions and advantageous technical effects to be solved by the present invention more apparent, the present invention will be further described in detail with reference to the accompanying drawings and a plurality of exemplary embodiments. It should be understood that the embodiments described herein are merely illustrative of the present invention and are not intended to limit the scope of the present invention.

Fig. 1 is a schematic diagram of an exemplary detection device providing ground truth data on the same vehicle according to one embodiment of the present application. The detection device that provides ground truth data is referred to herein as the ground truth device, GT (ground truth) device. Fig. 1 shows a vehicle V1 and a plurality of GT devices GT-D1, GT-D2, GT-D3 mounted on the vehicle V1. Those skilled in the art will appreciate that each GT device may include a set of sensors for providing ground truth values. For example, GT-D1 may include one or more of a high sensitivity lidar, ultra-high definition camera, high precision map, high precision inertial sensor, GPS navigation system to detect real world conditions, providing an accurate representation of the surrounding environment for vehicle V1. Similarly, GT-D2, GT-D3 also independently provide information about the surrounding environment for vehicle V1.

In this embodiment, three GT devices are each used to detect lane boundaries to generate ground truth data (GT data) of the lane. Three GT devices for acquiring ground truth data operate in the same environment for the same period of time as the perception system of the lane detection model of the vehicle (not shown in fig. 1). The perception system with which the vehicle V1 itself is equipped is adapted to estimate the surroundings of the vehicle and is configured to interpret the sensed information to identify obstacles, lanes, etc. Those skilled in the art will appreciate that the sensed information may be derived from one or more sensors configured on the vehicle, which may be one or more of radar, lidar, cameras, GPS navigation systems, inertial measurement units.

The data detected by the GT device may be processed and analyzed locally or by cloud computing to obtain GT data. The GT data of the lanes may be used to verify the accuracy of the lane detection model of the vehicle V1 and evaluate the overall performance of the various sensors of the vehicle body with the lane detection algorithm. Although three GT devices are shown in this embodiment, it will be appreciated by those skilled in the art that the individual GT devices mounted on the vehicle V1 are not limited to three.

Fig. 2 is a schematic diagram of the steps of a method of evaluating ground truth GT devices according to an embodiment of the present application. As shown in fig. 2, the method of evaluating ground truth GT devices includes a determining step 210, an obtaining step 220, and an evaluating step 230. In step 210, delay information between a plurality of GT devices mounted on the same vehicle is determined. In step 220, a respective set of GT data is acquired with each of the plurality of GT devices according to the delay information, wherein the sets of GT data are synchronized according to the delay information. In step 230, the plurality of GT devices is evaluated based on the sets of GT data. Here, the sets of GT data are synchronized according to delay information. The synchronized sets of GT data improve the efficiency and accuracy of the evaluation process.

As described above, the three GT devices are each used to detect lane boundaries to generate ground truth data (GT data) of the lane. Since each GT device will generate a large number of data records after lane detection, a series of sampling points may be selected from the data records with a specific rule to form a plurality of sets of GT data, one GT device for each set of GT data. The specific rule may be to determine the sampling points from the ground truth data record according to a specific time sequence and to determine the number and location of the sampling points on one time stamp for each time stamp of the time sequence. The amount of data analyzed and processed can be reduced, and the evaluation efficiency can be improved without reducing the evaluation accuracy.

The acquired set of GT data will optionally include a vehicle id and a time series including a plurality of time stamps. The time sequence comprising a plurality of time stamps may be a UTC time sequence based on a global navigation satellite system GNSS. Ideally, three GT devices would employ a consistent time sequence to determine the sampling points, with the determined sampling points being synchronized in time. In some cases, due to subtle differences in transmission delay and trigger delay of the hardware/software systems of the three GT devices, the time stamps of the resulting data may not be aligned, albeit with the same start acquisition time and time sequence set, thereby affecting the efficiency of the evaluation process.

In order to overcome the problem that the sampling point data cannot be aligned due to transmission delay, trigger delay, etc., the determination of delay information is introduced in step 210 of this embodiment. And synchronizing the detection results of the three GT devices through the delay information to form a new group or groups of GT data. For the case of multiple GT devices, one reference GT device may be selected and delay information for each other GT device relative to that reference GT device is calculated. By introducing delay information, the accuracy of time synchronization of different GT devices can be improved.

Delay information between GT devices may be determined based on specific vehicle behavior characteristic signals. For example, delay information between two GT devices is determined by performing an alignment process on their respective inertial measurement unit signals. More specifically, the delay information may be determined by the yaw rate and lateral acceleration signals over a period of time. Yaw rate and lateral acceleration signals may be obtained by an inertial measurement unit (Inertial measurement unit, IMU). The inertial measurement unit is a device that measures the three-axis attitude angle (or angular rate) and acceleration of an object. It will be appreciated by those skilled in the art that a three-axis gyroscope and three-directional accelerometers may be incorporated into an IMU, the accelerometers detecting acceleration signals of the object in the carrier coordinate system on three independent axes, and the gyroscope detecting angular velocity signals of the carrier relative to the navigational coordinate system, the attitude of the object being calculated after processing these signals.

Before starting to acquire the GT data record, the vehicle may be driven along a specific trajectory for a predetermined period of time, resulting in a varying angular velocity profile. And determining delay information between the two GT devices by performing alignment processing on the two angular speeds. In the case of a plurality of GT devices, one reference GT device is selected from the plurality of GT devices, and delay information of each other GT device except the above reference GT device with respect to the reference GT device is calculated.

Here, determination of delay information of two GT devices based on the angular velocity signal will be described. The two GT devices are relatively fixedly mounted on the vehicle and each include a sensor for measuring angular velocity. Thus, when the vehicle is traveling in a steering direction, the two sensors can acquire the same angular velocity information. The angular velocity sensor may be any sensor or sensor group capable of acquiring angular velocity, such as a gyro type. It will be appreciated by those skilled in the art that the angular velocity information from the two sensors may be denoised prior to the alignment process.

In one example, the delay information is determined from the correlation of two sets of angular velocity curves over the same period of time. Considering that the information sources of the two sensors are the angular velocity of the same vehicle, the change trend of the angular velocity recorded by the two sensors should be consistent. There may be a delay between the peaks/valleys of the variation curves of the two angular velocities due to hardware/software system problems. The curves of the two angular velocity signals can be aligned by varying the time interval of one of the curves of the angular velocity signals, at which time the two sets of angular velocity signal data will have the greatest correlation. Those skilled in the art will appreciate that covariance is a measure of the linkage relationship between X and Y, i.e., the correlation of X and Y is measured, and the covariance is divided by the product of the standard deviations of X and Y, respectively, to yield a normalized correlation therebetween. For signal data processing, a known function may be used to find the delay information that maximizes the correlation of the two sets of angular velocity signals. For example, the cross-correlation function xcorr of MATLAB, or the corelate function of the Python scientific calculation library NumPy. In another example, the delay information is determined from the correlation of two sets of lateral acceleration curves over the same period of time. In this example, the delay information may be determined by calculating the greatest correlation of the two sets of lateral acceleration signal data by aligning the change curves of the two in the same manner as the angular velocity is processed.

After the delay information is determined, a reference time sequence of the reference GT device is also determined, and the reference time sequence is compensated according to the delay information to obtain time sequences corresponding to other GT devices. For example, if the first timestamp in the reference time sequence is t1, it may be determined that the first timestamp of one other GT device is t1' t1+Δt according to the delay information Δt.

Returning to fig. 2, in step 220, respective sets of GT data are acquired from the plurality of GT devices configured to detect lanes according to detection conditions, wherein the detection conditions include at least one of: a specified time period, a specified weather condition, and a specified road segment. By specifying the detection conditions, it is advantageous to evaluate the reliability of GT data records obtained by each GT device under different conditions. For example, each GT device may be designated to acquire GT data records during the day and night (continuous or discrete time periods designated during the day), respectively. Or the lane detection is carried out under different weather conditions such as sunny days, foggy days or rainy days to evaluate the performance of the GT device for coping with various weather conditions. The specified road segments may also be used to evaluate the performance of the GT device on different road segments. The data quality of the GT device may be assessed from the part of the acquired GT data record and the GT device to be eventually employed is determined.

The set of GT data of the reference GT device is a set of sample point data of lane boundaries according to a reference time sequence, and each set of GT data of the other GT devices is a set of sample point data of lane boundaries according to respective compensated time sequences. FIG. 3 is a schematic diagram of sample points of ground truth according to one embodiment of the present application. In fig. 3, a vehicle V1 locates lane boundaries on both sides of the vehicle in a vehicle coordinate system. The vertical axis of the vehicle coordinate system is the x-axis and the horizontal axis is the y-axis. Three sets of sample points at time t1 are shown in the figure, schematically indicated by triangles, circles and squares, respectively. For example, a set of 6 sample points listed by triangles are points on the lane boundary determined by GT device GT-D1, while two sets of sample points shown by circles and squares are points on the lane line boundary determined by GT devices GT-D2, GT-D3. The sampling points may be determined from the ground truth data record according to a specific time sequence (t 1, t2, t3 …). The number and location of sampling points on one timestamp is determined for each timestamp (e.g., t 1) of the time series. The amount of data analyzed and processed can be reduced, and the evaluation efficiency can be improved without reducing the evaluation accuracy. The interval of each time stamp t may be, for example, 100 milliseconds. A set of sampling points corresponding to each timestamp is determined at a specified bit order to meet data interface requirements for evaluating ground truth data. For example, a set of samples at a time stamp includes 101 samples arranged in bit order within 200 meters in front of the host, each sample being spaced 2 meters apart. Those skilled in the art will appreciate that the sampling points may also be determined in other ways. It will also be appreciated by those skilled in the art that only one lane line is shown in fig. 3, and that the GT device may of course determine the point on the lane line on the other side in the same manner.

As shown in fig. 3, since the sets of GT data are provided according to the same specification, the three sets of sampling points at time t1 are all composed of 3 sampling points at the same bit order, as shown by the dashed oval in the figure. As described above, a predetermined distance in front of the vehicle may be specified, and the arrangement of the sampling points may be determined at predetermined intervals. For a set of samples at the same time t1, their values in the x-axis direction are equal, but have different values in the y-axis, due to the same bit times. Those skilled in the art will appreciate that the two-dimensional vehicle coordinate system herein is merely an example, and that a three-dimensional coordinate system may be employed in other situations.

In certain cases there is a subtle difference in the transmission delay and trigger delay for the hardware/software system of the three GT devices, although the same start acquisition time and time sequence are set, the data of the same timestamp t1 will not be aligned. As described above, in order to further improve the time synchronization accuracy of GT data, the time stamps of other GT devices will be compensated according to the delay information.

Fig. 4 is a schematic diagram of an exemplary composition of multiple sets of GT data according to an embodiment of the present application. Fig. 4 shows in tabular form three sets of GT data determined by GT devices GT-D1, GT-D2, GT-D3, each set of GT data comprising a plurality of sets of sampling point SP data. Each set of sample point data of the plurality of sets of sample point data corresponds to each time stamp of the time series, e.g. at time t1 to sample points sp1 to sp4. Wherein a set of sampling points sp1 to sp4 consists of a plurality of sampling point values with bit times, each sampling point value indicating the distance of the detected lane boundary to the longitudinal axis in the vehicle coordinate system. For example, for the GT device GT-D1, which is installed on the vehicle V1, it detects the generation of GT data records from which a set of GT data is determined. The set of GT data corresponds to each time stamp T1 to T5 of the time series T with a set of sampling points sp1 to sp4. It will be appreciated by those skilled in the art that the time stamps T1 to T5 and the sampling points sp1 to sp4 of the time series T are only examples herein. Accordingly, GT-D2 and GT-D3 also determine the sampling points of the same bit-order at the same time stamp. The data for each sampling point includes values of the abscissa of the detected lane boundary in the vehicle coordinate system. In the presence of delay information, the time sequences of GT-D2 and GT-D3 may be compensated, for example, by using GT-D1 as a reference device, resulting in time sequences t1' to t5' of GT-D2 and time sequences t1' to t5 "of GT-D3.

Returning to fig. 2, the plurality of GT devices is evaluated based on the GT data in step 230. As described above, each set of GT data is composed of a plurality of sets of sample point data, one set of sample point data corresponding to one time stamp of the time series, and one set of sample point data is composed of a plurality of sample point values having a bit number, each sample point value indicating a distance from a lane boundary detected under a vehicle coordinate system to a vertical axis. In the evaluating step, the plurality of GT devices may be evaluated according to positional relationships between the plurality of GT devices at a plurality of sampling points having the same bit times of the corresponding time stamps.

In some examples, a plurality of sample point values with the same bit order in each set of sample point data for each GT device at each same time stamp (e.g., t1 in fig. 4) are fused to obtain corresponding fused sample point values. For example, the values of the sampling points sp1 at time t1 of fig. 4 are fused for all GT devices. In other examples, for the case where delay information is present, a plurality of sample point values (e.g., a plurality of sp1 values) having the same bit times in each respective time stamp, i.e., each set of sample point data of compensated time stamps (e.g., t1', t1″ in fig. 4) are fused for each GT device to obtain a corresponding fused sample point value.

As shown in fig. 5, three sampling points A, B, C of the same bit times of the same/corresponding time stamps are shown. Here, the corresponding time stamp refers to a compensated time stamp. The position of the sample point A, B, C in the figure corresponds to the position of the horizontal axis of the sample point vehicle coordinate system. The distance between sampling points a and B is d1 and the distance between sampling points B and C is d2. In the first case at the top of the figure, when the sampling points B and C are closer and the ratio of d1 to d2 is greater than a first threshold (e.g., greater than 2), the sampling point a is regarded as an abnormal sampling point. At this time, the sampling point a is excluded when the fused sampling point is calculated, and only the point B and the point C are considered. The value yF of the fusion sampling point F on the abscissa of the vehicle coordinate system is the average of the values of point B and point C, i.e., (yB+yC)/2. In the second case at the bottom of the figure, when the sampling points a and B are closer and the ratio of d1 to d2 is smaller than the second threshold (for example, smaller than 0.5), the sampling point C is regarded as an abnormal sampling point. At this time, the sampling point C is excluded when the fused sampling point is calculated, and only the points a and B are considered. The value yF of the fusion sampling point F on the abscissa of the vehicle coordinate system is the average of the values of points A and B, i.e., (yA+yB)/2. In this example, regarding three samples A, B, C of the same order, regarding a closer sample as a higher accuracy sample and excluding an outlier sample may better evaluate the GT device providing ground truth data. This example shows only the case of three GT devices, and one skilled in the art will appreciate that other ways of excluding outlier samples may be used depending on the positional relationship between the plurality of samples. For example, an average value of values of a plurality of sampling points is obtained, and sampling points deviating from the average value by more than a threshold value are excluded.

FIG. 6 is a schematic diagram of an exemplary fused sample point based on weight calculation according to an embodiment of the present application. Similar to fig. 5, fig. 6 also shows three sampling points A, B, C of the same bit-times of the same/corresponding time stamps. The position of the sample point A, B, C in the figure corresponds to the position of the horizontal axis of the sample point vehicle coordinate system. The distance between sampling points a and B is d1 and the distance between sampling points B and C is d2. At this time, if the ratio of d1 to d2 is smaller than the first threshold (for example, smaller than 1.2) and larger than the second threshold (for example, larger than 0.8), three sampling points are simultaneously considered in calculating the fusion sampling point. In this case, there are no "outlier" sampling points. The value yF of the fusion sampling point F on the abscissa of the vehicle coordinate system is 2/3 (d 2/(d1+d2) yA+d1/(d1+d2) yC) +1/3yB. As shown, more weight is considered to be given to sample a that is closer to sample B when calculating the value of the fused sample. This example shows only the case of three GT devices, and one skilled in the art will appreciate that other means may be used to weight each sample point depending on the positional relationship between the multiple sample points. For example, an average value of values of a plurality of sampling points is obtained, and sampling points closer to the average value are given a larger weight.

After calculating the values of the fused sampling points, the GT device may be evaluated from these values. And calculating the distance between the sampling point of the GT device at the same level and the corresponding fusion sampling point aiming at each of the plurality of GT devices to obtain a group of distance data of the GT device at the level, and evaluating the GT device according to the standard deviation of the group of distance data. Taking GT-D1 as an example, calculating the distance between the sampling point of the same bit and the corresponding fusion sampling point at each time stamp refers to calculating the distance between sp1 of GT-D1 at time t1-t4 and the fusion sampling point of the fusion sampling point set at time t1-t4, which may be the absolute value of the difference between the two. From this, a set of distance data [ t1 (sp 1-sp 1), t2 (sp 1-sp 1), … ] between the sp 1-bit sampling point of the GT-D1 device and the corresponding fusion sampling point in time series can be obtained. t1 (sp 1-sp 1) refers to the distance between the sampling point of the bit sp1 and the fusion sampling point sp1 at time t 1. The degree of dispersion of the sampling points of the GT-D1 device at sp1 bit times can be estimated based on the set of standard deviations of the distance data, thereby estimating the stability of the detection device. The same can be done for other bit-wise sampling points (e.g., sp2 through sp 4).

The stability and authenticity of the GT device's provided ground truth data at a particular level is evaluated by calculating the standard deviation of a set of distance data (the set of absolute values of the differences) at that level. Similarly, the same calculation is performed for other GT devices to obtain the standard deviation of a set of distance data (the set of absolute values of the differences) for that GT device at a particular level. The GT device corresponding to the group of distance data with lower standard deviation has higher stability and authenticity of the sampling point data on the bit corresponding to the group of distance data. Thus, an efficient method of screening GT devices based on less data is provided herein. The selected GT device will provide more reliable ground truth data.

Fig. 7 is a schematic diagram of a system 700 for evaluating ground truth GT device data according to an embodiment of the present application. The system 700 is used to implement, for example, the method step operations in fig. 2, and is applied to the processes shown in fig. 3-6. The system 700 may include a processor 710 and a memory 720. Processor 710 and memory 720 communicate via a bus, or may communicate via other means such as wireless transmission. The memory 720 is used to store instructions and the processor 710 is used to execute the instructions stored by the memory 520. Memory 720 may include Read Only Memory (ROM), random Access Memory (RAM). Processor 710 may invoke program modules stored in memory 720 to perform the various steps of the methods described above. The system 700 also includes memory and communication interfaces not shown in fig. 7. The system 700 obtains multiple sets of GT data via input/output (I/O) interfaces. Computer program code stored in memory 720 is loaded into memory for execution by processor 720. In the example of fig. 7, the memory 720 includes three program modules, namely a determination module 721, an acquisition module 722, and an evaluation module 723.

The determination module 721 determines delay information between a plurality of GT devices mounted on the same vehicle. A reference GT device may be selected from the plurality of GT devices and delay information for each other GT device other than the reference GT device described above is calculated with respect to the reference GT device. Optionally, the delay information between the two GT devices is determined by performing an alignment process on the respective inertial measurement unit signals of the two GT devices. The inertial measurement unit signal may be an angular velocity signal or a lateral acceleration signal. The reference time sequence of the reference GT device can be determined, and the time sequence can be compensated according to the delay information to obtain the time sequences corresponding to other GT devices

The acquiring module 722 acquires a corresponding set of GT data with each of the plurality of GT devices according to the delay information, wherein the sets of GT data are synchronized according to the delay information. The acquisition module 722 may receive multiple sets of GT data through various communication means, such as wired, wireless, etc. The acquisition module 722 acquires respective sets of GT data from the plurality of GT devices configured to detect lanes according to detection conditions.

The evaluation module 723 evaluates the plurality of GT devices based on the sets of GT data. The plurality of GT devices may be evaluated in a positional relationship between a plurality of sampling points having the same bit times of the corresponding time stamps.

One or more of the embodiments, various examples described above, may be implemented in whole or in part by software, hardware, firmware, or a combination thereof. When implemented in software, embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from a network server to another computer by wire or wirelessly.

In various embodiments of the present application, the program modules and apparatus described are illustrative only. The division of the functional units is a logic function division, and other division modes can be adopted when the division is realized. Multiple units and devices may be physically separated, or may be distributed over a network unit, which may be combined or may be integrated into another system. The foregoing is merely a specific embodiment of the present application. Variations or alternatives will occur to those skilled in the art from the detailed description provided herein and are intended to be within the scope of the present application.

Claims (10)

1. A method of evaluating ground truth GT apparatus, comprising:

a determining step of determining delay information between a plurality of GT devices mounted on the same vehicle;

an acquisition step of respectively acquiring a corresponding group of GT data by each of the plurality of GT devices according to the delay information, wherein each group of GT data is synchronized according to the delay information;

an evaluation step of evaluating the plurality of GT devices based on the sets of GT data.

2. The method of claim 1, wherein,

in the determining step, a reference GT device is selected from the plurality of GT devices, and delay information of each other GT device except the above-mentioned reference GT device with respect to the reference GT device is calculated.

3. The method of claim 2, wherein,

in the determining step, delay information between the two GT devices is determined by performing an alignment process on respective inertial measurement unit signals of the two GT devices.

4. The method of claim 3, wherein,

in the determining step, the inertial measurement unit signal is an angular velocity signal or a lateral acceleration signal.

5. The method of claim 4, wherein,

in the determining step, a reference time sequence of the reference GT device is also determined, and the reference time sequence is compensated according to the delay information to obtain time sequences corresponding to other GT devices.

6. The method of claim 4, wherein,

in the acquiring step, respective sets of GT data are acquired from the plurality of GT devices configured to detect lanes according to detection conditions, wherein the detection conditions include at least one of: a specified time period, a specified weather condition, and a specified road segment.

7. The method of claim 4, wherein,

the set of GT data of the reference GT device is a set of sample point data of lane boundaries according to a reference time sequence, and each set of GT data of the other GT devices is a set of sample point data of lane boundaries according to respective compensated time sequences.

8. The method of claim 7, wherein,

each set of GT data is composed of a plurality of sets of sampling point data, one set of sampling point data corresponds to one time stamp of the time sequence, and one set of sampling point data is composed of a plurality of sampling point values having a bit order, each sampling point value indicating a distance from a lane boundary detected under a vehicle coordinate system to a vertical axis, wherein,

in the evaluating step, the plurality of GT devices are evaluated according to their positional relationship between the plurality of sampling points of the same rank of the corresponding time stamp.

9. A system for evaluating ground truth GT devices, characterized in that the system comprises a processor and a memory, wherein the memory stores computer program instructions that, when executed by the processor, are capable of performing the method according to any of claims 1-8.

10. A computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the method of any of claims 1 to 8.