KR20210108600A - Autonomous driving situation recognition program performance test method and apparatus for porceeding the same - Google Patents

Abstract

According to an embodiment of the present invention, an autonomous driving situation recognition program performance test method comprises: a step of obtaining original data related to autonomous driving of a vehicle; a step of obtaining reference data processed from the original data to conform to a predetermined data type; a step of executing the original data through a predetermined test program to generate result data; and a step of outputting verification data comparing the reference data with the result data.

Description

AUTONOMOUS DRIVING SITUATION RECOGNITION PROGRAM PERFORMANCE TEST METHOD AND APPARATUS FOR PORCEEDING THE SAME

The present invention relates to a method for testing performance of an autonomous driving situation recognition program, and more specifically, by executing original data acquired in an autonomous driving situation through a predetermined test program, and matching the result data and the original data obtained to a predetermined data format This is to measure the performance of the test program by comparing the reference data processed to will be.

When driving a vehicle, it is necessary to accurately recognize the surrounding conditions of the vehicle and to accurately control the vehicle, such as stopping so as not to collide with the vehicle. To this end, it is important to secure the reliability of the autonomous driving test program, and object recognition information is sometimes used to measure the performance of the test program. However, in the case of the conventional test program performance measurement method, there is a problem that the accuracy of the test program performance measurement result is relatively low because only object recognition information is used.

The present invention is to solve the above problems, and to measure the performance of the test program by comparing the original data with reference data processed to fit a predetermined data format with the result data executed through a predetermined test program. has its purpose in

The present invention also considers object type, object distance, and object speed information in addition to object recognition information included in the reference data, so that the accuracy of performance measurement of the autonomous driving test program is further improved. There is a purpose.

The present invention is also used to improve the accuracy of performance measurement of an autonomous driving test program by using a calculated value calculated by giving weight information to the type of object, the distance of the object, and the speed information of the object. There is a purpose.

Another object of the present invention is to obtain a more accurate object recognition result by using fusion information of image data and sensing data.

An autonomous driving situation recognition program performance test method according to an embodiment includes: acquiring original data related to autonomous driving of a vehicle; acquiring reference data processed so that the original data is suitable for a predetermined data format; the original data and generating result data by executing the ? through a predetermined test program and outputting verification data obtained by comparing the reference data with the result data.

The outputting of the verification data may be performed based on at least one of whether an object included in the reference data is recognized, a type of the object, a distance of the object, and speed information of the object.

The verification data may be calculated by giving weight information to at least one of the type of the object, the distance of the object, and the speed information of the object.

The verification data output is performed in units of frames including the object,

The method may further include measuring the performance of the predetermined test program with an average value of the verification data output in units of the frame.

Whether the object included in the reference data is recognized is a positive recognition when the object included in the reference data matches the object included in the result data, and object information not included in the reference data is included in the result data It may correspond to any one of erroneous recognition and unrecognized object information included in the reference data is not included in the result data.

The type of the object included in the reference data matches the type of the object included in the result data, and the object coordinate information included in the reference data and the object coordinate information included in the result data match each other. When the matching area is equal to or greater than a predetermined threshold, it may be determined as the correct recognition.

The reference data may be generated through a process of labeling the object and adding an annotation to fit the predetermined data format.

The predetermined data format may be a JavaScript Object Notation (JSON) format.

The original data may be generated by fusion of image data generated by photographing the surroundings of the vehicle and sensing data generated by a sensor of the vehicle.

An autonomous driving situation recognition program performance test apparatus according to another embodiment of the present invention includes an original data acquisition unit that acquires original data related to an autonomous driving situation of a vehicle, and reference data processed so that the original data conforms to a predetermined data format. a reference data acquisition unit for obtaining can do.

In addition to the object recognition information included in the reference data, by considering the object type, object distance, and object speed information, the accuracy of performance measurement of the autonomous driving test program can be further improved.

By using a calculated value calculated by giving weight information to the object type, the object distance, and the object speed information, the accuracy of the performance measurement of the autonomous driving test program can be further improved.

It is possible to obtain a more accurate object recognition result by utilizing the fusion information of the image data and the sensing data.

1 is an overall system diagram of an autonomous driving situation recognition program performance test apparatus and vehicle according to an embodiment;
2 is a flowchart illustrating a method for testing an autonomous driving situation recognition program according to an embodiment;
3 to 8 are diagrams referenced to describe a method for testing an autonomous driving situation recognition program according to an embodiment;

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0010] Reference is made to the accompanying drawings, which show by way of illustration specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein with respect to one embodiment may be embodied in other embodiments without departing from the spirit and scope of the invention. In addition, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the following detailed description is not intended to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all scope equivalents to those claimed. Like reference numerals in the drawings refer to the same or similar functions throughout the various aspects.

1 is a view showing a detailed block diagram of an autonomous driving situation recognition program performance test apparatus 1 together with an overall system diagram of an autonomous driving situation awareness program performance testing apparatus 1 and a vehicle 2 according to an embodiment, and FIG. 2 is a flowchart for describing a method for testing an autonomous driving situation recognition program according to an embodiment.

3 to 8 are diagrams referenced to describe a method for testing an autonomous driving situation recognition program according to an embodiment.

As shown in FIG. 1 , the autonomous driving situation recognition program test apparatus 1 according to the embodiment may include a control unit 10 and a storage unit 20 .

The control unit 10 is in charge of overall control for data transmission/reception and processing of the autonomous driving situation recognition program test device 1 , and includes an original data acquisition unit 110 , a reference data acquisition unit 120 , and a result data generation unit. 130 , and a verification result output unit 140 .

The storage unit 20 may store data processed by the control unit 10 or may store data received from an external device such as the vehicle 2 . The control unit 10 may read data stored in the storage unit 20 .

The storage unit 20 may include an original data DB 21 and a verification data DB 22 .

The original data DB 21 may store image data and sensing data related to autonomous driving of a vehicle for each scenario.

The verification data DB 22 may store verification data obtained by comparing reference data and result data.

Hereinafter, a method for testing an autonomous driving situation recognition program will be described in detail with reference to FIGS. 1 and 2 together.

First, the original data acquisition unit 110 may acquire original data related to autonomous driving of the vehicle from the vehicle 2 ( S210 ).

The original data related to autonomous driving of the vehicle 2 according to the embodiment includes image data generated in the vehicle 2 by photographing the surroundings of the vehicle 2 and sensing data generated from the sensor 222 of the vehicle 2 . may be generated by fusion.

The image data generated in the vehicle 2 is obtained by photographing the surrounding environment by the camera 221 of the vehicle 2, and includes omnidirectional image data obtained by photographing the surrounding environment of the vehicle in all directions, such as a 360-degree camera. can

The sensor 222 of the vehicle 2 is for detecting an object located outside the vehicle 2 , and may generate object information based on sensing data and transmit the generated object information to the ECU 210 . The objects may include, for example, various objects related to the operation of the vehicle 2 , for example, lanes, other vehicles, pedestrians, traffic signals, and roads, structures, speed bumps, animals, features, and the like. .

The sensor 222 according to the embodiment may include a lidar sensor (not shown), a radar sensor (not shown), an infrared sensor (not shown), an ultrasonic sensor (not shown), and the like.

The lidar sensor (not shown) may detect the size and arrangement of the object by irradiating a laser light pulse to the object and analyzing the light reflected from the object, and may map the distance to the object.

The sensor 222 may include a collision sensor, a wheel sensor, a speed sensor, a gyro sensor, a steering wheel sensor, and the like, and may include any kind of sensor capable of detecting a signal related to the state of the vehicle 2 . Vehicle collision information, vehicle direction information, vehicle location information, vehicle acceleration information, vehicle inclination information, and the like may be generated as sensing data.

Information of the sensor 222 and the camera 221 may be transmitted to the ECU 210 through in-vehicle communication such as CAN, LIN, or Ethernet.

The information of the sensor 222 and the camera 221 transmitted to the ECU 210 is transmitted to the autonomous driving situation awareness program test device 1 using wired/wireless communication through a communication unit (not shown) in the vehicle 2 . can be According to another embodiment, the information of the sensor 222 and the camera 221 transmitted to the ECU 210 is previously recorded in the storage unit 20 as the original data DB 21 , and then the original data acquisition unit 110 . can be obtained by In the original data DB 21 , fusion information of image data and sensing data may be prepared for each autonomous driving situation and recorded as a scenario list. Each scenario list may include image information, and each image may be composed of frames including object information.

The reference data acquisition unit 120 may acquire reference data processed so that the original data conforms to a predetermined data format ( S220 ).

Specifically, as illustrated in FIGS. 3A and 3B , the reference data is a process of labeling an object and annotating it to fit a predetermined data format.

It may be created through (Annotation). Here, the predetermined data format may be a JSON (JavaScript Object Notation) format, but the scope of the present invention is not limited thereto.

The original data is sensor data such as image, lidar, and radar, and an annotation process may be performed to output it as reference data. It can be performed by dividing the acquired image into frame units (eg 3 to 30 frames per second), using a pre-developed annotation tool to manually label objects within the frame, and input distance information using sensing data. When the annotation work is finished, it is output according to the data standard. At this time, the data format can be easily extended and JSON (JavaScript Object Notation) format can be used to organize the properties of each object.

As illustrated in FIGS. 3A and 3B , in the case of a moving object, the attribute value of the object includes object type (type), relative speed, absolute speed, distance information, angle, length, width, height, object detection position, etc. may contain information. And, in the case of a lane (non-moving fixed object), the attribute value of the object may include information such as lane style, lane type, and lane direction. In the process of storing the information related to each object shown in FIG. 3 in the form of a table using ANNOTAION, field information for identifying whether each object attribute information is included in the reference data and/or result data may also be reflected. have.

In particular, in the case of an object detection position, different information may be output according to the type of object. For example, a moving object such as a pedestrian or a car outputs the upper left and lower right coordinates of a bounding box (2D box), and a fixed object such as a lane or parking line can output the coordinates of vertices in a polyline type. For reference, a bounding box may be defined as a window area surrounding an object to be detected.

The result data generation unit 130 may generate result data by executing the original data through a predetermined test program (S230).

The result data is a result output when the original data is applied to a predetermined test program, and the format of the output result data at this time may also be output to conform to a predetermined data format. According to an embodiment, a predetermined data format of the result data may also use a JSON (JavaScript Object Notation) format.

The result data may be implemented to necessarily include detection position information (coordinate information) of a moving object and position information of a lane (coordinate information).

The verification data output unit 140 may output verification data obtained by comparing the reference data and the result data. The output verification data can be visualized and shown to the operator.

The verification data output unit 140 may compare the object recognition result of the result data with reference to the reference data, evaluate the result of comparison and measurement according to the reference, and output the verification data (S240).

The verification data output unit 140 may output verification data based on at least one of whether the object included in the reference data is recognized, the type of the object, the distance of the object, and the speed of the object.

The verification data output unit 140 may output verification data in units of frames including objects, and may measure the performance of the predetermined test program with an average value of the verification data output in units of frames.

According to an embodiment, whether an object included in the reference data is recognized is a positive recognition when an object included in the reference data and an object included in the result data match, and a misrecognition that object information not included in the reference data is included in the result data. , and object information included in the reference data may correspond to any one of unrecognized that is not included in the result data.

Specifically, the object included in the reference data and the type of the object included in the result data match (eg, vehicle-vehicle, pedestrian-pedestrian, lane-lane) and coordinate information of the object included in the reference data and When the matching area among the coordinate information of the object included in the result data is greater than or equal to a predetermined threshold (eg, when the IOU value is greater than or equal to 0.5), it may be determined as the correct recognition.

Here, whether the coordinate information of the object is equal to or greater than a predetermined threshold may be measured through an IOU (Intersection of Union) index. Object detection may be performed by measuring IOU, which is an indicator of how well the bounding box is predicted.

Accuracy calculation in object detection may be performed by comparing the correct answer (Ground Truth, GT) with the prediction result predicted by the test program (Predeciton). When the correct answer (GT) and the prediction result (Prediction) match 100%, the IOU value can have 1, and when the IOU value is 0.5 or more, it can be recognized as the correct answer. However, 0.5 is only an example and can be applied to other numerical values.

For example, as shown in FIG. 4B , two bounding boxes can be compared. If the types of objects match and the IOU calculation result is 0.5 or more, the corresponding object can be determined to be correctly recognized.

According to the embodiment, the IOU can be applied not only to the bounding box but also to the object recognized in the polygon shape. In particular, the polygon IOU can be used when comparing and measuring data output in the polyline shape among the objects. As shown in FIG. 4C , when recognizing a lane, vertices of a polyline are input on a data standard, and a Polygon can be created by connecting these vertices. And the IOU of the polygon can be measured by calculating the size of the part where the polygons of the reference data and the result data overlap and the total size.

In the case of misrecognition, an object that does not exist is recognized, and when object information not included in the reference data outputs information such as object coordinate information or object type from the result data, it can be classified as misrecognition.

In the case of unrecognized, an object exists but it is not recognized, and an error occurs even though the object information is not output in the result data even though there is an object to be recognized in the frame, or even though the object is recognized and information is output. It can be classified as unrecognized when the type of object does not match or the number of IOUs is less than a threshold.

For example, referring to FIG. 4A , the dotted line is a bounding box indicating the object coordinates of the reference data, and the solid line is a bounding box indicating the object coordinates of the result data. can

If A is Type = vehicle, and IoU is 0.5 or higher, it is correct recognition.

If A is Type ≠ vehicle or IoU is less than 0.5, it is not recognized.

If B is Type = Dotted Lane, and IoU is 0.5 or higher, it is correct.

If B is Type ≠ Dotted Lane or IoU is less than 0.5, it is not recognized.

C is a misrecognition because the coordinates of objects not in the reference data are output from the result data.

D is unrecognized because the coordinates of the object existing in the reference data cannot be output from the result data.

Hereinafter, a method of comparing object types will be described for reference.

In general, the calculation of accuracy in object detection is mainly performed through comparison between the correct answer (Ground Truth, hereinafter GT) and the result predicted by the model (Prediction). In object detection, GT means the corresponding bounding box of each object of the image and the class in the box. That is, high accuracy means that the model predicts (regression) the bounding box similar to GT and at the same time predicts the class of the object in the box well (classification). In other words, it is necessary to accurately predict the class and at the same time predict the area of the object well.

In the case of accuracy used in object detection, if the class cannot be predicted, it is considered a failure. The accuracy is measured based on the accuracy of the bounding box when the class is correctly predicted. Similarly in the present invention, if the class is not accurately predicted, it is a cognitive failure. However, in the present invention, instead of the term "Class", the name "Object Type" is used according to the data standard.

Hereinafter, an object detection metric and a performance measurement method of a test program according to an embodiment will be described with reference to FIGS. 5A and 5B .

In a problem such as statistical classification in the field of machine learning, a confusion matrix is a table that can visualize the performance of a classification algorithm, and FIG. 5A is an example thereof. Each row of the matrix represents an instance of the predicted class, and each column represents an instance of the actual class (or vice versa). A look at this for each case is as follows.

True Positive (TP): Predicting a true answer as true (correct answer)

False Positive (FP): Predicting a true false answer as true (wrong answer)

False Negative (FN): Predict a true true answer as False (wrong answer)

True Negative (TN): Predicting a correct answer that is actually False as False (correct answer)

When the above-described confusion matrix is applied to the test program of the present invention, it can be defined as follows.

True Positive - Recognizes an object that actually exists.

False Positive - Recognizes an object that does not actually exist (misrecognition)

False Negative - Does not recognize an actual object (unrecognized)

True Negative - Does not recognize objects that do not actually exist (Don't care)

Performance is measured by calculating indices such as Precision, Recall, and Average Precision based on the above four perceptions.

go. Precision

Precision is the ratio of objects that actually exist among recognized objects (output coordinates). It can be expressed in the following way.

In the present invention, it can be expressed as a ratio of the recognition results output by the result sheet that are genuine recognition.

me. Recall (recall rate)

Recall is the ratio of the perceived existence of the result map among the actually existing objects. It can be expressed in the following way.

In the present invention, it can be expressed as a ratio of the recognition results (recognition target object) outputted on the correct answer sheet (correct recognition) recognized by the correct answer sheet.

Precision and Recall are values that change dynamically according to the parameter adjustment of the algorithm. For example, in order to reduce the rate of misrecognition, if only clearly recognized results are outputted, the precision is close to 1. Conversely, if the result is output for all coordinates, the Recall becomes 1 because there is no recognition. Therefore, precision and recall have an inverse relationship, and in order to properly compare the performance of the test program, it is necessary to look at the entire performance change of precision and recall. The concept used here is Average Precision (AP). Average precision expresses the performance of the recognition algorithm as a single value. The higher the average precision, the better the overall performance of the algorithm. In the field of computer vision, the performance of object recognition algorithms is mostly evaluated with average precision.

As in the graph of Figure 5b, the calculation of Average Precision inevitably decreases the precision when the Recall is increased from 0 to 0.1 unit and increased to 1 (0, 0.1, 0.2, ..., 1). By calculating the Precision value for each unit, Calculate by averaging That is, the average of the precision values according to the 11 recall values means the AP. In this way, the average value obtained by calculating the AP for the entire class is the mean average precision (mAP). mAP is obtained by obtaining one AP value per one object and obtaining the mean value for several object-detectors.

Also in the present invention, the mAP can be measured by comparing the reference data and the result data. By comparing and measuring the object recognition result of the reference data and the result data with the IOU and Class (Type), it is possible to determine whether the object is recognized or not, and it can be used as a performance measurement index by calculating the mAP as an index for the result. However, unlike other object recognition, in the present invention, in particular, a test program for recognizing the autonomous driving situation of a vehicle needs to be verified. Since the specificity of the driving situation must be reflected, there are other factors to consider besides metrics such as mAP.

In consideration of this, as described above, the verification data output unit 140 according to an embodiment of the present invention provides information on at least one of the type of object, the distance of the object, and the speed of the object in addition to whether the object included in the reference data is recognized. Based on the verification data can be output.

One) Specifically, the type information of the object may be described as follows.

In general, since the occurrence of a traffic accident during autonomous driving may be more dangerous for a pedestrian than a vehicle, if the cognitive evaluation result of the test program is changed in consideration of the type of object, the accuracy of object detection increases.

2) Specifically, the distance information of the object may be described as follows.

In the present invention, the data acquired to verify the test program includes not only camera images but also various sensor data such as lidar, radar, and ultrasonic sensors. By synchronizing these sensor data to each image frame (hereinafter, sensor fusion), it is possible to know the degree of distance to the objects in the image frame. If this distance information is assigned to an object as an attribute value during the annotation process, the reference data has distance information between the vehicle and each object.

There are two reasons to consider the distance between the vehicle and the object in the evaluation of the driving situation awareness test program. First, you need to determine the maximum recognition distance. Depending on the acquisition conditions or environment, there may be objects that are too far away or appear small because they are obscured by other objects. Therefore, first, the maximum distance of the object to be recognized should be determined. However, since no international standard for cognitive test program verification has been established, standards can be established by referring to data such as individual international standards of the ADAS system.

First, referring to the standard for Adaptive Cruise Control System (ISO 15622), the detection range on a straight road for following vehicle in the ACC method is stipulated.

Referring to FIG. 6A , distance_max for the vehicle in front on a straight road is set to 100m.

Also, reference can be made to the method of calculating the stopping distance in traffic engineering.

Figure 6b shows a method of obtaining the stopping distance including the braking distance and the gonggong distance in terms of traffic engineering. When a vehicle traveling at a speed of 100 km/h (27.8 m/s) detects a risk factor in front and brakes, the safe distance is 100 m by adding the driver's reaction time to the idle distance and braking distance. Driver reaction time is a factor that can be excluded due to the nature of autonomous driving, but considering that SAE Level 3 is targeted, it is reasonable to include @ (reaction time) in the above calculation. Under the current traffic law, it is assumed that the speed limit of 100 km/h on highways is observed, and the maximum braking distance is applied to 100 m. Therefore, the maximum distance of the object to be recognized in this task is 100m from the acquisition vehicle.

In addition to the maximum distance of the perceived object, the distance between the perceived object and the acquired vehicle is also a consideration in the cognitive SW evaluation. This is because, as considered in Type, the distance between the acquisition vehicle and the object to be recognized is a factor that can affect the accident of an autonomous vehicle. For example, if there are two recognition target objects (vehicles) in one image frame, one is 20m away from the acquisition vehicle and the other is 60m away from the acquisition vehicle, it cannot be treated at the same level as in the existing mAP calculation. Assuming sensor fusion, it is necessary to evaluate the cognitive result differently according to the distance using the distance of each object.

3) Specifically, the speed information of the object may be described as follows.

Since it is an evaluation of cognitive SW based on the driving situation, the speed of the vehicle cannot be evaluated.

This is an important indicator to consider. The speed-related index required for cognitive SW evaluation is the speed of the vehicle and the relative speed of the object to be recognized.

In the present invention, the data acquired to verify the cognitive SW also includes CAN information of the acquired vehicle. Through CAN information, it is possible to know the speed of the vehicle at the time of acquisition. It is necessary to establish an evaluation index that takes into account the speed of the vehicle acquired while driving, taking into account the fact that the speed is input through CAN information and that the maximum speed is stipulated in the Road Traffic Act.

라 하고, B 물체의 속도를

라고 할 때, A가 측정한 B의 속도(A에 대한 B의 상대속도)는 In the present invention, an index to be referred to more important than the speed of the acquisition vehicle is the relative velocity with the recognition target object. Relative velocity is the velocity at which an observer moving with A sees the motion of B when objects A and B are in motion. The velocity of object A in one coordinate system

, and the velocity of object B is

, the speed of B measured by A (relative speed of B with respect to A) is

이다.

am.

The relative speed required in the present invention is the relative speed of the object to be recognized with respect to the acquisition vehicle. For example, if there are two recognition target vehicles that are the same distance apart in the frame acquired from a vehicle traveling at 50 km/h, and one has a relative speed of 50 km/h and the other has a relative speed of -50 km/h, the former vehicle is a vehicle running 50 km/h faster (100 km/h speed) than the acquired vehicle, and the latter vehicle is 50 km/h slower (stopped due to 0 km/h speed). The former perception object will move away from it as it goes to the next frame, but the latter object will get closer and closer. In this case, if the two objects are treated at the same level, the data will be inaccurate. Therefore, on the premise that the relative speed with each object can be measured and the data is obtained through sensor fusion, the above relative speed is also considered for evaluation.

The verification data output unit 140 according to an embodiment of the present invention may calculate verification data by assigning weight information to at least one of an object type, an object distance, and an object speed information.

One) Specifically, the weight reflecting the type of object is described as follows.

When there are two objects of different types in a situation where the remaining conditions such as distance are the same in a specific frame, how much difference in importance will be given between the two objects is the method to determine the weight for each type. According to the standard object classification for cognitive object properties, weights for each type were assigned as shown in FIG. 7A. In the present invention, the importance was selected based on only pedestrians and vehicles among the object properties, but the difference in each type was set to prevent a score reversal when calculating with other indicators to be weighted. will be.

The reason that pedestrians are more important than vehicles is that pedestrians in car-to-person traffic accidents have a higher mortality rate than occupants in car-to-vehicle accidents. The weight due to the difference in the type of two objects under the same condition can be selected in such a way that the difference in importance is given between the two objects.

2) Specifically, the weight reflecting the distance of the object is described as follows.

The distance considered when evaluating the cognitive SW is the distance between the acquisition vehicle and the object to be recognized. The closer the distance between the object and the acquisition vehicle, the higher the probability of an accident with the object in the driving situation. In the present invention, since the longest distance of the object to be recognized is set to 100 m, a weight section is made by subdividing it by 20 m inside 100 m, and the weight is set to increase as the distance increases (FIG. 7b). The weights within each section are reflected by interpolation.

The weight for each distance is suitable for the evaluation method in which a higher weight is given as the distance of the object to be recognized is closer, so that the object with a high probability of causing an accident is evaluated more strictly, but considering the autonomous driving situation, it is necessary to consider the vehicle speed there is also Therefore, in the present invention, time to collision (TTC), which is a combination of the distance index and the speed index, is used as an evaluation index.

3) Specifically, the weight reflecting the speed information of the object is described as follows.

Time to collision (TTC) refers to the time remaining until a collision between vehicles, and is currently used in functions such as Forward Collision Warning (FCW) and Autonomous Emergency Braking (AEB). TTC is calculated by dividing the relative distance by the relative speed. If the distance from the vehicle ahead is D, the vehicle traveling speed is Vr, and the speed of the vehicle in front is Vf, then the expected collision time is T = D / (Vr - Vf). The TTC used for FCW or AEB targets a vehicle in the same lane as the driving vehicle, so the calculation of relative speed, relative distance, and TTC is performed simply as above. However, in the present invention, an index having a different range is required to be applicable not only to a front collision, but also to a change in a surrounding lane, a vehicle crossing in a parking lot, or a collision at an intersection. Therefore, it is possible to extend the TTC so that it can be used in a two-dimensional space so that it can be used even if it is not in the same lane.

7C shows a conceptual diagram of a relative distance and a relative speed.

As shown in FIG. 7C , it is necessary to calculate the TTC by measuring the relative distance and relative speed with respect to the other vehicle (or object) located in a position other than the same lane.

7c, the relative distance and the relative speed can be expressed as follows.

The formula for calculating TTC using the above relative distance and relative speed is as follows.

The above values are calculated from the data acquired through sensor fusion, and if the recognition target object is a vehicle or a moving object such as a pedestrian, the TTC between the acquired vehicle and the recognition target object is calculated and included in the reference data. That is, the reference data JSON can be implemented to include at least the relative distance and speed (or TTC) as well as the type and coordinates of the object to be recognized.

This TTC can be reflected when performance is measured according to whether the recognition target object is authentic or unrecognized when evaluating the comparison result. For example, if the object to be recognized is a moving vehicle, the importance of the object varies depending on the TTC between the acquired vehicle and the vehicle, and different weights are given during evaluation. In the present invention, when there is a recognition target object (vehicle) within 100 m determined as the maximum recognition distance, the TTC calculation according to the relative speed is as shown in FIG. 7D.

If the relative speed is 100 km/h, the relative speed with the vehicle is 100 km/h (27.78 m/s) when the vehicle is traveling at the maximum speed of 100 km/h on the highway and the other vehicle is stopped, In this case, TTC can be calculated by dividing the relative speed by the distance between the two vehicles. In the same way, the TTC can also be calculated from the maximum relative speed that can come out of a general road, 60 km/h (16.67 m/s).

The TTC calculated in the above manner was assigned a weight as shown in FIG. 7E by dividing the boundary according to specific conditions. The weight on the far right is the method applied to the current simulation tool, and the final value can be changed through continuous testing.

If TTC is 0s or less, there is no weight because it has already collided. Each boundary is set to 1.6 sec, 3.6 sec, and 6.0 sec, and if TTC is included in the corresponding pole, the same weight as on the right is given. The threshold value set to 1.6 sec 3.6 sec 6 sec is according to the risk occurrence rate according to the TTC.

Then, the boundary between 3.6 seconds and 6.0 seconds is as described above, when the relative speed is measured as the highway maximum speed and general road maximum speed standards of 100 km/h (27.78 m/s) and 60 km/h (16.67 m/s), the maximum The TTC, which is calculated based on the distance of 100m, was taken as the standard. When the TTC exceeds 6.0 seconds, the weight was determined to be the lowest, considering that the probability of a collision in the autonomous driving situation was low. Whenever TTC corresponds to each section, whether the object is properly recognized is evaluated by reflecting the weight.

In addition, the weights within each section are interpolated so that the weights can be differentially applied according to the degree of TTC even within the same section.

If the TTC calculation is difficult because the relative speed part of the data acquired through Sensor Fusion is not accurately calculated, the distance weight may be reflected instead for evaluation.

As described above, the verification data output unit 140 according to an embodiment of the present invention can calculate the verification data by giving weight information to at least one of the object type, the object distance, and the object speed information. Hereinafter, a calculation example of calculating the verification data by assigning weights will be described.

When the type and TTC weight of the object to be recognized is applied, the object recognition score is as follows.

Equation 1: Object recognition score = 1 * (type weight) * (TTC weight)

The object recognition score is + if object recognition is positive, and 0 if not recognized.

For example, ① If there are 3 recognition target objects (vehicle, TTC weight is assumed to be 1) in a specific frame, 2 are positive and 1 is unrecognized, the score per object is 1*2(Type: vehicle)*1(TTC) ) = 2 points, and the score of the frame is 4/6 (only the recognition score of the recognized object is reflected in the numerator, the recognition score of the unrecognized object is calculated as 0) up to, count as 1*2*1 even for unrecognized objects).

② In the example of ①, if the TTC weight is different (only the TTC of the unrecognized object is assumed to be 1.5 seconds), the score of the unrecognized object is 1*2(type: vehicle)*30(TTC weight) = 60, and the score of the frame is 4/64, which is converted to a score of 100 points, is 6.25 points. The situation in this example failed to recognize a high-risk object with a TTC of 1.6 seconds or less, but two less important objects were recognized.

③ In the example of ②, if the type of unrecognized object (TTC 1.5 seconds) is human, the score is 1*3(type: person)*30(TTC weight) = 90, and the score of the frame is 4/94, converted This gives a score of 4.25 out of 100. Even if it is a high-risk object, if the Type is human, the performance evaluation is lower.

In the case of object unrecognition, performance evaluation is possible as above, but there is a problem in that misrecognition cannot be handled by weighting the above object recognition score. Since it is a method of calculating the score for each object that reflects the weight based on the object of the reference data, and deducting the score if the object is not recognized, it is a performance measurement method when an object that does not exist in the reference data is misrecognized and output to the result data I need this extra In order to evaluate the performance of the misrecognition, a method in which the misrecognition rate is reflected in the score obtained from the above calculation may be used. The number of false recognitions compared to the total number of output results included in the result data is called the false recognition rate, and when it is reflected in the formula, it is multiplied by (1 - false recognition rate). For example, ④ there are 5 objects in a specific frame, and the result data outputs 6 recognition results, and if 4 of them are correct and 1 is unrecognized, the object recognition score calculated through weight is (1 - false recognition rate) ) by 5/6. The score for one unrecognized object will be calculated in the same way as ①②③ above, and the performance evaluation result for one misrecognition will be lower according to the result reflecting the false recognition rate. result data

If the performance evaluation result in frame unit is obtained in the same way as above, it is measured by the average value of these frame scores in the image unit. Finally, the performance of the cognitive algorithm can be measured on a scale of 100 points.

8A and 8B show additional examples for evaluating the performance of the frame unit as a result of reflecting the object recognition score in the object recognition, unrecognized, and misrecognized information through comparison of the aforementioned reference data and result data.

For reference, the correct answer sheet refers to the reference data and the result sheet refers to the result data.

8A is an example of an answer sheet. It is assumed that there are 3 objects for each correct answer sheet, and the case where the type of each object is different (vehicle or person) and the case where the TTC is different were set. Accordingly, the perfect score is calculated differently for each result sheet.

FIG. 8B shows how the evaluation score differs according to the result recognized by the algorithm for the above correct answer. & represents the IoU value and must be 0.5 or higher to be correct.

The result sheet 1 outputs 3 results for 3 objects, and since they are all authentic, the cognitive score is a perfect score.

Result sheet 2 outputs 3 results for 3 objects, and since the recognition of object A among them is unrecognized, the recognition score is calculated by subtracting the corresponding object score.

Result Sheet 3 outputs 4 results for 3 objects, and 3 objects are correct, but one misrecognition is included. At this time, the result of calculating object A and IoU is X and W. Since the IoU of X is higher than W, X correctly recognizes object A, and W outputs the result for an object that does not exist, so W is misrecognized. . If two results on one object output IoU, the result with higher IoU is considered to have recognized the object. Therefore, the cognitive score is calculated by multiplying the score calculated by correct recognition by 3/4, which is (1-misrecognition rate).

Result sheet 4 outputs 5 results for 3 objects, 2 objects are correct recognition, object A is unrecognized, and 2 misrecognitions are included. IoU with object A is calculated for X and W, but A is unrecognized because both of them did not exceed 0.5, the threshold of positive recognition. At this time, X with a higher IoU value is the result of not recognizing A, and W is misrecognized. For object B, IoU is calculated for both results of Y and V and exceeds 0.5, but Y, which shows a higher value, becomes a positive recognition result for B, and V is misrecognized because IoU with other objects is not calculated. Therefore, the recognition score is calculated by multiplying the score obtained by subtracting the unrecognized object score of A from the full score in the case of all correct recognition, which is (1-misrecognition rate), 3/5.

The embodiments described above may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.

The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software.

Examples of the computer-readable recording medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM and DVD, and a magneto-optical medium such as a floppy disk. optical media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules for carrying out the processing according to the present invention, and vice versa.

Aspects herein may take the form of a computer program product embodied on one or more computer readable media embodied in hardware entirely, software entirely (including firmware, resident software, microcode, etc.) or computer readable program code. .

Features, structures, effects, etc. described in the above embodiments are included in one embodiment of the present invention, and are not necessarily limited to one embodiment. Furthermore, the features, structures, effects, etc. illustrated in each embodiment can be combined or modified for other embodiments by those of ordinary skill in the art to which the embodiments belong. Accordingly, the contents related to such combinations and modifications should be interpreted as being included in the scope of the present invention.

In addition, although the embodiment has been described above, it is merely an example and does not limit the present invention, and those of ordinary skill in the art to which the present invention pertains are exemplified above in a range that does not depart from the essential characteristics of the present embodiment. It can be seen that various modifications and applications that have not been made are possible. For example, each component specifically shown in the embodiment can be implemented by modification. And differences related to such modifications and applications should be construed as being included in the scope of the present invention defined in the appended claims.

Claims (18)

acquiring original data related to autonomous driving of the vehicle;
obtaining reference data processed so that the original data conforms to a predetermined data format;
generating result data by executing the original data through a predetermined test program; and
Including; outputting verification data comparing the reference data and the result data;
Autonomous driving situation awareness program performance test method. The method of claim 1,
The verification data output step includes:
It is performed based on at least one of whether the object included in the reference data is recognized, the type of the object, the distance of the object, and the speed information of the object,
Autonomous driving situation awareness program performance test method. 3. The method of claim 2,
The verification data is
It is calculated by giving weight information to at least one of the type of the object, the distance of the object, and the speed information of the object,
Autonomous driving situation awareness program performance test method. 4. The method of claim 3,
The verification data output is performed in units of frames including the object,
Measuring the performance of the predetermined test program with an average value of the verification data output in units of the frame;
Autonomous driving situation awareness program performance test method. 3. The method of claim 2,
Whether the object included in the reference data is recognized is a positive recognition when the object included in the reference data and the object included in the result data match, and object information not included in the reference data is included in the result data erroneous recognition, and the object information included in the reference data corresponds to any one of unrecognized not included in the result data,
Autonomous driving situation awareness program performance test method. 6. The method of claim 5,
The type of the object included in the reference data and the type of the object included in the result data match, and each other among the coordinate information of the object included in the reference data and the coordinate information of the object included in the result data If the matching area is greater than or equal to a predetermined threshold, it is determined as the correct recognition,
Autonomous driving situation awareness program performance test method. The method of claim 1,
The reference data is generated through the process of labeling the object and adding an annotation to suit the predetermined data format,
Autonomous driving situation awareness program performance test method. 8. The method of claim 7,
The predetermined data format is JSON (JavaScript Object Notation) format,
Autonomous driving situation awareness program performance test method. The method of claim 1,
The original data is generated by fusion of image data generated by photographing the surroundings of the vehicle and sensing data generated from a sensor of the vehicle,
Autonomous driving situation awareness program performance test method. an original data acquisition unit configured to acquire original data related to the autonomous driving situation of the vehicle;
a reference data acquisition unit for acquiring reference data processed so that the original data conforms to a predetermined data format;
a result data generation unit generating result data by executing the original data through a predetermined test program; and
A verification result output unit for outputting verification data obtained by comparing the reference data and the result data;
Autonomous driving situation awareness program performance test device. 11. The method of claim 10,
The verification result output unit,
outputting the verification data based on at least one of whether the object included in the reference data is recognized, the type of the object, the distance of the object, and the speed information of the object,
Autonomous driving situation awareness program performance test device. 12. The method of claim 11,
The verification result output unit,
calculating the verification data by giving weight information to at least one of the type of the object, the distance of the object, and the speed information of the object,
Autonomous driving situation awareness program performance test device. 13. The method of claim 12,
The verification result output unit outputs the verification data in units of frames including the object, and measures the performance of the predetermined test program with an average value of the verification data output in units of frames;
Autonomous driving situation awareness program performance test device. 12. The method of claim 11,
Whether the object included in the reference data is recognized is a positive recognition when the object included in the reference data and the object included in the result data match, and object information not included in the reference data is included in the result data erroneous recognition, and the object information included in the reference data corresponds to any one of unrecognized not included in the result data,
Autonomous driving situation awareness program performance test device. 15. The method of claim 14,
The verification result output unit,
The type of the object included in the reference data matches the type of the object included in the result data, and the object coordinate information included in the reference data and the object coordinate information included in the result data match each other. If the matching area is greater than or equal to a predetermined threshold, it is determined by the correct recognition,
Autonomous driving situation awareness program performance test device. 11. The method of claim 10,
The reference data is generated through the process of labeling the object and adding an annotation to suit the predetermined data format,
Autonomous driving situation awareness program performance test device. 17. The method of claim 16,
The predetermined data format is JSON (JavaScript Object Notation) format,
Autonomous driving situation awareness program performance test device. 11. The method of claim 10,
The original data is generated by fusion of image data generated by photographing the surroundings of the vehicle and sensing data generated from a sensor of the vehicle,
Autonomous driving situation awareness program performance test device.

Images