WO2020140684A1 - Method and device for evaluating vehicle damage identification model - Google Patents

Abstract

A method and device for evaluating a vehicle damage identification model, the method comprising: first acquiring a first test sample, the test sample corresponding to a first vehicle damage picture and a plurality of sets of labeling data, the plurality of sets of labeling data being generated on the basis of a plurality of labeling personnel labeling the first vehicle damage picture (21); then determining the intersection and the union of the plurality of sets of labeling data, and determining a significant damage object set in the test sample according to the intersection; and determining a non-significant damage object set in the test sample according to the difference between the intersection and the union (22). In addition, the first vehicle damage picture is inputted into a vehicle damage identification model to obtain a predicted damage object set composed of damage objects predicted by the vehicle damage identification model for the first test sample (23). Therefore, a test result of the vehicle damage identification model for the first test sample may be determined according to the relationship between the predicted damage object set and the significant damage object set and non-significant damage object set (24).

Description

Method and device for evaluating vehicle damage identification model Technical field

One or more embodiments of this specification relate to the field of machine learning, and in particular, to a method and device for evaluating a vehicle damage recognition model.

Background technique

In the traditional auto insurance claims process, the insurance company needs to send professional damage survey personnel to the accident site to conduct on-site damage assessment, give the vehicle's maintenance plan and compensation amount, and take photos of the scene, and the photos of the fixed losses are kept for background verification Personnel damage and price. Due to the need to manually investigate and assess damage, insurance companies need to invest a lot of labor costs, and professional knowledge training costs. From the experience of ordinary users, the claims process is due to waiting for human surveyors to take pictures on the spot, the loss adjuster fixes the damage at the repair site, and the loss loss personnel check the damage in the background. The claim period is as long as 1-3 days, and the user has a long wait time , The experience is poor.

Aiming at the pain points in the industry with huge labor cost in traditional auto insurance claims, some intelligent image loss determination schemes are proposed. Intelligent image loss determination is based on the computer vision image recognition technology in the field of artificial intelligence. Based on the on-site loss pictures taken by ordinary users, it automatically recognizes the lost parts and the degree of loss reflected in the pictures, and automatically gives a maintenance plan. Finally, by calling the price library corresponding to the insurance company's maintenance program, the compensation amount of the case is obtained. This solution eliminates the need to manually check and assess damage and loss, greatly reducing the cost of insurance companies and improving the average user's experience in auto insurance claims.

In order to achieve intelligent image loss determination, a variety of vehicle damage recognition models using machine learning algorithms have been proposed. However, how to evaluate the recognition effect of these vehicle damage recognition models has become a problem to be solved.

Therefore, it is hoped that there can be an improved plan to evaluate the vehicle damage identification model more effectively.

Summary of the invention

One or more embodiments of this specification describe a method and device for evaluating a vehicle damage identification model, by distinguishing between markedly damaged objects and non-significantly damaged objects in the test sample, to avoid evaluation interference caused by the ambiguity of non-significantly damaged objects .

According to the first aspect, a method for evaluating a vehicle damage identification model is provided, including:

Obtaining a first test sample, the first test sample corresponding to a first car damage picture and multiple sets of annotation data, the multiple sets of annotation data are based on at least a plurality of annotated persons to mark the damaged objects in the first car damage picture produce;

Determine the intersection and union of the multiple sets of labeled data, determine the set of significant damage objects in the first test sample according to the intersection; and determine the first according to the difference between the intersection and the union A collection of non-significantly damaged objects in the test sample;

Input the first car damage picture into a pre-trained car damage recognition model to obtain a predicted damage object set composed of the damage objects predicted by the car damage recognition model for the first test sample;

The test result of the first test sample by the vehicle damage identification model is determined according to the relationship between the set of predicted damage objects and the set of significant damage objects and the set of non-significant damage objects.

In one embodiment, the plurality of sets of labeling data include at least first labeling data and second labeling data, wherein the first labeling data is generated by labeling personnel with a first labeling ability level, and the second labeling data is composed of The tagging personnel of the second tagging capability level generate tags, and the second tagging capability level is higher than the first tagging capability level.

According to an embodiment, each set of labeling data in the plurality of sets of labeling data is generated by the labeling personnel and the verification personnel performing the verification.

In one embodiment, the test results include results that are incorrectly predicted or correctly predicted.

Further, in one embodiment, the test result of the vehicle damage identification model on the first test sample is determined in the following manner:

Determine whether the set of significant damage objects is a subset of the set of predicted damage objects;

If it is not a subset, it is determined that the test result of the vehicle damage identification model on the first test sample is a prediction error.

In another embodiment, the test result of the vehicle damage identification model on the first test sample is determined in the following manner:

For any first damage object in the predicted damage object set, if the first damage object does not belong to any of the significant damage object set and the non-significant damage object set, the vehicle damage identification model is determined The test result of the first test sample is a prediction error.

In another embodiment, the test result includes a single sample test score.

Further, in one embodiment, the single-sample test score is determined in the following manner:

Determining a first number of damage objects included in the set of predicted damage objects and belonging to the set of significant damage objects;

Determining a second number of damaged objects included in the set of predicted damaged objects and belonging to the set of non-significant damaged objects;

Based on at least the first number and the second number, a single-sample positive score of the vehicle damage identification model for the first test sample is determined.

In another embodiment, the one-sample test score is determined by:

Determining a third number of damaged objects included in the set of significant damaged objects but not included in the set of predicted damaged objects;

Determining a fourth number of damage objects included in the predicted damage object set that do not belong to the significant damage object set nor the non-significant damage object set;

According to the third number and the fourth number, a single-sample negative score of the vehicle damage identification model for the first test sample is determined.

According to a second aspect, a method for evaluating a vehicle damage identification model is provided, including:

Obtain a test sample set, which includes multiple test samples, the multiple test samples correspond to multiple car damage pictures, and each car damage picture corresponds to multiple sets of annotation data, the multiple sets of annotation data are based on at least multiple annotation personnel pairs The damage object in the car damage picture is generated by marking;

For each test sample, the method of the first aspect is executed to determine each test result of the vehicle damage identification model for each test sample:

According to each test result, the evaluation result of the vehicle damage identification model for the test sample set is determined.

In one embodiment, each test result includes a result of correct prediction or prediction error; correspondingly, determining the evaluation result may include determining the proportion of each test result that is predicted to be correct, as the evaluation result.

In another embodiment, each test result includes a single sample test score; correspondingly, determining the evaluation result of the vehicle damage identification model for the test sample set includes, according to the single sample test score included in each test result , Determine the total sample score for the test sample set, and use it as the evaluation result.

According to a third aspect, an apparatus for evaluating a vehicle damage identification model is provided, including:

A sample acquisition unit configured to acquire a first test sample, the first test sample corresponding to a first car damage picture and multiple sets of annotation data, the multiple sets of annotation data are based at least on the first car damage picture The damaged objects are marked and generated;

An annotation set determination unit, configured to determine an intersection and union of the plurality of sets of annotation data, determine a significant damage object set in the first test sample according to the intersection; and according to the intersection and the union To determine the set of non-significantly damaged objects in the first test sample;

A prediction set determining unit configured to input the first vehicle damage picture into a pre-trained vehicle damage recognition model to obtain a predicted damage object set composed of the damage objects predicted by the vehicle damage recognition model for the first test sample;

The test result determination unit is configured to determine the test result of the first test sample by the vehicle damage identification model according to the relationship between the set of predicted damage objects and the set of significant damage objects and the set of non-significant damage objects.

According to a fourth aspect, an apparatus for evaluating a vehicle damage identification model is provided, including:

The sample set acquisition unit is configured to acquire a test sample set, which includes multiple test samples, the multiple test samples correspond to multiple car damage pictures, and each car damage picture corresponds to multiple sets of labeled data, the multiple sets of labeled data It is generated based on at least a number of annotators marking the damaged objects in the car damage picture;

The test result obtaining unit is configured to determine the test results of the vehicle damage identification model for each test sample using the device of the third aspect for each test sample:

The evaluation result determination unit is configured to determine the evaluation result of the vehicle damage identification model for the test sample set according to the respective test results.

According to a fifth aspect, there is provided a computer-readable storage medium on which a computer program is stored, and when the computer program is executed in a computer, the computer is caused to perform the methods of the first and second aspects.

According to a sixth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the first aspect and the first Two ways.

Through the method and device provided in the embodiments of the present specification, multiple sets of labeling data generated by multiple labeling personnel labeling the same test picture are used to determine the significant damage object and the non-significant damage object in the test picture. When comparing the predicted damage objects and labeled data output by the model in order to evaluate the vehicle damage identification model, distinguish between significant damage objects and non-significant damage objects, using different comparison methods, based on the predicted damage objects and the significant damage objects respectively The relationship of non-significant damage objects determines the test results of the vehicle damage identification model on the test samples. Due to the distinction between significant damage objects and non-significant damage objects, when evaluating the vehicle damage identification model, these two kinds of marked objects are given different attention and different measurement standards, so as to avoid the ambiguity and non-significant damage objects. The controversy and noise in the evaluation caused by the uncertainty will find a better optimization goal for the model optimization.

The damage identification of the car body has a certain degree of professionalism. Some reflective stains on the car body are easily confused with the damage, which can easily cause mislabeling or missed labeling by the labeling personnel. For example, the car lamp is slightly damaged. Due to the complex structure of the car lamp, the glass material is easy to reflect and other factors It requires a person with strong visual ability and experience to make a more accurate label. This method can effectively distinguish between clear errors and ambiguous situations, help test the model's ability to optimize clear errors, and can increase the weight of clear errors in the model to reduce the weight of ambiguous situations, so that the model can better distinguish clear errors Sex. This method is also suitable for the evaluation and optimization of other scenes that require professional knowledge and high visual ability, such as the labeling of medical diseases.

BRIEF DESCRIPTION

In order to more clearly explain the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. Those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

FIG. 1 shows an example of a car damage picture according to an embodiment;

2 shows a flowchart of a method for evaluating a vehicle damage identification model according to an embodiment;

Figure 3a shows the first set of annotation data in an example;

Figure 3b shows the second set of annotation data in an example;

Figure 3c shows the third set of annotation data in an example;

Figure 4 shows the relationship between the set of significant damage objects, the set of non-significant damage objects, and the set of predicted damage objects;

FIG. 5 shows a method of evaluating a vehicle damage identification model according to an embodiment;

6 shows a schematic block diagram of an evaluation device according to an embodiment;

7 shows a schematic block diagram of an evaluation device according to an embodiment.

detailed description

The solution provided in this specification will be described below in conjunction with the drawings.

First, the overall concept of the embodiment scheme is described. The overall concept stems from the applicant's analysis and research on human visual ability.

After observation and research, the applicant believes that human visual ability can be divided into normal ordinary visual ability and super visual ability. Normal ordinary visual ability can accurately recognize salient objects, while super visual ability will observe and recognize non-significant objects based on normal ordinary visual ability.

In the scene of vehicle damage identification in order to determine the damage to the vehicle, people with normal ordinary visual abilities can notice and observe generally significant damage, while non-significant damage requires super visual ability to notice and can be compared with Distinguish between extremely disturbing reflective texture stains, etc.

FIG. 1 shows an example of a car damage picture according to an embodiment. In this example, people with normal ordinary visual abilities can observe that there are two scratches on the bumper and fender, so the damage is significant. However, a little opaque scratch damage on the lower left side of the left front headlight of the vehicle, and a little scratch damage on the steel ring are not so significant. Only people with strong visual abilities can observe the corners.

As known to those skilled in the art, machine learning models need to learn based on a large amount of labeled data in order to achieve good prediction performance. As a type of machine learning model, the car damage recognition model requires a large amount of car damage labeling data as training samples for training, in order to perform car damage recognition on unknown pictures. These labeling data are generated by the labeling personnel labeling the car damage pictures similar to Figure 1.

Generally speaking, the annotators who annotate car damage pictures are professionals with long-term visual training. Some of these people will have a strong visual observation ability, can observe and mark some non-significant damage, especially after special training, you can professionally mark lights or small parts. For the labelers as a whole, the labels with significant damage are easier, and there is basically no omission in labeling, but the labels with non-significant damage will vary from person to person and have some deviations. Based on the training samples generated by such labeling, the car damage recognition model is trained. The model will cover the significant damage very well, and it will also have a certain super visual ability to observe the perception ability, so the recognition ability of the model is better than the significant damage Large intersections, partly with non-significant damage.

When evaluating the recognition accuracy and ability of the vehicle damage identification model, one solution is to compare the damage identified by the vehicle damage identification model with the fixed loss order manually determined by the insurance company. However, this approach often brings great controversy. One case is that the damage order contains a damage that is not recalled by the vehicle damage identification model, and the damage is not significant. At this time, the recall ability of the model will be disputed because there are some damages due to visual ability The difference cannot be determined. Another situation is that the model has identified some non-significant damage that is not included in the fixed loss order. At this time, it is easy to dispute the accuracy of the model, because there may be damage here, and the fixed loss order is not reflected. Therefore, when directly comparing the consistency of the model identification result and the result of the fixed loss order, disputes will arise.

Based on the above observation, analysis and research, the applicant proposes a scheme to distinguish between markedly damaged objects and non-significantly damaged objects in the car damage pictures. When evaluating the recognition performance of the vehicle damage identification model, different judgment standards are made for the significantly damaged objects and non-significantly damaged objects, so as to evaluate the damage recognition model more effectively and avoid unnecessary disputes.

The following first describes how to evaluate the recognition effect of the trained vehicle damage recognition model for a single test sample, that is, a single vehicle damage picture.

FIG. 2 shows a flowchart of a method for evaluating a vehicle damage identification model according to an embodiment. The method can be performed by any device, device, computing platform, or computing cluster with computing and processing capabilities. As shown in FIG. 2, the method includes at least the following steps: Step 21: Obtain a first test sample, the first test sample corresponds to a first car damage image and multiple sets of annotation data, the multiple sets of annotation data are based on at least multiple annotations The person marks the damaged objects in the first car damage picture to generate; Step 22, determines the intersection and union of the multiple sets of labeled data, and determines the significant damaged objects in the first test sample according to the intersection Set; and according to the difference between the intersection and the union, determine the set of non-significant damage objects in the first test sample; Step 23, input the first car damage picture into a pre-trained car damage recognition A model to obtain a predicted damage object set composed of the damage objects predicted by the vehicle damage identification model for the first test sample; step 24, according to the predicted damage object set and the significant damage object set and the non-significant damage The relationship of the set of objects determines the test result of the first test sample by the vehicle damage identification model. The following specifically describes the execution method and execution process of the above steps.

First, in step 21, a single test sample is obtained. For simplicity of description, this test sample is referred to as the first test sample. The first test sample corresponds to a single car damage picture, hereinafter referred to as the first car damage picture, and multiple sets of annotation data for the car damage picture, wherein the multiple sets of annotation data are based on at least the first car damage The damaged objects in the picture are generated by annotation.

Specifically, for an original car damage picture, it can be distributed to multiple annotators to mark the vehicle damage objects respectively, thereby generating multiple sets of annotated data. The multiple sets of annotation data together with the car damage pictures constitute a test sample.

In one embodiment, in order to integrate the labeling results of the labelers with different visual abilities, the car damage pictures are distributed to the people with different labeling abilities. In actual operation, the labeling ability of the labeling personnel will be graded according to the strength of the visual observation ability of the labeling personnel, and the factors such as the length of training time. Generally, a higher level of annotation ability means a stronger visual observation ability. In the case where the labeling personnel are graded in advance, the car damage pictures can be distributed to the labeling personnel with different labeling ability levels, thereby generating multiple sets of labeling data.

In a specific example, the car damage image can be distributed to the labelers with the first labeling ability level for labeling, thereby generating the first labeling data; the same car damage image can be distributed to the labeling personnel with the second labeling ability level Annotation is performed to generate second annotation data, where the second annotation capability level is higher than the first annotation capability level. As such, the multiple sets of labeling data include at least the first labeling data and the second labeling data generated by labelers of different labeling ability levels (first and second ability levels), respectively.

More specifically, in one embodiment, the above-mentioned first annotation ability level corresponds to ordinary visual ability, and the second annotation ability level corresponds to super visual ability. Thus, the multiple sets of annotation data include data generated by an annotation person with ordinary visual abilities tagging a car damage picture, and data generated by an annotation person with super visual abilities tagging the picture. In other embodiments, the labeling ability levels can also be divided more carefully, for example, into three levels or even four levels, and the above-mentioned multiple sets of labeling data come from labelers of different labeling ability levels.

In one embodiment, for each set of labeling data, the labeling staff generates the labeling data, and then the verification staff performs verification and confirmation. In this way, to avoid the impact of individual labeling personnel's incorrect labeling on the entire test.

Specifically, in one example, the car damage picture shown in FIG. 1 is used as the original image, and it is distributed to three tagging personnel to tag the damaged object to obtain 3 sets of tagging data. Figures 3a-3c respectively show these three sets of annotation data. It can be seen that the three sets of labeling data are not the same due to the differences in the visual capabilities of different labelers. Figure 3a contains two marked damage objects, denoted as A1 and A2 respectively; Figure 3b contains marked damage objects A1, A2 and newly added A3; and Figure 3c contains five marked damage objects , A1, A2, A4, A5 and A6, respectively. These three sets of labeled data, together with the original car damage image shown in Figure 1, can be used as a test sample.

Next, in step 22, the intersection and union of multiple sets of labeled data are determined, and the set of significant damage objects in the first test sample is determined according to the intersection; according to the difference between the intersection and the union, the non-information in the first test sample is determined. Significant damage to the object collection.

As mentioned earlier, multiple sets of labeling data come from multiple different labelers, and the damaged objects marked are not the same. Therefore, in step 22, based on the similarities and differences between the multiple sets of labeled data, it is determined which damaged objects are significant damaged objects and which are non-significant damaged objects.

Specifically, if an injury is observed and marked by all annotators, the injury should be identified as a significant injury. This corresponds to the intersection of multiple sets of labeled data. Therefore, the damaged object in the intersection of the above-mentioned multiple sets of labeled data is determined as the significant damaged object. Correspondingly, the objects other than the significantly damaged objects are determined as non-significantly damaged objects, that is, the difference between the union of multiple sets of labeled data and the above intersection is determined as non-significantly damaged objects.

This is described in conjunction with the examples of Figures 3a-3c. The damaged object set included in FIG. 3a is {A1, A2}, the damaged object set included in FIG. 3b is {A1, A2, A3}, and the damaged object set included in FIG. 3c is {A1, A2, A4, A5, A6}. For these three sets of labeled data, the intersection is {A1, A2}, and the union is {A1, A2, A3, A4, A5, A6}. Therefore, in one embodiment, it is determined according to the intersection that for the car damage picture of FIG. 1, the set of significant damage objects is {A1, A2}, and the set of non-significant damage objects is {A3, A4, A5, A6}.

In order to evaluate the vehicle damage identification model, then, in step 23, the above-mentioned vehicle damage image is input into a pre-trained vehicle damage identification model to obtain a predicted damage object set composed of the damage object predicted by the vehicle damage identification model for the test sample .

The above vehicle damage recognition model can be implemented based on various machine learning algorithms and using various neural network structures, which is not limited herein. The vehicle damage identification model is trained based on the training sample set. The training sample set contains a large number of training samples, and each training sample corresponds to a car damage picture and the labeled data for the car damage picture. Using the labeled data as the sample label, you can train the vehicle damage recognition model. After the training is completed, the picture to be recognized is input into the vehicle damage recognition model, and the model will output the recognition result for the picture, also known as the prediction result, which contains each damage object that is recognized or predicted.

For the first car damage picture as the test sample in the previous step, input it into the already trained car damage recognition model, and the model will output each damage object predicted for the first car damage picture. The set of predicted damage objects is called a predicted damage object set.

Then, in step 24, according to the relationship between the predicted damage object set obtained in step 23 and the significant damage object set and the non-significant damage object set obtained in step 22, the test result of the vehicle damage identification model on the first test sample is determined.

FIG. 4 shows the relationship between the set of significant damage objects, the set of non-significant damage objects, and the set of predicted damage objects. As shown in FIG. 4, a circle 101 represents a set of significant damage objects, corresponding to the intersection of multiple sets of labeled data. The circle 102 corresponds to the union of the aforementioned multiple sets of labeled data, and represents all the damaged objects marked. Therefore, the circle 102 completely contains the circle 101, and the circle 101 completely falls into the circle 102. The part between the circle 102 and the circle 101, that is, the difference between the union and intersection of the multiple sets of labeled data, represents the non-significant damage object. This part of the damage object corresponds to the ambiguous damage, or only those with super visual ability The damage can be identified.

According to the concept of the embodiment of the present specification, in an ideal case, the predicted damage object set 103' should be located between the circle 101 and the circle 102, as shown by the dotted line in FIG. 4. That is, the predicted damage object set 103' completely includes the significant damage object set 101, but does not exceed the set 102 of all damage objects, and includes some non-significant damage objects. At this time, it can be considered that the result of the predicted damage object set 103' is correct, and the test result of the vehicle damage identification model for the current test sample is good.

However, in reality, there may be a case where the predicted damage object set 103 deviates from the ideal situation, as shown by the bold solid line in the figure. At this time, it is considered that the result of predicting the damaged object set 103 is wrong, and the test result of the vehicle damage identification model for the current test sample is not ideal.

The following describes a specific execution manner of determining the test result of the vehicle damage identification model in step 24 based on the relationship shown in FIG. 4.

In one embodiment, the test result of the vehicle damage identification model for a single sample is determined, that is, whether the prediction of the vehicle damage identification model for the single sample is correct. Correspondingly, the test results include the conclusion that the prediction is wrong or the prediction is correct.

As mentioned earlier, in the ideal case, the set of predicted damage objects should completely contain the set of significant damage objects, which means that the vehicle damage identification model is required to predict all significant damage objects. Therefore, in one embodiment, an error type that defines prediction errors is called a first type of error. This first type of error corresponds to the situation where the vehicle damage identification model fails to identify or predict a significant damage object somewhere. Since the significant damage object is the damage object that will be observed by people with ordinary visual abilities, in the case that the recognition result of the vehicle damage recognition model shows the above first type of error, the vehicle damage recognition model can be tested against the current first test sample The result is determined to be a prediction error.

To this end, in a specific implementation manner, it can be determined in step 24 whether the set of significant damage objects is a subset of the set of predicted damage objects, that is, whether the set of predicted damage objects completely contains the set of significant damage objects; if the above determination is negative, Then it is determined that the test result of the vehicle damage identification model is a prediction error.

In the examples of FIGS. 3a-3c, as mentioned above, the set of significant damage objects is {A1, A2}. If the predicted damage object set output by the vehicle damage identification model fails to fully contain {A1, A2}, it is considered that the test result for the test sample is a prediction error.

In addition, as mentioned above, it is also desirable to predict that the set of damaged objects does not exceed the set of all damaged objects. Therefore, in one embodiment, another type of error that defines a prediction error is called a second type of error. This second type of error corresponds to the situation where the damage identified by the vehicle damage identification model exceeds the range of all the damage objects marked. Since the range of all damaged objects (corresponding to the circle 102 in FIG. 4) includes significant damaged objects and non-significant damaged objects, or includes all possible damaged objects, the second result of the recognition of the vehicle damage recognition model appears above In the case of a class error, the test result of the vehicle damage identification model for the current first test sample may be determined as a prediction error.

For this reason, in the specific execution mode of step 24, for any damage object in the predicted damage object set, hereinafter referred to as the first damage object, if the first damage object does not belong to the significant damage object set and the non-significant damage object set Any one of them determines that the test result of the vehicle damage identification model on the current test sample is a prediction error.

In the examples of FIGS. 3a-3c, as mentioned above, the set of significant damage objects is {A1, A2}, the set of non-significant damage objects is {A3, A4, A5, A6}, and the union is {A1, A2 , A3, A4, A5, A6}. If the predicted damage object set output by the vehicle damage identification model contains a certain damage B1, it does not belong to any one of the significant damage object set and the non-significant damage object set, or does not belong to the union {A1, A2, A3, A4, A5, A6}, it is considered that the vehicle damage identification model recalled the wrong object, and the test result for the test sample is a prediction error.

Contrary to the above prediction error, the test result with the correct prediction can be determined correspondingly. In one embodiment, if the set of damaged objects is predicted to be free of the first type of error or the second type of error, the test result of the vehicle damage identification model for the current test sample is determined to be correct.

In the above, according to the relationship between the predicted damage object set, the significant damage object set and the non-significant damage object set, the test result of the vehicle damage identification model for the current test sample is divided into prediction errors or prediction predictions.

In contrast, according to another embodiment, a test score is used to characterize the test result of the vehicle damage identification model for a single test sample. The test score may include a positive score of a single sample. The positive score is used to indicate the predicted correct rate, correct number, correctness, etc. for the single sample; the test score may also include a negative score of a single sample, a negative The score indicates, for example, the prediction error rate for the single sample, the number of errors, and so on.

In one embodiment, the number of damage objects included in the set of predicted damage objects that belong to the set of significant damage objects can be determined, which is called the first number, and is denoted as N1; it is also determined that the number of damage objects included in the set of predicted damage objects belongs to The number of damaged objects in the non-significantly damaged object set is called the second number and is denoted as N2. Then, based on at least the first number N1 and the second number N2, a single-sample positive score of the vehicle damage identification model for the current test sample is determined.

For example, in one example, the sum of the first number N1 and the second number N2 is determined as the number N of correctly predicted damage objects, and the number N is divided by the total number M of elements in the predicted damage object set, and N/M is taken as the above The forward score is used to describe the proportion of the damaged objects that are predicted to be correct.

For another example, in another example, the ratio R1 of the first number N1 to the number S1 of elements in the significant damage set is determined, and the ratio R2 of the second number N2 to the number S2 of elements in the non-significant damage set is determined. For R1 and R2 On average, the single sample correct rate R is obtained as the above positive score, namely:

R=(R1+R2)/2=(N1/S1+N2/S2)/2

It is still described in conjunction with the examples of FIGS. 3a-3c. Assuming that the predicted damage object set P is {A1, A3, A4, A5, B1, B2}, then the elements in P that belong to the significant damage object set are A1, N1=1, according to the previous description, the number of elements in the significant damage object set S1=2; the elements in P belonging to the non-significant damage object set are A3, A4 and A5, so N2=3, and the number of elements in the non-significant damage object set S2=4. Thus, the forward score can be determined as R=(1/2+3/4)/2=0.625.

Further, in yet another example, when summing the first ratio R1 and the second ratio R2, different weights are given to the two, that is, R=w1*R1+w2*R2. Since the importance of correctly predicting significantly damaged objects is much higher than that of non-significantly damaged objects, w1 can be made greater than w2, and even w1 can be much larger than w2. For example, let w1=0.8 and w2=0.2. In one embodiment, for the above example, the positive score can be determined as R=0.8*0.5+0.2*0.75=0.55.

In one embodiment, considering the negative effects of the first type error and the second type error described above, a negative score of the vehicle damage identification model for a single test sample is determined. The negative score NS can be determined in various ways.

In one embodiment, a certain score is assigned to the first type of error and the second type of error, respectively, and the negative score NS is determined as the sum of the above scores, namely:

NS＝a*T1+b*T2

Among them, a is the score of the first type of error, in the case of the first type of error, T1=1, otherwise T1=0; b is the score of the second type of error, in the case of the second type of error , T2=1, otherwise T2=0.

In another embodiment, the number of damaged objects whose prediction is wrong is also determined. Specifically, in an example, the number of damaged objects included in the set of significant damage objects but not included in the set of predicted damage objects is called a third number, which is denoted as N3, and the number of damage objects in the set of predicted damage objects is determined. The number of included damaged objects that are not in the set of significant damage objects nor in the set of non-significant damage objects is called the fourth number and is denoted as N4. The above third number and fourth number correspond to the above-mentioned first type error and second type error, respectively, and can be considered as the number of prediction errors. Therefore, the negative score NS of the vehicle damage identification model for the current test sample can be determined according to the third number and the fourth number.

Specifically, in an example, the negative score can be determined as:

NS=w3*N3+w4*N4,

Among them, w3 and w4 are weighting factors.

The description will still be made in conjunction with the example where the predicted damage object set P is {A1, A3, A4, A5, B1, B2}. For this example, it can be determined that the damage object A2 included in the significant damage object set is not included in the predicted damage object set, therefore, N3=1; in addition, the damage object B1, B2 is included in the predicted damage object set. The object does not belong to the set of significant damage objects nor the set of non-significant damage objects, therefore, N4=2.

In one example, assuming a negative score NS=w3*N3+w4*N4, where w3=0.6 and w4=0.4, then the negative score for the sample is NS=1*0.6+2*0.4=1.4.

In the above manner, based on the consideration of the first type error/second type error described above, a negative score is given to the test result.

According to an embodiment, the above positive score and negative score can also be synthesized to obtain a single sample total score for the current test sample. For example, for the positive score PS and the negative score NS, the single-sample total score S of the vehicle damage identification model for the current test sample can be determined as: S=PS-NS.

In this way, according to the relationship between the predicted damage object set, the significant damage object set and the non-significant damage object set, the single-sample test result of the vehicle damage identification model against the current test sample is evaluated in various ways.

On the basis of evaluating the test results of a single sample, a similar method can be used to evaluate the effect of the vehicle damage identification model on the test sample set composed of multiple test samples.

FIG. 5 illustrates a method for evaluating a vehicle damage identification model according to an embodiment. The method is used to evaluate the overall recognition effect of the vehicle damage identification model on a plurality of test samples in a test sample set. As shown in FIG. 5, first in step 51, a test sample set is obtained, which includes multiple test samples. Each test sample corresponds to a car damage picture, and multiple sets of annotation data, the multiple sets of annotation data are generated based on at least multiple annotators marking the damaged objects in the car damage picture. For the description of multiple sets of labeling data and multiple labeling personnel, please refer to the description of the foregoing step 21, which will not be repeated here.

Next, in step 52, for each test sample, the method shown in FIG. 2 is executed, so as to determine each test result of the vehicle damage identification model for each test sample. In other words, for each test sample in the test sample set, take it as the first test sample in the method of FIG. 2 and execute the method shown in FIG. 2, so that the test result for the test sample can be obtained. By performing the above method on each test sample, each test result for each test sample in the test sample set can be obtained.

Then, in step 53, the evaluation result of the vehicle damage identification model for the test sample set is determined according to the above test results.

In one embodiment, as described above, each test result may include a result of correct prediction or incorrect prediction. In such a case, determining the evaluation result of the vehicle damage identification model for the test sample set may include determining the ratio of each test result that is predicted correctly, and using this ratio as the evaluation result. Correspondingly, when the ratio is higher than a certain threshold, for example 80%, it can be considered that the recognition effect of the vehicle damage recognition model meets the requirements and the test passes.

In another embodiment, each test result may include a single sample test score. In such a case, determining the evaluation result of the vehicle damage identification model for the test sample set may include determining the total sample score for the test sample set based on the single-sample test score included in each test result, and using the score as the evaluation result.

More specifically, you can calculate the sum, average, etc. of the test scores of each single sample to determine the total sample score. According to the specific meaning of the test score and the calculation method of the total sample score, a corresponding score threshold can be set. When the total sample score meets the score threshold, the recognition effect of the vehicle damage recognition model is considered to meet the requirements. For example, in a specific example, the single-sample test score is a negative score, and the total sample score is the average of each single-sample score. In such a case, if the total sample score is less than the preset score threshold, the test is considered passed.

In the above manner, the effect of the vehicle damage identification model on the entire test sample set is effectively evaluated.

Reviewing the evaluation process of the vehicle damage identification model, in this process, through multiple sets of annotation data generated by multiple annotators marking the same test picture, the significant and non-significant damage objects in the test picture are determined. When comparing the predicted damage objects and labeled data output by the model in order to evaluate the vehicle damage identification model, distinguish between significant damage objects and non-significant damage objects, using different comparison methods, based on the predicted damage objects and the significant damage objects respectively The relationship of non-significant damage objects determines the test results of the vehicle damage identification model on the test samples. Due to the distinction between significant damage objects and non-significant damage objects, when evaluating the vehicle damage identification model, different attention and different measurement standards are given to the two labeled objects, thereby avoiding the ambiguity and non-significant damage objects. Controversy and noise caused by uncertainty.

According to another embodiment, a device for evaluating a vehicle damage identification model is also provided. Fig. 6 shows a schematic block diagram of an evaluation device according to an embodiment. The evaluation device can be deployed in any device, device, computing platform, or computing cluster with computing and processing capabilities, and is used to evaluate the recognition results of a single test sample by the vehicle damage identification model. As shown in FIG. 6, the evaluation device 600 includes:

The sample obtaining unit 61 is configured to obtain a first test sample, the first test sample corresponding to a first car damage picture and multiple sets of annotation data, the multiple sets of annotation data are based at least on the first car damage picture The damaged objects in the label are generated;

An annotation set determination unit 62 is configured to determine the intersection and union of the multiple sets of annotation data, determine the set of significant damage objects in the first test sample according to the intersection; and according to the intersection and the union The difference between them, determine the set of non-significantly damaged objects in the first test sample;

The prediction set determining unit 63 is configured to input the first vehicle damage picture into a pre-trained vehicle damage recognition model to obtain a predicted damage object set composed of the damage objects predicted by the vehicle damage recognition model for the first test sample;

The test result determination unit 64 is configured to determine the test result of the first test sample by the vehicle damage identification model based on the relationship between the set of predicted damage objects and the set of significant damage objects and the set of non-significant damage objects .

In one embodiment, the plurality of sets of labeling data include at least first labeling data and second labeling data, wherein the first labeling data is generated by labeling personnel with a first labeling ability level, and the second labeling data is composed of The tagging personnel of the second tagging capability level generate tags, and the second tagging capability level is higher than the first tagging capability level.

According to an embodiment, each set of labeling data in the plurality of sets of labeling data is generated by the labeling personnel and the verification personnel performing the verification.

In one embodiment, the test results include results that are incorrectly predicted or correctly predicted.

Further, in an embodiment, the test result determination unit 64 is configured to:

Determine whether the set of significant damage objects is a subset of the set of predicted damage objects;

If it is not a subset, it is determined that the test result of the vehicle damage identification model on the first test sample is a prediction error.

In another embodiment, the test result determination unit 64 is configured to:

For any first damage object in the set of predicted damage objects, if the first damage object does not belong to any of the set of significant damage objects and the set of non-significant damage objects, the vehicle damage identification model is determined The test result of the first test sample is a prediction error.

In another embodiment, the test result includes a single sample test score.

Further, in an embodiment, the test result determination unit 64 is configured to:

Determine a first number of damage objects included in the set of predicted damage objects and belonging to the set of significant damage objects;

Determining a second number of damaged objects included in the set of predicted damaged objects and belonging to the set of non-significant damaged objects;

Based on at least the first number and the second number, a single-sample positive score of the vehicle damage identification model for the first test sample is determined.

In another embodiment, the test result determination unit 64 is configured to:

Determining a third number of damaged objects included in the set of significant damaged objects but not included in the set of predicted damaged objects;

Determining a fourth number of damage objects included in the predicted damage object set that do not belong to the significant damage object set nor the non-significant damage object set;

According to the third number and the fourth number, a single-sample negative score of the vehicle damage identification model for the first test sample is determined.

According to an embodiment of yet another aspect, another device for evaluating a vehicle damage identification model is also provided. 7 shows a schematic block diagram of an evaluation device according to an embodiment. The evaluation device 700 is used to evaluate the recognition result of the test sample set composed of multiple test samples by the vehicle damage identification model. As shown in FIG. 7, the evaluation device 700 includes:

The sample set acquisition unit 71 is configured to acquire a test sample set, which includes multiple test samples, the multiple test samples correspond to multiple car damage pictures, and each car damage picture corresponds to multiple sets of annotation data, the multiple sets of annotations The data is generated based on at least a number of annotators marking the damaged objects in the car damage picture

The test result acquisition unit 72 is configured to use the device 600 of FIG. 6 for each test sample to determine each test result of the vehicle damage identification model for each test sample:

The evaluation result determination unit 73 is configured to determine the evaluation result of the vehicle damage identification model for the test sample set according to the respective test results.

In an embodiment, wherein each of the test results includes a result of correct prediction or prediction error; correspondingly, the evaluation result determination unit 73 is configured to determine the ratio of the correct prediction of each test result as the evaluation result.

In another embodiment, wherein each of the test results includes a single-sample test score; correspondingly, the evaluation result determination unit 73 is configured to determine, based on the single-sample test score included in the respective test results, The total sample score of the test sample set is used as the evaluation result.

According to an embodiment of another aspect, there is also provided a computer-readable storage medium on which a computer program is stored, and when the computer program is executed in a computer, the computer is caused to perform the method described in conjunction with FIGS. 2 and 5.

According to an embodiment of still another aspect, a computing device is further provided, including a memory and a processor, where executable code is stored in the memory, and when the processor executes the executable code, the implementation is combined with FIG. 2 and FIG. 5 The method.

Those skilled in the art should realize that in one or more of the above examples, the functions described in this application may be implemented by hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of this application in detail. It should be understood that the above descriptions are only specific implementations of this application and are not intended to limit the scope of this application. The scope of protection, any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of this application, shall be included in the scope of protection of this application.

Claims (25)

1. A method for evaluating a vehicle damage identification model includes:

   Obtaining a first test sample, the first test sample corresponding to a first car damage picture and multiple sets of annotation data, the multiple sets of annotation data are based on at least a plurality of annotated persons to mark the damaged objects in the first car damage picture produce;

   Determine the intersection and union of the multiple sets of labeled data, determine the set of significant damage objects in the first test sample according to the intersection; and determine the first according to the difference between the intersection and the union A collection of non-significantly damaged objects in the test sample;

   Input the first car damage picture into a pre-trained car damage recognition model to obtain a predicted damage object set composed of the damage objects predicted by the car damage recognition model for the first test sample;

   The test result of the first test sample by the vehicle damage identification model is determined according to the relationship between the set of predicted damage objects and the set of significant damage objects and the set of non-significant damage objects.
2. The method according to claim 1, wherein the plurality of sets of labeling data include at least first labeling data and second labeling data, wherein the first labeling data is generated by labeling personnel with a first labeling ability level, the second The tagging data is generated by tagging personnel with a second tagging capability level, and the second tagging capability level is higher than the first tagging capability level.
3. The method according to claim 1, wherein each set of labeling data in the plurality of sets of labeling data is generated by a labeling staff and a verification staff performing verification.
4. The method according to claim 1, wherein the test result includes a wrong prediction or a correct prediction.
5. The method according to claim 4, wherein determining the test result of the vehicle damage identification model on the first test sample includes:

   Determine whether the set of significant damage objects is a subset of the set of predicted damage objects;

   If it is not a subset, it is determined that the test result of the vehicle damage identification model on the first test sample is a prediction error.
6. The method according to claim 4 or 5, wherein determining the test result of the vehicle damage identification model on the first test sample includes:

   For any first damage object in the set of predicted damage objects, if the first damage object does not belong to any of the set of significant damage objects and the set of non-significant damage objects, the vehicle damage identification model is determined The test result of the first test sample is a prediction error.
7. The method of claim 1, wherein the test result includes a single sample test score.
8. The method according to claim 7, wherein determining the test result of the vehicle damage identification model on the first test sample includes:

   Determine a first number of damage objects included in the set of predicted damage objects and belonging to the set of significant damage objects;

   Determining a second number of damaged objects included in the set of predicted damaged objects and belonging to the set of non-significant damaged objects;

   Based on at least the first number and the second number, a single-sample positive score of the vehicle damage identification model for the first test sample is determined.
9. The method according to claim 7 or 8, wherein determining the test result of the vehicle damage identification model on the first test sample includes:

   Determining a third number of damaged objects included in the set of significant damaged objects but not included in the set of predicted damaged objects;

   Determining a fourth number of damage objects included in the predicted damage object set that do not belong to the significant damage object set nor the non-significant damage object set;

   According to the third number and the fourth number, a single-sample negative score of the vehicle damage identification model for the first test sample is determined.
10. A method for evaluating a vehicle damage identification model includes:

    Obtain a test sample set, which includes multiple test samples, the multiple test samples correspond to multiple car damage pictures, and each car damage picture corresponds to multiple sets of annotation data, the multiple sets of annotation data are based on at least multiple annotation personnel pairs The damage object in the car damage picture is generated by marking;

    For each test sample, the method of claim 1 is executed to determine each test result of the vehicle damage identification model for each test sample:

    According to each test result, the evaluation result of the vehicle damage identification model for the test sample set is determined.
11. The method according to claim 10, wherein the respective test results include prediction correct or incorrect prediction results;

    The determining the evaluation result of the vehicle damage identification model for the test sample set includes:

    The ratio of correct predictions in each test result is determined as the evaluation result.
12. The method according to claim 10, wherein the respective test results include a single sample test score;

    The determining the evaluation result of the vehicle damage identification model for the test sample set includes:

    The total sample score for the test sample set is determined according to the test score of the single sample included in the respective test results, and used as the evaluation result.
13. A device for evaluating a vehicle damage identification model includes:

    A sample acquisition unit configured to acquire a first test sample, the first test sample corresponding to a first car damage picture and multiple sets of annotation data, the multiple sets of annotation data are based at least on the first car damage picture The damaged objects are marked and generated;

    An annotation set determination unit, configured to determine an intersection and union of the plurality of sets of annotation data, determine a significant damage object set in the first test sample according to the intersection; and according to the intersection and the union To determine the set of non-significantly damaged objects in the first test sample;

    A prediction set determining unit configured to input the first vehicle damage picture into a pre-trained vehicle damage recognition model to obtain a predicted damage object set composed of the damage objects predicted by the vehicle damage recognition model for the first test sample;

    The test result determination unit is configured to determine the test result of the first test sample by the vehicle damage identification model according to the relationship between the set of predicted damage objects and the set of significant damage objects and the set of non-significant damage objects.
14. The apparatus according to claim 13, wherein the plurality of sets of labeling data include at least first labeling data and second labeling data, wherein the first labeling data is generated by labelers with a first labeling ability level, and the second The tagging data is generated by tagging personnel with a second tagging capability level, and the second tagging capability level is higher than the first tagging capability level.
15. The apparatus according to claim 13, wherein each set of labeling data in the plurality of sets of labeling data is generated by a labeling staff and a verification staff performing verification.
16. The apparatus according to claim 13, wherein the test result includes a wrong prediction or a correct prediction.
17. The apparatus according to claim 16, wherein the test result determination unit is configured to:

    Determine whether the set of significant damage objects is a subset of the set of predicted damage objects;

    If it is not a subset, it is determined that the test result of the vehicle damage identification model on the first test sample is a prediction error.
18. The apparatus according to claim 16 or 17, wherein the test result determination unit is configured to:

    For any first damage object in the set of predicted damage objects, if the first damage object does not belong to any of the set of significant damage objects and the set of non-significant damage objects, the vehicle damage identification model is determined The test result of the first test sample is a prediction error.
19. The apparatus of claim 13, wherein the test result includes a single-sample test score.
20. The apparatus according to claim 19, wherein the test result determination unit is configured to:

    Determine a first number of damage objects included in the set of predicted damage objects and belonging to the set of significant damage objects;

    Determining a second number of damaged objects included in the set of predicted damaged objects and belonging to the set of non-significant damaged objects;

    Based on at least the first number and the second number, a single-sample positive score of the vehicle damage identification model for the first test sample is determined.
21. The apparatus according to claim 19 or 20, wherein the test result determination unit is configured to:

    Determining a third number of damaged objects included in the set of significant damaged objects but not included in the set of predicted damaged objects;

    Determining a fourth number of damage objects included in the predicted damage object set that do not belong to the significant damage object set nor the non-significant damage object set;

    According to the third number and the fourth number, a single-sample negative score of the vehicle damage identification model for the first test sample is determined.
22. A device for evaluating a vehicle damage identification model includes:

    The sample set acquisition unit is configured to acquire a test sample set, which includes multiple test samples, the multiple test samples correspond to multiple car damage pictures, and each car damage picture corresponds to multiple sets of labeled data, the multiple sets of labeled data It is generated based on at least a number of annotators marking the damaged objects in the car damage picture;

    The test result acquisition unit is configured to determine, for each test sample, using the device of claim 13 each test result of the vehicle damage identification model for each test sample:

    The evaluation result determination unit is configured to determine the evaluation result of the vehicle damage identification model for the test sample set according to the respective test results.
23. The apparatus according to claim 22, wherein the respective test results include prediction correct or incorrect prediction results;

    The evaluation result determination unit is configured to determine a ratio of correct prediction in each test result as the evaluation result.
24. The apparatus according to claim 22, wherein the respective test results include a single sample test score;

    The evaluation result determination unit is configured to determine the total sample score for the test sample set based on the single-sample test score included in each test result, and use it as the evaluation result.
25. A computing device, including a memory and a processor, wherein an executable code is stored in the memory, and when the processor executes the executable code, the processor according to any one of claims 1-12 is implemented method.

Images