DE102021205094A1 - Quality check of training data for image classifiers - Google Patents

Abstract

Method (100) for quality testing of training data (2) for an image classifier (1) with the steps: • an image classifier (1) is provided (110) that has been trained on the target, the training images (2a) on the respective associated Map target classification scores (2b);• a relevance evaluation function (5) is provided (120) that specifies classification scores (4) determined for an input image (3) and for this purpose by the image classifier (1), in which dimensions (3#) portions of the input image (3) are relevant for the decision of the image classifier (1);• a set of calibration images (6) is provided (130) for which it is known that the image classifier ( 1) it has mapped to correct classification scores;• with the relevance evaluation function (5), calibration evaluations (6#) are determined (140) to the extent to which parts of the calibration images (6) are relevant for the decision of the image classifier (1 ) in relation to this calibre r images (6) are relevant;• the relevance rating function (5) is used to determine (150) a test rating (2a#) to determine the extent to which parts of at least one training image (2a) are relevant for the decision of the image classifier (1) in are relevant with respect to this training image (2a);• in response to the test score (2a#) being consistent with the calibration scores (6#) (160), it is determined (170) that the target classification scores (2b) of the training image (2a) are correct;• otherwise it is determined (180) that the target classification scores (2b) of the training image (2a) are incorrect.

Description

The present invention relates to quality testing of training data for image classifiers to determine whether this training data contains correct target classification scores as “labels”.

State of the art

Driving a vehicle on public roads is a complex task that requires continuous detection of the vehicle's surroundings and a prompt reaction to the appearance of objects, such as traffic signs, and the behavior of other road users. A prerequisite for a correct reaction is that objects and other road users are classified correctly, for example a stop sign is always recognized as a stop sign.

In order to drive a vehicle at least partially automatically, it is necessary to mechanically reproduce the classification of objects that humans have learned long before the first driving lesson. the WO 2018/184 963 A2 discloses a method with which objects in the environment of a vehicle can be recognized with artificial neural networks, ANN.

Training data are required for training such ANNs. This training data contains, for example, training images and those values of the output variables onto which the ANN should ideally map these training images as labels. A training image can, for example, show at least part of the surroundings of the vehicle. The associated labels can, for example, indicate the objects that are present in the scenery shown in the image.

The success of the training is fundamentally dependent on the labels in the training data being correct with regard to the respective application. the DE 10 2019 204 139 A1 discloses a method for training an ANN that is particularly robust against errors in the labels.

Disclosure of Invention

A method for quality testing of training data for an image classifier was developed as part of the invention. This image classifier is configured to map input images to classification scores related to one or more classes. The training data contains training images and associated target classification scores.

For this method, an image classifier is first provided, which has been trained for the purpose of mapping the training images to the respectively associated target classification scores. The behavior of the image classifier is thus determined by the assignments of target classification scores to training images contained in the training data.

A relevance evaluation function is provided which, for an input image and classification scores determined for this by the image classifier, specifies the extent to which parts of the input image are relevant for the decision of the image classifier. Such a relevance evaluation function can, for example, output a "heat map" in which each pixel of the input image is assigned a numerical value for its relevance. However, it is also possible, for example, to simply output a binary mask which indicates which pixels of the input image are relevant and which are not.

For example, “randomized input sampling” or “deep Taylor decomposition” can be used as a relevance evaluation function, but any other function can also be used.

The motivation for the relevance evaluation function is that the classification scores alone do not allow a reliable assessment of whether the training data used are labeled with the correct target classification scores in the context of the respective application. A fundamental tendency of image classifiers to “memorize” (“overfitting”) the available training data is independent of whether the labels are correct or not. Therefore, it is not the classification scores that are analyzed here, but the output of the relevance rating function.

• • von vornherein mit Soll-Klassifikations-Scores, die bekanntermaßen korrekt sind, als Labels versehen waren und
• • von dem Bildklassifikator auf Klassifikations-Scores abgebildet wurden, die diesen Soll-Klassifikations-Scores zumindest näherungsweise entsprechen.

A set of calibration images is provided which is known to have been mapped to correct classification scores by the image classifier. For example, this could be images

• • a priori with target classification scores known to be correct, labeled and
• • were mapped by the image classifier to classification scores that at least approximately correspond to these target classification scores.

• • für die vorab kein Soll-Klassifikations-Score als Label bekannt war,
• • für die jedoch eine nachträgliche Prüfung der vom Bildklassifikator ermittelten Klassifikations-Scores ergibt, dass diese Klassifikations-Scores korrekt sind.

However, images can also be used as calibration images, for example,

• • for which no target classification score was previously known as a label,
• • for which, however, a subsequent check of the classification determined by the image classifier Scores shows that these classification scores are correct.

With the relevance evaluation function, calibration evaluations are determined to the extent to which parts of the calibration images are relevant for the decision of the image classifier in relation to these calibration images.

On the other hand, the relevance evaluation function is used to determine a test evaluation as to the extent to which parts of at least one training image are relevant for the decision of the image classifier in relation to this training image.

It is checked to what extent the test rating is consistent with the calibration ratings. In response to the test score being consistent with the calibration scores, the target classification scores of the training image are determined to be correct. If, on the other hand, the test evaluation is not consistent with the calibration evaluations, it is determined that the target classification scores of the training image are incorrect.

If there is a sufficient number of calibration images (i.e. not just a single calibration image that the image classifier may have randomly classified correctly), the image classifier can only assign correct classification scores to all these calibration images if it bases its decision on the image areas that are meaningful in relation to the classification. The image classifier must then have learned which image areas are relevant in relation to certain classes. This learning result is reflected in a certain characteristic distribution of the image parts considered relevant.

Conversely, this means that the image classifier will also assign the correct classification scores to other images with a high degree of probability if it is based on image parts from this distribution. For the training images, these classification scores correspond at least approximately to the target classification scores, since this is what the training was aimed at.

The case can now arise that the classification scores for a training image are determined using completely different image areas than is to be expected on the basis of the said distribution. This suggests that the image classifier is aiming for target classification scores that were more or less “forced” on it by training with an incorrect label, without being correct on the merits. Accordingly, in response to the test score not being consistent with the calibration scores, it may be determined that the target classification scores of the training image are incorrect.

The examination of the extent to which the test evaluation is consistent with the calibration evaluations can include, for example, evaluating at least one characteristic and/or at least one variable from at least one calibration evaluation and determining a corresponding characteristic from the test evaluation, or to determine a variable corresponding thereto. It can then be checked to what extent this corresponding feature or this corresponding variable is consistent with the feature or the variable for the calibration evaluation. In this way, the comparison of the test ratings with the calibration ratings is abstracted to a certain extent. In particular, this makes it easier to compare a specific test assessment with a large number of calibration assessments that belong to calibration images that have very different content.

• • ein quantitativer Anteil des Kalibrier-Bildes, dem die Relevanzbewertungsfunktion eine Relevanz oberhalb eines vorgegebenen Schwellwerts zuordnet; und/oder
• • eine Norm der Variabilität der mit der Relevanzbewertungsfunktion ermittelten Relevanz innerhalb des Kalibrier-Bildes; und/oder
• • Koordinaten von Orten im Kalibrier-Bild, denen die Relevanzbewertungsfunktion eine Relevanz oberhalb eines vorgegebenen Schwellwerts zuordnet, sowie beliebige von diesen

In particular, for example, are suitable as features or variables

• • a quantitative portion of the calibration image to which the relevance evaluation function assigns a relevance above a predetermined threshold; and or
• • a norm of the variability of the relevance determined with the relevance evaluation function within the calibration image; and or
• • Coordinates of locations in the calibration image to which the relevance evaluation function assigns a relevance above a predetermined threshold, and any of these

Coordinates derived quantities, such as a centroid of these places. The features or sizes determined for the training image and corresponding thereto will tend to deviate greatly from these reference values if the association between the training images and the classification scores determined from them does not satisfy the same systematics in terms of content that the calibration images correspond to the classifications determined from them -Scores binds.

Advantageously, a distribution and/or summarizing statistics are determined for the feature, or for the size, for all calibration images. It is then checked whether the corresponding feature or the corresponding variable for the training image is consistent with this distribution or with this summarizing statistic. In this way, the qualitatively very different roles and origins of calibration assessments on the one hand and test assessments on the other hand can be mapped particularly well: each A single test score is to be compared with a large number of calibration scores, because the calibration scores define the distribution against which the test score must be measured.

For this purpose, for example, the summarizing statistics can contain a mean value and a standard deviation. The corresponding characteristic or quantity can then be considered as being in line with this statistic if it lies within a range, measured in standard deviations, around the mean.

The behavior of the relevance evaluation function always depends to some extent on the image classifier used. This influence can be reduced by providing a plurality of image classifiers and, based on each of these image classifiers, determining the extent to which the target classification scores of the training image are correct. The findings obtained from the various image classifiers can then be combined into a final result, for example by a majority vote.

The multiple image classifiers particularly advantageously include multiple modifications of one and the same image classifier. Then the findings obtained in each case relate qualitatively to the same model on which the original image classifier was based. At the same time, the end result obtained by bringing these findings together is somewhat abstracted from small modifications of this model.

For this purpose, for example, starting from an image classifier, at least one modification of this image classifier can be generated by a random selection of neurons or other processing units of this image classifier being deactivated (“drop out”).

In a further advantageous embodiment, a set of counter-calibration images is provided for which it is known that the image classifier has mapped them to incorrect classification scores. Counter-calibration scores are determined for these counter-calibration images. The extent to which the test rating is consistent with the counter-calibration ratings is checked. Ideally, the distribution defined by the counter-calibration images should be complementary to the distribution defined by the calibration images. That is, a test score should be consistent with either the distribution of calibration scores or the distribution of counter-calibrate scores, but not both distributions at the same time.

This requirement can be used as feedback for optimizing the features or quantities that are used to compare the test ratings with the calibration ratings.

For example, at least one feature or at least one variable can be selected by optimization such that a maximum contrast is obtained when comparing the test evaluation with calibration evaluations on the one hand and with counter-calibration evaluations on the other hand. The test rating is then only consistent with either the calibration ratings or the counter-calibration ratings.

Alternatively or also in combination with this, at least one feature or at least one variable can be selected using an evaluation related to an additional classifier, which is trained using the calibration images and the counter-calibration images.

If it is determined during the process that the target classification scores of one or more training images are incorrect, this can be used as feedback to improve the training of the image classifier and ultimately increase its classification accuracy.

For this purpose, target classification scores that were determined to be incorrect can be changed in at least one training image. In particular, if target classification scores for a number of training images are determined to be incorrect, a new configuration of target classification scores for these training images can be determined using a suitable strategy. This strategy can include, for example, that the new target classification scores determined for the training images in question are not mutually contradictory.

This procedure is a bit analogous to the situation where a candidate in a multiple-choice exam first collects the questions he is unsure about answering and then again at the end of the exam in the time available in a way revised so that the new answers are at least internally consistent. On average, this increases the number of points achieved.

However, it is also possible, for example, to remove at least one training image from the training data whose target classification scores were determined to be incorrect. For the classification accuracy ultimately achieved by the image classifier, omitting the training image can be a lesser evil than doing so hold this training image with target classification scores that are inconsistent with the target classification scores of other training images.

This procedure is a bit analogous to the fact that when a large number of images of an object from different perspectives are merged into a three-dimensional model of this object by means of photogrammetry, poor-quality images should be discarded. A view from each perspective is then lost, but this is less detrimental to fusion than retaining the poor image, which could create inconsistencies with other images.

If the training data has been changed by adjusting target classification scores and/or by omitting training images, the image classifier can be retrained with this. It can then be assessed, for example, whether a higher proportion of the target classification scores is found to be correct when the method described here is carried out again. For example, it can also be assessed whether, after training with the changed training data, the classification accuracy in a measurement using test or validation data is better than in the originally trained state.

In a further advantageous embodiment, images that were recorded with at least one sensor are fed to the retrained image classifier as input images. These input images are mapped by the image classifier to one or more classification scores. A control signal is formed from the classification scores. A vehicle, a monitoring system and/or a system for the quality control of mass-produced products is controlled with this control signal.

The method is particularly advantageous in connection with quality control or other applications where training images with high target classification scores in relation to these classes are scarce for at least some of the classes. For example, in the series production of products, it is the normal case that a tested example of the product is free of defects, and the exceptional case that this example has a defect or damage. If an image classifier is now trained to classify images of the products into the classes "OK" and "not OK = NOK" in binary form, there is much more illustrative material for the class "OK" than for the class "NOK". This encourages “overfitting” during training, where an image classifier that is over-parameterized compared to the amount of training data available more or less “memorizes” the training data rather than drawing a generalized lesson from that training data. The consideration of the decision-relevant image parts creates an additional source of information for the purpose of assessing the extent to which target classification scores are correct with regard to the present application, which is not affected by a possible "overfitting" of the image classifier.

Furthermore, especially in quality control, there is also prior knowledge about which image areas can be relevant to the decision at all. The finished product is typically always in the same position in the image. It is therefore clear which image areas actually belong to the product and which image areas (e.g. a background) contain information that has nothing to do with the product itself. The image classifier may well make a right decision for the wrong reasons. However, if he is not considering the areas of the image he should be considering, he probably has not learned what he was supposed to learn, but found a "short cut" and "overfitted" on it.

In particular, the method can be fully or partially computer-implemented. The invention therefore also relates to a computer program with machine-readable instructions which, when executed on one or more computers, cause the computer or computers to carry out the method described. In this sense, control devices for vehicles and embedded systems for technical devices that are also able to execute machine-readable instructions are also to be regarded as computers.

The invention also relates to a machine-readable data carrier and/or a download product with the computer program. A downloadable product is a digital product that can be transmitted over a data network, i.e. can be downloaded by a user of the data network and that can be offered for sale in an online shop for immediate download, for example.

Furthermore, a computer can be equipped with the computer program, with the machine-readable data carrier or with the downloadable product.

Further measures improving the invention are presented in more detail below together with the description of the preferred exemplary embodiments of the invention with the aid of figures.

exemplary embodiments

• 1 Ausführungsbeispiel des Verfahrens 100;
• 2 Beispielhafte Unterscheidung zwischen Trainingsbildern 2a und 2a' mit korrekten (k) bzw. inkorrekten (i) Soll-Klassifikations-Scores 2b.

It shows:

• 1 embodiment of the method 100;
• 2 Exemplary distinction between training images 2a and 2a' with correct (k) or incorrect (i) target classification scores 2b.

1 is a schematic flowchart of an embodiment of the method 100 for quality testing of training data 2 for an image classifier 1.

In step 110, an image classifier 1 trained on this training data 2 is provided for the training data 2, which contains training images 2a and associated target classification scores 2b. According to block 111, a plurality of image classifiers 1 can also be provided, which, according to block 111a, can in particular be modifications 1′ of an image classifier 1, for example. Such modifications 1' can be generated by deactivating (dropping out) a random selection of neurons or other processing units from the original image classifier 1. This has the advantage that these modifications 1 ′, starting from a state of the original image classifier 1 trained with the training data 2 , can also be treated as having been trained with the training data 2 .

In step 120, the relevance evaluation function 5 is provided, which maps an input image 3 of the image classifier 1 to dimensions 3# for a decision relevance of image parts, for example in the form of a “heat map”.

In step 130, a set of calibration images 6 is provided, which the image classifier 1 is known to have correctly classified. In accordance with block 131, counter-calibration images 8 that are known to have been incorrectly classified by the image classifier 1 can also be provided.

In step 140, the relevance evaluation function 5 is used to determine calibration evaluations 6#, which indicate the decision relevance of portions of the calibration images 6. Likewise, according to block 141, counter-calibration evaluations 8# can be determined, which indicate the decision-making relevance of parts of the counter-calibration images 8.

In step 150, the relevance evaluation function 5 is used to determine a test evaluation 2a# as to the extent to which parts of at least one training image 2a are relevant for the decision of the image classifier 1 in relation to this training image 2a.

In step 160 it is checked whether the test score 2a# is consistent with the calibration scores 6#. If this is the case (truth value 1), it is determined in step 170 that the target classification scores 2b of the training image 2a are correct. Otherwise (truth value 0) it is determined in step 180 that the target classification scores 2b of the training image 2a are incorrect.

According to block 161, at least one feature and/or at least one variable can be evaluated from at least one calibration evaluation 6#. According to block 162, a feature corresponding thereto or a variable corresponding thereto can then be evaluated from the test evaluation 2a#. According to block 163, it can then be checked to what extent this corresponding feature or this corresponding variable is consistent with the feature or the variable for the calibration evaluation 6#.

In this case, in particular according to block 163a, a distribution and/or summarizing statistics for the feature or for the size can be determined for all calibration images 6#. According to block 163b, it can then be checked whether the corresponding feature or the corresponding variable is consistent with this distribution or with this summarizing statistic.

According to block 164, the cross-check can be performed as to how well the test score 2a# is consistent with the versus calibration scores 8#. As discussed previously, the test score 2a# should be consistent with either the distribution of calibration scores 6# or the distribution of counter-calibration scores 8# complementary thereto, but not both distributions at the same time. In particular, a feature for which the behavior with respect to the 6# calibration scores is at most contrary to the behavior with respect to the counter-calibration 8# scores can be selected from among several possible features as particularly meaningful.

According to block 165, determinations of the extent to which the target classification scores 2b of the training image 2a are correct can be combined to form a final result 7 for various image classifiers 1, such as drop-out modifications 1′ of an image classifier 1.

Insofar as target classification scores 2b are identified as incorrect, they can be changed according to block 190a. As an alternative or in combination with this, associated training images 2a can be removed from the training data 2 . The image classifier 1 can be trained again in step 200 with the training data 2 ′ changed in this way.

In step 210, the retrained image classifier 1* is assigned images containing at least one Sensor (9) were recorded as input images (3) are fed. In step 220, the input images 3 are mapped onto one or more classification scores 4 by the image classifier 1*. In step 230 a control signal 230a is formed from the classification scores 4 . In step 240, a vehicle 50, a monitoring system 60, and/or a system 70 for the quality control of mass-produced products is controlled with the control signal 230a.

2 illustrates how a training image 2a with a correct (k) target classification score 2b can be distinguished from a training image 2a′ with an incorrect (incorrect, i) target classification score 2b using the method described above.

Three calibration images 6, 6', 6" are recorded as an example, all of which show a product 10. In these calibration images 6, 6', 6" the calibration ratings are 6#, 6#' and 6 #" in the form of the respective decision-relevant parts.

Calibration image 6 shows an undamaged product 10, which as a whole also forms the decision-relevant portion 6# for the correct decision “OK” by the image classifier 1.

Calibration image 6′ shows a product 10 with a defect 11, which forms the decision-relevant portion 6#′ for the image classifier 1 to make the correct decision “not OK=NOK”.

Calibration image 6" shows a product 10 with a crack 12, the area around which forms the decision-relevant portion 6#" for the correct decision "not OK = NOK" of the image classifier 1.

Training image 2a shows a product 10 with a defect 11 that is located at a different location than in the calibration image 6'. The fact that this defect 11 is identified as a decision-relevant portion 2a# is consistent with the distribution of the calibration evaluations 6#, 6#' or 6#". It is thus established that the target classification score 2b of this training image 2a is correct (k) "not OK = NOK".

Training image 2a' shows a product 10 with a crack 12 that is similar to that in calibration image 6". The fact that an area in the image background outside the product 10 is identified as the decision-relevant portion 2a#' is not consistent with the distribution of the Calibration ratings 6#, 6#' and 6#". It is thus established that the target classification score 2b of this training image 2a′ is incorrectly (i) “OK”.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• WO 2018/184963 A2 [0003]
• DE 102019204139 A1 [0005]

Claims (15)

Method (100) for checking the quality of training data (2) for an image classifier (1), which is designed to map input images (3) onto classification scores (4) in relation to one or more classes, the training data (2 ) training images (2a) and associated target classification scores (2b), with the steps: • an image classifier (1) is provided (110) which has been trained for the purpose of mapping the training images (2a) to the respectively associated target classification scores (2b); • A relevance evaluation function (5) is provided (120) which, for an input image (3) and classification scores (4) determined for this by the image classifier (1), indicates the extent to which (3#) parts of the input image (3) are relevant to the decision of the image classifier (1); • providing (130) a set of calibration images (6) known to have been mapped by the image classifier (1) to correct classification scores; • with the relevance evaluation function (5) calibration evaluations (6#) are determined (140) to what extent proportions of the calibration images (6) for the decision of the image classifier (1) in relation to these calibration images (6) are relevant; • with the relevance evaluation function (5), a test evaluation (2a#) is determined (150) to what extent parts of at least one training image (2a) are relevant for the decision of the image classifier (1) in relation to this training image (2a). ; • in response to the test score (2a#) agreeing (160) with the calibration scores (6#), it is determined (170) that the target classification scores (2b) of the training image (2a ) are correct; • otherwise it is determined (180) that the target classification scores (2b) of the training image (2a) are incorrect. Method (100) according to claim 1 , wherein • at least one feature and/or at least one variable is evaluated (161) from at least one calibration evaluation (6#); • from the test evaluation (2a#), a feature corresponding thereto, or a variable corresponding thereto, is evaluated (162); and • it is checked (163) to what extent this corresponding feature or variable is consistent with the feature or variable for the calibration evaluation (6#). Method (100) according to claim 2 , wherein the feature, or the size, • a quantitative part of the calibration image (6#), to which the relevance evaluation function assigns a relevance above a predetermined threshold value; and/or • a norm of the variability of the relevance determined with the relevance evaluation function within the calibration image (6#); and/or • contains coordinates of locations in the calibration image (6#) to which the relevance evaluation function (5) assigns a relevance above a predetermined threshold value. Method (100) according to any one of claims 2 until 3 , where • for the feature, or for the size, a distribution and/or summary statistics is determined (163a) for all calibration images (6#), and • it is checked (163b) whether the corresponding feature , or the corresponding quantity, is consistent with this distribution, or with this summary statistic. Method (100) according to claim 4 , where • the summary statistic includes a mean and a standard deviation; and • the corresponding characteristic or quantity is considered consistent with that statistic if it falls within a range, measured in standard deviations, about the mean. Method (100) according to any one of Claims 1 until 5 , wherein • a plurality of image classifiers are provided (111); • Based on each of these image classifiers, it is determined (160) to what extent the target classification scores (2b) of the training image (2a) are correct; and • these determinations are combined (165) into a final result (7). Method (100) according to claim 6 , starting from an image classifier (1), at least one modification (1') of this image classifier (1) is generated (111a) by deactivating a random selection of neurons or other processing units of this image classifier (1). Method (100) according to any one of Claims 6 until 7 , whereby several determinations are brought together to a final result (7) by majority decision. Method (100) according to any one of Claims 1 until 8th wherein in addition • a set of counter-calibration images (8) is provided (131) which the image classifier (1) is known to have mapped to incorrect classification scores; • for these counter-calibration images (8) counter-calibration ratings (8#) are determined (141); and • It is checked (164) to what extent the test rating (2a#) is consistent with the counter-calibration ratings (8#). Method (100) according to claim 2 and 9 and optionally one of the claims 3 until 8th , wherein at least one feature, or at least one variable, • by optimization in such a way that when comparing the test rating (2a#) with calibration ratings (6#) on the one hand and with counter-calibration ratings (8#) on the other hand sets a maximum contrast, is selected, and/or • is selected using an evaluation related to an additional classifier, this additional classifier being trained on the basis of the calibration images (6#) and the counter-calibration images (8#). . Method (100) according to any one of Claims 1 until 10 , wherein • target classification scores (2b) that were determined to be incorrect are changed (190a) in at least one training image, and/or • at least one training image (2a) whose target classification scores (2b) were determined to be incorrect, is removed from the training data (2) (190b), and wherein • the image classifier (1) is trained again (200) using the training data (2') changed in this way. Method (100) according to claim 11 , wherein • the re-trained image classifier (1*) is supplied (210) with images that were recorded with at least one sensor (9) as input images (3); • the input images (3) are mapped (220) by the image classifier (1*) to one or more classification scores (4); • a control signal (230a) is formed (230) from the classification scores (4); and • a vehicle (50), a monitoring system (60), and/or a system (70) for the quality control of mass-produced products, with which the control signal (230a) is controlled (240). Computer program containing machine-readable instructions which, when executed on one or more computers, cause the computer or computers to perform a method (100) according to one of Claims 1 until 12 to execute. Machine-readable data carrier with the computer program Claim 13 . One or more computers with the computer program after Claim 13 , and/or with the machine-readable data carrier Claim 14 .

Images