DE102021205271A1 - Quality check of training data for classification models for semantic segmentation of images - Google Patents

Abstract

Method (100) for quality testing of training data (2) for a classification model (1), with the steps: • a classification model (1) is provided (110) that has been trained on the target, the training images (2a) on the respective associated target segmentation maps (2b);• a relevance evaluation function (5) is provided (120);• a set of calibration images (6) is provided (130);• calibration evaluations are made with the relevance evaluation function (5). (6#) determined (140);• with the relevance evaluation function (5), a test evaluation (2a#) is determined (150) to the extent to which parts of a training image (2a) are relevant for the decisions of the classification model (1). dividing the pixels of the training image (2a) specified by the selection mask (2c) into classes (4);• in response to the test score (2a#) being consistent with the calibration scores (6#) ( 160), it is stated (170) that s the portion of the target segmentation map (2b) indicated by the selection mask (2c) is correct for the training image (2a);• otherwise it is determined (180) that the portion of the target segmentation map (2b) indicated by the selection mask (2c) for the training image (2a) is incorrect.

Description

The present invention relates to the quality check of training data for classification models to determine whether this training data contains correct target segmentation maps for the respective training images as “labels”.

State of the art

Driving a vehicle on public roads is a complex task that requires continuous detection of the vehicle's surroundings and a prompt reaction to the appearance of objects, such as traffic signs, and the behavior of other road users. A prerequisite for a correct reaction is that objects and other road users are classified correctly, for example a stop sign is always recognized as a stop sign.

In order to drive a vehicle at least partially automatically, it is necessary to mechanically reproduce the classification of objects that humans have learned long before the first driving lesson. the WO 2018/184 963 A2 discloses a method with which objects in the environment of a vehicle can be recognized with artificial neural networks, ANN. In particular, semantic segmentation maps can be created with ANNs, which divide the pixels of an input image into classes, which in turn represent, for example, types of objects. Such ANNs are then trained with training data that contain training images and associated target segmentation maps with labels for each pixel of the training images.

The success of the training is fundamentally dependent on the labels in the training data being correct with regard to the respective application. the DE 10 2019 204 139 A1 discloses a method for training an ANN that is particularly robust against errors in the labels.

Disclosure of Invention

A method for quality testing of training data for a classification model was developed as part of the invention. This classification model is designed to assign a class to each individual pixel of an input image and thus create a semantic segmentation map of the input image. The training data includes training images and associated target segmentation maps.

For this method, a classification model is first provided that has been trained with the aim of mapping the training images onto the respectively associated target segmentation maps. The behavior of the classification model is thus determined by the assignments of target segmentation maps to training images contained in the training data.

A relevance rating function is provided. For an input image and a binary selection mask of pixels of this input image, this relevance evaluation function indicates the extent to which parts of the input image are relevant for the classification model's decisions to classify the pixels of the input image specified by the selection mask into classes .

The motivation for the relevance evaluation function is that the classification scores alone do not allow a reliable assessment of whether the training data used are labeled with the correct target classification scores in the context of the respective application. A fundamental tendency of image classifiers to “memorize” (“overfitting”) the available training data is independent of whether the labels are correct or not. Therefore, it is not the classification scores that are analyzed here, but the output of the relevance rating function.

For the classification, for example, of a coherent group of pixels that represent a specific object into a class that represents the type of object, it is often not just these pixels themselves that are relevant. Rather, the classification of an object often results from its context in the input image. For example, it is not possible to determine with certainty whether this person is a pedestrian, a cyclist or an e-scooter driver based only on the pixels that belong to a person. Distinguishing between a pedestrian and a scooter rider is particularly difficult because in both cases the person is in a standing position. However, the distinction is important for assessing a traffic situation, since an e-scooter driver travels much faster than a pedestrian and has to use the cycle path or, where there is no cycle path, the lane. The person can only be identified as the e-scooter driver based on other pixels that belong to the e-scooter. The same applies to a cyclist, who can only be distinguished from a person who is sitting for other reasons by the pixels belonging to the bicycle.

It is therefore to be expected that if a person is correctly recognized as an e-scooter driver or cyclist, the relevance evaluation function will mark the e-scooter or bicycle as relevant to the decision. On the other hand, the classification of an e-scooter driver or cyclist is incorrectly recognized th person does not depend on pixels belonging to a e-scooter or bicycle.

When it comes to quality control of products, too, the classification of certain pixels in the input image often depends on the context. For example, certain areas of the surface of a component can be coated with a functional layer. An image of the component then shows both areas of the surface that are coated with the functional layer and areas in which the base material of the component is visible. The finding that certain pixels reveal an uncoated area does not in itself allow a conclusive statement as to whether this is a regular uncoated area or a defect in the coating. But if, for example, the uncoated area is an “island” that is completely surrounded by a coated area, there is a high probability that there is a defect where the coating has flaked off or is locally missing for other reasons.

Thus, in the case of an uncoated area correctly identified as a defect, the relevance evaluation function can mark the coating material that surrounds this defect as relevant to the decision, for example.

In particular, the relevance evaluation function can be designed, for example, to indicate which pixels of the input image must be taken into account in order to correctly classify all pixels of the input image indicated by the selection mask.

• • von vornherein mit Soll-Segmentierungskarten versehen waren, die individuellen Pixeln ein bekanntermaßen korrektes Klassen-Label zuordnen und
• • von dem Klassifikationsmodell auf Segmentierungskarten abgebildet wurden oder werden, die diesen Soll-Segmentierungskarten zumindest näherungsweise entsprechen.

A set of calibration images is provided for the method, for which it is known that the classification model assigns or has assigned the correct classes to the pixels of these calibration images. For example, this could be images

• • were provided from the outset with target segmentation maps that assign a class label that is known to be correct to individual pixels, and
• • were or are mapped by the classification model to segmentation maps that at least approximately correspond to these target segmentation maps.

• • für die vorab keine Soll-Segmentierungskarte bekannt war,
• • für die jedoch eine nachträgliche Prüfung der vom Klassifikationsmodell ermittelten Segmentierungskarte ergibt, dass diese Segmentierungskarte korrekt ist.

However, images can also be used as calibration images, for example,

• • for which no target segmentation map was previously known,
• • for which, however, a subsequent check of the segmentation map determined by the classification model shows that this segmentation map is correct.

The number of calibration images can be significantly smaller than the total number of training images. For this, an effort can be invested in the guarantee that the pixels of each calibration image are each labeled with the correct classes, which is not possible for the total amount of training images. Thus, if the task is to check the quality of a large amount of training images, the effort required can be focused on acquiring and labeling a small amount of calibration images.

With the relevance evaluation function, on the one hand, calibration evaluations are determined for the calibration images and specified binary selection masks of pixels of these calibration images, to what extent parts of the calibration images are relevant for the decisions of the classification model, the pixels of the calibration indicated by the selection mask -divide each image into classes. A number of calibration evaluations can thus be generated from a calibration image, in that different binary selection masks are used to ask for the explanations for class divisions of different sets of pixels. For example, explanations for class divisions of pixels that belong to different classes can be asked for one after the other. If the calibration image contains several instances of an object of a certain class, explanations for the classification of these individual instances can be asked for one after the other.

On the other hand, based on a binary selection mask of pixels of a training image, the relevance evaluation function is used to determine a test evaluation of the extent to which parts of the training image are relevant for the decisions of the classification model to divide the pixels of the training image indicated by the selection mask into classes. For example, analogous to the calibration images, explanations can be asked for why a specific group of pixels is classified as a specific type of object.

It is checked to what extent the test rating is consistent with the calibration ratings. In response to the test score being consistent with the calibration scores, it is determined that the portion of the target segmentation map indicated by the selection mask is correct for the training image. If, on the other hand, the test evaluation is not consistent with the calibration evaluations, it is determined that the portion of the target segmentation map specified by the selection mask for the training image is incorrect.

If a sufficient number of calibration images are available and a sufficient number of binary selection masks are examined, the classification model can realistically classify each pixel selected according to the mask correctly in all cases only if its decision relies on meaningful image areas. As discussed previously, these image areas may also include the surrounding context of the pixels to be binned. The classification model must therefore have learned which image areas are relevant for the classification of which pixel groups into classes. This learning experience can include, for example, an e-scooter or bicycle turning a neighboring person into an e-scooter rider or cyclist, or surrounding coating material turning an uncoated area of a surface into a defect. The learning experience is reflected in a certain characteristic distribution of the parts of the picture that are considered relevant.

Conversely, this means that the classification model will also classify selected pixel groups from other images correctly with a high degree of probability if it is based on image parts from this distribution in each case. For the training images, these class divisions are given by the target segmentation maps, because the training was aimed at mapping the training images at least approximately onto these target segmentation maps.

The case can now arise that areas of a training image queried with the selection mask are divided into the classes according to the target segmentation map on the basis of completely different image areas than is to be expected on the basis of said distribution. This indicates that the classification model makes a classification that is more or less "forced" on it by way of "overfitting" through training with incorrect labels in the target segmentation map, without being correct in the matter. Accordingly, in response to the fact that the test rating does not agree with the calibration ratings, it can be determined that the portion of the target segmentation map indicated by the selection mask for the training image is incorrect.

Figuratively speaking, when training with incorrect labels, the classification model can reason, for example: “I think that this group of pixels belongs to a pedestrian. But if the boss says it's a cyclist, then it must be so... He's right! Here in the picture there are bicycles lying around everywhere. That convinces me. It is a cyclist.” The test evaluation for the corresponding training image thus marks a large number of areas distributed over the entire image, in which bicycles can be seen, as relevant for the decision-making for the classification of an individual person in the “cyclist” class. This does not match the distribution of the calibration ratings, which in this example all have in common that only one area immediately around the person in question can be relevant for the decision between pedestrian, cyclist and e-scooter driver.

It was recognized that the examination of whether the correct image context is taken into account for the classification of certain pixel groups into classes can be influenced to a much lesser extent by "overfitting" the classification model to certain training data than the classification itself. Especially when a classification is objectively incorrect, it cannot be based on a suitable image context, since this cannot exist (otherwise the classification would be objectively correct). In the above example, in which the pedestrian mistakenly becomes the cyclist, the classification model can "learn by heart" the classification as a cyclist, which "buzzes" him the loss function. However, this does not "conjure up" a bicycle in the immediate vicinity of the person in the training image, so that the classification model has to look for some other explanation (here: the bicycles distributed elsewhere in the image).

The examination of the extent to which the test evaluation is consistent with the calibration evaluations can include, for example, evaluating at least one characteristic and/or at least one variable from at least one calibration evaluation and determining a corresponding characteristic from the test evaluation, or to determine a variable corresponding thereto. It can then be checked to what extent this corresponding feature or this corresponding variable is consistent with the feature or the variable for the calibration evaluation. In this way, the comparison of the test ratings with the calibration ratings is abstracted to a certain extent. In particular, this makes it easier to compare a specific test assessment with a large number of calibration assessments that belong to calibration images that have very different content.

• • ein quantitativer Anteil des Kalibrier-Bildes, dem die Relevanzbewertungsfunktion eine Relevanz oberhalb eines vorgegebenen Schwellwerts zuordnet; und/oder
• • Klassen, in die das Klassifikationsmodell diejenigen Pixel des Kalibrier-Bildes einteilt, denen die Relevanzbewertungsfunktion eine Relevanz oberhalb eines vorgegebenen Schwellwerts zuordnet; und/oder
• • eine räumliche Lage von für die Einteilung in Klassen relevanten Pixeln relativ zu den durch die Auswahlmaske angegebenen Pixeln im Kalibrier-Bild.

In particular, for example, are suitable as features or variables

• • a quantitative portion of the calibration image to which the relevance evaluation function assigns a relevance above a predetermined threshold; and or
• • Classes into which the classification model divides those pixels of the calibration image to which the relevance evaluation function has relevance above a predetermined threshold; and or
• • a spatial position of pixels relevant for the division into classes relative to the pixels in the calibration image indicated by the selection mask.

For example, the number of pixels and/or other features of the calibration image to which the relevance evaluation function assigns a relevance above a predefined threshold value can be determined as a quantitative proportion.

For example, the classes into which the classification model divides the relevant pixels of the calibration image can be recorded as a frequency distribution, such as a histogram, over the classes.

A spatial position of the decision-relevant pixels can be recorded, for example, in the form of a direction and a distance between the pixels to be classified and a reference point of the group of decision-relevant pixels. The direction can, for example, be quantized according to the classic Cartesian directions north, south, east and west. The reference point can, for example, be a focal point of the group of pixels relevant to the decision.

Advantageously, a distribution and/or summarizing statistics are determined for the feature, or for the size, for all calibration images. It is then checked whether the corresponding feature or the corresponding variable for the training image is consistent with this distribution or with this summarizing statistic. In this way, the qualitatively very different roles and origins of calibration assessments on the one hand and test assessments on the other hand can be mapped particularly well: Each individual test assessment can be compared with a large number of calibration assessments, because the calibration assessments define the distribution , against which the test evaluation must be measured.

For this purpose, for example, the summarizing statistics can contain a mean value and a standard deviation. The corresponding characteristic or quantity can then be considered as being in line with this statistic if it lies within a range, measured in standard deviations, around the mean.

The behavior of the relevance evaluation function always depends to some extent on the classification model used. This influence can be reduced by providing a plurality of classification models and, starting from each of these classification models, determining the extent to which a portion of the target segmentation map of the training image is correct. The findings obtained from the different classification models can then be combined to form a final result, for example by a majority decision.

The multiple classification models particularly advantageously contain multiple modifications of one and the same classification model. Then the findings obtained in each case relate qualitatively to the same classification model. At the same time, the end result obtained by bringing these findings together is somewhat abstracted from small modifications of this model.

For this purpose, starting from a classification model, for example, at least one modification of this classification model can be generated by a random selection of neurons or other processing units of this classification model being deactivated (“drop out”).

Just like the original classification model, the modifications can be regarded as having been trained with the training data to be examined.

If it is determined during the process that the proportions of target segmentation maps for one or more training images are incorrect, this can be used as feedback to improve the training of the classification model and ultimately the accuracy of the semantic data provided by this model increase segmentation.

For this purpose, portions of target segmentation maps that have been determined to be incorrect can be changed into at least one training image. In particular, if there are several areas with incorrect labels in the target segmentation map in one and the same training image, a new configuration of these areas in the target segmentation map can be achieved with a suitable strategy. This strategy can in particular include, for example, that the new target class divisions determined for the areas in question are not mutually contradictory. For example, it makes no sense to classify a person as a cyclist while at the same time classifying the vehicle that person is on as an e-scooter.

This procedure is somewhat analogous to a candidate in a multiple-choice First collect the questions he is unsure about answering and at the end of the exam, within the time available, revise them again in such a way that the new answers are at least internally consistent. On average, this increases the number of points achieved.

However, it is also possible, for example, to remove at least one training image from the training data for which parts of the target segmentation map were determined to be incorrect. For the classification accuracy ultimately achieved by the classification model, omitting the training image may well be a lesser evil than retaining this training image with a target segmentation map that contradicts the target segmentation maps of other training images.

This procedure is a bit analogous to the fact that when a large number of images of an object from different perspectives are merged into a three-dimensional model of this object by means of photogrammetry, poor-quality images should be discarded. A view from each perspective is then lost, but this is less detrimental to fusion than retaining the poor image, which could create inconsistencies with other images.

If the training data has been changed by adjusting target segmentation maps and/or by omitting training images, the classification model can be retrained here. It can then be assessed, for example, whether a higher proportion of the target segmentation maps is found to be correct when the method described here is carried out again. For example, it can also be assessed whether, after training with the changed training data, the classification accuracy in a measurement using test or validation data is better than in the originally trained state.

In a further advantageous embodiment, the retrained classification model is supplied with images that were recorded with at least one sensor as input images. The pixels of the input images are divided into classes by the classification model. A control signal is formed from the segmentation maps obtained in this way. A vehicle, a monitoring system and/or a system for the quality control of series-produced products is controlled with this control signal.

The method is particularly advantageous in connection with quality control or other applications where examples of associations between classifications on the one hand and image contexts on the other hand are scarce. If, for example, the product mentioned above, where part of the surface is coated with a functional coating, is produced in large numbers, it is usually the case that an inspected sample is free of defects, and the exceptional case that the coating on this sample is defective is. At the same time, this damage usually affects only very small parts of the coating in terms of area. In the coated areas, there are many more pixels in the "Coating" class than in the "Base material" class. This favors "overfitting" during training, in which a classification model that is over-parameterized compared to the available amount of training data more or less "learns the training data by heart" instead of drawing a generalized lesson from this training data. The consideration of the decision-relevant image parts creates an additional source of information for the purpose of assessing to what extent target segmentation maps are correct with regard to the present application, which is not affected by a possible "overfitting" of the classification model.

Furthermore, especially in quality control, there is also prior knowledge about which image areas can be relevant to the decision at all. The finished product is typically always in the same position in the image. It is therefore clear which image areas actually belong to the product and which image areas (e.g. a background) contain information that has nothing to do with the product itself.

In particular, the method can be fully or partially computer-implemented. The invention therefore also relates to a computer program with machine-readable instructions which, when executed on one or more computers, cause the computer or computers to carry out the method described. In this sense, control devices for vehicles and embedded systems for technical devices that are also able to execute machine-readable instructions are also to be regarded as computers.

The invention also relates to a machine-readable data carrier and/or a download product with the computer program. A downloadable product is a digital product that can be transmitted over a data network, i.e. can be downloaded by a user of the data network and that can be offered for sale in an online shop for immediate download, for example.

Furthermore, a computer with the computer program with the machine-readable data carrier or be equipped with the download product.

Further measures improving the invention are presented in more detail below together with the description of the preferred exemplary embodiments of the invention with the aid of figures.

exemplary embodiments

• 1 Ausführungsbeispiel des Verfahrens 100;
• 2 Beispielhafte Unterscheidung zwischen Trainingsbildern 2a und 2a' mit korrekten (k) bzw. inkorrekten (i) Soll-Segmentierungskarten 2b.

It shows:

• 1 embodiment of the method 100;
• 2 Exemplary distinction between training images 2a and 2a' with correct (k) or incorrect (i) target segmentation maps 2b.

1 is a schematic flowchart of an embodiment of the method 100 for quality testing of training data 2 for a classification model 1 for the semantic segmentation of input images 3.

In step 110, a classification model trained on this training data is provided for the training data 2, which contains training images 2a and associated target segmentation maps 2b. According to block 111, a plurality of classification models 1 can also be provided, which, according to block 111a, can in particular be modifications 1′ of a classification model 1, for example. Such modifications 1' can be generated by deactivating (dropping out) a random selection of neurons or other processing units from the original classification model 1. This has the advantage that these modifications 1 ′, starting from a state of the original classification model 1 trained with the training data 2 , can also be treated as having been trained with the training data 2 .

In step 120, the relevance evaluation function 5 is provided, which for an input image 3 of the classification model 1 and a binary selection mask 3a of pixels of this input image 3 specifies the parts of the input image that are relevant for the classification of these selected pixels into classes 4 .

In step 130 a set of calibration images 6 is provided for which it is known that the classification model 1 assigns or has assigned the correct classes 4 to the pixels of these calibration images 6 in each case.

In step 140, calibration evaluations 6# are determined with the relevance evaluation function 5 for the calibration images 6 and predefined binary selection masks 6a of pixels of these calibration images. These calibration evaluations 6# indicate the portions of the calibration images 6 that are relevant for the classification of the pixels selected according to the masks 6a into classes 4.

In step 150, a test evaluation 2a# is determined with the relevance evaluation function 5 based on a binary selection mask 2c of pixels of a training image 2a. This test evaluation 2a# indicates the extent to which parts of the training image 2a are relevant for the decisions of the classification model 1 to classify the pixels of the training image 2a indicated by the selection mask 2c into classes 4 respectively.

In step 160 it is checked whether the test score 2a# is consistent with the calibration scores 6#. If this is the case (truth value 1), it is determined in step 170 that the portion of the target segmentation map 2b specified by the selection mask 2c for the training image 2a is correct. Otherwise (truth value 0), it is determined in step 180 that the portion of the target segmentation map 2b specified by the selection mask 2c for the training image 2a is incorrect.

According to block 161, at least one feature and/or at least one variable can be evaluated from at least one calibration evaluation 6#. According to block 162, a feature corresponding thereto or a variable corresponding thereto can then be evaluated from the test evaluation 2a#. According to block 163, it can then be checked to what extent this corresponding feature or this corresponding variable is consistent with the feature or the variable for the calibration evaluation 6#.

In this case, in particular according to block 163a, a distribution and/or summarizing statistics for the feature or for the size can be determined for all calibration images 6#. According to block 163b, it can then be checked whether the corresponding feature or the corresponding variable is consistent with this distribution or with this summarizing statistic.

According to block 165, determinations as to the extent to which parts of the target segmentation map 2b of the training image 2a are correct can be combined to form a final result 7 for various classification models 1, such as modifications 1′ of a classification model 1 generated by drop-out.

Insofar as parts of target segmentation maps (2b) are identified as incorrect, they can be changed according to block 190a. Alternatively or in combination with this, added relevant training images 2a are removed from the training data 2. The classification model 1 can be trained again in step 200 with the training data 2 ′ changed in this way.

In step 210, images that were recorded with at least one sensor 9 are supplied as input images 3 to the retrained classification model 1*. In step 220 the pixels of the input images 3 are divided into classes 4 by the classification model 1*. In step 230, a control signal 230a is formed from the segmentation maps obtained in this way. In step 240, a vehicle 50, a monitoring system 60, and/or a system 70 for the quality control of mass-produced products is controlled with the control signal 230a.

2 Using a simple example, illustrates how a training image 2a with a correct target segmentation map 2b can be distinguished from a training image 2a′ with an incorrect (incorrect, i) target segmentation map 2b using the method described above.

Three calibration images 6, 6', 6" are recorded as an example, all of which show a person 10. The binary selection mask 6a, 6a', 6a" for the image portion that is to be divided into classes 4 includes this person 10 in each case .

Calibration image 6 shows the person 10 on an e-scooter 11. The pixels selected with the selection mask 6a are therefore correctly classified in the “e-scooter driver” class. The image portion 6#, which shows the e-scooter 11, is relevant to the decision.

Calibration image 6' shows the person 10 on a bicycle 12. Therefore, the pixels selected with the selection mask 6a' are correctly divided into the "cyclist" class. The image portion 6#', which shows the bicycle 12, is relevant to the decision.

Calibration image 6″ shows person 10 as a pedestrian. Therefore, the pixels selected with selection mask 6a″ are correctly classified in the “pedestrian” class. An image portion 6#" is relevant to the decision, which includes not only the person 10 himself but also his immediate surroundings for the purpose of checking for bicycles, e-scooters or similar objects.

Training image 2a shows a person 10 on a bicycle 12. For the decision as to which class 4 the person 10 selected with the selection mask 2c is divided into, the relevance evaluation function 5 identifies the bicycle 12 as a decision-relevant image part 2a#. This is consistent with the distribution of the 6#, 6" and 6"' calibration scores. It is thus determined that the target segmentation map 2b, which assigns the “cyclist” class to the pixels belonging to the person 10, is correct (k).

Training image 2a' shows a person 10 as a pedestrian and two bicycles 12 that are not related to person 10. The relevance evaluation function 5 identifies these bicycles 12 as a decision-relevant part 2a#' for the classification of the person 10 selected with the selection mask 2c. This is not consistent with the distribution of the calibration evaluations 6#, 6" and 6"', all of which are only Identify parts of the image in the direct spatial connection with the person 10 as relevant to the decision. It is thus established that the target segmentation map 2b, which assigns the “cyclist” class to the pixels belonging to the person 10, is incorrect (i) in this point.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• WO 2018/184963 A2 [0003]
• DE 102019204139 A1 [0004]

Claims (14)

Method (100) for checking the quality of training data (2) for a classification model (1), which is designed to assign a class (4) to individual pixels of an input image (3) and thus create a semantic segmentation map of the input image (3 ) to create, wherein the training data (2) contain training images (2a) and associated target segmentation maps (2b), with the steps: • a classification model (1) is provided (110) which has been trained for the purpose of mapping the training images (2a) onto the respectively associated target segmentation maps (2b); • a relevance evaluation function (5) is provided (120) which indicates to an input image (3) and a binary selection mask (3a) of pixels of this input image (3) to what extent (3#) proportions of the input - image (3) are relevant for the decisions of the classification model (1), to divide the pixels of the input image (3) indicated by the selection mask (3a) into classes (4); • a set of calibration images (6) is provided (130) for which it is known that the classification model (1) assigns or has assigned the correct classes (4) to the pixels of these calibration images (6); • with the relevance evaluation function (5) for the calibration images (6) and predetermined binary selection masks (6a) of pixels of these calibration images (6) calibration evaluations (6#) are determined (140) to the extent to which proportions of Calibration images (6) are relevant for the decisions of the classification model (1) to divide the pixels of the calibration image (6) indicated by the selection mask (6a) into classes (4); • with the relevance evaluation function (5), starting from a binary selection mask (2c) of pixels of a training image (2a), a test evaluation (2a#) is determined (150) to determine the extent to which parts of the training image (2a) are relevant for the decisions of the classification model (1) are to divide the pixels of the training image (2a) indicated by the selection mask (2c) into classes (4); • in response to the test score (2a#) being consistent with the calibration scores (6#) (160), it is determined (170) that the portion of the target segmentation map indicated by the selection mask (2c). (2b) is correct for the training image (2a); • otherwise it is determined (180) that the portion of the target segmentation map (2b) specified by the selection mask (2c) for the training image (2a) is incorrect. Method (100) according to claim 1 , the relevance evaluation function (5) indicating which pixels of the input image (3) must be taken into account in order to correctly classify all pixels of the input image (3) indicated by the selection mask (3a) into classes (4). Method (100) according to any one of Claims 1 until 2 , wherein • at least one feature and/or at least one variable is evaluated (161) from at least one calibration evaluation (6#); • from the test evaluation (2a#), a feature corresponding thereto, or a variable corresponding thereto, is evaluated (162); and • it is checked (163) to what extent this corresponding feature or variable is consistent with the feature or variable for the calibration evaluation (6#). Method (100) according to claim 3 , wherein the feature or the size • a quantitative portion of the calibration image (6) to which the relevance evaluation function (5) assigns a relevance above a predetermined threshold value; and/or • classes (4) into which the classification model (1) divides those pixels of the calibration image (6) to which the relevance evaluation function (5) assigns a relevance above a predetermined threshold value; and/or • a spatial position of pixels relevant for the division into classes (4) relative to the pixels in the calibration image (6) specified by the selection mask (6a). Method (100) according to any one of claims 3 until 4 , where • for the feature, or for the size, a distribution and/or summary statistics is determined (163a) for all calibration images (6#), and • it is checked (163b) whether the corresponding feature , or the corresponding quantity, is consistent with this distribution, or with this summary statistic. Method (100) according to claim 5 , where • the summary statistic includes a mean and a standard deviation; and • the corresponding characteristic or quantity is considered consistent with that statistic if it falls within a range, measured in standard deviations, about the mean. Method (100) according to any one of Claims 1 until 6 , wherein • several classification models (1) are provided (111); • Based on each of these classification models (1), it is determined (160) to what extent a portion of the target segmentation map (2b) of the training image (2a) is correct; and • these determinations are combined (165) into a final result (7). Method (100) according to claim 7 , starting from a classification model (1), at least one modification (1') of this classification model (1) is generated (111a) by a random selection of neurons or other processing units of this classification model (1) being deactivated. Method (100) according to any one of Claims 7 until 8th , whereby several determinations are brought together to a final result (7) by majority decision. Method (100) according to any one of Claims 1 until 9 , wherein • portions of target segmentation maps (2b) that are determined to be incorrect are changed (190a) to form at least one training image (2a), and/or • at least one training image (2a) for which a portion of the target segmentation map (2b) was found to be incorrect is removed from the training data (2) (190b), and wherein • the classification model (1) is trained again (200) using the training data (2') changed in this way. Method (100) according to claim 10 , wherein • images that were recorded with at least one sensor (9) are supplied to the retrained classification model (1*) as input images (3) (210); • the pixels of the input images (3) are divided (220) into classes (4) by the classification model (1*); • a control signal (230a) is formed (230) from the segmentation maps obtained in this way; and • a vehicle (50), a monitoring system (60), and/or a system (70) for the quality control of mass-produced products, with which the control signal (230a) is controlled (240). Computer program containing machine-readable instructions which, when executed on one or more computers, cause the computer or computers to perform a method (100) according to one of Claims 1 until 11 to execute. Machine-readable data carrier with the computer program claim 12 . One or more computers with the computer program after claim 12 , and/or with the machine-readable data carrier Claim 13 .

Images