DE102022204364A1 - Quality determination of object recognition by a neural network - Google Patents

Abstract

The invention relates to a method for validating a trained artificial neural network for object recognition in image data, comprising the steps: providing (S1) a test image; Providing (S2) concepts related to the test image; Reading out (S3) latent representations from an inner level of the neural network; Determining (S4) respective distances between a respective latent representation and all concepts, and assigning the respective latent representation to that of the concepts with the shortest distance; for each of the concepts: determining (S5) an average value of the respective distances between the latent representations assigned to the respective concept and the respective concept, and determining (S6) a quality of object recognition by carrying out a respective comparison i) of the number of latent representations assigned to the respective concept with a first predetermined limit value and ii) the mean value of the respective distances associated with the respective concept with a second predetermined limit value.

Description

The invention relates to a method for testing and/or validating a trained artificial neural network for recognizing an object category of an object in image data.

Particularly in the area of automated driving of vehicles such as cars or trucks, it is becoming increasingly important to automatically analyze the vehicle's surroundings and classify objects such as pedestrians into their corresponding category. Such object recognition is typically carried out on the basis of visual data, for example from a camera, but can in principle alternatively or in addition to this also include other types of sensors. Regardless of the specific type of sensor used, a data set about the vehicle's surroundings is recorded with a certain repetition frequency. To recognize objects in such a current data set, a pre-trained artificial neural network can be used, which receives information from the above-mentioned data set as input data, and in the sense of output data, a classification of an object visible in the image data into one of several predefined object categories. To train such an artificial neural network, information about objects from these categories is typically used for the input data, while the specification of the respective category of the respective object under consideration together with the specification of the associated position represent the predetermined output data. Using the example of the artificial neural network used for an automated vehicle, these would be the categories: pedestrians, cyclists, passenger cars, construction vehicles, etc.;

Due to the high complexity of an artificial neural network, caused by an enormously high number of internal parameters that are practically incomprehensible to humans, the black-box character of the artificial neural network often stands out in comparison to alternative systems that can be intuitively observed by humans. This is particularly true for artificial deep neural networks, which have a very large number of levels (so-called “layers”). However, as the degree of automation of a vehicle increases, up to a fully automatic or autonomous vehicle that no longer requires manual intervention by a driver, the required level of safety also increases, as an error in the assignment of the object to a specific one of predetermined categories or even an error in localization, in short in “object recognition”, tends to have more serious effects on the safety of one's own vehicle or other road users, the greater the authority over the control of the vehicle by the automatic driving control system. However, a commonly observed property of artificial neural networks is that deviations in the current input data of the artificial neural network in its operation from the input data used for training purposes of the artificial neural network result in certain and sometimes large deviations in the output data of the artificial neural network network. It is therefore important to check and ideally be able to quantify the reliability and safety of an artificial neural network when used for object recognition, particularly in an automated driving application.

The following information does not refer to a specific document in the prior art, but is generally available specialist knowledge: Common artificial deep neural networks used for recognizing an object category based on visual data such as camera images are typically not interpretable and therefore not evidence-based . This makes broader use in safety-critical applications, e.g. in the area of automated driving of vehicles such as cars or trucks, more difficult. Unmodified artificial deep neural networks typically have a number of feature extraction levels and a number of uninterpretable perception levels. Here, the feature extraction levels generate a non-interpretable latent representation data set with a large number of latent representations for given input data such as 2D image data. The non-interpretable perception levels generate detections based on the latent representations of the respective representation data set, which means that the objects included in the image data are assigned to object categories.

The term “interpretability” and the expression “an interpretable comparison” are each used in the present context based on the understanding of the term “interpretable” in the sense of “algorithmically comprehensible”. “Interpretability” is understood as “algorithmic comprehensibility” and “an interpretable comparison” as “an algorithmically comprehensible comparison”. The term “interpretable” is therefore not fundamentally equivalent to the expression “intuitively understandable to a person”.

In the publication “Leveraging Interpretability: Concept-based Pedestrian Detection with Deep Neural Networks” by the authors Patrick Feifel, Frank Bonarens and Frank Köster, CSCS '21, November 30, 2021, Ingolstadt, Germany (available at the time of writing at https://dl.acm.org/doi/fullHtml/10.1145/3488904.3493379 and https://doi.org, respectively /10.1145/3488904.3493379) semantic segmentations as well as the detection of bounding boxes with the localization and classification of an object are shown. This publication is hereby incorporated by reference. This publication is particularly concerned with identifying so-called “concepts” in an image scene through the use of distances. Considered clearly using the example of a pedestrian as an object to be identified in a road traffic situation, these “concepts” in the sense of this publication preferably correspond to predefined body parts, which are recognized individually in the sense of a semantic segmentation of the “pedestrian” object. In order to avoid the above-described disadvantage of unmodified feature extraction levels with non-interpretable latent representations, this publication uses a modification that includes a transformation of the latent representations to generate downstream interpretable perception levels, which are based on the respective similarities of the interpretable latent representations final detections allowed. For the purpose of such a transformation, a number of transformation levels and a number of interpretable perception levels can be used. Here, the transformation levels map the non-interpretable latent space of the common artificial neural network onto the interpretable space of the modified artificial neural network. The transformed, interpretable, latent representation dataset containing the set of latent representations can then be compared for similarity to a number of learned vectors of concepts. For further details please refer to the publication cited above.

A central problem when evaluating common artificial neural networks is that the usual metrics for assessing the risk of false detection are not suitable. Therefore, in principle, a metric can be used that is representative of the latent representations in the latent space of the artificial neural network. However, the problem remains here as to how the decision about false detection can be made and is therefore often made dependent on a single, usually arbitrary threshold.

Against this background, the object of the invention is to test and/or validate an already trained artificial neural network, in particular an artificial neural network modified to make its latent space interpretable, for the purpose of classifying objects in an image, briefly referred to as “object recognition”. to appropriately determine the quality of the recognition of an object category by the artificial neural network, so that the reliability of the object recognition can be quantified.

The invention results from the features of the independent claims. Advantageous further developments and refinements are the subject of the dependent claims.

• - Bereitstellen eines Testbildes mit Informationen über mindestens ein Objekt aus einer von vorgegebenen Objektkategorien, die möglichen Ausgangsgrößen des künstlichen neuronalen Netzes entsprechen,
• - Bereitstellen von Konzeptmasken für auf das Testbild bezogene Konzepte, wobei ein jeweiliges Konzept vorab, insbesondere für das Training des künstlichen neuronalen Netzes, als Teilelement eines jeweiligen Objekts aus den vorgegebenen Objektkategorien definiert wurde,
• - Auslesen von latenten Repräsentationen und Konzeptparametern des jeweiligen Konzepts aus einer inneren Ebene des künstlichen neuronalen Netzes nach Verwendung des Testbildes für die Eingangsdaten des künstlichen neuronalen Netzes,
• - Ermitteln von jeweiligen Distanzen zwischen einer jeweiligen latenten Repräsentation und allen Konzepten mit den ausgelesenen Konzeptparametern unter Anwendung der jeweiligen Konzeptmaske eines jeweiligen Konzepts, und Zuordnen der jeweiligen latenten Repräsentation zu demjenigen der Konzepte mit der kürzesten Distanz,
• - Für jedes der Konzepte: Ermitteln eines Mittelwerts der jeweiligen Distanzen zwischen den dem jeweiligen Konzept zugeordneten latenten Repräsentationen und dem jeweiligen Konzept, und
• - Ermitteln einer Güte einer Objekterkennung zumindest auf Basis der Erkennung einer Objektkategorie durch das künstliche neuronale Netz durch einen für jedes der Konzepte ausgeführten jeweiligen Vergleich i) der Zahl der dem jeweiligen Konzept zugeordneten latenten Repräsentationen mit einem ersten vorgegebenen Grenzwert und ii) des zum jeweiligen Konzept zugehörigen Mittelwerts der jeweiligen Distanzen mit einem zweiten vorgegebenen Grenzwert.

A first aspect of the invention relates to a method for testing and/or validating a trained artificial neural network, which serves at least for the recognition of an object category of an object in image data, comprising the steps:

• - Providing a test image with information about at least one object from one of predetermined object categories that correspond to possible output variables of the artificial neural network,
• - Providing concept masks for concepts related to the test image, whereby a respective concept was defined in advance, in particular for training the artificial neural network, as a sub-element of a respective object from the predetermined object categories,
• - Reading out latent representations and concept parameters of the respective concept from an inner level of the artificial neural network after using the test image for the input data of the artificial neural network,
• - Determining respective distances between a respective latent representation and all concepts with the read concept parameters using the respective concept mask of a respective concept, and assigning the respective latent representation to that of the concepts with the shortest distance,
• - For each of the concepts: determining an average of the respective distances between the latent representations assigned to the respective concept and the respective concept, and
• - Determining a quality of object recognition at least based on the recognition of an object category by the artificial neural network by comparing i) the number of latent representations assigned to the respective concept with a first predetermined one, carried out for each of the concepts a limit value and ii) the mean value of the respective distances associated with the respective concept with a second predetermined limit value.

The artificial neural network that needs to be evaluated has already been trained, in particular with real or synthetic test data. In such a training process, input data is typically given to the artificial neural network at a regular input interface, and a corresponding calculation is carried out using the elements of the artificial neural network (in particular transformation levels, neurons, weights, levels, nonlinearities), followed by one of several using appropriate activation functions possible output variables is activated, which preferably corresponds to an object category. Such object categories for use in road traffic can be pedestrians, cyclists, trucks, cars, traffic signs and the like. By specifying the result, the parameters of the artificial neural network can be optimized using so-called “back propagation”.

Naturally, objects captured from image data during operation of the artificial neural network fundamentally differ from the objects used for training purposes. In order to be able to determine the quality of object recognition for an individual object, a test image is provided here, which can be a real camera image, a modified camera image, or a fully synthetic image, for example from a simulation.

The test image is preferably selected to be such that the source of the image data has comparable properties (resolution, etc.) in later operation of the artificial neural network in an automated vehicle. Example sources of image data in the later operation of the artificial neural network are: camera for light in the visible range, infrared camera, stereo camera, etc.;

To determine the quality of the recognition according to the first aspect of the invention, a modified artificial interpretable deep neural network is preferably used, which in particular has a number of transformation levels and a number of interpretable perception levels. Here, the transformation levels map the non-interpretable latent space of the common artificial deep neural network described above onto the interpretable space of the extended artificial interpretable deep neural network. For example, an artificial neural network modified in this way can provide a layer of vectors of concepts that structure the latent space of the artificial neural network and enable interpretability. For example, the “ProtoPNet” known in the prior art can be used (C. Chen, O. Li, A. Barnett, J. Su, C. Rudin, “This looks like that: deep learning for interpretable image recognition”. Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems. 2019,
NeuriPS 2019).

This interpretability in the sense of algorithmic comprehensibility allows an interpretable comparison to be made between the concepts mentioned at the beginning and the latent representations. For example, the body parts of an object “pedestrian” defined as segments can be compared with associated readable data in the latent space of the artificial interpretable deep neural network, simply by handling both in compatible data formats. In this case, a vector form is preferably used both for the definition of the respective concepts and the associated latent representations of the interpretable artificial deep neural network used in order to determine a measure of a deviation of these vectors from one another, preferably by means of a vector norm, particularly preferably with the 2nd -Norm for determining a Euclidean distance of the vector of a respective concept from the vector of the latent representation of the artificially interpretable deep neural network being considered.

The terms “concept” and “latent representation” are used analogously to the subject of the publication mentioned at the beginning. For further properties of these generally known terms, please refer to the publication mentioned above. For the sake of understanding, it should be mentioned repeatedly that a respective concept is preferably obtained together with the training (ie optimization of the parameters) of the artificial neural network and is therefore already available for the method according to the first aspect of the invention. A respective concept comes from the latent space of the artificial neural network and groups the latent representations according to structurable similarities, whereby the grouping according to these similarities serves to define a respective concept itself. For the method according to the first aspect of the invention, a respective concept is to be assumed as given and the entirety of the concepts is to be understood functionally as a means that structures the latent space of the artificial neural network, and in relation to the object as a partial element of the object. Such structuring into concepts results in: Interpretability of the artificial neural network.

This interpretability allows the use of the number of assignments of latent representations to each of the concepts, as well as the calculation of the average distance of all latent representations assigned to a concept to the concept itself, preferably by determining the distances as the respective distances by using an n- Norm, preferably the 2-norm (also called the “Euclidean distance”). For example, if there are six concepts, relevant latent representations are determined, each of which can be assigned to exactly one of the six concepts rather than to areas outside the concepts. A measure of how successful this is for a respective concept is determined by comparing the number of assigned latent representations to a specific concept with a first limit value (comparison i)). Another component of quality is the comparison of the average distance between these assignments, i.e. H. between a respective latent representation and the respective concept. The shorter this distance, the better the assignment of a latent representation to one of the concepts.

By repeating comparison i) and comparison ii) for each of the concepts of an object, it is easy to draw conclusions about the quality of object recognition with regard to the object, for example by determining the number of times the first limit value is exceeded and the number of times it falls below the limit value for each object of the second limit value - or the other way around to define the lack of quality.

The quality of the recognition of an object category by the artificial neural network is therefore preferably determined by comparing i) the number of latent representations assigned to the respective concept with a first predetermined limit value for each of the concepts and ii) the corresponding concept associated mean value of the respective distances with a second predetermined limit value.

The comparison of the number of latent representations assigned to the respective concept with a first predetermined limit value as well as the comparison of the mean value of the respective distances associated with the respective concept with a second predetermined limit value can be done both by forming a difference to the respective limit value, but can also be done by dividing with Reference is made to the respective limit value, so that, for example, a percentage is obtained relative to the respective limit value. The procedure can be as follows: Comparison i): (number of latent representations assigned to the respective concept minus the first specified limit value) or (number of latent representations assigned to the respective concept divided by the first specified limit value), and comparison ii): (the the mean value of the respective distances associated with the respective concept minus the second predetermined limit value) or (the mean value of the respective distances associated with the respective concept divided by the second predetermined limit value).

The method for testing and/or validating the trained artificial neural network for the recognition of an object category of an object in image data can be carried out with several objects of different object categories, but one or more objects from a single common object category can also be used, since also in In this case, the individual object can be checked for recognition.

It is an advantageous effect of the invention that a risk classification of objects is improved in a real scenario of object recognition, for example in an application for an automated vehicle. The method according to the invention advantageously provides an algorithmically comprehensible method with metrics and limit values, which is suitable for a risk assessment of tensors from the deep layers of an artificial neural network, namely the latent representations. Among other things, this enables the expansion of an existing and already powerful so-called Automated Driving System (ADS) in accordance with the state of the art and facilitates industrialization through improved safety arguments. In particular, taking into account the algorithmic distances in the latent space provides developers with important information about objects that have a higher risk of failing correct object recognition. This knowledge can itself be used to improve the artificial neural network used.

• - Bereitstellen eines idealen Begrenzungsrahmens für ein jeweiliges Objekt des Testbildes ohne die Anwendung des künstlichen neuronalen Netzes,
• - Ermitteln eines anwendungsbezogenen Begrenzungsrahmens für ein jeweiliges Objekt des Testbildes durch Anwendung des künstlichen neuronalen Netzes mit dem Testbild für die Eingangsdaten des künstlichen neuronalen Netzes, wobei ein jeweiliger idealer Begrenzungsrahmen und ein jeweiliger anwendungsbezogener Begrenzungsrahmen jeweils einen geometrischen Bereich des Testbildes mit dem jeweiligen Objekt eingrenzen, und
• - Ermitteln eines Maßes für die Übereinstimmung zwischen dem idealen Begrenzungsrahmen und dem anwendungsbezogenen Begrenzungsrahmen als zusätzlichen Qualitätsterm für die Güte der Erkennung der Objektkategorie durch das künstliche neuronale Netz.

According to an advantageous embodiment, the quality of the object recognition includes the recognition of an object category of an object and a localization of the object in the test image, and the method further comprises the steps:

• - Providing an ideal bounding box for a respective object of the test image without using the artificial neural network,
• - Determining an application-related bounding box for a respective object of the test image by using the artificial chen neural network with the test image for the input data of the artificial neural network, wherein a respective ideal bounding box and a respective application-related bounding box each delimit a geometric area of the test image with the respective object, and
• - Determining a measure of the agreement between the ideal bounding box and the application-related bounding box as an additional quality term for the quality of the recognition of the object category by the artificial neural network.

The ideal bounding box (also called “ground truth bounding box” or “ground truth 2D bounding box” in the professional world) is provided through prior knowledge and in particular with the help of appropriate annotations of the concepts, which is based on the previously known test image is present. The ideal bounding frame is therefore to be understood as the best possible bounding frame against which a respective application-related bounding frame can be compared. According to this embodiment, both the non-interpretable, recognized application-related bounding frames and the concept-based distances of the latent representations in the interpretable deep layers of an artificial neural network are advantageously used. The additional quality term is thus advantageously included in the quality of the recognition, as explained above.

According to a further advantageous embodiment, the quality of the recognition of an object category by the artificial neural network of a respective object in the test image is divided into predetermined categories, whereby if the first predetermined limit value falls below the result of the comparison i) in relation to one of the concepts or if the value falls below of the measure of agreement between the ideal bounding box and the application-related bounding box falls below a third predetermined limit value, the object associated with the concept is classified as incorrectly recognized or not recognized, and only otherwise comparison ii) is carried out.

Accordingly, only the above-explained comparison ii) of the mean value of the respective distances associated with the respective concept is carried out with a second predetermined limit value if the object associated with the concept is not recognized as “erroneous” or “not recognized” due to a lack of agreement between the ideal bounding box and the application-related bounding box, but is recognized as at least partially correct. If the application-related bounding box is found to be sufficiently suitable, all pixels that could be assigned to one of the concepts are then determined. The distances of all assigned concept pixels are then preferably averaged for each concept, after which the resulting mean value can be compared with the second limit value. Such a conditional, two-stage process advantageously provides developers and testers with relevant information about objects that are still recognized via an application-related bounding box, but have an algorithmically based, increased risk of false detection.

According to a further advantageous embodiment, the determination of the mean value of the respective distances between the latent representations assigned to the respective concept and the respective concept takes place only on the basis of latent representations which are assigned to pixels of the test image which are located within the ideal bounding box. This advantageously simplifies the calculation effort.

According to a further advantageous embodiment, a measure of the agreement between the ideal bounding box and the application-related bounding box is determined by comparing the common areas of the ideal bounding box and the application-related bounding box or the differing area between the ideal bounding box and the application-related bounding box.

According to this embodiment, an area-related comparison between the ideal and the application-related bounding frame is advantageously carried out. Preferably, the larger the overlapping area is compared to the non-overlapping area, the higher the degree of agreement is determined.

According to a further advantageous embodiment, several application-related bounding boxes are determined per object of the test image and the application-related bounding box with the greatest correspondence to the ideal bounding box per object is selected to determine the additional quality term.

Several application-related bounding boxes per object of the test image can be obtained when using the extended, interpretable, artificial neural network mentioned at the beginning. If this is the case, according to this embodiment, a selection of the most suitable application-related one is first made Bounding frame to the ideal bounding box with respect to their overlapping surfaces and the additional quality term is determined only depending on the selected application-related bounding box.

According to a further advantageous embodiment, all pixels of the test image are taken into account for assigning the latent representations to respective concepts.

According to a further advantageous embodiment, both a respective latent representation and the respective concept are used in vector form, with each of the distances between the latent representations assigned to a respective concept and the respective concept being specified by a scalar value. For example, each such vector has 128 entries, while, for example, six fundamentally different concepts are used.

According to a further advantageous embodiment, the test image comprises a pedestrian as an object, the concepts corresponding to body zones of the pedestrian.

According to a further advantageous embodiment, a respective latent representation is assigned to that of the concepts with the shortest distance only if the shortest distance among all distances to the concepts falls below a predetermined fourth limit value.

Further advantages, features and details emerge from the following description, in which at least one exemplary embodiment is described in detail - if necessary with reference to the drawing. Identical, similar and/or functionally identical parts are provided with the same reference numerals.

• 1: Eine typische Situation, für die das zu testende bzw. zu validierende künstliche neuronale Netz in der Anwendung an einem automatisierten Fahrzeug verwendet wird.
• 2: Ein Verfahren zum Testen und/oder Validieren eines trainierten künstlichen neuronalen Netzes für die Erkennung einer Objektkategorie eines Objekts in Bilddaten gemäß einem Ausführungsbeispiel der Erfindung.
• 3: Ein zweistufiges Einteilungsverfahren zur Ermittlung einer Güte der Erkennung einer Objektkategorie durch das künstliche neuronale Netz gemäß einem Ausführungsbeispiel der Erfindung.
• 4: Eine Ermittlung einer Übereinstimmung zwischen einem idealen und einem anwendungsbezogenen Begrenzungsrahmen gemäß einem Ausführungsbeispiel der Erfindung.
• 5: Ein Verfahren zum Testen und/oder Validieren eines trainierten künstlichen neuronalen Netzes für die Erkennung einer Objektkategorie eines Objekts in Bilddaten gemäß einem weiteren Ausführungsbeispiel der Erfindung.

Show it:

• 1 : A typical situation for which the artificial neural network to be tested or validated is used in an application on an automated vehicle.
• 2 : A method for testing and/or validating a trained artificial neural network for recognizing an object category of an object in image data according to an exemplary embodiment of the invention.
• 3 : A two-stage classification method for determining the quality of recognition of an object category by the artificial neural network according to an exemplary embodiment of the invention.
• 4 : A determination of a match between an ideal and an application-related bounding box according to an embodiment of the invention.
• 5 : A method for testing and/or validating a trained artificial neural network for recognizing an object category of an object in image data according to a further exemplary embodiment of the invention.

The representations in the figures are schematic and not to scale.

For better understanding is in the 1 a preferred application case for a trained artificial neural network is shown, which needs to be tested or validated. A camera C is installed in an automated vehicle F, which has a detection area directed towards the surroundings of the vehicle F. With a certain repetition frequency, this camera C records images of the environment and transmits each of these images to a computing unit with an implemented, already trained artificial neural network NN. While the artificial neural network NN was trained with real or synthetic camera images and as part of the so-called “supervised learning” using forward and backward calculations, the given results were used to classify objects in the area around the vehicle F into categories such as pedestrians, cyclists, etc. has been optimized, it can lead to incorrect object recognition in real operation of the vehicle F due to the natural, individually determined deviation between a real pedestrian and the large number of pedestrians that were used for training purposes for the input data of the artificial neural network NN, ie that a pedestrian as such could not be recognized by the neural network NN. It is therefore important to adequately test the trained artificial neural network NN and ensure its quality when categorizing objects in the environment. However, if corresponding objects are correctly recognized by the artificial neural network NN as belonging to a given category, they are as in 1 shown usually marked with an application-related bounding frame 2, which is assigned to the respective object in a body-fixed manner. In the example shown, three pedestrians are recognized as such and marked with an application-related bounding box 2.

• - Bereitstellen S1 eines Testbildes mit Informationen über mindestens ein Objekt aus einer von vorgegebenen Objektkategorien, die möglichen Ausgangsgrößen des künstlichen neuronalen Netzes entsprechen,
• - Bereitstellen S2 von auf das Testbild bezogenen Konzepten, wobei ein jeweiliges Konzept vorab, insbesondere für das Training des künstlichen neuronalen Netzes, als Teilelement eines jeweiligen Objekts aus den vorgegebenen Objektkategorien definiert wurde,
• - Auslesen S3 von latenten Repräsentationen aus einer inneren Ebene des künstlichen neuronalen Netzes nach Verwendung des Testbildes für die Eingangsdaten des künstlichen neuronalen Netzes,

• - Ermitteln S4 von jeweiligen Distanzen zwischen einer jeweiligen latenten Repräsentation und allen Konzepten, und Zuordnen der jeweiligen latenten Repräsentation zu demjenigen der Konzepte mit der kürzesten Distanz,
• - Für jedes der Konzepte: Ermitteln S5 eines Mittelwerts der jeweiligen Distanzen zwischen den dem jeweiligen Konzept zugeordneten latenten Repräsentationen und dem jeweiligen Konzept, und
• - Ermitteln S6 einer Güte der Erkennung einer Objektkategorie durch das künstliche neuronale Netz durch einen für jedes der Konzepte ausgeführten jeweiligen Vergleich i der Zahl der dem jeweiligen Konzept zugeordneten latenten Repräsentationen mit einem ersten vorgegebenen Grenzwert und ii des zum jeweiligen Konzept zugehörigen Mittelwerts der jeweiligen Distanzen mit einem zweiten vorgegebenen Grenzwert.

2 shows a method for testing and/or validating a trained artificial neural network for recognizing an object category of an object in image data, comprising the steps:

• - Providing S1 a test image with information about at least one object from one of predetermined object categories, which correspond to possible output variables of the artificial neural network,
• - Providing S2 of concepts related to the test image, whereby a respective concept was defined in advance, in particular for training the artificial neural network, as a sub-element of a respective object from the predetermined object categories,
• - reading out S3 of latent representations from an inner level of the artificial neural network after using the test image for the input data of the artificial neural network,

• - Determining S4 of respective distances between a respective latent representation and all concepts, and assigning the respective latent representation to that of the concepts with the shortest distance,
• - For each of the concepts: Determine S5 an average of the respective distances between the latent representations assigned to the respective concept and the respective concept, and
• - Determining S6 a quality of recognition of an object category by the artificial neural network by comparing i the number of latent representations assigned to the respective concept with a first predetermined limit value and ii the mean value of the respective distances associated with the respective concept, carried out for each of the concepts a second predetermined limit value.

• - „loU“ ein flächiges Maß für die Übereinstimmung zwischen dem idealen Begrenzungsrahmen 1 und dem anwendungsbezogenen Begrenzungsrahmen 2 im Sinne der bekannten „Intersection over Union“, angegeben in Prozent als Maß für die flächige Übereinstimmung;
• - „IoU_S“ den dritten vorgegebenen Grenzwert;
• - „n_K“ die Zahl der einem jeweiligen Konzept zugeordneten latenten Repräsentationen;
• - „n_K1“ den ersten Grenzwert;
• - „n_K2“ einen fünften Grenzwert;
• - „d_crit“ den zweiten Grenzwert;
• - „d_SM“ einen sechsten Grenzwert;
• - „NOK“ Kategorie „fehlerhaft-erkannt oder nicht-erkannt“;
• - „OK“ Kategorie „vollständig erkannt“;
• - „OK+“ und „OK-“ Annäherungen der Qualität an die Kategorie „vollständig erkannt“ („OK“) in besser („OK+“) und schlechter („OK-“);
• - (A) und (B) jeweilige Stufen der Bestimmung der Güte der Erkennung einer Objektkategorie durch das künstliche neuronale Netz;

3 and 4 give details about the in 5 procedures described. This shows the 3 a two-stage method for determining the quality of the recognition of an object category by the artificial neural network of a respective object in the test image. Here designate:

• - “loU” is an areal measure of the agreement between the ideal bounding box 1 and the application-related bounding box 2 in the sense of the well-known “Intersection over Union”, given in percent as a measure of the areal agreement;
• - “IoU_S” the third specified limit value;
• - “n_K” is the number of latent representations assigned to a respective concept;
• - “n_K1” the first limit value;
• - “n_K2” a fifth limit value;
• - “d_crit” the second limit value;
• - “d_SM” a sixth limit value;
• - “NOK” category “incorrectly recognized or not recognized”;
• - “OK” category “fully recognized”;
• - “OK+” and “OK-” approximations of the quality to the “fully detected” (“OK”) category in better (“OK+”) and worse (“OK-”);
• - (A) and (B) respective stages of determining the quality of recognition of an object category by the artificial neural network;

The quality is thus advantageously divided into predetermined categories. In the first stage (A), comparison i) is carried out, i.e. the number “n_K” of latent representations assigned to the respective concept is compared with the first predetermined limit value “n_K1”. If the number is greater than the first limit value "n_K1" and the areal measure of the agreement between the ideal bounding box 1 and the application-related bounding box 2 is greater than the third limit value "loU_S", the process continues with step (B). This means that comparison ii) in step (B ), namely the comparison of the mean value of the respective distances associated with the respective concept with the second specified limit value “d_crit”. In step (B), a comparison can again be made with the number “n_K” of the latent representations assigned to the respective concept with the first predetermined limit value “n_K1”, if this has not already been done. Concepts for which the first specified limit value “n_K1” and the second specified limit value “d_crit” are exceeded are classified in the “OK” category. All concepts that fall below the limit “n_K1” are placed in the “NOK” category. The remaining area is divided into the further categories “OK+ “ and “OK-“;

4 shows a possible process for determining a measure of the agreement between the ideal bounding box 1 and the application-related bounding box 2 as an additional quality term for the quality of the recognition of the object category by the artificial neural network for the exemplary embodiment of 5 . A person is used as an exemplary object in a test image to represent a pedestrian. After providing the ideal bounding box 1, the application-related bounding box 2 is determined (see middle part of the picture) and checked for surface correspondence with each other (see right part of the picture). 2 ). The hatched area is a deviation between the two bounding frames 1, 2. The ratio of overlapping areas to deviating areas is included in determining the measure of agreement and thus in the additional quality term, which in turn is used to determine the quality of the recognition an object category is used by the artificial neural network together with the comparison i) and ii). Furthermore, some of the concepts used here are indicated, which, for example, correspond to the body areas of head, torso, arm, hand, leg and foot. Further details of the use of the bounding boxes 1, 2 are provided with the description of the 5 explained.

• - Bereitstellen S7 eines idealen Begrenzungsrahmens 1 für ein jeweiliges Objekt des Testbildes ohne die Anwendung des künstlichen neuronalen Netzes,
• - Ermitteln S8 eines anwendungsbezogenen Begrenzungsrahmens 2 für ein jeweiliges Objekt des Testbildes durch Anwendung des künstlichen neuronalen Netzes mit dem Testbild für die Eingangsdaten des künstlichen neuronalen Netzes, wobei ein jeweiliger idealer Begrenzungsrahmen 1 und ein jeweiliger anwendungsbezogener Begrenzungsrahmen 2 jeweils einen geometrischen Bereich des Testbildes mit dem jeweiligen Objekt eingrenzen, und
• - Ermitteln S9 eines Maßes für die Übereinstimmung zwischen dem idealen Begrenzungsrahmen 1 und dem anwendungsbezogenen Begrenzungsrahmen 2 als zusätzlichen Qualitätsterm für die Güte der Erkennung der Objektkategorie durch das künstliche neuronale Netz.

5 shows a supplementary embodiment to that in 2 Procedure. The following additional steps are used, see also for details 3 and 4 :

• - Providing S7 an ideal bounding box 1 for a respective object of the test image without using the artificial neural network,
• - Determining S8 of an application-related bounding frame 2 for a respective object of the test image by using the artificial neural network with the test image for the input data of the artificial neural network, wherein a respective ideal bounding frame 1 and a respective application-related bounding frame 2 each have a geometric area of the test image with the limit the respective object, and
• - Determine S9 a measure of the agreement between the ideal bounding box 1 and the application-related bounding box 2 as an additional quality term for the quality of the recognition of the object category by the artificial neural network.

However, in the sense of the two-stage test procedure for the quality of recognition, a respective latent representation is only assigned to that of the concepts with the shortest distance if the shortest distance among all distances to the concepts falls below a predetermined fourth limit value - for details see 3 .

Although the invention has been illustrated and explained in detail by preferred embodiments, the invention is not limited by the examples disclosed and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention. It is therefore clear that a large number of possible variations exist. It is also to be understood that exemplary embodiments are truly examples only and should not be construed in any way as limiting the scope, application, or configuration of the invention. Rather, the preceding description and the description of the figures enable the person skilled in the art to concretely implement the exemplary embodiments, whereby the person skilled in the art can make a variety of changes with knowledge of the disclosed inventive concept, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

Reference symbol list

1

ideal bounding box

2

application-related bounding box

F

vehicle

C

camera

NN

Computing unit with implemented artificial neural network

S1

Provide

S2

Provide

S3

Read out

S4

Determine

S5

Determine

S6

Determine

S7

Provide

S8

Determine

S9

Determine

Claims (10)

Method for testing and/or validating a trained artificial neural network, comprising the steps: - providing (S1) a test image with information about at least one object from one of predetermined object categories which correspond to possible output variables of the artificial neural network, - providing (S2) of concept masks for concepts related to the test image, with a respective one Concept was defined in advance, in particular for training the artificial neural network, as a sub-element of a respective object from the predetermined object categories, - reading out (S3) latent representations and concept parameters of the respective concept from an inner level of the artificial neural network after using the test image for the input data of the artificial neural network, - determining (S4) respective distances between a respective latent representation and all concepts with the read-out concept parameters using the respective concept mask of a respective concept, and assigning the respective latent representation to that of the concepts with the shortest distance , - For each of the concepts: determining (S5) an average of the respective distances between the latent representations assigned to the respective concept and the respective concept, and - determining (S6) a quality of object recognition at least based on the recognition of an object category by the artificial neural Network by comparing i) the number of latent representations assigned to the respective concept with a first predetermined limit value and ii) the mean value of the respective distances associated with the respective concept with a second predetermined limit value. Procedure according to Claim 1 , wherein the quality of the object recognition includes the recognition of an object category of an object and a localization of the object in the test image, further comprising the steps: - Providing (S7) an ideal bounding box (1) for a respective object of the test image without the use of the artificial neural network , - Determining (S8) an application-related bounding frame (2) for a respective object of the test image by using the artificial neural network with the test image for the input data of the artificial neural network, with a respective ideal bounding frame (1) and a respective application-related bounding frame (2 ) each delimit a geometric area of the test image with the respective object, and - determining (S9) a measure for the correspondence between the ideal bounding frame (1) and the application-related bounding frame (2) as an additional quality term for the quality of the recognition of the object category by the artificial neural network. Procedure according to Claim 2 , whereby the quality of the object recognition by the artificial neural network of a respective object in the test image is divided into predetermined categories, whereby if the value falls below the first predetermined limit as a result of the comparison i) in relation to one of the concepts or if the measure for agreement is not reached between the ideal bounding frame (1) and the application-related bounding frame (2) below a third predetermined limit value, the object associated with the concept is classified as incorrectly recognized or not recognized, and only otherwise comparison ii) is carried out. Procedure according to one of the Claims 2 until 3 , whereby the determination of the mean value of the respective distances between the latent representations assigned to the respective concept and the respective concept takes place only on the basis of latent representations which are assigned to pixels of the test image which are located within the ideal bounding frame (1). Procedure according to one of the Claims 2 until 4 , wherein determining a measure of the agreement between the ideal bounding box (1) and the application-related bounding box (2) by comparing the common areas of the ideal bounding box (1) and the application-related bounding box (2) or the differing area between the ideal bounding box (1) and the application-related bounding box (2). Procedure according to one of the Claims 2 until 5 , wherein several application-related bounding frames (2) are determined per object of the test image and the application-related bounding box (2) with the greatest correspondence to the ideal bounding box (1) per object is selected to determine the additional quality term. Method according to one of the preceding claims, wherein all pixels of the test image are taken into account for assigning the latent representations to respective concepts. Method according to one of the preceding claims, wherein both a respective latent representation and the respective concept are used in vector form, a respective one of the distances between the latent representations assigned to a respective concept and the respective concept being specified by a scalar value. Method according to one of the preceding claims, wherein the test image comprises a pedestrian as an object, the concepts being associated with body zones of the pedestrian. Method according to one of the preceding claims, wherein the assignment of a respective latent representation to that of the concepts with the shortest distance only takes place if the shortest distance among all distances to the concepts falls below a predetermined fourth limit value.

Images