DE102022208480A1 - Method for evaluating a trained deep neural network - Google Patents

Abstract

The present disclosure relates to a method for evaluating a deep neural network (10) that has been trained to recognize objects in the environment of a motor vehicle. At least one reference value is determined based on a predetermined number of evaluation data sets and with the help of the deep neural network (10). Furthermore, a number of test data sets are provided, from which a number of additional training data sets are generated using a selected method for varying data sets. The deep neural network (10) is trained in an additional training step with the additional training data sets generated, with at least one evaluation variable being determined after going through the additional training step based on the predetermined number of evaluation data sets and with the help of the deep neural network (10). The deep neural network (10) is assessed as unsuitable for the selected method if a difference between the at least one reference variable and a corresponding evaluation variable exceeds a predetermined limit value.

Description

Technical area

The invention relates to a method for evaluating a trained deep neural network. The invention further relates to computer programs and computer program products for carrying out the method.

State of the art

The perception or modeling of a vehicle environment represents a significant challenge in the development of automated driving functions or advanced driver assistance systems (ADAS). For object detection, i.e. classification ) and localization of sensory objects, deep neural networks (DNN) play a crucial role due to their excellent performance.

Using a training algorithm and a number of training iteration steps, a feature space is learned in the internal structures of a deep neural network through which the objects to be recognized can be represented. By expanding a corresponding training data set, the performance of a DNN can generally be improved. After successful basic training, training data sets can be provided whose objects are difficult for the DNN to detect, although correct detection must still be expected depending on the requirements. Such training data sets can contain images with rare situations (corner cases) or images with limited image quality (e.g. in terms of contrast, brightness, etc.).

As a criterion, performance says nothing about how a DNN learns its feature space based on pre-leveled training data in a finite number of training iterations. The internal structures of a DNN, which can also be referred to as hidden or latent structures, are essentially incomprehensible when viewed from the outside. Therefore, during or after training, it is not clear from the outside whether learned representations stabilize in one area of the feature space or whether completely different areas are used from training step to training step to represent what has been learned.

In addition to performance, robustness is also a particularly important criterion for assessing sufficient training. The goal of training is not to supply a DNN with so many data sets that every relevant situation has been taken into account once in the training ("training by memorization all situations"), as such an approach carries the risk of over fitting. Instead, through training, the DNN should acquire the ability to generalize in object recognition, which can be assessed by sufficient robustness.

In the German patent application 10 2021 207 505.3 A method for training a deep neural network for object recognition in the environment of a motor vehicle was presented, which enables an evaluation of a current learning state of a training.

However, a subsequent evaluation of an already trained deep neural network is not possible with the described method.

Brief description of the invention

Against this background, the invention is based on the object of providing a method for evaluating a deep neural network whose training for object recognition in the environment of a motor vehicle has already been completed.

Accordingly, a method according to the main claim as well as a computer program and a computer program product according to the secondary claims are proposed. Further refinements are the subject of the respective dependent claims.

According to a first aspect of the invention, the object is achieved by a method for evaluating a deep neural network that has been trained to recognize objects in the environment of a motor vehicle. At least one reference value is determined based on a predetermined number of evaluation data sets and with the help of the deep neural network. Furthermore, a number of test data sets are provided, from which a number of additional training data sets are generated using a selected method for varying data sets. The deep neural network is trained in an additional training step with the additional training data sets generated, with at least one evaluation variable being determined after going through the additional training step based on the predetermined number of evaluation data sets and with the help of the deep neural network. The deep neural network is assessed as unsuitable for the selected method if there is a difference between at least one reference value and a corresponding evaluation size exceeds a predetermined limit value.

Accordingly, the deep neural network can be assessed as suitable for the selected method if the difference between the least one reference variable and a corresponding evaluation variable adheres to the predetermined limit value.

One idea behind the present invention is not to use additional training steps for extended training of a deep neural network, but rather to use them to evaluate an already trained deep neural network.

According to a further development of the method, it can therefore be provided that originally learned parameters of the trained deep neural network are first read out and saved before a first additional training step is carried out.

By freezing the training state of the deep neural network, the learned parameters of the deep neural network can be reset to the original learning state after an evaluation. The stored parameters can include, for example, weighting parameters and prototypes or concepts of feature extraction levels and perception levels for the internal structures of a deep neural network.

According to a further development of the method, all parameters of the deep neural network can be reset to the stored, originally learned parameters after the at least one evaluation variable has been determined.

Since the additional training steps are not intended to result in any additional training, for example to avoid the risk of over fitting, the original read and saved training status is reset as soon as the deep neural network has been assessed as suitable for the selected method. The additional training carried out therefore only has an impact on one evaluation step, which is based on the selected method for varying data sets. An additional training step therefore has no influence on the practical operation of an evaluated deep neural network.

According to a further development of the method, a number of different methods for varying data sets can be provided, with a number of additional training data sets being provided for each of the different methods for varying data sets, so that a number of successive additional training steps corresponding to the number of methods is carried out.

Thus, the deep neural network can be evaluated sequentially based on different critical limit cases.

In this context, parameters of the deep neural network can be reset to the saved, originally learned parameters after each additional training step as soon as the corresponding evaluation variables have been determined.

This means that each additional training for a respective method for varying data sets can begin at the same original training level of the deep neural network. A previous additional training step therefore has no influence on the evaluation of a currently selected method.

According to a further development, a corresponding latent representation data set can be generated for each evaluation data set by the deep neural network, with relevant latent representations being selected from the corresponding latent representation data set for annotated data of each evaluation data set. An average distance value in the latent space can be determined from the selected relevant latent representations of the evaluation data sets, with the determined average distance value being used as a reference variable and/or as an evaluation variable.

Alternatively or additionally, output data can be generated for each evaluation data set by the deep neural network, with a performance value being determined by means of a comparison between the generated output data and the reference data of the respective evaluation data set, the performance value being used as a reference variable and/or as an evaluation variable.

According to a further development, the method for varying data sets can include a parameterized change of all data in a data set.

Alternatively or additionally, the method for varying data sets may include adding objects to a respective data set.

According to a further aspect of the invention, the task is solved by a computer program which, when stored on a computing unit is carried out half of a maneuver planning device, which instructs the respective computing unit to carry out the method.

According to a further aspect of the invention, the object is achieved by a computer program product with a program code for carrying out the method, which is stored on a medium readable by a computer.

Brief description of the drawing figures

• 1 zeigt ein Blockdiagramm eines tiefen neuronalen Netzes;
• 2 zeigt eine Bildeingangsdatensatz mit einem annotierten Begrenzungsrahmen und einem latenten Repräsentationsdatensatz Z;
• 3 zeigt ein Ablaufdiagramm eines erfindungsgemäßen Evaluierungsverfahrens.

Further features and details emerge from the following description, in which at least one exemplary embodiment is described in detail - if necessary with reference to the drawing. Described and/or illustrated features form the subject matter on their own or in any meaningful combination, possibly also independently of the claims, and in particular can also be the subject of one or more separate applications. Identical, similar and/or functionally identical parts are provided with the same reference numerals. Show:

• 1 shows a block diagram of a deep neural network;
• 2 shows an image input data set with an annotated bounding box and a latent representation data set Z;
• 3 shows a flowchart of an evaluation method according to the invention.

Description of execution types

In the 1 a block diagram of a deep neural network 10 (DNN) is shown, which has been trained to detect objects, for example pedestrians, in camera-based 2D image data from a vehicle environment. The DNN 10 has a number of feature extraction levels 11 and a number of perception levels 12. The feature extraction levels 11 are designed to generate a latent representation data set Z for a current image input data set 1 and to pass it on to the perception levels 12. The perception levels 12 are designed to compare latent representations from a current latent representation data set Z with a number of learned prototypes for different classes of objects for similarity, so that objects in the 2D image data can be recognized, ie classified and localized, based on this.

The Indian 2 The image input data set 1 shown contains or describes a 2D image of a vehicle environment, which may have been recorded, for example, with a vehicle front camera. The image input data set 1 has a number of pixels which are arranged in rows and columns relative to one another in accordance with the height H and a width W of the image. Each pixel of the image input data set 1 is described, for example, by a vector in the three-dimensional RBG color space.

The DNN 10 can, for example, be designed as a convolutional neural network (CNN), the architecture of which provides special convolution and bundling structures in the feature extraction levels 11.

With the feature extraction levels 11, a latent representation data set Z is generated for each current image input data set 1, with a data reduction being achieved with regard to the number of data. For example, a latent representation data set Z with 512 x 256 latent representations Z _i,j is generated from an image input data set 1 with a size of 2048 x 1024 pixels.

While each pixel is defined by a vector with three color values in the RGB color space, a respective latent representation contains a vector with features in an n-dimensional latent space. Each latent representation encodes in the n features (e.g. 256 features) semantic relationships between image input data of a receptive field that has been incorporated into the respective latent representation.

How a respective image input data set 1 is mapped onto the latent representation data set Z is determined, on the one hand, by the network architecture of the feature extraction levels 11 of the DNN 10, and on the other hand by a number of associated weighting parameters that have been learned using machine learning based on training data.

The perception levels 12 of the DNN 10 have a number of prototypes that can be represented by a vector in the same latent space as the latent representations of the latent representation data set Z. In this respect, all latent representations can each be compared with a number of prototypes, which are also using machine learning have been learned based on the training data. From the comparison, the DNN 10 produces output data sets 3 which contain classes and positions of objects that have been recognized by the perception levels 12 in the latent representation data set Z.

The internal structures of DNN 10, which can also be referred to as hidden or latent structures, are fundamental when viewed from the outside additionally incomprehensible. However, the latent representations Z _i,j can be assigned to a spatially corresponding group of pixels of the image input data set 1.

For training or evaluation purposes, the DNN 10 is connected to a training algorithm 20. With the training algorithm 20, the weighting parameters and prototypes can be gradually optimized with a sequence of training iteration steps and by minimizing an internal cost function. For each training iteration step, a training data set is provided, which includes an image input data set 1 and an associated reference data set 2. With each image input data set 1, a reference data set 2 is provided, which is in the 2 is shown as an annotated bounding box, the training algorithm is given 20 classes and positions of objects to be recognized, which are present in an image of the corresponding image input data set 1 and which are to be recognized by the DNN 10. From the DNN 10, the training algorithm 20 is also supplied with an output data set 3, which contains classes and positions of objects that have been recognized by the perception levels 12 from the latent representation data set Z. The weighting parameters and prototypes of the DNN 10 are adjusted by the training algorithm 10 in each training iteration step in such a way that the classes and positions of objects of the output data set 3 match those of the reference data set 2 as closely as possible.

In the 3 a sequence of an evaluation method 100 according to the invention is shown, with which a trained deep neural network can be evaluated with regard to its robustness and/or performance.

In a first step 101, a DNN 10 and an associated training algorithm 20 are provided, which, as a fully trained hardware and/or software component, may have been supplied by a supplier, for example. The already learned weighting parameters and prototypes of the feature extraction levels 11 and the perception levels 12 are frozen and saved as a data record.

In a second step 102, a predetermined number of evaluation data sets is provided, each evaluation data set comprising an image input data set 1 and an associated reference data set 2. The evaluation data sets are fed to the DNN 10 and the associated training algorithm 20 in a defined order. For each evaluation data set, the DNN 10 generates a latent representation data set Z and an output data set 3 with data about classes and positions of recognized objects.

The data of reference data set 2 are compared with those of output data set 3. Based on a number of objects that were not recognized by the DNN 10 or were recognized as false positive compared to the reference data, an error rate F is determined and stored associated with each evaluation data set. For the number of all evaluation data sets, an average and/or maximum error rate is determined and stored as a reference value for the performance of the DNN 10.

Furthermore, based on the reference data set 2, all latent representations Z _i,j that match the positions of the annotated bounding boxes are determined and stored. A distance average is calculated from the stored latent representations Z _i,j of all evaluation data sets, which is saved as a reference value for the robustness of the DNN 10.

In a third step 103, a number of test data sets and a number of methods for varying data sets are provided. For each method of varying data sets, a number of additional training data sets are generated.

The test data sets each contain an image input data set 1 and an associated reference data set 2, with the classes and positions of the objects to be recognized, which are annotated or specified as usual.

Methods for varying data sets can be global changes to the image such as contrast or brightness changes. But additions of additional objects in the image (e.g. people, pedestrians, etc.) are also possible. Furthermore, an image can be overlaid with snowfall, rain or other image noise.

If, for example, the method of contrast changes is selected, an additional training data set is generated based on each test data set, the pixels of which in the image input data set 1 have been manipulated or adjusted by a predetermined value with regard to the contrast values.

In a fourth step 104, the DNN 10 is placed in a training mode and trained in additional training with a number of additional training data sets that have been generated for a first selected method for varying data sets. Became a corresponding one If the additional training step with the number of additional training data sets is completed, the training mode of the DNN 10 is ended again.

In a fifth step 105, the evaluation data sets are fed to the DNN 10 and the associated training algorithm 20 in the defined order. For each evaluation data set, the DNN 10 again generates a latent representation data set Z and an output data set 3 with data about classes and positions of recognized objects.

In a sixth step 106, all latent representations Z _i,j that match the positions of the annotated bounding boxes are determined for each evaluation data set based on the reference data set 2. From the determined latent representations Z _i,j of all evaluation data sets, a distance average is again determined, which is saved as an evaluation variable for the robustness of the DNN 10.

A difference is formed from the determined evaluation variable for the robustness and from the reference value for the robustness of the DNN 10 stored in the second step 102 and compared with a limit value for a change in robustness. If the limit value is not met, i.e. if the robustness has deteriorated to a certain extent after the additional training, the deep neural network 10 is assessed as not suitable for the selected method. The procedure for evaluating the DNN 10 can be completed with step 110.

If the limit value for the change in robustness has been met, in a seventh step 107 an error rate F is determined again for each evaluation data set and saved by comparing the reference data sets 2 with the output data sets 3. As previously described in the second step 102, the error rate F results from a number of objects that were not recognized by the DNN 10 or were recognized as false positives compared to the reference data. Accordingly, an average and/or maximum error rate is again determined and stored for the number of all evaluation data sets, which serves as an evaluation variable for the performance of the DNN 10.

From the determined evaluation variable for the performance and the reference value for the performance of the DNN 10 stored in the second step 102, a difference is further determined in the seventh step 107 and compared with a predetermined limit value for a change in performance. If the limit value is not met, i.e. if the performance has deteriorated to a certain extent after the additional training, the deep neural network 10 is assessed as not suitable for the selected method. The procedure for evaluating the DNN 10 can be ended 110 accordingly.

If the limit value for the change in performance has been met, the DNN 10 is assessed as suitable for the selected method. In a subsequent eighth step 108, the DNN 10 can be reset by playing back the originally learned weighting parameters and prototypes of the feature extraction levels 11 and the perception levels 12 stored as a data record. If it is also determined in the eighth step 108 that another method for varying data sets, i.e. one not yet used for additional training, is available, in the fourth step 104 the DNN 10 is again put into a training mode and with a number of additional training data sets additional training that was generated for another selected method for varying data sets.

If it is determined in the eighth step 108 that all additional training data sets have been used for all methods for varying data sets, the DNN 10 is assessed as suitable for all methods used. The method ends with a positive evaluation of the DNN 10 in step 109.

If in the sixth step 106 and/or seventh step 107 the deep neural network 10 is assessed as not suitable for a selected method, as an alternative to terminating the method 110 it can be provided to carry out further additional training for the methods for varying data sets that have not yet been used to get a complete list of which methods the deep neural network is suitable for and which is not.

Although the subject matter has been illustrated and explained in detail by exemplary embodiments, the invention is not limited by the disclosed examples and other variations can be derived therefrom by those skilled in the art. It is therefore clear that a large number of possible variations exist. It is also to be understood that exemplary embodiments are merely examples and should not be construed in any way as limiting the scope, application, or configuration of the invention. Rather, the preceding description and the description of the figures enable the person skilled in the art to concretely implement the exemplary embodiments, with the person skilled in the art making various changes, for example, with knowledge of the disclosed inventive concept with regard to the function or arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection which is defined by the claims and their legal equivalents, such as further explanations in the description.

List of reference symbols

1

Image input data set

2

Reference dataset

3

Output data set

10

deep neural network (DNN)

11

Feature extraction levels

12

Perception levels

20

Training algorithm

F

Error rate

Z

latent representation dataset

Zi,j

latent representation

100

Evaluation procedures

101

first step

102

second step

103

Third step

104

fourth step

105

fifth step

106

sixth step

107

seventh step

108

eighth step

109

End; rated as suitable for all methods used

110

End; assessed as not suitable for a selected method

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 1020212075053 [0006]

Claims (10)

Method for evaluating a deep neural network (10) which has been trained to recognize objects in the environment of a motor vehicle, wherein at least one reference variable is determined based on a predetermined number of evaluation data sets and with the aid of the deep neural network (10), and wherein a number of test data sets is provided, from which a number of additional training data sets are generated using a selected method for varying data sets, the deep neural network (10) being trained in an additional training step with the additional training data sets generated, after going through the additional training step based on the predetermined number of evaluation data sets and with the help of the deep neural network (10) at least one evaluation variable is determined, and wherein the deep neural network (10) is assessed as not suitable for the selected method if there is a difference between the at least one reference variable and a corresponding evaluation variable exceeds a predetermined limit. Procedure according to the above Claim 1 , whereby originally learned parameters of the deep neural network (10) are saved before a first additional training step is carried out. Procedure according to the above Claim 2 , whereby all parameters of the deep neural network (10) are reset to the stored, originally learned parameters after the at least one evaluation variable has been determined. Method according to one of the above Claims 1 until 3 , wherein a number of different methods for varying data sets are provided, with a number of additional training data sets being generated from the test data sets for each of the different methods for varying data sets, so that a number of successive additional training steps corresponding to the number of methods is passed through. Method according to one of the above Claims 1 until 4 , wherein for each evaluation data set a corresponding latent representation data set (Z) is generated by the deep neural network (10), relevant latent representations (Z _i,j ) being selected from the corresponding latent representation data set (Z) for annotated data (2) of each evaluation data set be, wherein a distance average value in the latent space is determined from the selected relevant latent representations of the evaluation data sets, and wherein the determined distance average value is used as a reference variable and / or as an evaluation variable. Method according to one of the above Claims 1 until 5 , wherein output data (3) are generated for each evaluation data set by the deep neural network (10), a performance being determined by means of a comparison between the generated output data (3) and the reference data (2) of the respective evaluation data set, the performance being used as a reference variable and/or is used as an evaluation variable. Method according to one of the preceding claims, wherein the method for varying data sets comprises a parameterized changing of all data in a data set. Method according to one of the preceding claims, wherein the method for varying data sets comprises adding objects into a respective data set. Computer program which, when executed on a computing unit, instructs the respective computing unit to carry out a method according to one of the Claims 1 until 8th to carry out. Computer program product with a program code that is stored on a computer-readable medium for carrying out the method according to one of the Claims 1 until 9 .

Images