DE102021212732A1 - REVIEWING TESTING AND/OR TRAINING RECORDS FOR A COMPUTER-BASED MACHINE LEARNING MODULE - Google Patents

Abstract

A first general aspect of the present disclosure relates to a computer-implemented method for checking test and/or training data sets for a computer-based machine learning module. The method includes receiving an embedding ("embedding" in German) for each datum of a first testing and/or training data set and a second testing and/or training data set for a computer-based machine learning module. An embedding represents a datum according to a specific rule by a vector with a lower dimensionality than the datum in question. The method further includes calculating a measure of similarity of a first distribution of the embeddings of the first data set and a second distribution of the embeddings of the second data set and outputting the calculated measure of similarity.

Description

State of the art

Computer-based machine learning modules - especially those for safety-critical applications such as autonomous or assisted driving - must be extensively tested in many cases (in the present disclosure, the term "testing" includes both a validation in the sense of determining a performance within the framework of a model selection of the computer-based machine learning module as also a check of a generalization error of the chosen model - often referred to as "testing" in a narrower sense). For this purpose, a trained computer-based machine learning module can be loaded with a test data set. The performance and/or reliability of the trained computer-based machine learning module can be determined based on the outputs (e.g. classification results) of the computer-based machine learning module for the data of the test data set. It may be necessary here that the training and test data sets are different but sufficiently similar (e.g. the training and test data sets can be independently and identically distributed and sampled from the same source). If this is not the case, in some cases the performance and/or reliability of the trained computer-based machine learning module cannot be determined with sufficient certainty using the test data set. For example, it is conceivable that a test dataset does not represent certain data of the training dataset or does so insufficiently (i.e. the training dataset does not cover the training dataset sufficiently). In these cases, the performance and/or reliability of the trained computer-based machine learning module in relation to the data of the training dataset that is not represented or is insufficiently represented cannot be reliably determined using the test dataset in some cases.

It is therefore worthwhile to check training and test datasets to see whether they are sufficiently similar and, if necessary, to establish a sufficient similarity between the training and test datasets.

Summary

A first general aspect of the present disclosure relates to a computer-implemented method for checking test and/or training data sets for a computer-based machine learning module. The method includes receiving an embedding for each datum of a first testing and/or training dataset and a second testing and/or training dataset for a computer-based machine learning module. An embedding represents a datum according to a specific rule by a vector with a lower dimensionality than the datum in question. The method further includes calculating a measure of similarity of a first distribution of the embeddings of the first data set and a second distribution of the embeddings of the second data set and outputting the calculated measure of similarity.

A second general aspect of the present disclosure relates to a method for generating an augmented data set for a computer-based machine learning module. The method includes checking a test and / or training data set using the method according to the first general aspect and augmenting the test and / or training data set by adding one or more data, so that a similarity of the first distribution of the embeddings of the first data set and the second distribution of the embeddings of the second data set in relation to the measure of similarity becomes larger.

A third general aspect of the present disclosure relates to a method for testing or training a computer-based machine learning module. The method includes receiving a verified test and/or training data set according to the first general aspect or an augmented test and/or training data set according to the second general aspect and testing or training the computer-based machine learning module with the received data set to generate a tested or receive a trained machine learning module.

A fourth general aspect of the present disclosure relates to a method for applying a computer-based machine learning module. The method comprises providing a tested or trained machine learning module according to the third general aspect and processing application data by the received machine learning module.

A fifth general aspect of the present disclosure relates to a system configured to carry out the methods of any one of the first to fourth general aspects.

The techniques of the first through fifth general aspects may have one or more advantages in some cases.

First, in some cases, using the techniques of the present disclosure, it can be verified whether a test data set and a training data set are sufficiently similar that a with the training data set trained computer-based machine learning module can be tested with sufficient reliability with the test data set. It can thus be ensured that the training and/or test adequately prepare the computer-based machine learning module with regard to its tasks in the field. By determining a similarity measure of the distributions of the two training datasets, a global statement can be made about the datasets. The reason why a measure of similarity between two data sets is (too) low can be located in one of the two data sets or in both. For example, data in certain areas in the training data set or in the test data set can be missing or too small (eg image data of certain signs at night and in the rain). In some cases, such and other problems can be detected relatively quickly and with reasonable effort by calculating the similarity measure based on the embeddings. In some cases, this can prevent insufficiently trained and/or tested computer-based machine learning modules from being used in the field and possibly not functioning satisfactorily there (which, depending on the technical environment, can have various consequences, from annoyances to significant security risks).

Second, the embeddings of the techniques of the present disclosure can be chosen flexibly and have more meaningfulness than other information in determining the similarity of the data sets. For example, it is possible that the vector of the embedding is generated both based on the useful information of a datum (e.g. the image data) and meta information of the datum (e.g. human-readable labels). In addition, customized embeddings can be created and used for various questions. For example, the embedding can contain a representation from an intermediate layer of a computer-based machine learning module or be determined from this. This can be helpful as the differences in the vectors of the embeddings can reflect differences in the information processed by the computer-based machine learning modules. Calculating a similarity measure based on this information can be more meaningful for the question whether e.g. the data of a training dataset is adequately represented in a test dataset than similarity measures of the data itself. In addition, in some cases the processing of the lower-dimensional vectors can be significantly faster and/or be more resource-efficient than the processing of the data itself. In some situations, the embedding can be orders of magnitude smaller than the data itself, which can correspondingly speed up its processing and/or make it resource-efficient. In some cases, both factors can interact so that a meaningful and efficient assessment of mutual coverage of two data sets becomes possible.

Third, using a measure of similarity determined on the distribution of embeddings, datasets can be augmented in order to avoid problems that may be caused by a similarity that is too low. Again, this can be facilitated in some cases by using the embeddings, because in some situations the embeddings can reflect features relevant to the performance of the computer-based machine learning module better and/or more clearly than the data itself. The insufficient similarity in space of the embeddings can thus be sometimes be more predictive of the existence of a problem than similarities found in the data itself.

Some terms are used in a specific manner in this disclosure.

A “computer-based machine learning module” in the present disclosure is any device that can be or has been trained to perform one or more tasks using machine learning. As part of a training, the computer-based machine learning module can be supplied with training data as input data and the properties of the computer-based machine learning module can be adjusted in response to the processing of the training data (e.g. by analyzing the output data) in order to perform the one or more tasks in a defined way (e.g. with a certain accuracy). A computer-based machine learning module can contain a model that determines the form of processing the respective input data into output data. The model may be parameterized, where the parameters of the computer-based machine learning module are fixed during learning. Computer-based machine learning modules may include, but are not limited to, (artificial) neural networks (eg, convolutional and/or recurrent and/or deep neural networks - the former are commonly referred to as CNNs and RNNs). Computer-based machine learning modules can be set up (and possibly trained) for various tasks. For example, a computer-based machine learning module can perform a classification task or a regression task. Computer-based machine learning modules can be implemented in any suitable form, ie, in software, in dedicated hardware, or in a hybrid of software and dedicated hardware. Therefore, the computer-based machine learning modules can be a software module (also in a higher-level Integrated software system) that can be run on a general-purpose processor. In other cases, a computer-based machine learning module may be implemented (at least in part) in circuitry.

In the present disclosure, an “embedding” represents a datum according to a specific regulation by a vector of lower dimensionality than the respective datum. An embedding is therefore a representation of the respective date in a space with a lower dimensionality than a space in which the data itself can be represented. The term embedding means that the respective datum is embedded in the space of lower dimensionality, i.e. it is transformed into this space by means of the specific rule. No other requirements are directed to the specific regulation. In some examples, the particular rule is processing a datum with a computer-based machine learning module, wherein the embeddings are read from the computer-based machine learning module (e.g., as a plurality of activations and/or outputs from units of the computer-based machine learning module). For example, an image datum containing 1024x2048 pixels, each pixel having three color values with a color depth of 16 bits, can be represented by an embedding with 100 elements (e.g. the activation values at a specific point in a neural network). Additional aspects of the nature and creation of the embeddings are discussed below.

The term "vector" in the present disclosure describes an ordered sequence of elements (e.g. ten or more elements, or fifty or more elements, or 100 or more elements). For example, an element can be a real number, an integer, or a natural number. A vector can be displayed and/or saved in various forms. For example, the entries in a matrix with multiple rows and columns are a vector because the contents contain an ordered sequence of elements. The same applies to a list of tuples (e.g. that assigns a rank number or an identifier to an element).

A "test data set" according to the present disclosure is a collection of a plurality of data suitable and selected for testing a computer-based machine learning module. As mentioned, the elements of the dataset are referred to as data (singular date). Depending on the nature of the computer-based machine learning module to be tested, the data of a test data set can contain one or more labels. Each datum of a test data set with image data can contain a single image (or a section from it) or a video. Meta information and other information associated with the individual datum (e.g. labels) are referred to as part of the datum. The part of the datum that is processed by the computer-based machine learning module as part of its execution is referred to as useful information (e.g. the image information in an image datum).

According to the present disclosure, a “training dataset” is a collection of a plurality of data suitable and selected for training a computer-based machine learning module. Incidentally, what was said for test data sets also applies to the training data set.

In the present disclosure, the term "database" encompasses any system in which data (e.g. data of a test and/or training data set) can be stored and made available. A database can have any physical implementation (e.g. distributed across multiple locations).

character list

• 1 FIG. 12 is a flow chart illustrating the methods of the present disclosure.
• 2 FIG. 12 schematically shows a system for performing the techniques of the present disclosure.
• 3 FIG. 12 schematically shows a technique for creating embeddings according to the present disclosure.
• 4 is a schematic representation of the distributions of embeddings of multiple datasets.

Detailed description

First, based on 1 and 2 an overview of the techniques of the present disclosure is provided. After that, the creation of the embeddings in relation to 3 explained in more detail. Finally follows with the help of 4 a discussion of techniques for using the similarity information obtained using the techniques of the present disclosure. Finally, some concrete applications of the computer-based machine learning modules of the present disclosure are presented.

1 FIG. 12 is a flow chart illustrating the methods of the present disclosure. 1 shows in three different columns a method for creating the embeddings 110 (left), a method for checking test and/or training data sets for a computer-based machine learning module 120 (middle) and a method for using the calculated similarity measures 130 (right).

A computer-implemented method for reviewing test and/or training datasets for a computer-based machine learning module includes receiving 121 an embedding for each datum of a first test and/or training dataset (also referred to as “first dataset” for short) and one second test and/or training dataset (also referred to as “second dataset” for short) for a computer-based machine learning module. An embedding represents a datum according to a specific rule by a vector with a lower dimensionality than the datum in question. In other words, a respective embedding is received for each datum of the first test and/or training data record and each datum of the second test and/or training data record.

In some examples, the first testing and/or training data set is a training data set and the second testing and/or training data set is a testing data set. However, it is also conceivable that both the first and the second test and/or training data record are a training data record or that both the first and the second test and/or training data record are a test data record.

In some examples, the method may include generating 110 the embeddings for each datum of the first testing and/or training dataset and the second testing and/or training dataset for the computer-based machine learning module (more on this with reference to 3 ). In other examples, the embeddings may be read from an embedding database (ie, the embeddings were previously created and stored in the database for later use).

The method further includes calculating 123 a measure of similarity of a first distribution of the embeddings of the first data set and a second distribution of the embeddings of the second data set. A distribution here is the distribution of the embeddings in a space of lower dimensionality of the vectors of the embeddings.

Calculating a measure of similarity may include using a similarity metric of the first and second distributions. In other words, it can be determined to what extent the first and second distribution differ in the space of the embeddings. Any similarity metric can be used that determines a global statement about the similarity of the first and second distributions based on the distributions. For example, a Kullback-Leibler distance of the first and second distributions, or a distance between the first and second distributions determined using a Wasserstein metric (where larger distance means less similarity) can be calculated. In some examples, the calculation of the measure of similarity can be approximated (e.g. a dimensionality of the embeddings can be reduced and/or a distribution of the embeddings can be approximated).

In some examples, the method may include determining 122 the first distribution of embeddings for the data of the first testing and/or training dataset and the second distribution of embeddings for the data of the second testing and/or training dataset. Determining the distributions 122 and calculating the degree of similarity 123 can be done in one step using the vectors of the embeddings.

Depending on the similarity metric used, a measure of similarity results for the first and second distribution. The similarity measure can take any form (e.g. a real value). Again depending on the similarity metric used, in some examples a higher value may mean a greater similarity of the first and second distributions than a lower value (or vice versa).

The method further includes outputting 124 the calculated measure of similarity.

In some examples, the first and/or second distribution may include a distribution of a subset of the first and second embeddings, respectively. For example, the embeddings of the first test and/or training data set and/or the embeddings of the test and/or training data set can be filtered according to a predefined criterion. The predefined criterion may relate to parts of the embeddings that are generated based on a respective datum's payload (e.g. the image data). Additionally or alternatively, the predefined criterion may refer to parts of the embeddings that are generated based on meta-information of the datum (e.g. human-readable labels). For example, you can filter according to certain characteristics of the data. In the case of image data, embeddings that show specific image content (e.g. specific objects and/or specific environmental conditions) can be filtered. In this case, filtering can be done using human-readable meta information (e.g. labels).

2 FIG. 1 schematically shows a system 20 for performing the techniques of the present disclosure payment. In this example, the system 20 is divided into a processing unit 21 and an output interface 22 .

The data of the first test and/or training data record can be stored in a first database 23 . The data of the second test and/or training data record can be stored in a second database 24 . A processing unit 25 is coupled to the first and second databases 23,24. The processing unit can have a first sub-unit 26, which calculates the respective first and second embeddings from the test and/or training data sets. In other examples, the first and second embeddings can be stored in an embedding database (not in 2 shown) must be filed. The processing unit can also have a second sub-unit 27, which is designed to receive the first and second embeddings and a similarity measure of a first distribution of the embeddings of the first test and/or training data set and a second distribution of the embeddings of the second test data set. and/or training data set (and, if necessary, carry out further steps). The calculated degree of similarity can be output via the output interface 22 .

The division according to 2 is just an example. In general, the present disclosure relates to any system configured to perform the methods according to the present disclosure (eg, a stand-alone system or a distributed system). The system can be implemented in any suitable hardware environment. In some examples, the system includes a memory containing instructions for performing the steps of the methods of the present disclosure and a processor configured to execute the instructions in the memory (a processor and a memory are not limited to a single physical component .

The nature and creation of the embeddings is described in more detail below. 3 FIG. 3 schematically shows a technique 30 for creating embeddings according to the present disclosure.

In some examples, the method may include generating the first testing and/or training data set and the second testing and/or training data set by filtering broader data sets that include the first and second testing and/or training data sets, include. In some examples, filtering may include data from the broader datasets with certain characteristics (e.g., image data with certain content).

In some examples, the vectors of the embeddings may be generated at least in part (i.e., the complete vector or only part of the vector) by processing the respective datum through a computer-based machine learning module. In other words, a piece of data from a test and/or training data set is sent to and processed by a computer-based machine learning module. Values arising during this processing can form the vector of the respective embedding (possibly after appropriate post-processing steps). The computer-based machine learning module for generating the embeddings can correspond to (e.g. be identical to) the computer-based machine learning module for which the first and second test and/or training data sets are intended. In other examples, the computer-based machine learning module for generating the embeddings can be a different computer-based machine learning module than the computer-based machine learning module for which the first and second test and/or training data sets are intended. The embeddings can include activation values of units and/or output values of units of the respective computer-based machine learning module when the respective data item is processed by the computer-based machine learning module (possibly after corresponding post-processing steps). For example, the computer-based machine learning module for generating the embeddings may include a convolutional neural network ("CNN"). The embeddings can then be generated from the data from filters (i.e. units) of the convolutional neural network.

In some examples, the activation values of units and/or output values of units of the respective computer-based machine learning module can be post-processed when processing the respective data and/or the embeddings (eg by means of an activation aggregation and/or normalization). In 3 a post-processing module 34 is shown in which these steps can take place.

Alternatively or additionally, the vectors of the embeddings may be generated at least in part (i.e., the complete vector or only part of the vector) by processing the respective datum by a computer-based module other than a computer-based machine learning module. For example, values from a computer-based edge or corner case detection module may be used to generate the embeddings.

In all examples, a computer-based machine learning module can process the respective datum in the embedding (or other computer-based module) reflect properties inherent in the datum when viewed "through the glasses" of the respective computer-based machine learning module. As a result, a similarity (or difference) calculated using the embeddings can in some cases be more meaningful than similarities (or differences) of the data itself. The similarity measures calculated using the techniques of the present disclosure can therefore in some cases be more meaningful about whether a first data set and a second data set are sufficiently similar to ensure meaningful testing of a trained computer-based machine learning module, for example.

In some examples, the embeddings can also be determined on parts of a datum of the respective test and/or training data set (e.g. on image sections or sub-images in the case of image data or on parts of time series in the case of time series data).

In 3 1, several exemplary computer-based machine learning modules 33 are shown for creating the embeddings 35, 37 (each using only one or a selection of the computer-based machine learning modules for creating the embeddings). The data of the first or second test and/or training data record is fed from a test and/or training database 31 to the respective computer-based machine learning module 33 (step 111 in 1 ). The embeddings 35 (or the elements of their vectors) can be obtained from the respective computer-based machine learning module 33 when the data of the first or second test and/or training data set is processed (e.g. as activation values of units and/or output values of units, possibly . after one or more post-processing steps) (step 112 in 1 ).

In some examples, the computer-based machine learning module 33 for generating the embeddings may be a computer-based machine learning module trained with a data set corresponding to the first and/or second test and/or training data set. The computer-based machine learning module 33 can be trained to solve the same task as the computer-based machine learning module for which the first and/or second test and/or training data sets are intended.

In other examples, the computer-based machine learning module 33 for generating the embeddings can be a computer-based machine learning module that is trained for a subtask of the task for which the computer-based machine learning module for which the first and/or second test and/or training data sets are intended. For example, a computer-based machine learning module may be trained to recognize specific features in the data (wherein the computer-based machine learning module for which the first and/or second test and/or training data sets are intended for recognizing those features and other features and/or solving a task for which recognition of the special features is necessary - in one example the special features may be edges in an image datum).

In other examples, the computer-based machine learning module 33 for creating the embeddings may be (e.g., a copy of) the computer-based machine learning module for which the first and/or second test and/or training data sets are intended.

In still other examples, the computer-based machine learning module may include an autoencoder (e.g., a variational autoencoder). The embeddings can be generated from an intermediate layer of the autoencoder when a respective datum is processed (e.g. a layer that represents a latent representation of an (input) datum).

Alternatively or additionally, in some examples, the embeddings may be generated using a datum itself (i.e., the payload and/or meta-information. In some examples, the datum may go through one or more pre-processing steps.

In a specific example, the data is image data. For example, an image datum can be represented by a number of pixels (e.g. N * M pixels, where N and M are natural numbers, e.g. greater than 50). One or more elements (e.g. C elements, where C is a natural number, e.g. a gray value and/or one or more color values) can now be given for each point. The image can thus be seen as a vector with N*M*C elements. The vector of an embedding can now be E-dimensional (that is, have E elements, where E is a natural number). In some cases, E is less than N* M * C. The vector of an embedding can now be generated by processing the image datum by an image processing computer-based machine learning module (e.g. based on activations and/or outputs from units of the image processing computer-based machine learning module). The image datum is now represented by an E-dimensional vector.

Alternatively or additionally, the vectors of the embeddings 37 can at least partially (ie, the complete vector or only a part of the vector) based on meta information of the respective date of the test and/or training data set. In some examples, the meta information includes human-readable information that describes the particular datum. The meta information can take on discrete or continuous values. In the present disclosure, the meta information is regarded as part of a respective data item. However, this does not mean (although it is possible) that the meta information is necessarily stored together with the useful information of the datum (e.g. in a data structure). Rather, the meta information is only assigned to the date. The same meta information may be associated with multiple pieces of data in some examples. In 3 a database for the meta information 32 is shown.

The meta information can contain, for example, parameter values that characterize the useful information of the respective date. In some examples, one or more properties of the datum payload may be parametrically characterized (e.g., contrast, saturation, brightness, and/or frame rates for image data). The parameter values can relate to parts of the useful information (e.g. a fixed section of an image datum or a semantic region).

Alternatively or additionally, the meta information can include one or more pieces of information that describe a content of the respective date and/or a context. For example, the one or more pieces of information may describe a data collection process (e.g., type of sensor used to collect the date, geographic information, and/or timestamp). In addition or as an alternative, the one or more items of information can describe the content of the respective datum through a classification into two or more classes. Additionally or alternatively, the one or more items of information can describe the content of the respective date by specifying whether a specific content is contained in the useful information of the respective date.

Additionally or alternatively, the meta information can include labels for the useful information of the respective date.

In an example of image data, the meta information can be, for example, a time of day when an image was recorded (e.g. day/night), geographic information on the image datum, domain information on the image datum (e.g. outside/inside), information on the perspective from which the image datum was recorded,
Camera properties and / or specific image content (e.g. existence of specific objects such as pedestrians, cars, ...) include.

In some cases, the vector of an embedding is generated solely based on the useful information of a given datum (e.g. the image data). Alternatively, the vector of an embedding can be generated solely based on the meta information of a given datum. In still other examples, the vector of an embedding 35, 37 may be generated based in part on a particular datum's payload (e.g., the image data) and based in part on a particular datum's meta-information. This type of embedding is also referred to below as combined embedding. In some examples, a combined may be generated by concatenating a first (partial) vector generated based on the payload of a datum and a second (partial) vector generated based on the meta information of a datum.

In an example of image data, as described above, a portion of the embedding generated based on the payload may be represented by an E-dimensional vector (i.e., having E elements, where E is a natural number). Another part of the embedding that is generated based on the meta-information can be represented by an F-dimensional vector (i.e., have F elements, where F is a natural number). The combined vector is created by concatenating the E-dimensional and F-dimensional vectors. For example, the F-dimensional vector may contain a value for the lightness and a value for the saturation of the image datum.

In other examples, a combined embedding may be generated based on each datum's payload (e.g., the image data), but processed in two or more different ways and combined in the combined embedding. For example, a first part of the vector of an embedding can be generated by means of a first computer-based machine learning module (e.g. one of the above computer-based machine learning modules) and a second part of the vector of an embedding can be generated by means of a second computer-based machine learning module (e.g. another of the above computer-based machine learning modules) ( the methods described above can be used).

The use of combined embeddings can (further) increase the significance of the calculated similarity measure in some cases. The meta information can make differences between two datasets more obvious than the (processed) payload alone. Thus, images that are captured during the day can usually be easily distinguished from those that were captured at night by a brightness value or a corresponding label. In contrast, this distinction may be less clear in the data processed by a computer-based machine learning module.

In some examples, the determined embeddings can be subjected to one or more post-processing steps (e.g. in module 36). In some examples, two or more embeddings may be combined (e.g., through an aggregation, averaging, and/or similar operation). The embeddings can be normalized additionally or alternatively.

Additionally or alternatively, in some examples, the embeddings (or the vectors) can be shortened (i.e. a dimensionality of the vectors can be reduced). One or more of the following techniques can be used for this. In some examples, a correlation analysis of the elements of the embeddings can be performed. If two or more elements of the embeddings are correlated more than a certain threshold value, a subset of the correlated elements can be selected as a representative (e.g. one of the elements). The other elements of the subset can be deleted in order to shorten the vectors of the embeddings. Additionally or alternatively, in some examples, a sensitivity analysis may be performed to identify one or more elements of the embedding that have the least impact on a performance of the computer-based machine learning module. These elements can be removed from the embeddings to shorten the embedding vectors.

In some examples, the techniques of the present disclosure also include
Check whether the embeddings meet a predetermined quality criterion (step 113 in 1 ). If the embeddings do not meet the predetermined quality criterion, a warning of incorrect results can be given and/or embeddings that do not meet the predetermined quality criterion can be disregarded when calculating the similarity measure and/or determining the first distribution and the second distribution. In some cases, the testing may include determining a loss value (ie, one loss value per datum) when the data associated with the embeddings is processed by a computer-based machine learning module that is used to at least partially generate the vectors of the embeddings. For example, if the computer-based computing module for which the first and second training and/or test data sets are intended is used to generate the embeddings (or parts thereof), by determining a training loss (when generating embeddings from training data) Data and associated embeddings are identified that are not adequately represented by the computer-based computer module (e.g. a training loss is above a certain threshold). Alternatively, when using an autoencoder to generate the vectors of the embeddings (or parts thereof) by determining a recovery loss, data and associated embeddings can be identified that are not adequately represented by the autoencoder (e.g. a recovery loss of a datum is located above a certain threshold). The method may include removing the identified embeddings and/or providing an indication to a user of a set of the data/embeddings that have been identified.

The embeddings generated using the above methods are stored in an embedding database 38 (from which they can be read, e.g., to calculate a similarity measure according to the methods of the present disclosure).

The measure of similarity calculated and reported using the techniques of the present disclosure may be used in some examples in one or more of the following ways.

In some examples, the calculated measure of similarity can be output to a human-machine interface. A user can thus be enabled to view the results of the methods for checking test and/or training data sets for a computer-based machine learning module according to the present disclosure (and, if necessary, to initiate further measures). In some examples, the method may further include visualizing the first and second distributions (eg, via a human-machine interface, e.g., a graphical user interface). 4 FIG. 400 is a schematic representation of the distributions of embeddings. A first distribution 410 and a second distribution 420 are relatively dissimilar in this example. This will then also be reflected in the calculated similarity measure.

Additionally or alternatively, the method can include recognizing 131 the first and/or second test and/or training data set as valid if the calculated degree of similarity satisfies a predetermined criterion. In other words, the first and/or second test and/or training data record can be validated automatically (ie without user intervention). As above described, the predetermined criterion can result from the similarity measure used. In some examples (e.g. Kullback-Leibler distance or other distance measures) a small value can indicate a high similarity. In this case, the predetermined criterion can define an upper limit value for the degree of similarity. In other examples, a large value may indicate a high level of similarity. In this case, the predetermined criterion can define a lower limit value for the measure of similarity.

If a test and/or training data set has been recognized as valid, the test and/or training data set can be released (again automatically or by a user) for use in testing or training the computer-based machine learning module.

If a test and/or training data set is determined to be invalid, the test and/or training data set may be blocked (again automatically or by a user) from being used to test or train the computer-based machine learning module. Additionally or alternatively, the augmentation procedures described herein may be performed.

The present disclosure also relates to methods for generating an augmented test and/or training data set for a computer-based machine learning module (see step 132 in 1 ). The method includes verifying a testing and/or training dataset using a method according to the present disclosure and augmenting the testing and/or training dataset by adding one or more data to the first dataset and/or the second dataset. In some examples, the adding may increase (eg, in an iterative and optionally automated process) a similarity of the first distribution of embeddings of the first data set and the second distribution of embeddings of the second data set using the measure of similarity. In some examples, the method further includes determining a change in the degree of similarity of the first distribution of the embeddings of the first data set and a second distribution of the embeddings of the second data set after adding data of a predetermined type (eg after applying a concrete augmentation method). If the similarity between the first and second data set increases after adding the data of a predetermined type, the first and/or second data set can be augmented with further data of the predetermined type.

The data of the predetermined type may be generated, in some examples, by translation and/or rotation of payloads within one or more data of the data sets (e.g., by translation and/or rotation of portions of image data). The method can then include checking whether the similarity of the first and second test and/or training data set increases after the insertion of the data generated by translation of useful information. Additionally or alternatively, in some examples, the data of the predetermined type may be "adversarial samples". Additionally or alternatively, in some examples, the data of the predetermined type may be interpolations or compositions of data.

The present disclosure also relates to methods for testing or training a computer-based machine learning module (see step 133 in 1 ). The method includes receiving a verified test and/or training data set according to any one of the present disclosure or an augmented test and/or training data set according to the present disclosure and testing or training the computer-based machine learning module with the received data set to generate a tested or receive a trained machine learning module.

The present disclosure also relates to methods of applying a computer-based machine learning module (see step 134 in 1 ). The method includes providing a tested or trained machine learning module according to the present disclosure and processing application data by the received machine learning module.

As previously discussed, the computer-based machine learning modules can be designed and/or trained for a variety of applications.

In some examples, the computer-based machine learning modules (e.g. the computer-based machine learning modules for which the first and second data sets are intended or the computer-based machine learning modules for generating the embeddings) are designed for image processing. In this example, the first and second test and/or training data sets may include image data (eg, still image data or video data). In some examples, the computer-based machine learning module may be an image classifier (eg, an image classifier that semantically segments image data pixel-by-pixel or region-by-region). Based on the classification result, a device (eg a vehicle, a robot, an industrial plant, a medical device or a household appliance) can be monitored and/or controlled. The image data can be captured using various sensors (e.g. cameras, radar, lidar, ultrasonic or Heat sensors) are generated and / or include synthetic image data.

Additionally or alternatively, the computer-based machine learning module may be configured to detect an anomaly in a data set (e.g., sensor data). In this example, the first and a second test and/or training data set may include data series (e.g., time series). The data series (e.g. time series) can be generated using various sensors (e.g. cameras, radar, lidar, ultrasonic or thermal sensors) and/or can comprise synthetic data series (e.g. time series). In turn, based on the result of the anomaly detection, a device (e.g., a vehicle, a robot, an industrial plant, a medical device, or a household appliance) can be monitored and/or controlled (e.g., brought into a safe state and/or stopped ).

In still other examples, the computer-based machine learning module can be designed to receive environment and/or status data of a device (e.g. sensor data) and based on the environment and/or status data the device (e.g. a vehicle, a robot, an industrial plant, a medical device or a household appliance).

The devices may be systems for vehicles (e.g. systems for assisted or autonomous driving). In this case, the computer-based machine learning modules may be integrated into the vehicle system (e.g., assisted or autonomous driving systems) to provide functionality to the vehicle

The present disclosure also relates to a computer program configured to perform the methods of the present disclosure.

The present disclosure also relates to a signal or computer-readable medium that encodes or contains the computer program according to the present disclosure.

Claims (15)

A computer-implemented method for checking test and/or training data sets for a computer-based machine learning module, comprising: Receiving (121) an embedding (35, 37) for each datum of a first test and/or training data set and a second test and/or training data set for a computer-based machine learning module, wherein an embedding (35, 37) represents a datum according to a specific rule by a vector of lower dimensionality than the respective datum; calculating (123) a similarity measure of a first distribution of the embeddings (35, 37) of the first data set and a second distribution of the embeddings of the second data set; and Outputting (124) the calculated measure of similarity. procedure according to claim 1 , wherein calculating the similarity measure (123) comprises calculating a distance measure of the first and second probability distributions, optionally calculating a Kullback-Leibler distance of the first and second distributions or a distance between the first and second distributions determined using a Wasserstein metric. Method according to one of Claims 1 until 2 , wherein the vectors of the embeddings (35, 37) are generated at least in part by processing the respective datum by a computer-based machine learning module (33). procedure according to claim 3 , wherein the embeddings (35, 37) include activation values of units and/or output values of units of the computer-based machine learning module (33) when the respective data item is processed by the computer-based machine learning module (33). Method according to one of Claims 1 until 4 , wherein the vectors of the embeddings (35, 37) are generated at least partially based on meta information of the respective date of the test and/or training dataset, optionally wherein the meta information includes human-readable information that describes the respective date. Method according to one of Claims 1 until 5 , further comprising: recognizing (131) the first and/or second data set as valid if the calculated degree of similarity satisfies a predetermined criterion. Method according to any of the foregoing Claims 1 until 6 , further comprising determining (122) the embeddings (35, 37) for each datum of the first test and/or training data set and the second test and/or training data set for a computer-based machine learning module. A method for generating an augmented (132) data set for a computer-based machine learning module, comprising: checking a test and/or training data set using a method according to claim 1 until 7 ; Augmenting the test and/or training data set by adding one or more data, so that a similarity of the first distribution of the embeddings (35, 37) of the first test and/or training data set and the second distribution of the embeddings (35, 37) of the second test and/or training data record in relation to the degree of similarity increases. Method for testing or training (133) a computer-based machine learning module: Receiving a checked test and/or training data set according to one of Claims 1 until 7 or according to an augmented test and/or training data set claim 8 ; and testing or training the computer-based machine learning module with the received data set to obtain a tested or trained machine learning module. Method for applying (134) a computer-based machine learning module: providing a tested or trained machine learning module according to claim 9 ; and processing application data by the received machine learning module. Method according to any of the foregoing Claims 1 until 10 , wherein the respective computer-based machine learning module is designed for image processing and wherein the first and a second test and/or training data set contain image data or wherein the computer-based machine learning module is designed for anomaly detection and the first and a second test and/or training data set -Records contain time series data. procedure according to claim 11 , wherein the respective computer-based machine learning module is an image classifier, optionally an image classifier that semantically segments image data pixel by pixel or by region. System (20) designed to implement the methods of any one of Claims 1 until 12 to execute. Computer program designed to perform the procedures of any of Claims 1 until 12 to execute. Signal or computer-readable medium corresponding to the computer program Claim 14 encoded or contained.

Images