DE102021212731A1 - REVIEWING TESTING AND/OR TRAINING RECORDS FOR A COMPUTER-BASED MACHINE LEARNING MODULE - Google Patents

Abstract

A first general aspect of the present disclosure relates to a computer-implemented method for checking test and/or training data sets for a computer-based machine learning module. The method includes receiving a first embedding for a datum of a first testing and/or training dataset for a computer-based machine learning module. An embedding represents a datum according to a specific rule by a vector with a lower dimensionality than the datum in question. The method further includes determining one or more distances of one or more embeddings of a second test and/or training dataset from the first embedding of the first test and/or training dataset and outputting test information for the second test and/or Training data set based on the determined one or more distances.

Description

State of the art

Computer-based machine learning modules - especially those for safety-critical applications such as autonomous or assisted driving - must be extensively tested in many cases (in the present disclosure, the term "testing" includes both a validation in the sense of determining a performance within the framework of a model selection of the computer-based machine learning module as also a check of a generalization error of the chosen model - often referred to as "testing" in a narrower sense). For this purpose, a trained computer-based machine learning module can be loaded with a test data set. The performance and/or reliability of the trained computer-based machine learning module can be determined based on the outputs (e.g. classification results) of the computer-based machine learning module for the data of the test data set. It may be necessary here that the training and test data sets are different but sufficiently similar (e.g. the training and test data sets can be independently and identically distributed and sampled from the same source). If this is not the case, in some cases the performance and/or reliability of the trained computer-based machine learning module cannot be determined with sufficient certainty using the test data set. For example, it is conceivable that a test dataset does not represent certain data of the training dataset or does so insufficiently (i.e. the test dataset does not cover the training dataset sufficiently). In these cases, the performance and/or reliability of the trained computer-based machine learning module in relation to the data of the training dataset that is not represented or is insufficiently represented cannot be reliably determined using the test dataset in some cases.

It is therefore worthwhile to check training and test datasets to see whether they are sufficiently similar and, if necessary, to establish a sufficient similarity between the training and test datasets.

Summary

A first general aspect of the present disclosure relates to a computer-implemented method for checking test and/or training data sets for a computer-based machine learning module. The method includes receiving a first embedding for a datum of a first testing and/or training dataset (also referred to as “first dataset” for short) for a computer-based machine learning module. An embedding represents a datum according to a specific rule by a vector with a lower dimensionality than the datum in question. The method further includes determining one or more distances of one or more embeddings of a second test and/or training data set (also referred to as “second data set” for short) from the first embedding of the first test and/or training data set and outputting of test information for the second test and/or training data set based on the determined one or more distances.

A second general aspect of the present disclosure relates to a method for generating an augmented data set for a computer-based machine learning module. The method includes determining that one or more training data is underrepresented in the test data set based on the verification information and/or determining that one or more test data is underrepresented in the training data set based on the verification information according to the first general aspect. The method further includes augmenting the second test and/or training dataset by adding one or more data such that one or more distances of embeddings of the second test and/or training dataset from the first embedding and optionally the one or more further embeddings of the first test and/or training dataset can be reduced.

A third general aspect of the present disclosure relates to a method for testing or training a computer-based machine learning module. The method includes receiving a verified test and/or training data set according to the first general aspect or an augmented test and/or training data set according to the second general aspect and testing or training the computer-based machine learning module with the received data set to generate a tested or receive a trained machine learning module.

A fourth general aspect of the present disclosure relates to a method for applying a computer-based machine learning module. The method comprises providing a tested or trained machine learning module according to the third general aspect and processing application data by the received machine learning module.

A fifth general aspect of the present disclosure relates to a system configured to carry out the methods of any one of the first to fourth general aspects.

The techniques of the first through fifth general aspects may have one or more advantages in some cases.

First, in some cases, the techniques of the present disclosure can be used to verify whether data from a test data set is adequately represented in a training data set (or vice versa) such that a computer-based machine learning module trained with the training data set can use the test with sufficient reliability -Record can be tested. It can thus be ensured that the training and/or test adequately prepare the computer-based machine learning module with regard to its tasks in the field. By determining distances between the embeddings of the two training datasets, "missing spots" in the second dataset (e.g. a test dataset or a training dataset) can be efficiently detected. For example, data in certain areas in the training data set or in the test data set may be missing or insufficient (e.g. image data of certain signs at night and in the rain). In some cases, these and other problems can be detected by analyzing the distances between the embeddings (also relatively quickly in some situations). On the other hand, by analyzing the distances, it can be recognized that certain data in a test and training data set is too similar (this too can be problematic in terms of reliable testing of the computer-based machine learning module). In some cases, this can prevent insufficiently trained and/or tested computer-based machine learning modules from being used in the field and possibly not functioning satisfactorily there (which, depending on the technical environment, can have various consequences, from annoyances to significant security risks).

Second, the embeddings of the techniques of the present disclosure can be chosen flexibly and have more meaningfulness than other information in determining the distances of the data of the data sets. For example, it is possible that the vector of the embedding is generated both based on the useful information of a datum (e.g. the image data) and meta information of the datum (e.g. human-readable labels). In addition, customized embeddings can be created and used for various questions. For example, the embedding can contain a representation from an intermediate layer of a computer-based machine learning module or be determined from this. This can be helpful as the differences in the vectors of the embeddings can reflect differences in the information processed by the computer-based machine learning modules. Calculating distances based on this information can be more meaningful in determining whether, for example, the data from a training dataset is adequately represented in a test dataset than distances or other comparisons of the data themselves. In addition, in some cases, processing the lower-dimensional Vectors can be significantly faster and/or more resource-efficient than processing the data itself. In some situations, the embedding can be orders of magnitude smaller than the data itself, which can correspondingly speed up its processing and/or make it resource-efficient. In some cases, both factors can interact so that a meaningful and efficient assessment of mutual coverage of two data sets becomes possible.

Thirdly, using a data spacing determined on the distribution of embeddings, the data sets can be augmented in order to avoid problems that may be caused by insufficient coverage (e.g. gaps in the data of a data set). Again, this can be facilitated in some cases by using the embeddings, because in some situations the embeddings can reflect features relevant to the performance of the computer-based machine learning module better and/or more clearly than the data itself. The insufficient coverage in space of the embeddings can thus be explained in In some cases, be more predictive of the presence of a problem than coverage determined from the data itself.

Some terms are used in a specific manner in this disclosure.

A “computer-based machine learning module” in the present disclosure is any device that can be or has been trained to perform one or more tasks using machine learning. As part of a training, the computer-based machine learning module can be supplied with training data as input data and the properties of the computer-based machine learning module can be adjusted in response to the processing of the training data (e.g. by analyzing the output data) in order to perform the one or more tasks in a defined way (e.g. with a certain accuracy). A computer-based machine learning module can contain a model that determines the form of processing the respective input data into output data. The model may be parameterized, where the parameters of the computer-based machine learning module are fixed during learning. Computer-based machine learning modules can include (artificial) neural networks (e.g. convolutional and/or recurrent and/or deep neural networks - the former are commonly referred to as CNNs and RNNs referred to), but are not limited to these. Computer-based machine learning modules can be set up (and possibly trained) for various tasks. For example, a computer-based machine learning module can perform a classification task or a regression task. Computer-based machine learning modules can be implemented in any suitable form, ie, in software, in dedicated hardware, or in a hybrid of software and dedicated hardware. Therefore, a computer-based machine learning module can be a software module (also integrated into a higher-level software system) that can be executed on a general-purpose processor. In other cases, a computer-based machine learning module may be implemented (at least in part) in circuitry.

In the present disclosure, an “embedding” represents a datum according to a specific regulation by a vector of lower dimensionality than the respective datum. An embedding is therefore a representation of the respective date in a space with a lower dimensionality than a space in which the data itself can be represented. The term embedding states that the respective datum is embedded in the space of lower dimensionality, i.e. it is transformed or projected into this space by means of the specific rule. No other requirements are directed to the specific regulation. In some examples, the particular rule is processing a datum with a computer-based machine learning module, wherein the embeddings are read from the computer-based machine learning module (e.g., as a plurality of activations and/or outputs from units of the computer-based machine learning module). For example, an image datum containing 1024x2048 pixels, each pixel having three color values with a color depth of 16 bits, can be represented by an embedding with 100 elements (e.g. the activation values at a specific point in a neural network). Additional aspects of the nature and creation of the embeddings are discussed below.

The term "vector" in the present disclosure describes an ordered sequence of elements (e.g. ten or more elements, or fifty or more elements, or 100 or more elements). For example, an element can be a real number, an integer, or a natural number. A vector can be displayed and/or saved in various forms. For example, the entries in a matrix with multiple rows and columns are a vector because the contents contain an ordered sequence of elements. The same applies to a list of tuples (e.g. that assigns a rank number or an identifier to an element).

A "test data set" according to the present disclosure is a collection of a plurality of data suitable and selected for testing a computer-based machine learning module. As mentioned, the elements of the dataset are referred to as data (singular date). Depending on the nature of the computer-based machine learning module to be tested, the data in a test data set can contain one or more labels. Each datum of a test data set with image data can contain a single image (or a section from it) or a video. Meta information and other information associated with the individual datum (e.g. labels) are referred to as part of the datum. The part of the datum that is processed by the computer-based machine learning module as part of its execution is referred to as useful information (e.g. the image information in an image datum).

According to the present disclosure, a “training dataset” is a collection of a plurality of data suitable and selected for training a computer-based machine learning module. Incidentally, what was said for test data sets also applies to the training data set.

In the present disclosure, the term "database" encompasses any system in which data (e.g. data of a test and/or training data set) can be stored and made available. A database can have any physical implementation (e.g. distributed across multiple locations).

character list

• 1 FIG. 12 is a flow chart illustrating the methods of the present disclosure.
• 2 FIG. 12 schematically shows a system for performing the techniques of the present disclosure.
• 3 FIG. 12 schematically shows a technique for creating embeddings according to the present disclosure.
• 4 12 is a schematic representation of the distances between embeddings of the first dataset and embeddings of the second dataset.

Detailed description

First, based on 1 and 2 an overview of the techniques of the present disclosure is provided. After that, the creation of the embeddings in relation to 3 explained in more detail. Finally follows with the help of 4 a discussion of techniques for using the techniques obtained using the techniques of the present disclosure test information. Finally, some concrete applications of the computer-based machine learning modules of the present disclosure are presented.

1 FIG. 12 is a flow chart illustrating the methods of the present disclosure. 1 shows in three different columns a method for generating the embeddings 110 (left), a method for checking test and/or training data sets for a computer-based machine learning module 120 (middle) and a method for using the determined test information 130 (right).

A computer-implemented method for checking test and/or training data sets for a computer-based machine learning module. The method includes receiving 121 a first embedding for a datum of a first testing and/or training dataset (also referred to as “first dataset” for short) for a computer-based machine learning module. An embedding represents a datum according to a specific rule by a vector with a lower dimensionality than the datum in question. In some examples, the first embedding can be selected by a user.

The method further includes determining 122 determining one or more distances of one or more embeddings of a second test and/or training dataset (also referred to as “second dataset” for short) from the first embedding of the first test and/or training dataset .

In some examples, determining one or more distances includes using a data model for the embeddings of the second testing and/or training dataset to determine the one or more distances from the first embedding of the first testing and/or training dataset. The data model can be a knn graph, k-nearest-neighbors graph, or any other data model that is suitable for determining the required distances between embeddings of the first and second datasets (and possibly a number of embeddings of the second dataset in an environment of the first embedding).

Determining the one or more distances may include an analysis of a neighborhood of the first embedding (i.e., in a sub-space of the embedding's space with predetermined maximum extents around the first embedding) in the embedding's space of the second data set. For example, a density of embeddings in the vicinity of the first embedding can be determined. The density may be smaller or larger depending on a number of embeddings of the second dataset in the vicinity of the first dataset (e.g. a density may be zero if no embeddings of the second dataset are found in the vicinity of the first dataset). As can be seen, an analysis of the environment of the first embedding (e.g. a density of embeddings) corresponds to determining one or more distances from embeddings of the second data set, since the distances between the embeddings of the second data set are determined at least implicitly. If, for example, it is recognized that a density of embeddings of the second dataset in the vicinity of the first embedding has a specific value, this means that the corresponding distances between the embeddings of the second dataset assume specific values. The operation of "determining the one or more distances" is therefore not limited to the explicit calculation of a distance measure.

Accordingly, a distance between two embeddings is (only) in some examples an explicit distance measure determined using a suitable distance metric. For example, distances between the vectors of the two embeddings can be calculated. In other examples, a distance is determined implicitly when calculating another quantity (e.g. density of embeddings). A distance can be post-processed, e.g. by aggregating several distances into several embeddings of the second data set.

In some examples, the one or more embeddings of the second testing and/or training dataset may be the nearest neighbors of the first embedding of the first testing and/or training dataset in the second testing and/or training dataset (i.e., in the room of the embeddings). In other words, the method includes determining one or more nearest neighbors (e.g., one nearest neighbor, five or more nearest neighbors, or ten or more nearest neighbors). Alternatively or additionally, the one or more embeddings of the second test and/or training dataset can be the embeddings that are in a specific environment (i.e., in a sub-space of the space of the embedding with predetermined maximum extents around the first embedding) of the first embeddings lie. In this case too, neighbors of the first embedding are determined under the second embeddings. A number of nearest neighbors among the embeddings of the second dataset whose distances to the first are determined may be user-defined in some examples.

The method also includes outputting 123 test information for the second test and/or training data set based on the determined ten one or more distances. The test information can include the one or more distances (or a characteristic value calculated therefrom) (possibly after one or more processing steps). For example, the check information can include an average value, a minimum value and/or a maximum value of the distances of the one or more embeddings of the second data set from the embedding of the first data set. In some examples, the test information includes a maximum distance of the one or more embeddings of the second test and/or training data set from the first embedding of the first test and/or training data set (e.g. a maximum distance of one or more nearest neighbors of the first embeddings under the embeddings of the second record). Additionally or alternatively, in some examples, the test information may include a minimum distance of the one or more embeddings of the second test and/or training data set from the first embedding of the first test and/or training data set (e.g. a minimum distance of one or several nearest neighbors of the first embedding among the embeddings of the second dataset). The check information can additionally or alternatively also include information about a density of embeddings of the second data set in an area surrounding the first embedding.

The verification information can take various formats. A format of the check information can depend on how the check information is to be used further. In some examples, the verification information is provided to a user. In this case, the check information can be output in a human-readable format. In other examples, the verification information can be processed in an automated manner. In this case, the test information can be output in a machine-readable format.

In some examples, the method includes creating a data model from the embeddings of the second test and/or training data set (e.g., a knn graph, k-nearest neighbors graph) suitable for determining the one or more distances.

In some examples, the first testing and/or training dataset is a training dataset and the second testing and/or training dataset is a testing dataset. However, it is also conceivable that both the first and the second test and/or training data record are a training data record or that both the first and the second test and/or training data record are a test data record.

In some examples, the first data set is a training data set for a computer-based machine learning module and the second data set is a testing data set for a computer-based machine learning module.

In these examples, the method may further include determining 131 that one or more test data is not adequately represented in the training data set based on the test information. In other words, the test information can be used to determine whether data is missing in the training data set. In some cases, the determining may include analyzing distance information contained in the verification information. For example, it can be checked whether one or more distances (e.g. a maximum distance or an average distance) meet a predetermined criterion. The predetermined criterion may be that the one or more distances (e.g., a maximum distance or an average distance) are greater than a certain threshold. For example, it may be determined whether a maximum distance from one or more nearest neighbors of the first embedding is greater than a certain threshold. In other examples, the check information may include information about a density of embeddings of the second dataset in a neighborhood of the first embedding. In these examples, the predetermined criterion may be that a density of embeddings of the second dataset in a vicinity of the first embedding is lower than a predetermined density. If the one or more distances and/or the density of embeddings of the second dataset in a vicinity of the first embedding meet the predetermined criterion, it can be determined that data is missing in the training dataset.

Additionally or alternatively, the method may further include determining 131 that one or more test data is too similar to the data of the training data set based on the test information. In some cases, the determining may include analyzing distance information contained in the verification information. For example, it can be checked whether one or more distances (eg a minimum distance or an average distance) meet a predetermined criterion. The predetermined criterion may be that the one or more distances (eg, a minimum distance or an average distance) are less than a certain threshold. For example, it can be determined whether a minimum distance from one or more nearest neighbors of the first embedding is less than a certain threshold. In other examples, the check information may include information about a density of embeddings of the second dataset in a neighborhood of the first embedding. In these examples, the predetermined criterion may be that a density of embeddings of the second dataset in a vicinity of the first embedding is higher than a predetermined density. If the one or more distances and/or the density of embeddings of the second dataset in a neighborhood of the first embedding meet the predetermined criterion, it can be determined that one or more test datasets are too similar to the data of the training dataset.

In other examples, the first data set is a test data set for a computer-based machine learning module and the second data set is a training data set for a computer-based machine learning module. In these examples, the method may further include determining 131 that one or more training data sets are not adequately represented in the test data set based on the testing information. In other words, the test information can be used to determine whether data is missing in the test data set. The options for designing the determination discussed in the last paragraph can be used in this case. The methods of determining 131 too much similarity based on the check information can also be used.

In some examples, the distances can be approximated (e.g., a dimensionality of the embeddings can be reduced and/or a distribution of the embeddings can be approximated).

In some examples, the method also includes receiving one or more further embeddings for data of the first test and/or training data set, determining one or more further distances of one or more embeddings of the second test and/or training data set from the one or several further embeddings of the first test and/or training data set outputting test information for the second test and/or training data set based on the determined one or more further distances. In some examples, distances for a plurality (e.g. for all) embeddings of the first test and/or training data set can be determined from corresponding embeddings of the second test and/or training data set and the test information can be output based on the distances determined in this way become.

In some examples, the method may include generating 110 the embeddings for the first datum and possibly further embeddings of the first test and/or training data set and/or the embeddings of the second test and/or training data set (more on this in relation to on 3 ). In other examples, the embeddings may be read from an embedding database (ie, the embeddings were previously created and stored in the database for later use).

2 FIG. 1 schematically shows a system 20 for performing the techniques of the present disclosure. In this example, the system 20 is divided into a processing unit 21 and an output interface 22 .

The data of the first test and/or training data record can be stored in a first database 23 . The data of the second test and/or training data record can be stored in a second database 24 . A processing unit 25 is coupled to the first and second databases 23,24. The processing unit can have a first sub-unit 26, which calculates the respective first and second embeddings from the test and/or training data sets. In other examples, the first and second embeddings can be stored in an embedding database (not in 2 shown) must be filed. The processing unit may also have a second sub-unit 27 configured to receive the first and second embeddings and one or more offsets of the first embedding of the first test and/or training dataset from embeddings of the second test and/or training dataset -Calculate the data set (and carry out further steps if necessary). An item of test information that has been determined can be output via the output interface 22 .

The division according to 2 is just an example. In general, the present disclosure relates to any system configured to perform the methods according to the present disclosure (eg, a stand-alone system or a distributed system). The system can be implemented in any suitable hardware environment. In some examples, the system includes a memory containing instructions for performing the steps of the methods of the present disclosure and a processor configured to execute the instructions in the memory (a processor and a memory are not limited to a single physical component .

The nature and creation of the embeddings is described in more detail below. 3 FIG. 3 schematically shows a technique 30 for creating embeddings according to the present disclosure.

In some examples, the method may include generating the first testing and/or training dataset and the second testing and/or training dataset by filtering larger Data sets containing the first and second test and/or training data sets include. In some examples, the filtering may include data from the broader datasets with certain characteristics (eg, image data with certain content).

In some examples, the vectors of the embeddings may be generated at least in part (i.e., the complete vector or only part of the vector) by processing the respective datum through a computer-based machine learning module. In other words, a piece of data from a test and/or training data set is sent to and processed by a computer-based machine learning module. Values arising during this processing can form the vector of the respective embedding (possibly after appropriate post-processing steps). The computer-based machine learning module for generating the embeddings can correspond to (e.g. be identical to) the computer-based machine learning module for which the first and second test and/or training data sets are intended. In other examples, the computer-based machine learning module for generating the embeddings can be a different computer-based machine learning module than the computer-based machine learning module for which the first and second test and/or training data sets are intended. The embeddings can include activation values of units and/or output values of units of the respective computer-based machine learning module when the respective data item is processed by the computer-based machine learning module (possibly after corresponding post-processing steps). For example, the computer-based machine learning module for generating the embeddings may include a convolutional neural network ("CNN"). The embeddings can then be generated from the data from filters (i.e. units) of the convolutional neural network.

In some examples, the activation values of units and/or output values of units of the respective computer-based machine learning module can be post-processed when processing the respective data and/or the embeddings (eg by means of an activation aggregation and/or normalization). In 3 a post-processing module 34 is shown in which these steps can take place.

Alternatively or additionally, the vectors of the embeddings may be generated at least in part (i.e., the complete vector or only part of the vector) by processing the respective datum by a computer-based module other than a computer-based machine learning module. For example, values from a computer-based edge or corner case detection module may be used to generate the embeddings.

In all examples, the processing of the particular datum by a computer-based machine learning module (or another computer-based module) may reflect in the embedding properties inherent in the datum when viewed "through the glasses" of the particular computer-based machine learning module. As a result, a distance (i.e., difference) calculated using the embeddings can in some cases be more meaningful than distances (or differences) of the data itself. The distance measures calculated using the techniques of the present disclosure can therefore in some cases be more meaningful about whether an embedding and thus a date of the first data set is sufficiently represented in the second data set, e.g. to ensure meaningful testing of a trained computer-based machine learning module.

In some examples, the embeddings can also be determined on parts of a datum of the respective test and/or training data set (e.g. on image sections or sub-images in the case of image data or on parts of time series in the case of time series data).

In 3 1, several exemplary computer-based machine learning modules 33 are shown for creating the embeddings 35, 37 (each using only one or a selection of the computer-based machine learning modules for creating the embeddings). The data of the first or second test and/or training data record is fed from a test and/or training database 31 to the respective computer-based machine learning module 33 (step 111 in 1 ). The embeddings 35 (or the elements of their vectors) can be obtained from the respective computer-based machine learning module 33 when the data of the first or second test and/or training data set is processed (e.g. as activation values of units and/or output values of units, possibly . after one or more post-processing steps) (step 112 in 1 ).

In some examples, the computer-based machine learning module 33 for generating the embeddings may be a computer-based machine learning module trained with a data set corresponding to the first and/or second test and/or training data set. The computer-based machine learning module 33 can be trained to solve the same task as the computer-based machine learning module for which the first and/or second test and/or training data sets are determined.

In other examples, the computer-based machine learning module 33 for generating the embeddings can be a computer-based machine learning module that is trained for a subtask of the task for which the computer-based machine learning module for which the first and/or second test and/or training data sets are intended. For example, a computer-based machine learning module may be trained to recognize specific features in the data (wherein the computer-based machine learning module for which the first and/or second test and/or training data sets are intended for recognizing those features and other features and/or solving a task for which recognition of the special features is necessary).

In other examples, the computer-based machine learning module 33 for creating the embeddings may be (e.g., a copy of) the computer-based machine learning module for which the first and/or second test and/or training data sets are intended.

In still other examples, the computer-based machine learning module may include an autoencoder (e.g., a variational autoencoder). The embeddings can be generated from an intermediate layer of the autoencoder when a respective datum is processed (e.g. a layer that represents a latent representation of an (input) datum).

Alternatively or additionally, in some examples, the embeddings may be generated using a datum itself (i.e., the payload and/or meta-information). In some examples, the datum may go through one or more pre-processing steps.

In a specific example, the data is image data. For example, an image datum can be represented by a number of pixels (e.g. N * M pixels, where N and M are natural numbers, e.g. greater than 50). One or more elements (e.g. C elements, where C is a natural number, e.g. a gray value and/or one or more color values) can now be given for each point. The image can thus be seen as a vector with N*M*C elements. The vector of an embedding can now be E-dimensional (that is, have E elements, where E is a natural number). In some cases, E is less than N* M * C. The vector of an embedding can now be generated by processing the image datum by an image processing computer-based machine learning module (e.g. based on activations and/or outputs from units of the image processing computer-based machine learning module). The image datum is now represented by an E-dimensional vector.

Alternatively or additionally, the vectors of the embeddings 37 can be generated at least partially (ie the complete vector or only part of the vector) based on meta information of the respective date of the test and/or training data set. In some examples, the meta information includes human-readable information that describes the particular datum. The meta information can take on discrete or continuous values. In the present disclosure, the meta information is regarded as part of a respective data item. However, this does not mean (although it is possible) that the meta information is necessarily stored together with the useful information of the datum (e.g. in a data structure). Rather, the meta information is only assigned to the date. The same meta information may be associated with multiple pieces of data in some examples. In 3 a database for the meta information 32 is shown.

The meta information can contain, for example, parameter values that characterize the useful information of the respective date. In some examples, one or more properties of the datum payload may be parametrically characterized (e.g., contrast, saturation, brightness, and/or frame rates for image data). The parameter values can relate to parts of the useful information (e.g. a fixed section of an image datum or a semantic region).

Alternatively or additionally, the meta information can include one or more pieces of information that describe a content of the respective date and/or a context. For example, the one or more pieces of information may describe a data collection process (e.g., type of sensor used to collect the date, geographic information, and/or timestamp). In addition or as an alternative, the one or more items of information can describe the content of the respective datum through a classification into two or more classes. Additionally or alternatively, the one or more items of information can describe the content of the respective date by specifying whether a specific content is contained in the useful information of the respective date.

Additionally or alternatively, the meta information can include labels for the useful information of the respective date.

In an example of image data, the meta information can be, for example, a time of day of an on Taking a picture datum (e.g. day/night) Geographical information about the picture datum Domains Information about the picture datum (e.g. outside/inside) Information about the perspective from which the picture datum was taken
Camera properties and / or specific image content (e.g. existence of specific objects such as pedestrians, cars, ...) include.

In some cases, the vector of an embedding is generated solely based on the useful information of a given datum (e.g. the image data). Alternatively, the vector of an embedding can be generated solely based on the meta information of a given datum. In still other examples, the vector of an embedding 35, 37 may be generated based in part on a particular datum's payload (e.g., the image data) and based in part on a particular datum's meta-information. This type of embedding is also referred to below as combined embedding. In some examples, a combined may be generated by concatenating a first (partial) vector generated based on the payload of a datum and a second (partial) vector generated based on the meta information of a datum.

In an example of image data, as described above, a portion of the embedding generated based on the payload may be represented by an E-dimensional vector (i.e., having E elements, where E is a natural number). Another part of the embedding that is generated based on the meta-information can be represented by an F-dimensional vector (i.e., have F elements, where F is a natural number). The combined vector is created by concatenating the E-dimensional and F-dimensional vectors. For example, the F-dimensional vector may contain a value for the lightness and a value for the saturation of the image datum.

In other examples, a combined embedding may be generated based on each datum's payload (e.g., the image data), but processed in two or more different ways and combined in the combined embedding. For example, a first part of the vector of an embedding can be generated by means of a first computer-based machine learning module (e.g. one of the above computer-based machine learning modules) and a second part of the vector of an embedding can be generated by means of a second computer-based machine learning module (e.g. another of the above computer-based machine learning modules) ( the methods described above can be used).

The use of combined embeddings can (further) increase the significance of the determined distances in some cases. The meta information can reveal differences between two data sets more clearly than the (processed) useful information alone. Thus, images that are captured during the day can usually be easily distinguished from those that were captured at night by a brightness value or a corresponding label. In contrast, this distinction may be less clear in the data processed by a computer-based machine learning module.

In some examples, the determined embeddings can be subjected to one or more post-processing steps (e.g. in module 36). In some examples, two or more embeddings may be combined (e.g., through an aggregation, averaging, and/or similar operation). The embeddings can be normalized additionally or alternatively.

Additionally or alternatively, in some examples, the embeddings (or the vectors) can be shortened (i.e. a dimensionality of the vectors can be reduced). One or more of the following techniques can be used for this. In some examples, a correlation analysis of the elements of the embeddings can be performed. If two or more elements of the embeddings are correlated more than a certain threshold value, a subset of the correlated elements can be selected as a representative (e.g. one of the elements). The other elements of the subset can be deleted in order to shorten the vectors of the embeddings. Additionally or alternatively, in some examples, a sensitivity analysis may be performed to identify one or more elements of the embedding that have the least impact on a performance of the computer-based machine learning module. These elements can be removed from the embeddings to shorten the embedding vectors.

In some examples, the techniques of the present disclosure also include
Check whether the embeddings meet a predetermined quality criterion (step 113 in 1 ). If the embeddings do not meet the predetermined quality criterion, a warning of incorrect results can be given and/or embeddings that do not meet the predetermined quality criterion can be disregarded when calculating the distances and/or creating the data model for the second dataset. In some cases, the checking can include determining a loss value in processing the data associated with the embeddings through computer-based machine learning module with which the vectors of the embeddings are at least partially generated (ie one loss value per data item). For example, if the computer-based computing module for which the first and second training and/or test data sets are intended is used to generate the embeddings (or parts thereof), by determining a training loss (when generating embeddings from training data) Data and associated embeddings are identified that are not adequately represented by the computer-based computer module (e.g. a training loss is above a certain threshold). Alternatively, when using an autoencoder to generate the vectors of the embeddings (or parts thereof) by determining a recovery loss, data and associated embeddings can be identified that are not adequately represented by the autoencoder (e.g. a recovery loss of a datum is located above a certain threshold). The method may include removing the identified embeddings and/or providing an indication to a user of a set of the data/embeddings that have been identified.

The embeddings generated using the above methods are stored in an embedding database 38 (from which they can be read, e.g., to determine distances and/or create a data model according to the methods of the present disclosure).

The audit information determined and output using the techniques of the present disclosure may, in some examples, be used in one or more of the following ways.

In some examples, the determined test information can be output to a human-machine interface. A user can thus be enabled to view the results of the methods for checking test and/or training data sets for a computer-based machine learning module according to the present disclosure (and, if necessary, to initiate further measures). In some examples, the verification information can also include visualizations of the distances between embeddings of the first and second data sets (eg via a human-machine interface, eg a graphical user interface). 4 12 is a schematic representation of verification information 400 in the form of distributions of distances (eg, maximum distances). A first part 410 of the distribution can contain embeddings (and thus data) of the first test and/or training data set, for which embeddings with smaller distances exist under the embeddings of the second test and/or training data set (number of on the plotted on the y-axis and maximum distance plotted on the x-axis). These first embeddings can be recognized as sufficiently represented in the second test and/or training data set. A second part 420 of the distribution may contain embeddings (and thus data) of the first test and/or training data set, to which embeddings with higher distances exist under the embeddings of the second test and/or training data set. These first embeddings can be identified as being insufficiently represented in the second test and/or training data set.

Additionally or alternatively, the method can include recognizing the second test and/or training data set as valid 124 if the test information (e.g. the distances and/or the density) meets a predetermined criterion (e.g. if it has been determined as above that in the second test and/or training data set is missing data). In other words, the second test and/or training data set can be validated automatically (i.e. without user intervention). As described above, the predetermined criterion can result from the test information used. In some examples (e.g. when using a distance measure), a larger value may signal a poorer representation of the first embedding (and thus the associated datum) in the second record. In this case, the predetermined criterion can define an upper limit value. In other examples, a large value may signal a better representation of the first embedding (and thus its datum) in the second record. In this case, the predetermined criterion can define a lower limit value. If a test and/or training data set has been recognized as valid, the test and/or training data set can be released (again automatically or by a user) for use in testing or training the computer-based machine learning module.

If a test and/or training data set is determined to be invalid, the test and/or training data set may be blocked (again automatically or by a user) from being used to test or train the computer-based machine learning module. Additionally or alternatively, the augmentation procedures described herein may be performed.

The present disclosure also relates to methods for generating an augmented test and/or training data set for a computer-based machine learning module (see step 132 in 1 ). The method includes examining a test and/or training data set using a method according to the present disclosure and augmenting the test and/or training data nings dataset by adding one or more data to the second dataset that is not adequately represented in the test dataset or the training dataset. In some examples, adding the one or more distances of embeddings of the second test and/or training data set from the first embedding and optionally the one or more further embeddings of the first test and/or training data set can reduce. (e.g. in an iterative and optionally automated process). In some examples, the method further includes determining a change in the distances of the first embeddings of the first dataset from the second embeddings of the second dataset upon adding data of a predetermined type to the second dataset. If the distance between embeddings of the first and second data set decreases after adding the data of a predetermined type, the second data set can be augmented with further data of the predetermined type.

The data of the predetermined type may be generated, in some examples, by translating payloads within one or more data of the data sets (e.g., by translating portions of image data). The method can then include checking whether the distances between embeddings of the first and second test and/or training data sets decrease after the insertion of the data generated by translation of useful information. Additionally or alternatively, in some examples, the data of the predetermined type may be "adversarial samples".

Additionally or alternatively, the data of the predetermined type may be generated (e.g. by interpolation) from data whose embeddings in the second data set are neighbors of the first embedding that is not sufficiently represented in the test data set or the training data set.

The present disclosure also relates to methods for testing or training a computer-based machine learning module (see step 133 in 1 ). The method includes receiving a verified test and/or training data set according to any one of the present disclosure or an augmented test and/or training data set according to the present disclosure and testing or training the computer-based machine learning module with the received data set to generate a tested or receive a trained machine learning module.

The present disclosure also relates to methods of applying a computer-based machine learning module (see step 134 in 1 ). The method includes providing a tested or trained machine learning module according to the present disclosure and processing application data by the received machine learning module.

As previously discussed, the computer-based machine learning modules can be designed and/or trained for a variety of applications.

In some examples, the computer-based machine learning modules (e.g. the computer-based machine learning modules for which the first and second data sets are intended or the computer-based machine learning modules for generating the embeddings) are designed for image processing. In this example, the first and a second test and/or training data set may include image data (e.g., still image data or video data). In some examples, the computer-based machine learning module may be an image classifier (e.g., an image classifier that semantically segments image data pixel-by-pixel or region-by-region). Based on the classification result, a device (e.g. a vehicle, a robot, an industrial plant, a medical device or a household appliance) can be monitored and/or controlled. The image data can be generated using various sensors (e.g. cameras, radar, lidar, ultrasonic or thermal sensors) and/or can comprise synthetic image data.

Additionally or alternatively, the computer-based machine learning module may be configured to detect an anomaly in a data set (e.g., sensor data). In this example, the first and a second test and/or training data set may include data series (e.g., time series). The data series (e.g. time series) can be generated using various sensors (e.g. cameras, radar, lidar, ultrasonic or thermal sensors) and/or can comprise synthetic data series (e.g. time series). In turn, based on the result of the anomaly detection, a device (e.g., a vehicle, a robot, an industrial plant, a medical device, or a household appliance) can be monitored and/or controlled (e.g., brought into a safe state and/or stopped ).

In still other examples, the computer-based machine learning module can be designed to receive environment and/or status data of a device (e.g. sensor data) and based on the environment and/or status data the device (e.g. a vehicle, a robot, an industrial plant, a medical device or a household appliance).

The devices may be systems for vehicles (e.g. systems for assisted or auto nome driving). In this case, the computer-based machine learning modules can be integrated into the system for vehicles (eg, the systems for assisted or autonomous driving) to provide functionality for the vehicle

The present disclosure also relates to a computer program configured to perform the methods of the present disclosure.

The present disclosure also relates to a signal or computer-readable medium that encodes or contains the computer program according to the present disclosure.

Claims (16)

A computer-implemented method for checking test and/or training data sets for a computer-based machine learning module, comprising: Receiving (121) a first embedding (35, 37) for a datum of a first test and/or training data set for a computer-based machine learning module, where an embedding represents a datum according to a specific rule by a vector with a lower dimensionality than the respective datum; determining (122) one or more distances of one or more embeddings (35, 37) of a second test and/or training data set from the first embedding of the first test and/or training data set; and Outputting (123) of test information (400) for the second test and/or training data set based on the determined one or more distances. Method according to any of the foregoing claim 1 , wherein determining one or more distances comprises: using a data model for the embeddings (35, 37) of the second test and/or training data set in order to calculate the one or more distances from the first embedding (35, 37) of the first test and/or training data set, optionally wherein the data model is a knn graph, k-nearest-neighbors graph. Method according to any of the foregoing Claims 1 until 2 , wherein the test information (400) based on a maximum or a minimum distance of multiple embeddings (35, 37) of the second test and / or training data set from the first embedding (35, 37) of the first test and / or training data record is determined. Method according to any of the foregoing Claims 1 until 3 , wherein the one or more embeddings (35, 37) of the second test and/or training data set are the nearest neighbors of the first embedding (35, 37) of the first test and/or training data set in the second test and/or or training data set. Method according to one of Claims 1 until 4 , wherein the vectors of the embeddings (35, 37) are generated at least in part by processing the respective datum by a computer-based machine learning module (33). procedure according to claim 5 wherein the embeddings (35, 37) comprise activation values of units and/or output values of units of the computer-based machine learning module (33) when the respective data item is processed by the computer-based machine learning module (33). Method according to one of Claims 1 until 6 , wherein the vectors of the embeddings (35, 37) are generated at least partially based on meta information of the respective date of the test and/or training dataset, optionally wherein the meta information includes human-readable information that describes the respective date. Method according to one of Claims 1 until 7 , wherein the first data set is a test data set for a computer-based machine learning module and wherein the second data set is a training data set for a computer-based machine learning module, the method further comprising: determining (131) that one or more training data sets are insufficient in are represented in the test data set based on the test information (400), or wherein the first data set is a training data set for a computer-based machine learning module and the second data set is a test data set for a computer-based machine learning module, the method further comprising: ascertaining (131) that one or more test data are not sufficiently represented in the training data set based on the test information (400). A method for generating an augmented (132) data set for a computer-based machine learning module, comprising: determining (131) that one or more training data sets are not adequately represented in the test data set based on the test information and/or determining (131) that one or more test data are not adequately represented in the training data set based on the test information according to claim 8 ; Augmenting the second test and/or training data set by adding one or more data that is not sufficiently represented in the test data set or the training data set. Method for testing or training (133) a computer-based machine learning module: receiving a data set recognized as valid according to one of Claims 1 until 8th or an augmented data set according to the claim 9 ; and testing or training the computer-based machine learning module with the received data set to obtain a tested or trained machine learning module. Method for applying (134) a computer-based machine learning module: providing a tested or trained machine learning module according to claim 10 ; and processing application data by the received machine learning module. Method according to any of the foregoing Claims 1 until 11 , wherein the respective computer-based machine learning module is designed for image processing and wherein the first and a second test and/or training data set contain image data or wherein the computer-based machine learning module is designed for anomaly detection and the first and a second test and/or training data set -Records contain time series data. procedure according to claim 12 , wherein the respective computer-based machine learning module is an image classifier, optionally an image classifier that semantically segments image data pixel by pixel or by region. System (20) designed to implement the methods of any one of Claims 1 until 13 to execute. Computer program designed to perform the procedures of any of Claims 1 until 13 to execute. Signal or computer-readable medium corresponding to the computer program claim 15 encoded or contained.

Images