JP2024059100A - Apparatus and method for determining an analysis of an image constructed by an encoder - Patents.com - Google Patents

Abstract

A computer-implemented method and system for training an encoder configured to determine a latent representation of an image includes the steps of determining a latent representation (w) and a noise image (ε) by providing training images (xi) to an encoder (70) that determines a latent representation and a noise image for the provided images, determining a masked noise image (εm) by a masking unit (74) masking out portions of the noise image, determining a predicted image by providing the latent representation and the masked noise image to a generator (80) of a generative adversarial network, and training the encoder (70) by adapting parameters of the encoder (70) based on a loss value that characterizes a difference between the predicted image and the training image.

Description

The present invention relates to a computer-implemented method for training an encoder, a method for determining an augmentation of an image, a method for training a machine learning system, a method for determining a control signal, a training system, a control system, a computer program, and a computer-readable storage medium.

Prior Art
“Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation” by Richardson et al., 2021, https://arxiv.org/pdf/2008.00951.pdf, presents a generic framework for image-to-image translation.

“A style-based generator architecture for generative adversarial networks” by Karras et al., 2019, https://arxiv.org/pdf/1812.04948.pdf, discloses StyleGAN, a neural network architecture that automatically learns and separates high-level attributes and probabilistic changes in generated images in an unsupervised manner.

“Analyzing and Improving the Image Quality of StyleGAN” by Karras et al., 2020, https://arxiv.org/pdf/1912.04958.pdf, discloses StyleGAN2, an improved version of the StyleGAN neural network.

Zhang et al., "The Unreasonable Effectiveness of Deep Features as a Perceptual Metric", 2018, https://arxiv.org/pdf/1801.03924.pdf, discloses the Learned Perceptual Image Patch Similarity (LPIPS) metric.

2. Background Art Automatically analyzing latent factors of an image is a task faced by practitioners in several technical fields. While determining an image from a latent representation can be easily achieved, for example, by neural networks known as generative adversarial networks (GANs), the opposite direction, i.e., finding a latent representation for a given image, remains a difficult problem. Particularly when considering machine learning systems, finding such latent factors is a problem that would be desirable to solve, since it would allow extending existing datasets to train machine learning systems on semantic aspects encoded in the image. For example, a latent factor of an image may be the weather conditions currently depicted in the image. By adapting the values of this latent factor and feeding the adapted latent representation to the GAN, an extension can be created that characterizes different weather conditions for a given image. These extensions can then be used to train the machine learning system. The performance of the machine learning system for classification and/or regression analysis can be improved, since the machine learning system will be trained with more diverse data on the latent factors, e.g., semantic factors, of the images used for training.

The process of determining latent factors for an image can be accomplished based on GANs. Such methods are also referred to in the art as "GAN inversion". Previous work on GAN inversion has shown promising results for simple face datasets such as FFHQ. Using a GAN generator, Richardson et al. propose to train an encoder to extract features from a given image and map these features to intermediate latent variables, where the latent variables can be used to manipulate the image, e.g., to change hair color and other facial details. However, when it comes to datasets with a relatively high degree of structural complexity, such as driving scene datasets, known methods are not fully capable of reconstructing all objects in the scene, i.e., recovering all details in the image. For example, in the case of face datasets, the human face is a single object that is roughly in the center, whereas in datasets depicting driving scenes, for example, there are multiple objects such as cars in the image, resulting in much more diverse image layouts.

Richardson et al., “Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation”, 2021, https://arxiv.org/pdf/2008.00951.pdf Karras et al., "A style-based generator architecture for generative adversarial networks", 2019, https://arxiv.org/pdf/1812.04948.pdf Karras et al., “Analyzing and Improving the Image Quality of StyleGAN”, 2020, https://arxiv.org/pdf/1912.04958.pdf Zhang et al., "The Unreasonable Effectiveness of Deep Features as a Perceptual Metric", 2018, https://arxiv.org/pdf/1801.03924.pdf

Advantageously, the method having the features of independent claim 1 makes it possible to train an encoder that is able to accurately analyze the latent factors of an image. This has the additional advantage that the encoder is suitable for enhancing images with a high structural complexity and thereby for determining accurate enhancements.

DISCLOSURE OF THEINVENTION In a first aspect, the present invention provides a computer-implemented method for training an encoder configured to determine a latent representation of an image, the method comprising:
- determining a latent representation and a noisy image by providing training images to an encoder, the encoder being configured to determine the latent representation and the noisy image for the provided images;
- determining a masked noise image by masking out parts of the noise image;
Determining a predicted image by providing the latent representation and the masked noise image to a generator of a generative adversarial network;
- Training the encoder by adapting its parameters based on a loss value, the loss value characterizing the difference between a predicted image and a training image.

The encoder may be understood as a machine learning system configured to receive an image as input and to predict a latent representation based on pixel values of the image. Preferably, the encoder is or includes a neural network. In the method, the encoder is provided with training images for predicting a latent representation. Determining the latent representation may be understood as a particular form of image analysis. The encoder is trained to analyze images with respect to particular latent factors that characterize the images, where the latent factors are included in the latent representation.

The latent factors contained in the latent representation may also be referred to in the art as "style". In other words, the latent representation may also be understood as characterizing at least one latent style of an image. The latent factors may be generally understood as the appearance of an image. For example, one latent factor may be the brightness of the situation depicted in the image. A specific value of this latent factor may then characterize, for example, an image depicting a daytime scene. Another value of this latent factor may characterize a nighttime scene depicted by the image.

The encoder is further configured to predict the second component during training, namely the noise image. This noise image can be understood as an image that preferably has the same aspect ratio as the training image. The name noise image has been chosen with respect to the similar naming of similar entities in StyleGAN. In other words, the noise image should not be understood as a prediction of noise in the image. The noise image is an entity that characterizes one part of the inverse of the image as provided by the generator (the other part is the latent representation). In other words, given a training image generated by the generator, the encoder learns to determine the noise that was used as input to the generator to generate that training image. The noise image may include, for example, a value between 0 and 1 that characterizes a percentage value of how much noise a pixel in the training image is subject to. The noise image may be of the same size as the training image, in which case there is a one-to-one correspondence between pixels in the noise image and pixels in the training image. However, it is also possible for the encoder to predict a noise image that has a scaled-down size compared to the training image.

It can be understood that the encoder can be configured to process images from different types of sensors. In this sense, an image can be understood as a sensor measurement obtained from a camera, a LIDAR sensor, a radar sensor, an ultrasonic sensor or a thermal camera.

In the method, parts of the noise image are masked out. This can be understood as replacing pixels in the noise image by other values. For example, pixel values in the noise image can be masked out by replacing them by pixel values drawn randomly, preferably from a Gaussian distribution. To select which pixels to mask out, pixels in the image can be randomly assigned to those that should be masked out or those that should not be masked out. Alternatively, it is also possible to determine, for example randomly, areas of the image to be masked out, e.g. rectangular areas. Such rectangular areas may also be referred to as patches.

The latent representation and the masked noise image are provided to a generator, which is configured to determine an image based on the latent representation and the noise image. The generator may be a neural network, in particular a neural network configured to receive the latent representation and the noise image at different layers of the neural network. Preferably, the encoder can provide the latent representation to all inputs of the generator that require a latent representation. Alternatively, the encoder can be configured to predict multiple different latent representations and provide these multiple different latent representations to inputs that require a latent representation. With respect to the noise image, the encoder can preferably predict a single noise image. In that case, the generator can be provided with the noise image at all inputs that require it. Alternatively, the noise image can be provided only to a single input of the generator that requires a noise image, and all other inputs that require a noise image can be provided with a single randomly drawn noise image or copies of multiple differently drawn noise images.

Preferably, the generator is a generator configured according to the StyleGAN or StyleGAN2 architecture. Such a generator is also referred to as a "StyleGAN or StyleGAN2 generator." In embodiments using a StyleGAN or StyleGAN2 generator, the latent representation is preferably provided directly to the StyleGAN or StyleGAN2 generator, i.e., the use of a mapping network is omitted. Since StyleGAN and StyleGAN2 can also be configured to receive different latent representations and/or noise images for different parts of the StyleGAN, the encoder can also be configured to determine multiple latent representations and/or noise images that serve as inputs to the StyleGAN or StyleGAN2. StyleGAN or StyleGAN2 are the preferred generative adversarial networks used to obtain the generator, but other machine learning systems are possible as well, as long as they determine the image based on at least the latent representation and a noise image.

During training of the encoder, the generator parameters are preferably fixed, i.e., not adapted. However, it is generally possible to update the generator parameters as part of the method.

The generator determines a predicted image based on the latent representation and the masked noise image. The encoder is then trained by adapting the parameters of the encoder based on the difference between the training image and the predicted image. This is achieved by determining a loss value and adapting the parameters based on this loss value. Preferably, this is achieved by determining the gradient of the parameters with respect to the loss by a backpropagation algorithm and adapting the parameters according to the negative gradient. Alternatively, other optimization methods, e.g. evolutionary optimization methods, can be used as well.

The method can also be understood as a method for inverting a generator, i.e. for GAN inversion, which allows the encoder to recover latent factors from a latent space determined during training of the generator. The inventors have discovered that masking parts of a noisy image leads to an improvement in the performance of the encoder in accurately determining the latent factors for a provided image, i.e. the encoder is better able to analyze the image. This is especially true for images depicting a high structural complexity, such as images from natural scenes or images containing multiple, potentially different, objects.

The loss value can be determined based in particular on a loss function, the first term of which characterizes the difference between the predicted image and the training image.

Preferably, the first term further characterizes a masking of the difference, which removes from the difference pixels that fall within the masked out portion.

This difference may be, for example, the average _L2 norm of corresponding pixels from the training and predicted images. The inventors have discovered that it is beneficial for the performance of the encoder to not consider pixels that are subject to being masked out in this difference.

Preferably, the loss function includes a second term that characterizes the norm of the noisy image predicted by the encoder.

This is advantageous because the second term constrains the encoder from learning to predict noisy images with high variance, thereby limiting the amount of information provided by the noisy images. This allows the latent factors of the image to be more faithfully encoded into the latent representation and not leak into the noisy images.

を弁別器に提供することによって弁別器によって決定される。 Preferably, the loss function includes a third term characterizing the negative log-likelihood of the output signal of the discriminator, the output signal being a predicted image

is determined by the discriminator by providing to the discriminator

The inventors have discovered that the use of a discriminator further improves the accuracy of the encoder to determine an accurate latent representation for a provided image. The parameters of the discriminator are also preferably fixed during the training of the encoder. The use of the decoder further encourages the encoder to determine a latent representation that characterizes the exact values for each latent factor. Since the generator and the discriminator may be fixed, the encoder is preferably the only entity that can adapt during training, i.e., the only entity that can change the predicted image. As the inventors have discovered, this term provides a favorable incentive to have the generator generate images that look real during optimization. In other words, the encoder is motivated to determine latent representations that are mapped to images that look real by the generator.

Preferably, the training images can be determined by providing randomly sampled or user-defined latent representations to the generator, and the loss function includes a fourth term that characterizes the difference between the randomly sampled or user-defined latent representations and the latent representation determined from the encoder.

This can be understood as providing cyclic consistency in mapping back and forth between latent representations and predicted images. Thus, the starting point may be a randomly selected latent representation, or a latent representation selected at the user's discretion, which is then provided to the generator to determine a training image. The training image is then provided to the encoder to determine a latent representation as predicted by the encoder. This latent representation should be close to the previously selected latent representation, i.e., mapping back and forth between the latent representation and the image should yield similar results. The fourth term advantageously motivates the encoder to ensure such cyclic consistency. Thus, the inventors have found that the fourth term advantageously further improves the accuracy of the encoder.

In embodiments that include the fourth term, the noise image required to generate the predicted image from the generator may be randomly sampled or may be a predetermined noise image.

Preferably, the loss function includes a fifth term characterizing the difference between a first feature representation determined by providing training images to the feature extractor and a second feature representation determined by providing a predicted image to the feature extractor, the difference preferably not characterizing features characterizing pixels that are within the masked out portion.

This can be understood as adding a term to the loss function that characterizes the LPIPS metric. The feature extractor can be understood as a machine learning system configured to determine features in a machine learning sense from the training images and the predicted images provided, respectively. For example, the feature extractor may be a neural network, such as the convolutional part of VGGnet. The inventors have found that adding a fifth term further improves the accuracy of the encoder.

It is possible to combine any of the first through fifth terms in the loss function. In other words, it is possible to train with some of these terms or to exclude some of these terms.

In another aspect, the present invention provides a computer-implemented method for determining an augmentation of an image, the method comprising:
- obtaining an encoder based on training the encoder by the method as described above;
- determining a first latent representation and a noise image by providing the image to an encoder;
- determining a second latent representation by modifying the first latent representation;
- determining the augmentation by providing the second latent representation and the noise image as input to a generator used in training the encoder;
The present invention relates to a method comprising the steps of:

Obtaining an encoder based on training can be understood as implementing a method for training as part of a method for determining an extension. Alternatively, obtaining an encoder based on training can also be understood as obtaining an already trained encoder, in which case the encoder has been trained by the method for training as presented above.

In a method for determining the augmentation, the augmentation is determined by extracting a latent representation and a noise image from the image using an encoder, modifying latent factors in the latent representation, and providing the modified latent representation and the noise image to a generator used in training the encoder to determine the augmentation.

Advantageously, the method makes it possible to create images that can be used to train a machine learning system. Due to the variation of the latent factors, the augmentation features different styles of the image while preserving at least some of its content. In this way, using the augmentation to train a machine learning system provides the machine learning system with a more diverse set of images, since the augmentation features different styles. The inventors have found that this improves the performance of the machine learning system.

Thus, in another aspect, the present invention provides a computer-implemented method for training a machine learning system, the machine learning system being configured to determine an output signal characterizing a classification and/or regression analysis of an image, the method comprising:
- determining an extension of the training images according to claim 9;
Training a machine learning system based on the augmentation;
The present invention relates to a method comprising the steps of:

Embodiments of the present invention will be described in more detail with reference to the following drawings.

FIG. 2 illustrates a schematic diagram of a part of a training method for training an encoder; FIG. 2 shows a schematic diagram of an example for masking a noise image; FIG. 1 shows a dilation device for dilating an image. FIG. 1 illustrates a training system for training a machine learning system. FIG. 1 illustrates a control system including a machine learning system for controlling an actuator in an environment of the actuator. FIG. 1 illustrates a control system for controlling at least a semi-autonomous robot. FIG. 1 illustrates a control system for controlling a manufacturing machine. FIG. 1 illustrates a control system for controlling an automated personal assistant. FIG. 2 illustrates a control system that controls the access control system. FIG. 2 is a diagram showing a control system that controls the monitoring system. FIG. 2 illustrates a control system for controlling the imaging system. FIG. 1 illustrates a control system for controlling a medical analysis system.

1 shows a portion of an embodiment of a method for training an encoder (70). During the method, the encoder (70) is trained to determine a latent representation (w) that characterizes a latent factor (also known as style) of an image and a noise image (ε), which can be understood as predicting areas of noise in the image.

The encoder (70) can be trained based on a single training image (x _i ). Preferably, however, the method uses multiple training images (x _i ) to train the encoder (70). The training image (x _i ) or multiple training images (x _i ) preferably depict scenes with a high degree of structural complexity, e.g. scenes encountered while driving a car and/or scenes of natural environments such as city scenes.

In this embodiment, the encoder (70) is characterized by a neural network for predicting the latent representation (w) and/or the noise image (ε). In other embodiments, other machine learning models may be used for predicting the latent representation (w) and/or the noise image (ε). The encoder (70) preferably includes a feature extractor (71) for extracting features (f) from training images (x _i ) provided to the encoder (70). The features (f) may be transferred to a style unit (72) preferably configured to determine the latent representation (w) and to a noise unit (73) configured to determine the noise image (ε). The style unit (72) and/or the noise unit (73) may preferably be a neural network. However, in general, other machine learning models may be used for the style unit (72) and/or the noise unit (73). In other embodiments, the encoder (70) may include a single neural network for predicting the latent representation (w) and the noise image (ε).

In this embodiment, the latent representation (w) is configured to be a matrix or tensor, and the noise image (ε) is configured to be a matrix. The encoder (70) is configured such that the width and height dimensions of the latent representation (w) and the noise image (ε) have the same ratio as the width and height dimensions of the training images (x _i ). This can be achieved by preferably using aspect-preserving operations in the feature extractor (71), style unit (72), and noise unit (73), for example by using convolution operations with equal strides along the width and height.

The noise image (ε) is provided to a masking unit (74) configured to mask the noise image (ε). In the present embodiment, masking is performed by randomly selecting elements of the noise image (ε) and replacing each selected element by a value randomly drawn from a Gaussian distribution, thereby determining a masked noise image (ε _m ). In further embodiments, the randomly drawn values may also be drawn from other probability distributions.

The latent representation (w) and the masked noise image (ε _m ) are then provided as input to a generator (80) of a generative adversarial network. The generative adversarial network is preferably trained prior to performing the method for training the encoder (70). However, it is also possible to train the generative adversarial network as an additional step in training the encoder (70). The generative adversarial network is configured to determine an image based on the provided latent representation and the noise image. Preferably, the generative adversarial network is a StyleGAN or StyleGAN2.

The latent representation is preferably provided to the generator (80) without using a StyleGAN or StyleGAN2 mapping network. This is advantageous because the encoder learns to determine the latent representation from an intermediate latent space of StyleGAN or StyleGAN2. This advantageously further improves the performance of the encoder in determining latent factors for the image, since this intermediate latent space has better disentanglement than the original latent space of StyleGAN or StyleGAN2.

を決定する。次いで、トレーニング画像（ｘ_ｉ）と予測画像

との間の差を特徴付ける損失値を決定することができる。次いで、損失値を最小化するようにトレーニングを実施することができる。例えば、損失値は、損失関数に基づいて決定可能である。損失関数は、特に、差を特徴付ける第１の項、すなわち、

を特徴付けることができ、ここで、ｘ_ｉ及び

は、それぞれトレーニング画像及び予測画像であり、

は、アダマール積である。１－Ｍの項は、差におけるピクセルの好ましい重み付けを示し、すなわち、マスキングされたノイズ画像（ε_ｍ）においてマスキングアウトされたピクセルは、第１の項Ｌ_ｒｅｃを決定する際には考慮されない。１は、トレーニング画像（ｘ_ｉ）及び予測画像

と同一の形状の全て１の行列として理解されるべきであり、ノイズ画像（ε）がトレーニング画像（ｘ_ｉ）とは異なる形状を有する場合には、ノイズ画像（ε）をマスキングするために使用されるマスクが、トレーニング画像（ｘ_ｉ）のサイズにスケーリングされる。差

に対するＬ_２－ノルムを決定することは、特にｘ_ｉ及び

からの対応するピクセルのユークリッド距離の平均を求めることとして理解される。 A generator (80) generates a predicted image based on the latent representation (w) and the noise image (ε).

Then, the training image (x _i ) and the predicted image

A loss value can be determined that characterizes the difference between the mean and mean scores. Training can then be performed to minimize the loss value. For example, the loss value can be determined based on a loss function. The loss function includes, in particular, a first term that characterizes the difference, i.e.,

can be characterized, where x _i and

are the training image and the predicted image, respectively,

is the Hadamard product. The 1-M term indicates the preferred weighting of pixels in the difference, i.e., the pixels masked out in the masked noise image (ε _m ) are not considered in determining the first term L _rec . 1 is the weighting factor of the training image (x _i ) and the predicted image

should be understood as an all-ones matrix of the same shape as x i , and if the noise image (ε) has a different shape than the training images (x _i ), then the mask used to mask the noise image (ε) is scaled to the size of the training images (x _i ).

Determining the L ₂ -norm _for

This can be understood as taking the average of the Euclidean distances of corresponding pixels from

Preferably, the loss function includes a second term that characterizes the norm of the noise image (ε), which is preferably a sum of the values in the noise image ε, thereby promoting sparsity in predicting the noise image. The second term is expressed as follows:
L _{noise_reg} = |ε|
It can be expressed as:

を弁別器に提供することによって弁別器によって決定される。換言すれば、敵対的生成ネットワークをトレーニングする際に使用される弁別器は、エンコーダ（７０）をトレーニングする際における追加的なガイドとして使用可能である。デコーダを介して、エンコーダは、予測画像

がどの程度「現実的に」見えるかに関する追加的な情報を取得し、それにより、「現実的に」見える画像を予測するための潜在表現の有用性に関する情報を取得する。第３の項は、以下の式：

によって表現可能であり、ここで、Ｄは、弁別器であり、

は、期待値関数である。 Preferably, the loss function includes a third term characterizing the negative log-likelihood of the output signal of the discriminator, the output signal being the predicted image

In other words, the discriminator used in training the generative adversarial network can be used as an additional guide in training the encoder (70).

We obtain additional information about how “realistic” an image looks, and thereby obtain information about the usefulness of the latent representation for predicting “realistic” looking images. The third term is expressed as follows:

where D is the discriminator,

is the expectation function.

によって表現可能であり、ここで、ｗ_ｇｔは、ランダムにサンプリングされた潜在表現又はユーザ定義された潜在表現である。 Preferably, the loss function includes a fourth term characterizing a difference between the randomly sampled or user-defined latent representation and the latent representation determined from the encoder (70), and the randomly sampled or user-defined latent representation is provided to the generator (80) to determine the training images (x _i ). In other words, the training images (x _i ) are determined based on the randomly sampled or user-defined latent representation. The fourth term is expressed by the following formula:

where w _gt is a randomly sampled or user-defined latent representation.

を特徴抽出器に提供することによって決定される第２の特徴表現との差を特徴付ける第５の項を含み、この差は、好ましくはマスキングアウトされた部分内にあるピクセルを特徴付ける特徴を特徴付けない。このことは、エンコーダ（７０）をトレーニングする際における追加的なガイドとしてＬＰＩＰＳ指標を使用することとして理解することが可能である。第５の項は、以下の式：

によって表現可能であり、ここで、Ｖは、特徴抽出器であり、マスクＭは、第１の項Ｌ_ｒｅｃに関して行われるのと同様に、特徴の幅及び高さにスケーリングされている。 Preferably, the loss function is determined by providing a first feature representation (x _i ) to the feature extractor and a predicted image (x i )

to a feature extractor, which preferably does not characterize features that characterize pixels that are within the masked-out portion. This can be understood as using the LPIPS indices as an additional guide in training the encoder (70). The fifth term is expressed as the following formula:

where V is the feature extractor and the mask M has been scaled to the feature width and height as is done for the first term _Lrec .

によって表現可能であり、ここで、α_１乃至α_５は、それぞれの項の重みである。これらの重みは、トレーニング方法のハイパーパラメータとして理解することが可能である。 Any combination of these terms can be used to determine the loss function L. Preferably, each different term is assigned a weight, which controls the importance of each term relative to the other terms. Thus, the loss function is expressed as the following formula:

where α ₁ to α ₅ are the weights of the respective terms. These weights can be understood as hyperparameters of the training method.

The encoder (70) can then be trained by gradient descent, which can be understood as, among other things, adapting the parameters according to the negative gradient of the loss with respect to the parameters.

Fig. 2 shows how a noise image (ε) can be masked to determine a masked noise image (ε _m ). The noise image (ε) is characterized by a matrix, whose elements are noise values. A number of elements are selected to be masked out. These elements are also referred to as the masked out portion (p) of the noise image (ε). These elements can be characterized by a binary matrix M, which contains a value of 1 for the masked out portion (p) and a value of 0 for all other elements. The masked out portion can then be replaced by randomly sampled values, for example by values sampled from a Gaussian distribution.

を決定する。第２の潜在表現

とノイズ画像（ε）とは、エンコーダ（７０）をトレーニングする際に使用された生成器（８０）に提供される。次いで、生成器（８０）は、拡張

として提供される画像を決定する。 Fig. 3 shows an embodiment of an augmentation unit (90) configured to augment a provided image (b _i ). The augmentation unit (90) comprises an encoder (70) that has been trained by the training method presented above. The encoder (70) receives the provided image (b _i ) and determines a noise image (ε) and a latent representation (w). The latent representation (w) is provided to a transformation unit (91). The transformation unit (91) is configured to vary one or more latent factors of the latent representation. Preferably, the transformation unit (91) randomly determines and varies the one or more factors. The amount of variation can also be randomly selected, with an interval that is understood as a hyperparameter of the transformation unit (91). The transformation unit (91) generates a second latent representation (w) by varying the latent factors of the latent representation (w).

Determine the second latent representation

The noise image (ε) is provided to a generator (80) that was used in training the encoder (70). The generator (80) then

Determine the image that will be provided as.

4 shows an embodiment of a training system (140) using an augmentation unit (90) for training the machine learning system (60) with a training data set (T) comprising a plurality of images (b _i ) used to train the machine learning system (60), the training data set (T) further comprising a desired output signal (t _i ) for each image (b _i ), the desired output signal (t _i ) corresponding to the image (b _i ) and characterizing a desired classification and/or a desired regression analysis result for the image (b _i ).

For training, the training data unit (150) accesses a computer-implemented database (St ₂ ), which provides a training data set (T). The training data unit ( ₁₅₀ ) determines, preferably randomly, at least one image (b _i ) from the training data set (T) and a desired output signal (t _i ) corresponding to this image (b _i ), and transmits this image (b _i ) to the machine learning system (60). The machine learning system (60) determines an output signal (y _i ) based on the image (b _i ).

The desired output signal (t _i ) and the determined output signal (y _i ) are sent to a modification unit (180).

The modification unit (180) then determines new parameters (Φ') for the machine learning system (60) based on the desired output signals (t _i ) and the determined output signals (y _i ). To this end, the modification unit (180) compares the desired output signals (t _i ) and the determined output signals (y _i ) using a loss function. The loss function determines a first loss value that characterizes how much the determined output signals (y _i ) deviate from the desired output signals (t _i ). In the given embodiment, a negative log-likelihood function is used as the loss function. In alternative embodiments, other loss functions are also possible.

Furthermore, it is also conceivable that the determined output signal (y _i ) and the desired output signal (t _i ) each comprise a plurality of sub-signals, for example in the form of a tensor, where the sub-signals of the desired output signal (t _i ) correspond to the sub-signals of the determined output signal (y _i ). For example, it is conceivable that the machine learning system (60) is configured for object detection, where a first sub-signal characterizes the probability of occurrence of an object with respect to a portion of the image (b _i ), and a second sub-signal characterizes the exact location of the object. In case the determined output signal (y _i ) and the desired output signal (t _i ) comprise a plurality of corresponding sub-signals, a second loss value is preferably determined for each corresponding sub-signal by a suitable loss function, and these determined second loss values are suitably combined to form a first loss value, for example by a weighted sum.

The correction unit (180) determines new parameters (Φ') based on the first loss value. In a given embodiment, this is performed using a gradient descent method, preferably a stochastic gradient descent method, Adam or AdamW. In further embodiments, the training can also be based on evolutionary algorithms or second-order methods for training neural networks.

In another preferred embodiment, the training is repeated iteratively for a predetermined number of iteration steps or until the first loss value falls below a predetermined threshold. Alternatively or additionally, training may be terminated when the first average loss value for the test data set or the validation data set falls below a predetermined threshold. In at least one of the multiple iterations, the new parameters (Φ') determined in the previous iteration are used as parameters (Φ) of the machine learning system (60).

Further, the training system (140) may include at least one processor (145) and at least one machine-readable storage medium (146), the at least one machine-readable storage medium (146) including instructions that, when executed by the processor (145), cause the training system (140) to perform a training method according to one of the aspects of the present invention.

Figure 5 shows an embodiment of an actuator (10) in its environment (20). The actuator (10) interacts with a control system (40), which uses a machine learning system (60) to control the actuator (10). The actuator (10) and its environment (20) are collectively referred to as the actuator system. A sensor (30), preferably at equally spaced time points, senses the state of the actuator system. The sensor (30) may include multiple sensors. Preferably, the sensor (30) is an optical sensor that takes an image of the environment (20). An output signal (S) of the sensor (30) (or an output signal (S) for each of these sensors, if the sensor (30) includes multiple sensors), encoding the sensed situation, is sent to the control system (40).

Thereby, the control system (40) receives the stream of sensor signals (S). The control system (40) then calculates a set of control signals (A) depending on the stream of sensor signals (S), which are then transmitted to the actuator (10).

The control system (40) receives a stream of sensor signals (S) of the sensor (30) at an optional receiving unit (50). The receiving unit (50) converts the sensor signals (S) into an image (x). Alternatively, if the receiving unit (50) is not provided, the respective sensor signal (S) may be directly acquired as an image (x). The image (x) may be provided, for example, as an excerpt of the sensor signal (S). Alternatively, the sensor signal (S) may be processed to generate the image (x). In other words, the image (x) is provided according to the sensor signal (S).

The image (x) is then transmitted to a machine learning system (60).

The machine learning system (60) is parameterized by parameters (Φ), which are stored in and provided by a parameter store (St _{1 )} .

The machine learning system (60) determines an output signal (y) from an image (x). The output signal (y) includes information that assigns one or more labels to the image (x). The output signal (y) is sent to an optional transformation unit (80), which transforms the output signal (y) into a control signal (A). The control signal (A) is then sent to the actuator (10) to control the actuator (10) accordingly. Alternatively, the output signal (y) may be obtained directly as the control signal (A).

Actuator (10) receives control signal (A) and is accordingly controlled to perform an action corresponding to control signal (A). Actuator (10) may include control logic that converts control signal (A) into a further control signal, which is then used to control actuator (10).

In further embodiments, the control system (40) may include a sensor (30). In yet other embodiments, the control system (40) may alternatively or additionally include an actuator (10).

In yet other embodiments, it is contemplated that the control system (40) controls the display (10a) instead of or in addition to the actuator (10).

Further, the control system (40) may include at least one processor (45) and at least one machine-readable storage medium (46) having instructions stored thereon that, when executed, cause the control system (40) to perform a method according to an aspect of the present invention.

Figure 6 shows an embodiment in which the control system (40) is used to control an at least semi-autonomous robot, such as an at least semi-autonomous vehicle (100).

The sensors (30) may include one or more video sensors, one or more radar sensors, one or more ultrasonic sensors, and/or one or more LiDAR sensors. Some or all of these sensors are preferably, but not necessarily, on board the vehicle (100).

The machine learning system (60) can be configured to detect an object in the vicinity of the at least semi-autonomous robot based on the image (x). The output signal (y) can include information characterizing where the object is located in the vicinity of the at least semi-autonomous robot. A control signal (A) can then be determined according to this information, for example to avoid a collision with the detected object.

The actuators (10), preferably on board the vehicle (100), can be provided by the brakes, propulsion system, engine, drivetrain or steering of the vehicle (100). A control signal (A) can be determined such that the actuators (10) are controlled such that the vehicle (100) avoids a collision with the detected object. The detected objects can also be classified according to their identity, e.g. pedestrian or tree, which the machine learning system (60) considers to be most plausible, and the control signal (A) can be determined depending on the classification.

Alternatively or additionally, the control signal (A) can also be used to control the display (10a) so that objects detected, for example, by the machine learning system (60) are displayed. It is also conceivable that the control signal (A) can control the display (10a) so that a warning signal is generated if the vehicle (100) is about to collide with at least one of the detected objects. The warning signal can be an audible warning and/or a haptic signal, for example a vibration on the steering wheel of the vehicle.

In further embodiments, the at least semi-autonomous robot may be provided by another mobile robot (not shown), which may for example move by flying, swimming, diving or walking. The mobile robot may in particular be an at least semi-autonomous lawnmower or an at least semi-autonomous cleaning robot. In all the above embodiments, the control signal (A) may be determined such that the propulsion unit and/or steering and/or braking of the mobile robot are controlled so that the mobile robot can avoid a collision with said identified object.

In a further embodiment, the at least semi-autonomous robot can be provided by a gardening robot (not shown), which uses a sensor (30), preferably an optical sensor, to identify the state of the plants in the environment (20). The actuator (10) can control a nozzle for spraying a liquid and/or a cutting device, e.g. a blade. Depending on the identified species of the plant and/or the identified state, a control signal (A) can be determined to cause the actuator (10) to spray an appropriate amount of a suitable liquid on the plant and/or to cut the plant.

In yet another embodiment, the at least semi-autonomous robot can be provided by a household appliance (not shown), such as a washing machine, stove, oven, microwave or dishwasher. A sensor (30), such as an optical sensor, can detect the state of an object to be processed by the household appliance. For example, if the household appliance is a washing machine, the sensor (30) can detect the state of the laundry in the washing machine. A control signal (A) can then be determined depending on the detected material of the laundry.

Figure 7 shows an embodiment in which a control system (40) is used to control a manufacturing machine (11) (e.g., a punch cutter, cutter, gun drill, or gripper) of a manufacturing system (200), for example as part of a production line. The manufacturing machine may include a conveying device, for example a conveyor belt or an assembly line, that moves the manufactured product (12). The control system (40) controls an actuator (10), which in turn controls the manufacturing machine (11).

The sensor (30) can be provided, for example, by an optical sensor that captures characteristics of the manufactured product (12). The machine learning system (60) can therefore be understood as an image classifier.

The machine learning system (60) can determine the position of the manufactured product (12) relative to the conveying device. The actuator (10) can then be controlled depending on the determined position of the manufactured product (12) for a subsequent manufacturing process of the manufactured product (12). For example, the actuator (10) can be controlled to cut the manufactured product at a specific point on the manufactured product itself. Alternatively, it can be envisaged that the machine learning system (60) classifies whether the manufactured product is damaged and/or exhibits a defect. In that case, the actuator (10) can be controlled to remove the manufactured product from the conveying device.

Figure 8 shows an embodiment in which the control system (40) is used to control an automated personal assistant (250). The sensor (30) may be, for example, an optical sensor for receiving video images of the gestures of the user (249). Alternatively, the sensor (30) may be, for example, an acoustic sensor for receiving voice commands of the user (249).

The control system (40) then determines a control signal (A) for controlling the automated personal assistant (250). The control signal (A) is determined according to the sensor signal (S) of the sensor (30). The sensor signal (S) is transmitted to the control system (40). For example, the machine learning system (60) can be configured to execute a gesture recognition algorithm, for example, to identify a gesture performed by the user (249). The control system (40) can then determine a control signal (A) for transmission to the automated personal assistant (250). The control system (40) then transmits the control signal (A) to the automated personal assistant (250).

For example, the control signal (A) may be determined according to an identified user gesture recognized by the machine learning system (60). The control signal (A) may include information for causing the automated personal assistant (250) to retrieve information from a database and output the retrieved information in a form suitable for receipt by the user (249).

In a further embodiment, instead of an automated personal assistant (250), it can be envisaged that the control system (40) controls a home appliance (not shown) which is controlled according to the identified user gestures. The home appliance may be a washing machine, a stove, an oven, a microwave oven or a dishwasher.

Figure 9 shows an embodiment in which the control system (40) controls an access control system (300). The access control system (300) can be designed to physically control access. The access control system (300) can include, for example, a door (401). The sensor (30) can be configured to detect a relevant scene to determine whether access should be allowed. For example, the sensor (30) can be an optical sensor to provide image or video data, for example to detect a person's face. Thus, the machine learning system (60) can be understood as an image classifier.

The machine learning system (60) can be configured to classify the identity of the person, for example by matching the detected person's face with the faces of other known people stored in a database, thereby determining the identity of the person. A control signal (A) can then be determined depending on the classification of the machine learning system (60), for example according to the determined identity. The actuator (10) can be a lock that opens or closes a door depending on the control signal (A). Alternatively, the access control system (300) can be a non-physical and logical access control system. In this case, the control signal can be used to control a display (10a) to display information about the identity of the person and/or whether the person should be allowed access.

10 shows an embodiment in which the control system (40) controls the monitoring system (400). This embodiment is largely identical to the embodiment shown in FIG. 9. Therefore, only the different aspects will be described in detail. The sensor (30) is configured to detect the scene under surveillance. The control system (40) does not necessarily control the actuator (10), but can alternatively control the display (10a). For example, the machine learning system (60) can determine a classification of the scene, e.g., whether the scene detected by the optical sensor (30) is normal or whether the scene shows an anomaly. The control signal (A) sent to the display (10a) can then be configured to, for example, cause the display (10a) to adjust the content it displays depending on the determined classification, e.g., to highlight objects determined to be anomalous by the machine learning system (60).

Figure 11 shows an embodiment of a medical imaging system (500) controlled by a control system (40). The imaging system may be, for example, an MRI device, an X-ray imaging device, or an ultrasound imaging device. The sensor (30) may be, for example, an imaging sensor that takes at least one image of a patient, for example displaying different types of body tissue of the patient.

The machine learning system (60) can then determine a classification of at least a portion of the sensed image. Thus, at least a portion of the image is used as an input image (x) to the machine learning system (60).

A control signal (A) can then be selected according to this classification, thereby controlling the display (10a). For example, the machine learning system (60) can be configured to detect different types of tissue in the sensed image, e.g., by classifying tissues displayed in the image as either malignant or benign tissue. This can be done by semantic segmentation of the input image (x) by the machine learning system (60). A control signal (A) can then be determined to cause the display (10a) to display the input image (x) and show different tissues, e.g., by coloring different regions of the same tissue type with the same color.

In further embodiments (not shown), the imaging system (500) can be used for non-medical purposes, for example to identify material properties of a workpiece. In these embodiments, the machine learning system (60) can be configured to receive an input image (x) of at least a portion of the workpiece and perform a semantic segmentation of the input image (x) to thereby classify material properties of the workpiece. A control signal (A) can then be determined to cause the display (10a) to display the input image (x) and information related to the detected material properties.

Figure 12 shows an embodiment of a medical analysis system (600) controlled by a control system (40). The medical analysis system (600) is provided with a microarray (601) that includes a number of spots (602, also known as features) that are exposed to a medical sample. The medical sample may be, for example, a human sample or an animal sample, for example obtained from a swab.

The microarray (601) may be a DNA microarray or a protein microarray.

The sensor (30) is configured to sense the microarray (601). The sensor (30) is preferably an optical sensor, such as a video sensor.

The machine learning system (60) is configured to classify the sample result based on the input image (x) of the microarray provided by the sensor (30). In particular, the machine learning system (60) can be configured to determine whether the microarray (601) indicates the presence of a virus in the sample.

The control signal (A) can then be selected so that the display (10a) displays the results of the classification.

The term "computer" may be understood to encompass any device for processing certain computational rules. These computational rules may be in the form of software, hardware, or a mixture of software and hardware.

In general, plurals can be understood to be subscripted, i.e., each element of a plural is assigned a unique subscript, preferably by assigning consecutive integers to the elements in the plural. Preferably, if a plural has N elements, and N is the number of elements in the plural, then the elements are assigned integers from 1 to N. It can also be understood that each element in a plural is accessible via the subscript of the element.

Claims (15)

1. A computer-implemented method for training an encoder (70) configured to determine a latent representation of an image (x _i ), comprising:
Training the encoder comprises:
- determining a latent representation (w) and a noise image (ε) by providing training images (x _i ) to said encoder (70), said encoder (70) being configured to determine the latent representation and the noise image for the provided images;
- determining a masked noise image (ε _m ) by masking out a portion (p) of said noise image (ε);
A predicted image is generated by providing the latent representation (w) and the masked noise image (ε _m ) to a generator (80) of a generative adversarial network.

determining
training the encoder (70) by adapting its parameters based on a loss value, the loss value being determined by the predicted image

characterizing the difference between the training images (x _i ) and
A method comprising: masking out a portion (p) of the noise image (ε) comprises replacing values in the portion (p) by randomly drawn values.
The method of claim 1. The loss value is determined based on a loss function;
The first term of the loss function is the predicted image

and the training images (x _i );
The method according to claim 1 or 2. The first term further characterizes the masking of the difference,
The masking removes pixels from the difference that fall within the masked out portion (p).
The method according to claim 3. the loss function includes a second term that characterizes the norm of the noisy image (ε) predicted by the encoder (70).
The method according to claim 3 or 4. the loss function includes a third term that characterizes the negative log-likelihood of the discriminator output signal;
The output signal is the predicted image

to the discriminator,
6. The method according to any one of claims 3 to 5. The training images (x _i ) are determined by providing randomly sampled or user-defined latent representations to the generator;
the loss function includes a fourth term that characterizes a difference between the randomly sampled or user-defined latent representation and the latent representation determined from the encoder (70).
7. The method according to any one of claims 3 to 6. The loss function is a function of a first feature representation determined by providing the training images (x _i ) to a feature extractor and a function of the predicted image (x i )

a fifth term characterizing a difference between the first feature representation determined by providing
said difference preferably does not characterize features that characterize pixels that are within said masked out portion (p);
8. The method according to any one of claims 3 to 7. Extending the image (b _i )

1. A computer-implemented method for determining
- obtaining an encoder (70) based on training said encoder (70) by a method according to any one of claims 1 to 8;
- determining a first latent representation (w) and a noise image (ε) by providing said images (b _i ) to said encoder (70);
- modifying the first latent representation (w) to generate a second latent representation

determining
the second latent representation as input to a generator (80) used in training the encoder (70);

and the noise image (ε),

determining
The method includes: 1. A computer-implemented method for training a machine learning system (60), comprising:
the machine learning system is configured to determine an output signal (y) characterizing a classification and/or regression analysis of an image (x);
The method comprises:
Extension of the training images (b _i ) according to claim 9

determining
・The above-mentioned extension

training the machine learning system (60) based on
A method comprising: 1. A computer-implemented method for determining a control signal (A) for an actuator (10), comprising:
The control signal (A) is determined based on an output signal (y) of a machine learning system (60) trained according to claim 10,
The output signal (y) is determined based on an image (x).
Method. A training system (140) configured to implement the training method according to any one of claims 1 to 8. A control system (40) configured to implement the method of claim 11. A computer program configured to cause a computer to perform all steps of the method according to any one of claims 1 to 11 when executed by a processor (45, 145). A machine-readable storage medium (46, 146) on which the computer program of claim 14 is stored.