DE112019007762T5 - Method for training a generative adversarial network, modified image generation module and system for detecting features in an image - Google Patents

Abstract

A computer-implemented method for training a generative adversarial network GAN comprising a generator (G) and a discriminator (GD, LD). The generator is an image codec that inputs an input image (111, 112) of a scene and outputs a modified image (MI) of the scene suitable for processing by an image-based detector (SSD). The GAN network is trained by alternately training the discriminator (GD, LD) and the generator (G). The generator (G) is trained to generate, for each input image among input images (II2) captured under any lighting conditions, a modified image (MI) that represents the same scene as the respective input image, but under predetermined first lighting conditions . This training is done using a generator loss function (G_loss) that includes a detector loss term (SSD_loss) that is calculated based on an output (PBB) of the image-based detector (SSD). A modified image generation module and system for detecting features in an image.

Description

FIELD OF THE INVENTION

The present invention relates to the detection of features in images, and more particularly in images of outdoor scenes where identifying features can become difficult due to the variety of different lighting conditions to which the scene is exposed.

BACKGROUND OF THE INVENTION

Cameras have become a widely used sensor for gathering information. The high-resolution images they provide can be processed to extract information that can be used in many subsequent applications. In this process, the images output from the cameras are not used directly; they are processed by a processor running a program that processes the image or images to extract useful information from them. The processor and its image processing program form an image-based detector capable of outputting useful information based on one or more images (or "input images") fed to the processor. The images that can be processed by such an image-based detector can be provided by a camera (CCD, CMOS cameras operating in the visible, infrared, etc.) or - more generally - any sensor that can output an image of its surroundings , to be included. The images can also be calculated by a computer.

Image-based detectors of the type defined above can be of various types depending on the type of information to be detected in the images. For example, the image-based detector can be an object detector that outputs information about the presence of one or more objects in the images or certain types of objects in the images; in some implementations, the information output includes the bounding boxes of the objects detected in the images.

Deep neural networks can be used as an exemplary embodiment of image-based object detectors. For example, a "Single Shot Detector - Generative Adversarial Network" (SSD-GAN) has been proposed in the publication by Literaturquelle [2].

While the process outlined above (image capture with a camera and subsequent processing by an image-based detector) has been successfully implemented in a number of use cases, one of the biggest challenges in dealing with real-world images has proven to be the big one Variability in the appearance of physical objects, people and environments, resulting in some instances of less than expected quality.

Among these variations, an important factor affecting quality is lighting conditions. This factor is discussed in particular in the publication [1] mentioned below.

A first solution to overcome this problem was to perform pre-processing on the images to normalize the illumination in the images before feeding them to the image-based detector. Traditionally, this pre-processing involved altering the light histogram of the images. However, this feature, or the traditional contrast normalization techniques, proved insufficient to correctly compensate for the different lighting conditions in real images, resulting in poor quality.

Accordingly, there is a need for methods and systems capable of processing images such that the modified images can be processed by a high quality image-based detector network.

Current trends for achieving insensitivity to changes in lighting conditions through deep neural networks are illustrated by the publications below (particularly publication [1]).

Literature sources:

• [1] Rad Mahdi, Peter M. Roth and Vincent Lepetit: "ALCN: Adaptive Local Contrast Normalization for Robust Object Detection and 3D Pose Estimation" (British Machine Vision Conference, 2017).
• [2] Phillip Isola et al.: "Image-to-Image Translation with Conditional Adversarial Networks." 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR, IEEE, 2017).
• [3] Liu, Wei, et al.: "SSD: Single shot multibox detector" 14th European Conference on Computer Vision, ECCV 2016. Springer Verlag, 2016.
• [4] Justin Johnson, Alexandre Alahi and Li Fei-Fei: "Perceptual losses for real-time style transfer and super-resolution." European conference on computer vision. Springer, Cham, 2016.
• [5] Karen Simonyan and Andrew Zisserman: "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv:1409.1556 (2014).
• [6] Olaf Ronneberger, Philipp Fischer and Thomas Brox: "U-net: Convolutional networks for biomedical image segmentation." International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.

SUMMARY OF THE INVENTION

Despite the performance that has been achieved in today's methods for processing and interpreting images, it is still desirable to have methods and systems that make it possible to maintain a high quality level of image-based processes despite the different lighting conditions that a scene may be exposed to to obtain.

Therefore, according to a first aspect of the present invention, a modified image generation module is proposed. This module is designed to generate a modified image of a scene based on an initial image of the scene. This module includes a generator that is the generator of a Generative Adversarial Network (GAN), where the Generative Adversarial Network is trained using a training method that is described below.

More specifically, this modified image generation module is configured to input an initial image captured or computed under arbitrary lighting conditions and output a modified image representing the same scene but as it appears under "optimal lighting conditions". These “optimal lighting conditions” are lighting conditions that are considered to be optimal lighting conditions for the image-based detector to perform its image-based process.

Accordingly, the modified images generator is able to pre-process an image to output a modified image that closely resembles an image of the same scene but under optimal lighting conditions. Therefore, if modified images obtained thanks to this pre-processing (instead of the raw initial images) are input to the image-based detector, it is advantageously possible to obtain a high quality level of the image-based detector - despite the different lighting conditions of the initial images.

The modified image generation module can be coupled with an image-based detection module to detect features in an image. The system for detecting features in an image obtained in this way comprises the modified-image generation module defined above and an image-based detection module comprising an image-based detector arranged to detect the modified image output by the modified-image generation module to process the scene.

According to the present invention, the generator is coupled to a discriminator to form a generative adversarial network. The generator is an image codec configured to input an input image of a scene and output a modified image of the scene suitable for processing by an image-based detector.

The training procedure for the generative adversarial network includes:

• wobei der Generator so trainiert wird, dass er für jedes Eingabebild aus einem Beleuchtungstrainingssatz von Eingabebildern einer oder mehrerer Szenen, die unter beliebigen Lichtverhältnissen aufgenommen oder berechnet wurden, ein modifiziertes Bild erzeugt, das dieselbe Szene wie das jeweilige Eingabebild darstellt, jedoch unter zuvor festgelegten ersten Lichtverhältnissen;
• wobei der Generator unter Verwendung einer Generatorverlustfunktion trainiert wird, die einen Detektorverlustterm umfasst;
• wobei der Detektorverlustterm auf der Grundlage einer Eingabe des bildbasierten Detektors berechnet wird.

B30) training the generative adversarial network by alternately training the discriminator and the generator;

• wherein the generator is trained to produce, for each input image, from a lighting training set of input images of one or more scenes captured or computed under any lighting condition, a modified image that represents the same scene as the respective input image, but under predetermined firsts lighting conditions;
• wherein the generator is trained using a generator loss function comprising a detector loss term;
• wherein the detector loss term is calculated based on an input from the image-based detector.

In the above method, of course, "alternatively training the discriminator and the generator" also includes repeating the steps of training the discriminator one or more times and training the generator one or more times.

In the proposed method, the detector loss term is preferably calculated to represent how well the image-based detector performs its function based on a modified image.

For example, in one embodiment, for each input image, the detector loss term is computed by combining an image-based detector output based on a ground truth image corresponding to the input image and captured or obtained under the first lighting conditions with an image-based detector output based on the modified image obtained on the basis of each input image is compared.

During its training, for each input image it receives (which may be, for example, a modified image output by the generator), the discriminator is trained to determine whether the input image is a real image or a fake image, where a real image is an image originally taken or calculated under the optimal conditions (i.e. under the first lighting conditions), and conversely a fake image is a modified image output by the generator.

On the other hand, during its training, the generator learns to generate modified images which - at least for the discriminator - are indistinguishable from the real images. Thus, when training is complete, the generator generates images that appear to represent the scene under the optimal (or initial) lighting conditions. Since the optimal conditions are lighting conditions under which the processing performed by the image-based detector works best, the pre-processing performed by the generator thus enables the image-based detector to process the modified images with a high level of quality.

Advantageously in this method the generator loss function comprises the detector loss term, which is calculated in particular on the basis of an output of the image-based detector when the modified image output by the modified-images generator is processed. By including this detector loss term in the generator loss function, it was possible to increase the performance of the generator so that the image-based detector performs particularly well in processing modified images obtained by the generator trained using this generator loss function.

The generator loss function typically includes other terms as well. For example, the generator loss function may also include an LI norm as a reconstruction error.

Both the generator and the discriminator are essentially formed by neural networks.

The generator can preferably be a coder-decoder. In particular, it may have a U-network architecture (a U-network architecture is described for example by publication [6]).

• - einen Codierer, der eine Darstellung des Bildes erlernt, wobei er weniger Neuronen verwendet als das Eingabebild
• - einen Decodierer, der lernt, die originale Eingabe aus der komprimierten Darstellung zu rekonstruieren.

An image codec is a neural network, essentially comprising an encoder and a decoder, configured to learn a compressed representation of an image. A coder-decoder consists of two parts:

• - an encoder that learns a representation of the image using fewer neurons than the input image
• - a decoder that learns to reconstruct the original input from the compressed representation.

A U-net is an image encoder-decoder modified by adding skip-connections connecting encoder layers directly to decoder layers: usually a layer at position i is connected to a layer at position ni, where n is the total number of layers.

The discriminator includes one or more neural networks that are used to calculate the loss function of the generator.

In one embodiment, the discriminator comprises two sub-neural networks, ie a global discriminator and a local discriminator. The global discriminator is a neural network that gives an estimate of whether the input image is a fake or a true image based on the entire input image.

In contrast, the local discriminator comprises a pre-processing unit with a convolution layer and a main processing unit. The pre-processing unit is arranged to output patches, each patch being a representation of a sub-patch of the input image. The main processing unit comprises a neural network and is arranged to output an estimate based on the patches as to whether the input image is a fake or a true image.

The patches can preferably be based on non-overlapping regions of the input image.

For example, the global discriminator can be a neural network trained using a structured loss function that penalizes the output of the global discriminator only at a large scale; and the local discriminator is trained using a structured loss function that penalizes the local discriminator output only at a small scale smaller than the large scale.

Since the local discriminator is only based on local information, a patch output by the pre-processing unit usually represents a local texture of the input image.

By using both the global and local discriminator instead of just one, it is possible to obtain an improved result that takes into account the scale of the overall image and the local scale. The global discriminator is set up to give an estimate based on the global similarity of the overall image to a "true" image, i.e. an image obtained under normal lighting conditions. However, using only one such discriminator can result in training the generator to generate blurred images, which is unsatisfactory. To compensate for this aspect, the local discriminator is used, which forces the generator to consider seldom occurring details (as they appear in the different regions) of the input image. The local discriminator is therefore used to judge whether an image is "real" or "fake" (that is, whether an image was taken under normal lighting conditions or not) by looking at the texture of the image locally, rather than looking at large-scale indices to consider.

For example, the global objective function of the GAN network may include an L2 loss term computed for the global discriminator. Conversely, the global objective function of the GAN network can include an L1 loss term computed for the local discriminator, which forces the GAN network to consider the local scale. (This is described by the above-mentioned publication [2]; see in particular Section 3.1).

An example of a possible architecture for the generator, the global discriminator and the local discriminator is proposed by the above-mentioned publication [2]: see in particular the architectures proposed in Section 3.2: Section 3.2.1 for the generator G; Section 3.2.3 for the local discriminator LD, with the example of PatchGAN.

In one embodiment, the image-based detector is an object detector, in particular a neural network, which is set up to detect objects in an image. In particular, the object detector can detect the bounding boxes of objects shown in the image. Now, in a broader sense, the function of the image-based detector can be any computer vision task: activity detection, semantic segmentation, SLAM, etc.

In one embodiment, the generator loss function also accounts for perceptual loss, where perceptual loss is an image-based function that increases as sharpness loss in the image increases.

Such a function is described in particular by publication [4].

In one embodiment, the image-based function is based on information that is output by two different layers of a convolutional neural network, respectively. This convolution neural network can be, for example, a VGG16 neural network or a similar neural network.

In this case, the loss of perception can be expressed by the sum of two terms as follows: The input image and the output image are input to the VGG16 or a similar neural network. The outputs of a low-order convolutional layer and a high-order convolutional layer are then collected; they are called "low-level traits" and "high-level traits," respectively.

The loss of perception can then be expressed as follows: $$\begin{array}{l}{L}_{pe\mathrm{right}c}=100\ast (\left|HiGH\_level\left(inp\mathrm{and}t\right)-HiGH\_level\left(O\mathrm{and}tp\mathrm{and}t\right)\right|\\ +\left|lOw\_level\left(inp\mathrm{and}t\right)-lOw\_level\left(O\mathrm{and}tp\mathrm{and}t\right)\right|)\end{array}$$

• B20) Erstellen des GAN-Trainingssatzes, wobei der GAN-Trainingssatz umfasst:
• als Eingabe, Bilder von mindestens einer Szene, wobei die mindestens eine Szene unter mehreren Lichtverhältnissen abgebildet wird;
• als Grundwahrheit, Bilder der mindestens einen Szene unter den ersten Lichtverhältnissen.

In one embodiment, the method also includes the step of:

• B20) creating the GAN training set, the GAN training set comprising:
• as input, images of at least one scene, the at least one scene being imaged under multiple lighting conditions;
• as a basic truth, images of at least one scene under the first lighting conditions.

According to a further aspect of the present invention, a computer-implemented method for training an image-based neural network is proposed.

In this method, the image-based neural network comprises a generator coupled to an image-based detector; the image-based detector is a neural network arranged to process a modified image of the scene output by the generator; the generator is an image codec arranged to input an input image of a scene and output a modified image of the scene suitable for processing by the image-based detector; the generator is part of a generative adversarial network that includes the generator and a discriminator.

• B) Trainieren des generativen kontradiktorischen neuronalen Netzes unter Verwendung des oben definierten Trainingsverfahrens; wobei das Verfahren so implementiert wird, dass während des Trainingsschrittes B30) der Verlust unter Verwendung der Verlustfunktion des bildbasierten Detektors (SSD) berechnet wird.

The procedure includes the following step:

• B) training the generative adversarial neural network using the training method defined above; wherein the method is implemented such that during the training step B30) the loss is calculated using the loss function of the image-based detector (SSD).

• A10) Erstellen eines ersten Trainingssatzes, der umfasst:
  • als eine Eingabe, erste Bilder von mindestens einer Szene unter den ersten Lichtverhältnissen;
  • als Grundwahrheit, gewünschte Ausgaben des bildbasierten Detektors für die ersten Bilder.
• A20) Trainieren des bildbasierten Detektors anhand des ersten Trainingssatzes.

The method may further include the following optional steps for training the image-based detector:

• A10) Create an initial training set that includes:
  • as an input, first images of at least one scene under the first lighting conditions;
  • as a ground truth, desired outputs of the image-based detector for the first images.
• A20) Training the image-based detector using the first training set.

• C10) Erstellen eines dritten Trainingssatzes, der umfasst:
  • als eine Eingabe, Bilder, die durch den Generator bei der Verarbeitung des ersten Trainingssatzes ausgegeben werden; und
  • als Grundwahrheit, die gewünschten Ausgaben des bildbasierten Detektors für die ersten Bilder.
• C20) Trainieren des bildbasierten Detektors anhand des dritten Trainingssatzes.

In addition, the method may include the following optional steps to optimize the image-based detector:

• C10) Create a third training set that includes:
  • as an input, images output by the generator in processing the first training set; and
  • as a ground truth, the desired outputs of the image-based detector for the first images.
• C20) Training the image-based detector using the third training set.

For example, if the image-based detector is an object detector, the desired outputs of the image-based detector may be the actual bounding boxes of the various objects represented in the image.

In one embodiment, the training method further comprises repeating step B) training the generative adversarial neural network and steps C10) creating the third training set and C20) training the image-based detector at least once.

In a particular implementation, the proposed training method is determined by computer program instructions.

Accordingly, another purpose of the present invention is to propose a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of a method. The computer program is preferably stored on a non-transitory, computer-readable storage medium. The computer program can use any programming language and can be in the form of source code, object code or a code between source code and object code, for example in partially compiled form, or in any other convenient form. The computer can be any data processing means, for example a personal computer, an electronic control unit adapted to be installed in a vehicle, etc.

The present invention also includes a non-transitory computer-readable medium on which the computer program is stored. The computer-readable medium can be any entity or device capable of storing the program. For example, the computer-readable medium can be a storage medium, such as a read-only memory (ROM), such as a compact disk (CD)-ROM or a microelectronic circuitry ROM, or it can be a magnetic recording medium, such as a floppy disk or a hard drive, include. Alternatively, the computer-readable medium may be an integrated circuit incorporating the program, which circuit is configured to perform or participate in the performance of the control method discussed herein.

Another object of the present invention is to propose a data processing device adapted to carry out the steps of the training method defined above.

character list

• 1 12 shows a schematic representation of an embodiment of an image-based detector;
• 2 FIG. 12 shows a schematic representation of an input image detected by the image-based detector of FIG 1 to be processed;
• 3 Figure 12 shows a schematic representation of a generative adversarial network according to an embodiment of the present invention during training of its discriminator;
• 4 shows a schematic representation of the generative adversarial mesh 3 while training his generator;
• 5 Figure 12 shows a block diagram of a training method according to an embodiment of the present invention; and
• 6 FIG. 12 shows a schematic representation of a vehicle comprising a data processing device according to an embodiment of the present invention.

DESCRIPTION OF THE EMBODIMENTS

We turn now in detail to exemplary embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

ARCHITECTURE OF A NEURAL NETWORK

The neural network architectures according to the present invention are essentially based on a Generative Adversarial Network (GAN) complemented by auxiliary loss function calculation modules including an image-based detector SSD.

In the present exemplary embodiment, the image-based detector is an algorithm capable of determining the bounding boxes of objects shown in an image. As previously discussed, the methods and systems of the present invention may be implemented using an image-based detector that is also configured for a function other than object detection.

In the present embodiment, the image-based detector is a single-shot detector (SSD) configured to detect objects in an image (hereinafter the “SSD mesh”). This can be, for example, the single-shot multibox detector proposed by publication [3] (see in particular Section 2.1 and 2 ).

The SSD network is arranged to receive input images (II) and to output the predicted bounding boxes (PBB) of the objects detected in the input image. 1 Figure 12 shows, for example, an image IIA, which is the first image of a set of input images II that is input to the SSD mesh. This image IIA represents two apples ( 2 ). The SSD mesh processes the image IIA and returns the predicted values PBB1 and PBB2 of the bounding boxes ((ulmin,vlmin); (ulmax,vlmax)), (u2min,v2min); (u2max,v2max)) of the representations of the two apples in Figure IIA. By processing all input images of the set of input images II, the SSD network outputs a set of bounding boxes PBB.

The adversarial generative network GAN comprises a first neural network G representing the generator and two neural networks, namely a global discriminator GD and a local discriminator LD, forming the discriminator ( 3 , 4 ).

The generator G is a coder-decoder arranged to input an input image II of a scene and to output a modified image MI of the scene. Usually the modified image MI has the same size (height and width) as the input image II.

The input image II can of course have been recorded or calculated under any lighting conditions.

According to the present invention, the generator G is trained so that the modified image MI is as close as possible to an image of the scene as it would appear under the first lighting conditions. That is, the generator G is arranged to correct the lighting conditions of the scene so that it outputs a modified image in which the lighting conditions are modified to correspond to optimal lighting conditions.

For example, the generator G may have a coder-decoder architecture. In the present embodiment, it has a U-network architecture, ie a coder-decoder architecture supplemented by skip connections between the coder and decoder layers. Such an architecture is disclosed by publication [2], section 3.2.1, and by publication [6], particularly in the architecture of 1 , presents.

In this embodiment, the discriminator D comprises, in addition to the global discriminator GD, an additional discriminator called the "local discriminator" and denoted by LD.

These two networks are set up to input a first image, which is a "raw" image of the scene, taken under any lighting conditions, and a second image, which is an image of the same scene but now under optimal lighting conditions.

This second image can be of two types: it can be either a ground truth image, i.e. an image of the scene under optimal lighting conditions, or a modified image MI output by the generator G (this image is intended to represent the optimal lighting conditions).

Based on the first and second images, each of these networks returns an estimate, E _GD and E _LD , respectively (a real number in a range [0-1]) that expresses how likely the second image is to the ground truth image of a modified image. If the net returns the value 1, this means for each of these nets that the second picture is the ground truth picture with absolute certainty. On the other hand, if the network returns the value 0, this means that the image for the neural network is a modified image with absolute certainty.

In another embodiment, the discriminator networks GD and LD may be arranged to output binary values instead of values in a range (in the range [0-1] in the present embodiment).

Each of the networks G, GD and LD is formed from modules of the form Convolution-BatchNorm-ReLu.

The local discriminator LD comprises two units, a pre-processing unit LD1 comprising a convolution layer LD11 and a main processing unit LD2.

The pre-processing unit LD1 is set up to issue patches. Each of these patches is a representation of a sub-patch of the input image. In this embodiment, each input image is divided into eight sub-patches p11-p41 and p12-p42, collectively referred to as the sub-patches p. Accordingly, the pre-processing unit LD1 outputs a patch for each partial patch p of the input image II. This patch is based only on the considered partial patch p and represents its texture.

In a second step, the main processing unit LD2 outputs its estimate E _LD based on the eight patches output by the pre-processing unit LD1.

CONFIGURATION AND TRAINING PROCESS

An exemplary implementation of a method for configuring and training the GAN network and the SSD network according to the present invention is now presented.

In particular, the present invention can be implemented to improve the performance of a prior art image-based detector.

In the present case, the image-based detector is a detector (namely the SSD network) that can be trained. In such a case, a first step consists of an initial training of the image-based detector. Accordingly, in the present embodiment, the first step is to train the SSD network.

A. Configuring and initial training of the image-based detector SSD

The SSD network is trained by performing the following steps A10, A20:

A10 - Creating a first training set SSD-TS1

For the initial training of the SSD network, a first training set SSD-TS1 is generated. This first training set SSD-TS1 serves to train the SSD network to determine the position of the bounding boxes of objects present in images.

The first training set SSD-TS1 includes first input images II1 of different scenes. All of these input images are captured or computed under optimal lighting conditions. As defined above, these optimal lighting conditions are, in the present case, the lighting conditions that are considered best for performing bounding box detection in images.

In the first training set SSD-TS1, the ground truth is the correct or desired bounding boxes for the objects shown by Images III.

A20 - Training the SSD network

Then the SSD network is trained using the first training set SSD-TS1.

B. Configure and train the generator G and the discriminator GD, LD

B10 - Configure the generator G and the discriminator D

Then the generator G, the global discriminator GD and the local discriminator LD are set up; in particular, each of their weights is assigned an initial value.

B20 - Prepare the training set GAN-TS for training the GAN

In order to train these networks, a GAN training set GAN-TS is then created. This training set includes all the data used to train the generator and discriminator. This training set includes a set of input images II2 and a set of "ground truth images" TGI.

In contrast to the input images of the first training set SSD-TS1, the images of the second set of input images 112 are images taken or computed under a variety of different (arbitrary) lighting conditions.

As ground truth, for each input image II2 of a scene, the training set GAN-TS comprises a corresponding ground truth image TGI, which is the image of the same scene but under the optimal lighting conditions.

B30 - Iterative training of the discriminator D and the generator G

Then the generator G, the global discriminator GD and the local discriminator LD are trained by performing an iterative process of one or more stages. Each stage of this process includes the following steps for training the discriminator or generator:

Discriminator Training

First, the discriminators GD and LD are trained by performing the following actions ( 3 , 5 ).

a) Processing of the input images 112 of the GAN training set GAN-TS by the generator G and the discriminator GD, LD

Each input image II2 of the GAN training set GAN-TS (as a "first image") goes through the generator G, which yields as many modified images MI (as a "second image").

Then, for each input image 112, the global and local discriminators GD, LD each provide their estimates (estimates E _GD , E _LD ) of whether the corresponding second image is an original image with optimal lighting or a modified image. For this purpose, for each input image II2 of the GAN training set GAN-TS, a pair of images comprising both the input image 112 and the corresponding modified image MI output by the generator G (as the first and second images) are sent to the transmit global discriminator GD and local discriminator LD. In addition, pairs of images comprising an input image II2 and the corresponding ground truth image TGI (as the first and second images) are also forwarded to the global and local discriminators GD and LD.

For each pair of images, both the global discriminator GD and the local discriminator LD output their estimate EGD or ELD, respectively, that the second image they received is a "true" image, i.e. a ground truth image obtained under the optimal lighting conditions , and not a "false" image, that is, an image produced by the generator G to resemble an original image imaged under the optimum lighting conditions.

b) Calculate the losses of the discriminators GD and LD

Then the values of the global discriminator loss and the local discriminator loss are calculated for the global discriminator GD and for the local discriminator LD, respectively. These values are based on the global and local discriminator outputs EGD and ELD, which are compared to the true values TV expected from the global and local discriminators. (These true values TV are annotations previously prepared in the training data set GAN-TS).

In this embodiment, the two loss functions of the global discriminator and the local discriminator are sigmoidal cross-entropy loss functions. The discriminator loss is calculated as a weighted sum of the global discriminator loss and the local discriminator loss.

c) Back propagation by the discriminator

Based on the discriminator loss calculated in step b), the weights of the global discriminator GD and the local discriminator LD are then updated by backpropagation.

generator training

Then the generator G is trained by performing the following actions ( 4 , 5 ).

a) Reprocessing of the data by the architecture

The modified images MI are then computed using the generator G based on the input images 112 of the GAN training set GAN-TS.

Furthermore, the modified images MI are passed through the image-based detector SSD, which yields calculated or predicted values PBB of the bounding boxes of the objects identified in the modified image MI. Then the pairs of images, comprising an input image II2 and a corresponding modified image MI, are passed through the updated global discriminator GD and the local discriminator LD, which yield updated values for their estimates E _GD and E _LD .

b) Calculating the generator loss G Loss

wobei
L1 eine L1-Normfunktion ist. Der Ll-Verlustterm wird durch den Vergleich des Grundwahrheitsbildes GTI und des durch den Generator G ausgegebenen modifizierten Bildes MI berechnet;
Fool_global_discriminator und Fool_local_discriminator sind zwei Terme, die für den globalen Diskriminator GD bzw. den lokalen Diskriminator LD berechnet werden und die die Fähigkeit des Generators G darstellen, diese Diskriminatoren zu täuschen („fool“);
Perceptual_Loss ist ein Term, um eine Nähe zwischen einem modifizierten Bild MI und einem Grundwahrheitsbild GTI zu berücksichtigen. Die Nähe zwischen diesen beiden Bildern wird bevorzugt in einem Merkmalsraum bewertet. Das heißt, anstatt die Nähe zwischen diesen beiden Bildern direkt zu bewerten, wird auf jedes dieser Bilder eine Transformation unter Verwendung eines neuronalen Netzes angewendet. Die Nähe wird dann zwischen den Ausgaben dieser jeweiligen Transformationen und nicht zwischen den beiden Bildern selbst bewertet. Bevorzugt ist das neuronale Netz, das zum Anwendung der Transformation verwendet wird, ein neuronales Netz, das für die Ausführung von Bilderkennungsaufgaben geeignet ist. Zum Beispiel kann ein neuronales Netz, wie zum Beispiel ein VGG16, für diesen Zweck verwendet werden.The generator loss G loss is then calculated as follows: $$\begin{array}{l}\text{G\_Loss}=\text{L1}+\text{Fool\_global\_discriminator}+0.5\text{Fool\_local\_discriminator}+\\ \text{Perceptual\_Loss}+\text{SSD\_loss}\end{array}$$

whereby
L1 is an L1 norm function. The Ll loss term is calculated by comparing the ground truth image GTI and the modified image MI output by the generator G;
Fool_global_discriminator and Fool_local_discriminator are two terms calculated for the global discriminator GD and the local discriminator LD, respectively, which represent the ability of the generator G to "fool" these discriminators;
Perceptual_Loss is a term to consider a proximity between a modified image MI and a ground truth image GTI. The proximity between these two images is preferably evaluated in a feature space. That is, instead of directly assessing the proximity between these two images, a transformation is applied to each of these images using a neural network. Proximity is then assessed between the outputs of these respective transforms, rather than between the two images themselves. Preferably, the neural network used to apply the transformation is a neural network suitable for performing image recognition tasks. For example, a neural network such as a VGG16 can be used for this purpose.

Finally, SSD_loss is a term that accounts for the generator G's ability to output images for which the SSD mesh can efficiently (that is, with little loss) determine the bounding boxes of the objects shown in the image.

The terms of the sum G_loss above are calculated as follows.

L1

The L1 term is computed using the L1 norm based on the compared inputs of the generator, i.e. based on the ground truth images GTI and the corresponding modified image MI output by the generator.

Fool Discriminator Terms

The two fool discriminator terms are Fool_global_discriminator and Foollocal_discriminator, which are computed for the global discriminator GD and the local discriminator LD, respectively.

• ■ Wie oben erwähnt, geben der lokale Diskriminator LD und der globale Diskriminator GD auf der Grundlage des modifizierten Bildes MI ihre Schätzungen E_GD und E_LD, dass diese Bilder Grundwahrheitsbilder sind, aus.
• ■ Diesen Schätzungen (und damit diesen Bildern) wird ein „Wahr“-Label (oder „Grundwahrheits“-Label) zugewiesen, so als ob dieses modifizierte Bild MI ein Grundwahrheitsbild wäre.
• ■ Es wird ein sigmoidaler Kreuzentropieverlust zwischen der durch den betrachteten Diskriminator ausgegebenen Schätzung (E_GD bzw. E_LD) und den „Echtbild-Labels“, welche die echten Werte sind, die auf der Grundlage der tatsächlichen Grundwahrheitsbilder zugewiesen werden, berechnet. Der auf diese Weise bestimmte Wert ist der Fool-Diskriminator-Term (bzw. die Fool_global discriminator- und Fool_local_discriminator-Terme).

Each of these terms is calculated as follows:

• ■ As mentioned above, based on the modified image MI, the local discriminator LD and the global discriminator GD output their estimates E _GD and E _LD that these images are ground truth images.
• ■ These estimates (and thus these images) are assigned a “true” (or “ground truth” label) as if this modified image MI were a ground truth image.
• ■ A sigmoidal cross-entropy loss is calculated between the estimate (E _GD or E _LD ) given by the considered discriminator and the “real image labels”, which are the real values assigned on the basis of the actual ground truth images. The value so determined is the Fool discriminator term (or Fool_global discriminator and Fool_local_discriminator terms).

Accordingly, the Fool_global_discriminator and Fool_local_discriminator terms restrict the generator G to generating modified images MI, which are commonly identified as ground truth images (with high values of EGD and ELD) by the discriminators GD and LD. By comparing the estimates (EGD and ELD, respectively) output by the discriminators to real image labels, the loss function G_loss accounts for the extent to which the discriminators estimate that the outputs of generator G are ground truth images. If the discriminators GD and LD give estimates indicating that the modified images MI are fake images, then the loss is high (indicating poor performance of the generator G); otherwise the loss is low.

loss of perception

Perceptual loss is a term that measures the generator's performance in reconstructing images that look similar to the input. First, the input image 112 and the modified image MI are input to a neural network. In the present embodiment, this neural network has a VGG16 architecture (this neural network is called VGG16 for convenience). For example, VGG16 can be a pre-trained network, specifically trained on the Imagenet dataset as suggested in reference [4]. An example of a VGG16 architecture is known from publication [5], for example.

wobei high_level (Bild) die Merkmale auf hoher Ebene darstellt, die erhalten werden, wenn ein Bild in das VGG16-Netz eingegeben wird, und low level (Bild) die Merkmale auf niedriger Ebene darstellt, die erhalten werden, wenn das Bild in das VGG16-Netz eingegeben wird.When the input image is processed by the neural network VGG16, the output of the second convolution layer becomes a low-level convolution layer (hereinafter referred to as "low level features") and the output of the 13th convolution layer becomes a convolution layer high level features (hereafter referred to as “high level features”). Based on these outputs, the loss of perception is calculated as $$\begin{array}{l}\text{Perceptual\_Loss}=100*(\left|\text{high\_level}\left(\text{II}\right)-\text{high\_level}\left(\text{MI}\right)\right|+|\text{low\_level}\left(\text{II}\right)-\\ \text{low\_level}\left(\text{MI}\right)|)\end{array}$$

where high_level (image) represents the high-level features obtained when an image is input to the VGG16 network and low level (image) represents the low-level features obtained when the image is input to the VGG16 network is entered.

SSD loss

The SSD loss incorporates the values reported by the loss function of the image-based detector. In the present embodiment, the image-based detector is the SSD single-shot detector proposed in publication [3]. Accordingly, the SSD loss is the loss function used to train this network. The loss term is therefore a weighted sum of a loss of localization and a loss of confidence. The localization loss is a smooth L1 loss term between a predicted position and a correct (ground truth) position of a bounding box. A detailed presentation of the SSD loss function can be found in publication [3] (see function L presented in the training objective section).

• - der vorhergesagten Positionen PBB der Begrenzungsrahmen gemäß Ausgabe durch das SSD-Netz auf der Grundlage der modifizierten Bilder MI; und
• - der tatsächlichen Positionen GTBB der Begrenzungsrahmen der Grundwahrheitsbilder GTI, die Annotationen sind, die zuvor in dem Trainingsdatensatz GAN-TS erstellt wurden.

During generator training, the SSD_loss term is calculated based on the compared values of:

• - the predicted positions PBB of the bounding boxes as output by the SSD network based on the modified images MI; and
• - the actual positions GTBB of the bounding boxes of the ground truth images GTI, which are annotations previously created in the training data set GAN-TS.

In the generator loss, the various terms (L1, Fool_local_discriminator, Fool_local_discriminator, Perceptual Loss, SSD loss) are weighted to account for their respective impacts. The weights can be modified depending on the impact to prioritize. In the present embodiments, the weights 1, 1, 0.5, 1, and 1 have been found to be optimal, with the Fool_local_discriminator being weighted half as much as the Fool_global_discriminator.

c) back propagation by generator G

Then, based on the G_loss function of the generator, the weights of the generator G are updated by backpropagation.

The generator and the discriminator are repeatedly alternately trained by performing the operations described above. Training stops when the training algorithm is considered to have converged or when stable values for the weights of these networks have been determined.

C. Final training of the image-based detector SSD

When the training of the generator G, the global discriminator GD and the local discriminator LD has been stopped, the SSD network is retrained on the basis of the modified images output by the updated generator G.

The image-based detector SSD is thus trained by performing the following steps C10, C20:

C10 - Creating a second training set SSD-TS2

A second training set SSD-TS2 is generated for training the SSD network. The purpose of this second training set SSD-TS2 is to train the SSD network to determine the position of the bounding boxes of the objects present in an image. However, while the first training set SSD-TS1 comprised images of scenes under normal lighting conditions, in contrast, the input images included in the second training set are modified images output by the generator G . It has been observed that training the image-based detector (the SSD mesh in the present embodiment) on the modified images as output by the generator G instead of on the original input images increases the quality of the SSD mesh.

Accordingly, the second training set SSD-TS2 includes as input to the generator G modified images, which are output by the generator G based on the initial input images III of the first training set SSD-TS1. That is, instead of training the SSD mesh on the basis of original images imaged under the first lighting conditions (that is, under normal lighting conditions), the SSD mesh is trained on fake images thus generated by the generator G that they resemble these original images.

As in the first training set SSD-TS1, the ground truth is the correct or desired bounding boxes for the objects represented by the input images.

C20 - Retrain SSD mesh

The SSD network is then retrained - but now using the second training set SSD-TS2.

Exemplary computerized system

6 10 shows a passenger car 1000 (an example of a vehicle) equipped with an automated driving system 500 . The automated driving system 500 includes an image-based detection system 100 as an exemplary computerized system or computing device on which the present invention may be implemented in whole or in part.

The image-based detection system 100 (or system 100 for short) comprises a plurality of sensor units, including in particular a forward-facing camera 110. The camera 110 is mounted slightly above the windshield of the car on a mounting bracket, not shown.

The image-based detection system 100 includes a computer system 150, which includes a mass storage device 151, one or more processors 152, a data memory 153, an operating system 154 and a communication infrastructure 155.

The communication infrastructure 155 is a data bus to which all of the sensor units mentioned above are connected and over which the signals output by these sensor units are therefore transmitted to the other components of the system 100 .

The mass storage 151, the one or more processors 152, the data storage 153 and the operating system 154 are communicatively coupled via the communication infrastructure 155. Optionally, the computer system 150 can interact with a user or the environment via one or more input/output devices 156 and one or more networks 157 . The operating system 154 can interact with other components to control one or more applications 158 . All components of the image-based detection system 2100 are shared or possibly shared with other units of the automated driving system 500 or the passenger car 1000 .

A computer program for performing object detection according to the present invention is stored in data memory 153 . This program and the data memory 153 are examples of a computer program and a computer-readable recording medium, respectively, according to the present invention.

The data storage 153 of the computer system 150 effectively represents a recording medium according to the invention, readable by the one or more processors 152 and on which the program is recorded.

The systems and methods described herein may be implemented in software or hardware, or a combination thereof. The systems and methods described herein may be implemented using one or more computing devices, which may be physically or logically separate from one another, as appropriate.

The systems and methods described herein can be implemented using a combination of hardware, firmware, and/or software. The present systems and methods described herein (or any portion or function thereof) may be implemented using hardware, software, firmware, or a combination thereof, and may be implemented in one or more computer systems or other processing systems.

In one or more embodiments, the present embodiments are embodied in machine-executable instructions. The instructions may be used to cause a processing device, such as a general purpose or special purpose processor, programmed with the instructions to perform the steps of the present invention. Alternatively, the steps of the present invention may be performed by dedicated hardware components that are hardwired logic to perform the steps, or any combination of programmed computer components and specially adapted hardware components. For example, the present invention may be provided as a computer program product as set out above. In this environment, embodiments may include a machine-readable medium on which instructions are stored. The instructions may be used to program any processor(s) (or other electronic device) to perform a process or method consistent with the present example embodiments. In addition, the present invention can also be downloaded and stored on a computer program product. Here, the program may be transmitted from a remote computer (e.g., a server) to a requesting computer (e.g., a client) using data signals embodied in a carrier wave or other propagation medium over a communications link (e.g., a modem or network connection ) and finally such signals can be stored on the computer systems for later execution.

The methods may be implemented in a computer program product, accessible from a computer-usable or computer-readable storage medium, that provides program code for use by, or in connection with, a computer or any instruction execution system. A computer-usable or computer-readable storage medium may be any device that can include or store the program for use by, or in connection with, the computer or instruction execution system, apparatus, or device.

A data processing system suitable for storing and/or executing the corresponding program code may include at least one processor connected directly or indirectly to computerized data storage devices, such as data storage elements.

The systems and methods described herein may be implemented in a computer system that includes a back-end component, such as a data server, or that includes a middleware component, such as an application server or a web server, or that includes a front-end component such as a client computer having a graphical user interface or an Internet browser, or any combination thereof. The components of the system may be connected by any form or medium of digital data communication, for example a communication network.

The terms "computer program medium" and "computer-readable medium" may be used to refer generally to media such as a removable disk drive or a hard disk installed in a hard disk drive. These computer program products can provide software for a computer system. The systems and methods described herein may relate to such computer program products.

All of the neural networks mentioned herein are artificial neural networks composed of one or more processors (e.g., microprocessors, integrated circuits, field-programmable gate arrays, or the like). These neural networks are divided into two or more layers, which are an input layer that can receive images, for example, an output layer that can output, for example, an image or a loss function (for example errors, as described below), and one or more intermediate layers include. The layers of the neural networks G, GD and LD represent different groups or sets of artificial neurons that can represent different functions performed on the images by the processors to calculate modified images and/or errors in the calculation of the modified images to determine.

Throughout the specification, including the claims, the term "comprises a" is synonymous with "comprises at least one" unless otherwise specified.

Furthermore, each range given in the specification as well as in the claims should be understood to include the values at both ends unless otherwise specified. Specific values for items described are to be understood to be within accepted manufacturing or industry tolerances known to those skilled in the art and any use of the terms "substantially" and/or "approximately" and/or "generally" is intended to be so understand that they are within these accepted tolerances.

Although the present invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the present invention.

It is intended that the specification and examples be considered as exemplary only, with the true scope of the invention being indicated by the claims.

Claims (14)

A computer-implemented method for training a Generative Adversarial Network (GAN) comprising a generator (G) and a discriminator (GD, LD); wherein the generator is an image codec arranged to input an input image (111, 112) of a scene and to output a modified image (MI) of the scene suitable for processing by an image-based detector (SSD); the procedure includes: B30) training of the generative adversarial network (GAN) by alternating training of the discriminator (GD, LD) and the generator (G); wherein the generator (G) is trained to generate, for each input image from a lighting training set of input images (II2) of one or more scenes captured or computed under any lighting conditions, a modified image (MI) that has the same scene as the represents respective input image but under predetermined first lighting conditions; wherein the generator (G) is trained using a generator loss function (G_loss) comprising a detector loss term (SSD loss) calculated on the basis of an output (PBB) of the image-based detector (SSD). training procedure claim 1 , the discriminator (D) comprising two sub-neural networks, namely a global discriminator (GD) and a local discriminator (LD); the global discriminator is a neural network arranged to output an estimate based on the entire input image as to whether the input image is a fake or a true image; the local discriminator (LD) comprises a pre-processing unit (LD1) with a convolution layer and a main processing unit (LD2); the pre-processing unit is arranged to output patches, each patch being a representation of a sub-patch of the input image; and the main processing unit comprises a neural network and is arranged to output an estimate based on the patches as to whether the input image is a fake or a true image. procedure after claim 1 or 2 , wherein the image-based detector (SSD) is an object detector, in particular a neural network arranged to detect objects in an image. Procedure according to one of Claims 1 until 3 , wherein the generator (G) is an image coder-decoder, in particular a U-network. Procedure according to one of Claims 1 until 4 , wherein the generator loss function (G_loss) further accounts for a perceptual loss, the perceptual loss being an image-based function that increases as a sharpness loss in the image increases. procedure after claim 5 , where the image-based function is based on information each output by two different layers of a convolutional neural network. Procedure according to one of Claims 1 until 6 , further comprising: B20) creating a generator training set (GAN-TS), the generator training set comprising: as input, images (II2) of at least one scene, the at least one scene being imaged under multiple lighting conditions; as a basic truth, images (GTI) of at least one scene under the first lighting conditions. Computer-implemented method for training an image-based neural network (G-SSD), comprising a generator (G) coupled to an image-based detector (SSD), wherein the image-based detector (SSD) is a neural network arranged to process a modified image of the scene output by the generator (G); the generator is an image codec arranged to input an input image (111, 112) of a scene and to output a modified image (MI) of the scene suitable for processing by the image-based detector (SSD); the generator (G) is part of a generative adversarial network (GAN) comprising the generator (G) and a discriminator (GD, LD); the method comprising: B) training the generative adversarial neural network using the training method of any one of Claims 1 until 7 ; wherein the method is implemented such that during the training of the generative adversarial neural network generator (G) the loss is calculated using the loss function of the image-based detector (SSD). Computer-implemented method for training an image-based neural network claim 8 , further comprising repeating step B) of training the generative adversarial neural network (GAN) and a step (C20) of training the image-based detector (SSD) at least twice. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to perform steps of the method according to any one of Claims 1 until 9 to execute. Non-transitory computer-readable medium on which a computer program claim 10 is saved. Data processing device set up to carry out the steps of the method according to one of Claims 1 until 9 to execute. A modified image generation module (150) arranged to generate a modified image of a scene based on an initial image of the scene, the modified image generation module comprising a generator (G) that is a generative adversarial network generator (GAN) is obtained using the method of any one of Claims 1 until 7 is trained. A system (150) for detecting features in an image, comprising: a modified image generation module according to Claim 13 and an image-based detection module comprising an image-based detector (SSD) configured to process the modified image of the scene output by the modified-image generation module.

Images