DE102020110243A1 - Computer-implemented method for data augmentation - Google Patents

Abstract

The invention relates to a computer-implemented method for data augmentation, comprising the steps of: providing (21) scene image data which represent a first scene, the scene image data including spatial depth information; Providing (22) object image data which represent at least one object (3); Generating (23), as a function of the scene image data, an insertion map which indicates suitable positions at which an object can be inserted into the first scene; Using the insertion map, selecting (24) a suitable position for inserting the at least one provided object (3) into the first scene; Mixing (25) the scene image data and the object image data and thereby generating augmented scene image data which represent a second scene, the second scene comprising the first scene together with the at least one object (3) inserted at the selected position.

Description

The invention relates to a computer-implemented method for data augmentation. Such a method can be used, for example, in the field of autonomous driving for generating synthetic training data for artificial neural networks. The invention also relates to a computing device and a computer program for executing such a method, as well as a computer-readable (storage) medium which comprises instructions for executing such a method.

In the transition from pure research to application, autonomous driving must be able to deal with a wide range of traffic scenarios that occur in the real world. This represents one of the most important challenges in the development of autonomous vehicles and their perceptual components. The ability to master previously unknown situations will determine whether such a self-driving vehicle is able to drive completely autonomously.

Artificial neural networks have significantly improved the generalization capabilities of the perceptual systems of today's autonomous vehicles. However, it is well known that their performance and their ability to generalize are very data-dependent. This fact has a strong influence on the development and quality assurance of perception algorithms. To ensure that a specific scenario can be handled by a perception model, it should already be present in a training data set that is used.

Developers continuously improve their models by capturing new scenarios in road traffic and integrating them into the model by means of fine-tuning. This approach requires the annotation of each newly recorded scene, i.e. individual objects in the respective scene have to be provided with semantic labels in a complex manner (mostly by hand).

This procedure, which is based on the acquisition of complete real scenes, reaches its limits, particularly in connection with so-called “long-tail” scenarios, which occur very rarely and are difficult to capture. The same applies, for example, to near-miss scenarios or comparable scenarios that cannot be reproduced for ethical reasons, for example because they would endanger vulnerable road users (VRUs).

The use of synthetically generated data appears to be very promising for the examples mentioned, since any rare scenario or near-miss scenario can be simulated safely and as required. The idea of using simulated data has been addressed by many researchers in their work (see e.g. Ref. [4], [5], [35], [24]).

Because of the extreme complexity of the data and the enormous variety of possible scenarios, the idea of synthetic data generation is particularly attractive for the field of autonomous driving. The use of synthetic data in the field of autonomous driving has a relatively long history. Ref. [26] asked whether models trained with synthetic data for pedestrian detection could also work in real scenarios, and Ref. [33] used 3D models to learn 2D shapes based on natural images can be adjusted.

Aside from autonomous driving, synthetic data has also been used in various other areas. In order to accurately predict a 3D human pose, Ref. [30] used a synthetic training data set with a high variability to learn models that are invariant to poses, the observer's point of view, and other factors. Ref. [36] provided an entire data set, called SURREAL, based on realistic human augmentations. In contrast to the 3D pose estimation, Ref. [3] and [34] used 3D models for multi-category 2D object recognition.

In connection with autonomous driving, virtual simulators, such as in Ref. [7] and CARLA in Ref. [12], have been used. More recently, several research papers have presented synthetic data sets made up of rendered images. One of them is the data set SYNTHIA (see Ref. [29]), which provides 9,400 frames of scenes in different lighting and weather conditions. Another data set (see Ref. [28]) was generated using a video game engine to annotate 25,000 images. Ref. [14] simulated a clone of the real data set KITTI (see Ref. [15]).

Some work made the task easier and rendered individual objects instead of rendering the whole image. For example, Ref. [17] used simulated pedestrians to use a pedestrian detector for a To learn the monitoring system. Another use of virtual humans has been disclosed in Ref. [9] aimed at estimating a human pose. MixedPeds from Ref. [10] made use of the idea of pedestrian augmentation in the field of autonomous driving. Ref. [20] specialized in synthetic pedestrians in unexpected traffic situations, with real data being enriched, and Ref. [2] used vehicle augmentation for vehicle instance segmentation. An end-to-end trainable network from Ref. [23] dealt with the spatial and semantic correctness when inserting objects into the picture.

The approaches mentioned above have produced very useful training data; however, these are still distinguishable from actual "real-world" images. This is related to a so-called domain shift, which is introduced by the simulation. This means that there are generally so-called “domain gaps” between synthetic and real data (domains). In particular, simulated data typically differ from real data in terms of data distribution and the appearance of contents. This phenomenon is commonly referred to as a covariate shift. It is considered to be the main reason that models that were only trained with synthetic data decrease significantly in their performance compared to models that were trained with real data when the models are evaluated with real data (cf. Ref. [13]) ). For example, Ref. [28] reports a reduction in the size meanloU by almost 20% for a segmentation model (see Ref. [22]) when this is done with the data set CamVid (train) (see Ref. [6]) and a synthetic data set (see Ref. [28]), but was evaluated with the data set CamVid (val).

Due to the domain shift or the covariate shift, the approach, which in itself appears very promising, of using synthetic training data in the development of autonomous driving, is rarely used in practice.

In order to solve the above-mentioned problem, many researchers are trying to minimize the discrepancy between synthetic and real domains by means of generative networks. Most of them make use of adversarial training (cf. Ref. [16], [31], [38]), which, however, can itself be subject to a covariate shift: Although generating adversarial networks (English: generative adversarial networks - GAN) learn to deliver visually realistic data, such models usually suffer from a semantic mismatch and tend to integrate visual artifacts into generated samples. Therefore, much research focuses on improving adversarial networks by incorporating certain restrictions in order to compensate for inconsistencies introduced.

Some sophisticated approaches have tried to learn the appearance of scenes and directly synthesize target-like images. The vast majority of these approaches are based on generative models and make use of a GAN architecture. Ref. [1] introduced an augmentation network that synthesizes geometrically consistent and realistic insertions of vehicles. Other work has focused on creating the whole image in such a way that it is similar to the target data. For example, with Pix2Pix in Ref. [25], conditioning of an opponent's training according to the prior information about the semantic layout was introduced. Another conditioning strategy was suggested in Ref. [27], using predefined patches from a database to create a canvas. Ref. [37] has improved "Conditional GANs" and integrated instance information together with the possibility of image manipulation. Ref. [38] added a cycle loss in addition to the adversarial loss in order to achieve stable and consistent image transmission results. Further works, such as Ref. [25], [32], which were based on a tandem of cycle loss and opposing loss, attempted to disentangle the appearance and content by learning the latent representation space.

It is the object of the present invention to specify an improved method for data augmentation.

This object is achieved by a computer-implemented method, a computing device, a computer program and a computer-readable (storage) medium according to the independent patent claims. Advantageous embodiments are specified in the dependent claims.

It should be noted that additional features of a patent claim dependent on an independent patent claim without the features of the independent patent claim or in Combination with a subset of the features of the independent claim can form a separate invention that is independent of the combination of all the features of the independent claim and can be made the subject of an independent claim, a divisional application or a subsequent application. This applies equally to technical teachings described in the description, which can form an invention that is independent of the features of the independent patent claims. For example, a (domain) adaptation step of the type as claimed in one of the dependent claims 7-12 can in principle also be carried out independently of the specific steps specified in claim 1 in order to change any augmented image data.

A first aspect of the invention relates to a computer-implemented method for data augmentation. The method can, for example, be part of a more comprehensive data processing pipeline, which can be set up in particular for the generation of augmented (e.g. at least partially synthetic) training data for training VRU recognition for autonomous driving.

The inventive method according to the first aspect of the invention comprises the steps of: providing scene image data which represent a first scene, the scene image data including spatial depth information; Providing object image data which represent at least one object; Generating, as a function of the scene image data, an insertion map which indicates suitable positions at which an object can be inserted into the first scene; Using the insertion map, selecting a suitable position for inserting the at least one provided object into the first scene; Mixing the scene image data and the object image data and thereby generating augmented scene image data which represent a second scene, the second scene including the first scene together with the at least one object inserted at the selected position.

The scene shown can be a road traffic scene, for example. The at least one inserted object can be, for example, a VRU, such as a cyclist or a pedestrian. However, it is also possible that the at least one object represents another vehicle, such as a car or truck. It is therefore particularly within the scope of the invention that data augmentation can be used to generate tailor-made traffic scenarios with VRUs. In this way, for example, pedestrian detection used in the context of autonomous driving can be improved.

According to the invention, an insertion map is initially generated which indicates positions in the first scene which are suitable for the geometrically and / or semantically correct insertion of the at least one object. Such an insertion map can also be referred to using the English term “spawn map”, which comes from 3D gaming.

The insertion map preferably represents a (free) ground level (English: ground level) of the first scene. can be placed semantically correct.

The subsequent mixing of the scene image data and the object image data, which can also be referred to as "blending", includes, for example, a rendering step that can be carried out using standard methods known per se to the person skilled in the art, such as ray tracing.

The result is augmented scene image data that represent a second scene into which the object was inserted, preferably geometrically and / or semantically correct, namely at a position selected on the basis of the insertion map. Inserting the object into the scene can also be referred to as "in-painting" in English.

In a preferred embodiment, the scene image data, together with the spatial depth information contained therein, is based on recording the scene by means of a stereo camera and / or by means of a LIDAR system. Accordingly, according to a development, the method can additionally also include the generation of the scene image data by means of a stereo camera and / or a LIDAR system.

Provision can be made for the scene image data to be present in at least one processing step (ie at least in an intermediate step or as an intermediate stage of data processing) in the form of a so-called disparity map. Such disparity maps are known per se to the person skilled in the art, for example in connection with the derivation of depth information from image data from stereo cameras. In this case, depth or distance information relating to detected objects can be derived from an apparent pixel difference or shift between a pair of stereo images. In this comparison, objects that are close to the stereo camera seem to “jump” a considerable distance, whereas objects further away “move” only a little. The apparent “movement” of the corresponding pixels is the disparity shown in the disparity map.

It is also possible for the scene image data to be present in at least one processing step (i.e. at least in an intermediate step or as an intermediate stage of the data processing) in the form of a depth map. The depth map can be derived (e.g. estimated) from a disparity map of the type mentioned above.

Alternatively or additionally, it can optionally be provided that the scene image data are present in at least one processing step (i.e. at least in an intermediate step or as an intermediate stage of the data processing) in the form of a point cloud. The point cloud can in particular be a 3D point cloud (such as a stereo point cloud). The point cloud can be reconstructed from a disparity map and / or from a depth map of the type mentioned above, e.g. by estimating depth information for each pixel.

In the combination of some of the above-mentioned aspects, an advantageous embodiment of the method according to the invention can thus implement a data processing pipeline in which the generation of the insertion map comprises the following steps in particular: Creation of a disparity map which contains spatial information about the scene and which is based, for example, on is created from detection data from a stereo camera and / or a LIDAR system; Generating a depth map based on the disparity map; optionally generating a 3D point cloud based on the depth map; and generating a spawn map on the basis of the point cloud, wherein the spawn map can in particular indicate a (free) ground plane on which the at least one object can be inserted geometrically and / or semantically correct.

As already mentioned above, the mixing of the scene image data and the object image data can comprise a rendering step. Such a rendering step can be carried out using a (virtual) render camera, for example. In the event that the scene image data are based on a capture of the scene by means of a (real) camera, such as a stereo camera, it is preferred that the render camera is calibrated with one or more calibration parameters that were used when capturing the scene on the (real) camera ) Camera were set. This can help improve the visual correctness of the augmentation process.

It is also within the scope of the invention that the method further comprises creating and / or updating (i.e. updating) a collision map, the collision map indicating which positions in the second scene are already occupied by inserted objects. This means that the collision map can be used to track which positions in the scene are occupied and which, in return, can still be occupied. For example, every pedestrian who is placed in the scene expands the collision map so that an overlap of the corresponding graphical networks (English: Meshes) that represent the pedestrian with networks that belong to objects inserted afterwards can be prevented.

In one embodiment, several objects are accordingly inserted into the first scene, a suitable position for inserting the object being selected using the collision map in which at least one previously inserted object is recorded. The collision map shows, e.g. in a free space representation, suitable positions that are suitable for inserting the (further) object, taking into account any previously inserted objects.

The above-described steps and variants of the method according to the invention for data augmentation enable in particular a geometrically and / or semantically correct insertion of an object into a scene. For example, these steps as a whole can implement a phase of a more comprehensive data processing pipeline (or a part thereof), which can also be referred to as the augmentation phase.

According to one development, the method also includes a (domain) adaptation step, in which an appearance of the at least one inserted object in the second scene is changed by means of a GAN method.

The adjustment step is used to give the inserted object a more realistic appearance, adapted to the surrounding scene, in order to bridge a domain gap if possible. This can relate, for example, to aspects of a more realistic lighting that is part of the adjustment step eg in the form of removed and / or added shadows and / or points of light or light spots can be taken into account.

The adaptation step can, for example, be carried out in the context of a domain adaptation phase of the above-mentioned data processing pipeline, which is subordinate to the augmentation phase. Such a domain adaptation phase can, in accordance with some embodiments, also be referred to as an appearance learning phase.

In a preferred embodiment, within the scope of the GAN method, a generator generates changed image data on the basis of the augmented scene image data. The changed image data can be present in particular in the form of one or more changed frames. For example, after a corresponding GAN algorithm has converged, it is possible in this way to obtain adapted image data, in particular one or more adapted frames, in which the inserted object has a more realistic or plausible appearance in the context of the surrounding scene.

With regard to a GAN architecture used here, it is provided according to an advantageous embodiment that a multi-discriminator is used in the context of the GAN method, which includes at least two partial discriminators. Each of the partial discriminators evaluates one of several disjoint partial areas of a frame generated by the generator which contains the changed image data (or at least a part thereof). In accordance with some embodiments, this can in particular mean that each partial discriminator is exclusively responsible for its own partial area, i.e. there is no provision for two partial discriminators to evaluate the same partial area.

The disjoint sub-areas are preferably assigned to a respective semantic class (such as "street", "pedestrian", "vegetation" etc.).

It is further preferred that each of the partial discriminators is designed as a convolution neural network (CNN) which comprises at least one class-specific masking layer. The masking layer can in particular be assigned to a specific semantic class. The effect of the masking layer can in particular be such that at the input of the relevant partial discriminator, those pixels that belong to irrelevant classes (i.e. not to the specific class assigned to the masking layer or the partial discriminator) are set to 0.

In an advantageous further development of this embodiment variant, one, several or all of the partial discriminators (each) comprise several convolution layers, with a sampled version of the original masking layer in front of each subsequent convolution layer. is applied.

According to a second aspect of the invention, a computing device is proposed, the computing device being designed to carry out a method according to the first aspect of the invention. The computing device can be designed, for example, in the form of one or more processors (such as CPUs and / or GPUs) on which the necessary computing operations for carrying out the method run.

A third aspect relates to a computer program which comprises commands which, when the computer program is executed by a computing device, cause it to execute a method according to the first aspect of the invention.

A fourth aspect of the invention relates to a computer-readable (storage) medium which comprises instructions which, when executed by a computing device, cause the computing device to execute a method according to the first aspect.

It goes without saying that the computing device mentioned above in connection with the third and fourth aspect of the invention can in particular be a computing device according to the second aspect of the invention.

• 1 zeigt ein schematisches Ablaufdiagramm eines computerimplementierten Verfahrens zur Datenaugmentation gemäß einer oder mehreren Ausführungsformen.
• 2A zeigt beispielhaft und schematisch eine Punktwolke, die aus einer Tiefenkarte einer erfassten ersten Szene rekonstruiert wurde, mitsamt einem in die erste Szene einzufügenden Fußgänger.
• 2B zeigt beispielhaft und schematisch eine auf der Grundlage der Punktwolke aus 2A abgeschätzte Einfügungskarte (Spawn Map) der ersten Szene, mitsamt einer (freien) Bodenebene, auf welcher der Fußgänger platziert werden kann.
• 2C zeigt beispielhaft und schematisch eine zweite Szene, die die erste Szene mitsamt dem eingefügten Fußgänger umfasst, als das Ergebnis eines Mischens der Szenenbilddaten der ersten Szene und der Objektbilddaten des Fußgängers.
• 3 zeigt beispielhaft und schematisch eine Multi-Diskriminator-Architektur mit einem Generator und drei klassenspezifischen Teildiskriminatoren, die im Rahmen eines GAN-basierten Verfahrens zur Domänenanpassung eingesetzt werden kann, um das Erscheinungsbild eingefügter Objekte in einem augmentierten Datensatz zu verändern.
• 4 zeigt eine beispielhafte und schematische Überblicksansicht eines klassenspezifischen Diskriminators in Form eines faltenden neuronalen Netzwerks, wobei nach jedem Faltungsblock jeweils eine (ggf. heruntergesampelte) Maskierungsschicht eingefügt ist.
• 5A-C veranschaulichen beispielhaft und schematisch ein Ergebnis einer Datenaugmentation mit anschließender Domänenanpassung mittels einer GAN-Multi-Diskriminator-Architektur.

The invention will now be explained in more detail using exemplary embodiments and with reference to the accompanying drawings. The features and feature combinations mentioned in the description and / or shown alone in the drawings are not only in the respectively specified combination, but can also be used in other combinations or on their own without departing from the scope of the invention.

• 1 shows a schematic flow diagram of a computer-implemented method for data augmentation according to one or more embodiments.
• 2A shows, by way of example and schematically, a point cloud that was reconstructed from a depth map of a recorded first scene, together with a pedestrian to be inserted into the first scene.
• 2 B shows by way of example and schematically one based on the point cloud 2A Estimated spawn map of the first scene, including a (free) ground plane on which the pedestrian can be placed.
• 2C shows, by way of example and schematically, a second scene that includes the first scene together with the inserted pedestrian as the result of mixing the scene image data of the first scene and the object image data of the pedestrian.
• 3 shows an example and schematically a multi-discriminator architecture with a generator and three class-specific partial discriminators, which can be used in the context of a GAN-based method for domain adaptation in order to change the appearance of inserted objects in an augmented data set.
• 4th shows an exemplary and schematic overview view of a class-specific discriminator in the form of a convolutional neural network, with a masking layer (possibly downsampled) being inserted after each convolution block.
• 5A-C illustrate by way of example and schematically a result of a data augmentation with subsequent domain adaptation by means of a GAN multi-discriminator architecture.

An embodiment of the present invention is described below. Data augmentation is used to generate tailor-made traffic scenarios with vulnerable road users (VRUs). This can serve to improve pedestrian recognition.

In the context of the exemplary embodiment, a pipeline for the augmentation of the so-called cityscape data set (cf. Ref. [11]) with virtual pedestrians is presented (below in Section I). In order to make the augmentation achieved by the pipeline more realistic, a novel architecture of a generating network (English: generative network) for adversarial learning of the lighting conditions of the data set is proposed. The proposed approach is evaluated on the basis of semantic segmentation and instance segmentation tasks (below in Section II).

In the following description of the exemplary embodiment, reference is made to the method steps from the schematic flow charts shown in FIG 1 are shown by reference.

I. Solution

In the present exemplary embodiment, the proposed pipeline for data generation comprises two parts - a data augmentation phase and an appearance learning phase.

In the first phase (data augmentation phase), which includes the steps 21-25 in 1 virtual VRUs are inserted into an existing scene and the resulting rendering is then blended with the camera snapshot of the scene (the actual frame). This part focuses on the geometric and optical correctness of the blending process. Virtual pedestrians should only be placed in suitable locations in the scene, such as on sidewalks or streets. Placing the 3D models also requires avoiding collisions under the 3D models and with objects already in the scene (vehicles, trees, posts, etc.).

In order to achieve the optical correctness of the augmentation process, a render camera is calibrated with the parameters of the data record camera. This phase is based only on spatial information about the scene and ignores properties relating to the appearance, such as lighting conditions. The result of this phase are real data record frames with virtual pedestrians on them, which are clearly distinguishable thanks to their synthetic optics.

In the second phase (appearance learning phase) the step 26th in 1 The aim is to learn visual features of the record scenes and apply them to the inserted pedestrians so that they can look more realistic. This part is based on a dedicated adversarial network architecture, which can be assumed to be robust against a distribution discrepancy between synthetic and real domains. For example, a generator network from Ref. [38] can be used for the opposing architecture, which has shown very stable and visually appealing results. A multi-discriminator architecture proposed here, which is described in subsection IB, is used for the discriminator part.

A. Data augmentation

1) Spawn Map:

• - Bereitstellen 21 von Szenenbilddaten, die eine erste Szene darstellen, wobei die Szenenbilddaten räumliche Tiefeninformationen umfassen;
• - Bereitstellen 22 von Objektbilddaten, die wenigstens ein Objekt 3 darstellen;
• - Erzeugen 23, in Abhängigkeit der Szenenbilddaten, einer Einfügungskarte (Spawn Map), welche geeignete Positionen angibt, an denen ein Objekt in die erste Szene eingefügt werden kann;
• - Auswählen 24, anhand der Einfügungskarte, einer geeigneten Position zum Einfügen des wenigstens einen bereitgestellten Objekts 3 in die erste Szene;
• - Mischen 25 der Szenenbilddaten und der Objektbilddaten und hierdurch Erzeugen augmentierter Szenenbilddaten, die eine zweite Szene darstellen, wobei die zweite Szene die erste Szene mitsamt dem an der ausgewählten Position eingefügten wenigstens einen Objekt 3 umfasst.

The data augmentation phase includes the in 1 steps shown schematically:

• - Provide 21 scene image data representing a first scene, the scene image data including spatial depth information;
• - Provide 22nd of object image data containing at least one object 3 represent;
• - Produce 23 , depending on the scene image data, an insertion map (spawn map) which indicates suitable positions at which an object can be inserted into the first scene;
• - Choose 24 , based on the insertion map, a suitable position for inserting the at least one provided object 3 in the first scene;
• - Mix 25th the scene image data and the object image data and thereby generating augmented scene image data which represent a second scene, the second scene being the first scene together with the at least one object inserted at the selected position 3 includes.

The data augmentation phase accordingly includes in particular the assessment of a so-called spawn map (step 23 ). The main goal of this stage is to calculate suitable places where virtual pedestrians could be placed without the risk of colliding with other objects already present in the scene, such as buildings, vegetation or vehicles.

The estimation of the spawn map is based on spatial information about the scene. This information can typically be obtained directly from a lidar system or stereo camera. In the exemplary embodiment described here, spatial data are used which are reconstructed from disparity maps, the disparity maps being obtained from stereo cameras from cityscapes data sets.

Based on the disparity maps and on extrinsic and intrinsic parameters of the camera, a corresponding depth map is calculated for each individual image. 2A shows, by way of example and schematically, a point cloud that was reconstructed from a depth map of a recorded first scene, together with a pedestrian to be inserted into the first scene 3 .

Using a simple threshold-based heuristic together with an "isolation forest" algorithm, a ground level 4th (ground level) are estimated and outliers are eliminated. The ground level 4th , including sidewalks and road surface, allows pedestrians to be inserted without the risk of them being placed on unsuitable places such as buildings or walls. Using the spawn map that is now available, every location can be sampled in order to insert a pedestrian there in a semantically correct manner.

2 B shows by way of example and schematically one based on the point cloud 2A Estimated spawn map of the first scene, including a (free) ground plane 4th on which the pedestrian 3 can be placed

2) Collision Tracking:

In addition, the aim is to avoid overlapping between multiple virtual pedestrians as well as overlapping with other dynamic objects in the scene. Every location in the scene that is already occupied is tracked using a collision map. This creates a kind of free space representation. Initially, each reconstructed stereo point is inserted into the collision map if it does not belong to the spawn map. Every pedestrian which is placed in the scene, expands the collision map so that an overlap of the corresponding graphical networks (English: Meshes) that represent the pedestrian with networks that belong to objects inserted afterwards (e.g. other pedestrians) can be prevented.

In the last step 25th During the data augmentation phase, the slice with the 3D pedestrian model is rendered and mixed with an existing frame from the data set. It should be pointed out at this point that the render camera can be configured with the extrinsic and intrinsic parameters of the original camera that captured the frame, so that when mixing 25th to achieve optical correctness

2C shows, by way of example and schematically, a second scene, which includes the first scene together with the inserted pedestrian, as the result of the mixing 25th the scene image data of the first scene and the object image data of the pedestrian 3 . The mixing 25th (Blending / rendering), together with the described process of placing a pedestrian, including point cloud reconstruction, the estimation of the spawn map (step 23 ) and the collision map and the placed virtual pedestrian 3 is thus in the 2A-C illustrated in context.

B. Domain customization

In the domain adaptation model part of the pipeline, properties of the appearance of the target data set are to be learned and applied to the augmented models. This is done in an in 1 adjustment step shown 26th , in which an appearance of the at least one inserted object in the second scene is changed by means of a GAN-based method. The pedestrians should be given a realistic appearance, for example with regard to more realistic lighting.

1) Disappearing pedestrians:

The adaptation part is based on the widespread opposing training (see Ref. [16]), in which two neural networks act against each other. The first neural network is called a generator. It receives samples from source data or noise vectors in order to synthesize an image indistinguishable from the target data. In return, the second neural network, called the discriminator, tries to distinguish the synthesized images and the target images and punishes the generator accordingly. During training, the aim is to find a balance in a zero-sum game between the networks. This is continued until, in the ideal case, the prediction of the discriminator is only equivalent to random guessing. Such GAN methods are known per se to the person skilled in the art and as such are therefore not explained in detail here.

When the adversary learning converges, it provides a set of synthesized samples with a distribution similar to the target data. Although adversarial learning is a powerful tool for a domain matching task, it generally creates synthetic mismatches between source data and generated data. While the generator is encouraged to introduce perturbations in the source images, the discriminator identifies discrepancies between two domains very well and controls the perturbations to compensate for the discrepancies. However, the opposing training can introduce inconsistencies into the image data.

For example, in the exemplary embodiment presented here of a setup for adapting from adapted to real, the discriminator detects unfamiliar pedestrians with ease and encourages the generator to render them away in order to restore the output distribution. This has an undesirable effect in the data generation pipeline, as it can cause inserted objects to disappear again.

Such behavior can be observed in many Domain Adaptation Model setups. In general, such inconsistencies can strongly influence learning with synthesized data, since the so-called ground truth of the source no longer matches the corresponding images. The ground truth means "true" reference data, which can be in the form of semantic annotations, for example, which have been created manually or automatically in advance.

2) Multi-discriminator:

In the context of the present approach, a novel GAN model (GAN = Generative Adversarial Networks - generating opposing networks) is presented, which is designed to with Dealing with distribution discrepancies by splitting the discriminator into several discriminators. The motivation on which this method is based is to reduce the discriminator's freedom of choice in order to overcome the distribution discrepancy between the source data set and the target data set. It is proposed to eliminate the degree of freedom related to the content and instead let the discriminator focus only on certain features that affect the appearance of a particular sample.

For this purpose, the number of discriminators is increased, each of the discriminators evaluating only a part of the generated image which corresponds to a respective semantic class. This can be achieved by splitting the synthesized image into several disjoint areas (English: patches), each area containing a semantic class, and by only passing this respective area to a corresponding discriminator.

A full view of the multi-discriminator architecture is shown schematically in 3 shown. In this representation, a generator gene can be seen, to which an image of an augmented data set is transferred as input, with pedestrians in particular being augmented 3 were added. The generator Gen translates augmented images into realistic images, and its output is then split into several areas using semantic maps. Each of these areas is then made available to a dedicated partial discriminator Dis _road , Dis _ped , Dis _veg . For example, the partial discriminator Dis _{road is} assigned image areas that show a street, the partial discriminator Dis _ped is assigned image areas that depict pedestrians, and the partial discriminator Dis _veg is responsible for image areas that depict vegetation.

By training such a multi-discriminator, each of the split (partial) discriminators is specialized in the appearance features of a respective class. In order to achieve this, optimization is carried out using a special aggregated loss function (English also: objective).

This aggregated loss function, which is provided for the multi-discriminator, is composed of all class-specific loss functions: $${adv}_{}\left(G_r,{\mathrm{D.}}_r\right)={\displaystyle \sum _c^{N_c}\left({\mathrm{D.}}_r^c,G_r\right)}$$

In equation (1), c stands for a respective class and N _c is a total number of classes in a data set. In the present exemplary embodiment of a transmission of augmented images, the number of classes N _c = 2, so that the architecture is simplified to only two discriminators: One for the “pedestrian” class and one for the rest. The simplified version of the loss function according to equation (1 ) is reduced accordingly to: $${adv}_{}\left(G_r,{\mathrm{D.}}_r\right)=\left({\mathrm{D.}}_r^p,G_r\right)+\left({\mathrm{D.}}_r^r,G_r\right)$$

3) Masking:

In the present exemplary embodiment, the patch GAN architecture from Ref. [21] is adopted for each class-specific discriminator. To ensure that only single-class patches of the input image are passed to each of these discriminators, all pixels of irrelevant classes are masked by replacing them with 0 values. However, masking the input images alone does not reduce the propagation of the signal from the entire image, because due to the convolution, applying a mask to features during the forward pass through the neural network cannot prevent the propagation of the signal from undesired parts of the image.

Therefore, in the present solution, activations that originate in undesired areas of the input image are cut off at each level of the discriminator. For this purpose a masking layer (MaskLayer) is introduced, and after each convolutional layer of the discriminator a downsampled version is used M ^c applied to the original mask, where c denotes a respective class on the feature cards.

A more detailed overview of a class-specific discriminator architecture is given in 4th shown. This shows a class-specific partial discriminator for a class c (here: pedestrian) in the form of a convolutional neural network with a masking layer M ^c . After each convolution block there is a correspondingly downsampled version of the original masking layer M ^c inserted.

4) Cost sensitive loss:

aus der realen Domäne.The adaptation pipeline proposed here consists of pairs of input images x ⁱ of size 3 × h × w together with corresponding labels y ⁱ from the augmented data set ${\left\{\left(x_a^i,y_a^i\right)\right\}}_{i=1}^{N_a}\text{and couples}{\left\{\left(x_r^j,y_r^j\right)\right\}}_{j=1}^{N_r}$

from the real domain.

und im Label-Verteilungsraum y die Werte $y_a^i$

an, welche unabhängig und identisch verteilt sind und der gemeinsamen Wahrscheinlichkeitsverteilung P_a(x,y) folgen: $$\begin{array}{c}{x}_a^i\in X_a\subset X\subset {\mathbb{N}}^{3\times h\times w},i=\mathrm{0,}\mathrm{1,}\dots ,N_a\\ y_a^i\in Y_a\subset Y\subset {\mathbb{N}}^{h\times w},i=\mathrm{0,}\mathrm{1,}\dots ,N_a\\ {\left\{\left(x_a^i,y_a^i\right)\right\}}_{i=1}^{N_a}\sim P_a\left(x,y\right)\end{array}$$

The random variable x takes the values in the input distribution space x $x_a^i$

and the values in the label distribution space y $y_a^i$

which are independently and identically distributed and _{follow the common probability distribution P a} (x, y): $$\begin{array}{c}{x}_a^i\in X_a\subset X\subset {\mathrm{<figure-callout\; id="3"\; label="N"\; filenames="DE102020110243A1\_0020.png,DE102020110243A1\_0021.png"\; state="\{\{state\}\}">\mathbb{N}</figure-callout>}}^{3\times H\times w},i=\mathrm{0,}\mathrm{1,}...,N_a\\ y_a^i\in Y_a\subset Y\subset {\mathbb{N}}^{H\times w},i=\mathrm{0,}\mathrm{1,}...,N_a\\ {\left\{\left(x_a^i,y_a^i\right)\right\}}_{i=1}^{N_a}\sim {\mathrm{P.}}_a\left(x,y\right)\end{array}$$

folgen ihrerseits einer anderen Wahrscheinlichkeitsverteilung P_r: $$\begin{array}{c}{x}_r^j\in X_r\subset X\subset {\mathbb{N}}^{3\times h\times w},j=\mathrm{0,}\mathrm{1,}\dots ,N_r\\ y_r^j\in Y_r\subset Y\subset {\mathbb{N}}^{h\times w},\text{ j}=\mathrm{0,}\mathrm{1,}\dots ,N_r\\ {\left\{\left(x_r^j,y_r^j\right)\right\}}_{j=1}^{N_r}\sim P_r\left(x,y\right)\end{array}$$

The real samples $x_r^j$

in turn follow a different probability distribution P _r : $$\begin{array}{c}{x}_r^j\in X_r\subset \mathrm{<figure-callout\; id="3"\; label="X\; \subset \; N"\; filenames="DE102020110243A1\_0020.png,DE102020110243A1\_0021.png"\; state="\{\{state\}\}">X</figure-callout>}\subset {\mathbb{N}}^{}{}^{3\times H\times w},j=\mathrm{0,}\mathrm{1,}...,N_r\\ y_r^j\in Y_r\subset Y\subset {\mathbb{N}}^{H\times w},\text{j}=\mathrm{0,}\mathrm{1,}...,N_r\\ {\left\{\left(x_r^j,y_r^j\right)\right\}}_{j=1}^{N_r}\sim {\mathrm{P.}}_r\left(x,y\right)\end{array}$$

In the case of the class-specific discriminator, the error values are only calculated with reference to the relevant areas; the masks M ^c are therefore also used in the calculation of errors. Accordingly, the loss function of the class discriminator takes the following form: $$\begin{array}{l}\left({\mathrm{D.}}_r^c,G_r\right)={\mathbb{E.}}_{\left(x_r,y_r\right)}\left[\frac{1}{wH}{\Vert {\mathrm{D.}}_r^c\left(x_r,{\mathrm{M.}}^c\right)\circ {\mathrm{M.}}^c\left(y_r\right)\Vert }_{{\mathrm{F.}}^2}\right]\\ +{\mathbb{E.}}_{\left(x_a,y_a\right)}\left[\frac{1}{wH}{\Vert \left({\mathrm{D.}}_r^c\left(G_r\left(x_a\right),{\mathrm{M.}}^c\right)-J\right)\circ {\mathrm{M.}}^c\left(y_a\right)\Vert }_{{\mathrm{F.}}^2}\right]\end{array}$$

den Erwartungswert des jeweiligen Ausdrucks bei Mittelung über die betreffenden, im Index angegebenen Datensamples. Ohne den Einfluss der hier eingeführten Maske M^c würde die Gleichung (5) eine dem Fachmann an sich geläufige Formulierung der Verlustfunktion eines GANs angeben. Dabei ist $D_r^c$

eine Mapping-Funktion, die durch den jeweiligen Klassen-Diskriminator (d.h. durch das entsprechende neuronale Netzwerk) approximiert ist, und G_r ist eine Mapping-Funktion, die durch den Generator approximiert ist. Ferner bezeichnet J eine Einheitsmatrix der Dimension h × w und || · ||_{F 2} ist eine Frobenius-Norm.Here designated

the expected value of the respective expression when averaged over the relevant data samples specified in the index. Without the influence of the mask introduced here M ^c Equation (5) would indicate a formulation of the loss function of a GAN that is familiar to the person skilled in the art. It is ${\mathrm{D.}}_r^c$

a mapping function which is approximated by the respective class discriminator (ie by the corresponding neural network), and G _r is a mapping function which is approximated by the generator. Furthermore, J denotes an identity matrix of the dimension h × w and || · || _{F. 2} is a Frobenius norm.

It is intended that masked MSE be normalized to the size of the actual sample, as this will result in masks of different sizes contributing differently to each loss. The model is encouraged to learn more from samples with more prominent instances of classes of interest (e.g. pedestrians), thereby providing more information about appearance.

A naive application of this loss function in the procedure of the opponent's training naturally leads to the background, which normally occupies the largest part of an image (street, buildings, etc.), being emphasized more strongly. In the case of the augmented cityscapes data set, around 95% of the pixels represent non-pedestrian classes. This makes the task highly unbalanced, since non-pedestrian pixels would contribute 19 times the intensity to the overall loss function. It is therefore desirable Eliminate the effect of dominating classes across the data set when data is available that is unbalanced with regard to the classes.

This can be achieved by means of a weight factor λ as a hyper parameter. Experiments have shown that this works best when λ reflects the actual class ratio in the data set: $$\lambda =\frac{{\displaystyle {\sum }_y{\Vert {\mathrm{M.}}_p\left(y\right)\Vert }_1}}{{\displaystyle {\sum }_y{\Vert {\mathrm{M.}}_r\left(y\right)\Vert }_1}}$$

An analogous calculation is used in the case of a multi-discriminator with more than two classes of interest.

The overall cost-sensitive objective therefore takes the following form: $$\begin{array}{l}={\lambda }_{cyc}{cyc}_{}+{adv}_{}\left({\mathrm{D.}}_r^p,G_r\right)+\lambda {adv}_{}\left({\mathrm{D.}}_r^r,G_r\right)+{adv}_{}\left({\mathrm{D.}}_a^p,G_a\right)\\ +\lambda {adv}_{}\left({\mathrm{D.}}_a^r,G_a\right)\end{array}$$

den zyklischen Konsistenzverlust (cyclic-consistency loss) zusammen mit seinem Gewicht λ_cyc, wie in Ref. [38] eingeführt.Inscribed therein ${cyc}_{}$

the cyclic-consistency loss together with its weight λ _cyc , as introduced in Ref. [38].

Now that the loss function has been defined, the opponent's training procedure is carried out to optimize as follows: $$\underset{G_r,G_a}{{\displaystyle \text{min}}}\underset{{\mathrm{D.}}_r^p,{\mathrm{D.}}_a^p,{\mathrm{D.}}_r^r,{\mathrm{D.}}_a^r}{{\displaystyle \text{Max}}}\left(G_r,G_a,{\mathrm{D.}}_r^p,{\mathrm{D.}}_a^p,{\mathrm{D.}}_r^r,{\mathrm{D.}}_a^r\right)$$

II. Experimental evaluation

Experiments were carried out on the basis of the widely used, publicly accessible data set Cityscapes (cf. Ref. [11]). This provides all the data required for data generation: camera frames and disparity maps along with calibration parameters to enable augmentation, as well as the ground truth for the computer vision task to carry out evaluations.

A. Records

The data set to be used must meet several criteria: it must provide information about spatial properties of the scene to be augmented, and it must also provide the ground truth for performing the computer vision task and for evaluation with the data generated.

Cityscapes is an extensive computer vision dataset that provides 5000 camera snapshots with a size of 2048 × 1024 pixels together with dense labels for pixel segmentation. The annotations provide the ground truth for both the semantic segmentation and the instance segmentation. The data set is based on measurements in cities across Germany (and Strasbourg) and at different times of the year, weather and time of day conditions. The images are divided into the sets train, val and test with 2975, 500 and 1525 samples. In addition to the fine-meshed annotations, the data set provides 20,000 weakly annotated images; however, the latter were not used here. Semantic maps contain information about 30 classes, such as road, sidewalk, person, car, etc. and the instance segmentation includes 8 categories: person, rider, car, truck, bus, train, motorcycle, bicycle.

With the augmentation pipeline proposed here, the data set augmented citiyscapes is generated, which consists of 2975 images with a resolution of 2048 × 1024, with 5 virtual pedestrians being inserted at random. Both a semantic map and an instance map are generated for all images. The standard Cityscapes annotation format is followed for classes and categories.

After augmented citiyscapes has been generated, the GAN model multidiscriminator GAN, which was introduced in Section III-B, is trained for 200 epochs. The model was trained “from scratch”, with cyclic weight set to 10 and cost sensitive λ = 0.2. As in Ref. [38], a learning rate of 0.0002 is started and this is held constant for the first 100 epochs before falling to zero over a further 100 epochs. For the training, the pictures were increased by a factor 2 scaled down and no so-called random crops were used. (Typically, neural networks are not trained on the basis of images with full resolution, but rather on randomly selected image sections of lower resolution, which are known as random crops.)

Several experiments are carried out to evaluate the translated results and the results are assessed both qualitatively and quantitatively. An example of augmentation and customization is in the 5A-C illustrated schematically. In order to visualize the effects brought about by the present adaptation approach, both the augmented and the translated image are shown in schematic form together with manually selected crops where these effects are characteristic. So illustrated 5A an adapted scene with inserted pedestrians 3 . 5C illustrates schematically (within the scope of the possibilities of a black and white line drawing) the result of the domain adaptation. In 5B is an enlarged section on the left and right 5A respectively. 5C shown.

First of all, it can be stated that the multi-discriminator architecture presented here effectively helps in solving the problem of vanishing objects, which is due to the discrepancy between the distributions of the augmented data and the original data. There are no semantic inconsistencies between synthetic and translated samples, and the inserted objects remain in place. Another characteristic of the translated images that should be considered is the actual appearance of the objects being rendered. In the translated images, the rendered objects follow the color scheme of the entire target data set (in the black and white representation of the 5A-C this is not recognizable). Finally, lighting effects learned from the scenes were added as features to the transmitted images. This can be seen from the enlarged sections, which have inserted light spots and smoothed edges. It is, for example, based on a light spot inserted by the domain adaptation on the shoulder and chest of the pedestrian 3 It can be seen that the appearance of the pedestrians has been adapted to the scene and thus appears more realistic as a result. In the merely schematic line drawings according to 5B-C light spots are drawn darker than the respective surroundings for reasons of illustration (deviating from the actual image obtained in this experiment).

In order to carry out a qualitative evaluation of the results, the data generated are assessed with reference to two computer vision tasks: semantic segmentation and instance segmentation. Semantic segmentation predicts for each pixel which semantic class it belongs to, e.g. pedestrian or vehicle. If, however, individual instances are still differentiated within a class, one speaks of instance segmentation.

For the purpose of the evaluation, state-of-the-art computer vision algorithms such as Deeplabv3 (see Ref. [8]) and Mask-RCNN (see Ref. [18]) were used.

B. Instance segmentation

The transfer quality with regard to instance segmentation is evaluated using the standard metric COCO average precision (AP). One of the most powerful detection models, called Mask-RCNN (see Ref. [18]), is used, which is pre-trained on a COCO data set and fine-tuned on the augmented cityscapes data set. The evaluation is then carried out on the basis of 500 images from Cityscapes val. We follow the experimental set-up from the original work Ref. [18] and give the results of the instance segmentation in Table I. Table I: Results of the instance segmentation for Mask-RCNN, which was trained on cityscapes (above) and a data set generated according to the invention (below). method AP _avg AP50 _avg AP _person AP50 _person CS 31.8 59.0 33.0 67.7 invention 32.6 60.4 35.6 74.2

C. Semantic segmentation

The effectiveness of the generated data for the semantic segmentation task was assessed using the Deeplabv3 model. Similar to the previous experiment on instance segmentation, the baseline method is trained both on cityscapes (train) and on the generated data set. In both cases, the images are sampled down to 1024 × 512 pixels in a preprocessing step. The evaluation takes place on Cityscapes (val).

Deeplabv3 (see Ref. [8]) uses xception65 as the backbone. It was trained for 90K reps with batch 16 on 513 × 512 random crops. The learning rate in this case remained at 0.007. For the best snapshots, the meanloU metric is given in Table II. Table II: meanloU values for the prediction of the semantic segmentation by Deeplabv3, which was trained on Cityscape and the data set generated according to the invention method accuracy meanloU person CS 95.6 75.6 77.1 invention 95.3 75.3 77.3

_{In both evaluation experiments} , the generated data show a similar performance with regard to overall metrics such as meanloU and AP avg, but they show a slight improvement for the class of interest here - pedestrians. The improvement for AP _pedestrian in Mask-RCNN experiments is more than 2.5%. For Deeplab, the improvement in the pedestrian class is rather moderate (less than 0.5%). At the same time, the pixel accuracy and the meanloU metric show almost no reduction in performance (0.3%) across all 19 classes.

As an exemplary embodiment, a pipeline for data generation was presented, which comprises a pedestrian augmentation part and a data record appearance learning part, which is based on a novel class-specific multi-discriminator architecture. Experiments have shown that the proposed pipeline can generate semantically and geometrically consistent training images with target data set optics, which helps to bridge the domain gap between augmented and real data.

The exemplary embodiment presented deals in particular with pedestrian detection and specifies a pipeline for augmentation that allows a real data set, such as cityscapes (see Ref. [11]), to be expanded with virtual pedestrians in various scenarios. This pipeline enables geometrically correct inpainting of pedestrian CAD models in Cityscapes scenes. However, the augmentation in and of itself does not yet take into account the lighting conditions of the respective data set scene. Therefore, as a second contribution of the invention, a domain adaptation model is provided which is based on an opposing network and makes it possible to learn appearances and to make the CAD model more realistic.

Using a masking approach, the proposed model is robust against distribution discrepancies between the real and synthetic data sets, and it is able to produce consistent images with realistic lighting conditions and an appearance that matches the original data set.

The proposed method makes it possible to simulate various scenarios with VRUs without introducing a relevant discrepancy between simulation and reality (“simulation-to-real gap”).

credentials

• [1] Geometry Image Synthesis, 2018 .
• [2] Hassan Abu Alhaija, Siva Karthik Mustikovela, Lars Mescheder, Andreas Geiger and Carsten Rother. Augmented reality meets deep learning for car instance segmentation in urban scenes. In the British Machine Vision Conference (BMVC), 2017 .
• [3] Mathieu Aubry, Daniel Maturana, Alexei A. Efros, Bryan C. Russell and Josef Sivic. Seeing 3d chairs: Exemplar part-based 2d-3d alignment using a large dataset of cad models. 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 3762-3769, 2014 .
• [4] John L. Barron, David J. Fleet, Steven S. Beauchemin, and TA Burkitt. Performance of optical flow techniques. Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 236-242, 1992 .
• [5] Alberto Broggi, Alessandra Fascioli, Paolo Grisleri, Thorsten Graf and Marc-Michael Meinecke. Model-based validation approaches and matching techniques for automotive vision based pedestrian detection. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops, pages 1-1, 2005 .
• [6] Gabriel J. Brostow, Julien Fauqueur and Roberto Cipolla. Semantic object classes in video: A high-definition ground truth database. Pattern Recognition Letters, 30: 88-97, 2009 .
• [7] Chenyi Chen, Ari Seff, Alain L. Kornhauser, and Jianxiong Xiao. Deepdriving: Learning affordance for direct perception in autonomous driving. CoRR, abs / 1505.00256, 2015 .
• [8th] Liang-Chieh Chen, George Papandreou, Florian Schroff and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. CoRR, abs / 1706.05587, 2017 .
• [9] Wenzheng Chen, Huan Wang, Yangyan Li, Hao Su, Changhe Tu, Dani Lischinski, Daniel Cohen-Or and Baoquan Chen. Synthesizing training images for boosting human 3d pose estimation. 2016 Fourth International Conference on 3D Vision (3DV), pages 479-488, 2016 .
• [10] Ernest Cheung, Anson Wong, Aniket Bera, and Dinesh Manocha. Mixedpeds: Pedestrian detection in unannotated videos using synthetically generated human-agents for training. In AAAI, 2017 .
• [11] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. CoRR, abs / 1604.01685, 2016 .
• [12] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio López and Vladlen Koltun. CARLA: an open urban driving simulator. CoRR, abs / 1711.03938, 2017 .
• [13] Aysegul Dundar, Ming-Yu Liu, Ting-Chun Wang, John Zedlewski and Jan Kautz. Domain stylization: A strong, simple baseline for synthetic to real image domain adaptation. ArXiv, abs / 1807.09384, 2018 .
• [14] Adrien Gaidon, Qiao Wang, Yohann Cabon and Eleonora Vig. Virtualworlds as proxy for multi-object tracking analysis. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4340-4349, 2016 .
• [15] Andreas Geiger, Philip Lenz and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012 .
• [16] lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, ND Lawrence and KQ Weinberger, editors, Advances in Neural Information Processing Systems 27, pp. 2672-2680. Curran Associates, Inc., 2014 .
• [17] H. Hattori, VN Boddeti, K. Kitani, and T. Kanade. Learning scene-specific pedestrian detectors without real data. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3819-3827, June 2015 .
• [18] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross B. Girshick. Mask r-cnn. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2980-2988, 2017 .
• [19] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A. Efros and Trevor Darrell. Cycada: Cycle consistent adversarial domain adaptation. CoRR, abs / 1711.03213, 2017 .
• [20] Shiyu Huang and Deva Ramanan. Expecting the unexpected: Training detectors for unusual pedestrians with adverse imposters. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4664-4673, 2017 .
• [21] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Imageto-image translation with conditional adversarial networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5967-5976, 2016 .
• [22] Abhijit Kundu, Vibhav Vineet and Vladlen Koltun. Feature space optimization for semantic video segmentation. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3168- 3175, 2016 .
• [23] Donghoon Lee, Sifei Liu, Jinwei Gu, Ming-Yu Liu, Ming-Hsuan Yang and Jan Kautz. Context-aware synthesis and placement of object instances. In NeurIPS, 2018 .
• [24] Joerg Liebelt, Cordelia Schmid and Klaus Schertier. Viewpoint-independent object class detection using 3d feature maps, 2008 .
• [25] Ming-Yu Liu, Thomas Breuel and Jan Kautz. Unsupervised image-to-image translation networks. In I. Guyon, UV Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 700-708. Curran Associates, Inc., 2017 .
• [26] J. Marin, AM Lopez, D. Geronimo, and D. Vazquez. Learning appearance in virtual scenarios for pedestrian detection. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), volume 00, pages 137-144, 06 2010 .
• [27] Xiaojuan Qi, Qifeng Chen, Jiaya Jia and Vladlen Koltun. Semiparametric image synthesis. 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 8808-8816, 2018 .
• [28] Stephan R Richter, Vibhav Vineet, Stefan Roth and Vladlen Koltun. Playing for data: Ground truth from computer games. In European Conference on Computer Vision, pages 102-118. Springer, 2016 .
• [29] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez and Antonio M. Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016 .
• [30] Jamie Shotton, Ross B. Girshick, Andrew W. Fitzgibbon, Toby Sharp, Mat Cook, Mark Finocchio, Richard Moore, Pushmeet Kohli, Antonio Criminisi, Alex Kipman and Andrew Blake. Efficient human pose estimation from single depth images. IEEE transactions on pattern analysis and machine intelligence, 35 12: 2821-40, 2013 .
• [31] Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang and Russell Webb. Learning from simulated and unsupervised images through adversarial training. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2242-2251, 2016 .
• [32] Rui Shu, Hung H. Bui, Hirokazu Narui and Stefano Ermon. A DIRTT approach to unsupervised domain adaptation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings, 2018 .
• [33] Michael Stark, Michael Goesele and Bernt Schiele. Back to the future: Learning shape models from 3d cad data. In the Proceedings of the British Machine Vision Conference, pp. 106.1-106.11. BMVA Press, 2010. doi: 10.5244 / C.24.106 .
• [34] Baochen Sun and Kate Saenko. From virtual to reality: Fast adaptation of virtual object detectors to real domains. In the Proceedings of the British Machine Vision Conference. BMVA Press, 2014 .
• [35] Geoffrey R. Taylor, Andrew J. Chosak, and Paul C. Brewer. Ovvv: Using virtual worlds to design and evaluate surveillance systems. 2007 IEEE Conference on Computer Vision and Pattern Recognition, Pages 1-8, 2007 .
• [36] Gül Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev and Cordelia Schmid. Learning from synthetic humans. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4627-4635, 2017 .
• [37] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 8798-8807, 2017 .
• [38] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. 2017 IEEE International Conference on Computer Vision (ICCV), pages 2242-2251, 2017 .

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Non-patent literature cited

• Geometry Image Synthesis, 2018 [0105]
• Hassan Abu Alhaija, Siva Karthik Mustikovela, Lars Mescheder, Andreas Geiger and Carsten Rother. Augmented reality meets deep learning for car instance segmentation in urban scenes. In British Machine Vision Conference (BMVC), 2017 [0105]
• Mathieu Aubry, Daniel Maturana, Alexei A. Efros, Bryan C. Russell and Josef Sivic. Seeing 3d chairs: Exemplar part-based 2d-3d alignment using a large dataset of cad models. 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 3762-3769, 2014 [0105]
• John L. Barron, David J. Fleet, Steven S. Beauchemin, and T. A. Burkitt. Performance of optical flow techniques. Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 236-242, 1992 [0105]
• Alberto Broggi, Alessandra Fascioli, Paolo Grisleri, Thorsten Graf and Marc-Michael Meinecke. Model-based validation approaches and matching techniques for automotive vision based pedestrian detection. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops, pages 1-1, 2005 [0105]
• Gabriel J. Brostow, Julien Fauqueur and Roberto Cipolla. Semantic object classes in video: A high-definition ground truth database. Pattern Recognition Letters, 30: 88-97, 2009 [0105]
• Chenyi Chen, Ari Seff, Alain L. Kornhauser, and Jianxiong Xiao. Deepdriving: Learning affordance for direct perception in autonomous driving. CoRR, abs / 1505.00256, 2015 [0105]
• Liang-Chieh Chen, George Papandreou, Florian Schroff and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. CoRR, abs / 1706.05587, 2017 [0105]
• Wenzheng Chen, Huan Wang, Yangyan Li, Hao Su, Changhe Tu, Dani Lischinski, Daniel Cohen-Or and Baoquan Chen. Synthesizing training images for boosting human 3d pose estimation. 2016 Fourth International Conference on 3D Vision (3DV), pages 479-488, 2016 [0105]
• Ernest Cheung, Anson Wong, Aniket Bera, and Dinesh Manocha. Mixedpeds: Pedestrian detection in unannotated videos using synthetically generated human-agents for training. In AAAI, 2017 [0105]
• Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. CoRR, abs / 1604.01685, 2016 [0105]
• Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio López and Vladlen Koltun. CARLA: an open urban driving simulator. CoRR, abs / 1711.03938, 2017 [0105]
• Aysegul Dundar, Ming-Yu Liu, Ting-Chun Wang, John Zedlewski and Jan Kautz. Domain stylization: A strong, simple baseline for synthetic to real image domain adaptation. ArXiv, abs / 1807.09384, 2018 [0105]
• Adrien Gaidon, Qiao Wang, Yohann Cabon and Eleonora Vig. Virtualworlds as proxy for multi-object tracking analysis. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4340-4349, 2016 [0105]
• Andreas Geiger, Philip Lenz and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012 [0105]
• lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N.D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pp. 2672-2680. Curran Associates, Inc., 2014 [0105]
• H. Hattori, V. N. Boddeti, K. Kitani, and T. Kanade. Learning scene-specific pedestrian detectors without real data. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3819-3827, June 2015 [0105]
• Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross B. Girshick. Mask r-cnn. 2017 IEEE International Conference on Computer Vision (ICCV), pages 2980-2988, 2017 [0105]
• Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A. Efros and Trevor Darrell. Cycada: Cycle consistent adversarial domain adaptation. CoRR, abs / 1711.03213, 2017 [0105]
• Shiyu Huang and Deva Ramanan. Expecting the unexpected: Training detectors for unusual pedestrians with adverse imposters. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4664-4673, 2017 [0105]
• Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Imageto-image translation with conditional adversarial networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5967-5976, 2016 [0105]
• Abhijit Kundu, Vibhav Vineet and Vladlen Koltun. Feature space optimization for semantic video segmentation. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3168- 3175, 2016 [0105]
• Donghoon Lee, Sifei Liu, Jinwei Gu, Ming-Yu Liu, Ming-Hsuan Yang and Jan Kautz. Context-aware synthesis and placement of object instances. In NeurIPS, 2018 [0105]
• Joerg Liebelt, Cordelia Schmid and Klaus Schertier. Viewpoint-independent object class detection using 3d feature maps, 2008 [0105]
• Ming-Yu Liu, Thomas Breuel and Jan Kautz. Unsupervised image-to-image translation networks. In I. Guyon, U.V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 700-708. Curran Associates, Inc., 2017 [0105]
• J. Marin, A. M. Lopez, D. Geronimo, and D. Vazquez. Learning appearance in virtual scenarios for pedestrian detection. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), volume 00, pages 137-144, 06 2010 [0105]
• Xiaojuan Qi, Qifeng Chen, Jiaya Jia and Vladlen Koltun. Semiparametric image synthesis. 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 8808-8816, 2018 [0105]
• Stephan R Richter, Vibhav Vineet, Stefan Roth and Vladlen Koltun. Playing for data: Ground truth from computer games. In European Conference on Computer Vision, pages 102-118. Springer, 2016 [0105]
• German Ros, Laura Sellart, Joanna Materzynska, David Vazquez and Antonio M. Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016 [0105]
• Jamie Shotton, Ross B. Girshick, Andrew W. Fitzgibbon, Toby Sharp, Mat Cook, Mark Finocchio, Richard Moore, Pushmeet Kohli, Antonio Criminisi, Alex Kipman and Andrew Blake. Efficient human pose estimation from single depth images. IEEE transactions on pattern analysis and machine intelligence, 35 12: 2821-40, 2013 [0105]
• Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang and Russell Webb. Learning from simulated and unsupervised images through adversarial training. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2242-2251, 2016 [0105]
• Rui Shu, Hung H. Bui, Hirokazu Narui and Stefano Ermon. A DIRTT approach to unsupervised domain adaptation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings, 2018 [0105]
• Michael Stark, Michael Goesele and Bernt Schiele. Back to the future: Learning shape models from 3d cad data. In the Proceedings of the British Machine Vision Conference, pp. 106.1-106.11. BMVA Press, 2010. doi: 10.5244 / C.24.106 [0105]
• Baochen Sun and Kate Saenko. From virtual to reality: Fast adaptation of virtual object detectors to real domains. In the Proceedings of the British Machine Vision Conference. BMVA Press, 2014 [0105]
• Geoffrey R. Taylor, Andrew J. Chosak, and Paul C. Brewer. Ovvv: Using virtual worlds to design and evaluate surveillance systems. 2007 IEEE Conference on Computer Vision and Pattern Recognition, Pages 1-8, 2007 [0105]
• Gül Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev and Cordelia Schmid. Learning from synthetic humans. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4627-4635, 2017 [0105]
• Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 8798-8807, 2017 [0105]
• Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. 2017 IEEE International Conference on Computer Vision (ICCV), pages 2242-2251, 2017 [0105]

Claims (15)

Computer-implemented method for data augmentation, comprising the steps of: - providing (21) scene image data which represent a first scene, the scene image data including spatial depth information; - providing (22) object image data which represent at least one object (3); - Generating (23), as a function of the scene image data, an insertion map which indicates suitable positions at which an object can be inserted into the first scene; - Using the insertion map, selecting (24) a suitable position for inserting the at least one provided object (3) into the first scene; - Mixing (25) the scene image data and the object image data and thereby generating augmented scene image data which represent a second scene, the second scene including the first scene together with the at least one object (3) inserted at the selected position. Procedure according to Claim 1 , the scene image data being based on a recording of the scene by means of a stereo camera and / or a LIDAR system. Method according to one of the preceding claims, wherein the scene image data is present in at least one processing step in the form of a disparity map. Method according to one of the preceding claims, wherein the scene image data are present in at least one processing step in the form of a depth map. Method according to one of the preceding claims, wherein the scene image data are present in at least one processing step in the form of a point cloud. Method according to one of the preceding claims, wherein the insertion map represents a ground plane (4) of the first scene, wherein the ground plane (4) in particular comprises road surface (s) and / or sidewalk (s) present in the first scene. Method according to one of the preceding claims, further comprising an adaptation step (26) in which an appearance of the at least one inserted object (3) in the second scene is changed by means of a GAN method. Procedure according to Claim 7 , with a generator (gene) generating changed image data on the basis of the augmented scene image data as part of the GAN method. Procedure according to Claim 8 In the context of the GAN method, a multi-discriminator is used which comprises at least two partial discriminators (Dis _road , Dis _ped , Dis _veg ), and each of the partial discriminators (Dis _road , Dis _ped , Dis _veg ) has a sub-area assigned to it evaluated by several disjoint sub-areas of a frame generated by the generator (gene). Procedure according to Claim 9 , the disjoint sub-areas being assigned to a respective semantic class. Procedure according to Claim 9 or 10 , wherein at least one of the partial discriminators (Dis _road , Dis _ped , Dis _veg ) is designed as a folding neural network which comprises at least one class-specific masking layer (M ^c ). Procedure according to Claim 11 , wherein at least one partial discriminator (Dis _road , Dis _ped , Dis _veg ^{) comprises several folding layers, and wherein a sampled version of the original masking layer (M c} ) is used before at least one subsequent folding layer, in particular before each subsequent folding layer. Computing device, wherein the computing device is designed to carry out a method according to one of the preceding claims. Computer program, comprising instructions which cause the computer program to be executed by a computing device, a method according to one of the Claims 1 until 12th to execute. Computer-readable (storage) medium, comprising instructions which, when executed by a computing device, cause them, a method according to one of the Claims 1 until 12th to execute.

Images