WO2019192746A1

WO2019192746A1 - Image rendering from texture-less cad models

Info

Publication number: WO2019192746A1
Application number: PCT/EP2018/082794
Authority: WO
Inventors: Benjamin PLANCHE; Sergey Zakharov; Andreas Hutter; Slobodan Ilic; Ziyan Wu
Original assignee: Siemens Aktiengesellschaft
Priority date: 2018-04-06
Filing date: 2018-11-28
Publication date: 2019-10-10

Abstract

The invention relates to a method for rendering an augmented image (213) of an object from a texture-less CAD model (211) of the object, the method comprising the steps of: - generating a normal map (212) of the object from the CAD model (211) for a specific viewpoint, and - augmenting the normal map (212) by applying one or more actions defined in an augmentation pipeline (A) to the normal map (212). Furthermore, the invention relates to a renderer for render- ing an augmented image (213) of an object from a texture-less CAD model (211), a computer program product and a computer-readable storage medium.

Description

Specification

Image rendering from texture-less CAD models

The invention relates to a concept for rendering an augmented image of an object from a texture-less CAD model of the ob ject. The invention is in particular relevant for the field of object recognition.

Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba and Pieter Abbeel presented the concept of "domain randomization" in their scientific article "Domain Randomiza tion for Transferring Deep Neural Networks from Simulation to the Real World", first published on arXiv with the ID

1703.06907vl . Herein, it is proposed to generate a large amount of synthetic, augmented color images based on texture less 3D CAD models. The texture and further features of the output image such as the background or shadows are chosen en tirely randomly. Their concept is based on the hypothesis that if the variability in simulation of the augmented images is large enough, a recognition system trained in simulation would generalize to the real world with no additional train ing necessary. In the case of texture, one could say that the recognition system becomes "texture-blind" and focuses on other aspects in the image for recognizing the object to be recognized.

According to the cited article of Tobin et al . , this concept has been proven to work quite well in certain tests. A draw back of the concept is, however, that the generation of the cluttered color images is carried out by the graphics pro cessing unit (GPU) of the computer. In other words, Tobin et al . use a 3D engine to directly render random color images from the 3D CAD models, especially randomizing the object's texture .

The operation of using a 3D engine to render color images from 3D models is heavy and must run of the GPU for efficien- cy. As the training of neural networks is also generally done on GPU(s), this means that either the data generation must be done beforehand (which means generating and storing a finite dataset, instead of generating new data every training itera tion) , or that network training and data generation must com pete for the GPU resource, which is far from optimal for lim ited GPU setups .

Thus, there is the desire to develop a concept how to be able to generate a large amount of augmented images based on CAD models, without burdening the GPU of the system.

This problem is solved by the subject-matters of the inde pendent claims. Advantageous details or specifications are disclosed in the dependent claims.

Accordingly, there is provided a method for rendering an aug mented image of an object from a texture-less CAD model of the object, wherein the method comprises:

- generating a normal map of the object from the CAD model for a specific viewpoint, and

- augmenting the normal map by applying one or more actions defined in an augmentation pipeline to the normal map.

"Rendering", which is also referred to as image synthesis, is the automatic process of generating an image from a two- dimensional (2D) or three-dimensional (3D) model by means of computer programs .

An "augmented" image of an object is understood as an image wherein some kind of disturbance, in other words nuisance, has been added to. The type of augmentation, in other words "augmentation modality" comprises, for instance, adding a background behind the object which is depicted in the image, adding noise in the illustration of the object and/or the background, shadows, surface texture, blurring, rotation, translation, flipping or resizing of the object, partial oc clusions of the object and changing the color of the object. In general, a normal map or a depth map which is directly ob tained from the CAD model and has not been augmented, is also referred to as a "clean" normal or depth map, respectively.

A "CAD model" (computer-aided design model) , which is some times also referred to as "CADD model" (computer-aided design and drafting model) , is understood as a design for which cre ation computer systems including workstations have been used. CAD output is often in the form of electronic files for print, machining, or other manufacturing operations. CAD mod els may be two-dimensional (2D) or three-dimensional (3D) .

A "texture-less CAD model" is understood as a CAD model which only contains pure semantic and geometrical information, but no information regarding e.g. its appearance (color, texture, material type) , scene (position of light sources, cameras, peripheral objects) or animation (how the model moves, if ap plicable) . It will be one of the tasks of the augmentation pipeline to add random appearance or scene features to the clean normal map of the texture-less CAD model.

A normal map is a representation of the surface normals of a 3D model from a particular viewpoint, stored in a two- dimensional colored image, also referred to as an RGB (i.e. red color/ green color/ blue color) image. Herein each color corresponds to the orientation of the surface normal. A "nor mal map" therefore creates the impression of a three- dimensional image, but only occupies little storage space. Normal mapping, sometimes also referred to as "Dot3 bump map ping" is a known technique from 3D computer graphics. It is primarily used in video games.

"Augmenting" refers to the transformation of a normal map in to a color image by means of adding certain types of nuisanc es (or "clutter") to the clean normal map. Note that by transforming a normal map into a color image, on the one hand information is lost as no more precise 3D representation of the object surface is present at the color image. On the oth- er hand, external information is added to the image, due to e.g. lighting condition and texture information. If, however, the added information is randomly defined, it is more noise than semantic information.

A key aspect of the present invention is to introduce the step of generating normal maps when rendering color images from 3D models. In other words, the process of generating color images is made lightweight by separating the 3D projec tion step (i.e. computing the projected geometry as seen from target viewpoints) in terms of the generation of normal maps and the augmentation step, namely computing the color repre sentation. The introduction of normal maps as an intermediary dataset has the advantage that the GPU-intensive step of di rectly rendering augmented images from 3D models is substi tuted by two separate steps wherein the second step, i.e. the conversion from normal maps into augmented images, can be carried out by the CPU. In particular, the first step of con verting the geometric information of the CAD model into a normal map can in principle be performed beforehand, while the step of augmenting the clean normal map can be done online, for instance in parallel of training a recognition unit which uses the augmented images as input data.

Advantageously, normal maps for different perspectives are generated and further processed. This can e.g. be realized by creating a virtual hemisphere around and above the CAD model, thus defining a desired number of viewpoints. For each view point, i.e. for each perspective, a significant amount of augmented images may then be created. By this procedure, a large number of images depicting the same object from differ ent viewpoints with different types of augmentation is ob tained .

In an embodiment of the invention, the augmentation pipeline is built in a modular manner such that the actions to be ap plied to the normal map can be individually chosen. This means that different kinds of augmenting the normal map can be defined, which will be described in more detail in the following. It can be attributed a specific weight to each kind, so that by processing the normal map through the pipe line the normal maps are modified according to the weights given to the individual augmentation steps.

An augmentation step which is advantageously implemented in and comprised by the augmentation pipeline is the shading of the object by varying levels of darkness, in particular by simulating directional and/or ambient light sources.

By multiplying the normal maps to a random directional light vector and adding to the results a random ambient light in tensity value, one can easily simulate an infinity of simple lighting conditions. For each foreground normal map, this op eration yields a lightness map, as used e.g. in the HSL (hue, saturation, lightness) color format.

Another augmentation step which is advantageously implemented in and comprised by the augmentation pipeline is adding a random texture to the object.

To compensate for the presumed lack of texture information in the CAD models, random object textures can be generated using the same functions to generate noise as those used for the background. Random noise maps are generated for the hue and saturation channels of the foreground. Hue and saturation noise maps may be generated separately using different param eters to add further complexity to the resulting textures.

Another augmentation step which is advantageously implemented in and comprised by the augmentation pipeline is color domain transformation, in particular from RGB to HSL or vice versa.

This means that a color image which is obtained by the aug mentation pipeline can also be converted from one color do- main to another. It may, for instance, be beneficial to con vert RGB images into HSL images in order to have semantically more meaningful channels and facilitate the augmentation and training. This operation can either be done inline or of fline, wherein offline in implies that the converted dataset is saved before training.

Another augmentation step which is advantageously implemented in and comprised by the augmentation pipeline is modifying the size and/or position of the object, in particular by translating, rotating, flipping and/or resizing of the ob ject .

Translation of the object comprises x- and/or y-translations . This augmentation step is useful as in real-life color image the object to be recognized is not always perfectly centered.

Rotation comprises a twist (i.e. turn) of the object about a certain angle.

Flipping of the object comprises tilting the object e.g. from left to right or from up to down.

Resizing of the object comprises enlarging or shrinking the object while maintaining its relative dimensions.

All these operations may be referred to as linear transfor mations. Additionally or alternatively, the object may also be distorted, which signifies that x-extension of the object is modified differently to the y-extension.

Another important augmentation step which is advantageously implemented in and comprised by the augmentation pipeline is adding a background to the image.

The background may comprise random colors and patterns. The background is also referred to as "noise" or "background noise". It may include: fractal Perlin noise, cellular noise and white noise. The noise patterns may be generated using a vast frequency range further increasing the number of possi ble background variations.

The background may also include real images from the target domain or from any public image dataset, which the recogni tion system using the augmented images needs to distinguish from the object to be recognized.

Another augmentation step which is advantageously implemented in and comprised by the augmentation pipeline is blurring, in particular uniform, median or Gaussian blurring, of the ob ject .

To reproduce possible motion blur or unfocused images, simple blur operations (e.g. Gaussian, uniform or median blur) can be applied with variable intensity.

Yet another augmentation step which is advantageously imple mented in and comprised by the augmentation pipeline is add ing a partial occlusion of the object.

Occlusions are introduced to serve two different purposes: First is to teach a network to reconstruct the parts of the object that are partially occluded. The second purpose is to enforce the invariance to additional objects within the patch, i.e. to ignore them, in other words treat them as a background. Occlusion objects are generated by walking around the circle taking random angular steps and random radii at each step. Subsequently, the generated polygons are filled with arbitrary depth values and painted on top of the patch.

The invention is also directed towards a renderer for render ing an augmented image of an object from a texture-less CAD model of the object. The renderer comprises:

- a unit for generating a normal map of the object from the CAD model for a specific viewpoint, and - an augmentation pipeline for augmenting the normal map by applying one or more predefined actions defined to the normal map.

The unit for generating the normal maps from the CAD model may be a conventional renderer engine running on the GPU of the computer. The augmentation pipeline may be designed in a modular manner comprising one or several of the properties as described in the context of the inventive method above.

Furthermore, the invention is directed to a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the method described above.

Lastly, the invention is directed to a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the steps of the method as described above.

Embodiments of the invention are now described, by way of ex ample only, with the help of the accompanying drawings, of which :

Figure 1 shows a rendering process according to the prior art ;

Figure 2 shows a rendering process using an augmentation

pipeline according to the invention; and

Figure 3 shows the rendering process of Figure 2 highlight ing the modular character of the augmentation pipe line .

Figure 1 illustrates a method for rendering an augmented col or image of an object from a texture-less CAD model as known in the prior art. A texture-less CAD model 111 is directly rendered, i.e. con verted, to a color image 112, wherein the color image 112 comprises some kind of nuisance, or "clutter", such as back ground noise, blurring, etc. The CAD model 111 comprises geo metrical 3D (three-dimensional) information about the object, but no information about the e.g. the texture or material type of the object.

The rendering process is executed on the graphics processing unit (GPU) of the computer. It consumes relatively much ca pacity of the GPU, i.e. it is a relatively heavy, GPU

intensive process.

The obtained augmented images need to be stored somewhere be fore they can be used, for instance, as synthetic training data of a recognition system.

In contrast, the rendering process according to the present invention comprises the generation and use of intermediary normal maps 212. These normal maps 212 are "clean" normal maps and obtained from texture-less CAD models 211 of the ob ject. The normal maps 212 are used as input of an augmenta tion pipeline A, which converts the clean normal maps 212 in to augmented images 213.

The advantage of the inventive rendering process is that it can be executed on the central processing unit (CPU) instead of the GPU. It is a lightweight process and can be carried out in parallel to other tasks of the computer. In particu lar, it can be carried out in parallel to e.g. a training of a recognition unit. The training data of the recognition unit, which are in this case the augmented images obtained through the rendering process, can even be generated "in situ", in other words "online". This means that they need not to be generated and stored beforehand. Thus, a quasi-infinite number of training data can be generated and used for train ing a recognition unit. Figure 3 shows an advantageous embodiment of the invention, namely the use of a modular augmentation pipeline A. The mod ular augmentation pipeline A comprises n pipeline modules Ai, A₂, A₃,... A_n. Each pipeline module A_± corresponds to one aug- mentation step, for example Ai the inclusion of simple direc tional or ambient lighting, A₂ performing random texturing of the object, A₃ resizing of the object, etc.

A weight is attributed to each pipeline module A_± . If the weight is zero, the augmentation step is not performed, oth erwise it is performed with a certain probability and a cer tain intensity.

Claims

1. Method for rendering an augmented image (213) of an object from a texture-less CAD model (211) of the object, the method comprising the steps of:

- generating a normal map (212) of the object from the CAD model (211) for a specific viewpoint, and

- augmenting the normal map (212) by applying one or more ac tions defined in an augmentation pipeline (A) to the normal map (212) .

2. Method according to claim 1,

wherein the augmentation pipeline (A) is built in a modular manner such that the actions to be applied to the normal map (212) can be individually chosen.

3. Method according to one of the preceding claims,

wherein the method is carried out on a central processing unit (CPU) of a computer.

4. Method according to one of the preceding claims,

wherein the augmentation pipeline (A) comprises shading of the object by varying levels of darkness, in particular by simulating directional and/or ambient light sources.

5. Method according to one of the preceding claims,

wherein the augmentation pipeline (A) comprises adding a ran dom texture to the object.

6. Method according to claim 5,

wherein texturing comprises application of a noise map of the hue and saturation channel to hue and saturation values of the object.

7. Method according to one of the preceding claims,

wherein the augmentation pipeline (A) comprises color domain transformation, in particular from RGB to HSL or vice versa.

8. Method according to one of the preceding claims,

wherein the augmentation pipeline (A) comprises modifying the size and/or position of the object, in particular by trans lating, rotating, flipping and/or resizing of the object.

9. Method according to one of the preceding claims,

wherein the augmentation pipeline (A) comprises adding a background to the image of the object.

10. Method according to one of the preceding claims,

wherein the augmentation pipeline (A) comprises blurring, in particular uniform, median or Gaussian blurring, of the ob ject .

11. Method according to one of the preceding claims,

wherein the augmentation pipeline (A) comprises the partial occlusion of the object.

12. Renderer for rendering an augmented image (213) of an ob ject from a texture-less CAD model (211) of the object, the renderer comprising:

- a unit for generating a normal map (212) of the object from the CAD model (211) for a specific viewpoint, and

- an augmentation pipeline (A) for augmenting the normal map (212) by applying one or more predefined actions to the normal map (212) .

13. A computer program product comprising instructions which, when the program is executed by a computer, cause the comput er to carry out the steps of the method according to one of the claims 1 to 11.

14. A computer-readable storage medium comprising instruc tions which, when executed by a computer, cause the computer to carry out the steps of the method of claim according to one of the claims 1 to 11.