DE102019209282A1 - Method for obtaining an infrared representation of an image situation and computing unit, system and computer program for its implementation - Google Patents

Abstract

The invention relates to a method for obtaining an IR representation (20) of an image situation from an RGB representation (10) of the image situation, with a mapping function that maps an RGB image into a corresponding IR image on the RGB representation ( 10) is applied, the mapping function comprising an algorithm trained on the basis of RGB images and associated IR images. The invention also relates to a computing unit (3), a system, a computer program and a storage medium.

Description

The present invention relates to a method for obtaining an infrared representation of an image situation from an RGB representation as well as a computing unit, a system and a computer program for its implementation.

State of the art

The simulation or, in general, computer-aided generation of pictorial representations plays an increasingly important role, since not all image situations of people can be recorded in advance. One example is a road that should be shown once with rain, once with snow and once in a dry state, whereby the movement of vehicles and people should remain the same in all situations.

DE 100 01 252 B4 discloses a surveillance system that breaks down signals from cameras into objects and then transfers the objects to an RGB display to display a surveillance scene, adds artificial objects and deletes other objects and, if the cameras fail, simulates the surveillance scene using sensor values.

Since such methods from the prior art have mainly been used to generate RGB images, there are a number of programmed computer tools to simulate RGB images and to compare or display them with real camera recordings. In principle, the known methods can also be transferred to other frequency ranges such as the infrared range. Near-infrared (NIR) recordings use areas of the wavelength spectrum that humans cannot see. However, the already existing programmed computer tools do not work easily for simulations in the infrared range, since they are usually intended for the RGB range. In order to transfer them to the infrared range, material properties must also be simulated, which is extremely complex.

Disclosure of the invention

According to the invention, a method for obtaining an infrared or IR representation of an image situation from an RGB representation as well as a computing unit, a system and a computer program for its implementation with the features of the independent patent claims are proposed. Advantageous configurations are the subject of the subclaims and the description below.

The invention makes use of the measure of mapping or converting an RGB representation, which is obtained, for example, by a simulation or computer-aided generation or by recording an image situation, into an IR representation of the image situation using a mapping function, the mapping function being one comprises an algorithm trained on the basis of training RGB images and associated training IR images.

In this way, IR representations can be generated using conventional RGB tools. Obtaining IR representations of image situations is possible in particular without the use of appropriate cameras. The invention makes it possible to generate a large number of IR images (e.g. from existing / simulated RGB images) and then to process the IR images further, in particular to use them again as training data for systems that are to process IR images later during operation.

An important component here is the use of machine learning to maintain the trained algorithm.

Machine learning is a generic term for the "artificial" generation of knowledge from experience: an artificial system learns from examples and can generalize them after the learning phase has ended. This means that the examples are not simply learned by heart, but rather it "recognizes" patterns and regularities in the learning data. In this way, the system can also assess unknown data (learning transfer).

Deep learning describes a class of optimization methods of artificial neural or neural networks that have numerous intermediate layers (English hidden layers) between the input layer and the output layer and thus have an extensive internal structure. In extensions of the learning algorithms for network structures with very few or no intermediate layers, such as with the single-layer perceptron, the methods of deep learning enable stable learning success even with numerous intermediate layers.

The algorithm is advantageously trained by recording at least one image situation by means of an RGB camera for obtaining a training RGB image and by means of an IR camera for obtaining a training IR image. The recording is preferably carried out by means of a camera structure which specifies a predetermined arrangement and alignment of the cameras. This can in particular be a stereo camera structure in which the image situation is recorded by both cameras at the same time, which the other Editing made easier. In an advantageous embodiment, the RGB camera has an IR cut filter. This ensures that no IR light is detected by the RGB camera.

Then, according to a preferred embodiment, the training RGB image and the training IR image of the at least one image situation are rectified, i. E. Geometric distortions are eliminated and an epipolar rectification is carried out in order to facilitate the subsequent assignment of the rectified training images or their image points (pixels). After a dense stereo matching, the associated pixel in the training IR image is known for each pixel of the training RGB image and the rectified training images can be used to train the algorithm.

Alternatively, training with the training images subject to distortion is also possible if one of the training images - possibly after rectification - is converted into a training image subject to distortion in the other domain.

In particular, training the algorithm includes applying the mapping function to the training RGB image to obtain a training IR representation of the RGB image and comparing the training IR representation with the training IR image. The imaging function can then be changed or improved on the basis of deviations between the training IR display and the training IR image. For example, artificial neural networks can be trained using what is known as “supervised learning”. For example, this includes minimizing a loss function, which describes a difference between pixel values of the training IR representation and the training IR image.

An RGB pixel value usually consists of three numerical values for each color, each numerical value being able to assume 256 values, for example with a bit depth of 8 bits. A pixel value for an IR display preferably has the same range of values as the RGB pixel value, for example 3x8 = 24 bits. The image is highly non-linear and varies with the object class currently being viewed, the material properties and the illumination of the recorded image situation with IR light.

The algorithm preferably comprises an artificial neural network or a Generative Adversial Network (GAN), which consists of two artificial neural networks. One of these networks acts as a generator and generates a training IR display from a training RGB image, initially based on the trial and error principle. The second of these artificial neural networks acts as a discriminator and compares the generated training IR representation with the actually recorded training IR image. Structures are determined from the differences in order to improve the future generation of IR representations. The first neural network is now applied to a next training RGB image and a further training IR display is generated, which in turn is compared with the associated recorded training IR image. With each additional representation generated, the GAN learns bit by bit the relationships between the training RGB image and the corresponding training IR image. As a result, the generated training IR representations correspond better and better to the actually recorded training IR images.

In an advantageous embodiment, objects in the training images are classified using additionally annotated training data during training. The mapping function can advantageously be improved by the classification, since the reflectivity for IR light depends strongly on the material and is therefore very different for clothing, skin, etc. In particular, the mapping function can receive class-specific components in this way. An algorithm trained in this way can then also classify objects in the RGB representation and thus apply the mapping function in a class-specific manner.

The classification can be improved if the RGB representation has depth information, e.g. B. because it was generated by computer. A 3D world can be reconstructed from the depth, so that it is easier to see which parts in the RGB representation e.g. People, inventory, nature, cars, etc. can be.

Possible classifications preferably include the category of clothing, face, vegetation, part of the body, in particular hand, head or fingers. These classifications are particularly relevant in the context of autonomous driving or for interior sensors in a motor vehicle. But other classifications are also conceivable.

A computing unit according to the invention is set up, in particular in terms of programming, to carry out a method according to the invention.

According to a further aspect of the invention, a system for carrying out a method according to the first aspect of the invention is proposed, which has an RGB camera and an IR camera as well as a computing unit according to the invention.

The RGB camera and the IR camera are expediently calibrated to one another and theirs Orientation is known, so that each pixel of an RGB image can be assigned a pixel of an IR image if dense matching has been carried out.

The implementation of a method according to the invention in the form of a computer program or computer program product with program code for performing all method steps is advantageous, since this causes particularly low costs, in particular if an executing control device is also used for other tasks and is therefore available anyway. Suitable data carriers for providing the computer program are, in particular, magnetic, optical and electrical memories, e.g. Hard disks, flash memories, EEPROMs, DVDs, etc. A program can also be downloaded via computer networks (Internet, intranet, etc.).

Further advantages and configurations of the invention emerge from the description and the accompanying drawing.

The invention is shown schematically in the drawing using an exemplary embodiment and is described below with reference to the drawing.

Figure list

• 1 shows an embodiment of a system according to the invention in a schematic drawing.
• 2 shows an embodiment of a computing unit according to the invention in a schematic drawing.

Embodiment of the invention

One embodiment of a system according to the invention is shown in 1 shown in a schematic view and denoted by 1.

The system 1 is suitable for carrying out an embodiment of a method according to the invention according to a preferred embodiment of the invention. In the case shown, it has a stereo camera structure with an RGB camera 5 and an IR camera 4th . The RGB camera 5 and the IR camera 4th are here calibrated to each other and their orientation is known. In this way, a pixel of an IR image can be assigned to each image point or pixel of an RGB image after dense matching has been carried out.

The RGB camera 5 advantageously has an IR cut filter 6th so that no IR light is detected by the RGB camera and an RGB image that is as pure as possible is generated. The IR cut filter does not necessarily have to be installed.

The system 1 furthermore has a computing unit 3 which is set up to train an algorithm as part of a mapping function for obtaining an IR representation of an image situation from an RGB representation of the image situation. According to 2 then this mapping function, which in a computing unit 3 ' is implemented, used to represent an RGB representation 10 in an IR representation 20th to depict or transfer the image situation.

In the embodiment shown, the algorithm is implemented by recording at least one image situation by means of the RGB camera 5 for obtaining a training RGB image and using the IR camera 4th trained to obtain a training IR image. The training RGB image and the training IR image of the at least one image situation are then rectified. An associated pixel of the training IR image is therefore known for each pixel of the training RGB image after the matching has been carried out.

The arithmetic unit 3 has in the present case, for example, a GAN that consists of two artificial neural networks 31 , 32 consists. One of these networks acts as a generator 31 and generates a training IR representation from the training RGB image. The second of these networks acts as a discriminator 32 and compares the generated training IR representation with the actual recorded training IR image. The GAN “learns” from the differences and modifies the mapping function. The more training data are available, the more precise the mapping function becomes.

A possible field of application of the invention is the generation of a large number of IR images from existing or simulated RGB images. In particular, the RGB images can be generated using established methods and tools that are not available for generating IR images. These RGB images are then transformed into IR images and can be further processed. In particular, the IR images generated in this way can function as training data for systems that later have to process IR images during operation, e.g. in the context of interior sensors in a vehicle or environmental sensors (e.g. for autonomous driving).

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• DE 10001252 B4 [0003]

Claims (15)

A method for obtaining an IR representation (20) of an image situation from an RGB representation (10) of the image situation, comprising: - Applying a mapping function that maps an RGB image into a corresponding IR image to the RGB representation (10), the mapping function comprising an algorithm trained on the basis of RGB images and associated IR images. Procedure according to Claim 1 , wherein the algorithm is trained by: - recording at least one image situation by means of an RGB camera (5) for obtaining a training RGB image and by means of an IR camera (4) for obtaining a training IR image; - Training the algorithm using the training images. Procedure according to Claim 2 , the training IR image and / or the training RGB image being rectified prior to the training so that the associated pixel of the training IR image is known for each pixel of the training RGB image. Procedure according to Claim 2 or 3 , wherein the algorithm is trained by: - applying the mapping function to the training RGB image to obtain a training IR representation of the training RGB image; - Comparing the training IR representation (20) with the training IR image and changing the mapping function to reduce detected deviations. Procedure according to Claim 4 wherein comparing the training IR representation with the training IR image and changing the mapping function to reduce detected deviations comprises a loss function, which is a difference between pixel values of the training IR representation and the training IR image describes how to minimize. Method according to one of the preceding claims, wherein during the training objects in the training images are classified using additionally annotated training data in order to obtain class-specific components of the mapping function. Procedure according to Claim 6 , whereby objects are classified in the RGB representation (10) and the mapping function is applied class-specifically. Procedure according to Claim 7 , whereby depth information of the RGB representation is used for the classification. Method according to one of the preceding claims, wherein the RGB representation (10) is obtained by a simulation or computer-aided generation or by recording an image situation. Method according to one of the preceding claims, wherein the algorithm comprises at least one artificial neural network or at least one Generative Adversarial Networks system. Method according to one of the preceding claims, wherein the RGB camera (5) has an IR cut filter (6). Computing unit (3, 3 ') which is set up to carry out all method steps of a method according to one of the preceding claims. System (1) for carrying out a method according to one of the Claims 1 to 11 , an RGB camera (5) and an IR camera (4) as well as a computing unit (3 ') Claim 12 having. Computer program that causes a computing unit (3, 3 ') to perform all process steps of a process according to one of the Claims 1 to 11 to be carried out when it is executed on the computing unit (3, 3 '). Machine-readable storage medium with a computer program stored thereon Claim 14 .

Images