CN112740264A

CN112740264A - Design for processing infrared images

Info

Publication number: CN112740264A
Application number: CN201980063468.6A
Authority: CN
Inventors: S·克洛姆普; W·泰默; T·比申费尔德; S·格尔林; C·耶尔登斯
Original assignee: Volkswagen Automotive Co ltd
Current assignee: Volkswagen Automotive Co ltd
Priority date: 2018-09-28
Filing date: 2019-09-20
Publication date: 2021-04-30
Also published as: WO2020064521A1; DE102018216806A1

Abstract

A method, apparatus, and computer-readable storage medium having instructions for processing infrared images and a method and computer-readable storage medium having instructions for training a neural network. In a first step (10), an infrared image of an infrared camera is read in. Next, an output image is generated (11) by applying the neural network to the infrared image. For this application, neural networks have been trained for filtering shadows or for reducing lighting effects. Finally, the output image is output (12) for further use.

Description

Design for processing infrared images

Technical Field

The invention relates to a method, a device and a computer-readable storage medium with instructions for processing infrared images and a motor vehicle using such a method or such a device. The invention also relates to a method for training a neural network and a computer-readable storage medium having instructions and a neural network trained using such a method.

Background

Video conferencing systems have been implemented in the ISDN (Integrated Services Digital Network); Integrated voice and data networks) era. However, such systems are not further widespread with the expansion of mobile radio capacity on mobile terminal devices, such as smart phones. Such systems are also becoming more and more commonly accepted in the PC environment, especially in the business sector. All these systems usually have a color camera which photographs the user from the front.

Cameras are also currently used in motor vehicles for interior monitoring, for example to photograph the driver. The main application of such cameras is autopilot, in which it is checked whether the driver can take over the driving task again. If such a camera is present in a motor vehicle, it can be advantageous: the camera is also used for video telephony applications. This is a very interesting application situation for the driver, especially during autonomous driving. However, so-called driver viewing cameras are designed to also function in darkness. In order not to dazzle the driver, illumination in the near infrared range at a wavelength of approximately 940nm is used, which is not visible to the human eye. The camera is correspondingly designed specifically for this wavelength.

However, images are not well suited for video conferencing due to the taking in the near infrared range and the active illumination of the face from below. A kind of "ghost face" is formed, which has an abnormal and unnatural-looking shadow image. The mostly darkened eye sockets appear particularly bright here, while the bridge of the nose is abnormally darkened. The resulting image is undesirable from a customer perspective.

Another disadvantage is that: the corresponding camera provides only a grayscale image, while the customer expects a color image from the videophone. In addition, due to the mounting position of the camera, the face is photographed from below, and there is no line-of-sight contact between the driver and the camera.

The mentioned problems can currently only be compensated by installing an additional color camera that is aimed exactly at the driver. However, this results in considerable additional costs. The provision of additional installation space is also a problem in the design of the interior space of the vehicle.

For special applications like E-Call (electronic telephone), an unaltered infrared image may be a luxury for the customer. It is therefore desirable that: the available infrared images are processed and made available for other applications, in particular for video telephony.

In this context, CN 105023269 a describes a method for coloring an infrared image of an infrared camera of a vehicle. First, infrared images are captured in the vehicle to train the classifier using the infrared images. After training is complete, the infrared image to be rendered is handed over to a classifier, which classifies each pixel and generates a result map. The result map is segmented into superpixels and a result histogram is created for each superpixel. The primary classification attribute is assigned to the entire superpixel to generate an optimized result graph. Finally, an RGB image of the same size as the infrared image to be rendered is first generated, and then the color space of the RGB image is converted into an HSV color space. Then, pixel values of the image converted into the HSV color space are determined based on the optimized result map and the grayscale values of the infrared image.

DE 102006044864 a1 describes a method for computer-aided image processing in a night vision system of a motor vehicle. The detection device detects a night vision image of the surroundings of the motor vehicle. Other detection devices detect parameters of the surroundings of the motor vehicle. Image regions of the night vision image which are respectively associated with the image content classes are determined as a function of the parameters of the surroundings. One or more image processing criteria are defined for each image content category. The detected night vision image is then processed for display on a display device. The night vision image is processed in accordance with the one or more image processing criteria for the image content category of the respective image region. Finally, the processed night vision image is reproduced on a display device.

US 6792136B 1 describes a method for producing a high resolution color image from an infrared image of a monitored scene. The captured infrared image is analyzed to ascertain whether an object, such as a face, is recognized in the image. If the object is identified, the object characteristics are compared to a plurality of stored object characteristics. If there is consistency, the color characteristics of the object are determined and the object is colored based on the characteristic information stored in the database. If there is no correspondence or no identifiable object and the object color cannot be identified, the image is analyzed to see if a pattern such as clothing can be recognized within the image. If the pattern can be recognized, the color characteristics of the pattern are determined and the pattern is colored according to the infrared reflection characteristics in combination with the stored pattern information. If the pattern cannot be recognized, the unpatterned and featureless portion of the image is colored according to infrared reflection characteristics.

Disclosure of Invention

It is an object of the invention to provide an improved design for processing infrared images.

This object is achieved by a method having the features of claim 1 or 7, by a computer-readable storage medium having instructions according to claim 10, by a device having the features of claim 11 and by a neural network having the features of claim 12. Preferred embodiments of the invention are the subject matter of the dependent claims.

According to a first aspect of the invention, a method for processing an infrared image comprises the steps of:

-reading in an infrared image of an infrared camera;

-generating an output image by applying a neural network to the infrared image, wherein the neural network has been trained for filtering shadows or for reducing lighting effects; and is

-outputting the output image.

Accordingly, a computer-readable storage medium contains instructions which, when executed by a computer, cause the computer to perform the steps of processing an infrared image:

-reading in an infrared image of an infrared camera;

-outputting the output image.

The term computer is to be understood broadly herein. In particular, the computer also includes workstations, control devices and other processor-based data processing devices.

Similarly, the apparatus for processing an infrared image has:

-an input module for reading in an infrared image of an infrared camera;

-an image processing unit for generating an output image by applying a neural network to the infrared image, wherein the neural network has been trained for filtering shadows or for reducing lighting effects; and

-an output for outputting the output image.

The solution according to the invention is based on the idea that: infrared images of a user, in particular of a driver of a motor vehicle, are used as input data, and improved output images are calculated by means of a neural network, which can then be used for further applications. Here, the improvement may consist in: reducing the influence due to unnatural lighting or filtering shadows. In recent years, great progress has been made in the field of machine learning. As computers become more powerful, complex neural networks can also be taught and applied. With these complex neural networks, difficult image processing tasks can also be achieved. An example of complex image processing by a neural network is described, for example, in [1 ]. In this example, two different images are used in order to generate a new image therefrom. Here, one of the input images specifies the content of the image to be generated. The second input image defines a structure or texture and color scheme. By means of the solution according to the invention, additional cameras can be dispensed with, which on the one hand saves costs and on the other hand enables an efficient use of the available installation space.

According to one aspect of the invention, the neural network has additionally been trained to change the perspective of the infrared image. By converting the viewing angle, it is possible to realize: the output image appears as if the person in the image is looking at the camera from the front. This significantly improves the perceived naturalness of the presentation.

According to one aspect of the invention, the output image is a grayscale image. Instead, the output image is a color image. In this case, the neural network has additionally been trained for coloring the infrared image. In the case of a grayscale image, the output image is not always a color image, but a more natural perception is achieved. This again improves the perception significantly if the neural network also adds color to the image.

According to one aspect of the invention, a neural network may access a user's color image for rendering an infrared image. Alternatively, the user may select between two or more neural networks for rendering the infrared image. The result of the processing does not directly correspond to reality, rather only the style of the presentation taught is mimicked. Thus, colors in the output image may deviate from true colors. This is not critical in terms of the user's clothing, but these deviations are rather annoying, for example, in terms of hair color. This problem can be solved in two ways. In one aspect, the neural network may also be taught such that a color photograph of the current user is additionally provided to the system as well. From the image, the system can make a determination regarding color. Alternatively, it is possible: different neural networks are taught that take into account different individual characteristics. In this case, the user may select the neural network that best represents him.

According to one aspect of the invention, the infrared image is part of an infrared video and the output image is part of an output video. The described solution is not limited in application to still images. The solution is equally applicable to moving images or image sequences. In this way, the infrared image of the infrared camera can also be used for applications based on video data.

According to one aspect of the invention, the output image is used for a videophone. In this way, the already existing infrared camera can be used for additional applications, which provide a significant added value for the customer.

The method according to the invention or the device according to the invention is particularly advantageously used in a vehicle, in particular a motor vehicle.

According to another aspect of the invention, a method for training a neural network comprises the steps of:

-reading in an infrared image of an infrared camera;

-reading in images of the additional camera; and is

-training the neural network, wherein the infrared image of the infrared camera is used as input image and the image of the additional camera is used as output image.

Correspondingly, a computer-readable storage medium contains instructions that, when executed by a computer, cause the computer to train a neural network by:

-reading in an infrared image of an infrared camera;

-reading in images of the additional camera; and is

A challenge in machine learning is to provide sufficient training data. For this application scenario, a second camera may be used to generate training data. The second camera is only needed for the training phase, which can be omitted later. The second camera photographs the user in a visible light range under good lighting conditions. Now, various testers are photographed using this structure. The image data from the two cameras is then used to teach the neural network. In this way, the required training data can be easily generated.

According to one aspect of the invention, the image of the additional camera has a similar viewing angle to the infrared image of the infrared camera or has a preferred target viewing angle. In particular, the situation in which a user present in the motor vehicle is photographed from below and does not look at the camera can be taken into account in two ways. A first possibility consists in training the neural network such that it also adapts to the perspective. For this purpose, the second camera is not placed next to the infrared camera for generating the training data, but is directed at the user from the front. Thus, the neural network implicitly learns the preferred perspective. In the case of the second possibility, the perspective conversion is carried out in a second step in a conventional manner (such as is customary in photographic processing) by vertical or horizontal deformation of the image. To this end, the infrared camera provides a 3D position of the head and a head direction. In this case, it is also possible: by changing the intensity gradient in the image, the unnatural lighting situation from below is modified.

According to one aspect of the invention, the image of the additional camera is a grayscale image or a color image. Depending on the application, that is to say depending on whether a grayscale image or a color image is to be output as output image, a grayscale image or a color image can be captured by the additional camera. In this way, the training can be matched in a simple manner to the requirements of future applications.

Preferably, a neural network for application to the infrared image is trained with the method according to the invention. The neural network trained in this way is particularly well suited for processing infrared images of infrared cameras in motor vehicles.

Drawings

Other features of the invention will be apparent from the subsequent description and the appended claims, taken in conjunction with the accompanying drawings.

Fig. 1 schematically shows a method for processing an infrared image;

fig. 2 shows a first embodiment of a device for processing infrared images;

fig. 3 shows a second embodiment of a device for processing infrared images;

fig. 4 schematically shows a motor vehicle in which the solution according to the invention is implemented;

FIG. 5 schematically illustrates a method for training a neural network;

fig. 6 schematically shows an infrared camera arranged in a motor vehicle;

FIG. 7 schematically illustrates an architecture for applying a neural network in an automotive vehicle; and

fig. 8 schematically shows an architecture for teaching the structure of a neural network.

Detailed Description

For a better understanding of the principles of the invention, embodiments thereof are explained in more detail below with reference to the accompanying drawings. It is easy to understand that: the invention is not limited to these embodiments and the described features may also be combined or modified without departing from the scope of protection of the invention, as defined in the appended claims.

Fig. 1 schematically shows a method for processing an infrared image. In a first step 10, an infrared image of an infrared camera is read in. The infrared image may be part of an infrared video. Next, an output image is generated 11 by applying a neural network to the infrared image. For this application, neural networks have been trained for filtering shadows or for reducing lighting effects. Additionally, the neural network may have been trained for changing the perspective of the infrared image or for coloring the infrared image. To color the infrared image, the neural network preferably has access to the user's color image. Alternatively, the user may select between two or more neural networks for rendering the infrared image. Finally, the output image is output 12 for further use, e.g. for videotelephony. The output image may be a grayscale image or a color image, in particular. The output image may also be part of the output video.

Fig. 2 shows a simplified schematic diagram of a first embodiment of a device 20 for processing infrared images. The device 20 has an input 21 via which, in particular, infrared images of an infrared camera 41 can be read in by an input module 22. The infrared image may be part of an infrared video. The image processing unit 23 generates an output image by applying a neural network to the infrared image. For this application, neural networks have been trained for filtering shadows or for reducing lighting effects. Additionally, the neural network may have been trained for changing the perspective of the infrared image or for coloring the infrared image. For the purpose of rendering the infrared image, the neural network preferably has access to a color image of the user, which may be stored in a memory 25 of the device 20 or received via the input 21, for example. Alternatively, the user may select between two or more neural networks for the purpose of coloring the infrared image. The output image is provided for further use, e.g. for videotelephony, via an output 26 of the device 20. The output image may be a grayscale image or a color image, in particular. The output image may also be part of the output video.

The input module 22 and the image processing unit 23 may be controlled by a control unit 24. The settings of the input module 22, the image processing unit 23 or the control unit 24 can be changed as necessary via the user interface 27. The data accumulated in the device 20 can be stored in the memory 25 as needed, for example for later analysis or for use by components of the device 20. The input module 22, the image processing unit 23 and the control unit 24 may be implemented as dedicated hardware, for example as an integrated circuit. However, they may of course also be combined in part or in whole or be implemented as software running on a suitable processor, e.g. on a GPU or CPU. The input 21 and the output 26 may be implemented as separate interfaces or may be implemented as a combined bi-directional interface.

Fig. 3 shows a simplified schematic diagram of a second embodiment of a device 30 for processing infrared images. The device 30 has a processor 32 and a memory 31. The device 30 is, for example, a computer or a control device. In the memory 31 there are stored instructions which, when executed by the processor 32, cause the device 30 to carry out the steps according to one of the described methods. The instructions stored in the memory 31 thus represent a program that can be executed by the processor 32, which program implements the method according to the invention. The device 30 has an input 33 for receiving information, in particular an image of an infrared camera. Data generated by the processor 32 is provided via an output 34. These data may also be stored in the memory 31. The input 33 and output 34 may be combined into a bi-directional interface.

The processor 32 may include one or more processor units, such as a microprocessor, a digital signal processor, or a combination thereof.

The

memories

25, 31 of the described embodiments may have not only volatile storage areas but also nonvolatile storage areas, but may include various storage devices and storage media such as hard disks, optical storage media, or semiconductor memories.

Fig. 4 schematically shows a motor vehicle 40 in which the solution according to the invention is implemented. An infrared camera 41 is installed in the motor vehicle 40, for example in the dashboard in the region of the steering wheel 42. The device 20 for processing infrared images processes the infrared images captured by the infrared camera 41 and provides the resulting output images for further use. In fig. 4, the device 20 is a separate component, but the separate component may also be mounted in the control device 43 of the infrared camera 41. A further component of motor vehicle 40 is a data transmission unit 44, via which, in particular, a connection to a service provider, for example for video telephony, can be established. For storing data, a memory 45 is present. Data exchange between the various components of the motor vehicle 40 is effected via a network 46.

Fig. 5 schematically illustrates a method for training a neural network. In a first step 70, an infrared image of an infrared camera is read in. The image of the additional camera is also read in 71. The image of the additional camera preferably has a similar viewing angle to the infrared image of the infrared camera or has a preferred target viewing angle. The image of the additional camera may be, in particular, a grayscale image or a color image. The neural network is trained 72 based on the images and the infrared images. Here, the infrared image of the infrared camera is used as an input image and the image of the additional camera is used as an output image.

Subsequently, a preferred embodiment of the invention shall be described with reference to fig. 6 to 8, using as an example in a motor vehicle. Of course, other uses are possible.

Fig. 6 schematically shows an infrared camera 41 arranged in a motor vehicle. In this example, an infrared camera 41 is mounted in the dashboard in the region of the steering wheel 42. For sufficient illumination of the field of view of the infrared camera 41, infrared light sources 47 are mounted on the right and left sides of the infrared camera 41. Due to the positioning of the infrared camera 41 and the active illumination of the face from below by the infrared light source 47, the captured image has an abnormal viewing angle and an abnormal and unnatural-looking shadow image.

Fig. 7 schematically shows an architecture for applying the architecture of a neural network in a motor vehicle. The idea on which this structure is based is that: a trained neural network 51 is used which produces an improved output image 52 or output video from the input image or input video. In this case, for example, so-called "Generative adaptive Networks" (GANs, german for example, "erzeugend geogerische Netzwerke") can be used, which produce a reasonable image [2 ]. In particular, for video telephony, a color image should be generated as an output image 52 from an infrared image 50 as an input image. In the example shown, the infrared camera 41 provides image data or video data captured in Infrared (IR) or Near Infrared (NIR). Here, the output data is RGB image data or RGB video data.

It should be noted here that: the result does not correspond directly to reality, but only the style of presentation taught is mimicked. Thus, in this application scenario, even if the driver wears a red sweater, the algorithm may produce an image of the driver wearing a blue sweater. For example, deviations from reality can have a stronger effect on hair color.

This problem can be solved in two ways. On the one hand, different neural networks 51 may be taught, which take into account different personal characteristics. In this case, the driver may select the neural network 51 that best represents him. On the other hand, the neural network may also be taught such that the system is additionally provided with a color photograph 53 of the current driver. From the image, the system can make a determination regarding color. The color picture 53 can be taken, for example, by means of a smartphone 54 and transferred to a motor vehicle by means of Car-Net (internet of vehicles).

Fig. 8 schematically shows an architecture for teaching the structure of the neural network 51. A challenge in machine learning is to provide sufficient training data. For this application scenario, an additional camera 61 may be embedded in the vehicle in order to generate the training data. Of course, the second camera 61 is only necessary in the development stage, and a mass-production vehicle may not have the second camera. The additional camera 61 photographs the driver from a similar perspective as the embedded infrared camera 40. In this case, the image 60 of the additional camera 61 is taken in the visible range under good lighting conditions. Depending on the application, either a grayscale image or a color image can be taken. In the example shown, an additional camera 61 provides RGB image data or RGB video data. Now, various testers are photographed using this structure. The image data of the two

cameras

41, 61 are then used in a training stage 62 to teach the neural network 51.

The situation where the driver is shot from below and does not look at the

cameras

41, 61 can be solved in two ways. In one aspect, the neural network 51 can be trained such that the neural network also adapts to the perspective. For this purpose, the additional camera 61 is not placed next to the infrared camera 41 for the purpose of generating training data, but is directed at the driver from the front. Thus, the neural network 51 implicitly learns the preferred perspective. On the other hand, the perspective transformation can be carried out in a second step in a conventional manner by vertical or horizontal deformation of the image, as is customary in photographic processing. For this purpose, the infrared camera 41 provides a 3D position of the head and a head direction. In this case, it is also possible: by changing the intensity gradient in the image, the unnatural lighting situation from below is modified.

Reference to the literature

[1] Gatys et al: "Image Style Transfer Using relational Neural Networks", 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), page 2414-.

[2] Zhu et al: "Ungained Image-to-Image transformation using Cycle-dependent additive Networks", 2017 IEEE International reference on Computer Vision (ICCV), p.2242-2251.

List of reference numerals

10 reading in an infrared image

11 generating an output image by applying a neural network to the infrared image

12 outputting the output image

20 device

21 input terminal

22 input module

23 image processing unit

24 control unit

25 memory

26 output terminal

27 user interface

30 device

31 memory

32 processor

33 input terminal

34 output terminal

40 motor vehicle

41 Infrared camera

42 steering wheel

43 control device

44 data transmission unit

45 memory

46 network

47 Infrared light source

50 infrared image

51 neural network

52 output image

53 colour photograph

54 smart phone

60 images

61 additional camera

62 training stage

70 reading in infrared image of infrared camera

71 reading in the image of the additional camera

72 training the neural network using the read-in images and the infrared images

Claims

1. A method for processing an infrared image (50), the method having the steps of:

-reading in (10) an infrared image (50) of an infrared camera (41);

-generating (11) an output image (52) by applying a neural network (51) to the infrared image (50), wherein the neural network (51) has been trained for filtering shadows or for reducing lighting effects; and is

-outputting (12) the output image (52).

2. The method of claim 1, wherein the neural network (51) has additionally been trained for changing the perspective of the infrared image (50).

3. The method according to claim 1 or 2, wherein the output image (52) is a grayscale image, or the neural network (51) has additionally been trained for coloring the infrared image (50) and the output image (52) is a color image.

4. The method of claim 3, wherein the neural network (51) has access to a user's color image (53) for rendering the infrared image (50) or a user can select between two or more neural networks (51) for rendering the infrared image (50).

5. The method of any of the above claims, wherein the infrared image (50) is part of an infrared video and the output image (52) is part of an output video.

6. The method according to any of the preceding claims, wherein the output image (52) is for a videophone.

7. A method for training a neural network (51), the method having the steps of:

-reading in (70) an infrared image (50) of an infrared camera (41);

-reading in (71) an image (60) of the additional camera (61); and is

-training (72) the neural network (51), wherein the infrared image (50) of the infrared camera (41) is used as input image and the image (60) of the additional camera (61) is used as output image.

8. Method according to claim 7, wherein the image (60) of the additional camera (61) has a similar viewing angle as the infrared image (50) of the infrared camera (41) or has a preferred target viewing angle.

9. The method according to claim 7 or 8, wherein the image (60) of the additional camera (61) is a grayscale image or a color image.

10. A computer-readable storage medium having instructions which, when implemented by a computer, cause the computer to carry out the steps of the method for processing infrared images (50) according to any one of claims 1 to 6 or the steps of the method for training a neural network (51) according to any one of claims 7 to 9.

11. An apparatus for processing an infrared image (50), the apparatus having:

-an input module (22) for reading in (10) an infrared image (50) of the infrared camera (41);

-an image processing unit (23) for generating (11) an output image (52) by applying a neural network (51) to the infrared image (50), wherein the neural network (51) has been trained for filtering shadows or for reducing lighting effects; and

-an output (26) for outputting (12) the output image (52).

12. A neural network (51) for application to an infrared image (50), characterized in that it is trained with a method according to any one of claims 7 to 9.

13. A motor vehicle (40), characterized in that the motor vehicle (40) has at least a device (20) according to claim 11 or is set up to carry out a method for processing infrared images (50) according to one of claims 1 to 6.