EP4612646A1

EP4612646A1 - Method, computer program and electronic device for tone mapping based on a neural network

Info

Publication number: EP4612646A1
Application number: EP24700922.8A
Authority: EP
Inventors: Peter TODOROV; Julian HARTIG; Jan MEYER-SIEMON; Gregor Schewior
Original assignee: Dream Chip Technologies GmbH
Current assignee: Dream Chip Technologies GmbH
Priority date: 2024-01-12
Filing date: 2024-01-12
Publication date: 2025-09-10
Also published as: WO2025149174A1

Abstract

The invention is related to a method for tone mapping of a high dynamic range (HDR) input image into a low dynamic range (LDR) output image. The invention is further related to a computer program having program code means configured for performing such a method and an image processing chip configured for performing such a method. The invention is further related to a method for training a neural network of an image processing chip.

Description

METHOD, COMPUTER PROGRAM AND ELECTRONIC DEVICE FOR TONE MAPPING BASED ON A NEURAL NETWORK

The invention is related to a method for tone mapping of a high dynamic range (HDR) input image into a low dynamic range (LDR) output image. The invention is further related to a computer program having program code means configured for performing such a method and an electronic device configured for performing such a method. The invention is further related to a method for training a neural network of an electronic device, e.g. an image processing chip.

Image processing chips are electronic devices for processing input image data and converting them into output image data. In general, the invention concerns the field of high dynamic range (HDR) imaging. HDR images typically offer a dynamic range of 14-16 bits to 20-24 bits, while traditional LDR images (or standard dynamic range images) typically only offer a dynamic range of 8-10 bits. Therefore, HDR images allow preserving the details of real-world scenes that contain very bright, but also very dark areas much better than conventional LDR images. Luminance is a photometric quantity that is used in photo and video technology to measure the brightness of pixels. HDR images allow capturing dynamic ranges similar to the human eye, which can detect a luminance range of up to approximately 10¹⁴.

However, common displays are only able to reproduce the luminance range of conventional LDR images. Currently, there are no displays available that can reproduce the luminance range of HDR images, but only very few expensive displays are able to reproduce a luminance range of approximately 12 bits. Due to these limitations of display contrast, the luminance range of input HDR images has to be reduced in order to be displayed on a regular display. This process of converting an HDR image to an LDR image is commonly referred to as tone mapping.

In the context of this application, which is related to a method for tone mapping a high dynamic range input image into a low dynamic range output image, the term “low dynamic range output image” can generally refer to any image with a dynamic range that is smaller than the dynamic range of the high dynamic range input image. In particular, in certain embodiments, the low dynamic range can be a conventional low dynamic range image, e. g. with a dynamic range of 8 to 10 bits and/or a standard dynamic range (SDR) image.

The input image and the output image can be part of an input video sequence and an output video sequence, respectively.

From the state of the art, a variety of different tone mapping operators is known that have been developed to meet these challenges. They can be divided into two main types: global tone mapping operators and local tone mapping operators.

On the one hand, global tone mapping operators (or global tone mapping methods) map every pixel of the image in the same way, independent of its position or surrounding pixels in the image. In general, global tone mapping operators are non-linear functions (also called tone mapping curves) that are based on a global luminance and/or other global properties of the particular image. As the same function is applied to every pixel of the particular image, global tone mapping methods are simple and fast and require only a small amount of processing power. Therefore, they can be implemented, for example based on look-up tables, using low-cost hardware. However, these global methods often cause a significant loss of contrast, in particular with regard to local image details, as only global properties of the image are considered.

On the other hand, local tone mapping operators (or local tone mapping methods) map each pixel of the image differently, in particular depending on local image properties, for example depending on surrounding pixels. When designed correctly, these methods can achieve very good results in terms of preserving local contrast as it is beneficial with regard to the intended application. However, these local tone mapping methods are prone to artifacts like halo effect and ringing and their output may appear unrealistic to the human observer. Moreover, these methods are generally more complicated than global tone mapping methods and hence require significantly more processing power in most cases.

From US 2018/0097992 A1, for example, a global tone mapping method and a corresponding system are known that involve computing a first histogram for luminance values of the input image, accessing a target histogram for the image, and computing a transfer function based on the first histogram and the target histogram. The tone-mapped image is produced by applying the transfer function to the pixel values of the input image.

It is an object of the present invention to provide a method for tone mapping of HDR input images into LDR output images that offers an improved preservation of the image’s details and contrast compared to conventional tone mapping methods, but requires a smaller amount of processing power.

The object of the invention is achieved by a method for tone mapping of a high dynamic range input image into a low dynamic range output image, the method comprising at least the following steps: a) Obtaining a brightness component of the input image, b) Generating a tone mapping curve or at least parts thereof by a neural network based upon the obtained brightness component of the input image, c) Generating the output image by applying the tone mapping curve to at least one brightness component of the input image.

In particular, the tone mapping can be done by an electronic device in real time.

In step a), a brightness component of the input image is obtained. Such brightness component can be, for example, the luminance component of the input image, the chrominance component or any other component characterizing the brightness, or a combination of such components.

A luminance component can be, for example, the luma component (luma channel) of a color space in which luminance intensity values and color tone values are separated, for example the luma component Y of the YCbCr color space (or Y’ of the Y’CbCr color space). If the luminance component is a luma component of a color space in which luminance intensity values and color tone values are separated, as the luma component Y of the YCbCr color space, it can be necessary to transform the input HDR image to such a color space first. For example, if the input image is an RGB image, the RGB input image could be transformed to the YCbCr color space first.

However, the luminance component can generally also be any other component or combination of components of an arbitrary color space that represents the luminance of the image. For example, the luminance component could also be the green component G of the RGB color space, as it can represent the image’s luminance sufficiently well. As a matter of course, the luminance component comprises a plurality of luminance values. In particular, the luminance component of the input image can comprise one luminance value for each pixel of the input image.

The terms “dynamic range” and “luminance range” are used equivalently in the context of this application.

In step b) either the whole tone mapping curve can be generated, or at least parts of the whole tone mapping curve, for example parameters of a predefined shape of the tone mapping curve. It is also possible that most or all discrete values of a tone mapping curve can be generated out of the neural network. Then the neural network has the potential to produce arbitrary curve shapes which are best adapted to the requirements of the actual image situation.

In step c), the output image is generated by applying the adapted tone mapping curve to at least one brightness component of the input image, e.g. to the same kind of brightness component used in step a) regarding the input image.

The tone mapping curve generated by the method of the invention can be global tone mapping curve or a local tone mapping curve. The invention has the advantage that a very efficient tone mapping can be done using the advantages of neural networks. In particular, the capabilities of neural networks allow the processing and tone mapping of images with extreme contrasts, for example in the case of a tunnel exit.

According to an advantageous embodiment of the invention, it is proposed that the method comprises calculating a brightness histogram that represents the brightness distribution of the input image and feeding in the brightness histogram into the neural network as input data for the generation of the tone mapping curve or the parts thereof. In this way, the amount of data to be processed is reduced significantly. The use of histograms provides very efficient data compression. It is also possible to perform an initial data compression algorithm and to obtain the brightness histogram from the compressed data delivered by this algorithm.

The brightness histogram is obtained that represents the brightness distribution of the input image. In general, this brightness histogram can be obtained from the original HDR input image. However, as will be explained in greater detail below, the brightness histogram can also be obtained from a compressed version of the input image in order to reduce the processing power that is required for step c).

According to an advantageous embodiment of the invention, it is proposed that the brightness histogram is calculated based upon at least two previously calculated brightness histograms. In this way, the amount of calculations to be performed can be reduced, whereby the latency can be minimized. For example, the previously calculated brightness histograms can be the histograms of previously processed frames of a video sequence. In this way, the latency of each frame of a video sequence between the calculation of histograms and applying tone mapping to the frame can be reduced.

According to an advantageous embodiment of the invention, it is proposed that the brightness histogram is calculated as a combination, e.g. a combination of a linear image histogram and a logarithmic image histogram. This further allows reducing latency between the calculation of a histogram and applying tone mapping to a frame. In addition, the loss of information caused by the compression is minimized. For example, a brightness histogram of the current frame can be calculated by a logarithmic histogram from the current frame and a predicted linear histogram from one or more previous frames.

According to an advantageous embodiment of the invention, it is proposed that the neural network generates only tone curve parameters of a predefined shape of the tone mapping curve, wherein the tone mapping curve is then created by applying the tone curve parameters generated by the neural network to the predefined shape of the tone mapping curve. In this way, the functional safety of the method can be improved because a certain basic shape of the final tone mapping curve is predefined which reduces possible variations in the result. Only some parameters of the final curve can be changed. The curve parameters are then used in a predetermined curve function. This also reduces the processing time. By predicting curve parameters, we can guarantee the shapes of the curves and that they will always be monotonous and have smooth transitions. According to an advantageous embodiment of the invention, it is proposed that the predefined shape of the tone mapping curve is the shape of a Reinhard curve, an extended Reinhard curve, a freely defined curve and/or one of the aforementioned curve shapes blended with a sigmoid function. This allows for significant improvements in the tone mapping of high dynamic range input images.

By applying a sigmoid function with a raised starting point to the shape of the tone mapping curve, the problem of the tunnel exit image can be resolved with outperforming quality.

According to an advantageous embodiment of the invention, it is proposed that the generated tone mapping curve or the parts thereof, the generated output image and/or any other data based upon output data of the neural network is fed back in at least one feedback loop as new input data for the neural network. In this way, the prediction produced by the neural network can be made more accurate. The feedback procedure using the feedback loop can be done only once or more than once, for example two times or three times. For the first round of the loop, predefined initializing values can be used as input data, for example a histogram with predefined initializing values. It is also possible to use a real input image as input data. Also, for the first round of the feedback loop the predicted tone curve parameters can be set to initializing values. After the neural network has predicted the tone mapping curve or the parts thereof, the original histogram can be tone mapped using the tone mapping curve and then fed back to the neural network for another round of the feedback loop. It is also possible to implement a feedback loop of the predicted parameters from the previous round and/or a feedback loop of the predicted tone mapping curve from the previous round.

According to an advantageous embodiment of the invention, it is proposed that the neural network is trained using training data based upon a high dynamic range training image and a corresponding low dynamic range ground truth image. In this way, high quality training data can be provided for efficient training of the neural network. Training of the neural network can be done prior to the run time of the method, for example when the electronic device is manufactured. It is also possible to perform training of the neural network at run time, e.g. during an initialization phase of the method or the electronic device performing the method.

According to an advantageous embodiment of the invention, it is proposed that the tone mapping curve is generated for the whole image value range from 0 to 2ⁿ-1 or at least 90% of this range, wherein n is the number of bits of the high dynamic range input image. This produces a more stable outcome of the result of the tone mapping, in contrast to approaches where the image value range is more limited. It was experimented with this image value range and found out that it produces a more stable outcome than an alternative solution where the curve was generated for the range between the minimum and maximum pixel values of the actual image. Situations when the brightest spot changes rapidly in the image will cause severe global effect if the image was processed from minimum value to maximum value. However, processing an image with a curve that was always produced for the whole value range resulted in a stable and pleasing outcome.

According to an advantageous embodiment of the invention, it is proposed that a curve loss is calculated based upon the tone mapping curve using the generated curve parameters and a ground truth curve. For example, the area under the tone mapping curve and the area under a ground truth curve can be calculated, and curve loss can be calculated as the difference or the ratio between these areas. As an alternative, it is also possible to conduct summing up the differences between the two curves at discrete sampling points.

In this way, curve loss can be minimized during run time of the method, whereby the image quality of the produced output images can significantly be improved. It is possible to use mean average error (MAE) as a measure of the loss for parameter estimation. However, it was noticed that MAE doesn’t correspond one-to-one with the outcoming curve and that leads to a bias in the error. Specifically, using MAE the bias causes the predictions to be mostly darker than what they should be and very rarely brighter. To solve this, it is proposed to create the predicted tone mapping curve and the ground truth curve and calculate the loss from them. The comparison can be done, for example, by computing the area of the curves using the integral. This results in a much faster calculation than comparing 2ⁿ values of the two curves.

The object of the invention is also achieved by a computer program having program code means configured for performing a method of the aforementioned kind when the computer program is executed on a computer. The computer can be a computer, particularly a processor, of an electronic device configured for image processing.

The object of the invention can also be achieved by an electronic device configured for performing a method of the aforementioned kind. The electronic device is configured for image processing, e.g. in the form of an image processing chip or a chip set having more than one chip. The electronic device can be configured as image processing engine, image processing unit (IPU) or image signal processor (ISP). The electronic device can have own memory, e.g. for buffering the video stream. The electronic device can be buffer free, for example without any memory, and is configured for direct connection to a camera streaming its video data “through” the electronic device.

In addition, the object of the invention can be achieved by a method for training a neural network of an electronic device, which is configured for tone mapping of a high dynamic range input image into a low dynamic range output image, particularly an electronic device of the aforementioned kind, comprising the following steps: a) Obtaining a low dynamic range ground truth image from a data set, b) Generating a high dynamic range training image by adding degradation to the low dynamic range ground truth image, c) Training the neural network using the high dynamic range training image and the low dynamic range ground truth image.

This provides for very efficient training of the neural network.

According to an advantageous embodiment of the invention, it is proposed that the high dynamic range training image is generated by applying an inverse tone mapping curve to the low dynamic range ground truth image. In this way, the high-quality training data for training the neural network can be generated very efficiently. The inverse tone mapping curve can be an inverse version of the tone mapping curve used for the conversion from the high dynamic range input image into the low dynamic range output image, particularly a tone mapping curve of a predefined shape, e.g., in the shape of a Reinhard curve.

In the following, the invention will be explained in more detail using exemplary embodiments schematically shown in the attached drawings. The drawings show the following:

Figure 1 a representation of an exemplary image processing system,

Figure 2 steps of a tone mapping method,

Figure 3 steps of a further tone mapping method,

Figure 4 an example of a tone mapping curve,

Figure 5 an example of a better tone mapping curve shape in wide light distributions, Figure 6 steps of a training method of the neural network. Figure 1 shows an exemplary image processing system. The image processing system comprises an HDR image sensor 1, which can be an HDR video camera. Moreover, the image processing system of Figure 1 comprises a display unit 5, which can be a conventional LDR display.

Furthermore, the image processing system shown in Figure 1 comprises an electronic device 2, which is an image processing unit in this exemplary embodiment. The image processing unit 2 has an image processing chip 4 and a memory 3 to store image data. The image processing chip 4 can be, for example, an appropriately programmed microprocessor, a digital signal processor (DSP), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC) or similar. The image processing chip 4 reads from and writes to memory 3.

The HDR image sensor 1 generates a sequence of HDR input images 6. The HDR image sensor 1 is directly or indirectly connected with the image processing unit 2, which allows the image processing unit 2 to read the HDR input images 6 generated by the HDR image sensor 1. Each HDR input image 6 read by the image processing unit 2 can be stored in memory 3.

Alternatively, the image processing unit 2 can be an image signal processor without memory 3 which is directly connected to the HDR image sensor 1 , thereby streaming its video data “through” the image processing chip 4 without buffering.

The image processing unit 2 is configured for performing the method as described above for tone mapping the HDR input image 6 into a LDR output image 7.

After tone mapping the HDR input image 6 into an LDR output image 7, i.e. after generating the LDR output image 7 as previously explained, the generated LDR output image 7 can be transmitted to the display unit 5, where the output image 7 can be displayed.

This procedure can be repeated for each HDR input image 6 of the HDR video sequence generated by the HDR image sensor 1. This results in a generation of a LDR output video sequence, which is a sequence of LDR output images 6. The LDR output video sequence can be displayed on the display unit 5. Additionally or alternatively, the output image and/or the output video sequence can be stored in a memory and/or stored on a data storage unit and/or can be transmitted via a data transmission link.

The method depicted in Figure 2, which can be executed by image processing unit 2, provides for tone mapping of the HDR input image 6 from camera 1 into the LDR output image 7. In block 10 a brightness histogram 15 is calculated from the input image data, e.g. as a combination of a linear image histogram and a logarithmic image histogram. The brightness histogram 15 is provided to a neural network 11 which predicts in a block 12 parameters 16 of a predefined shape of the tone mapping curve. The parameters 16 are then provided to a block 13 where the tone mapping curve 17 is created from the parameters 16. In block 14 the tone mapping curve 17 is applied to the input image 6 for generating the output image 7.

The method depicted in Figure 3 is based on the method shown in Figure 2, with the difference that one or more feedback loops 18, 19, 20, 21 are added. Through feedback loop 18, output data 7a of block 14 can be fed back to block 10 as new input data for neural network 11. The flow of data can be controlled by switch 22. First, the HDR input image 6 could be fed as input data into block 10. Neural network 11 produces again the tone mapping curve parameters in 16. These parameters 16 can be used in block 13 for creating the tone mapping curve 17. In block 14, the tone mapping curve 17 can be applied to the input image 6 for generating a preliminary output image data 7a. Then the feedback loop 18 is used for feeding back the preliminary output image data 7a to block 10, where, for example, the histogram can be calculated based upon the preliminary output image data 7a. Then the process is repeated in neural network 11, which means that new and more accurate parameters 16 of the tone mapping curve are predicted by neural network 11. The improved parameters 16 are then again provided to block 13, and the tone mapping curve 17 created in block 13 is applied in block 14 generating new preliminary output image data 7a, which is more accurate than the previous preliminary output image data 7a.

The feedback loop 18 process can now be terminated, which means that the preliminary output image data 7a is then output as final output image data 7. It is also possible to proceed again with the feedback loop 18 one or more times.

As can be seen in Figure 3, other feedback loops 19, 20, 21 can be implemented as well.

Feedback loop 19 can be used for feeding back the predicted parameters 16 from the previous round, thereby feeding back the output data of neural network 11 as new input data for neural network 11 , controlled by a switch 23. Feedback loop 20 can be used for feeding back the predicted tone mapping curve 17 from the previous round as new input data for neural network 11 , controlled by a switch 24. In feedback loop 21 , a tone mapped input logarithmic or linear histogram 15a can be generated separately in an additional block 14a, based upon the predicted tone mapping curve 17. This tone mapped input logarithmic or linear histogram 15a can be fed back as new input data for neural network 11 within feedback loop 21 controlled by switch 25. This might be less accurate than the feedback loop 18 but this could be done if the hardware sets such limitations that no output image 7, 7a could be accessed.

The method depicted in Figure 3 can be implemented with only one of the feedback loops 18, 19, 20, 21, or with several or all of them. It is also possible that several or all of the feedback loops 18, 19, 20, 21 are implemented in the method, the computer program and/or in the electronic device, and that one or several of them are activated only upon detection of predefined activation criteria, e.g. by changing the switching status of the corresponding switch 22, 23, 24, 25. The feedback loops may be inactivated upon detection of predefined deactivation criteria.

Figure 4 shows an advantageous embodiment of a tone mapping curve 17 which is based upon a predefined shape in the form of a Reinhard curve. Figure 5 shows another advantageous embodiment of a tone mapping curve 17 which is based on a predefined shape in the form of a Reinhard curve blended with a modified sigmoid curve. The modified sigmoid curve has a raised starting point which makes it possible to have a steep start and swiftly change to a curve with slower growth.

It has been a known challenge to use a single global tone mapping curve for an image with at least two very different distributions of light, for example, when driving a car out of a tunnel. The tunnel exit is bright but the light in the tunnel is dim, and we would like to be able to see both - the tunnel inside and outside - clearly. In the tunnel exit image case, it is advantageous to have a steep start of the curve to brighten the dark parts but also slow slope until the end to not make the outside of the tunnel overexposed. The difference in this new curve compared to the Reinhard curve is in the creation of the sigmoid blending curve. The 2 graphs in figures 4 and 5 show that they have a required steep start in a tone curve. With the Reinhard equation, the curve of Figure 4 will be steep almost until the very max mapping value. But with the curve of Figure 5 we can have the same steep start but cut the steep start faster and continue with almost linear line until the end. In a high contrast image, the results are as follows: the dark parts will get brighter but bright parts will remain almost the same.

In Figure 6 an advantageous method for generating the training data for neural network 11 is shown. As one input stream for training data 27 to be generated, a low dynamic range ground truth image 26 is taken from a dataset and provided to block 30. As a second input stream, in block 27, parameters 33 for generating a tone mapping curve based upon a predefined shape of such tone mapping curve are calculated. Parameters 33 can be generated randomly. In the following block 28 tone mapping curve 34 is created from parameters 33. In the following block 29 tone mapping curve 34 is inverted, resulting in an inverse tone mapping curve 35. This inverse tone mapping curve 35 is provided to block 30.

In block 30, ground truth image 26 is mapped by applying inverse tone mapping curve 35 to image 26. The result is a degraded image in the form of a high dynamic range training image 36 which is provided to block 31. In block 31 , a histogram 37 of the high dynamic range training image 36 is calculated. The result of block 31 is provided as training data 32 for training neural network 11.

*****

Claims

Claims:

1. A method for tone mapping of a high dynamic range input image (6) into a low dynamic range output image (7), the method comprising at least the following steps: a) Obtaining a brightness component of the input image (6), b) Generating a tone mapping curve (17) or at least parts (16) thereof by a neural network (11) based upon the obtained brightness component of the input image (6), c) Generating the output image (7) by applying the tone mapping curve (17) to at least one brightness component of the input image (6).

2. The method of claim 1 , characterized in that the method comprises calculating a brightness histogram (15) that represents the brightness distribution of the input image (6) and feeding in the brightness histogram (15) into the neural network (11) as input data for the generation of the tone mapping curve (17) or the parts (16) thereof.

3. The method of claim 2, characterized in that the brightness histogram (15) is calculated based upon at least two previously calculated brightness histograms (15).

4. The method according to any of the claims 2 to 3, characterized in that the brightness histogram (15) is calculated as a combination of a linear image histogram and a logarithmic image histogram.

5. The method according to any of the preceding claims, characterized in that the neural network (11) generates only tone curve parameters (16) of a predefined shape of the tone mapping curve (17), wherein the tone mapping curve (17) is then created by applying the tone curve parameters (16) generated by the neural network (11) to the predefined shape of the tone mapping curve (17).

6. The method according to any of the preceding claims, characterized in that the predefined shape of the tone mapping curve (17) is the shape of a Reinhard curve, an extended Reinhard curve, a freely defined curve and/or one of the aforementioned curve shapes blended with a sigmoid function.

7. The method according to any of the preceding claims, characterized in that the neural network (11) is trained using training data (26) based upon a high dynamic range training image (30) and a corresponding low dynamic range ground truth image (20).

8. The method according to any of the preceding claims, characterized in that the tone mapping curve (17) is generated for the whole image value range from 0 to 2ⁿ-1 or at least 90% of this range, wherein n is the number of bits of the high dynamic range input image (6).

9. The method according to any of the preceding claims, characterized in that a curve loss is calculated based upon the tone mapping curve (17) using the generated curve parameters (16) and a ground truth curve.

10. The method according to any of the preceding claims, characterized in that the generated tone mapping curve (17) or the parts (16) thereof, the generated output image (7) and/or any other data based upon output data of the neural network (11) is fed back in at least one feedback loop (18, 19, 20, 21) as new input data for the neural network (11).

11. Computer program having program code means configured for performing a method according to any of the preceding claims when the computer program is executed on a computer.

12. Electronic device (2) configured for performing a method according to any of the claims 1 to 10.

13. A method for training a neural network (11) of an electronic device (2), which is configured for tone mapping of a high dynamic range input image (6) into a low dynamic range output image (7), in particular an electronic device (2) of claim 11, comprising the following steps: a) Obtaining a low dynamic range ground truth image (20), b) Generating a high dynamic range training image (30) by adding degradation to the low dynamic range ground truth image (20), c) Training the neural network (11) using the high dynamic range training image (30) and the low dynamic range ground truth image (20).

14. The method according to claim 13, characterized in that the high dynamic range training image (30) is generated by applying an inverse tone mapping curve (29) to the low dynamic range ground truth image (20).