KR20220028814A - Apparatus for inverse tone mapping and method for inverse mapping through exposure changing using the same - Google Patents

Abstract

The present invention relates to an inverse tone mapping method for allowing an inverse tone mapping apparatus to perform inverse tone mapping on an LDR image. The method includes: generating an LDR image, a target exposure value to be reflected in the LDR image, and training data obtained by mapping the target exposure value to a correct answer image; when a deep neural network is trained using the training data and a target image for reverse tone mapping and a target exposure value are received, generating a target exposure value formed by reflecting image reflecting the target exposure value in the target image by using the learned deep neural network.

Description

Apparatus for inverse tone mapping and method for inverse mapping through exposure changing using the same}

The present invention relates to an inverse tone mapping apparatus and inverse tone mapping through exposure change using the same.

As the maximum output brightness of the display increases and the color gamut increases, high dynamic range (HDR) images are created from low dynamic range (LDR) images that lose information in overexposed and underexposed areas. Interest in the Inverse Tone Mapping algorithm is increasing.

A general inverse tone mapping algorithm captures several low dynamic region images with different exposures for the same scene, and then merges several low dynamic region images into a high dynamic region image. In the case of using the inverse tone mapping algorithm, ghosting and blurring may occur due to camera or object movement in the process of photographing the same scene multiple times.

In order to solve this problem, inverse tone mapping algorithms based on deep learning are being actively studied recently, and they show high performance compared to existing algorithms. Deep learning-based inverse tone mapping algorithms can be divided into direct approaches and stack-based approaches.

The direct approach is a technique for directly generating a high dynamic region image from a single low dynamic region image. The direct approach is the simplest deep learning-based inverse tone mapping algorithm, which directly learns end-to-end mapping of high dynamic region images from low dynamic region images. However, since one low dynamic region image may correspond to a plurality of high dynamic region images having different dynamic regions, an object to be generated by the network becomes unclear.

The stack-based approach is a technique to generate multiple images with different exposure values from a single low dynamic region image and merge them into a high dynamic region image through a general inverse tone mapping algorithm. Deep Recursive HDRI, a typical stack-based approach, generates multiple images with different exposure values by recursively using two convolutional neural networks that raise and lower the exposure value, respectively.

With the stack-based approach, the network does not get confused in the learning process because what needs to be created is clear. However, when the same network is used recursively, a problem arises in that errors generated in each reasoning process are accumulated or amplified to the next reasoning process.

For this reason, in the process of generating an image having a large difference between the input image and the exposure value, the image quality is deteriorated. As a result, in the high dynamic area image merging process, the detail of the severely overexposed or underexposed area due to the degraded image is not restored and a crushed image is obtained.

Accordingly, the present invention provides an inverse tone mapping apparatus and an inverse tone mapping method using an exposure control network that directly changes an arbitrary target exposure value in order to solve the error accumulation problem due to recursive reasoning.

As a method for inverse tone mapping an LDR image by an inverse tone mapping apparatus, which is a feature of the present invention for achieving the technical problem of the present invention,

Generating the LDR image, the target exposure value to be reflected in the LDR image, and training data mapping the target exposure value to the correct answer image, training a deep neural network using the training data, and performing reverse tone mapping and receiving a target image and a target exposure value, and generating a target exposure value reflection image in which the target exposure value is reflected in the target image by using the learned deep neural network.

The trained deep neural network includes an LDR image and an exposure control network trained to output a target exposure value reflected image by reflecting the weight in the LDR image when a weight generated based on the target exposure value is input, and a target exposure When a value and luminance information extracted from the LDR image are input, the display may include a luminance characteristic generator learned to generate luminance information to adjust exposure for each of a plurality of pixels constituting the LDR image as the characteristic map.

The exposure control network may be based on Auxiliary classifier-generative adversarial networks.

The step of training the deep neural network includes generating an exposure control image by reflecting the initial weight of the exposure control network in the LDR image, a difference value between each pixel value of the exposure control image and each pixel value of the correct image , and learning the exposure tidal network by adjusting an arbitrary value corresponding to the pixel in which the difference value occurs by the difference value.

The generating of the target exposure value reflection image may include receiving a target LDR image to be reverse tone mapped and a target exposure value, extracting a luminance value from the LDR image, and based on the luminance value and the target exposure value The method may include generating a characteristic map, and generating an image reflecting the target exposure value by reflecting the characteristic map in the target LDR image.

After the step of generating the target exposure value reflection image, the target exposure value reflection image and the received target LDR image are received as one set, and the target exposure value reflection image included in the received one set is, The method may further include the step of discriminating whether the target exposure value is the reflected target exposure value or the correct answer image.

As an inverse tone mapping system for inverse tone mapping of an LDR image, which is another feature of the present invention for achieving the technical problem of the present invention,

a memory for storing training data for each correct answer image, including at least one instruction, an interface for receiving a target LDR image and a target exposure value to be reflected in the target LDR image, and a processor, wherein the processor includes the correct answer image Check LDR images and exposure values from star learning data, train a deep neural network using the LDR images and the exposure values, and use the received target LDR image and target exposure value with the learned deep neural network An image reflecting the target exposure value is generated from the target LDR image.

The processor may initialize the weights when the LDR image included in the training data is input.

The processor generates an exposure control image by reflecting the initialized weight in the LDR image, calculates a difference value between each pixel value of the exposure control image and each pixel value of the correct answer image, and the difference value occurs The weight corresponding to the pixel may be adjusted by the difference value.

The processor may extract a luminance value from the LDR image and generate a characteristic map based on the luminance value and the target exposure value.

The processor may distinguish whether the target exposure value reflection image is an image reflecting the target exposure value or a correct answer image based on the target exposure value reflection image and the received target LDR image.

The processor may generate an HDR image for the LDR image by merging the plurality of images reflecting the target exposure value.

According to the present invention, details of severely overexposed or underexposed areas including light sources can be reconstructed distinctly by using the inverse tone mapping technique through exposure change.

1 is a block diagram of an inverse tone mapping apparatus according to an embodiment of the present invention.
2 is an exemplary diagram illustrating a learning process of an exposure control network according to an embodiment of the present invention.
3 is an exemplary diagram of an exposure control network according to an embodiment of the present invention.
4 is a flowchart illustrating a method of operating an inverse tone mapping apparatus according to an embodiment of the present invention.
5 is an exemplary diagram illustrating a performance comparison of an inverse tone mapping apparatus according to an embodiment of the present invention and a general technique.
6 is a hardware configuration diagram of a computing device according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part "includes" a certain element, it means that other elements may be further included, rather than excluding other elements, unless otherwise stated.

Hereinafter, an apparatus for inverse tone mapping according to an embodiment of the present invention and a method for inverse tone mapping by changing an exposure using the apparatus will be described with reference to the drawings.

1 is a block diagram of an inverse tone mapping apparatus according to an embodiment of the present invention.

1 , the inverse tone mapping apparatus 100 receives a low dynamic range image (LDR) and an exposure value (EV), and an exposure value having an exposure changed by the target exposure value. This reflected image is created. The reverse tone mapping apparatus 100 generates a high dynamic range (HDR) image using images to which a plurality of exposure values are reflected.

The inverse tone mapping apparatus 100 includes an exposure control network 110 which is a deep neural network and a brightness feature generator 120 , and in addition to the deep neural network, a converter 130 , a merge unit 140 , and a discriminator 150 . ) is included. In the embodiment of the present invention, although the exposure control network 110 and the brightness characteristic generator 120 are represented as distinct components, they may be included in one deep neural network. That is, the brightness characteristic generator 120 may be included in the exposure control network 110 .

The exposure control network 110 receives an LDR image and a target exposure value from the outside. Then, brightness information is received from the brightness characteristic generator 120, which will be described later.

The exposure control network 110 reflects the input target exposure value in the LDR image to generate and output the exposure value reflection image.

To this end, the exposure control network 110 is trained using the training data in advance. In the embodiment of the present invention, learning the exposure control network 110 using the AC-GANs (Auxiliary Classifier-Generative Adversarial Networks, Auxiliary Classifier-Generative Adversarial Networks) structure will be described as an example. When the training data having the input image, the exposure value to be reflected in the image, and the correct answer image with the exposure value reflected as a pair is input to the exposure control network 110, the exposure control network 110 in advance so that the correct answer image with the exposure value is outputted to learn

The brightness characteristic generator 120 receives the target exposure value and the luminance value of the input image, and generates brightness information to be reflected in the LDR image in the exposure control network 110 . Here, the brightness information is information indicating which point is to be adjusted to how much brightness when the target exposure value is reflected in the LDR image, and the brightness information to be reflected for each pixel is generated in the form of a characteristic map as an example .

Both the exposure control network 110 and the brightness characteristic generator 120 are configured with weights. Accordingly, the initial exposure control network 110 is initialized to have an arbitrary weight, and the brightness information initially generated by the brightness characteristic generator 120 is also an arbitrary value based on the initialized weight.

When the LDR image is input to the initialized exposure control network 110 , the brightness value of the image is adjusted with an arbitrary weight, so that a meaningless image is output. In the case of the brightness characteristic generator 120, an output value is generated as a characteristic map rather than an image. Therefore, if an input image is entered into the initialized brightness characteristic generator 120, the brightness value of the image is adjusted with an arbitrary weight, resulting in a meaningless characteristic map. is output In this way, when the initial exposure control network 110 is trained, the brightness characteristic generator 120 is also learned at the same time.

That is, when the exposure control network 110 generates an exposure control image from the LDR image with weights, the exposure control network 110 compares the exposure control image with the correct answer image reflecting the exposure value included in the training data. Then, each pixel of the exposure control image and the correct answer image is compared, and the weights of the initial exposure control network and the brightness characteristic generator are modified to have the learned weights. In the embodiment of the present invention, the procedure for adjusting the initial weights will be described as an example that is executed when the exposure control network 110 is trained.

The converter 130 converts the input RGB LDR image into a color space such as YUV, Y-Pr-Pb, YCbCr, or Y'CbCr, and calculates luminance information Y from the color space. The converter 130 transmits the calculated luminance information to the luminance characteristic generator 120 . Here, the method for the converter 130 to calculate the luminance information from RGB is already known, and detailed description will be omitted in the embodiment of the present invention.

The merging unit 140 merges a plurality of exposure value-reflecting images generated in the exposure control network 110 to generate an HDR image. A method for the merging unit 140 to generate an HDR image by merging a plurality of exposure value-reflecting images is already known, and a detailed description thereof will be omitted in the exemplary embodiment of the present invention.

The discriminator 150 receives the exposure value reflection image and the LDR image generated by the exposure control network 110 as one set, or receives the correct answer image when the LDR image and the exposure value are reflected as one set.

And, the discriminator 150 discriminates whether the image is an exposure value-reflected image or a correct image when the exposure value is reflected. In addition, the discriminator 150 classifies which of the seven target exposure values from EV-3 to EV+3. In the exemplary embodiment of the present invention, seven exposure values ranging from -3 to +3 are described as examples, but the present invention is not limited thereto.

A process of learning the exposure control network with the inverse tone mapping apparatus 100 described above will be described with reference to FIG. 2 .

2 is an exemplary diagram illustrating a learning process of an exposure control network according to an embodiment of the present invention.

As shown in FIG. 2 , the exposure control network 110 of the inverse tone mapping apparatus 100 according to the embodiment of the present invention receives an LDR image and a target exposure value, and the exposure reflected image changed by the target exposure value. to create

In the case of the existing stack-based approach, once trained, the target exposure value is fixed by the training data set. To change this fixed target exposure value, it can be changed only by changing the training data set and re-learning.

On the contrary, since the exposure control network according to the embodiment of the present invention can adjust an arbitrary exposure value including a non-integer type, that is, a real number without re-learning, it is possible to very precisely restore a lost dynamic region.

That is, it is assumed that the training data set includes exposure values of -3, -2, -1, 0, 1, 2, and 3 and images to which each exposure value is applied. In this case, since the images to which the exposure value is applied continuously exist in the polyhedral space when the dimension reduction technique is used, an image in which an exposure value between two integer exposure values is reflected can also be identified.

Accordingly, in an embodiment of the present invention, even if a non-integer, that is, a real number, is input as the target exposure value, the exposure control network 110 may generate an exposure-reflecting image in which the exposure value corresponding to the real number is reflected. Here, the dimension reduction technique is a known technique, and detailed description thereof will be omitted in the embodiment of the present invention.

In addition, the existing stack-based approach has a limitation in that it cannot recover details of information lost due to over- or under-exposure. In the case of exposure adjustment, the brightness change width for each pixel is determined differently according to the luminance of the area by a non-linear function.

Therefore, a composite product operation having spatially equivalent properties in which a filter of the same weight is applied to all pixels is not suitable for adaptively changing the brightness value. Therefore, an embodiment of the present invention proposes a new spatial adaptive normalization that is easy to adaptively change a pixel brightness value by determining a normalization parameter based on a luminance component of an input image and a target exposure value.

At this time, the brightness characteristic generator 120 generates brightness information to estimate an area requiring brightness adjustment, and adds an exposure value conditional convolution that restores a low-frequency component to the network, so that spatial adaptive normalization is performed at a high frequency. Induces you to focus on restoring the ingredients.

3 is an exemplary diagram of an exposure control network according to an embodiment of the present invention.

3, in the embodiment of the present invention, the LDR image, which is the input image, is compressed into a latent representation by the encoder (not shown), and changed by the target exposure value in the decoder (not shown) restore as an image. Here, the compression to the latent representation means to generate a compressed version that best represents the input LDR image, and the method of compressing the latent representation is already known in the field of auto-encoders that generate the input image, an embodiment of the present invention A detailed description is omitted.

To this end, in the embodiment of the present invention, designing a decoder using spatial adaptive normalization and exposure value conditional synthesis product is described as an example, but the present invention is not limited thereto.

In shortcut linking, the target exposure values extended to the spatial dimension of the feature map are connected to the feature map, and then applied to the convolutional product layer to change the low-frequency components. Spatial adaptive normalization receives the brightness information generated by the brightness characteristic generator 120 and changes the high-frequency component.

As the low frequency component reconstructed through the exposure value conditional synthesis product and the high frequency component reconstructed through spatial adaptive normalization are added for each element, the exposure value of the image is changed according to the target exposure value.

For quantitative evaluation of tone-mapped HDR images, in the embodiment of the present invention, the use of Peak Signal-to-Noise Ratio (PSNR) and Structure Similarity Index (SSIM) as indicators will be described as an example. In addition, although quantitative evaluation is performed using Reinhard's tone mapping algorithm as an example, the description is not necessarily limited thereto.

For quantitative evaluation of HDR images, the HDR-VDP-2 score may be used. Table 1 expresses the quantitative performance comparison of the proposed method and Deep Recursive HDRI numerically.

PSNR SSIM HDR-VDP-2 Average Standard Deviation Average Standard Deviation Average Standard Deviation embodiment of the present invention 35.79 3.17 0.965 0.042 60.64 4.98 Deep Recursive HDRI 33.86 3.45 0.947 0.053 58.19 4.79

As shown in Table 1, it can be seen that the method according to the embodiment of the present invention achieves 1.93 dB higher average PSNR, 0.018 higher average SSIM, and 2.45 higher average HDR-CDP-2 than Deep Recursive HDRI. .

4 is a flowchart illustrating a method of operating an inverse tone mapping apparatus according to an embodiment of the present invention.

As shown in FIG. 4 , the inverse tone mapping apparatus 100 trains the exposure control network 110 and the brightness characteristic generator 120 as training data ( S100 ). Here, the training data includes an LDR image, an exposure value, and a correct answer image in which the exposure value is reflected in the LDR image.

That is, the inverse tone mapping apparatus 100 receives the input image, the exposure value to be reflected in the corresponding image, and the learning data having the correct image in which the exposure value is reflected as a pair is input to the exposure control network 110 and the brightness characteristic generator 120 . , the exposure control network 110 and the brightness characteristic generator 120 are trained in advance so that the exposure value-reflected image to which the exposure value is reflected is output. In addition, the inverse tone mapping apparatus 100 compares the exposure value reflection image generated by the exposure control network 110 with the correct image, and adjusts the initial weights of the exposure control network 110 and the brightness characteristic generator 120 .

After learning the exposure control network 110 and the brightness characteristic generator 120 in this way, the inverse tone mapping apparatus 100 receives the LDR image and the target exposure value ( S110 ). Then, the learned exposure control network 110 reflects the exposure value in the LDR image, and generates an exposure value reflection image (S120).

In this case, the inverse tone mapping apparatus 100 extracts a luminance value from the LDR image and provides it to the luminance characteristic generator 120 . The brightness characteristic generator 120 generates a characteristic map based on the luminance value and the target exposure value and transmits it to the exposure control network 110 .

The exposure control network 110 applies the characteristic map to the LDR image to generate an image reflecting the exposure value. The inverse tone mapping apparatus 100 generates an HDR image by merging a plurality of images reflecting the exposure value.

Next, the performance of converting an LDR image into an HDR image when the inverse tone mapping apparatus 100 according to an embodiment of the present invention is used and a general technique is compared with reference to FIG. 5 .

5 is an exemplary diagram illustrating a performance comparison of an inverse tone mapping apparatus according to an embodiment of the present invention and a general technique.

As shown in FIG. 5 , a qualitative performance comparison between the HDR image and Deep Recursive HDRI generated by the inverse tone mapping apparatus 100 according to the embodiment of the present invention can be confirmed. In FIG. 5, (a) is an image of the correct answer, (b) is an image in which a target exposure value is reflected through the inverse tone mapping apparatus 100 according to an embodiment of the present invention, and (c) is an image in the case of using general Deep Recursive HDRI indicates

When an LDR image is input, in the case of Deep Recursive HDRI, the details of severely overexposed or underexposed areas are not restored and the result is crushed due to error accumulation due to recursive reasoning. However, it can be seen that, when the inverse tone mapping apparatus 100 according to the method according to the embodiment of the present invention is used, the details of the severely overexposed or underexposed area including the light source are clearly restored.

6 is a hardware configuration diagram of a computing device according to an embodiment of the present invention.

As shown in FIG. 6 , the inverse tone mapping apparatus 100 executes a program including instructions described to execute the operation of the present invention in the computing device 200 operated by at least one processor. run

Hardware of the computing device 200 may include at least one processor 210 , a memory 220 , a storage 230 , and a communication interface 240 , and may be connected through a bus. In addition, hardware such as an input device and an output device may be included. The computing device 200 may be loaded with various software including an operating system capable of driving a program.

The processor 210 is a device for controlling the operation of the computing device 200, and may be various types of processors 210 that process instructions included in a program, for example, a CPU (Central Processing Unit), MPU (Central Processing Unit) It may be a micro processor unit), a micro controller unit (MCU), a graphic processing unit (GPU), or the like. The memory 220 loads the corresponding program so that the instructions described to execute the operation of the present invention are processed by the processor 210 . The memory 220 may be, for example, read only memory (ROM), random access memory (RAM), or the like. The storage 230 stores various data and programs required for executing the operation of the present invention. The communication interface 240 may be a wired/wireless communication module.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims are also provided. is within the scope of the right.

Claims (12)

A method for an inverse tone mapping device to inverse tone-mapping an LDR image, the method comprising:
generating an LDR image, a target exposure value to be reflected in the LDR image, and learning data in which the target exposure value is mapped to the correct answer image;
training a deep neural network using the training data, and
receiving a target image to be reverse tone mapped and a target exposure value, and generating a target exposure value-reflected image in which the target exposure value is reflected in the target image using the learned deep neural network
An inverse tone mapping method comprising: According to claim 1,
The learned deep neural network is
When an LDR image and a weight generated based on a target exposure value are input, an exposure control network trained to output an image reflecting the target exposure value by reflecting the weight in the LDR image; and
When a target exposure value and luminance information extracted from the LDR image are input, the brightness characteristic generator is trained to generate brightness information to adjust exposure for each pixel constituting the LDR image as the characteristic map.
An inverse tone mapping method comprising: 3. The method of claim 2,
wherein the exposure control network is based on Auxiliary classifier-generative adversarial networks. 4. The method of claim 3,
The step of learning the deep neural network is,
generating an exposure control image by reflecting an initial weight of the exposure control network in the LDR image;
calculating a difference value between each pixel value of the exposure control image and each pixel value of the correct image; and
learning the exposure tidal network by adjusting an arbitrary value corresponding to the pixel in which the difference value occurs by the difference value
An inverse tone mapping method comprising: The method of claim 1,
The step of generating an image reflecting the target exposure value,
receiving a target LDR image to be reverse tone mapped and a target exposure value;
extracting a luminance value from the LDR image;
generating a characteristic map based on the luminance value and the target exposure value; and
generating an image reflecting the target exposure value by reflecting the characteristic map on the target LDR image
An inverse tone mapping method comprising: 6. The method of claim 5,
After generating the image reflecting the target exposure value,
Receive the target exposure value reflection image and the received target LDR image as one set, and whether the target exposure value reflection image included in the received one set is the target exposure value reflection image reflecting the target exposure value Steps to distinguish whether it is an image
Further comprising, an inverse tone mapping method. An inverse tone mapping system for inverse tone mapping of an LDR image, comprising:
A memory that contains at least one instruction and stores learning data for each correct image,
An interface for receiving a target LDR image and a target exposure value to be reflected in the target LDR image, and
processor
including,
The processor is
Check LDR images and exposure values from the training data for each correct image, train a deep neural network using the LDR images and the exposure values, and use the learned deep neural network to determine the received target LDR image and target exposure value An inverse tone mapping system that generates an image reflecting a target exposure value from the target LDR image by using it. 8. The method of claim 7,
The processor is
When the LDR image included in the training data is input, the weight is initialized, an inverse tone mapping system. 9. The method of claim 8,
The processor is
generating an exposure control image by reflecting the initialized weight in the LDR image, calculating a difference value between each pixel value of the exposure control image and each pixel value of the correct image, corresponding to the pixel in which the difference value occurs adjusting a weight by the difference value. 10. The method of claim 9,
The processor is
extracting a luminance value from the LDR image and generating a feature map based on the luminance value and the target exposure value. 11. The method of claim 10,
The processor is
An inverse tone mapping system for discriminating whether the target exposure value reflection image is an image reflecting the target exposure value or a correct answer image based on the target exposure value reflection image and the received target LDR image. 12. The method of claim 11,
The processor is
An inverse tone mapping system for generating an HDR image for the LDR image by merging the plurality of images reflecting the target exposure value.

Images