KR20220108664A - System and method for image restoration based on multiple exposure image - Google Patents

Abstract

A method in which an image restoration system operated by at least one processor operates is provided. Learning data is generated by collecting one low dynamic range (LDR) image and multi-exposure images generated by transforming an exposure value for the LDR image, and a multi-exposure image stack generator is trained with the learning data. When one LDR image is received from a photographing means, an exposure value of the LDR image is extracted as a reference exposure value. An inverse camera response function is estimated using a multi-exposure image stack including multi-exposure images generated based on the LDR image and the reference exposure value in the trained multi-exposure image stack generator. The LDR image is restored into an HDR image by using a gradient collected through the inverse camera response function and the inverse camera response function. An end-to-end learning technique is used to restore a desired HDR image accurately.

Description

System and method for image restoration based on multiple exposure image

The present invention relates to an image processing system and method for reconstructing a high dynamic range (HDR) image based on a differentiable multiple exposure image.

Recently, in order to provide a better aesthetic appreciation than general imaging technology with a limited dynamic range, HDRI (High Dynamic Range Imaging) technology has been used in various fields. In addition, HDRI aims to restore underexposed and overexposed areas so that the reconstructed HDR image conveys a lot of information, such as image details, regardless of illuminance changes.

Various methods are used to restore HDR images. Among them, deep learning-based HDR image restoration techniques are widely used, and deep learning-based HDR image restoration techniques are largely divided into direct restoration techniques and multi-exposure image-based restoration techniques.

The direct restoration technique is a technique for restoring a given single low dynamic range (LDR) image to an HDR image. The multiple-exposure image-based restoration technique is a technique for restoring a multiple-exposure image, which is an intermediate step, using a given single LDR image.

In the case of the direct restoration technique, there exist a method of restoring an image of an overexposed or underexposed area and a method focusing on restoring an HDR image directly from an LDR image. In the direct restoration technique, the network needs to know the various environments in which the input LDR image was created to be able to restore the HDR image. Therefore, a large amount of LDR-HDR data pairs is required because data sets for various environments are required.

In the case of the multi-exposure image-based technique, learning can be performed only with images having different exposure values. Accordingly, training data for various scenes is not required. In addition, when only image data and a corresponding exposure value are given, there is an advantage that it shows performance similar to the direct restoration technique even with a small amount of data.

However, in the case of the multiple exposure image method, a discrete function structure is mainly used in the process of estimating (radiometric calibration) a mapping function between a pixel brightness value and an actual light brightness in order to reconstruct an HDR image from an LDR image. Therefore, an HDR image cannot be learned as a target image because non-differentiable elements exist. Accordingly, even if the multiple-exposure image is accurately restored, a difference between the actually generated or synthesized HDR image and the generated HDR image occurs.

Therefore, the present invention does not focus on restoring a multiple-exposure image, but an image restoration system and method for restoring an HDR image based on a multiple-exposure image using an end-to-end learning technique for an accurate HDR image restoration. provides

As a method of operating an image restoration system operated by at least one processor, which is a feature of the present invention for achieving the technical problem of the present invention,

Collecting one LDR (Low Dynamic Range) image and multiple exposure images generated by modifying the exposure value for the LDR image, generating training data including the collected LDR image and the multiple exposure images, and the learning training a multiple exposure image stack generator with data, when receiving one LDR image from a photographing means, extracting an exposure value of the LDR image as a reference exposure value, the learned multiple exposure image stack generator with the reference exposure value and the estimating an inverse camera response function using a multiple exposure image stack including multiple exposure images generated based on the LDR image; restoring the image to an HDR image.

The step of training the multiple exposure image stack generator may include: when the LDR image is input to the multiple exposure image stack generator, learning to generate an increased exposure value generated by increasing the exposure value, applying the LDR image to the multiple exposure When input to the image stack generator, learning to generate a reduced exposure image generated by reducing the exposure value, and generating a plurality of exposure value increased images and a plurality of reduced exposure value images as one multi-exposure image stack may include steps.

The multiple exposure image stack generator may include an exposure value increase image generation model trained to generate the exposure value increase image, and an exposure value decrease image generation model learned to generate an exposure value decrease image.

The estimating of the inverse camera response function may include generating pixel brightness samples by mapping a brightness value for each pixel extracted from each pixel of a plurality of images included in the multi-exposure image stack to an actual brightness value of light; and The method may include estimating the inverse camera response function based on the generated pixel brightness samples.

The restoring to the HDR image may include converting the estimated inverse camera response function into a linearized inverse camera response function, and obtaining the slope from the linearized inverse camera response function.

The restoring to the HDR image may include obtaining the gradient by differentiating the linearized inverse camera response function.

As an image restoration system for restoring a plurality of multiple exposure images and a High Dynamic Range (HDR) image from one LDR (Low Dynamic Range) image, which is another feature of the present invention for achieving the technical problem of the present invention,

An interface for receiving the single LDR image from an image collection device, and a processor, wherein the processor collects multiple exposure images generated by modifying a reference exposure value extracted from the LDR image as training data, and the collected training data When the LDR image included in is input, the multiple exposure image stack generator is trained so that the multiple exposure images are output, the inverse camera response function is estimated using the generated multiple exposure images, and the estimated inverse camera response function is linearized. The gradient is extracted, and the HDR image is restored by inputting the gradient and the one received LDR image.

The processor is configured to increase the reference exposure value to generate a plurality of exposure value increase images and decrease the reference exposure value to generate a plurality of exposure value decrease images, wherein the plurality of exposure value increase images and the plurality of exposure value increase images are generated. Reduced value images can be created as one multi-exposure image stack.

The processor generates a plurality of pixel brightness samples by mapping a brightness value for each pixel extracted from each pixel of a plurality of images included in the multiple exposure image stack to an actual light brightness value, and using the generated pixel brightness samples The inverse camera response function may be estimated.

The processor may convert the estimated inverse camera response function into a linearized inverse camera response function, and obtain the slope by differentiating the linearized inverse camera response function.

According to the present invention, since the non-differentiable mapping function estimation module is changed to be differentiable and used, it is possible to restore the HDR image from the LDR image.

In addition, since a structure to which a cyclic structure and a conditional instance normalization layer are applied is employed to increase network connectivity, connectivity between images having variously changed exposure values can be strengthened when generating a multi-exposure image.

In addition, a neural network image reconstructor can be trained using a small image data set, and HDR restoration performance can be improved.

1 is an exemplary diagram of an environment to which an image restoration system according to an embodiment of the present invention is applied.
2 is a structural diagram of an image restoration system according to an embodiment of the present invention.
3 is a flowchart of an image restoration method according to an embodiment of the present invention.
4 is an exemplary diagram of deriving a differentiable linear function through an interval linear model according to an embodiment of the present invention.
5 is an exemplary diagram of a sub-network structure according to an embodiment of the present invention.
6 is an exemplary diagram comparing the HDR image restoration results using the image restoration method according to an embodiment of the present invention and the conventional image restoration methods.
7 is a structural diagram of a computing device according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

Hereinafter, a multi-exposure image-based image restoration system and method according to an embodiment of the present invention will be described in detail with reference to the drawings.

1 is an exemplary diagram of an environment to which an image restoration system according to an embodiment of the present invention is applied.

As shown in FIG. 1 , the image restoration system 100 restores an HDR image based on a multiple exposure image using an end-to-end learning technique. That is, the image restoration system 100 generates a multi-exposure image stack using one input LDR image. Then, the image restoration system 100 synthesizes the HDR image using the generated multi-exposure image stack.

In this case, the image restoration system 100 generates a multi-exposure image stack by using one LDR image. Here, the image restoration system 100 trains a multiple exposure image stack generator to generate a plurality of multiple exposure images from one LDR image. And, by sampling pixel values of a plurality of multiple exposure images included in the generated multiple exposure image stack, g, which is an inverse camera response function, is estimated.

The inverse camera response function is a function having discrete characteristics that cannot be differentiated, and the image restoration system 100 transforms this function into a differentiable function. Then, the gradient is extracted using the transformed inverse camera response function.

The image restoration system 100 restores the luminance value of the image by using the extracted gradient, and restores the HDR image from the LDR image.

Here, the structure of the image restoration system 100 will be described with reference to FIG. 2 .

2 is a structural diagram of an image restoration system according to an embodiment of the present invention.

As shown in FIG. 2 , the image restoration system 100 includes an LDR image receiver 110 , a multiple exposure image stack generator 120 , and an image restoration unit 130 . The multiple exposure image stack generator 120 includes an exposure value increase image generator 121 and an exposure value decrease image generator 122 .

When the LDR image receiving unit 110 receives one LDR image, the LDR image is extracted from the LDR image by an exposure value of an image collecting means (not shown) in which the corresponding LDR image is collected. Here, the form of the LDR image may be collected in the form of JPEG, TIFF, Bmp, etc., and the LDR image is not limited to any one.

Since the method for the LDR image receiver 110 to extract the exposure value from the LDR image can be performed in various ways, the embodiment of the present invention will not be limited to any one method. In this case, the exposure value extracted by the LDR image receiver 110 may be referred to as a reference exposure value for convenience of explanation, or may be expressed as 0.

The exposure value increase image generator 121 generates an image having an exposure value increased by one from a reference exposure value that is an exposure value of the input LDR image. Then, an image having one more increased exposure value is generated by using the image having one increased exposure value.

Similarly, the exposure reduction image generator 122 generates an image having an exposure value reduced by one from a reference exposure value that is an exposure value of the input LDR image. Then, an image having a further reduced exposure value is generated by using the image having one reduced exposure value.

In an embodiment of the present invention, the exposure value increase image generator 121 generates three exposure value increase images, and the exposure value decrease image generator 122 generates three exposure value decrease images to generate a multi-exposure image stack of 6 sheets. An example will be described. At this time, the exposure value increase image generator 121 and the exposure value decrease image generator 122 increase or decrease the exposure value one by one from the reference exposure value to generate an image in which the exposure value is increased or decreased. The exposure value increase image generator 121 and the exposure value decrease image generator 122 are trained.

That is, the exposure value increase image generator using training data including the LDR image and the first exposure value increase image generated by increasing the exposure value by one from the reference exposure value, and the second exposure value increase image generated by increasing the exposure value of the LDR image twice (121) is learned. Similarly, the exposure value reduction image generator using the training data including the LDR image, the first reduced exposure image generated by reducing the exposure value by one from the reference exposure value, and the second reduced exposure value image generated by reducing the exposure value of the LDR image twice (122) is learned. In the embodiment of the present invention, it is described as an example of including two images of increased exposure value and decreased image of exposure value in the training data, but is not necessarily limited thereto.

The image restoration unit 130 restores multiple exposure images and an HDR image from a single LDR image. Here, the neural network image restorer obtains a gradient from the first model estimating the inverse camera response function to output the LDR image as a multi-exposure image and the inverse camera response function estimated through the multi-exposure image stack to convert the LDR image into an HDR image It consists of a second model, which is an HDR synthesis layer that is restored to .

The image restoration unit 130 estimates a mapping function, that is, an inverse camera response function, that maps the brightness value for each pixel extracted from the pixels of the multiple exposure images to the actual luminance of light. Through this, the image restoration unit 130 generates an inverse camera response function with the brightness value as the X axis and luminance as the actual light brightness as the Y axis, and maps the values to multiple exposure image pixel brightness samples (hereinafter, described below). For the convenience of , referred to as a 'pixel brightness sample').

In addition, the image restoration unit 130 converts the inverse camera response function estimated by the first model into a differentiable linear function based on a piece wise linear model.

Here, the slope is a value obtained by transforming the inverse camera response function of the discrete function structure into a differentiable linear function by applying the piecewise linear method, and differentiating the graph generated from the differentiable linear function. This will be described in detail later.

A method for the image restoration system 100 described above to restore the multiple exposure images and the HDR image from the LDR image will be described with reference to FIG. 3 .

3 is a flowchart of an image restoration method according to an embodiment of the present invention.

As shown in FIG. 3 , the image restoration system 100 trains a neural network image restoration using training data. That is, a neural network image reconstructor is trained using a multiple exposure image stack including one LDR image and a plurality of multiple exposure images.

To this end, the image restoration system 100 extracts an exposure value from a single LDR image as a reference exposure value (S100). In addition, the image restoration system 100 generates training data including images of which a plurality of exposure values are adjusted by using the LDR image and the reference exposure value (S110).

When the image restoration system 100 generates a multi-exposure image stack, it generates images in which the exposure value is increased by increasing the reference exposure value by one, and images in which the exposure value is decreased by decreasing the reference exposure value by one. Since the image restoration system 100 increases or decreases the exposure value by one to generate the images with the changed exposure value can be executed in various ways, the embodiment of the present invention will not be limited to any one method.

The image restoration system 100 trains the exposure value increase image generator 121 and the exposure value decrease image generator 122 constituting the multi-exposure image stack generator 120 with the generated training data ( S120 ).

That is, when the LDR image is input to the exposure value increase image generator 121, a first exposure value increase image having an exposure value increased by one from the reference exposure value of the LDR image, and a second exposure value increase image having an exposure value increased by two from the reference exposure value are output learn as much as possible. Similarly, when the LDR image is input to the reduced exposure image generator 122, a first reduced exposure image having an exposure value reduced by one from the reference exposure value of the LDR image, and a second reduced exposure image having an exposure value reduced by two from the reference exposure value are output learn as much as possible.

In this way, after learning the multiple exposure image stack generator 120 , the image restoration system 100 extracts a reference exposure value from an input LDR image, and then multiple exposures through the trained multiple exposure image stack generator 120 . An image stack is created (S130).

The image restoration system 100 estimates an inverse camera response function that maps the brightness value for each pixel extracted from each pixel of the multiple exposure images to the actual light brightness ( S140 ). In this case, the image restoration system 100 estimates the inverse camera response function by using the following Equation (1).

Here, O is the objective function, g is the inverse camera response function, and Z _ij is the brightness value of the i-th pixel having the j-th exposure value. And Z _min and Z _max mean the minimum and maximum intensity values of a given LDR image.

In addition, N denotes the number of images, and P denotes the number of images included in the multi-exposure image stack. i refers to the index of the image, and refers to a value indicating the number of the image. And, j means an index of an exposure value, and corresponds to a value ranging from -3 to +3 in the embodiment of the present invention.

And E _i represents the luminance value of the i-th pixel, and EV _j represents the j-th exposure value. The second term in the objective function is a value constraining the size of the second derivative of the inverse camera response function to be minimized. play a role

By minimizing the objective function obtained in this way, it is possible to obtain a discrete camera response function that maps an 8-bit pixel-specific brightness value to a 32-bit luminance value. And, by using the restored inverse camera response function, the brightness value of each pixel can be mapped to the luminance as shown in Equation 2 below.

The image restoration system 100 samples pixels from a multi-exposure image stack to sample pixels of the same coordinates having different exposure values, and then estimates the inverse camera response function by using Equation (1). Then, the image restoration system 100 converts the function into an inverse camera response function of a differentiable linear form using the linear model ( S150 ).

That is, the luminance value of the image can be mapped by Equation 2, but the inverse camera response function is a non-differentiable function. Accordingly, the image restoration system 100 converts a discrete inverse camera response function into a linearized inverse camera response function by using a linear approximation technique.

At this time, the image restoration system 100 calculates the inverse camera response function g = [p ₀ , p ₁ , ... , p _N ], where N is the maximum intensity value of the multiple exposure image. The image restoration system 100 defines the derivative of the linearized inverse camera response function as in Equation 3 below.

As such, when the linearized inverse camera response function is obtained, the image restoration system 100 obtains a slope by differentiating the linearized inverse camera response function (S160). That is, such as the slope for the first interval between the first sample and the second sample, the slope for the second interval between the second sample and the third sample, etc., the image restoration system 100 is a linearized inverse camera response function g Obtain the slopes for the intervals between samples of , respectively.

Then, the image restoration system 100 outputs an HDR image using the gradients and the LDR image ( S170 ).

Here, an example of deriving the inverse camera response function estimated in step S140 as a differentiable linear function in step S150 will be described with reference to FIG. 4 .

4 is an exemplary diagram of deriving a differentiable linear function through an interval linear model according to an embodiment of the present invention.

When a multi-exposure image stack is generated from the input LDR image as shown in (a) of FIG. 4, the image restoration system 100 samples pixels as shown in (b) of FIG. Aggregates pixel brightness samples of the coordinates.

The image restoration system 100 estimates the inverse camera response function using Equation 1 as shown in FIG. 4C . And, as shown in FIG. 4(d), the inverse camera response function estimated using the linearization for each image fragment is converted into a differentiable linear form.

5 is an exemplary diagram of a sub-network structure according to an embodiment of the present invention.

5, the exposure value increase image generator 120 and the exposure value decrease image generator 130 according to the embodiment of the present invention each include three sub-networks of a U-Net structure. Then, using three sub-networks, an image is generated by increasing the relative exposure value or decreasing the exposure value.

Here, the three sub-networks correspond to the global network, the local network, and the redefining network. Conventionally, it is difficult to generate a recursive multiple exposure image, but in an embodiment of the present invention, an image with an increased or decreased exposure value according to the number of recursions can be easily generated using the global network and the local network, which are sub-networks.

That is, the global network minimizes the histogram difference between the image generated by adjusting the generated exposure value and the target exposure value image, and the local network focuses on generating a gradient-based edge structure. Then, each image is applied to the redefining network to create one multi-exposure image stack.

6 is an exemplary diagram comparing the HDR image restoration results using the image restoration method according to an embodiment of the present invention and the conventional image restoration methods.

As shown in FIG. 6 , images were restored using six deep learning-based methods, that is, various conventionally known direct methods and multiple exposure stack-based methods, and the results of restoration of HDR images restored according to an embodiment of the present invention were compared. . Here, HDR images generated by Single HDR, Deep Recursive, FHDR, ExpandNet, DrTMO, and HDRCNN using six deep learning-based methods were compared with HDR images generated by the restoration method according to an embodiment of the present invention.

Way training dataset
number VDS HDR-Eye RAISE m±σ m±σ m±σ embodiment of the present invention 48 scenes 58.807±5.413 55.914±1.917 59.493±3.420 JDRCMM 3,700 scenes 53.031±4.957 50.804±5.790 57.154±3.642 DrTMO 1,043 scenes 55.227±4.662 51.800±5.933 57.645±4.028 Deep recursive HDRI 48 scenes 56.347±3.492 52.832±2.944 57.570±3.697 ExpandNet 1,013 scenes 44.720±9.432 50.428±4.493 54.717±1.998 FHDR 39,460 scenes 57.708±6.373 53.815±3.603 59.144±2.764 SingleHDR 10,289 scenes 55.237±4.487 54.509±3.714 59.304±3.541

At this time, when measuring the quality of the generated HDR image, a conventionally known method as a stack-based method was applied. As shown in Table 1, it can be seen that the size of the training data set for various conventional methods is unbalanced.

However, it can be seen that the restoration method according to the embodiment of the present invention performs better than the conventional direct method or multi-exposure stack-based method, which has an advantageous HDR-VDP-2 score, despite learning with much fewer images compared to the conventional methods. can

7 is a structural diagram of a computing device according to an embodiment of the present invention.

As shown in FIG. 7 , the image restoration system 100 executes a program including instructions described to execute the operation of the present invention in the computing device 200 operated by at least one processor do.

The hardware of the computing device 200 may include at least one processor 210 , a memory 220 , a storage 230 , and a communication interface 240 , and may be connected through a bus. In addition, hardware such as an input device and an output device may be included. The computing device 200 may be loaded with various software including an operating system capable of driving a program.

The processor 210 is a device for controlling the operation of the computing device 200 , and may be a processor 210 of various types for processing instructions included in a program, for example, a central processing unit (CPU), an MPU ( It may be a micro processor unit), a micro controller unit (MCU), a graphic processing unit (GPU), or the like. The memory 220 loads the corresponding program so that the instructions described to execute the operation of the present invention are processed by the processor 210 . The memory 220 may be, for example, read only memory (ROM), random access memory (RAM), or the like. The storage 230 stores various data, programs, etc. required for executing the operation of the present invention. The communication interface 240 may be a wired/wireless communication module.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improved forms of the present invention are also provided by those skilled in the art using the basic concept of the present invention as defined in the following claims. is within the scope of the right.

Claims (10)

A method of operating an image restoration system operated by at least one processor, the method comprising:
Collecting one LDR (Low Dynamic Range) image and multiple exposure images generated by modifying the exposure value for the LDR image;
generating training data including the collected LDR image and the multiple exposure images, and training a multiple exposure image stack generator with the training data;
When receiving one LDR image from the photographing means, extracting the exposure value of the LDR image as a reference exposure value;
estimating an inverse camera response function using a multiple exposure image stack including multiple exposure images generated based on the reference exposure value and the LDR image in the trained multiple exposure image stack generator;
Restoring the LDR image to an HDR image using the gradient collected through the inverse camera response function and the inverse camera response function
comprising, a method of operation. According to claim 1,
The step of training the multiple exposure image stack generator comprises:
learning to generate an increased exposure value generated by increasing the exposure value when the LDR image is input to the multiple exposure image stack generator;
when the LDR image is input to the multiple exposure image stack generator, learning to generate an exposure value reduced image generated by reducing the exposure value; and
generating a plurality of increased exposure value images and a plurality of reduced exposure value images as one of the multiple exposure image stacks;
comprising, a method of operation. 3. The method of claim 2,
The multiple exposure image stack generator comprises:
An operating method comprising: an exposure value increase image generation model trained to generate the exposure value increase image; and an exposure value decrease image generation model learned to generate an exposure value decrease image. 4. The method of claim 3,
Estimating the inverse camera response function comprises:
generating pixel brightness samples by mapping a brightness value for each pixel extracted from each pixel of a plurality of images included in the multiple exposure image stack to an actual brightness value of light; and
estimating the inverse camera response function based on the generated pixel brightness samples
comprising, a method of operation. 5. The method of claim 4,
Restoring the HDR image includes:
converting the estimated inverse camera response function into a linearized inverse camera response function, and
obtaining the slope from the linearized inverse camera response function;
comprising, a method of operation. 6. The method of claim 5,
Restoring the HDR image includes:
and obtaining the slope by differentiating the linearized inverse camera response function. An image restoration system that restores a plurality of multiple exposure images and a High Dynamic Range (HDR) image from a single LDR (Low Dynamic Range) image,
an interface for receiving the one LDR image from an image acquisition device; and
processor
including,
The processor is
The multiple exposure images generated by modifying the reference exposure value extracted from the LDR image are collected as training data, and the multiple exposure image stack generator is trained so that the multiple exposure images are output when the LDR image included in the collected training data is input. , estimating the inverse camera response function using the generated multiple exposure images, extracting the slope by linearizing the estimated inverse camera response function, and restoring the HDR image by inputting the slope and the one received LDR image , image restoration system. 8. The method of claim 7,
The processor is
increasing the reference exposure value to generate a plurality of exposure value increasing images, decreasing the reference exposure value to generate a plurality of exposure value decreasing images, and dividing the plurality of exposure value increase images and the plurality of exposure value decrease images An image restoration system that creates a single multi-exposure image stack. 9. The method of claim 8,
The processor is
A plurality of pixel brightness samples are generated by mapping a brightness value for each pixel extracted from each pixel of a plurality of images included in the multiple exposure image stack to an actual light brightness value, and the inverse camera response is performed using the generated pixel brightness samples. Estimating function, image restoration system. 10. The method of claim 9,
The processor is
Transforming the estimated inverse camera response function into a linearized inverse camera response function, and differentiating the linearized inverse camera response function to obtain the gradient.

Images