KR20230088213A - Deep learning decomposition-based multi-exposure image fusion method and apparatus - Google Patents

Abstract

A multi-exposure image fusion method and apparatus based on deep learning decomposition are disclosed. The deep learning decomposition-based multi-exposure image fusion method includes: (a) decomposing each multiple-exposure image into a common component and a residual component at a feature level by applying a deep learning model; (b) fusing each individual component and fusing each common component; and (c) generating a reconstruction image by adding the fused individual component and the fused common component.

Description

Deep learning decomposition-based multi-exposure image fusion method and apparatus

The present invention relates to a deep learning decomposition-based multi-exposure image fusion method and apparatus therefor.

The dynamic range visible to the human eye is much wider than commercial camera sensors. For natural scenes, images shot at single exposure levels often do not yield satisfactory quality in terms of dynamic range. The visibility of the scene is low due to the low dynamic range of the imaging sensor.

The low dynamic range of the image sensor reduces the visibility of the scene in terms of image detail and contrast. To solve this dynamic range problem, Mertens et al. studied multiple exposure image fusion (MEF, hereinafter referred to as MEF). MEF is a High Dynamic Range (HDR) imaging technology for merging multiple Low Dynamic Range (LDR) images with different exposure levels into a high-quality image. In LDR images, visibility is greatly affected by non-uniform lighting conditions and camera exposure levels. For example, details in bright areas are lost through overexposure, while details in dark areas are lost through underexposure. To solve these problems, many non-deep learning-based studies have been conducted and MEF performance has been dramatically improved.

Nonetheless, conventional methods have problems of creating serious visual unnaturalness (detail or color distortion) under some bad conditions (too bright or too dark).

The present invention is to provide a deep learning decomposition-based multi-exposure image fusion method and apparatus therefor.

In addition, the present invention is to provide a deep learning decomposition-based multi-exposure image fusion method and apparatus capable of improving fusion performance by decomposing components on a deep learning network, confirming their characteristics, and fusing them according to characteristics.

In addition, the present invention is to provide a deep learning decomposition-based multi-exposure image fusion method and apparatus capable of solving the problem of detail loss and degradation caused by tidal waves by separating components from a deep learning network and introducing a convergence technique suitable for the components. will be.

In addition, the present invention is to provide a deep learning decomposition-based multi-exposure image fusion method and apparatus capable of contributing to performance improvement in terms of color restoration by introducing fusion in the RGB-domain instead of fusion in the Y-domain.

According to an aspect of the present invention, a multi-exposure image fusion method based on deep learning decomposition is provided.

According to an embodiment of the present invention, (a) decomposing each multi-exposure image into a common component and a residual component at a feature level by applying a deep learning model; (b) fusing each individual component and fusing each common component; and (c) generating a reconstruction image by adding the fused individual component and the fused common component.

The deep learning model may be trained in consideration of equalization loss so that each of the multi-exposure images obtained under different exposure conditions for the same scene has the same common component.

In the deep learning model, weights may be learned in consideration of visualization loss so that each decomposed common component and each individual component are combined into a single image identical to each multi-exposure image.

In step (b), each of the individual components may be fused using a spatial attention weight map.

The deep learning model may be trained to minimize reconstruction loss considering a difference between an output image reconstructed with the fused common component and the fused individual component and the multi-exposure image.

According to another aspect of the present invention, a multi-exposure image fusion device based on deep learning decomposition is provided.

According to an embodiment of the present invention, a decomposition unit for decomposing each multi-exposure image into a common component and a residual component at a feature level by applying a deep learning model to each multi-exposure image; a fusing unit fusing each of the individual components and fusing each of the common components; and a reconstruction unit generating a reconstructed output image using the fused individual component and the fused common component, wherein the deep learning model minimizes equalization loss between common components obtained by decomposing each multi-exposure image. A multi-exposure image fusion device based on deep learning decomposition, characterized in that it is learned, can be provided.

By providing a deep learning decomposition-based multi-exposure image fusion method and apparatus according to an embodiment of the present invention, it is possible to improve fusion performance by decomposing components on a deep learning network, confirming their characteristics, and fusing them according to characteristics. .

In addition, the present invention also has the advantage of solving the detail loss problem and the deterioration problem caused by tidal waves by separating components from the deep learning network and introducing a convergence technique suitable for the components.

In addition, the present invention has an advantage of contributing to performance improvement in terms of color restoration by introducing convergence in the RGB-domain instead of convergence in the Y-domain.

1 is a flowchart illustrating a multi-exposure image fusion method based on deep learning decomposition according to an embodiment of the present invention.
2 is a diagram showing a deep learning network architecture according to an embodiment of the present invention;
3 is a diagram showing a detailed structure of a disassembly module according to an embodiment of the present invention;
4 is a diagram for explaining visualization and a loss function according to an embodiment of the present invention;
5 is a diagram showing a detailed structure of a fusion module according to an embodiment of the present invention.
6 is a diagram for explaining a multi-exposure image fusion method based on deep learning decomposition according to an embodiment of the present invention.
7 is a block diagram schematically showing the internal configuration of a multi-exposure image fusion device based on deep learning decomposition according to an embodiment of the present invention.
8 to 10 are diagrams comparing multiple exposure image fusion results according to an embodiment of the present invention and the conventional one.

Singular expressions used herein include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "consisting of" or "comprising" should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or some of the steps It should be construed that it may not be included, or may further include additional components or steps. In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software. .

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart showing a deep learning decomposition-based multi-exposure image fusion method according to an embodiment of the present invention, FIG. 2 is a diagram showing a deep learning network architecture according to an embodiment of the present invention, and FIG. 3 is a diagram illustrating the present invention. A diagram showing the detailed structure of a decomposition module according to an embodiment of the, Figure 4 is a diagram for explaining visualization and loss function according to an embodiment of the present invention, Figure 5 is an embodiment of the present invention 6 is a diagram showing a detailed structure of a fusion module according to , and FIG. 6 is a diagram for explaining a multi-exposure image fusion method based on deep learning decomposition according to an embodiment of the present invention.

In step 110, the multi-exposure image fusion apparatus 100 based on deep learning decomposition receives a plurality of multiple-exposure images.

Here, the multi-exposure image may be an image acquired (captured) under different exposure conditions for the same scene.

For example, as shown in FIG. 2 , the multi-exposure image may be a low-exposure image and an over-exposure image. In this way, since multiple exposure images are obtained for the same scene with different exposure levels, they may differ from each other in terms of brightness, contrast, and visibility according to exposure time (FIG. 2). reference).

In step 115, the deep learning decomposition-based multi-exposure image fusion apparatus 100 applies a plurality of multi-exposure images to a deep learning model and decomposes each into a common component and a residual component at a feature level.

Figure 2 shows the overall architecture of the deep learning model.

The deep learning model may consider an equalization loss of each decomposed common component when decomposing a plurality of multi-exposure images into common components and individual components at a feature level.

This will be described in more detail.

A multi-exposure image is an image obtained by varying exposure times for the same scene, and structural information is the same, but individual information other than the structural information may be different. That is, overall structural components such as edge components for a scene are common components that do not change even if the exposure time is changed.

In an embodiment of the present invention, in view of this, a plurality of multi-exposure images may be applied to a deep learning model to be decomposed into common components and individual components at a feature level.

3 shows a detailed structure of a decomposition block of a deep learning model. Referring to FIG. 3 , the decomposition block of the deep learning model may decompose a plurality of multi-exposure images into common components and individual components at a feature level.

In the case of a plurality of multi-exposure images, since common components are almost identical to each other, they do not need to be elaborately fused, and can be fused through several convolutional layers to enhance structural information (eg, edges). Individual components can significantly contribute to the quality of the fused image, such as detail restoration and halo artifacts during image fusion.

Therefore, in order to correct the exact decomposition between a common component and an individual component, a deep learning model can be trained in consideration of constraints such as Equation 1.

Here, the constraint is the equalization loss, and all input images (ie, multi-exposure images) must have the same common components. Equalization loss is expressed as Equation 1.

은 입력 이미지(

)의 공통 성분을 나타내고, n은 다중 노출 이미지의 개수를 나타낸다. 손실 함수(

)는 다중 노출 이미지들 사이의 차이를 최소로 하며, 유사한 공통 성분을 가지는 것을 의미한다. 즉, 딥러닝 모델은 등화 손실이 최소가 되도록 학습될 수 있다. here,

is the input image (

), and n represents the number of multiple exposure images. loss function (

) means minimizing the difference between multiple exposure images and having similar common components. That is, the deep learning model can be trained to minimize equalization loss.

)과 개별 성분(

)을 시각화하면 도 4와 같이 나타낼 수 있다. 이러한 공통 성분과 개별 성분을 시각적으로 확인하는 것은 매우 어렵다. 따라서, 분해된 공통 성분과 개별 성분을 이용하여 RGB 채널로 매핑하여 시각화할 수 있다. In addition, common components (

) and the individual components (

) can be visualized as shown in FIG. It is very difficult to visually identify these common and individual components. Therefore, it can be visualized by mapping to RGB channels using the decomposed common components and individual components.

When combining the decomposed common components and individual components, they must be combined into a single image identical to the input image (ie, multiple exposure image). At this time, reconstruction of the decomposed common feature component and individual feature component may be measured as a mean square error (MSE).

Each common component and individual component respectively pass through a mapping block, and the mapping block may include a plurality of convolutional layers. A weight of a convolution layer included in each mapping block may be shared in each feature.

Figure 4 schematically shows feature decomposition and visualization and the resulting loss function. This will be described in more detail with reference to FIG. 4 .

For example, as shown in FIG. 4 , a first exposure image I ₁ and a second exposure image I ₂ will be assumed and described.

The first exposure image I ₁ may be decomposed into a first common component C ₁ , vis and a first individual component R ₁ , vis at a feature level through a deep learning model. In addition, the second exposure image I ₂ may be decomposed into a second common component C ₂ , vis and a second individual component R ₂ , vis at a feature level through a deep learning model.

In this case, since the first exposure image (I ₁ ) and the second exposure image (I ₂ ) are images obtained by changing only the exposure conditions for the same scene, the first common component (C) decomposed at the feature level through the deep learning model Structural information (eg, edge information) included in ₁ , vis and the second common component C ₂ , vis may be the same.

Therefore, the deep learning model is a first exposure image (I ₁ ) and a second exposure image (I ₂ ) so that the equalization loss of the first common component (C ₁ , vis) and the second common component (C ₂ , vis) is minimized. ) can be learned to decompose.

)은 제1 노출 영상과 동일하도록 매핑 블록의 가중치가 조정될 수 있다. In addition, an image reconstructed of the first common component (C ₁ , vis) and the first individual component (R ₁ , vis) (

) may have the weight of the mapping block adjusted to be the same as that of the first exposure image.

That is, the deep learning model can adjust weights to minimize visualization loss. The visualization loss can be expressed as Equation 2.

이다. here,

am.

는 입력 영상을 나타내고,

는 재구성된 영상을 나타내고,

는 공통 성분을 나타내며,

는 개별 성분을 나타낸다. here,

represents the input image,

represents the reconstructed image,

represents a common component,

represents an individual component.

In other words, the deep learning model is trained to minimize equalization loss and visualization loss, and multi-exposure images can be decomposed into common components and individual components through the trained deep learning model.

In step 120, the multi-exposure image fusion apparatus 100 based on deep learning decomposition fuses common components and individual components.

For example, the deep learning decomposition-based multi-exposure image fusing apparatus 100 may fuse individual components by applying a spatial attention weight map when merging. The contribution of each input image is different depending on the usefulness of the information for HDR reconstruction. Accordingly, the deep learning model may generate a spatial attention weight map in units of pixels through a learning process. Spatial attention weight maps for each individual component can be learned separately for high-quality fusion.

5 shows the process of fusing the individual components. As shown in FIG. 5 , each component passes through a convolution layer and a spatial attention weight map may be generated.

The result of fusing the individual components can be expressed as Equation 3.

는 공간 어텐션 가중치 맵을 나타내고,

는 개별 특징 성분을 나타낸다. 이와 같이, 공간 어텐션 가중치 맵을 적용하여 개별 성분들을 융합함으로써 후광 아티팩트를 피할 수 있는 이점이 있다. here,

denotes a spatial attention weight map,

represents an individual feature component. In this way, there is an advantage in that halo artifacts can be avoided by fusing individual components by applying a spatial attention weight map.

Common components can be constrained to be identical to each other in the decomposition process and consequently decompose very similarly to each other. Thus, common components of all inputs (i.e. multiple exposure images) can be fused by connecting several convolution layerings.

In step 125, the deep learning decomposition-based multi-exposure image fusion apparatus 100 reconstructs an output image using the fused common component and the fused individual component.

Reconstruction loss can be calculated using the real image and the reconstructed image.

If this is expressed as an equation, it is equivalent to Equation 4.

Here, Mout represents a reconstructed image, and GT represents a real image. Also, SSIM (structural similarity index measure) is a visual quality difference evaluation function. Since SSIM is a well-known technology, a detailed description thereof will be omitted. Therefore, the deep learning model can be learned taking into account the training loss (L) as a result. Here, the training loss is an equalization loss, a visualization loss, and a reconstruction loss.

,

,

은 각각 가중치를 나타낸다. here,

,

,

represent weights, respectively.

In summary, according to an embodiment of the present invention, a multi-exposure image may be decomposed into common components and individual components at a feature level by a deep learning model, and then fused at a feature level. In addition, according to an embodiment of the present invention, the multi-today image fusion method applies a spatial attention weight map to individual components and fuses them, thereby enabling detailed restoration of bright and dark parts, reducing halo artifacts, and restoring natural colors. There are benefits to being able to

), 분해된 개별 성분(R₁, vis 내지 R₄, vis)을 융합하며(

), 융합된 공통 성분과 융합된 개별 성분을 이용하여 재구성된 출력 영상(Mout)을 생성하는 과정에 대한 설명이 예시되어 있다. In FIG. 6, each of the four multiple exposure images is decomposed into common components and individual components through a deep learning model, and the decomposed common components (C ₁ , vis to C ₄ , vis) are fused (

), fusing the decomposed individual components (R ₁ , vis to R ₄ , vis) (

), a description of a process of generating a reconstructed output image Mout using the fused common component and the fused individual component is exemplified.

7 is a block diagram schematically showing the internal configuration of a deep learning decomposition-based multi-exposure image fusion device according to an embodiment of the present invention, and FIGS. 8 to 10 are multiple exposures according to the prior art and an embodiment of the present invention. It is a drawing comparing image fusion results.

Referring to FIG. 7 , the multi-exposure image fusion apparatus 100 based on deep learning decomposition according to an embodiment of the present invention includes a decomposition unit 710, a visualization unit 720, a fusion unit 730, and a reconstruction unit 740. , a memory 750 and a processor 760.

The decomposition unit 710 is a means for decomposing the multi-exposure image into common components and individual components at a feature level by applying the deep learning model.

The visualization unit 720 is a means for visualizing a single image identical to an input image (multiple exposure image) by combining common components and individual components.

The deep learning model can be trained to minimize equalization loss and visualization loss, as already described above. Since this is the same as that described in FIG. 1, redundant description will be omitted.

The fusion part 730 is a means for fusing common components and individual components. The fusion part 730 may fuse a common component by connecting it to a plurality of convolution layers. Also, when fusing individual components, the fusing unit 730 may fuse individual components by applying a spatial attention weight map.

The reconstruction unit 740 is a means for reconstructing an output image using fused common components and individual components. Deep learning models can be trained to decompose common and individual components such that reconstruction loss is minimal. As described above, in learning to decompose a multi-exposure image into common components and individual components, the deep learning model can be trained to minimize loss considering equalization loss, visualization loss, and reconstruction loss.

The memory 750 is a means for storing instructions for performing a multiple exposure image fusion method based on deep learning decomposition according to an embodiment of the present invention.

The processor 760 includes internal components (eg, a decomposition unit 710, a visualization unit 720, a fusion unit ( 730), a reconfiguration unit 740, a memory 750, etc.).

8 shows a result of fusion of multiple exposure images according to the prior art and an embodiment of the present invention. As shown in (a) to (h) of FIG. 8 , it can be seen that conventional methods have problems in restoring details of dark areas, and the fusion method (i) according to an embodiment of the present invention It can be seen that there is an advantage in restoring the detail of the region.

9 shows a result of fusing multiple exposure images according to an embodiment of the present invention with respect to different input images. In FIG. 9, the red boxed area is a detailed area saturated with excessive light, and as shown in (a) to (h), it can be seen that the performance of the conventional methods is poor in terms of detail restoration. However, it can be seen that the fusion method (i) according to an embodiment of the present invention shows stable and natural color and the best detail restoration ability.

10 is a comparison between a fusion result and a weight map according to an embodiment of the present invention and the conventional one for another input image.

As shown in (a) of FIG. 10, in the case of the conventional method, an unnatural boundary phenomenon is observed. As shown in the weight map, the four input weight distributions have sharp fluctuations and the contrast is higher. can As a result, it can be seen that an unnatural boundary phenomenon appears due to a biased weight for a specific input.

However, it can be seen that the present invention has a softer weight distribution compared to the prior art. Due to this, it can be seen that halo artifacts are further reduced in the fusion result according to an embodiment of the present invention.

Devices and methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in computer readable media. Computer readable media may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on a computer readable medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in the art in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - Includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media and ROM, RAM, flash memory, etc. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

So far, the present invention has been looked at mainly by its embodiments. Those skilled in the art to which the present invention pertains will be able to understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from a descriptive point of view rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent scope will be construed as being included in the present invention.

100: Deep learning-based multi-exposure image fusion device
710: disassembly
720: visualization unit
730: fusion part
740: reconstruction unit
750: memory
760: processor

Claims (7)

(a) decomposing each multi-exposure image into a common component and a residual component at a feature level by applying a deep learning model;
(b) fusing each individual component and fusing each common component; and
(c) generating a reconstruction image by adding the fused individual component and the fused common component;
According to claim 1,
The deep learning model is learned in consideration of equalization loss so that each of the multiple exposure images obtained under different exposure conditions for the same scene has the same common component.
According to claim 2,
The deep learning model,
Deep learning decomposition-based multi-exposure image fusion method, characterized in that weights are learned in consideration of visualization loss so that each of the decomposed common components and each individual component are combined into a single image identical to each of the multiple-exposure images.
According to claim 1,
In step (b),
Deep learning decomposition-based multi-exposure image fusion method, characterized in that for fusing each individual component using a spatial attention weight map.
According to claim 3,
The deep learning model,
Deep learning decomposition-based multi-exposure image fusion method, characterized in that learning to minimize reconstruction loss considering the difference between the multi-exposure image and the output image reconstructed with the fused common component and the fused individual component.
A computer-readable recording medium recording program codes for performing the method according to any one of claims 1 to 5.
a decomposition unit that applies each multi-exposure image to a deep learning model and decomposes each into a common component and a residual component at a feature level;
a fusing unit fusing each of the individual components and fusing each of the common components; and
A reconstruction unit generating a reconstructed output image using the fused individual component and the fused common component;
The deep learning model is a deep learning decomposition-based multi-exposure image fusion device, characterized in that it is learned to minimize equalization loss between each common component decomposed of each multiple-exposure image.

Images