KR102403947B1 - Composite image generation method, apparatus performing the same, and computer program for executing the same - Google Patents

Abstract

The present invention relates to a method for generating a synthesized image, a synthesized image generating apparatus for performing the same, and a computer program stored in a computer readable medium for executing the same, and more particularly, to a single synthesis by synthesizing a plurality of different input images. To a synthetic image generating method for generating an image, to a synthetic image generating apparatus for performing the same, and to a computer program stored in a computer readable medium for executing the same.
The present invention discloses a synthesized image generating method for generating a synthesized image by synthesizing K different input images (X(k)) (K is a natural number greater than or equal to 2, 1≤k≤K).

Description

A method for generating a synthetic image, an apparatus for generating a synthetic image for performing the same, and a computer program stored in a computer readable medium for executing the same

The present invention relates to a method for generating a synthesized image, an apparatus for generating a synthesized image for performing the same, and a computer program stored in a computer readable medium for executing the same, and more particularly, to a single synthesis by synthesizing a plurality of different input images. To a synthetic image generating method for generating an image, to a synthetic image generating apparatus for performing the same, and to a computer program stored in a computer readable medium for executing the same.

In general, the human visual system does not have any difficulty in recognizing information even when both dark and bright areas exist in the scene it is looking at. On the other hand, when a digital device receives light reflected from an object as a sensor, information cannot be obtained depending on the degree to which the light is exposed.

In other words, before acquiring an image, the digital device determines the exposure according to the brightness of a specific area and acquires an image. Under-exposure phenomenon or Over-exposure occurs. And there is a problem that the information of the part where this phenomenon appears cannot be expressed in the image. This problem occurs because the dynamic range of digital devices is narrower than that of the human eye.

In order to overcome this difference, a high dynamic range (HDR) technology for obtaining an image having a wide dynamic range is being studied. The ultimate goal of HDR technology is to obtain HDR images that look similar to human-perceived scenes.

In addition, since the HDR image contains a lot of information, it is easy to recognize objects in the image with the human eye, and it can be used as a preprocessing algorithm to improve the performance of computer vision algorithms such as detection and recognition. However, the HDR image by synthesis mainly has several problems such as a halo effect, color shift, and saturation.

HDR technology can be divided into a high dynamic range imaging (HDRI) method and a multi-exposure image fusion method. HDRI obtains a camera response function using several LDR (Low dynamic range) images with different exposures and the exposure time of the images, and constructs a radiance map to obtain a single HDR image. way.

Since HDR images obtained using HDRI have 32 bits of floating-point representation per channel, they cannot be expressed in general digital devices, and it is common to convert HDR images to LDR images using tone mapping to express them.

The multiple exposure synthesis method also uses several LDR (Low dynamic range) images with different exposures. However, it is a method of obtaining a synthesized LDR image at once without a process of obtaining an HDR image and a process of performing tone mapping.

However, the conventional image synthesis method still has problems in which synthesis processing speed needs to be improved along with various problems such as a halo effect, color shift, and saturation.

SUMMARY OF THE INVENTION An object of the present invention is to recognize the above problems, and to generate a composite image by synthesizing a plurality of input images to greatly improve the processing speed, a synthesized image generating apparatus for performing the same, and the same To provide a computer program stored in a computer-readable medium for execution.

), 및 평균밝기(

)로 이루어지는 K개의 패치벡터(

, 1≤k≤K)들로 정의하는 패치벡터정의단계(S204)와; 상기 K개의 패치벡터(

)들을 이용해 평균밝기(

)가 제거된 K개의 평균제거패치벡터(

, 1≤k≤K)들을 산출하는 평균제거패치벡터산출단계(S206)와; 상기 K개의 평균제거패치벡터(

)들 각각의 신호강도(c(k))를 기초로 정규화된 가중치함수(ω(k))를 산출하는 가중치함수산출단계(S208)와; 상기 가중치함수(ω(k))와 상기 K개의 평균제거패치벡터(

)들에 대한 가중합벡터를 산출하는 가중합벡터산출단계(S210)와; 상기 가중합벡터에 상기 합성영상의 합성평균밝기(l)를 더해 상기 합성영상의 합성패치벡터(

)를 산출하는 합성패치벡터산출단계(S212)를 포함하는 것을 특징으로 하는 합성영상생성방법을 개시한다.The present invention was created to achieve the object of the present invention as described above, and a synthesized image is generated by synthesizing K different input images (X(k)) (K is a natural number of 2 or more, 1≤k≤K) As a synthetic image generating method, a patch set {A including K color image patches (CP(k), 1≤k≤K) extracted from the same position of each of the K input images X(k) a patch set definition step (S202) of defining _k }; The K color image patches (CP(k)) are analyzed for signal intensity (c(k)), signal structure (

), and average brightness (

) of K patch vectors (

, 1≤k≤K) defining a patch vector defining step (S204); The K patch vectors (

) using the average brightness (

) of the K mean removed patch vectors (

, 1≤k≤K) and calculating an average removed patch vector (S206); The K average removed patch vectors (

) a weighting function calculation step (S208) of calculating a normalized weighting function (ω(k)) based on each of the signal strengths (c(k)); The weight function (ω(k)) and the K average removed patch vectors (

), a weighted sum vector calculating step (S210) of calculating a weighted sum vector for them; Synthetic patch vector (

), a synthetic patch vector calculating step (S212) for calculating a synthetic image generating method is disclosed.

)들을 신호강도에 따라 정렬하여 가장 큰 신호강도를 가지는 최대신호강도벡터(

)를 산출하는 최대신호강도벡터산출단계와; 상기 가중합벡터를 상기 최대신호강도벡터(

)로 근사하는 가중합벡터근사단계를 포함할 수 있다.The K average removed patch vectors (

) are sorted according to the signal strength, and the maximum signal strength vector (

) a maximum signal intensity vector calculating step; The weighted sum vector is the maximum signal intensity vector (

) may include a weighted sum vector approximation step.

)를 기초로 상기 합성영상을 생성하는 합성영상생성단계를 추가로 포함할 수 있다.The synthetic patch vector (

) may further include a synthetic image generating step of generating the synthetic image based on the.

)을 산출하는 고주파영상가중합단계와; 상기 저주파영상(Mk(i))에 저주파가중치함수(Bk(i))을 가중합하여 저주파영상가중합(

를 산출하는 저주파영상가중합단계와; 상기 고주파영상가중합(

)과 저주파영상가중합(

을 더해 상기 합성영상을 생성하는 합성영상생성단계를 포함하는 것을 특징으로 하는 합성영상생성방법을 개시한다.In another aspect, there is provided a synthetic image generating method for generating a synthesized image by synthesizing K different input images X(k) (K is a natural number greater than or equal to 2, 1≤k≤K), wherein the K input images (X (k)) are divided into K high-frequency images (Nk(i), 1≤k≤K) and K low-frequency images (Mk(i), 1≤k≤K) for the i-th pixel. separation step; A weighting function calculation step of calculating a normalized high-frequency weight function Ak(i) based on the signal intensity ck(i) of the K high-frequency images Nk(i), and the high-frequency image Nk(i) )) by weighting the high-frequency weighting function (Ak(i))) to the high-frequency image weighting sum (

) a high-frequency image weighting step of calculating; By weighting the low-frequency image (Mk(i)) with the low-frequency weighting function (Bk(i)), the low-frequency image weighting sum (

a low-frequency image weighted polymerization step of calculating The high-frequency image weighting (

) and low-frequency image weighting (

Disclosed is a synthetic image generating method, characterized in that it comprises a synthetic image generating step of generating the synthesized image by adding

)을 상기 최대신호강도(Cmax(i))로 근사하는 고주파영상가중합근사단계를 포함할 수 있다.a maximum signal intensity calculation step of arranging the K high-frequency images (Nk(i)) according to signal intensity and calculating a maximum signal intensity Cmax(i) having the greatest signal intensity; The high-frequency image weighting (

) may include a high-frequency image weighted sum approximation step of approximating the maximum signal intensity (Cmax(i)).

The synthetic image generating method includes a low-frequency weighting function calculating step of calculating the low-frequency weighting function (Bk(i)) based on the global brightness value μk and the local brightness value lk of the K input images X(k). may further include.

In another aspect, the present invention executes a synthetic image generating method for generating a synthetic image by synthesizing K different input images (X(k)) (K is a natural number greater than or equal to 2, 1≤k≤K) using a computer. Discloses a computer program stored in a computer-readable medium to do so.

In addition, the present invention discloses a synthesized image generating apparatus for performing a synthesized image generating method for generating a synthesized image by synthesizing K different input images (X(k)) (K is a natural number greater than or equal to 2, 1≤k≤K) do.

A method for generating a synthetic image according to the present invention, an apparatus for generating a synthetic image for performing the same, and a computer program stored in a computer readable medium for executing the method greatly increase the processing speed in generating a synthetic image by synthesizing a plurality of input images There are benefits that can be improved.

1 is a block diagram showing an apparatus for generating a composite image according to an embodiment of the present invention.
FIG. 2 is a flowchart illustrating a method for generating a synthesized image performed in the apparatus for generating a synthesized image of FIG. 1 .
3 is a flowchart illustrating a modified example of the method for generating a composite image of FIG. 2 .
4A to 4B are diagrams illustrating input images for generating a composite image.
5A to 5C are views showing synthesized images generated according to the method for generating a synthesized image according to the present invention.

Hereinafter, an apparatus for generating a composite image according to the present invention will be described with reference to the accompanying drawings.

As shown in FIG. 1, the synthesized image generating apparatus 100 according to the present invention synthesizes and synthesizes K different input images X(k) (K is a natural number greater than or equal to 2, 1≤k≤K). Various configurations are possible as a configuration for performing a synthetic image generating method for generating an image.

Here, as shown in FIGS. 4A to 4C , the K input images X(k) are images with different exposures, and the number K of input images may be variously set.

The input image X(k) is an image acquired by an image sensor such as a CCTV or a camera, and is not limited to a specific image, and may include various images.

In one embodiment, the synthesized image generating apparatus 100 includes, as shown in FIG. 1 , an image input unit 110 to which K different input images X(k) are input, and the image input unit 110 ), the image analysis unit 120 for analyzing the K input images (X(k)), and the K input images ( It may include a weighted sum operator 130 that calculates a weighted sum of X(k)), and a synthesized image generator 140 that generates a composite image using the calculated weighted sum.

The image input unit 110 can be configured in various configurations such that K different input images X(k) are inputted.

The image input unit 110 may be connected to an image sensor for photographing/photographing the input image X(k) through a wired/wireless network.

The original image input through the image input unit 110 may be stored in a memory (not shown) or transmitted to the outside through a wired/wireless network.

In this case, the synthesized image generating apparatus 100 includes an image acquisition means (not shown) such as a camera for acquiring the input image X(k) to be input to the image input unit 110 by itself or by an external device. The input image X(k) may be acquired by being connected to the image acquisition means through a wired/wireless network.

The image analysis unit 120 may be configured in various configurations by analyzing the K input images X(k) input through the image input unit 110 .

For example, the image analysis unit 120 may convert the K input images X(k) into a patch unit to calculate a patch unit or pixel unit weighted sum of the K input images X(k). It can be analyzed or it can be analyzed on a pixel-by-pixel basis.

The analysis process performed by the image analysis unit 120 will be described in detail below by explaining a method for generating a synthesized image.

The weighted sum calculating unit 130 has various configurations, such as calculating the weighted sum of the K input images X(k) based on the analysis result of the K input images X(k). It is possible.

For example, the weighted sum operation unit 130 may calculate a patch unit or pixel unit weighted sum of the K input images X(k).

The analysis process performed by the image analysis unit 120 and the weighted sum operation unit 130 will be described in detail below with reference to the synthesis image generation method.

The synthesized image generating unit 140 may be configured to generate a synthesized image by using the weighted sum calculated by the weighted sum operation unit 130 , and various configurations are possible.

Through the synthesized image generator 140 , K input images X(k) may be generated as one synthesized image.

In this case, the synthesized image generating apparatus 100 according to the present invention may further include an output unit (not shown) for outputting the synthesized image synthesized through the synthesized image generating unit 140 .

The output unit (not shown) is configured to output a synthesized composite image, and various configurations are possible. For example, the output unit (not shown) may include a display unit that outputs the synthesized image using a visual method.

The composite image as a final result output through the output unit (not shown) may be transmitted to the outside through a wired/wireless network.

Meanwhile, the synthesized image generating apparatus 100 may further include a wireless communication unit for wireless communication and a user input unit for controlling the operation of the synthesized image generating apparatus 100 through a user in addition to the above-described configuration.

Hereinafter, a synthetic image generating method performed by the synthetic image generating apparatus 100 will be described in detail with reference to FIGS. 2 to 5C .

The synthetic image generating method is a synthetic image generating method that generates a synthesized image by synthesizing K different input images X(k) (K is a natural number greater than or equal to 2, 1≤k≤K). Disclosed is a synthesis method that can significantly improve processing speed while generating an image.

In the first embodiment, as shown in FIG. 2 , the synthesized image generating method may generate a synthesized image based on a patch for K input images X(k).

), 및 평균밝기(

)로 이루어지는 K개의 패치벡터(

, 1≤k≤K)들로 정의하는 패치벡터정의단계(S204)와; 상기 K개의 패치벡터(

)들을 이용해 평균밝기(

)가 제거된 K개의 평균제거패치벡터(

, 1≤k≤K)들을 산출하는 평균제거패치벡터산출단계(S206)와; 상기 K개의 평균제거패치벡터(

)들 각각의 신호강도(c(k))를 기초로 정규화된 가중치함수(ω(k))를 산출하는 가중치함수산출단계(S208)와; 상기 가중치함수(ω(k))와 상기 K개의 평균제거패치벡터(

)들에 대한 가중합벡터를 산출하는 가중합벡터산출단계(S210)와; 상기 가중합벡터에 상기 합성영상의 합성평균밝기(

)를 더해 상기 합성영상의 합성패치벡터(

)를 산출하는 합성패치벡터산출단계(S212)를 포함할 수 있다.Specifically, in the method for generating a composite image, a patch including K color image patches (CP(k), 1≤k≤K) extracted from the same position of each of the K input images X(k). a patch set definition step of defining a set {CP _k }; The K color image patches (CP(k)) are analyzed for signal intensity (c(k)), signal structure (

), and average brightness (

) of K patch vectors (

, 1≤k≤K) defining a patch vector defining step (S204); The K patch vectors (

) using the average brightness (

) of the K mean removed patch vectors (

, 1≤k≤K) and calculating an average removed patch vector (S206); The K average removed patch vectors (

) a weighting function calculation step (S208) of calculating a normalized weighting function (ω(k)) based on each of the signal strengths (c(k)); The weight function (ω(k)) and the K average removed patch vectors (

), a weighted sum vector calculating step (S210) of calculating a weighted sum vector for them; In the weighted sum vector, the synthesized average brightness of the synthesized image (

) to the composite patch vector of the composite image (

) may include a synthetic patch vector calculating step (S212) for calculating.

The K input images X(k) may be input to the synthesized image generating apparatus 100 through the image input unit 110 .

The synthetic image generating method may further include an input image input step S202 of receiving the K input images X(k).

The patch set definition step includes a patch set {CP _k including K color image patches (CP(k), 1≤k≤K) extracted from the same location of each of the K input images X(k). }, which may be performed by the image analysis unit 120 .

Each of the color image patches (CP(k), 1≤k≤K) is a patch of a color channel extracted from the same position of the input image X(k), for example, an RGB color patch (single channel is also possible) can be

The patch set {CP _k } may be defined as a set including K color image patches (CP(k), 1≤k≤K) for K input images X(k) as elements. .

=

수식 (1)

=

Formula (1)

), 및 평균밝기(

)로 이루어지는 K개의 패치벡터(

, 1≤k≤K)들로 정의하는 단계로서, 상기 영상분석부(120)에 의해 수행될 수 있다.In the patch vector definition step S204, the K color image patches CP(k) are identified by the signal intensity c(k), the signal structure (

), and average brightness (

) of K patch vectors (

, 1≤k≤K), which may be performed by the image analysis unit 120 .

)로 표현되는 신호일 수 있다.That is, the K color image patches CP(k) are the patch vectors (

) may be a signal represented by

)는 입력영상(X(k))의 컬러채널의 개수가 n이고 이미지패치의 사이즈가 m일 때, n*m²의 차원을 가지는 열벡터(Column vector)로 정의될 수 있다.The patch vector (

) may be defined as a column vector having a dimension of n*m ² when the number of color channels of the input image X(k) is n and the size of the image patch is m.

)는 아래 수식들에 의해 신호강도(c(k)), 신호구조(

), 및 평균밝기(

)로 정의될 수 있다.At this time, the patch vector (

) is the signal strength (c(k)) and the signal structure (

), and average brightness (

) can be defined as

)의 평균밝기

를 구하고 상기 패치벡터(

)에서 평균밝기(

를 제거하여 평균-제거된(Mean-removed) 평균제거패치벡터(

)를 구한다. First, the patch vector (

) average brightness

to obtain the patch vector (

) in average brightness (

Mean-removed mean-removed patch vector (

) to find

)는,

로 표기(notation)될 수 있다.Here, the average removed patch vector (

)Is,

It can be marked as (notation).

를 그 크기(Magnitude,

)와 방향(Direction,

)으로 구분할 수 있다.Next, the average removal patch vector (

is its magnitude (Magnitude,

) and direction (Direction,

) can be distinguished.

)는 평균제거패치벡터(

에 평균밝기

를 더하는 방식으로 정의될 수 있다.The patch vector (

) is the average removed patch vector (

average brightness on

It can be defined by adding

수식(2)

Formula (2)

수식(3-1)

Formula (3-1)

는 패치벡터(

)의 평균밝기(mean intensity), 상기 C(k)는 패치벡터(

)의 신호강도(signal strength)로서 평균제거패치벡터(

)의 크기이고,

는 패치벡터(

)의 신호구조(signal structure)로서 평균제거패치벡터(

)의 단위벡터일 수 있다.here,

is the patch vector (

) of the mean intensity, the C(k) is the patch vector (

) as the signal strength of the average removed patch vector (

) is the size of

is the patch vector (

) as the signal structure of the mean removed patch vector (

) may be a unit vector of

)가 수식 (3-1)과 같이 신호강도(c(k)), 신호구조(

), 및 평균밝기(

)로 이루어지는 벡터로 정의될 수 있다.Through this, the patch vector (

) is the signal strength (c(k)) and the signal structure (

), and average brightness (

) can be defined as a vector consisting of

)들이 합성된 합성영상의 합성패치벡터(

)는 아래 수식 (3-2)와 같이 정의될 수 있다.As a result, the K patch vectors (

) of the synthesized image synthesized patch vector (

) can be defined as in Equation (3-2) below.

수식 (3-2)

Formula (3-2)

은 합성영상의 합성신호강도,

은 합성영상의 합성신호구조,

은 합성영상의 합성평균밝기로 정의될 수 있다.Similarly to Equation (3-1), in Equation (3-2) above,

is the composite signal strength of the composite image,

is the composite signal structure of the composite image,

may be defined as the composite average brightness of the composite image.

)들을 이용해 평균밝기(

)가 제거된 K개의 평균제거패치벡터(

, 1≤k≤K)들을 산출하는 단계로서, 상기 영상분석부(120)에 의해 수행될 수 있다.In the step of calculating the average removed patch vector (S206), the K patch vectors (

) using the average brightness (

) of the K mean removed patch vectors (

, 1≤k≤K), which may be performed by the image analysis unit 120 .

수식 (4)

Formula (4)

The average removal patch vector calculation step (S206) may be performed together with the patch vector defining step (S204) or before the patch vector defining step (S204).

)의 평균밝기(

)를 연산함으로써 상기 수식 (1) 내지 (4)에 의해 수행될 수 있다.The patch vector definition step (S204) and the average removal patch vector calculation step (S206) include the patch vector (

) of the average brightness (

) can be performed by Equations (1) to (4) above.

)들 각각의 신호강도(c(k)), 즉 상기 K개의 평균제거패치벡터(

)들의 벡터크기를 기초로 정규화된 가중치함수(ω(k))를 산출하는 단계로서, 상기 가중합연산부(130)에 의해 수행될 수 있다.The weight function calculation step (S208) includes K average removed patch vectors (

) of each signal intensity (c(k)), that is, the K average removed patch vectors (

) is a step of calculating a normalized weight function ω(k) based on the vector magnitudes, and may be performed by the weighted sum operation unit 130 .

로부터 요구되는 크기(신호강도(c(k))와 방향(신호구조(

)을 산출할 때, 중복 계산(Redundancy)을 줄이기 위해 평균제거패치벡터(

의 크기와 방향을 함께 고려할 수 있다.The present invention provides an average removed patch vector (

Required magnitude (signal strength (c(k))) and direction (signal structure (

), to reduce redundancy when calculating the average removed patch vector (

size and direction can be considered together.

)들의 벡터크기

, 즉, c(k)는 다음과 같이 크기에 따라 정렬될 수 있다.First, the K average removed patch vectors (

) of the vector size

, that is, c(k) can be sorted according to size as follows.

수식 (5)

formula (5)

수식 (6)

formula (6)

는 상술한 바와 같이 평균제거패치벡터(

)의 크기(신호강도)로, 평균제거패치벡터(

)의 square-root 값(L²-norm)일 수 있고, K는 상기 K개의 평균제거패치백터(

)들을 신호강도에 따라 정렬하여 가장 큰 신호강도를 가지는 평균제거패치벡터(

)의 인덱스넘버일 수 있다.here,

is the average removed patch vector (

) as the magnitude (signal strength), the average removed patch vector (

) may be a square-root value (L ² -norm), and K is the K average removed patch vector (

) are sorted according to the signal strength, and the average removed patch vector (

) can be an index number.

)는 최대신호강도벡터(

)로 정의될 수 있다.At this time, the average removed patch vector (

) is the maximum signal strength vector (

) can be defined as

)들 각각의 신호강도(c(k)), 즉 상기 K개의 평균제거패치벡터(

)들의 벡터크기를 기초로 정규화된 함수로서, 아래 수식들을 통해 정의될 수 있다.The weight function (ω(k)) is the K average removed patch vectors (

) of each signal intensity (c(k)), that is, the K average removed patch vectors (

) as a normalized function based on the vector size, it can be defined through the following equations.

수식 (7)

formula (7)

Considering the relationship between the signal strength (c(k)) and the weight,

으로 정의하면, 수식 (7)의 가중치함수(ω(k))는 아래 수식 (8)과 같이 산출될 수 있다.

, the weight function ω(k) of Equation (7) can be calculated as Equation (8) below.

Here, p is an exponential weight parameter.

수식 (8)

formula (8)

)들에 대한 가중합벡터를 산출하는 단계로서, 상기 가중합연산부(130)에 의해 수행될 수 있다.In the weighted sum vector calculation step S210, the weighting function ω(k) and the K average removed patch vectors (

), which is a step of calculating a weighted sum vector for the values, which may be performed by the weighted sum operation unit 130 .

First, if the exponential weight parameter p (p≥0) is applied to Equation (5), it can be arranged as in Equation (9) below.

수식 (9)

formula (9)

)들의 신호강도(c(k) 또는

에 대한 가중합을 산출하면 아래 수식 (10)과 같이 표현될 수 있다.Based on Equation (9), the weight function (ω(k)) and the K average removed patch vectors (

) of the signal strengths (c(k) or

Calculating the weighted sum for can be expressed as Equation (10) below.

수식 (10)

formula (10)

가 같은 값을 가진다면, 상기 수식 (10)의 가중합의 결과는

과 동일한 값일 것이다.for all k

has the same value, the result of the weighted sum of Equation (10) is

will be the same value as

)의 신호강도

(또는

와 유사할 것이다.Alternatively, if the exponential weight p is a sufficiently large value, the result of the weighted sum of Equation (10) is the maximum signal intensity vector (

) of the signal strength

(or

will be similar to

수식 (11)

Formula (11)

Since the weight function ω(k) is a value normalized from 0 to 1, if the p (exponential weight value) is sufficiently large, it can be arranged as follows.

수식 (11)

Formula (11)

)의 가중치함수 값

가 가장 우세(dominant)하며 나머지 가중치

들의 값은 무시할 수 있다.That is, when k is K, the maximum signal intensity vector (

) of the weight function

is the dominant and the remaining weights

Their values can be ignored.

From this, Equation (3-2) can be approximated as follows.

수식 (12)

Formula (12)

)에 포함된 벡터컴포넌트(component)들은 음수를 포함할 수 있기 때문에, 상기 수식 (12)를 보다 간략화 시켜야 할 필요가 있다.In Equation (12), the average removed patch vector (

Since vector components included in ) may include negative numbers, it is necessary to simplify Equation (12).

)들을 모두 양수값으로 바이어스(쉬프트) 시켜 바이어스된 평균제거패치벡터(

, Biased mean-removed patch vector)를 산출하는 평균제거패치벡터 바이어스단계를 추가로 포함할 수 있다.To this end, the weighted sum vector calculation step S210 includes K average removed patch vectors (

) are all biased (shifted) to positive values and the biased average removal patch vector (

, a biased mean-removed patch vector) may further include a biasing step of calculating an average removed patch vector.

)는 아래 수식 (13)과 같이 도출될 수 있다.The biased average removed patch vector (

) can be derived as in Equation (13) below.

수식 (13)

Formula (13)

)는 항상 0보다 크거나 같은 벡터로 정의될 수 있고, 평균제거패치벡터(

)를 바이어스된 평균제거패치벡터(

) 대체하여 상기 수식 (12)를 간략화하여 수식 (14)를 도출할 수 있다.Through this, the biased average removed patch vector (

) can always be defined as a vector greater than or equal to 0, and the average removed patch vector (

) is the biased mean removed patch vector (

), by simplifying Equation (12) above, Equation (14) can be derived.

수식 (14)

Formula (14)

는 합성패치벡터

에 요구되는 신호강도(합성신호강도)로 아래 수식 (15)를 기초로 수식 (14)를 도출하였다.Meanwhile,

is a composite patch vector

Equation (14) was derived based on Equation (15) below as the required signal strength (synthetic signal strength).

수식 (15)

Formula (15)

는 합성패치벡터

에 요구되는 신호구조(합성신호구조)로, 가중합에 의해 재정의될 수 있다.In addition,

is a composite patch vector

A signal structure (composite signal structure) required for

를 산출하는 식은 아래 수식 (16)을 통해 도출될 수 있다.Here, the required signal structure

The formula for calculating ? can be derived through Equation (16) below.

수식 (16)

Formula (16)

는 신호구조를 도출하기 위한 가중치함수로, 신호강도에 의해 결정될 수 있다. 신호강도가 클수록 신호구조도 잘 표현된 신호일 확률이 높기 때문이다.Here, the

is a weighting function for deriving the signal structure, and may be determined by the signal strength. This is because the higher the signal strength, the higher the probability that the signal structure is a well expressed signal.

는 아래 수식 (17)과 같이 정의하였다.At this time, the

is defined as Equation (17) below.

수식 (17)

Formula (17)

)들에 대한 가중합벡터를 산출할 수 있다.As a result, in the weighted sum vector calculation step S210, as shown in Equation (14), the weighting function ω(k) and the K average removed patch vectors (

), a weighted sum vector can be calculated.

)를 더해 상기 합성영상의 합성패치벡터(

)를 산출하는 합성패치벡터산출단계(S212)를 포함할 수 있다.At this time, as derived from Equation (14), the synthesized image generation method includes the combined average brightness of the synthesized image in the weighted sum vector.

) to the composite patch vector of the composite image (

) may include a synthetic patch vector calculating step (S212) for calculating.

The composite patch vector calculation step S212 may be performed by the composite image generator 140 .

)는, 가중합벡터산출단계(S210)에서 도출된 가중치함수(ω(k))와 상기 K개의 평균제거패치벡터(

)들에 대한 가중합벡터

에 합성평균밝기(

)를 더함으로써 얻어질 수 있다.As a result, the synthetic patch vector (

) is the weight function (ω(k)) derived in the weighted sum vector calculation step (S210) and the K average removed patch vectors (

) weighted sum vector

Synthetic average brightness (

) can be obtained by adding

는 합성패치벡터

의 평균밝기(합성평균밝기)로 아래 적정한 노출의 정도와 관련있다.here,

is a composite patch vector

The average brightness of (synthetic average brightness) is related to the degree of appropriate exposure below.

는 아래 수식 (18)를 기초로 도출될 수 있다.The composite average brightness

can be derived based on Equation (18) below.

수식 (18)

formula (18)

)는 전역적 밝기값

와, 지역적 밝기값

에 의한 가중치 함수이고,

은 합성패치벡터

, 즉 합성영상의 요구되는 평균밝기이다.here,

) is the global brightness value

Wow, local brightness values

is a weight function by

silver composite patch vector

, that is, the required average brightness of the composite image.

는 아래 수식 (19)와 같이, 전역적 밝기와 지역적 밝기가 0.5 (0~1 로 밝기 값이 정규화되었을 때 수치임)에 가까울 수록 가중치가 커지는 형태로 정의된다.

As shown in Equation (19) below, as the global brightness and local brightness are closer to 0.5 (the value is when the brightness value is normalized to 0 to 1), the weight increases.

수식 (19)

formula (19)

와

은 가우시안형(shape)을 결정하는 파라미터로서, 거리에 따른 가중치를 부여하는 정도를 결정하는 파라미터이다.here,

Wow

is a parameter that determines a Gaussian shape, and is a parameter that determines the degree of weighting according to distance.

)를 기초로 합성영상이 생성하는 합성영상생성단계(S214)를 포함할 수 있다.The synthetic image generation method includes the synthetic patch vector (

) may include a synthetic image generating step (S214) of generating a synthetic image based on the synthetic image.

5A shows the synthesized image obtained through the first embodiment when the input image X(k) of FIGS. 4A to 4C is used as an input image. A composite image can be obtained.

)를 산출할 수 있다.As a second embodiment, the synthesized patch vector (

) can be calculated.

)들을 신호강도에 따라 정렬하여 가장 큰 신호강도를 가지는 최대신호강도벡터(

)를 산출하는 최대신호강도벡터산출단계와, 상기 가중합벡터를 상기 최대신호강도벡터(

)로 근사하는 가중합벡터근사단계를 추가로 포함할 수 있다.In the second embodiment, the method for generating a composite image includes the K average removed patch vectors (

) are sorted according to the signal strength, and the maximum signal strength vector (

a maximum signal intensity vector calculating step of calculating

) may further include a weighted sum vector approximation step.

)들을 신호강도에 따라 정렬하여 가장 큰 신호강도를 가지는 최대신호강도벡터(

)를 산출하는 단계로서, 상기 가중치함수산출단계(S208)에서 수행될 수 있다.In the step of calculating the maximum signal intensity vector, K average removed patch vectors (

) are sorted according to the signal strength, and the maximum signal strength vector (

), which may be performed in the weight function calculation step (S208).

)로 근사하는 단계로서, 상기 가중합벡터산출단계(S210) 이후 수행될 수 있다.In the weighted sum vector approximation step, the weighted sum vector is converted to the maximum signal intensity vector (

), which may be performed after the weighted sum vector calculation step (S210).

In the weighted sum vector approximation step, the weighted sum vector calculated in the weighted sum vector calculation step S210 may be further approximated.

Specifically, in the second embodiment, Equation (14) can be simplified as Equation (20) below through weighted sum vector approximation.

수식 (20)

formula (20)

Equation (20) has the advantage of being simpler and faster processing because it does not need to calculate the weight function (ω(k)) compared to Equation (14).

)는, 최대신호강도벡터(

)에 합성평균밝기(

)를 더함으로서 얻어질 수 있다.At this time, the composite patch vector (S212) calculated in the composite patch vector calculation step (S212)

) is the maximum signal strength vector (

) to the composite average brightness (

) can be obtained by adding

FIG. 5B shows the synthesized image obtained through the above-described second embodiment when the input image X(k) of FIGS. 4A to 4C is used as an input image. A composite image can be obtained.

를 잘 구성할 필요가 있다. 일반적으로 가중치함수는 영상의 품질을 측정(Measure) 하는 방법을 이용하여 중요도(Saliency) 맵을 구하고, 중요도에 따라 가중치함수가 구성된다. On the other hand, in the case of the conventional pixel-based synthetic image generating method, a weight function for each input image (X(k)) to generate a synthetic image is

needs to be well configured. In general, the weight function obtains a saliency map by using a method of measuring image quality, and a weight function is constructed according to the importance.

If the weight function constructed in this way is used as it is, the image may become unnatural or noisy due to a sharp transition at the boundary.

In order to obtain a natural result, the input images X(k) must be properly mixed, and in order to be mixed, neighboring weight values must be similar to each other.

For example, the weight function should be smoothed, and various methods have been tried for this purpose.

In other words, pixel-based synthetic image generation methods require a process of modifying the weight function to reduce a sudden boundary change or halo phenomenon. has a

On the other hand, the patch-based synthetic image generation method, unlike pixel-based synthetic image generation methods, requires pre/post-processing such as Gaussian filtering, recursive filtering, and guided filtering. It is not required and gives better synthesis results.

However, since the input image is divided into overlapping patches and processed for all patches, there is a problem in that the calculation cost is relatively high.

In order to solve this problem, the present invention proposes a pixel-based composite image generating method in consideration of factors used in the patch-based method through the third embodiment.

)을 산출하는 고주파영상가중합단계(S308)와, 상기 저주파영상(Mk(i))에 저주파가중치함수(Bk(i))을 가중합하여 저주파영상가중합(

를 산출하는 저주파영상가중합단계(S310)와; 상기 고주파영상가중합(

)과 저주파영상가중합(

을 더해 상기 합성영상을 생성하는 합성영상생성단계(S312)를 포함할 수 있다.Specifically, the method for generating a synthesized image according to the third embodiment, similar to the first and second embodiments, generates K different input images X(k) (K is a natural number greater than or equal to 2, 1≤ A method for generating a synthetic image by synthesizing k ≤ K) to generate a synthesized image, wherein the K input images (X(k)) are combined with K high-frequency images (Nk(i), 1≤k≤K for the i-th pixel). ) and K low-frequency images (Mk(i), 1≤k≤K), an image separation step (S304), and signal strengths (ck(i)) of the K high-frequency images (Nk(i)) ), a weighting function calculation step (S306) of calculating a normalized high-frequency weight function Ak(i) based on the high-frequency weight function Ak(i)), and weighting the high-frequency weight function Ak(i)) to the high-frequency image Nk(i)) High-frequency image weighted summing (

) of the high-frequency image weighting step (S308), and the low-frequency image weighted sum (

a low-frequency image weighted polymerization step (S310) for calculating ; The high-frequency image weighting (

) and low-frequency image weighting (

It may include a synthetic image generating step (S312) of generating the synthesized image by adding .

The K input images X(k) may be input to the synthesized image generating apparatus 100 through the image input unit 110 .

The synthetic image generating method may further include an input image input step S302 of receiving the K input images X(k).

In the image separation step (S304), the K input images X(k) are divided into K high-frequency images (Nk(i), 1≤k≤K) for the i-th pixel and K low-frequency images ( As a step of classifying into Mk(i), 1≤k≤K), it may be performed by the image analysis unit 120 .

Assuming that K input images X(k) are input, the synthesized image Y(i) may be defined as the brightness value at the ith pixel, and may be expressed as Equation (21) below. .

수식 (21)

formula (21)

는 k번째 영상의 제i번째 픽셀에서의 가중치값,

는 k번째 영상의 제i번째 픽셀에서의 밝기값일 수 있다.Here, K is the number of synthesized input images (X(k)),

is the weight value at the i-th pixel of the k-th image,

may be a brightness value in the i-th pixel of the k-th image.

In the image separation step S304, each input image X(k) may be divided into a high frequency image Nk(i) and a low frequency image Mk(i).

Here, the high-frequency image Nk(i) may be a local mean-removed image.

Conversely, the low-frequency image Mk(i) may be a local mean intensity image.

As the input image X(k) is divided into a high-frequency image Nk(i) and a low-frequency image Mk(i), the high-frequency image Nk(i) is separated from the low-frequency image X(k) in the input image X(k). It can be obtained by removing the image Mk(i).

수식 (22)

formula (22)

The weight function calculation step S306 may calculate a normalized high frequency weight function Ak(i) based on the signal intensities ck(i) of the K high frequency images Nk(i).

For example, in the weight function calculation step S306, the weight function ω(k)) derived from Equation (14) of the first embodiment described above is applied as the high frequency weight function Ak(i). can

)을 산출할 수 있다.The high-frequency image weighting step S308 is performed by weighting the high-frequency image Nk(i) with the calculated high-frequency weighting function Ak(i) to perform the high-frequency image weighting (

) can be calculated.

를 산출할 수 있다.Similarly, the low-frequency image weighting step S310 is performed by weighting the low-frequency image Mk(i) with the low-frequency weighting function Bk(i) to perform the low-frequency image weighting (

can be calculated.

Of course, the low-frequency image weighting step (S310) may be performed simultaneously with the high-frequency image weighting step (S308) or before the high-frequency image weighting step (S308).

The high frequency image weighting step (S308) and the low frequency image weighting step (S310) can be expressed by Equation (23).

수식 (23)

formula (23)

의 합성고주파영상은

, 합성저주파영상은

로 정의될 수 있다.At this time, the composite image

The synthesized high-frequency image of

, synthetic low-frequency images are

can be defined as

)을 상기 최대신호강도(Cmax(i))로 근사하는 고주파영상가중합근사단계를 포함할 수 있다.In this case, further from Equation (23), the synthesis image generating method arranges the K high-frequency images Nk(i) according to the signal strength and has the maximum signal strength (Cmax(i)) ) a maximum signal intensity calculation step of calculating; The high-frequency image weighting (

) may include a high-frequency image weighted sum approximation step of approximating the maximum signal intensity (Cmax(i)).

In the calculating of the maximum signal strength, the K high-frequency images Nk(i) may be arranged according to the signal strength to calculate the maximum signal strength Cmax(i) having the largest signal strength.

)을 상기 최대신호강도(Cmax(i))로 근사하여 상기 수식 (23)을 추가적으로 간략화할 수 있다.Through this, the high-frequency image weighted approximation step is the high-frequency image weighted sum (

Equation (23) can be further simplified by approximating ) to the maximum signal intensity (Cmax(i)).

Through the high-frequency image weighted sum approximation step, Equation (23) can be simplified to Equation (24) below.

수식 (24)

formula (24)

In the high-frequency image weighted sum approximation step, by applying the maximum value derived from Equation (20) of the second embodiment described above, there is an advantage that the processing speed can be increased by omitting the calculation for the weight function.

On the other hand, in the synthetic image generating method, a low frequency weighting function for calculating the low frequency weighting function (Bk(i)) based on the global brightness value μk and the local brightness value lk of the K input images X(k) It may further include a calculation step.

)이 적용될 수 있다.The low-frequency weighting function (Bk(i)) calculates the low-frequency weighting function (Bk(i)) based on the global brightness μk and the local brightness lk of the K input images X(k). As a step, the weighting function of the average brightness of the first embodiment

) can be applied.

)과 저주파영상가중합(

을 더해 상기 합성영상을 생성할 수 있다.The composite image generating step (S312) is a high-frequency image weighted sum (

) and low-frequency image weighting (

can be added to generate the composite image.

5C shows the synthesized image obtained through the third embodiment when the input image X(k) of FIGS. 4A to 4C is used as an input image. A composite image can be obtained.

Table 1 below shows the simulation evaluation results through the synthetic image generating method according to the first to third embodiments.

As shown in Table 1, in the present invention, the SSIM INDEX (Structural similarity) is very close to 1, despite the efficient approach by greatly simplifying the synthetic image generation method compared to the prior art, so that the synthesis result of the existing synthetic image generation method and It can be seen that the results are very similar.

input images number of images Method according to the first embodiment
SSIM INDEX. Method according to the third embodiment
SSIM INDEX Arno 3 0.9972 0.9958 Balloons 9 0.9954 0.9964 Belgium 9 0.9925 0.9942 Ostrow 3 0.9951 0.9951 Garden 3 0.9970 0.9973 Garage 6 0.9783 0.9795 Grand canal 3 0.9862 0.9815 Iglue 6 0.9645 0.9635 Laurenziana 3 0.9957 0.9954 Hasselt 4 0.9912 0.9912 Chairs 5 0.9952 0.9935 Mask 3 0.9979 0.9984 Eiffel Tower 3 0.9975 0.9975 Memorial 16 0.9880 0.9913 Cave 4 0.9965 0.9967 Kluki 3 0.9953 0.9954 Average 0.9921 0.9914

<Table 1- Image quality performance evaluation result>

In addition, Table 2 below shows the speed required to generate a composite image through the method for generating a synthetic image according to the present invention. gave.

Method Implementation image size
320×240 image size
720×480 image size
1366×768 image size
1920x1080 prior art 1 toolbox 0.366 1.466 33.620 48.878 prior art 2 toolbox 0.194 0.404 1.258 2.415 prior art 3 toolbox 0.391 0.769 2.154 4.240 prior art 4 toolbox 0.129 0.656 2.124 4.465 prior art 5 toolbox 0.432 2.025 6.502 12.869 Method according to the first embodiment toolbox 0.067 0.275 0.865 1.742 Method according to the third embodiment toolbox 0.049 0.229 0.688 1.382 Method according to the first embodiment C/C++ 0.031 0.135 0.405 0.835 Method according to the third embodiment C/C++ 0.009 0.039 0.121 0.231

When the synthesis image generating method according to the present invention is applied, it is expected that real-time synthesis is possible for images of all sizes.

The key to image synthesis depends on how quickly the weighting function is obtained. If you use the method of finding the weights by reducing the size of the input image and then expanding it back to the size of the original image, then full high definition images can be obtained in real time. It has a very big advantage in that it is a technology that can display synthesized images by synthesizing signals directly from the camera sensor stage as well as post-processing the acquired image by greatly reducing the amount of computation.

In another aspect, the present invention may be defined as a computer program stored in a computer-readable medium for executing the above-described synthetic image generating method using a computer.

Since the above has only been described with respect to some of the preferred embodiments that can be implemented by the present invention, as noted, the scope of the present invention should not be construed as being limited to the above embodiments, and It will be said that the technical idea and the technical idea accompanying the fundamental are all included in the scope of the present invention.

100: synthetic image generating device

Claims (8)

A synthetic image generating method for generating a synthetic image by synthesizing K different input images (X(k)) (K is a natural number greater than or equal to 2, 1≤k≤K),
A patch set defining a patch set {CP _k } including K color image patches (CP(k), 1≤k≤K) extracted from the same position of each of the K input images X(k) defining step (S202);
The K color image patches (CP(k)) are analyzed for signal intensity (c(k)), signal structure (

), and average brightness (

) of K patch vectors (

, 1≤k≤K) defining a patch vector defining step (S204);
The K patch vectors (

) using the average brightness (

) of the K mean removed patch vectors (

, 1≤k≤K) and calculating an average removed patch vector (S206);
The K average removed patch vectors (

) a weighting function calculation step (S208) of calculating a normalized weighting function (ω(k)) based on each of the signal strengths (c(k));
The weight function (ω(k)) and the K average removed patch vectors (

), a weighted sum vector calculating step (S210) of calculating a weighted sum vector for them;
Synthetic patch vector (

), a synthetic patch vector calculating step (S212) for calculating a synthetic image generating method. The method according to claim 1,
The K average removed patch vectors (

) are sorted according to the signal strength, and the maximum signal strength vector (

) a maximum signal intensity vector calculating step;
The weighted sum vector is the maximum signal intensity vector (

), and a weighted sum vector approximation step. The method according to claim 1,
The synthetic patch vector (

) based on the synthesized image generating method, characterized in that it further comprises a synthesized image generating step of generating the synthesized image. delete delete delete using a computer
A method for generating a synthesized image according to any one of claims 1 to 3 to generate a synthesized image by synthesizing K different input images (X(k)) (K is a natural number greater than or equal to 2, 1≤k≤K) A computer program stored on a computer readable medium for execution. A method for generating a synthesized image according to any one of claims 1 to 3 to generate a synthesized image by synthesizing K different input images (X(k)) (K is a natural number greater than or equal to 2, 1≤k≤K) Composite image generating apparatus including a synthesis processing module to perform.

Images