KR102089084B1 - High dynamic range image generation system, method and computer readable recording medium thereof - Google Patents

Abstract

Disclosed is an HDR image generation system, method and computer readable recording medium. The HDR image generation system according to an embodiment of the present invention includes an image receiving unit receiving an LDR image, a saturation region removing unit detecting a saturation region from the LDR image, and removing the saturation region from the LDR image. And an image replacement unit that generates a replacement image corresponding to and inserts the replacement image into the LDR image in which the saturation region is removed.

Description

HDR DYNAMIC RANGE IMAGE GENERATION SYSTEM, METHOD AND COMPUTER READABLE RECORDING MEDIUM THEREOF

The present invention relates to an HDR image generation system, method, and computer-readable recording medium, and more specifically, an HDR image generation system, method, and computer readable capable of generating an HDR image by applying an alternative image to a portion of an LDR image. It is about the record carrier.

In the meantime, the input image source of the display is SDR (Standard Dynamic Range), and the range of darkness is narrower than the range seen by the human eye, and there is a problem that the color of a specific object cannot be expressed. For example, in the case of welding, when the spark of the welding is taken with a general camera, the intensity of the corresponding part may appear saturated and appear white, and even in the case of a normal picture, the color intensity of the corresponding part may be Saturation occurs in white.

In the high light source (welding, furnace, laser pointing, etc.) industry, there is an urgent need for an HDR (High Dynamic Range) camera that does not cause loss of image information even in a high light source in order to inspect a production line or inspect an article (product) being manufactured. In addition, HDR cameras are expected to play an important role in machine vision and autonomous driving systems where video plays an important role.

In general, the price of a high-end HDR camera image sensor is high, so the unit price is high. Therefore, if an HDR camera image can be effected while using a low dynamic range (LDR) sensor, this can have a great advantage in terms of cost.

For this, an image processing algorithm is needed to expand the image captured by the LDR sensor into an HDR image.

An object of the present invention is to provide an HDR image generation system, method and a computer-readable recording medium capable of generating an HDR image from an LDR image by applying an algorithm having low complexity.

The HDR image generation system according to an embodiment of the present invention includes an image receiving unit receiving an LDR image, a saturation region removing unit detecting a saturation region from the LDR image, and removing the saturation region from the LDR image. And an image replacement unit that generates a replacement image corresponding to and inserts the replacement image into the LDR image in which the saturation region is removed.

In addition, the saturation region removal unit may detect the saturation region by using an average of minimum values among pixel values of each pixel included in the LDR image and a standard deviation of the minimum values.

Also, the image replacement unit may generate the replacement image by calculating an average difference between a pixel value of a pixel corresponding to the saturation region and a pixel value of all pixels included in the LDR image.

In addition, the tone mapping unit may perform a tone mapping (Tone Mapping) on the image generated by the image replacement unit, the tone mapping unit may average the brightness value of each pixel included in the replacement image and the brightness of each pixel. A brightness value to be applied to each pixel may be determined from a ratio of values.

Meanwhile, a method for generating an HDR image according to an embodiment of the present invention includes receiving an LDR image, detecting a saturated region from the LDR image, removing the saturated region from the LDR image, and corresponding to the saturated region. And generating the replacement image and inserting the replacement image into the LDR image in which the saturation region is removed.

In addition, in the step of removing the saturation region, the saturation region may be detected using an average of minimum values among pixel values of each pixel included in the LDR image and a standard deviation of the minimum values.

Also, in the step of generating the replacement image, the replacement image may be generated by calculating a difference between an average of pixel values of pixels corresponding to the saturation region and pixel values of all pixels included in the LDR image.

In addition, the method may further include performing tone mapping on an image in which the replacement image is inserted, and in performing the tone mapping, the average of the brightness values of the pixels included in the replacement image and each pixel. A brightness value to be applied to each pixel may be determined from a ratio of the brightness values of.

Meanwhile, a computer-readable recording medium in which a program for performing the HDR image generation method according to the present invention is recorded may be provided.

The present invention can provide an HDR image generation system, method, and computer-readable recording medium capable of generating an HDR image from an LDR image by applying an algorithm having low complexity.

1 is a view schematically showing the configuration of an HDR image generation system according to an embodiment of the present invention.
2 is a diagram schematically showing the configuration of an HDR image generation system according to another embodiment of the present invention.
FIG. 3 is a diagram exemplarily showing a process of generating an HDR image from an LDR image using the HDR image generation system according to the present invention.
4 is a flowchart schematically showing a method for generating an HDR image according to an embodiment of the present invention.
5 is a flowchart schematically illustrating a method for generating an HDR image according to another embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent by referring to embodiments described in detail with reference to the accompanying drawings. However, the present invention is not limited to the embodiments presented below, but can be implemented in various different forms, and should be understood to include all transformations, equivalents, and substitutes included in the spirit and scope of the present invention. do. The embodiments presented below are provided to make the disclosure of the present invention complete, and to fully disclose the scope of the invention to those skilled in the art to which the present invention pertains. In the description of the present invention, when it is determined that detailed descriptions of related known technologies may obscure the subject matter of the present invention, detailed descriptions thereof will be omitted.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, terms such as "comprises" or "have" are intended to indicate the presence of features, numbers, steps, actions, components, parts or combinations thereof described in the specification, but one or more other features. It should be understood that the existence or addition possibilities of fields or numbers, steps, operations, components, parts or combinations thereof are not excluded in advance. Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from other components.

1 is a view schematically showing the configuration of an HDR image generation system according to an embodiment of the present invention.

Referring to FIG. 1, the HDR image generation system 100 according to an embodiment of the present invention includes an image receiving unit 110, a saturation region removal unit 120, and an image replacement unit 130. The HDR image generation system 100 according to the present invention aims to generate an HDR (High Dynamic Range) image from an LDR (Low Dynamic Range) image, and the image receiving unit 110 generates an LDR image for generating an HDR image. To receive.

The dynamic range (DR) of a digital image may be defined as a ratio of the brightest pixel value and the darkest pixel value in the image, and the HDR image may be understood as an image having a high dynamic range. On the other hand, the dynamic band can also be understood as representing the ratio of the minimum intensity of light (dark intensity) to the maximum intensity of light (white intensity) that can be measured, and can be defined by the following equation.

The saturation region removing unit 120 detects a highlight area in the LDR image received by the image receiving unit 110 and removes the saturation region from the LDR image. The saturation region removing unit 120 extracts a minimum value of RGB data (pixel values) of pixels included in the LDR image before detecting the saturation region. The HDR image generation system 100 according to the present invention generates an HDR image by processing an LDR image frame by frame, so the minimum value of the RGB data is the RGB data of a pixel included in one LDR frame. Means the minimum value. For example, when RGB data of a pixel corresponding to (10, 10) of the LDR frame is (100, 150, 200), 100 may be extracted as a minimum value in the corresponding pixel.

Thereafter, the saturation region removing unit 120 may calculate a standard deviation and an average of the minimum values, and calculate a threshold and an offset using the minimum value, the standard deviation, and the average. Specifically, the threshold and the offset may be calculated through the following equation.

은 각 화소의 RGB 데이터 중 최소 값들의 평균, I_std는 각 화소의 RGB 데이터 중 최소 값들의 표준 편차,

는 표준 편차의 계수, 그리고

는 (x, y) 위치의 화소에 대한 오프셋을 의미한다. 한편, 상기 표준 편차의 계수는 실험적으로 약 2.5로 정해질 수 있으나 그 수치가 특정한 값으로 제한되는 것은 아니다.Where T _hd is the threshold,

Is the average of the minimum values of the RGB data of each pixel, I _std is the standard deviation of the minimum values of the RGB data of each pixel,

Is the coefficient of standard deviation, and

Denotes an offset for a pixel at (x, y). On the other hand, the coefficient of the standard deviation may be experimentally determined to be about 2.5, but the number is not limited to a specific value.

Different values may be applied to the coefficient according to the LDR image, and subscript hd means highlight detection.

Meanwhile, the saturation region removal unit 120 may detect the saturation region through the following equation.

는 포화 영역,

는 포화 영역에 해당하지 않는 영역을 의미한다. 따라서, 상기 LDR 영상에 포함되는 화소의 총 개수는 상기 포화 영역에 해당하는 화소의 개수와 나머지 영역에 해당하는 화소의 개수의 합과 같은 것으로 이해할 수 있다.here,

Is the saturated region,

Means a region that does not correspond to a saturated region. Therefore, it can be understood that the total number of pixels included in the LDR image is equal to the sum of the number of pixels corresponding to the saturation region and the number of pixels corresponding to the remaining region.

값은 포화 영역에 대해서는 1, 나머지 영역에 대해서는 0의 값을 가질 수 있으며,

값은 포화 영역에 대해서는 0, 나머지 영역에 대해서는 1의 값을 가질 수 있다.At this time,

The value can have a value of 1 for the saturated region and 0 for the remaining region,

The value can have a value of 0 for the saturated region and 1 for the remaining region.

Meanwhile, in Equation 3, the saturation region removing unit 120 sets a threshold using the average and standard deviation of the minimum values of the pixel values of each pixel included in the LDR image, and the threshold and the minimum values It can be understood that the saturation region is detected by comparing the means.

After detecting the saturation region, the saturation region removal unit 120 removes the saturation region from the LDR image. The image removed corresponding to the saturated region is replaced with another image by the image replacement unit 130.

The image replacement unit 130 generates a replacement image corresponding to the saturation region, and the replacement image may be generated using the following equation.

는 상기 대체 영상을 의미하며,

는 상기 LDR 영상에 포함된 픽셀(x, y)의 RGB 값,

는 픽셀(x, y)의 RGB 값 중 최소 값을 의미한다. 그리고,

는 다음의 수식으로 정의될 수 있다.here,

Means the replacement image,

Is an RGB value of a pixel (x, y) included in the LDR image,

Denotes the minimum value of the RGB values of the pixels (x, y). And,

Can be defined by the following equation.

는 상기 대체 영상에 대한 임계값,

는

에 대한 계수이다.here,

Is the threshold for the replacement image,

The

Is the coefficient for.

The image replacement unit 130 generates the replacement image through Equations 4 and 5, and replaces the saturation region detected by the saturation region removal unit 120 with the replacement image.

The replacement process of the saturated region with the replacement image may be expressed by the following equation.

는 포화 영역이 대체 영상으로 대체된 영상을 의미하며, 상기 수식은 포화 영역(

)은 대체 영상(

)으로 대체되고, 나머지 영역(

)은 원래의 LDR 영상(

)으로 유지되는 것을 의미한다.

Means an image in which the saturated region is replaced by an alternative image, and the above formula is a saturated region (

) Is an alternative video (

), And the rest of the area (

) Is the original LDR image (

).

항은 다음과 같이

항으로 대체될 수 있다.On the other hand, the image generated through Equation 6 may have a brightness imbalance between pixels, and in order to solve this imbalance

The term is as follows

Term.

는

의 밝기를 의미하고, V 값은 이미지에 따라 다른 값이 적용되는 계수, 그리고

는 밝기 값이 조정된 대체 영상을 의미한다.here,

The

Means the brightness, V value is the coefficient to which different values are applied depending on the image, and

Indicates an alternative image in which the brightness value is adjusted.

Accordingly, Equation 6 may be expressed as follows.

는 포화 영역이 제거된 HDR 영상으로, Highlight Free Imgae를 의미한다.here,

Is the HDR image with the saturation region removed, which means Highlight Free Imgae.

2 is a diagram schematically showing the configuration of an HDR image generation system according to another embodiment of the present invention.

Referring to FIG. 2, the HDR image generation system 200 according to another embodiment of the present invention includes an image receiving unit 210, a saturation region removal unit 220, an HDR image replacement unit 230, and a tone mapping unit 240 ). The image receiving unit 210, the saturation region removal unit 220, and the image replacement unit 230 are the image receiving unit 110, the saturation region removal unit 120, and the image replacement unit 130 described with reference to FIG. 1. And performs substantially the same function, so detailed descriptions will be omitted only for overlapping contents.

The tone mapping unit 240 performs tone mapping on the image generated by the image replacement unit 230. The tone mapping may be understood as applying a new brightness value so that the image (ie, Highlight Free Image) output from the image replacement unit 230 is more natural.

The tone mapping unit 240 may first calculate the logarithm average of the brightness value of the image calculated through Equation (8).

는

의 로그 평균 밝기 값을 의미하며,

는

의 밝기 값을 의미한다. 또한,

를 계산하는 방법과 동일한 방법을 이용하여 계산될 수 있다.here,

The

Means the logarithmic average brightness value of

The

Means the brightness value of In addition,

It can be calculated using the same method as the method for calculating.

Meanwhile, a new brightness value for performing tone mapping may be defined as follows.

Here, a means a key value applied differently according to the target image, and a high value may be applied to a relatively bright frame and a low value to a relatively dark frame.

In the present invention, a brightness value finally applied as shown in Equation 11 may be applied by modifying the conventional method.

값은 상기 키 값과 마찬가지로 더 나은 HDR 영상을 얻기 위하여 대상 이미지마다 서로 다른 값이 적용될 수 있다.here,

As for the key value, different values may be applied to each target image to obtain a better HDR image.

Meanwhile, the tone mapping unit 240 may calculate each pixel value as follows in order to apply the brightness value L _d calculated through Equation (11).

는 톤 맵핑을 거쳐 최종적으로 생성되는 HDR 영상을 의미하며,

를 만족한다.here,

Means the HDR image that is finally generated through tone mapping,

Is satisfied.

3 is a view exemplarily showing a process of generating an HDR image from an LDR image using the HDR image generation system according to the present invention.

3 (a) to 3 (f) respectively show an LDR image, a saturated region, an alternative image, an image replaced by the replacement image, an image in which brightness imbalance is resolved, and an image to which tone mapping is applied.

값은 0.5가 적용되었으며, 밝기 불균형을 해소하기 위한 V 값은 1.2가 적용되었다. 한편, a 값과 L_white 값은 최종 결과물인 HDR 영상의 밝기 값의 표준 편차가 최소가 되도록 선택될 수 있으며, 도 3에서는 각각 0.05와 0.19가 적용되었다.In the image shown in Figure 3

The value of 0.5 was applied, and the V value of 1.2 was applied to solve the brightness imbalance. Meanwhile, the a value and the L _white value may be selected so that the standard deviation of the brightness value of the final result HDR image is minimum, and in FIG. 3, 0.05 and 0.19 were applied, respectively.

Referring to FIG. 3, as a result of generating an HDR image from an LDR image by applying the HDR image generation system according to the present invention, an overexposed or underexposed portion may be removed and an overall image may have improved visibility.

4 is a flowchart schematically illustrating a method for generating an HDR image according to an embodiment of the present invention.

Referring to FIG. 4, an HDR image generation method according to an embodiment of the present invention includes an LDR image reception step (S110), a saturation region removal step (S120), an alternate image generation step (S130), and an alternate image insertion step (S140). ).

The HDR image generation method according to the present invention aims to generate a high dynamic range (HDR) image from a low dynamic range (LDR) image, and the LDR image reception step (S110) receives an LDR image for generating an HDR image. do. Meanwhile, a dynamic range (DR) of a digital image may be defined as in Equation 1.

In the saturation region removal step (S120), a highlight area is detected in the LDR image, and the saturation region is removed from the LDR image. In the saturation region removal step (S120), a minimum value among RGB data of pixels included in the LDR image is extracted before detecting the saturation region. In the HDR image generating method according to the present invention, an HDR image is generated by processing an LDR image frame by frame, so the minimum value of the RGB data is the minimum value among the RGB data of pixels included in one LDR frame. Means For example, when RGB data of a pixel corresponding to (10, 10) of the LDR frame is (100, 150, 200), 100 may be extracted as a minimum value in the corresponding pixel.

Thereafter, in the saturation region removal step (S120), a standard deviation and an average of the minimum values are calculated, and a threshold and an offset can be calculated using the minimum value, the standard deviation, and the average. Specifically, the threshold and the offset may be calculated through Equation (2).

Meanwhile, in the saturation region removal step (S120), the saturation region may be detected using the threshold and offset, and as an example, the saturation region may be detected through Equation (3).

In the replacement image generation step S130, the replacement image corresponding to the saturated region is generated, and the replacement image may be generated through Equations 4 and 5. Equations 4 and 5 may be interpreted as calculating a difference between a pixel value of a pixel corresponding to the saturation region and a pixel value of all pixels included in the LDR image.

Finally, in the replacement image insertion step (S140), the replacement image is inserted into the LDR image in which the saturation region is removed, and the process of replacing the replacement image with the replacement image may be expressed through Equation (6). In addition, after inserting the replacement image, the brightness between pixels may be balanced through Equation 7 in order to solve the brightness imbalance between pixels.

5 is a flowchart schematically illustrating a method for generating an HDR image according to another embodiment of the present invention.

5, the HDR image generation method according to another embodiment of the present invention, LDR image receiving step (S210), saturation region removal step (S220), replacement image generation step (S230), replacement image insertion step (S240) ) And a tone mapping step (S250). In the LDR image reception step (S210), the saturation area removal step (S220), the replacement image generation step (S230), and the replacement image insertion step (S240), the LDR image reception step (S110) and the saturation area described with reference to FIG. Since substantially the same operation is performed in the removal step (S120), the replacement image generation step (S130), and the replacement image insertion step (S140), detailed description of the overlapped content will be omitted.

In the tone mapping step (S250), tone mapping is performed on an image in which the replacement image is inserted. The tone mapping may be understood as applying a new brightness value so that the image generated in the replacement image insertion step (S240) is more natural. In the tone mapping step (S250), first, a logarithm average of the brightness value of the image calculated through Equation 8 may be calculated.

In the tone mapping step (S250), a brightness value to be applied to each pixel may be determined from an average of the brightness values of pixels included in the replacement image and a brightness value of each pixel, and specifically, Equations 9 to 11 Tone mapping can be performed. And, finally, an HDR image to which tone mapping is applied may be generated through Equation (12).

Meanwhile, the present invention can be embodied in computer readable codes on a computer readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data readable by a computer system are stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, and optical data storage devices.

In addition, the computer-readable recording medium can be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion. In addition, functional programs, codes, and code segments for implementing the present invention can be easily inferred by programmers in the technical field to which the present invention pertains.

Unless explicitly stated or contrary to the steps constituting the method according to the invention, the steps can be done in a suitable order. The present invention is not necessarily limited to the description order of the above steps.

The use of all examples or exemplary terms (eg, etc.) in the present invention is merely for describing the present invention in detail, and the scope of the present invention is due to the examples or exemplary terms, unless it is limited by the claims. It is not limited. In addition, a person skilled in the art can recognize that various modifications, combinations, and changes can be configured according to design conditions and factors within the scope of the appended claims or equivalents thereof.

Accordingly, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the present invention, as well as the claims to be described later, and all ranges that are equivalent to or equivalently changed from the claims of the present invention It would be said to fall into the category.

100, 200: HDR image generation system
110, 210: video receiver
120, 220: saturation region removal unit
130, 230: video replacement
240: tone mapping unit

Claims (11)

An image receiving unit that receives a low dynamic range (LDR) image;
A saturation region removal unit detecting a saturation region from the LDR image and removing the saturation region from the LDR image; And
An image replacement unit generating an alternate image corresponding to the saturation region, and inserting the replacement image in the LDR image from which the saturation region is removed to obtain a first HDR image; And
A tone mapping unit calculating a tone mapping mapping brightness value and obtaining a second tone mapped HDR image by performing tone mapping on a first HDR image based on the tone mapping brightness value;
Including,
The tone mapping brightness value L _d (x, y) is

Is calculated through,
The L (x, y) is a second brightness value obtained by dividing the first brightness value of the first HDR image by the log average brightness value of the first HDR image, and L _white is the standard of the brightness value of the second HDR image. HDR (High Dynamic Range) image generation system, which is a value set so that the deviation is minimum. According to claim 1,
The saturation region removing unit detects the saturation region using an average of minimum values among pixel values of each pixel included in the LDR image and a standard deviation of the minimum values. According to claim 1,
The image replacement unit HDR image generation system to generate the replacement image by calculating the difference between the average of the pixel values of the pixels included in the LDR image and the pixel value of the pixel corresponding to the saturation region. delete delete Receiving a low dynamic range (LDR) image;
Detecting a saturation region in the LDR image and removing the saturation region from the LDR image;
Generating an alternate image corresponding to the saturated region; And
Obtaining a first HDR image by inserting the replacement image into the LDR image from which the saturation region is removed; and
Calculating a tone mapping brightness value, and performing tone mapping on a first HDR image based on the tone mapping brightness value to obtain a tone-mapped second HDR image;
Including,
The tone mapping brightness value L _d (x, y) is

Is calculated through,
The L (x, y) is a second brightness value obtained by dividing the first brightness value of the first HDR image by the log average brightness value of the first HDR image, and L _white is the standard of the brightness value of the second HDR image. A method of generating a high dynamic range (HDR) image, which is a value set to have a minimum deviation. The method of claim 6,
In the step of removing the saturation region, an HDR image generation method of detecting the saturation region using an average of minimum values among pixel values of each pixel included in the LDR image and a standard deviation of the minimum values. The method of claim 6,
In the generating of the replacement image, an HDR image generation method of generating the replacement image by calculating an average difference between a pixel value of a pixel corresponding to the saturation region and a pixel value of all pixels included in the LDR image. delete delete A computer-readable recording medium in which a program for performing the method according to any one of claims 6 to 8 is recorded.

Images