KR20190097838A - High dynamic range image generation system, method and computer readable recording medium thereof - Google Patents

Abstract

Disclosed are a high dynamic range (HDR) image generation system, a method and a computer readable recording medium thereof. An HDR image generation system according to an embodiment of the present invention comprises: an image receiving unit receiving a lower dynamic range (LDR) image; a saturated area removing unit detecting a saturated area from the LDR image and removing the saturated area from the LDR image; and an image substituting unit generating a substitution image corresponding to the saturated area and inserting the substitution image to the LDR image of which the saturated area is removed.

Description

HDR DYNAMIC RANGE IMAGE GENERATION SYSTEM, METHOD AND COMPUTER READABLE RECORDING MEDIUM THEREOF

The present invention relates to an HDR image generating system, a method and a computer readable recording medium, and more particularly, to an HDR image generating system, a method and a computer readable that can generate an HDR image by applying a replacement image to a portion of an LDR image. It relates to a recording medium.

In the meantime, the input image source of the display is SDR (Standard Dynamic Range), which has a problem that the range of bright and dark is narrower than that of the human eye so that the color of a specific object cannot be expressed. For example, in case of welding, when the spark of welding is taken by a general camera, the intensity of the part may be saturated and appear white. In the case of the general picture, when the exposure is over-exposed to sunlight, the color intensity of the part is Saturation occurs in white.

In the high light source (welding, furnace, laser pointing, etc.) industry, there is an urgent need for high dynamic range (HDR) cameras that do not cause loss of image information even in high light sources to inspect production lines or products (products) to be manufactured. In addition, HDR cameras are expected to play an important role in the field of machine vision and autonomous driving systems, where video plays an important role.

In general, the price of HDR camera image sensor is high because of the high price. Therefore, if you can use the HDR dynamic sensor (LDR) sensor to produce the effect of HDR camera image, this can be a great advantage in terms of cost.

This requires an image processing algorithm that extends the image captured by the LDR sensor to the HDR image.

An object of the present invention is to provide an HDR image generating system, a method and a computer-readable recording medium capable of generating an HDR image from an LDR image by applying an algorithm having a low complexity.

An HDR image generating system according to an embodiment of the present invention, an image receiving unit for receiving an LDR image, a saturation region removal unit for detecting a saturation region in the LDR image, and removes the saturation region from the LDR image and the saturation region And an image substitute unit for generating a substitute image corresponding to and inserting the substitute image into the LDR image from which the saturation region is removed.

The saturation region removal unit may detect the saturation region by using an average of minimum values of pixel values of each pixel included in the LDR image and a standard deviation of the minimum values.

The image replacement unit may generate the replacement image by calculating a difference between a pixel value of a pixel corresponding to the saturation region and an average of pixel values of the entire pixel included in the LDR image.

The apparatus may further include a tone mapping unit that performs tone mapping on the image generated by the image replacement unit, wherein the tone mapping unit may include an average of brightness values of pixels included in the replacement image and brightness of each pixel. The brightness value to be applied to each pixel may be determined from the ratio of the values.

On the other hand, the HDR image generation method according to an embodiment of the present invention, receiving an LDR image, detecting a saturation region in the LDR image, and removing the saturation region from the LDR image, corresponding to the saturation region Generating a replacement image and inserting the replacement image into the LDR image from which the saturation region is removed.

In the removing of the saturation region, the saturation region may be detected using an average of minimum values of pixel values of each pixel included in the LDR image and a standard deviation of the minimum values.

In the generating of the replacement image, the replacement image may be generated by calculating a difference between a pixel value of a pixel corresponding to the saturation region and an average of pixel values of all pixels included in the LDR image.

The method may further include performing tone mapping on the image into which the substitute image is inserted. In performing the tone mapping, the average of brightness values of pixels included in the substitute image and each pixel may be further included. A brightness value to be applied to each pixel may be determined from a ratio of brightness values of.

Meanwhile, a computer-readable recording medium having recorded thereon a program for performing the HDR image generating method according to the present invention may be provided.

The present invention can provide an HDR image generating system, a method and a computer readable recording medium capable of generating an HDR image from an LDR image by applying an algorithm having a low complexity.

1 is a diagram schematically illustrating a configuration of an HDR image generation system according to an embodiment of the present invention.
2 is a diagram schematically illustrating a configuration of an HDR image generating system according to another exemplary embodiment of the present invention.
FIG. 3 is a diagram exemplarily illustrating a process of generating an HDR image from an LDR image using the HDR image generation system according to the present invention.
4 is a flowchart schematically illustrating a method of generating an HDR image, according to an exemplary embodiment.
5 is a flowchart schematically illustrating a method of generating an HDR image, according to another exemplary embodiment.

Advantages and features of the present invention, and methods of achieving them will be apparent with reference to the embodiments described in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments set forth below, but may be embodied in many different forms and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. do. The embodiments set forth below are provided to make the disclosure of the present invention complete, and to fully inform the scope of the invention to those skilled in the art. In the following description of the present invention, when it is determined that the detailed description of the related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprises" or "having" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof. Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

1 is a diagram schematically illustrating a configuration of an HDR image generation system according to an embodiment of the present invention.

Referring to FIG. 1, the HDR image generating system 100 according to an exemplary embodiment of the present invention includes an image receiver 110, a saturation region remover 120, and an image substitute unit 130. The HDR image generation system 100 according to the present invention aims to generate a high dynamic range (HDR) image from a low dynamic range (LDR) image, and the image receiver 110 generates an LDR image for generating an HDR image. Receive.

The dynamic range (DR) of the digital image may be defined as a ratio of the brightest pixel value and the darkest pixel value in the image, and the HDR image may be understood as an image having a high dynamic range. On the other hand, the dynamic band can also be understood as representing the ratio of the minimum light intensity (dark intensity) to the maximum light intensity (white intensity) that can be measured, can be defined by the following equation.

The saturation region remover 120 detects a saturation region from the LDR image received by the image receiver 110 and removes the saturation region from the LDR image. The saturation region removal unit 120 extracts a minimum value of RGB data (pixel values) of pixels included in the LDR image before detecting the saturation region. The HDR image generation system 100 according to the present invention generates an HDR image by processing an LDR image frame by frame, so the minimum value of the RGB data is RGB data of a pixel included in one LDR frame. Means the minimum value. For example, when RGB data of a pixel corresponding to (10, 10) of the LDR frame is (100, 150, 200), 100 may be extracted as a minimum value in the corresponding pixel.

Subsequently, the saturation region removal unit 120 may calculate a standard deviation and an average of the minimum values, and calculate a threshold and an offset using the minimum value, the standard deviation, and the average. Specifically, the threshold value and the offset may be calculated through the following formula.

은 각 화소의 RGB 데이터 중 최소 값들의 평균, I_std는 각 화소의 RGB 데이터 중 최소 값들의 표준 편차,

는 표준 편차의 계수, 그리고

는 (x, y) 위치의 화소에 대한 오프셋을 의미한다. 한편, 상기 표준 편차의 계수는 실험적으로 약 2.5로 정해질 수 있으나 그 수치가 특정한 값으로 제한되는 것은 아니다.Where T _hd is the threshold,

Is the average of the minimum values of the RGB data of each pixel, I _std is the standard deviation of the minimum values of the RGB data of each pixel,

Is the coefficient of standard deviation, and

Denotes an offset with respect to the pixel at the position (x, y). On the other hand, the coefficient of the standard deviation may be experimentally set to about 2.5, but the numerical value is not limited to a specific value.

The coefficient may have different values according to the LDR image, and the subscript hd means highlight detection.

Meanwhile, the saturation region removal unit 120 may detect the saturation region through the following equation.

는 포화 영역,

는 포화 영역에 해당하지 않는 영역을 의미한다. 따라서, 상기 LDR 영상에 포함되는 화소의 총 개수는 상기 포화 영역에 해당하는 화소의 개수와 나머지 영역에 해당하는 화소의 개수의 합과 같은 것으로 이해할 수 있다.here,

Is the saturation zone,

Means a region that does not correspond to a saturation region. Therefore, the total number of pixels included in the LDR image may be understood as the sum of the number of pixels corresponding to the saturation region and the number of pixels corresponding to the remaining region.

값은 포화 영역에 대해서는 1, 나머지 영역에 대해서는 0의 값을 가질 수 있으며,

값은 포화 영역에 대해서는 0, 나머지 영역에 대해서는 1의 값을 가질 수 있다.At this time,

The value can have a value of 1 for saturated regions and 0 for the remaining regions.

The value may have a value of 0 for the saturated region and 1 for the remaining region.

Meanwhile, in Equation 3, the saturation region removing unit 120 sets a threshold value using an average and a standard deviation of minimum values of pixel values of each pixel included in an LDR image, and sets the threshold value and the minimum values. It can be understood that the saturation region is detected by comparing the averages.

The saturation region removal unit 120 detects the saturation region and then removes the saturation region from the LDR image. The image removed corresponding to the saturated region is replaced by another image by the image replacing unit 130.

The image replacement unit 130 generates a replacement image corresponding to the saturation region, and the replacement image may be generated using the following equation.

는 상기 대체 영상을 의미하며,

는 상기 LDR 영상에 포함된 픽셀(x, y)의 RGB 값,

는 픽셀(x, y)의 RGB 값 중 최소 값을 의미한다. 그리고,

는 다음의 수식으로 정의될 수 있다.here,

Means the substitute image,

Is an RGB value of a pixel (x, y) included in the LDR image,

Denotes a minimum value among the RGB values of the pixels (x, y). And,

May be defined by the following equation.

는 상기 대체 영상에 대한 임계값,

는

에 대한 계수이다.here,

Is a threshold value for the substitute image,

Is

Is the coefficient for.

The image replacement unit 130 generates the replacement image through Equations 4 and 5, and replaces the saturation region detected by the saturation region removal unit 120 with the replacement image.

The replacement process of the saturated region with the replacement image may be expressed by the following equation.

는 포화 영역이 대체 영상으로 대체된 영상을 의미하며, 상기 수식은 포화 영역(

)은 대체 영상(

)으로 대체되고, 나머지 영역(

)은 원래의 LDR 영상(

)으로 유지되는 것을 의미한다.

Denotes an image in which the saturation region is replaced with the replacement image, and the equation is the saturation region (

) Is the fallback video (

) And the rest of the area (

) Shows the original LDR image (

) To be maintained.

항은 다음과 같이

항으로 대체될 수 있다.Meanwhile, in the image generated through Equation 6, there may exist a brightness imbalance between pixels, in order to solve such an imbalance.

Terms are as follows

Can be replaced by a term.

는

의 밝기를 의미하고, V 값은 이미지에 따라 다른 값이 적용되는 계수, 그리고

는 밝기 값이 조정된 대체 영상을 의미한다.here,

Is

The brightness of, the value of V is the coefficient to which different values are applied depending on the image, and

Denotes a substitute image whose brightness value is adjusted.

Therefore, Equation 6 may be expressed as follows.

는 포화 영역이 제거된 HDR 영상으로, Highlight Free Imgae를 의미한다.here,

Is an HDR image from which the saturation region is removed, which means Highlight Free Imgae.

2 is a diagram schematically illustrating a configuration of an HDR image generating system according to another exemplary embodiment of the present invention.

Referring to FIG. 2, the HDR image generating system 200 according to another exemplary embodiment of the present invention may include an image receiver 210, a saturation region remover 220, an HDR image replacer 230, and a tone mapper 240. ). The image receiver 210, the saturation region remover 220, and the image replacer 230 include the image receiver 110, the saturation region remover 120, and the image replacer 130 described with reference to FIG. 1. Since it performs substantially the same function and detailed description will be omitted only in the overlapping content.

The tone mapping unit 240 performs tone mapping on the image generated by the image replacement unit 230. The tone mapping may be understood as applying a new brightness value so that the image (ie, Highlight Free Image) output from the image replacement unit 230 is more natural.

The tone mapping unit 240 may first calculate a log average of brightness values of an image calculated through Equation 8.

는

의 로그 평균 밝기 값을 의미하며,

는

의 밝기 값을 의미한다. 또한,

를 계산하는 방법과 동일한 방법을 이용하여 계산될 수 있다.here,

Is

Mean the logarithmic brightness value of,

Is

Means the brightness value. Also,

It can be calculated using the same method as that for calculating.

Meanwhile, a new brightness value for performing tone mapping may be defined as follows.

Here, a means a key value applied differently according to the target image, and a high value may be applied to a relatively bright frame and a low value to a relatively dark frame.

In the present invention, a brightness value that is finally applied as shown in Equation 11 may be applied by modifying the conventional method.

값은 상기 키 값과 마찬가지로 더 나은 HDR 영상을 얻기 위하여 대상 이미지마다 서로 다른 값이 적용될 수 있다.here,

Like the key value, different values may be applied to each target image to obtain a better HDR image.

Meanwhile, the tone mapping unit 240 may calculate each pixel value as follows to apply the brightness value L _d calculated through Equation 11 above.

는 톤 맵핑을 거쳐 최종적으로 생성되는 HDR 영상을 의미하며,

를 만족한다.here,

Refers to the HDR image finally generated by tone mapping.

Satisfies.

FIG. 3 is a diagram exemplarily illustrating a process of generating an HDR image from an LDR image using the HDR image generation system according to the present invention.

3 (a) to 3 (f) show an LDR image, a saturation region, a substitute image, an image replaced by the substitute image, an image in which brightness imbalance is eliminated, and an image to which tone mapping is applied.

값은 0.5가 적용되었으며, 밝기 불균형을 해소하기 위한 V 값은 1.2가 적용되었다. 한편, a 값과 L_white 값은 최종 결과물인 HDR 영상의 밝기 값의 표준 편차가 최소가 되도록 선택될 수 있으며, 도 3에서는 각각 0.05와 0.19가 적용되었다.In the image shown in Figure 3

The value 0.5 was applied, and the V value 1.2 was used to solve the brightness imbalance. On the other hand, a value and L _white value may be selected so that the standard deviation of the brightness value of the HDR image as the final result is the minimum, 0.05 and 0.19 were applied in FIG.

Referring to FIG. 3, as a result of generating the HDR image from the LDR image by applying the HDR image generating system according to the present invention, an overexposed or underexposed portion may be removed and the visibility of the entire image may be improved.

4 is a flowchart schematically illustrating a method of generating an HDR image, according to an exemplary embodiment.

Referring to FIG. 4, in the HDR image generation method according to an embodiment of the present invention, an LDR image reception step S110, a saturation region removal step S120, a replacement image generation step S130, and a replacement image insertion step S140 are performed. ).

The HDR image generating method according to the present invention aims to generate a high dynamic range (HDR) image from a low dynamic range (LDR) image, and in the step of receiving an LDR image (S110), an LDR image for generating an HDR image is received. do. Meanwhile, the dynamic range (DR) of the digital image may be defined as in Equation 1.

In the saturation region removal step (S120), a saturation region is detected in the LDR image, and the saturation region is removed from the LDR image. In the saturation region removal step (S120), the minimum value of the RGB data of the pixels included in the LDR image is extracted before the saturation region is detected. In the HDR image generating method according to the present invention, an HDR image is generated by processing an LDR image frame by frame, and thus the minimum value of the RGB data is the minimum value among the RGB data of pixels included in one LDR frame. Means. For example, when RGB data of a pixel corresponding to (10, 10) of the LDR frame is (100, 150, 200), 100 may be extracted as a minimum value in the corresponding pixel.

Subsequently, in the saturation region removing step S120, a standard deviation and an average of the minimum values may be calculated, and a threshold and an offset may be calculated using the minimum value, the standard deviation, and the average. Specifically, the threshold value and the offset may be calculated through Equation 2.

Meanwhile, in the saturation region removal step (S120), the saturation region may be detected using the threshold value and the offset, and as an example, the saturation region may be detected through Equation 3.

In the substitute image generation step (S130), a substitute image corresponding to the saturation region is generated, and the substitute image may be generated through the equations (4) and (5). Equations 4 and 5 may be interpreted as calculating a difference between a pixel value of a pixel corresponding to the saturation region and an average of pixel values of the entire pixel included in the LDR image.

Finally, in the replacement image insertion step (S140), the replacement image is inserted into the LDR image from which the saturation region has been removed, and the process of replacing the replacement image with the replacement image may be expressed by Equation 6. In addition, after the replacement image is inserted, the brightness between the pixels may be balanced through Equation 7 to solve the brightness imbalance between the pixels.

5 is a flowchart schematically illustrating a method of generating an HDR image, according to another exemplary embodiment.

Referring to FIG. 5, in the HDR image generating method according to another embodiment of the present invention, an LDR image receiving step (S210), a saturation region removing step (S220), a replacement image generating step (S230), and a replacement image insertion step (S240) ) And tone mapping step (S250). In the LDR image receiving step (S210), the saturation region removing step (S220), the replacement image generating step (S230), and the replacement image insertion step (S240), the LDR image receiving step (S110) and the saturation region described with reference to FIG. 4. Since substantially the same operations are performed as in the removing step S120, the replacing image generating step S130, and the replacing image inserting step S140, detailed descriptions of overlapping contents will be omitted.

In the tone mapping step (S250), tone mapping is performed on the image into which the substitute image is inserted. The tone mapping may be understood as applying a new brightness value to make the image generated in the sub-image insertion step S240 more natural. In the tone mapping step (S250), first, a log average of brightness values of an image calculated through Equation 8 may be calculated.

In the tone mapping step (S250), the brightness value to be applied to each pixel may be determined from a ratio of the average brightness value of the pixels included in the substitute image and the brightness value of each pixel. Tone mapping can be done via: Finally, an HDR image to which tone mapping is applied may be generated through Equation 12.

Meanwhile, the present invention can be embodied as computer readable codes on a computer readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion. And functional programs, codes and code segments for implementing the present invention can be easily inferred by programmers in the art to which the present invention belongs.

If the steps constituting the method according to the invention are not explicitly stated or contrary to the steps, the steps may be performed in a suitable order. The present invention is not necessarily limited to the description order of the above steps.

The use of all examples or exemplary terms (eg, etc.) in the present invention is merely for describing the present invention in detail, and the scope of the present invention is limited by the examples or exemplary terms unless the scope of the claims is defined. It is not limited. Also, one of ordinary skill in the art appreciates that various modifications, combinations and changes can be made in accordance with design conditions and factors within the scope of the appended claims or their equivalents.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and all the scope equivalent to or equivalent to the scope of the claims, as well as the claims to be described later, are defined by the spirit of the invention. It belongs to the category.

100, 200: HDR video generation system
110, 210: video receiver
120, 220: Saturation area removal unit
130, 230: image replacement unit
240: tone mapping unit

Claims (11)

An image receiver configured to receive an LDR image;
A saturation region removal unit for detecting a saturation region in the LDR image and removing the saturation region in the LDR image; And
An image substitute unit generating a substitute image corresponding to the saturated region and inserting the substitute image into the LDR image from which the saturated region is removed;
HDR image generation system comprising a. The method of claim 1,
And the saturation region removal unit detects the saturation region by using an average of minimum values of pixel values of each pixel included in the LDR image and a standard deviation of the minimum values. The method of claim 1,
And the image replacement unit generates the replacement image by calculating a difference between a pixel value of a pixel corresponding to the saturation region and an average of pixel values of all pixels included in the LDR image. The method of claim 1,
And a tone mapping unit configured to perform tone mapping on the image generated by the image replacement unit. The method of claim 4, wherein
And the tone mapping unit determines a brightness value to be applied to each pixel from a ratio of an average of brightness values of pixels included in the substitute image and a brightness value of each pixel. Receiving an LDR image;
Detecting a saturation region in the LDR image and removing the saturation region in the LDR image;
Generating a substitute image corresponding to the saturated region; And
Inserting the substitute image into the LDR image from which the saturation region is removed;
HDR image generation method comprising a. The method of claim 6,
In the removing of the saturation region, the HDR image generation method detects the saturation region by using the average of the minimum values of the pixel values of each pixel included in the LDR image, and the standard deviation of the minimum values. The method of claim 6,
In the generating of the replacement image, the HDR image generation method generates the replacement image by calculating a difference between a pixel value of a pixel corresponding to the saturation region and an average of pixel values of all pixels included in the LDR image. The method of claim 6,
And performing tone mapping on the image into which the substitute image is inserted. The method of claim 9,
In the performing of the tone mapping, the HDR image generation method of determining a brightness value to be applied to each pixel from a ratio of the brightness value of each pixel included in the substitute image and the brightness value of each pixel. A computer-readable recording medium having recorded thereon a program for performing the method according to any one of claims 6 to 10.

Images