KR101774158B1 - Apparatus and method of processing fog in image based on statistical vertor ellipsoid - Google Patents

Abstract

The present invention relates to a device which displays color information or brightness information as vector data in an RGB color space, wherein the color information or the brightness information is extracted from an image reduced due to fog in a foggy environment to effectively expand a statistical ellipsoid in which color information of the image is gathered, thereby reducing the fog of the image.

Description

[0001] APPARATUS AND METHOD OF PROCESSING FOG IN IMAGE BASED ON STATISTICAL VERTOR ELLIPSOID [0002]

Field of the Invention [0002] The present invention relates to image processing for improving visibility and sharpness, and more particularly, to an apparatus and method for processing fog images based on statistical vector ellipses.

Many single-image dehazing methods are based on a dichromatic model. This model describes a hazy image as a mixture of a transmitted portion of a hazy-free image and a light source portion reaching the camera.

The fog free image can be recovered using the estimated transmission rate and light source measurements. Thus, the accuracy of transmission estimation determines the performance of the fog process using a dichromatic model. Most transmission estimation methods used for a single image are based on a dark channel prior (hereinafter referred to as DCP). The DCP principle is that the contrast in the haze-free image area is sufficiently large such that the minimum color component among the pixels in the area is sufficiently close to zero. The minimum color component may be referred to as a dark channel value. The rate is estimated based on the weighted dark channel value. The weight adjusts the overall brightness or global contrast of the fog removed images, the weights being determined experimentally for the user's particular application. The dark channel value evaluates the local contrast or visibility within the detailed image signal. Dark channel values have larger values for images with more fog. The dark channel value helps to recover the refined image signals. Therefore, the ultimate performance of the DCP-based fog processing method depends on how well you estimate the dark channel value.

Depending on the characteristics of the dark channel, the randomness of the image signals has to be considered. Due to randomness, the image signals in the local region may include exceptional signals with little correlation with the overall characteristics of the underlying local image. Figure 1 shows the distribution of pixel values for two adjacent regions in a fogged image. One of the areas includes exceptional textures that appear as small dark areas. Fig. 1 (a) is for an area with no extreme texture and (b) is for an area with extreme texture. Even though the one region includes exceptional textures, the two adjacent regions have generally similar characteristics, and the haziness degrees of the regions are equal. The overall pixel value distributions within the image are similar, except that some extreme pixels are located far away from the dominant distribution. According to the DCP, the minimum color component of an area having exceptional textures is found among the extreme pixels. Thus, even though the fog extent of most areas is approximately the same, the dark channel value of an area having an exceptional texture may be less than the dark channel value of the other area. This means that the dark channel value may not be statistically robust without considering the randomness of the image signals. Therefore, appropriate dark channel values should be provided in the DCP theory and robust against statistical randomness.

On the other hand, the randomness of the image signal causes statistical instability in conventional DCP-based fogging schemes, thus making the details of the fog removed image more difficult to see. Transmission estimation mismatches can also cause halo artifacts along the boundaries of an object. Also, in situations where the DCP theory is not well maintained, color distortion of fog removed images may occur. In order to obtain a stable dark channel value corresponding to the randomness of the signal and to maintain the DCP theory well, a fog processing apparatus and method based on a statistical vector ellipse is required.

An object of the present invention is to provide an apparatus and method for processing fog images based on statistical vector ellipses.

Another object of the present invention is to provide a statistically robust transmission estimation apparatus and method using a pixel vector elliptic geometry in RGB space.

It is another object of the present invention to provide a fog processing apparatus and method for embedding a fuzzy division in an ellipse-based dark channel value estimator to avoid transmission discordance in a heterogeneous region.

Another technical problem of the present invention is to display color information or brightness information extracted from a reduced image due to fog in a foggy environment as vector data in an RGB color space to maximally expand statistical ellipsis in which color information of the image is crowded To reduce the fog of the image.

According to an aspect of the present invention, there is provided a fog processing method of an image performed by an image processing apparatus. The method comprises the steps of mapping pixels of a patch associated with the image to vectors of a color space according to their values, respectively, Constructing an approximated geometric ellipsoid in a vector cluster region, computing a dark channel value from vectors on the surface of the ellipse, calculating a dark channel value based on the dark channel value, Calculating an estimate, and obtaining a fog removed image based on the transmission estimate.

As an example, the dark channel value may be the minimum color element value among the vectors in the ellipse.

As another example, calculating the dark channel value may be performed based on a minimum value and a variance value of the color element averages within the ellipse.

As another example, calculating the dark channel value may be performed based on an average value and a variance value of the minimum color elements in the ellipse.

As another example, calculating the transmission estimate may be performed based on a product of the dark channel value and a fog processing weight.

As another example, the step of constructing the ellipse may comprise the steps of: determining a probability that a pixel is in an area including the center of the patch, if the patch of the image includes a foreground pixel and a background pixel; and And can be performed based on fuzzy segmentation.

According to another aspect of the present invention, there is provided an apparatus for processing fog of an image. The apparatus includes an input interface for inputting an original image, maps pixels of a patch of the original image to vectors of a color space according to the values, Constructing an approximated geometric ellipsoid in a vector cluster region of the vectors based on a moment, calculating a dark channel value from vectors on the surface of the ellipse, A storage for storing at least one of a processor for calculating a transmission estimate based on the channel value and a fog removed image based on the transmission estimate, vectors of the color space, the dark channel value, And an output interface for outputting the fog removed image.

As an example, the dark channel value may be the minimum color element value among the vectors in the ellipse.

As another example, the processor may calculate the dark channel value based on the minimum and variance values of the color element averages within the ellipse.

As another example, the processor may calculate the dark channel value based on an average value and a variance value of the minimum color elements in the ellipse.

As another example, the processor may calculate the transmission estimate based on a product of the dark channel value and a fog processing weight.

As another example, the processor may be configured to determine the probability that a pixel is in an area containing the center of the patch if the patch of the image includes a foreground pixel and a background pixel, segmentation of the ellipse.

According to another aspect of the present invention, there is provided a method of processing fog of an image performed by an image processing apparatus. The method includes determining whether a first pixel in the image belongs to the same area as a second pixel that requires transmission rate estimation to prevent backlight phenomenon, Constructing a geometric ellipse comprising corresponding vectors in a color space, computing a dark channel value from the vectors on the surface of the ellipse, calculating a transmission estimate based on the dark channel value, And obtaining a fog removed image based on the transmission estimate.

As an example, the second pixel requiring the estimation of the bit rate may be the center pixel of the patch.

As another example, determining whether to belong to the same area includes measuring a probability that the pixel is in an area containing the center of the patch, and obtaining a fuzzy decision value based on the probability , The ellipse may be configured based on the fuzzy decision value.

As another example, the step of embedding may be performed based on fuzzy partitioning using a guided filter.

The present invention provides a more detailed image with little color distortion and non-visible halo phenomenon and much less computational burden. Formulas that directly calculate the dark channel values from the configured geometric ellipses based on statistical measurements can be used. Further, the present invention can maintain the DCP principle, but also ensure statistical robustness. In order to avoid the halo phenomenon, the present invention can use a fast fuzzy division method as a transmission estimation method. In addition, the present invention does not scan all the pixels in the local region and can therefore have significantly lower computational complexity.

Figure 1 shows the distribution of pixel values for two adjacent regions in a fogged image.
2A and 2B illustrate a fog processing procedure based on DCP and MDCP in the RBG space.
Figure 3 is a collection of inconsistent cases by DCP and MDCP.
FIG. 4 is a view showing the occurrence of a halo phenomenon in a fog removed image. FIG.
FIG. 5 shows statistical ellipses in the RGB color space for two adjacent patches.
6 is a flowchart illustrating a fog processing method according to an embodiment.
Figure 7 shows the geometry of the vectors z ^* _r , z ^* _g and z ^* _b .
FIG. 8 is a diagram comparing DCP, MDCP, and SEDCP-based fog processing in an area where extreme pixels exist in the RGB color space.
FIG. 9 shows the dark channel values for images of adjacent patches obtained based on DCP, MDCP, and SEDCP.
10 compares the process of removing fog of an original image by DCP, MDCP and SEDCP according to the present embodiment.
11 is a flowchart showing a fog processing method according to another embodiment.
Figure 12 is a diagram illustrating a method used to construct an ellipse with only pixels in an area at the center of the patch.
Figure 13 shows various views of the halo phenomenon within the fog removal processed images.
Figure 14 compares the fogged images of trees, lakes, forests, and hills.
15 is a block diagram illustrating a fog treatment apparatus according to an embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when an element is referred to as "comprising ", it means that it can include other elements as well, without excluding other elements unless specifically stated otherwise. Also, the terms "to" and the like described in the specification mean a unit for processing at least one function or operation, which can be implemented by hardware, software, or a combination of hardware and software.

Throughout the specification, it is examined how randomness of an image signal affects fog performance, analyzing how transmission mismatch occurs in object boundaries, A method of fogging images that reduces distortion and halo is posted.

The DCP-based dark channel estimation method selects a dark channel value from an exceptional texture when extreme textures are present. Therefore, the DCP method predicts the dark channel value smaller than the actual fog and does not fully recover the details of the fog removed image. An intermediate DCP (hereinafter referred to as "MDCP") estimates a dark channel value by selecting a median value among minimum color components. Since the dark channel estimation method based on MDCP uses a statistical average value, it is hardly influenced by the values according to the exceptional texture, and therefore is statistically robust. However, the medium dark channel value is typically larger than the dark channel value, which means that the MDCP does not sustain the principle of the DCP. Due to the larger dark channel value, MDCP can cause fog removed images to be darker and produce over-contrast that distorts the color.

According to another MDCP-based dark channel estimation method, the dark channel value is regarded as the distance between the origin of the vector space and the ellipsoid centroid, and the center of the ellipse is proved to be the middle value of the color vectors. The problem of the DCP-based method is overcome by adding a compensation term to the dark channel estimation process to compensate for the darkness of the fog removed image. According to this method, a just noticeable distortion (JND) profile is employed to search for a dark channel range that produces a fogging image that is friendly to the human visual sense. However, this method does not provide optimal dark channel values for local areas. Thus, even though image quality can be improved in certain areas when following this method, details of the textures may still be sacrificed in other areas.

In the following, DCP or MDCP-based fog processing methods, particularly basic single image fog processing methods, will be described in more detail.

를 복원하며, 이때 아래의 수학식이 사용될 수 있다. The first problem with DCP or MDCP based fog processing is that it is statistically non-optimal in estimating dark channel values. A dichromatic model is a foggy image x _i located at position i,

The following equation can be used.

Referring to Equation (1), A = [A _r A _g A _b ] ^T is the global atmospheric light and t _i is the transmission estimate (or transmission value). Here, DCP and MDCP can be used as an estimation method of the transmission estimation value t _i as shown in the following equation.

및

은 추정된 다크 채널값이다. Referring to Equation (2), a is a dehazing weight, w _i is a patch like a local window centered at i,

And

Is an estimated dark channel value.

For better statistical observation of the fog image signal, the color pixel values may be mapped to the RGB color space and the pixel values as random vectors, as shown in FIGS. 2A and 2B.

는 x_j의 안개가 제거된 픽셀이다. 도 2a는 영역 내에 극단적 픽셀이 포함된 경우 DCP에 기반한 분포 확장을 보여주고, 도 2b는 MDCP에 기반한 컬러 벡터 분포 확장을 보여준다.2A and 2B illustrate a fog processing procedure based on DCP and MDCP in the RBG space. 2a and 2b, _xj is a pixel having a dark channel value,

Is the fog removed pixel of x _j . FIG. 2A shows DCP-based distribution extensions when extreme pixels are included in the region, and FIG. 2B shows MDCP-based color vector distribution extensions.

이 성립함을 알 수 있다. 따라서, RGB 공간에서 안개 처리 방법은 안개 컬러 벡터 분포를 선형적으로 확장(linearly expansion)함으로써 안개 이미지 신호의 콘트라스트(contrast)를 증가시킨다. The color vectors of the pixels with the more highly correlated cluster more densely and are generally distributed. From Equation (1)

Can be seen. Therefore, the fog processing method in the RGB space increases the contrast of the fog image signal by linearly expanding the fog color vector distribution.

In Figures 2a and 2b, each fog color vector value moves down from the atmospheric light to each fog vector until the color element designated as the dark channel reaches zero. Thus, the dark channel value is used to adjust the dehazing degrees at the pixel level, and the ultimate performance of the fogging method depends on determining the appropriate dark channel value.

2A, due to the randomness of the image signal, each region may contain extreme pixels x _j , such as noise or a small sporadic texture. The color values of the correlation pixels are clustered densely while the color values of the extreme pixels are generally much smaller or much larger than the clustered pixel values in the region. Since the DCP scheme selects the smallest value ( _xj in FIG. 2A) as the dark channel value, a dark channel value based on the DCP may be selected from one of the extreme pixels when the local region includes extreme textures. Although statistically the minimum value of the clustered pixels is more suitable as the channel value, the dark channel value is smaller than the minimum value of the clustered pixels. The expansion of the distribution based on the smaller dark channel value may not be sufficient to obtain a maximally permissible contrast. Therefore, the DCP-based fog processing method is difficult to sufficiently remove the fog of the image, making the fog processing image brighter and less detailed. This insufficient fog processing occurs because the DCP method does not properly handle the randomness of the image signals.

Referring next to Fig. 2b, _xj is located close to the center of the cluster. That is, the MDCP scheme approximates the dark channel value by selecting a median value among the minimum color elements of all the pixels in the region. Since a pixel vector having an intermediate dark channel value is usually located near the center of the distribution, the intermediate dark channel value is greater than the minimum value of the clustered pixels. The MDCP scheme extends the pixel distribution to an extreme degree due to the large dark channel value, which can make many of the fogged pixel values smaller than zero, which eventually saturates and darkens and color distortion can occur . The excessive dehazing of the fog by the MDCP method does not follow the principle of the DCP method.

Therefore, a dark channel estimation method capable of ensuring statistical robustness to the randomness of an image signal while maintaining the physical principle of the DCP is needed.

A second problem with DCP or MDCP based fog processing is that halo occurs due to transmission mismatches.

는 최소 DCP 연산부(operator) 또는 중간 MDCP 연산부 중 어느 하나에 의해 선택된 픽셀의 위치(location)이고, 이것은 전송이 측정되어야 하는 패치 중심(patch center) i와는 구별된다. 전경은 안개를 더 포함한 배경보다는 일반적으로 더 어둡기 때문에, 최소 DCP 연산부는 전경 픽셀을 선택한다. 중간 MDCP 연산부는 패치의 지배적인 영역 내에 있는 픽셀을 선택한다. 따라서, 선택된 픽셀

의 위치들은 하기 수학식과 같이 표현될 수 있다. If a patch covers heterogeneous regions, then a general transmission estimation method often causes transmission mismatches along the region boundaries, which ultimately leads to halo phenomena. The mixed patch w _i can be separated into a set of foreground pixels w _i ^{f and} a set of background pixels w _i ^b based on a rational method. Accordingly, the ^{_{w i = w i f ∪w i}} b, = w i f ∩w i b = ㅨ, and w _i ≠ ^f ㅨ, w _i ≠ ^b ㅨ in mixed patches is established.

Is the location of the pixel selected by either the minimum DCP operator or the intermediate MDCP operator and is distinct from the patch center i where the transmission is to be measured. Since the foreground is generally darker than the background with more fog, the minimum DCP operator selects foreground pixels. The intermediate MDCP operator selects pixels within the dominant region of the patch. Thus,

Can be expressed as the following equation.

Meanwhile, FIG. 3 is a collection of inconsistent cases by DCP and MDCP. When the selected pixel and the patch center belong to different areas, the value placed on the selected pixel does not indicate transmission at the patch center. Thus, a transmission phase mismatch is induced. Since the minimum DCP operation unit selects pixels in the foreground, a discrepancy occurs when the patch center is in the background. Thus, typically the DCP mismatch appears in the background and is condensed along the boundary. Because the intermediate MDCP operator selects pixels in the dominant region, MDCP creates a mismatch when the patch center is not in the dominant region. Thus, even though MDCP can reduce the number of inconsistencies compared to DCP, there is still a mismatch at the corner boundaries. Of course, while soft matting, bidirectional, and guide filters can be used to reduce mismatches in the DCP scheme, these methods may expose halo to the background area, and the halo phenomenon may be less obvious, The phenomenon is not removed, but rather the burden of the calculation amount can be increased.

FIG. 4 is a view showing the occurrence of a halo phenomenon in a fog removed image. FIG.

Referring to FIG. 4, (a) is an original image, (b) is a DCP applied to an original image, (c) is a DCP applied to an original image based on a guided filter, Is the MDCP applied to the original image. (b), the DCP causes a halo phenomenon along the tree branch boundary, and in (d), the MDCP causes a halo phenomenon at the edge boundary. (c) provides a less obvious halo phenomenon spread by the refinement filter, but the halo phenomenon becomes less obvious and can not be completely eliminated.

In the following, a statistical elliptic DCP (Statistical Ellipsoid DCP: SEDCP) based fog processing method is published. Here, since the shape of the ellipse is determined by the statistical moments of the pixel vector distributions, it is hardly influenced by pixel values (e.g., exceptional textures or noise pixels) that are not well correlated to the overall vector distribution Do not.

A fog processing apparatus and method in accordance with an embodiment can perform the step of directly determining the dark channel value from the vectors on the elliptical surface. To this end, a formula for determining the dark channel value directly from the vectors on the elliptical surface can be provided. An apparatus and method for fog treatment according to one aspect may be based on DCP theory. This is because the obtained dark channel value is the minimum color component in the vector distribution. According to one aspect, the apparatus and method for fog processing are based on the DCP theory, but also the statistical robustness can be guaranteed since the ellipse is constructed from the statistical averages. In addition, the fog processing apparatus and method according to one aspect can significantly reduce the computational complexity compared to the prior art, since there is no need to scan every pixel within the underlying region. In addition, the fog processing apparatus and method according to one aspect can avoid a transmission mismatch in a mixed patch.

The fog processing apparatus and method according to another embodiment may include a step of including a first pixel having a dark channel value in the same area as a second pixel for which a rate estimation is required in order to avoid a halo phenomenon. In order to collect pixels within an area containing pixels that require estimation of such a transmission rate, a fog processing apparatus and method according to one aspect can perform the step of embedding a fuzzy segmentation in an ellipse configuration procedure have. Here, a guided filter can be used for fuzzy division, which can provide improved performance while reducing complexity.

The present invention also can significantly reduce complexity because it calculates the dark channel value instead of scanning all the pixels in the local region.

According to the experimental results, the present invention not only avoids the halo phenomenon but also can provide a more detailed image with fog removed by a computational burden, and greatly reduce the color distortion.

In the RGB color space, the vectors from the foggy image signals cluster more densely. The vector cluster region can be statistically approximated by an ellipse, wherein the shape of the ellipse can be determined by the statistical moment of the vector distribution. The advantage of analyzing the ellipses is that they can provide statistical robustness to arbitrary signals.

Hereinafter, a method of constructing geometric ellipses by statistical moments in the RGB color space will be described. Here, Z = [z _r z _g z _b ] ^T is defined as a vector variable in the RGB color space. Also, Ω _i is defined as an ellipse fit to the vector cluster region of color pixels in w _i . Therefore,? _I can be defined, for example, by the following equation.

로서 정의되고, 평균 벡터(mean vector)

로서 정의되며, 공분산 행렬(covariance matrix)

로서 정의된다. Referring to Equation (4), a normalized pixel < RTI ID = 0.0 >

And a mean vector,

And a covariance matrix,

.

In the RGB color space, μ is located at the center of the ellipse, and Σ determines the shape and orientation of the ellipse.

FIG. 5 shows statistical ellipses in the RGB color space for two adjacent patches.

5, patch w _i and patch w _j are adjacent patches having similar overall characteristics. FIG. 5 (a) shows patch w _i in RGB color space and FIG. 5 (b) shows patch w _j In the RGB color space. (a), there is no extreme texture in the region, and (b), there is an extreme texture in the region. As in (b), even though one patch contains extreme pixels, the two adjacent patches have similar characteristics and the degree of fog is comparable. Since the two ellipses Ω _i and Ω _j in (a) and (b) are made by statistical averaging, extreme vectors from sporadic textures that are hardly correlated with the overall distribution are averaged out, It hardly affects the shape of the body. That is, even though the distributions of the pixels at patch w _i and patch w _j due to extreme pixels are different from each other, the two ellipses Ω _i and Ω _j are nearly identical. Therefore, it can be seen that the analysis using the ellipse is stronger than the method of analyzing the pixel value itself.

6 is a flowchart illustrating a fog processing method according to an embodiment.

Referring to FIG. 6, in this embodiment, a dark channel value can be obtained by using an ellipse in an RGB color space rather than a pixel value in an image domain. To this end, the present embodiment may first perform the step of mapping the pixels of the original image to the vectors of the color space, respectively, as shown in FIG. In this embodiment, as shown in FIG. 5, a geometric ellipsoid including a vector cluster region among the vectors may be constructed based on statistical moments in which the vectors are distributed.

Thereafter, the fog processing method according to this embodiment includes a step S600 of obtaining a dark channel value θ ^s _i of a statistical ellipse at a position i, a step S605 of calculating a transmission estimation value based on the dark channel value, And a step (S610) of acquiring or generating a fog removed image by performing fog processing based on the estimated value.

In step S600, in contrast to the DCP and MDCP method of selecting dark channel value from all the pixels in the local area, from this example is the vector in the oval Ω _i as shown in Equation 5 (z∈Ω _i) Select the dark channel value.

Referring to Equation (5), the selected dark channel value may be the smallest element value among the vectors in the ellipse (implemented by the min function in Equation (5)). In this case, it is possible to provide a statistically robust dark channel value while maintaining the DCP principle, and the dark channel value can be calculated immediately from the geometric elliptical structure.

가 성립할 수 있다. z ^* _r, z ^* _g 및 z ^* _b 는 각각 H _r, H _g, H _b 평면에 가장 가까운 벡터이다. 이하 도 7을 참조하여 타원 내의 벡터들 중에서 가장 작은 요소값을 구하는 방법을 설명한다. In step S600, the parameter may be defined as follows before describing a method for obtaining the smallest element value among the vectors in the ellipse. Let each unit color element vector be e _r = [1 0 0] ^T , e _g = [0 1 0] ^T , and e _b = [0 0 1] ^T. Let H _{c be a} plane perpendicular to the unit vector e _c (c ∈ {r, g, b}) and pass through the origin. z _c = e ^T _c · z is the c-element value of z (cε {r, g, b}) and is the distance from z to H _c . Let z ^* _{c be} a vector that minimizes the distance e ^T _c · z ,

Can be established. z ^* _r , z ^* _g, and z ^* _b are vectors closest to the H _r , H _g , and H _b planes, respectively. Hereinafter, a method for obtaining the smallest element value among the vectors in the ellipse will be described with reference to FIG.

Figure 7 shows the geometry of the vectors z ^* _r , z ^* _g and z ^* _b .

Referring to Figure 7 (a), z ^* _r is the vector closest to the BG plane (H _r ). The shadow plane is tangent to the ellipsoidal surface and parallel to H _r . e ^T _r · z ^* _r is the distance from z ^* _r to H _r . Referring to (b) of 7, z ^* _g is the closest to the vector BR plane (H _g). Shadow plane is in contact with the elliptical surface, H _g, it is parallel. e ^T _g · z ^* _g is the distance from z ^* _g to H _g . Referring to Figure 7 (c), z ^* _b is the closest vector to the RG plane (H _b ). The shadow plane is tangent to the elliptical surface, and parallel to H _b . e ^T _b · z ^* _b is the distance from z ^* _b to H _b .

The z ^* _c closest to plane H _c may be on the surface of Ω _i . In this case, considering that the vectors located on the surface of the ellipse satisfy S ( z ) = ( z - μ) ^T Σ ^-1 ( z - μ) = 1, the process of determining the dark channel value in Equation ≪ / RTI >

Here, S ( z ^* _c ) = ( z ^* _c- mu) ^{T? -1} ( z ^* _c- mu) = 1.

As shown in (a) of Figure 7, z ^* _c that must yirueoya the plane H _c and the equilibrium contact with the elliptical surface from the _c ^* z order to be most close to H _c. Thus, the outward normal vector of the ellipse surface at z ^* _c is - e _c . Since the gradient direction of the vector on the ellipse surface is perpendicular to the tangent plane, the outward law vector can be the same as the oblique direction of the ellipse surface. Therefore, the following equation can be established.

Here, the oblique direction can be calculated as shown in Equation (8).

을

로 대체하면, 수학식 9가 성립한다.

of

, Equation (9) holds.

And S ( z ^* _c ) can be expressed as Equation (10).

가 성립한다. 따라서, 수학식 11이 성립한다. However, since? ^T =?,

. Therefore, Equation (11) holds.

To minimum distance from the equation 9, and 11, to the plane of the H _c from the vector Ω _i can be expressed by equation (12).

By the equation (12), a dark channel value based on statistical ellipse θ ^s _i may be defined as shown in Equation 13 by the μ _c and σ _c.

Referring to Equation (13), the dark channel value can be obtained through the average value μ _c , the variance value σ _c and the minimum value (min) of the color elements. Comparing the MDCP and guide filter based DCP to the sorting of all pixel values to determine the dark channel value, the fog processing method according to this embodiment measures only the average and variance values of the minimum color elements, Therefore, it provides excellent performance in terms of computational complexity as described in the experimental results described later.

Referring to FIG. 6 again, in step S605, according to the SEDCP-based fog processing method according to the present embodiment, the transmission estimation value (or transmission rate estimation value) t _i ^S can be defined as shown in equation (14).

를 획득(또는 복원)할 수 있다. Finally, in step S610, based on the transmission estimation value, the fog removed image

(Or restoring) the same.

FIG. 8 is a diagram comparing DCP, MDCP, and SEDCP-based fog processing in an area where extreme pixels exist in the RGB color space.

는

의 안개 제거 처리된 픽셀이고, z^* _c는 타원 표면상에서 최소요소를 가지는 벡터이다.

는

의 안개 제거 처리된 벡터이다. Referring to FIG. 8, (a) is a DCP-based fog process, (b) is a MDCP-based fog process, and (c) is a SEDCP-based fog process. here,

The

And z ^* _c is a vector having the smallest element on the surface of the ellipse.

The

Is a fog removal processed vector.

In Figures 8 (a) and 8 (b), the DCP achieves the fog process insufficiently, while the MDCP overcomes the fog process. However, the fog processing method according to the present embodiment performs the process of completely expanding the foggy pixel vector cluster until the minimum element of the vectors in the ellipse reaches zero, as shown in FIG. 8 (c) Provides a larger contrasting image compared to DCP, and provides images with less color distortion than MDCP.

FIG. 9 shows the dark channel values for images of adjacent patches obtained based on DCP, MDCP, and SEDCP. The degree of fog in areas is nearly equal, and one area contains extreme textures.

Referring to FIG. 9, in the case of the DCP, a lower dark channel value is generated in an area including an extreme texture, which means that it is not stable when an extreme texture is involved. In the case of MDCP, it produces almost the same dark channel value for the two regions, but is higher than the value produced by the DCP. This may indicate that MDCP does not follow the DCP principles well. Finally, in the case of the SEDCP according to the present embodiment, it produces almost the same dark channel value, which is closest to the dark channel value of the DCP for the region not including the extreme texture. Thus, the method according to this embodiment is stable to extreme textures and maintains the DCP principle well.

10 compares the process of removing fog of an original image by DCP, MDCP and SEDCP according to the present embodiment.

Referring to FIG. 10, (a) are original images, (b) are images dehazed by DCP, (c) are images subjected to fog removal by MDCP, As shown in FIG. Images fog removed by DCP are less detailed than MDCP or SEDCP. Images fog removed by MDCP are generally darker, saturated to dark and contain more color-distorted areas. On the other hand, within the fog removal processed image based on SEDCP, well contrasted, saturated or color distorted areas are not well visible.

Hereinafter, a fog processing method for minimizing or eliminating a halo phenomenon that may occur in the presence of a mixed region will be described in detail. The halo phenomenon occurs in the mixed region when the dark channel pixels used to estimate the transmission do not belong to the same region as the region containing the patch center. therefore,

11 is a flowchart showing a fog processing method according to another embodiment.

11, the fog processing method according to the present embodiment includes a step of separating a mixed patch w _i into a set of foreground (FG) pixels w _i ^{f and} a set of background (BG) pixels w _i ^b (S1100), constructing (S1105) the statistical ellipses from pixels that are only in regions covering the patch center to avoid transmission mismatches, calculating (S1110) a dark channel value from the statistical ellipses (S1115) using the calculated dark channel value, and generating a final image by performing fog removal processing on the original image based on the transmission estimation value (S1120).

In steps S1100 and S1105, statistical ellipses Ω _i ^b and Ω _i ^f corresponding to the separated w _i ^f and w _i ^b , respectively, can be configured.

와 공분산 행렬

은 다음의 수학식과 같이 계산될 수 있다. In particular, in step S1105, the center that is used to construct a vector Ω _i ^S

And covariance matrix

Can be calculated by the following equation.

In step S1110, calculating the dark channel value from the statistical ellipses may be based on Equation (17).

이다. here,

to be.

To implement the calculation of the dark channel value, a statistical ellipse Ω _i ^S must first be constructed or determined, for which step S1105 may comprise determining whether the pixel belongs to the same area as the area containing the patch center have. And if it does not belong to the same area, step S1105 may further include positioning (or moving) the selected dark channel pixel and patch center to the same area. Since the actual boundaries of the image between the background and foreground regions may then be ambiguous at this time, a fuzzy decision may be used that divides the pixel regions by examining pixel value variables and pixel positions. Here, the fuzzy determination value can be obtained by measuring the probability P _ij that the pixel at position j is in the region including the patch center at position i. Thus, the statistical ellipse Ω _i ^S can be constructed by filtering the patch image (or image pixels) through P _ij (or masking P _ij on the image pixels).

Various filters may be used to measure (or estimate) P _ij . A filter according to one aspect may assign a larger value to a probability P _ij where the pixel at position j is closer to the patch center at position i and the pixel value at position j is closer to the pixel value at the patch center.

The filter along the other side reduces P _ij smoothly with the distance from i to j when pixel j is in the region containing the patch center and is close to 0 when j is not in the region of the patch center . In a mixed patch, P _ij can be symmetrical with the patch center, so Ω _i ^S = Ω _i can be established.

Figure 12 is a diagram illustrating a method used to construct an ellipse with only pixels in an area at the center of the patch. Figure 12 is a specific implementation of the fog processing method according to Figure 11, and relates to a method for constructing an ellipse using fuzzy division in a mixed patch.

The patch image shown in FIG. 12 is composed of a foreground (FG) and a background (GB) as a mixed patch. The patch center is located in the background. Pixels corresponding to the patch center are indicated by "+" in the RGB color space, pixels located in the background are indicated by "o" in the RGB color space, and pixels located in the foreground are indicated by "x". And an example of a guide filter for calculating a probability map according to an estimated probability is shown. The adopted guide filter adopted in this embodiment gradually reduces P _ij with the distance from i to j when pixel j is in the region including the patch center, Lt; / RTI > can be designed to be close to zero.

The patch image is masked by a probability map with a probability P _ij close to zero in the foreground and output as a fuzzy partitioned image from which a statistical ellipse Ω _i ^S containing the patch center . Thus, irrespective of boundary uncertainty and area geometry, the statistical ellipses Ω _i ^S can be constructed using only those pixels in the background that contain the patch center. This is a clear difference from prior art techniques that evaluate all pixels equally in mixed regions.

Referring back to Figure 11, in step S1110, the dark channel value based on statistical ellipse θ ^s _i may be defined as the equation (13) by μ _c and σ _c.

On the other hand, in step S1115, according to the SEDCP-based fog processing method according to the present embodiment, the transmission estimate (or transmission estimation value) t _i ^S can be defined as Equation (14).

를 획득(또는 복원)할 수 있다. Finally, in step S1120, based on the transmission estimation value, the fog removed image

(Or restoring) the same.

Figure 13 shows various views of the halo phenomenon within the fog removal processed images.

13 (a) is an original image, (b) is an image subjected to fog removal processing based on a guide filter and DCP method, (c) is an image subjected to fog removal processing based on the MDCP method, and d is an image subjected to fog removal processing according to the present embodiment. (d) does not cause a halo phenomenon visually distinguishable from (b) and (c).

Hereinafter, a fog processing method of another embodiment, which improves the speed of calculation in terms of statistics, is published.

Local image signals tend to have a significantly higher correlation, and foggy local images have a higher correlation. Thus, the minimum of the color component averages in Equation 13 is approximately equal to the average of the minimum color components. This is because the minimum value among the minimum elements among the color pixels is almost the same as the minimum elements of the color vectors on the elliptical surface. Therefore, the fog processing method of this embodiment can perform quick calculation of each statistic by the following equation.

Therefore, according to the fog processing method according to the present embodiment, the transmission estimation value can be summarized as the following equation.

로서 정의되고, 최소요소의 평균

이며, 최소요소의 분산(variance)은

이다. 이러한 파라미터들을 기반으로 통계적인 측면에서 계산의 속도를 향상시키는 안개 처리 방법은, i) 각 픽셀에서 최소 컬러 요소를 선택하는 단계(

), ii) 타원 내에서 벡터들의 최소 요소를 결정하기 위한 평균값

및 분산값

을 계산하는 단계, iii) 평균 및 분산을 기반으로 다크 채널값을 구하는 단계(

), iv) 각 픽셀에서의 전송 추정값을 계산하는 단계(t_i ^S=1-αθ_i ^S), 그리고 v) 안개 처리를 수행하는 단계(

)를 포함한다. In equations (19) and (20), the minimum element

And the average of the minimum elements

, And the variance of the minimum factor is

to be. A fog processing method that improves the speed of the calculation on the statistical side based on these parameters comprises the steps of i) selecting a minimum color element at each pixel

), ii) an average value for determining the minimum element of the vectors in the ellipse

And variance value

(Iii) obtaining a dark channel value based on the mean and the variance (

), iv) calculating a transmission estimate for each pixel (t _i ^S = 1-αθ _i ^S ), and v) performing a fog process

).

Hereinafter, the effects according to the present embodiment will be described in detail on the basis of experimental results in terms of the quality of the fogged image and the implementation complexity. In this experiment, the overall fog weighting weight α is fixed to 0.95 and the patch size is set to 15 × 15, which is the parameter value of the fog filter used mostly in image fog processing.

First, in the subjective evaluation, this experiment refers to the mean opinion score (MOS) test technique recommended by ITU-R BT, 500-11 standard as shown in Table 1. Thirty participants participated in the experiment to assess the visual quality of fogged images according to the conventional method and herein. Experiments were trained to compare fog images with original fog images in terms of degree of color distortion, presence of halo, and visibility of detail. The original image and the fogged image pair were identical and were output on adjacent full-size HD monitors. Participants were located at a distance of 85 to 95 from the monitor and selected scores corresponding to {-3, -2, -1, -, 1, 2, 3}. The scores corresponding to 3, 2, and 1 indicate that the quality of the fogged image is very good, good, or slightly better than the quality of the original image. A score of 0 indicates that the quality of the fogged image is the same as the quality of the original image. On the other hand, the scores corresponding to -3, -2, and -1 indicate that the quality of the fogged image is very poor, poor, and slightly poor compared to the quality of the original image.

In order to statistically analyze the synergistic effect achieved by the fogging method according to the present invention, a Student T-Test was performed. The null hypothesis indicates that there is no difference between the conventional method and the fog control method according to the present invention. The reliability was set at 95%, and the rejection region (rr) was 2.093σ _t / N ^1/2 . Where μ _t , σ _t and N represent the mean score, the standard deviation, and the number of participants, respectively. The improvement in performance can be expressed as (μ _t -rr).

Figure 14 compares the fogged images of trees, lakes, forests, and hills.

14 (a) is an original image, (b) is an image subjected to fog removal processing by DCP using a guide filter, (c) is an image subjected to fog removal processing by MDCP, Is an image subjected to fog removal processing by the method according to the present invention.

Table 1 shows MOS scores of fogged images compared to the original image in Fig.

How to deal with fog tree Lake forest Hill DCP + Guide Filter 0.475 1.079 0.921 0.659 MDCP 0.921 1.166 0.147 1.011 Invention 1.036 1.251 1.013 1.131

Referring to Table 1, the fogging method according to the present invention scores 0.085 to 0.866 higher than MDCP and 0.092 to 0.561 higher than DCP + guide filter. In the case of a DCP with a guide filter, there is a distinct drop in detail in areas of extreme textures such as black dots on stairs or stones. The guide filter spreads the halo phenomenon at the object boundary, but the post-orphan phenomenon remains and the fog remains around the branches and leaves. MDCP provides more detailed textures in the strong fog area, but excessive fog processing with MDCP causes saturated pixels as well as darkens the overall image, eventually distorting the object color to dark, reddish color or dark green. MDCP also creates an identifiable halo effect near the corners of branches. However, the fogging method (SEDCP) according to the present invention provides more visible detail, but excludes obvious color distortion and detectable halo phenomenon.

Next, in the objective evaluation, this experiment used a blind or referenceless image space quality evaluation method (BRISQUE). BRISQUE reports 89% accuracy of information for various images, quantifying the impairment of naturalness in the image due to the presence of distortion. BRISQUE uses a score range from 0 to 100, with scores closer to 0 indicating better image quality.

Table 2 shows the fog removal processed images scored by the BRISQUE method according to each method in Fig.

How to deal with fog tree Lake forest Hill DCP + Guide Filter 11.68 14.17 15.09 12.03 MDCP 19.22 25.14 19.15 17.17 Invention 10.01 12.36 14.58 11.18

Referring to Table 2, the score according to the present invention is at least about 4.57 and 0.51, respectively, better than the score by MDCP or guide filter based DCP. Therefore, it is proved that the fogging method according to the present invention is excellent in terms of objective evaluation.

Hereinafter, the conventional method and the fog processing method according to the present invention are compared in terms of the complexity of the operation.

DCP and MDCP sort all pixel values to determine the dark channel value. Moreover, the guide filter used by the DCP also requires a measurement of the mean and variance values of all color elements. However, the fog processing method according to the present invention suffices to measure only the average and variance values of the minimum color elements.

The fast sorting method according to one example requires addition of 1 + L times per pixel and subtraction of 1 + L times. Where L is the number of pixel value levels, e.g., L = 256 for a pixel value of 8 bits. Determining the median value requires 0.5L-1 additional additions and 0.5L additional comparison procedures. The transmission refinement process using a guide filter requires 40 additions, 58 subtractions, 56 multiplications and 6 divisions per pixel.

The fog processing method according to the present invention also uses a guide filter for fuzzy division, wherein the guide filter is characterized by directly weighting mean and variance by concatenating simple mean filters. Therefore, the fog processing method according to the present invention does not require an additional procedure for measuring P _ij . The weighted mean and variance require 10 additions, 10 subtractions, 4 multiplications and 4 divisions per pixel.

Table 3 compares the number of operations for each fog control method.

calculate DCP + GF MDCP Invention Plus 298 385 21 subtract 317 259 24 compare 2 130 2 multiply 57 One 11 division 30 4 12 Square root 0 0 One

Referring to Table 3, the number of executions per operation required by the DCP using the guide filter is significantly greater than the fog processing method according to the present invention. The MDCP has a smaller number of times of multiplication and division compared to the fog processing method according to the present invention, but the major computational burden of the MDCP is compared with addition and subtraction, which greatly exceeds the number of times the method according to the present invention performs. Thus, MDCP is more computationally burdensome than the method according to the present invention.

To measure the actual computational burden using different processors, Table 4 compares the actual clock cycles per pixel.

Processor DCP + GF MDCP Invention Intel Core i5-2500 1088 817 210 AMD Steam Roller A10-7850K 1115 814 209 Intel Core i7-920 998 805 191

Referring to Table 4, the average number of clock cycles required by the present invention is about 29% and 21%, respectively, as compared with MDCP and guide filter based DCP. For the execution time on Intel Core i5, the time required by the present invention to process 10 ⁶ pixels was measured as 115 ms, while the MDCP and strong filter based DCP were measured as 350 ms and 510 ms, respectively. Since the present invention computes the dark channel value directly from the geometric ellipse rather than scanning all the pixels within that area, the computational complexity can be significantly reduced according to the present invention.

This invention has little color distortion and can provide a more detailed image with a non-visible halo phenomenon and much less computational burden. Formulas that directly calculate the dark channel values from the configured geometric ellipses based on statistical measurements can be used. Further, the present invention can maintain the DCP principle, but also ensure statistical robustness. In order to avoid the halo phenomenon, the present invention can use a fast fuzzy division method as a transmission estimation method. In addition, the present invention does not scan all the pixels in the local region and can therefore have significantly lower computational complexity.

15 is a block diagram illustrating a fog treatment apparatus according to an embodiment.

15, a fog processing apparatus 1500 according to the present invention includes a processor 1510, a storage device 1520, an output interface 1530 for outputting a result of the processor 1510, an input An interface 1540, and a communication module 1550 for transmitting and receiving data through a wired network or a wireless network. Here, the processor 1510 may include an application-specific integrated circuit (ASIC), other chipset, logic circuit, and / or data processing device. The storage device 1520 may include a disk drive, a read-only memory (ROM), a random access memory (RAM), a flash memory, a memory card, and a storage medium. The communication module 1550 may be comprised of an RF antenna or a modem, and may in particular comprise a baseband circuit for processing radio signals.

When the fog processing method according to the present disclosure is implemented in software, the fog processing method or algorithms according to the present disclosure may be applied to the processor 1510, the storage device 1520, the output interface 1530, the input interface 1540, May be implemented as a combination of all or a portion of module 1550. [ The fog processing algorithm herein may be stored in storage device 1520 and executed by processor 1510. [ The storage device 1520 may be internal or external to the processor 1510 and may be coupled to the processor 1510 in a variety of well known ways.

Specifically, the input interface 1540 receives the original image and / or image processing related parameters and the processor 1510 receives fog processing from the original image using the fog processing algorithm according to various embodiments described herein Thereby outputting the fog removal processed image.

Specifically, in processing the fog in the unmixed patch, the processor 1510 calculates a dark channel value from the statistical ellipsis as in step S600, calculates a transmission estimate based on the dark channel value as in step S605, The fog processing can be performed based on the transmission estimation value as in S610.

Further, in processing the fog in the mixed patch, the processor 1510 separates the mixed patch into the foreground pixel and the background pixel as in step S1100, and constructs a statistical ellipse based on the pixels in the patch center as in step S1105 A dark channel value is calculated from a statistical ellipse as in step S1110, a transmission estimation value is calculated based on a dark channel value as in step S1115, and fog processing is performed based on a transmission estimation value as in step S1120.

The storage device 1520 stores not only the fog processing algorithm but also the color elements of the vectors that store the input parameters or constitute the geometric ellipse and stores the average value and the variance value of the minimum elements in the geometric ellipse obtained during the execution of the fog processing algorithm And store the dark channel values obtained therefrom, and store the data to the processor 1510 at the request of the processor 1510. [

An apparatus and method for effectively improving the visibility and sharpness of an image by removing fog in an image according to the present invention is useful for restoring a variety of information such as damaged text or objects and visibility of an object or a person in a vehicle or a ship, And can be used as a pre-processing technology for the field of image recognition. Specifically, the apparatus and method for processing fog according to the present invention are mounted on a smart phone, and in a situation in which a rescue operation due to fire smoke is difficult due to a fire disaster, the smoke is removed from the scene image, By helping to see better, the rescue can be done quickly and accurately. This is also the case for vessels sailing on the misty sea. Also, it can be applied to everyday life, for example, the user can acquire fog-free photographs and images even in situations where it is very difficult to ensure the visibility of the scene or the person due to the foggy weather.

Claims (16)

A fog processing method of an image performed by an image processing apparatus,
Mapping pixels of a patch associated with the image to vectors of a color space according to the values, respectively;
Constructing a geometric ellipsoid approximated to a vector cluster region among the vectors based on statistical moments in which the vectors are distributed;
Calculating a dark channel value from the vectors on the surface of the ellipse;
Calculating a transmission estimate based on the dark channel value; And
And obtaining a fog removed image based on the transmission estimate,
The step of constructing the ellipse comprises:
Expanding the ellipse until the minimum element of the vectors in the ellipse reaches zero. The method according to claim 1,
Wherein the dark channel value is a minimum color element value among the vectors in the ellipse. 2. The method of claim 1, wherein calculating the dark channel value comprises:
Characterized in that the method is performed based on a minimum value and a variance value of the color element averages within the ellipse. 2. The method of claim 1, wherein calculating the dark channel value comprises:
The average value and the variance value of the minimum color elements in the ellipse. 2. The method of claim 1, wherein calculating the transmission estimate comprises:
Characterized in that the method is performed based on a product of the dark channel value and a fog processing weight. 2. The method of claim 1, wherein the step of constructing the ellipse comprises:
Characterized in that, if the patch of the image comprises a foreground pixel and a background pixel, the probability that the pixel is in the region containing the center of the patch and the fuzzy segmentation are performed How to. An apparatus for processing fog of an image,
An input interface for receiving an original image;
Mapping pixels of patches related to the original image to vectors of a color space according to the values, respectively, and calculating, based on statistical moments in which the vectors are distributed, wherein the vector estimator is configured to compute an approximated geometric ellipsoid in a vector cluster region, calculate a dark channel value from vectors on the surface of the ellipse, calculate a transmission estimate based on the dark channel value, A processor for obtaining a fog removed image based on the transmission estimate;
A storage device for storing at least one of the vectors of the color space, the dark channel value, and the transmission estimate; And
And an output interface for outputting the fog removed image,
The processor comprising:
And extends the ellipse until the minimum element of the vectors in the ellipse reaches zero. 8. The method of claim 7,
Wherein the dark channel value is a minimum color element value among the vectors in the ellipse. 8. The apparatus of claim 7,
And calculates the dark channel value based on the minimum and variance values of the color element averages within the ellipse. 8. The apparatus of claim 7,
And calculates the dark channel value based on an average value and a variance value of the minimum color elements in the ellipse. 8. The apparatus of claim 7,
And to calculate the transmission estimate based on a product of the dark channel value and a fog processing weight. 8. The apparatus of claim 7,
If the patch of the image includes a foreground pixel and a background pixel, then the ellipse is constructed based on fuzzy segmentation and the probability that the pixel is in the region containing the center of the patch . ≪ / RTI > A fog processing method of an image performed by an image processing apparatus,
Determining whether a first pixel in the image belongs to the same area as a second pixel that requires transmission rate estimation to prevent a halo phenomenon;
Constructing in a color space a geometric ellipse including vectors corresponding to pixels included in the same region;
Calculating a dark channel value from the vectors on the surface of the ellipse;
Calculating a transmission estimate based on the dark channel value; And
And obtaining a fog removed image based on the transmission estimate,
The step of configuring the ellipse in a color space comprises:
Expanding the ellipse until the minimum element of the vectors in the ellipse reaches zero. 14. The method of claim 13,
Wherein a second pixel requiring an estimate of a bit rate is the center pixel of the patch for the image. 15. The method of claim 14, wherein determining whether the region belongs to the same region comprises:
Measuring a probability that the pixel is in an area containing the center of the patch; And
And obtaining a fuzzy decision value based on the probability,
Wherein the ellipse is constructed based on the fuzzy decision value. 14. The method of claim 13, wherein configuring the ellipse in a color space comprises:
Characterized in that it is carried out on the basis of a fuzzy division using a guided filter.

Images