KR20210143031A - Method and system for removing haze in image using polarization image - Google Patents

Abstract

The present invention discloses a system for removing fog in an image. More particularly, the present invention relates to a method and system for removing fog in an image using a polarization image for rapid detection and update of road condition changes for the autonomous driving field. According to an embodiment of the present invention, a plurality of polarization images are obtained by photographing the same subject through a plurality of polarization sensors, and the haze component included in each polarization image is removed based on image characteristics, so as to provide an image having low noise and high contrast.

Description

Method and system for removing fog in images using polarized images

The present invention relates to a system for removing fog in an image, and more particularly, to a method and system for removing fog in an image using a polarized image for rapid detection and update of changes in road conditions for the autonomous driving field.

Recently, a video surveillance system or a vehicle video black box has been used for accident prevention or detection. Also, in the case of autonomous vehicles, research is being conducted to provide lane departure and vehicle collision warnings by detecting images of lanes and vehicles from images acquired from a video camera using computer vision technology.

In order to improve the performance of such an image processing or computer vision application system, a clean input image is required. In particular, when detecting and tracking an object appearing in an image or using edge information of an image, the cleaner the input image, the better the result.

However, in the case of an image taken in an outdoor situation, since the brightness and color reflected from the object are mixed with particles in the air, the original vivid color and brightness contrast cannot be provided. In particular, when there are large particles in the air, such as fog or smoke, it is difficult to accurately obtain the original color and shape of an object.

Registered Patent Publication No. 10-1445577 (Announcement Date: 2014.11.04.)

An object of the present invention is to provide a method and system for removing fog in an image using a polarized image that removes fog so that an object appearing in an image can be more accurately detected and tracked based on an image obtained using a plurality of polarization sensors there is

In order to solve the above problems, a method for removing fog in an image using a polarized image according to a preferred embodiment of the present invention comprises the steps of: (a) receiving a plurality of polarized images obtained by photographing the same subject from a plurality of polarization sensors, ( b) generating one unpolarized image (I) by using the plurality of polarized images, (c) determining the existence of a sky region by applying a vanishing point estimation algorithm to the unpolarized image (I); _{(d) extracting a sky area and estimating a first brightness (A ∞} ) for an object light located at an infinity distance appearing in the unpolarized image (I) using the sky area, (e) the sky area using an area polarized light in the air scattered light also (P _a) and a second brightness third brightness by estimating the (a _p), and removes the degree of polarization (P _a) component of the air scattering light at the second brightness (a _p) (A) estimating and (f) using the first brightness (A _∞ ) and the third brightness (A) to generate a dehaze image (L) from which the haze component is removed from the unpolarized image (I) may include the step of

In addition, in order to solve the above problems, in the method for removing fog in an image using a polarized image according to an embodiment of another aspect of the present invention, (a) input a plurality of polarized images obtained by photographing the same subject from a plurality of polarization sensors receiving, (b) generating one unpolarized image (I) using the plurality of polarized images, (c) applying a vanishing point estimation algorithm to the unpolarized image (I) to determine whether a sky region exists determining, (d) if it is determined that the sky region does not exist in step (c), generating a maximum brightness image (I ^⊥ ) and a minimum brightness image (I ^│ ) using the plurality of polarized images; (e) calculating a gap image (ΔI) according to the difference between the maximum brightness image (I ^⊥ ) and the minimum brightness image (I ^│ ), and using the gap image (ΔI) to appear in the unpolarized image (I) estimating a _{first brightness (A ∞} ) for object light located at an infinity distance of 1 Using the brightness (A _∞ ), the total polarization degree (p), and the gap image (ΔI), generating a dehaze image (L) in which the haze component is removed from the unpolarized image (I) have.

In addition, in order to solve the above problems, the fog removal system in an image using a polarized image according to another embodiment of the present invention receives a plurality of polarized images of the same subject from a plurality of polarization sensors and receives one An image generation unit for generating an unpolarized image (I) of the sky area determination unit for determining the existence of a sky area by applying a vanishing point estimation algorithm to the unpolarized image (I), extracting the sky area, the unpolarized image (I) the second estimate of the first brightness (a _∞) estimation, and the degree of polarization (P _a) of the air scattered light to and the second brightness (a _p) for the emergence object light in the infinite distance to the image (I), and using the brightness (a _p) brightness estimating unit and the first brightness (a _∞) and the third brightness (a) for estimating a third brightness (a) by removing the degree of polarization (p _a) component of the air scattered light from , a dehaze unit generating a dehaze image L from which a haze component is removed from the unpolarized image I.

In addition, in order to solve the above problems, the fog removal system in an image using a polarized image according to another embodiment of the present invention receives a plurality of polarized images of the same subject from a plurality of polarization sensors and receives one An image generator for generating an unpolarized image (I) of the sky area determination unit for determining the existence of a sky area by applying a vanishing point estimation algorithm to the unpolarized image (I), the sky area by the sky area determination unit If it is determined that there is no ^{image synthesizing unit for generating a maximum brightness image (I ⊥} ) and a minimum brightness image (I ^│ ) using the plurality of polarized images, the maximum brightness image (I ^⊥ ) and the minimum brightness image ( calculating a gap image (ΔI) according to a difference between I ^∥) and the first brightness (a _∞) for the object light in the infinite distance to appear in the non-polarized images (I) by using the gap image (ΔI) A brightness estimator for estimating , a polarization degree estimator for estimating the total polarization degree (p) according to the relationship between the atmospheric scattered light and the object light, and the first brightness (A _∞ ), the total polarization degree (p), and the gap image ( A dehaze unit for generating a dehaze image L from which a haze component is removed from the unpolarized image I by using ΔI) may be included.

According to an embodiment of the present invention, a plurality of polarized images are obtained by photographing the same subject through a plurality of polarization sensors, and a haze component included in each polarized image is removed based on image characteristics to achieve low noise and high contrast. There is an effect that can provide an image having a.

1 is a diagram illustrating a method of removing fog in an image using a polarized image according to an embodiment of the present invention.
2 is a diagram illustrating a method of extracting a sky region in a method of removing fog in an image using a polarized image according to an embodiment of the present invention.
3 is a view for explaining a method of setting a vanishing point in a method of removing fog in an image using a polarized image according to an embodiment of the present invention.
4 is a diagram illustrating a system for removing fog in an image using a polarized image according to an embodiment of the present invention.

It should be noted that the technical terms used in the present invention are only used to describe specific embodiments, and are not intended to limit the present invention. In addition, the technical terms used in the present invention should be interpreted as meanings generally understood by those of ordinary skill in the art to which the present invention belongs, unless otherwise defined in particular in the present invention, and excessively comprehensive It should not be construed in the meaning of a human being or in an excessively reduced meaning. In addition, when the technical term used in the present invention is an incorrect technical term that does not accurately express the spirit of the present invention, it should be understood by being replaced with a technical term that can be correctly understood by those skilled in the art. In addition, general terms used in the present invention should be interpreted as defined in advance or according to the context before and after, and should not be interpreted in an excessively reduced meaning.

Also, as used herein, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present invention, terms such as "consisting of" or "comprising" should not be construed as necessarily including all of the various elements or several steps described in the invention, some of which elements or some steps are included. It should be construed that it may not, or may further include additional components or steps.

In addition, terms including ordinal numbers such as first, second, etc. used in the present invention may be used to describe the components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted.

In addition, in the description of the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, it should be noted that the accompanying drawings are only for easy understanding of the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the accompanying drawings.

Meanwhile, the device referred to in the present invention may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. The systems, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a PLU (PLU). It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit, microprocessor, or any other device capable of executing and responding to instructions. In addition, such a server or device may execute an operating system (OS) and one or more software applications running on the operating system. The device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one server or device is sometimes described as being used, one of ordinary skill in the art will recognize that a processing device includes a plurality of processing elements and/or a plurality of types of processing. It can be seen that elements can be included. For example, a server or device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors. In addition, the device can control various sensors directly through an interface or through a separate driver or hardware, software board or module provided separately for sensor control, and can encompass various logical, electrical, and optical transformations for this purpose. have.

In the following description, the term 'fog removal system in an image using a polarized image' may be abbreviated as 'fog removal system' or 'system' for convenience of description.

Hereinafter, a method and system for removing fog in an image using a polarized image according to an embodiment of the present invention will be described with reference to the drawings.

1 is a diagram illustrating a method of removing fog in an image using a polarized image according to an embodiment of the present invention. In the following description, each step execution entity becomes the fog removal system of the present invention and its constituent parts, even if there is no separate description.

Referring to FIG. 1 , in the method for removing fog in an image using a polarized image according to an embodiment of the present invention, receiving a plurality of polarized images obtained by photographing the same subject from a plurality of polarization sensors (S100), the plurality of polarized images Generating one unpolarized image (I) by using the image (S110), applying a vanishing point estimation algorithm to the unpolarized image (I) to determine the existence of a sky area (S120), the sky area _{Extracting and estimating the first brightness (A ∞} ) of the object light located at an infinity distance appearing in the unpolarized image (I) (S130), using the sky area, the polarization degree (P) of the atmospheric scattered light _a), and estimates the second brightness (a _p), and wherein the step of estimating the third brightness (a) to remove the component (p _a) polarized light in the air scattered light from the second brightness (a _p) (S140) and The method may include generating a dehaze image L from which a haze component is removed from the unpolarized image I by using the first brightness A _{∞ and the third brightness A.}

In addition, in the method for removing fog in an image using a polarized image according to another embodiment of the present invention, receiving a plurality of polarized images obtained by photographing the same subject from a plurality of polarization sensors (S100), the plurality of polarized images Generating one unpolarized image (I) by using an image (S110), applying a vanishing point estimation algorithm to the unpolarized image (I) to determine the existence of a sky region (S120), the step S120 If it is determined that the sky region does not exist, generating a maximum brightness image (I ^⊥ ) and a minimum brightness image (I ^│ ) using the plurality of polarized images (S160), the maximum brightness image (I ^⊥ ) and the first brightness to the minimum brightness image (I ^∥) difference calculating a gap image (ΔI), and the object light in the infinite distance to appear in the non-polarized images (I) by using the gap image (ΔI) according to the between Estimating (A _∞ ) (S170), estimating the total polarization degree (p) according to the relationship between the atmospheric scattered light and the object light (S180), and the first brightness (A _∞ ), total polarization degree (p) ) and the gap image ΔI to generate a dehaze image L from which a haze component is removed from the unpolarized image I ( S190 ).

That is, according to an embodiment of the present invention, when a sky region exists or does not exist on a polarized image input from a polarization sensor, different methods are applied to calculate element values for generating a dehaze image, respectively. can

sky realm existence

First, explaining the case where the sky area exists, in the step of receiving a plurality of polarized images of the same subject from a plurality of polarization sensors ( S100 ), the system inputs images of surrounding conditions from a polarization sensor installed in the vehicle In this case, a polarized image having high noise and low contrast with respect to the subject may be input due to fog, dust, or the like. Preferably, four polarized images of the same subject are simultaneously received through an image sensor equipped with polarizers having four different degrees of polarization and dehazed by using them. ) can be derived.

Next, in the step (S110) of generating one unpolarized image (I) by using the plurality of polarized images, four input unpolarized images, that is, 0° polarized image (I ₀ ), 45° polarized image I ₄₅ , 90° polarized image (I ₉₀ ) and 135° polarized image (I ₁₃₅ ) can be used to generate an unpolarized image (I) that can be acquired when photographing with an unpolarized general image sensor.

To this end, Stokes parameters (S ₀ , S ₁ , S ₂ ) representing polarization characteristics may be used, and these Stokes parameters may be calculated according to Equation 1 below.

And, according to Equation 1 above, the unpolarized image may be calculated according to Equation 2 below.

Next, in step S120 of determining whether a sky region exists by applying a vanishing point estimation algorithm to the unpolarized image I, it is determined whether a sky region exists in each polarized image.

As a specific embodiment for this, a vanishing point existing in each polarized image may be found, and an arbitrary predetermined region above the vanishing point may be set as a region of interest (ROI). Such an ROI may be located at a certain height upward from the vanishing point, and its width may be set to 50% or less of the total width of the image.

Here, the vanishing point may be a point where two or more straight lines corresponding to the road, which are objects appearing on the unpolarized image, meet, or a point where virtual extension lines for two or more straight lines meet.

In addition, when the average brightness of pixels included in the ROI is equal to or greater than a predetermined value, it may be determined that the image is an image in which the sky region exists. In addition, when no vanishing point is found or the average brightness of pixels included in the ROI is less than a predetermined value, it is determined as an image in which the sky region does not exist.

As the vanishing point exists, if it is determined that there is a sky area in the polarized image, the sky area is extracted and the infinity distance appearing in the unpolarized image (I) using a sky pixel included in the sky area _{A first brightness (A ∞} ) of the object light located at is estimated ( S130 ). Specifically, in step S130, a sky pixel is extracted within the ROI region. The sky pixel may be calculated based on a well-known Dark Channel Prior (DCP). The DCP image is determined as the minimum value among 3 channel (R, G, B) pixels in the image, and a sky area of a certain size centered on the DCP image can be defined.

_{Thereafter, the first brightness (A ∞} ) of the object light located at an infinite distance appearing in the unpolarized image I may be estimated using the sky region.

Specifically, a plurality of pixels belonging to the sky region are arranged in a bright order, a brightness patch of a certain size is generated centering on a pixel having the maximum brightness among the plurality of aligned pixels, and all pixels belonging to the brightness patch are According to Equation 3 of , it is checked whether the condition with the patch constant δ is satisfied.

Here, the patch constant (δ) may be a constant determined experimentally.

In this step, if the brightness patch satisfies Equation 3 above, the size of the brightness patch is increased by a predetermined unit and the comparison step with the patch constant can be performed again.

_{And, if the condition with the patch constant δ is not satisfied, the first brightness A ∞} can be estimated by applying the brightness patch of the size used in the previously performed step (d2) to Equation 4 below.

Here, S ₀ (m,n) is the brightness of all pixels in the brightness patch, and |Δ| indicates the number of pixels belonging to the brightness patch.

On the other hand, according to the embodiment of the present invention, when the second pixel of the initial size does not satisfy the condition with the patch constant (δ), the brightness patch is centered on the pixel having the next lower brightness than the pixel having the maximum brightness. _{The first brightness (A ∞} ) may be estimated by re-generation and performing a comparison step using the aforementioned patch constant (δ).

Next, using the sky area, estimating the degree of polarization (P _A) and a second brightness (A _p) of the air scattered light, and wherein the polarization of the air scattered light from the second brightness (A _p) is also (P _A) component as the step (S140) for estimating a third brightness (a) is removed, by using the stroke parameters according to equation (1) above can calculate the degree of polarization (P _a) of the air scattering.

FIG polarized light in the air scattered light (P _A) can be calculated according to the following equation (5).

Here, Ω indicates the extracted sky area, and |Ω| indicates the number of pixels in the sky area.

In addition, in the polarized images, the brightness of atmospheric scattered light including a polarization component is reflected, and the brightness (A _p ) of the polarized atmospheric scattered light can be calculated according to Equation 6 below.

Meanwhile, in Equation (6), if the bias factor ε is multiplied by the total polarization degree (p) component (ε×p), the noise of the subsequent dehaze image can be reduced. When the above Equation 6 is summarized by reflecting the bias factor ε, the following Equation 7 is obtained.

By using such a bias factor (ε), noise in the image can be reduced, but at the same time, the contrast is also lowered. Accordingly, in the embodiment of the present invention, both a dehaze image to which a bias factor ε is applied and a dehaze image to which the bias factor ε is not applied are generated in the step of calculating the brightness A of the atmospheric scattered light, and the difference between the two images is calculated to obtain the dehaze image By reflecting in , a dehaze image having low noise and high contrast characteristics can be derived.

The optimal value of the bias factor ε may be determined by experiment.

In addition, in Equation 6 above, p(x,y) is the total brightness of the atmospheric scattered light including the polarization component and the object light by the object appearing in the polarized image, and θ _A is the angle of polarization of the atmospheric scattered light. of the airlight), each of which can be calculated by the following Equation (8).

Then, the brightness (A) of the non-polarized atmosphere scattered light is polarized light in the polarized components included in the air scattered light from the brightness of the polarized air scattering (A _p) air scattering light. Fig removing (Degree of polarization of the airlight P _A) Since it can be calculated, if it is expressed as an equation, the following equation (9) is satisfied.

Next, using the first brightness (A _∞ ) and the third brightness (A), in the step of generating a dehaze image (L) from which the haze component is removed from the unpolarized image (I) (S150), the polarized image By removing the scattering and polarization components by atmospheric scattered light and the polarization component reflected on the object in the , a dehazed image L directly transmitted by the image sensor without the effect of fog is calculated.

The relational expression for the dehaze image L satisfies Equation 10 below.

Here, I(x,y) indicates the brightness of each pixel constituting the unpolarized image I.

sky realm non-existence

Further, the it is determined if not the sky area is present at the S120 step, not to the sky area is present at the S120 step, the maximum brightness image (I ^⊥) and the minimum brightness image (I ^∥) using the plurality of polarized images generated (S160).

Specifically, the unpolarized image (I) and the Stokes parameters (S ₀ , S ₁ , S ₂ ) can be calculated using the four polarized images, and the calculated data is used for the same subject as a polarized image sensor It is possible to generate a maximum brightness image (I ^⊥ ) and a minimum brightness image (I ^│ ) that can be obtained when shooting at an arbitrary polarization angle.

The above-described maximum brightness image (I ^⊥ ) and minimum brightness image (I ^│ ) can be calculated by Equation 11 below.

Next, the infinite distance to appear in the maximum brightness image (I ^⊥) minimum brightness image (I ^∥) a sensitive gap image (ΔI) non-polarized images (I) by using the generated, and the gap image (ΔI) from the A first brightness (A _∞ ) of the located object light is estimated ( S170 ).

The above-mentioned gap image (ΔI) can be calculated by calculating the difference value of the pixel of the maximum brightness image (I ^⊥) minimum brightness image (I ^∥) at the same position as each pixel of the.

This gap image (ΔI) is used to estimate _{the first brightness (A ∞} ) of the object light located at an infinity distance appearing in the unpolarized image (I). In detail, a plurality of pixels belonging to the brightest upper 0.1% to 0.5% among pixels for each channel in the gap image ΔI are selected as a candidate region, and a plurality of pixels at the same position as the candidate region in the unpolarized image I are selected. Among the pixels, the virtual bright upper 0.1% to 0.5% is selected, and the average brightness of the plurality of selected pixels is determined as _{the first brightness (A ∞ ).}

Next, the total polarization degree p is calculated according to the correlation between atmospheric scattered light and object light, which is reflected light by an object existing at a virtual infinity distance (S180).

Since there is practically no correlation between the brightness caused by atmospheric scattered light and the brightness of the light reflected from the object, the correlation between the two variables should be 0 when expressed as a covariance (Cov).

However, in an actual situation, since the covariance value does not become 0 due to noise and measurement error, it is possible to estimate the covariance variable through the variable value when the value becomes the minimum.

That is, the covariance equation between the brightness of the atmospheric scattered light and the object light is expressed as Equation 12 below.

는 대기 산란광,

는 객체광을 가리킨다.here,

is atmospheric scattered light,

indicates object light.

Accordingly, the total polarization degree p may be determined as a p value when the value expressed by Equation 13 below has a minimum value.

Next, using the first brightness (A _∞ ), the total polarization degree (p), and the gap image (ΔI), a dehaze image (L) in which the haze component is removed from the unpolarized image (I) is generated ( S190).

The dehaze image L can be calculated by the following Equation 14.

On the other hand, according to the embodiment of the present invention, it is possible to improve the quality of the image by applying a bias factor (ε) for removing the noise component due to the atmospheric scattered light (p) included in the unpolarized image (I). As the bias factor ε is applied to Equation 14 above, the formula for calculating the dehaze image L satisfies Equation 15 below.

According to the method described above, according to the method for removing fog in an image using a polarized image according to an embodiment of the present invention, low noise and high noise level and high noise are eliminated by removing scattering and polarization components by atmospheric scattered light included in an image using a plurality of polarized images. An image with a contrast characteristic can be created.

Hereinafter, a method of extracting a sky region in a method for removing fog in an image using a polarized image according to an embodiment of the present invention will be described in detail with reference to the drawings.

2 is a diagram illustrating a method of extracting a sky region in a method for removing fog in an image using a polarized image according to an embodiment of the present invention, and FIG. 3 is a method for removing fog from an image using a polarized image according to an embodiment of the present invention It is a diagram for explaining a method of setting a vanishing point in .

2 and 3 , the method of extracting the sky region in the embodiment of the present invention describes in detail the method of extracting the sky region using the DCP image described above. generating a DCP image by selecting a minimum value among each of the three color channels of a pixel (S121), dividing the entire DCP image into a plurality of DCP patches having a predetermined size (S122), Sorting the DCP patches in order of brightness based on the pixel value with a low value, selecting one or more DCP patches corresponding to the brightest upper 0.1% to 0.5% (S123), and selecting the selected DCP patch as the sky area (S124) may be included.

The step (S121) of generating a DCP image by selecting the minimum value among the three color channels of a plurality of pixels included in the ROI above the vanishing point (VP) is a vanishing point (VP) for extracting the sky region from the acquired polarization images. is the step to find When the image sensor is installed to photograph the front of the vehicle in the driving direction, it is present on the upper region of the vanishing point VP in the front on the road on which the sky vehicle travels in the photographed image.

The vanishing point VP relates to a geometric characteristic in which a plurality of parallel straight lines PL existing on the same plane on the image meet at one point. A plurality of vanishing points VP may exist, and the vanishing point VP When connected to each other, it becomes a vanishing line (VL). That is, any one point on the vanishing line VL becomes the vanishing point VP.

Accordingly, it is possible to estimate a certain range above the vanishing point VP as the sky area. As such a vanishing point determination method, a vanishing point is a point where two or more straight lines PL corresponding to a road, which are objects appearing on the plurality of polarized images, meet, or a point where virtual extension lines for two or more straight lines PL meet. (VP) may be determined, but is not limited by a specific method.

Next, if the vanishing point VP exists, in the step S112 of setting an upper region of the vanishing point as a region of interest (ROI), a rectangular region is set as the ROI with a constant size in the upper direction of the identified vanishing point VP. will do The ROI may be set as an area above the vanishing point VP, in which the average brightness of pixels is equal to or greater than a predetermined value.

Next, the entire DCP image is divided into a plurality of DCP patches having a predetermined size (S122). In detail, a DCP image is generated according to a dark channel prior (DCP), and a sky area is set based on this DCP image. According to , in general, among the three channel (R, G, B) pixels of a certain area in the image taken by the camera, at least one is 0 with high probability, and the DCP of the image distorted by fog, etc. α is increased, and thus, a dehazed image can be obtained by estimating α and removing α from the entire image.

Next, the DCP patches are arranged in the order of brightness based on the pixel value with the lowest value within each DCP patch, and one or more DCP patches corresponding to the brightest upper 0.1% to 0.5% are selected (S123).

Then, the sky area is determined by selecting the selected DCP patch as the sky area.

Hereinafter, a system for performing a method of removing fog in an image using a polarized image according to an embodiment of the present invention will be described in detail with reference to the drawings.

4 is a diagram illustrating a system for removing fog in an image using a polarized image according to an embodiment of the present invention. In the following description, the fog removal system and each component included therein may be implemented as a computer program executable by a known microprocessor, and may be recorded on a readable and writable recording medium and mounted on a computer device.

Referring to FIG. 4 , the fog removal system 100 according to an embodiment of the present invention receives a plurality of polarized images obtained by photographing the same subject from a plurality of polarization sensors and generates one unpolarized image (I). The generator 110, the sky area determination unit 120 that determines whether a sky area exists by applying a vanishing point estimation algorithm to the unpolarized image (I), extracts the sky area, and adds it to the unpolarized image (I) estimate the first brightness (a _∞) for the object light in the appearing infinite distance, and the polarization of the air scattered light also (p _a) and a second brightness (a _p) and the second brightness (a _p) estimated, and the using the degree of polarization (P _a), a third brightness (a) brightness estimating unit 130, the first brightness (a _∞) and the third brightness (a) of estimating by removing the components of the air scattering from the The first dehaze unit 140 for generating the dehaze image L from which the haze component is removed from the unpolarized image I, and when it is determined that the sky area does not exist by the sky area determination unit, the plurality of polarized light gap in accordance with the image to the difference between the maximum brightness image (I ^⊥) and the minimum brightness image (I ^∥), the image combination unit 150 to generate the maximum brightness image (I ^⊥) and the minimum brightness image (I ^∥) A brightness estimator 160 _{for calculating an image ΔI and estimating a first brightness A ∞} for an object light located at an infinity distance appearing in the unpolarized image I using the gap image ΔI ), the polarization degree estimator 170 for estimating the total polarization degree (p) according to the relationship between the atmospheric scattered light and the object light, and the first brightness (A _∞ ), the total polarization degree (p), and the gap image (ΔI) may include a second dehaze unit 180 that generates a dehaze image L from which a haze component is removed from the unpolarized image I by using .

The image generator 110 may generate one unpolarized image I by receiving polarized images having four different polarization angles from a plurality of polarization sensors mounted on the vehicle, for example, four polarization sensors.

The sky area determination unit 120 may determine whether the sky area exists based on the vanishing point using the unpolarized image I provided from the image input unit 110 . The vanishing point may be determined by an object such as a road or a building included in the polarized image, and in the case of an image in which the vanishing point does not exist, it may be determined that there is no sky area.

In addition, if a vanishing point exists in the polarized image, the sky area determination unit 120 sets a rectangular area of a certain range upward based on the vanishing point as the ROI, and the existence of the sky area based on the brightness averages of pixels in the ROI can determine whether

The first brightness estimator 130 extracts the sky area when the sky area exists by the sky area determination unit 120, and based on Equations 1 to 9 described above, the polarization degree P _{A of the atmospheric scattered light.} ), the first brightness A _∞ , the second brightness A _p , and the third brightness A may be calculated.

The first dehaze unit 140 _{generates a dehaze image L from which the haze component is removed using the first brightness A ∞} , the third brightness A, and the brightness I of the unpolarized image. can When the sky area exists, the first dehaze unit 140 may calculate the dehaze image L based on Equation 10 described above.

In addition, when the sky area does not exist (non-sky), the image synthesizing unit 150 may generate a maximum brightness image (I ^⊥ ) and a minimum brightness image (I ^│ ) using Equation 11 described above. .

Second brightness estimating unit 160 includes a maximum brightness image (I ^⊥) and the minimum brightness image (I ^∥) calculating a gap image (ΔI) according to the difference, and by using a gap image (ΔI), the non-polarized image between the ( _{The first brightness (A ∞} ) of the object light located at an infinite distance appearing in I) can be estimated.

The polarization degree estimator 170 may estimate the total polarization degree p through covariance as shown in Equation 13 above according to the relationship between the atmospheric scattered light and the object light.

The second dehaze unit 180 uses the total polarization degree (p), the first brightness (A _∞ ), the unpolarized image (I), and the gap image (ΔI) to remove the haze component (L) can create The second dehaze unit 180 may calculate the dehaze image L based on Equation 15 described above.

In addition, although not shown, the fog removal system 100 of the present invention dehazes when the third brightness A is calculated by the first brightness estimator 120 or dehazes by the second dehaze unit 180 . When the image L is generated, the dehaze image L may be converted to have low noise and low contrast characteristics by applying a bias factor ε.

In the above, preferred embodiments according to the present invention have been described in detail, but the present invention is not limited to the above-described embodiments, and the technical field to which the present invention belongs without departing from the gist of the present invention appended in the claims Anyone with ordinary skill in the art will be able to implement various modifications.

Ap: Brightness of polarized atmospheric scattered light (first brightness)
A: Brightness of atmospheric scattered light (second brightness)
A _∞ : Brightness due to atmospheric scattered light for an object located at infinity (third brightness)
p: degree of polarization P _A : degree of polarization of atmospheric scattered light
x,y : pixel coordinates in polarized image I ₀ : 0° polarized image
I ₄₅ : 45° polarized image I ₉₀ : 90° polarized image
I ₁₃₅ : 135° polarized image S ₀ , S ₁ , S ₂ , S ₃ : Stokes parameter
L: Dehaze image 100: Fog removal system
110: image generating unit 120: sky area determining unit
130: first brightness estimator 140: first dehaze unit
150: image synthesizing unit 160: second brightness estimating unit
170: polarization degree estimating unit 180: second dehaze unit

Claims (22)

A method of removing fog in an image by a fog removal system, the method comprising:
(a) receiving a plurality of polarized images obtained by photographing the same subject from a plurality of polarization sensors;
(b) generating one unpolarized image (I) by using the plurality of polarized images;
(c) determining whether a sky region exists by applying a vanishing point estimation algorithm to the unpolarized image (I);
(d) extracting a sky region and estimating a _{first brightness (A ∞} ) of an object light located at an infinity distance appearing in the unpolarized image (I);
(e) using the sky area, even the polarization of the air scattered light (P _A) and a second brightness (A _p) estimated, and the second brightness (A _p) the degree of polarization (P _A) of the air scattering from the elements a estimating the third brightness (A) by removing and
(f) generating a dehaze image (L) from which a haze component is removed from the unpolarized image (I) by using the first brightness (A _{∞ ) and the third brightness (A)}
A method of removing fog in an image using a polarized image comprising a. The method of claim 1,
The plurality of polarized images are 0 ° polarized image (I ₀ ), 45 ° polarized image I ₄₅ , 90 ° polarized image (I ₉₀ ) and 135 ° polarized image (I ₁₃₅ ),
_{Stokes parameters (S 0} , S ₁ , S ₂ , S ₃ ) for representing the polarization characteristics of each polarized image are,

A method of removing fog in an image using a polarized image calculated as . 3. The method of claim 2,
The unpolarized image (I) is

A method of removing fog in an image using a polarized image calculated as . 3. The method of claim 2,
The step (c) is,
(b1) if there is a vanishing point in the unpolarized image, setting a predetermined area above the vanishing point as an ROI; and
(b2) calculating the average brightness of a plurality of pixels existing in the ROI and determining that the sky region exists if it is greater than or equal to a reference value
A method of removing fog in an image using a polarized image comprising a. 5. The method of claim 4,
The vanishing point is
A method of removing fog in an image using a polarized image, which is a point where two or more straight lines corresponding to a road, which are objects appearing on the unpolarized image, meet or a point where virtual extension lines for two or more straight lines meet. 5. The method of claim 4,
The step (b2) is,
generating a DCP image by selecting a minimum value from among three color channels of a plurality of pixels included in the ROI;
dividing the entire DCP image into a plurality of DCP patches having a predetermined size;
selecting one or more DCP patches corresponding to the brightest upper 0.1% to 0.5% based on the minimum values of pixels included in each DCP patch; and
selecting the selected DCP patch as the sky area
A method of removing fog in an image using a polarized image comprising a. 7. The method of claim 6,
Step (d) is,
(d1) arranging a plurality of pixels belonging to the sky region in a bright order;
(d2) Among the plurality of aligned pixels, a brightness patch of a certain size is generated centering on a pixel having the maximum brightness, and all pixels belonging to the brightness patch are expressed by the following equation,

checking whether a condition with the patch constant (δ) is satisfied; and
(d3) If the condition in step (d2) is satisfied, the size of the brightness patch is increased by a certain unit and step (d2) is performed again. If the condition with the patch constant (δ) is not satisfied, the previously performed The brightness patch of the size used in step (d2) is expressed by the following equation,

estimating the first brightness (A _∞ ) by applying
Within the mist removal image by using the polarized image including a method (where, δ is the brightness patch constant, S ₀ (m, n) is brightness, brightness, within all the pixel patches | Δ | is the number of pixels belonging to the brightness patch). 8. The method of claim 7,
The step (d2) is,
If the second pixel of the initial size does not satisfy the condition with the patch constant (δ), the brightness patch is regenerated with the pixel having the next lower brightness than the pixel having the maximum brightness as the center, and the patch constant ( Checking whether the condition with δ) is satisfied
A method of removing fog in an image using a polarized image comprising a. 3. The method of claim 2,
And, the degree of polarization (P _A) of the air are scattered light,
the following formula,

A method of removing fog in an image using a polarized image calculated as 3. The method of claim 2,
The second brightness (A _p ) is the following equation,

is calculated as
The third brightness (A) is the following equation,

A method of removing fog in an image using a polarized image calculated as (where p(x,y) is the total polarization degree of atmospheric scattered light and object light, and θ _A is the polarization angle of atmospheric scattered light). 11. The method of claim 10,
The dehaze image (L) is
the following formula,

A method of removing fog in an image using a polarized image calculated as (where I(x,y) is the brightness of each pixel in the unpolarized image). 11. The method of claim 10,
As a bias factor (ε) for removing a noise component caused by atmospheric scattered light (p) included in the unpolarized image (I) is applied, the second brightness (A _p ) is
the following formula,

A method of removing fog in an image using a polarized image calculated as . A method of removing fog in an image by a fog removal system, the method comprising:
(a) receiving a plurality of polarized images obtained by photographing the same subject from a plurality of polarization sensors;
(b) generating one unpolarized image (I) by using the plurality of polarized images;
(c) determining whether a sky region exists by applying a vanishing point estimation algorithm to the unpolarized image (I);
^{(d) generating a maximum brightness image (I ⊥} ) and a minimum brightness image (I ^│ ) using the plurality of polarized images if the sky region does not exist in step (c);
(e) calculating a gap image (ΔI) according to the difference between the maximum brightness image (I ^⊥ ) and the minimum brightness image (I ^│ ), and using the gap image (ΔI) to appear in the unpolarized image (I) estimating a _{first brightness (A ∞} ) of an object light located at an infinity distance;
(f) estimating a total polarization degree (p) according to the relationship between the atmospheric scattered light and the object light; and
(g) using the first brightness (A _∞ ), the total polarization degree (p), and the gap image (ΔI) to generate a dehaze image (L) in which the haze component is removed from the unpolarized image (I) step
A method of removing fog in an image using a polarized image comprising a. 14. The method of claim 13,
The plurality of polarized images are 0 ° polarized image (I ₀ ), 45 ° polarized image I ₄₅ , 90 ° polarized image (I ₉₀ ) and 135 ° polarized image (I ₁₃₅ ),
_{Stokes parameters (S 0} , S ₁ , S ₂ , S ₃ ) for representing the polarization characteristics of each polarized image are,

A method of removing fog in an image using a polarized image calculated as . 15. The method of claim 14,
The unpolarized image (I) is

A method of removing fog in an image using a polarized image calculated as . 16. The method of claim 15,
The maximum brightness image (I ^⊥ ) and the minimum brightness image (I ^│ ) are,
Each of the following formulas,

A method of removing fog in an image using a polarized image calculated as . 17. The method of claim 16,
Step (e) is,
Generating an image of the gap (ΔI) obtained by subtracting the minimum brightness image (I ^∥) in the maximum brightness image (I ^⊥);
selecting, as a candidate region, a plurality of pixels belonging to the brightest upper 0.1% to 0.5% among pixels for each channel in the gap image ΔI; and
From among a plurality of pixels in the same position as the candidate region in the unpolarized image (I), virtual bright upper 0.1% to 0.5% are selected, and the average brightness of the selected plurality of pixels is set as the first brightness (A _∞ ) step to decide
A method of removing fog in an image using a polarized image comprising a. 18. The method of claim 17,
The step (f) is,
According to the correlation between the atmospheric scattered light and the object light, the total polarization degree (p) is the following equation,

A method of removing fog in an image using a polarized image, which is a step of determining a value when it has this minimum value. 19. The method of claim 18,
The dehaze image (L) is,
the following formula,

A method of removing fog in an image using a polarized image calculated as . 20. The method of claim 19,
As the bias factor (ε) for removing the noise component by the atmospheric scattered light (p) included in the unpolarized image (I) is applied, the dehaze image (L) is,

A method of removing fog in an image using a polarized image calculated as . an image generator configured to receive a plurality of polarized images obtained by photographing the same subject from a plurality of polarization sensors and generate one unpolarized image (I);
a sky area determination unit for determining whether a sky area exists by applying a vanishing point estimation algorithm to the unpolarized image (I);
Extracting a sky area, and estimate the first brightness (A _∞) for the object light in the infinite distance to appear in the non-polarized images (I), and the polarized light in the air scattered light also (P _A) and a second brightness (A _a brightness estimator for estimating p ) and estimating a third brightness A by removing the _{polarization degree P A} component of the atmospheric scattered light from _{the second brightness A p ;} and
A first dehaze unit generating a dehaze image L from which a haze component is removed from the unpolarized image I by using the first brightness A _{∞ and the third brightness A}
In-image fog removal system using a polarized image comprising a. 22. The method of claim 21,
If it is determined the sky region does not exist in the sky area determining unit, an image synthesizing unit for generating a maximum brightness image (I ^⊥) and the minimum brightness image (I ^∥) using the plurality of the polarization image;
A gap image (ΔI) is calculated according to the difference between the maximum brightness image (I ^⊥ ) and the minimum brightness image (I ^│ ), and the infinity distance appearing in the unpolarized image (I) using the gap image (ΔI) a brightness estimator for estimating a _{first brightness (A ∞} ) of an object light located in ;
a polarization degree estimator for estimating a total polarization degree (p) according to the relationship between the atmospheric scattered light and the object light; and
Using the first brightness (A _∞ ), the total polarization degree (p), and the gap image (ΔI), a second dehaze image (L) in which the haze component is removed from the unpolarized image (I) is generated. haze department
Fog removal system in the image using a polarized image further comprising a.

Images