KR102151750B1 - Apparatus for Removing Fog and Improving Visibility in Unit Frame Image and Driving Method Thereof - Google Patents

Abstract

The present invention relates to a device for removing fog and improving visibility for a single image, and a driving method of the device thereof. According to an embodiment of the present invention, the device for removing fog and improving visibility for a single image comprises: an adaptive constraint condition unit for generating an adaptive constraint condition which is adaptively changed in accordance with a change in a pixel by using estimated atmospheric light and a pixel of a preprocessed fog image; a transfer map estimating unit for estimating a transfer map related to the subtraction amount of fog by applying a depth map which estimates the correlation between the saturation and brightness of the fog image having background noise removed therefrom to the pre-generated adaptive constraint condition unit; and an image restoring unit for restoring a fog-removed image by applying the weight to the fog image based on the pre-estimated transfer map when the fog is removed from the fog image having the background noise removed therefrom.

Description

Apparatus for Removing Fog and Improving Visibility in Unit Frame Image and Driving Method Thereof}

The present invention relates to an apparatus for removing fog and improving visibility of a single image, and a method of driving the same, and more particularly, to a method of removing fog contained in the atmosphere in an image and removing the fog. The present invention relates to a device for removing fog and improving visibility for a single image, which improves visibility for a region of visibility impairment, and enables fast processing from a hardware design point of view based on a linear model, and a driving method thereof.

Recently, in the field of determining unnatural situations through images, such as an autonomous driving system, an image with high visibility has been developed to accurately recognize surrounding objects even in sudden weather changes such as fog, yellow dust, and fine dust. Is required. Methods of removing fog to improve visibility include a method of using multiple images to remove unwanted elements, and a method of removing fog using a single image.

The first is a method to improve visibility by using multiple images. This method is a method of removing fog by extracting information of fog using images captured in multiple environments at the same location, and improving visibility of images. If multiple images are used, the performance is superior to that of a single image, but it is difficult to capture images of multiple environments at the same location, so the utilization is low.

The second method is to improve visibility using DCP (Dark Channel Prior) in a single image created by Kaiming He, Jian Sun, and Xiaoou Tang. This method is a method of removing fog as a result of observation that, through many experiments, pixels with high color clarity have very low and near zero pixels among color channel values. However, this method has a disadvantage that it does not handle sky areas well and requires a large amount of computation.

Lastly, this is a visibility enhancement method using CAP (Color Attenuation Prior) of Qingsong Zhu, Jiaming Mai, and Ling Shao. This method creates a linear model for the depth map of an image using the difference between brightness and saturation, and removes fog by obtaining the parameters of the linear model through machine learning. This method effectively removes fog and improves visibility, but there is a lot of room for improvement in terms of quality, such as complex computational volume, fixed constraints, and color distortion.

Korean Patent Publication No. 10-1788660 (2017.10.20. Announcement) Fog removal device and method in a single image Korean Patent Publication No. 10-1568971 (announcement on November 13, 2015) Method and system for removing fog from images and videos Korean Patent Publication No. 10-2014-0142381 (published on Dec. 12, 2014) Method and apparatus for removing fog in a single image U.S. Patent Publication No.9754356 (registered on September 5, 2017) Method and system for processing an input image based on a guidance image and weights determined thereform

"Single Image Haze Removal Using Dark Channel Prior" IEEE Transactions on Pattern Analysis and Machine Intelligence (2011), K He, J Sun, X Tang et al., pp. 2341-2353 "A Fast Single Image Haze Removal Algorithm Using Color Attenuation Prior" IEEE Transactions on Image Processing (2015), Q Zhu, J Mai, L Shao et al., pp. 3522-3533 "Improved Dark Channel Prior Fog Removal Algorithm Using Correction of Mispredicted Fog Volume" Journal of the Korean Society of Broadcast Engineers, Vol. 21, No. 5, Kim Jong-Hyeon and Cha Hyung-myung, September 2016

The embodiment of the present invention removes fog contained in the atmosphere in an image, improves visibility for areas of visibility loss that may occur in the process of removing fog, and also improves visibility from a hardware design perspective based on a linear model. An object of the present invention is to provide an apparatus for removing fog and improving visibility of a single image capable of fast processing, and a method of driving the apparatus.

An apparatus for removing fog and improving visibility for a single image according to an embodiment of the present invention is an adaptive constraint that adaptively changes according to a change of a pixel using an estimated atmospheric light and a pixel of a preprocessed fog image. An adaptive constraint unit that creates a condition, a depth map that estimates the correlation between the saturation and brightness of the haze image from which the background noise has been removed, is applied to the generated adaptive constraint, and transfers related to the amount of haze subtracted. And a transmission map estimating unit for estimating a map, and an image restoration unit for restoring the fog image by applying a weight to the fog image based on the estimated transmission map when the fog image from which the background noise is removed is removed.

The apparatus includes a preprocessor for performing preprocessing by adjusting a white balance of an input fog image, a background noise removal unit for removing background noise by applying a low pass filter to a saturation channel of the preprocessed fog image, and the preprocessed fog An atmospheric light estimating unit for estimating atmospheric light using a luminance channel of the image, and a depth map estimating unit for estimating the depth map, wherein the depth map estimating unit uses a training image having an equally divided uniform distribution Thus, the depth map may be estimated by further calculating a parameter estimated by machine learning with the saturation and the brightness.

The adaptive constraint unit may prevent black pixels generated when fog is removed from the fog image by using a constraint using the average and standard deviation of the pixels.

The image restoration unit may apply a weight when removing the fog in a dark area from the fog image from which the background noise has been removed, so that channel values of red (R), green (G), and blue (B) may be consistently operated.

The image restoration unit may apply a weight when removing the fog in the dark area so that channel values of red (R), green (G), and blue (B) are not consistently decreased or changed.

In addition, the method of driving an apparatus for removing fog and improving visibility for a single image according to an embodiment of the present invention provides for removing fog and improving visibility for a single image including an adaptive constraint conditioner, a transfer map estimation unit, and an image restoration unit. A method of driving an apparatus for driving a preprocessed fog image, comprising: generating an adaptive constraint that changes adaptively according to a pixel change using an estimated atmospheric light and a pixel of a pre-processed fog image, in the adaptive constraint conditioner, and the background noise is removed Estimating a transfer map related to the amount of fog subtracted from the transfer map estimation unit by applying a depth map that estimates the correlation between the saturation and brightness of the image to the generated adaptive constraint, and a fog image from which background noise has been removed And restoring the fog removal image by the image restoration unit by applying a weight to the fog image based on the estimated transmission map when the fog is removed.

In the driving method, a preprocessing unit pre-processing by adjusting a white balance of an input fog image, a background noise removing unit removing background noise by applying a low-pass filter to a saturation channel of the pre-processed fog image. A light estimating unit, estimating atmospheric light using a luminance channel of the pre-processed fog image, and a depth map estimating unit, further comprising estimating the depth map, and the step of estimating the depth map may include: The depth map may be estimated by further calculating parameters estimated by machine learning with the saturation and brightness using a training image having a discounted uniform distribution.

In the generating step, a black pixel generated when the fog is removed from the fog image may be prevented by using a constraint using the average and standard deviation of the pixels.

In the restoring, a weight is applied to the fog image from which the background noise is removed when the fog is removed from a dark area, so that channel values of red (R), green (G), and blue (B) may be consistently operated.

In the restoring, a weight is given when the fog is removed in the dark area so that channel values of red (R), green (G), and blue (B) are not consistently decreased or changed.

According to an exemplary embodiment of the present invention, it is possible to improve visibility of a region of visibility deterioration that occurs during a process of reconstructing a fog image. In other words, there is an effect of improving visibility while restoring a fog-free image by removing fog through adaptive constraints, low-pass filters, and adaptive weighting elements for problems such as background noise and color distortion that occur while removing fog. .

In addition, according to an embodiment of the present invention, it will be easy to develop a device capable of applying a high-resolution image of about 4K (Ultra High Definition) to a system such as an autonomous vehicle in real time. In other words, by using only linear models of equations, there is an effect that it can operate quickly when implementing hardware.

1 is a view showing an example of an autonomous vehicle system to which an image processing device according to an embodiment of the present invention is applied;
2 is a block diagram illustrating a detailed structure of an apparatus for removing fog and improving visibility for a single image according to an embodiment of the present invention;
3 to 5 are diagrams for explaining color distortion that occurs when fog is removed;
6 is a view showing a comparison between an input image in which fog exists and an output image to which a technology according to an embodiment of the present invention is applied;
7 is a flowchart of a fog image processing process according to an embodiment of the present invention, and
8 is a flowchart of a process of processing a fog image according to another embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.

As shown in FIG. 1, the autonomous vehicle system 90 according to the embodiment of the present invention includes a photographing device 100 and a part or all of the autonomous vehicle 110.

Here, "including some or all" means that some or all of the photographing device 100 can be integrated and configured in the autonomous vehicle 110, and all in order to help a sufficient understanding of the invention It is described as including.

Prior to a detailed description, the autonomous vehicle system 90 according to an embodiment of the present invention may be replaced with a traffic control system or a security control system. In other words, the traffic control system or the security control system may be composed of a server operated by a traffic control center or a security control center, and a photographing device such as CCTV or IP camera interlocking with the server, and according to an embodiment of the present invention. An image processing device may be mounted on a server and used. For example, although the autonomous vehicle 110 will operate in the fog, even when a traffic accident occurs, an image of the fog removed may be used to correct the situation. Accordingly, the traffic control server operated by the traffic control sensor may include an image processing apparatus according to an embodiment of the present invention. As described above, the image processing apparatus according to the embodiment of the present invention can be applied to various devices in various fields, but in the embodiment of the present invention, for convenience of explanation, the autonomous vehicle 110 is described as an example, and image processing It is only described as being made in the autonomous vehicle 110, it will not be specifically limited to any one form.

The photographing apparatus 100 of FIG. 1 includes at least one camera, and may include a camera capable of pan, tilt, and zoom operations like a PTZ camera. The photographing device 100 may be provided in the autonomous vehicle 110, for example, to photograph the driving direction of the autonomous vehicle 110 and provide the photographed image to the autonomous vehicle 110.

The autonomous vehicle 110 removes the fog when there is fog or the like in the image received from the photographing device 100 to generate an image with improved visibility, and performs image analysis based on the image with improved visibility, based on the analysis result. You can perform various actions with. Of course, the captured image is provided in the form of a moving picture, so that tens to hundreds of unit frame images may be received per second. Since the autonomous vehicle 110 recognizes surrounding objects through the analysis of the captured image, it is clear that an accurate analysis of the received captured image must be performed for accurate recognition. In this respect, the image processing apparatus according to the embodiment of the present invention may be applied to the autonomous vehicle 110 and be usefully utilized.

Of course, the autonomous vehicle 110 may include various components such as a front wheel, a rear wheel, a steering wheel, and a sensor, but in FIG. 1, only the driving unit is illustrated in terms of image processing. In general, the driving unit of a vehicle is referred to as an electronic control device or an electronic control unit (ECU). Accordingly, the communication interface unit 111, the control unit 113, and the fog image processing unit (or image processing unit) 115 of FIG. 1 may mean an ECU or a part thereof.

The communication interface unit 111 is controlled by the control unit 113 and communicates with the photographing apparatus 100. For example, when a vehicle accident occurs in front, when the vehicle bypasses the corresponding place, the photographing apparatus 100 photographs the direction to be bypassed. Accordingly, the photographing apparatus 100 may perform communication with the communication interface unit 111 and perform various operations such as pan, tilt, and zoom under the control of the controller 113.

The control unit 113 is in charge of overall control operations of the communication interface unit 111 and the fog image processing unit 115. When a photographed image photographed by the photographing apparatus 100 is received through the communication interface unit 111, the control unit 113 provides the received photographed image to the fog image processing unit 115 to be processed. Further, the control unit 113 may perform various operations such as performing communication with the photographing apparatus 100 through the communication interface unit 111 based on a control command commanded by the fog image processing unit 115 again. For example, the control unit 113 may operate a flashing or the like when a vehicle suddenly interrupted is found through analysis of the captured image.

In addition to this, the controller 113 performs various control operations, but in FIG. 1, only the viewpoint of image processing is described, and other details are apparent to those skilled in the art, so further description will be omitted.

The fog image processing unit 115 may be more accurately referred to as an image processing unit. In other words, in the embodiment of the present invention, fog is described as an example because fog may become a more dangerous factor for vehicle operation than yellow dust or fine dust in the atmosphere, but the embodiment of the present invention will not be specifically limited to fog. .

The fog image processing unit 115 may first determine whether fog exists in the received photographed image. Of course, the presence or absence of fog is not particularly limited as it may be provided with relevant information from the Meteorological Agency or detected through an external sensor. When the saturation of the currently received photographed image suddenly drops from the photographed image of the previous time, the fog image may be determined based on the experience of fog characteristics or patterns through learning. For example, when there is fog in the received image, the fog image processing unit 115 may remove the fog and perform analysis using the image from which the fog has been removed, and provide it to the controller 113. For example, when the fog image processing unit 115 provides the analysis result of the binary bit information, the control unit 113 compares the binary bit information with the data on the look-up table (LUT) to recognize what operation to perform. can do.

The fog image processing unit 115 will be described in detail later with reference to FIG. 2, but it removes the fog contained in the atmosphere from the captured image and improves the visibility for the visibility loss area that may occur in the process of controlling the fog, and the linear model Based on this, fast processing is possible from a hardware design point of view. In order to improve visibility, more precisely, the fog image processing unit 115 is equipped with a low pass filter in order to prevent the occurrence of a visible area (e.g., background noise, color distortion, etc.) The visibility is improved by restoring the image without fog through constraints and adaptive weighting factors. The low-pass filter is applied to the saturation channel to remove background noise. In addition, the adaptive constraint is generated by generating an adaptive constraint that changes adaptively according to the pixels of the image, and is calculated using at least one of a minimum value, an average, and a standard deviation of the pixels of the preprocessed image. Constraints using mean and standard deviation can be used to avoid black pixels that can occur when removing the fog. In addition, when removing fog from an input image, a weight factor is used to remove fog in a dark area, that is, a weight is given to ensure that the values of each red (R), green (G), and blue (B) channel are operated consistently. Improves. Here, "adaptive" refers to changing or attempting to change to appropriately respond to changes in the external environment (eg, pixels).

More specifically, when a fog image is input, the fog image processing unit 115 performs a pre-processing operation by processing a white balance that restores the distortion caused by lighting in the image in which the fog exists, and estimates atmospheric light after the pre-processing operation. , Remove the background noise of the image. Here, "white balance" is to correct the color temperature of light through a lighting environment or an image device to prevent the color of the subject from being expressed differently depending on the temperature of light, that is, color temperature, so that it becomes ideal white. Adjust the gain for each color of R, G, and B. In other words, white is expressed as white. In addition, the fog image processing unit 115 generates adaptive constraints that change according to the pixels of the image, estimates a relaxed depth map of the image, estimates a new transmission map, and color distortion. Components are removed to create a fog removal image with improved visibility. After that, image analysis may be performed on the image from which the fog has been removed. The image analysis operation may be performed in a separate device.

The fog image processing unit 115 applies a low-pass filter to the saturation channel to alleviate background noise. In addition, the fog image processing unit 115 generates an adaptive constraint condition using the atmospheric light and the minimum value, average, and standard deviation of the pixels after the preprocessing step. After estimating the relaxed depth map by calculating the parameters estimated (or obtained) through machine learning using (or using) training images with uniform distribution of brightness and saturation and equally divided, adaptive constraints are applied. Create a new delivery map. Thereafter, the fog image processing unit 115 improves visibility by applying a reduced depth map and a weighting factor to prevent color distortion occurring while removing the fog.

For example, white balance, a pre-processing process, is performed so that white is expressed as white by correcting the color temperature of the image changed by lighting or atmospheric particles in the image with fog. In addition, the fog image processing unit 115 performs a task of mitigating background noise by estimating atmospheric light using the luminance channel of the image and applying a low pass filter to the saturation channel of the image. After that, adaptive constraints that change according to the pixels of the image are created, and the CAP parameters obtained through machine learning are applied to the brightness and saturation channels to estimate the relaxed depth map. Then, the fog image processing unit 115 estimates a new transmission map through brightness, saturation, parameters, and adaptive constraints, and generates an adaptive weighting factor using the depth map information, thereby removing the fog. It plays a role of preventing color distortion components that occur.

Meanwhile, the photographing apparatus 100 of FIG. 1 may itself include a fog image processing apparatus according to an embodiment of the present invention. Accordingly, an image with improved visibility may be provided directly to the autonomous vehicle 110 by removing a factor of visibility change from a captured fog image, more precisely, an image changed by the surrounding environment. Above all, the image processing apparatus according to an embodiment of the present invention forms visibility for an existing visibility deterioration area that occurs in the process of reconstructing an image by removing fog, and also displays a high-resolution image of about 4K in an autonomous vehicle ( 110) can be applied in real time, regardless of the location. Here, the existing visibility impaired area may be understood as a visibility enhancement method using a CAP.

On the other hand, the control unit 113 of FIG. 1 may include a CPU and a memory as another embodiment of the present invention. The CPU includes a control circuit, an operation unit (ALU), an instruction analysis unit, a registry, and the like, and the memory may include RAM. Therefore, when the autonomous vehicle 110 is initially driven, the CPU copies the program for performing the operation according to the embodiment of the present invention stored in the fog image processing unit 115, loads it into memory, such as RAM, and executes it. So that the data operation processing speed can be accelerated. The control circuit is in charge of the control operation, the operation unit performs the operation of binary bit information, and the instruction analysis unit performs the operation of converting high-level language into machine language or machine language into high-level language, and the registry stores software data. Can be involved in

Subsequently, an image processing process for removing fog according to an embodiment of the present invention will be described in detail with reference to FIG. 2.

2 is a block diagram illustrating a detailed structure of an apparatus for removing fog and improving visibility for a single image according to an embodiment of the present invention, and FIGS. 3 to 5 are diagrams illustrating color distortion occurring when fog is removed. 6 is a view showing an input image in which fog exists and an output image to which a technology according to an embodiment of the present invention is applied.

As shown in FIG. 2, an apparatus for removing fog and improving visibility for a single image according to an embodiment of the present invention (hereinafter, the color distortion prevention apparatus 115 ′ is an image input unit 200, a pre-processing unit (or Preprocessing unit) 210, atmospheric light estimating unit 220, background noise removing unit 230, adaptive constraint condition unit 240, (relaxed) depth map estimating unit 250, transmission map estimating unit 260 , A part or all of the image restoration unit 270 and the image output unit 280 are included.

Here, "including some or all" means that the above components are configured by hardware, software, or a combination thereof, and some components can be integrated into other components to be configured. Explain that it is all inclusive to help you understand it.

The color distortion prevention apparatus 115 ′ of FIG. 2 according to an exemplary embodiment of the present invention may use the atmospheric dispersion model proposed by McCartney to remove fog from a fog image. The atmospheric dispersion model can be expressed by <Equation 1>.

Here, x,y means the location of the pixels, I(x,y) is the input image including the fog captured by the camera, J(x,y) is the image from which the fog has been removed, and t(x,y) ) Denotes a transmission map indicating the amount of fog transmitted, and A denotes the atmospheric light obtained by estimation of the atmospheric intensity. J(x,y)t(x,y) is the direct attenuation, and A[1-t(x,y)] is the additional distortion caused by the scattered light.

The image input unit 200 inputs R, G, and B images I(x,y) including fog components. The image presented in FIG. 3 may be an input image. A unit frame image is formed by a combination of R, G, and B pixels, and a moving picture usually implements 60 unit frame images per second on the screen.

The pre-processing step unit 210 performs a pre-processing operation for the input I(x,y) so that the color temperature changed by lighting or atmospheric particles can be expressed as white by using a white balance. Here, the color temperature is a method of indicating color, and is expressed in Kelvin (K). When the temperature increases, the color temperature becomes blue and when the temperature decreases, the color temperature becomes red.

The atmospheric light estimating unit 220 estimates the atmospheric light in a method that does not use a frame buffer. For example, the atmospheric light estimating unit 220 can estimate atmospheric light by estimating atmospheric intensity according to the atmospheric dispersion model proposed by McCartney, and can estimate atmospheric light using a luminance channel of a fog image from which preprocessing or background noise has been removed. have.

When removing the fog, the background noise removing unit 230 generates nonlinear noise in the saturation channel, such as a “salt-and-pepper” phenomenon. This background noise phenomenon is a common problem for all fog removal algorithms. In order to remove this unwanted noise, it is solved by using a low pass filter in the saturation channel.

The adaptive constraint conditioner 240 may estimate from the atmospheric variance model. <Equation 1> can be refined to obtain unprocessed t(x,y), whereby <Equation 2> can be inferred. I(x,y) used in the adaptive constraint (Equation 2-7) means the luminance channel of the image in which the fog passing through the white balance exists, and I ^c (x,y) is the It is a foggy image with R, G, and B channels of the image.

Here, whether or not the haze is removed J(x,y) is correctly removed is within the range of normalized 0 to 1. Therefore, this is expressed as <Equation 3>.

Here, the atmospheric light corresponding to A operates as a constant.

In addition, the first constraint of the adaptive constraint is as shown in Equation 4.

는 r,g,b 채널의 max값을 의미하고,

는 c ∈ {r,g,b}이므로 I(x)(안개가 있는 영상)의 x,y 좌표를 의미하므로, 종합하면 r,g,b채널의 안개가 있는 영상의 x,y 좌표이다.

는 c ∈ {r,g,b}이므로 대기광의 r,g,b 채널을 의미한다.here,

Means the max value of the r,g,b channels,

Is c ∈ {r,g,b}, so it means the x,y coordinates of I(x) (image with fog), so in sum, it is the x,y coordinates of the image with fog in r,g,b channels.

Is c ∈ {r,g,b}, meaning the r,g,b channels of atmospheric light.

The second adaptive constraint is influenced by the definition of the mean and standard deviation of the statistics. Fog removal means subtracting fog from an image with fog. Here, the amount of fog subtracted is controlled by the transmission map. The smaller this value, the more fog is removed, and the brightness of the image from which the fog is removed decreases. In particular, in the case of a dark image before the fog is removed, a negative number occurs after the fog is removed. Negative numbers are rounded to zero, and these values become black pixels. In order to prevent such black pixels from occurring, the second constraint uses the property that the average of the image must be greater than or equal to the standard deviation of the image. This is the same as Equation 5.

는 i,j∈Ω{x,y}의 의미는 x,y좌표 중심으로 사각형 윈도우 필터(window filter)를 의미하며, 윈도우 내의 구성성분 좌표를 i,j라 한다. 즉, meani,j는 x,y를 기준으로 사각형의 평균값 필터를 사용할 때, 그 사각형 평균값 필터의 구성성분을 i,j라고 말한다.

는 J는 안개 없는 영상이며, i,j는 구성성분을 의미한다.

의 의미는 x,y 좌표 중심으로 사각 윈도우 필터를 의미한다.

에서 f는 제약조건을 제어하기 위한 상수이고, std는 표준편차이며, i,j는 표준편차 내의 성분을 의미한다.

는 meani,j∈Ω{x,y}의 의미는 x,y좌표 중심으로 사각 윈도우 필터를 의미하며, 윈도우의 각각 좌표를 i,j라 한다.here,

The meaning of i,j∈Ω{x,y} means a rectangular window filter centered on the x,y coordinates, and the component coordinates in the window are called i,j. That is, meani,j refers to i,j as the constituent components of the square mean value filter when using a square mean value filter based on x,y.

Where J is the image without fog, and i,j is the component.

Means a square window filter centered on the x,y coordinates.

In, f is a constant for controlling the constraint, std is the standard deviation, and i,j are the components within the standard deviation.

Meani,j∈Ω{x,y} means a rectangular window filter centered on the x,y coordinates, and the coordinates of each window are referred to as i,j.

In <Equation 5>, f is used as a constant for controlling the constraint. If <Equation 5> is substituted into <Equation 1> and refined, the second constraint of the adaptive constraint appears as in <Equation 6>.

By combining the first constraint and the second constraint, an adaptive constraint is expressed as shown in Equation 7.

The relaxed depth map estimation unit 250 is an area for estimating a depth map of an image. <Equation 8> is a linear model that explains the correlation between the brightness and saturation of an image.

v(x,y) and s(x,y) denote the brightness of the image and the saturation passed through the background noise removal unit 230, respectively, and θo, θ ₁ , θ ₂ are parameters of the linear model. ε(x,y) means error. In order to obtain the value of the parameter that minimizes the error, it is estimated through supervised learning. Although training images are required for supervised learning, there is no means for obtaining reliable and accurate depth maps according to the training images. In the past, the depth map was refined using a random number generator. By producing a training image using an equally divided uniform distribution that guarantees a uniform distribution, the parameters of a linear model with higher reliability than before can be obtained.

The transfer map estimator 260 applies the parameters obtained through brightness, saturation, and machine learning to the relaxed depth map estimator 250, and based on the adaptive constraint unit 240, the new transfer map t(x, Find the y) value.

<Equation 9> is an expression of the transfer map of the Atmospheric scattering model. It can be seen from Equation 9 that it is exponentially proportional to the relaxed depth map.

Here, ß denotes a scattering coefficent, and d(x,y) denotes a relaxed depth map.

The image restoration unit 270 receives the newly obtained value from the transmission map estimation unit 260, removes fog, and solves a color distortion problem to improve visibility. The cause of color distortion is recognized as a region close to a dark region, that is, a region with low brightness and saturation in the Color Attenuation Prior method, that is, the a priori color attenuation method. x,y) is recognized as being small or 0, which has a disadvantage in that the t(x,y) value is set to a value close to 1 and cannot be properly processed.

When the image is restored from which the fog has been removed through Equation 10, the pixel value in the dark area of the image may be slightly decreased or may not be changed.

Referring to FIGS. 3 to 5, it can be seen that the blue (B) channel value includes a larger value than the red (R) and green (G) channel values in the dark area. In the image after removing the fog, it can be seen that the blue (B) channel value does not decrease and contains almost the same value. Therefore, when the fog is removed for a dark area, a color distortion phenomenon that looks like a blue area occurs. A method of solving this problem can solve the problem by adding a weighting factor to consistently reduce the value of each channel in the dark image and using the value of the relaxed depth map.

Fig. 4(a) shows the pixel values of the R, G, and B channels in the dark area in the image where the fog exists, and Fig. 4(b) shows the pixel values of the R, G, and B channels in the image with the fog removed. 5A shows an image in which color distortion has occurred, and FIG. 5B shows an image in which color distortion is reduced by applying an adaptive weight. According to an embodiment of the present invention, color distortion occurring in the red box of FIG. 5, that is, a dark area is prevented.

<Equation 11> is an equation for adding weights, which may be referred to as adaptive weights. w ₀ denotes the starting value of the weight, and d ₀ denotes the value of the depth map designating the near-field.

<Equation 12> is an equation in which an adaptive weight is applied to <Equation 10> to obtain an image from which the fog is removed from which the color distortion phenomenon is solved.

As a result of the above configuration, the color distortion prevention apparatus 115 ′ according to the embodiment of the present invention does not cause color distortion when removing the fog from the image (Fig. 6(a)) where the fog exists as shown in FIG. By doing so, it is possible to restore the image without fog (Fig. 6(b)).

7 is a flowchart of a process of processing a fog image according to an embodiment of the present invention.

For convenience of explanation, referring to FIG. 7 together with FIG. 1, a control device (eg, an ECU, a control unit) of an autonomous vehicle 110 according to an embodiment of the present invention is a fog in which fog exists from the photographing device 100. An image may be received (S700).

Subsequently, the control device restores the image from which the fog has been removed by applying a low-pass filter, an adaptive constraint, and an adaptive weight factor to prevent color distortion that occurs when the fog is removed from the received fog image (S710). .

Here, the low-pass filter is applied to the saturation channel to remove background noise. Also, the adaptive constraint uses a constraint that changes adaptively according to the pixel. To create an adaptive constraint, the atmospheric light of the preprocessed fog image is estimated, and the constraint is created using the mean and standard deviation of pixels. Furthermore, the image is restored by applying a weight to the fog image based on the transmission map when the fog image is removed from the background noise, and the transmission map is the saturation of the fog image from which the background noise is removed, the brightness of the fog image, and further, the training image. The depth map is estimated based on the parameters obtained by using and the estimated depth map is applied to the adaptive constraint to estimate the transmission map of the fog.

Here, the transmission map is related to the amount of subtracting fog, and removing fog means subtracting fog from an image with fog. Thus, the amount of fog subtracted is controlled by the transfer map. For example, the smaller the value, the more fog is removed. Also, the depth map corresponds to a linear model that attempts to explain the correlation between the brightness and saturation of an image.

The control device can perform various operations in addition to the above descriptions, and other detailed information has been sufficiently described above, and thus the contents will be substituted.

8 is a flowchart of a process of processing a fog image according to another embodiment of the present invention.

For convenience of explanation, referring to FIG. 8 together with FIG. 2, the preprocessing step unit 210 of the color distortion prevention apparatus 115 ′ according to an embodiment of the present invention receives a fog image input to the image input unit 200 and A pre-processing operation is performed on an input image having a presence through white balance (S800). It mechanically changes the color of the light by adjusting the white balance.

In addition, the atmospheric light estimating unit 220 estimates atmospheric light from the preprocessed fog image (S810). The atmospheric intensity can be calculated for atmospheric light estimation, and can be obtained using the luminance channel.

The background noise removal unit 230 removes background noise from the preprocessed fog image. To this end, a low-pass filter is applied to the saturation channel.

In addition, the adaptive constraint conditioner 240 generates an adaptive constraint that adaptively changes according to a change in a pixel after preprocessing (S830). To this end, an adaptive constraint may be generated using the estimated atmospheric light after preprocessing and the average and standard deviation of the pixels after the preprocessing, and the minimum value of the pixel may be further calculated to generate the adaptive constraint.

The relaxed depth map estimation unit 250 is an area for estimating a depth map of an image.

Next, the transfer map estimator 260 calculates the brightness and saturation of the haze image from which the background noise has been removed, and the parameters estimated through machine learning, and applies the estimated relaxed depth map to the adaptive constraint to create a new transfer map. It is estimated (S840). The transmission map estimating unit 260 estimates a transmission map related to an amount of subtracting fog. For this, the brightness and saturation of the fog image from which the background noise has been removed is used.

Finally, the image restoration unit 270 restores the image by applying the relaxed depth map and weighting factor to prevent color distortion when removing the fog from the fog image (S850). When removing the fog of dark areas from a fog image, a weighting factor is used to make each R, G, and B channel values operate consistently. For example, each channel value can be consistently reduced in a dark area.

On the other hand, even if all the constituent elements constituting an embodiment of the present invention are described as being combined into one or operating in combination, the present invention is not necessarily limited to this embodiment. That is, within the scope of the object of the present invention, all of the constituent elements may be selectively combined and operated in one or more. In addition, although all the components may be implemented as one independent hardware, a program module that performs some or all functions combined in one or more hardware by selectively combining some or all of the components. It may be implemented as a computer program having Codes and code segments constituting the computer program may be easily deduced by a person skilled in the art of the present invention. Such a computer program is stored in a non-transitory computer readable media that can be read by a computer and is read and executed by a computer, thereby implementing an embodiment of the present invention.

Here, the non-transitory readable recording medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. . Specifically, the above-described programs may be provided by being stored in a non-transitory readable recording medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, or the like.

In the above, preferred embodiments of the present invention have been illustrated and described, but the present invention is not limited to the specific embodiments described above, and is generally used in the technical field to which the present invention pertains without departing from the gist of the present invention claimed in the claims. Various modifications are possible by those skilled in the art of course, and these modifications should not be individually understood from the technical idea or perspective of the present invention.

100: photographing device 110: autonomous vehicle
111: communication interface unit 113: control unit
115: fog image processing unit 200: (video) input unit
210: pre-processing step 220: atmospheric light estimation unit
230: background noise removal unit 240: adaptive constraint unit
250: relaxed depth map estimation unit 260: transmission map estimation unit
270: image restoration unit 280: (image) output unit

Claims (10)

An image input unit for inputting an R, G, B image I(x,y) including a fog component;
A preprocessor for performing preprocessing by adjusting the white balance of the input fog image;
An atmospheric light estimating unit for estimating atmospheric light using a luminance channel of the preprocessed fog image;
A background noise removal unit that removes background noise by applying a low-pass filter to a saturation channel of the preprocessed fog image;
An adaptive constraint conditioner that generates an adaptive constraint that adaptively changes according to a change of a pixel by using the estimated atmospheric light and a pixel of the preprocessed fog image;
A depth map estimating unit for estimating a depth map by further calculating parameters estimated by machine learning with saturation and brightness using the training image having an equally divided uniform distribution;
A transfer map estimating unit for estimating a transfer map related to a subtraction amount of fog by applying a depth map obtained by estimating a correlation between saturation and brightness of a fog image from which background noise has been removed to the generated adaptive constraint;
An image restoration unit for restoring a fog image by applying a weight according to [Equation 12] below to the fog image based on the estimated transmission map when the background noise is removed from the fog image; And
An image output unit that compares an input image in which fog is present and an output image of an image obtained by removing the fog;
An apparatus for removing fog and improving visibility for a single image comprising a.
[Equation 12]

In the above, x,y are the location of the pixels, J(x,y) is the image taken by the camera from which the fog has been removed, I(x,y) is the input image including the fog captured by the camera, and A is Atmospheric light obtained by estimation of atmospheric intensity, w _{t (x, y)} is an adaptive weight, and t (x, y) is a transmission map representing the amount of fog transmitted. delete The method of claim 1,
The adaptive constraint unit, by using a constraint using the average and standard deviation of the pixels, removes fog and improves visibility for a single image that prevents black pixels from removing fog from the fog image. Device for. The method of claim 1,
The image restoration unit applies a weight when removing the fog in a dark area from the fog image from which the background noise has been removed, so that the channel values of red (R), green (G) and blue (B) are consistently operated. A device for removing fog and improving visibility. The method of claim 4,
The image restoration unit applies a weight when removing the fog in the dark area so that the channel values of red (R), green (G), and blue (B) are not consistently decreased or changed, and A device for improved visibility. Image input unit, preprocessing unit, atmospheric light estimating unit, background noise removing unit, adaptive constraint conditioner, depth map estimation unit, transmission map estimation unit, image restoration unit and image output unit. As a driving method of the device for,
An image input step of inputting an R, G, B image I(x,y) including a fog component from an image input unit;
Preprocessing by adjusting a white balance of the input fog image by a preprocessor;
An atmospheric light estimating unit estimating atmospheric light using a luminance channel of the preprocessed fog image;
A background noise removal unit removing background noise by applying a low pass filter to a saturation channel of the preprocessed fog image;
Generating, by an adaptive constraint condition unit, an adaptive constraint that changes adaptively according to a change of the pixel using the estimated atmospheric light and the pixel of the preprocessed fog image;
Estimating a depth map by further calculating a parameter estimated by machine learning with saturation and brightness using the training image having a uniform distribution divided by the depth map;
Estimating a transfer map related to a subtraction amount of fog from the transfer map estimating unit by applying a depth map obtained by estimating a correlation between saturation and brightness of a fog image from which background noise has been removed, to the generated adaptive constraint condition unit;
Restoring the fog image by the image restoration unit by applying a weight according to the following [Equation 12] to the fog image based on the estimated transmission map when the fog image from which the background noise has been removed; And
An image output step of comparing an input image in which fog is present and an output image of an image obtained by removing the fog, in the image output unit;
A method of driving an apparatus for removing fog and improving visibility for a single image comprising a.
[Equation 12]

In the above, x,y are the location of the pixels, J(x,y) is the image taken by the camera from which the fog has been removed, I(x,y) is the input image including the fog captured by the camera, and A is Atmospheric light obtained by estimation of atmospheric intensity, w _{t (x, y)} is an adaptive weight, and t (x, y) is a transmission map representing the amount of fog transmitted. delete The method of claim 6,
The generating step,
A method of driving an apparatus for removing fog and improving visibility for a single image, which prevents black pixels occurring when fog is removed from the fog image by using a constraint using the average and standard deviation of the pixels. The method of claim 6,
The restoring step,
To remove fog and improve visibility for a single image in which the channel values of red (R), green (G), and blue (B) are consistently operated by weighting when removing the fog in the dark area from the fog image from which the background noise has been removed Method of driving the device for. The method of claim 9,
The restoring step,
A device for removing fog and improving visibility for a single image in which the channel values of red (R), green (G) and blue (B) are not consistently decreased or changed by assigning a weight when removing the fog in the dark area Method of driving.

Images