TW201308251A - Underwater image enhancement system - Google Patents

Abstract

Light scattering and color shift are two major sources of distortion for underwater photography. Light scattering is caused by light incident on objects reflected and deflected multiple times by particles present in the water before reaching the camera. This in turn lowers the visibility and contrast of the image captured. Color shift corresponds to the varying degrees of attenuation encountered by light traveling in the water with different wavelengths, rendering ambient underwater environments dominated by bluish tone. This Underwater Image Enhancement System employs a novel approach to enhance underwater images by a dehazing algorithm with wavelength compensation and consider the impact of artificial light irradiation. Once the depth map, i.e., distances between the objects and the camera, is estimated by dark channel prior, the light intensities of foreground and background are compared to determine whether an artificial light source is employed during image capturing process. After compensating the effect of artificial light, the haze phenomenon from light scattering is removed and the color shift distortion is compensated by the dehazing algorithm. Next, estimation of the image scene depth according to the residual energy ratios of different wavelengths in the background is performed. Based on the amount of attenuation corresponding to each light wavelength, color shift compensation is conducted to restore color balance.

Description

Underwater image enhancement system

The invention relates to a method for image enhancement, in particular to strengthening underwater images to enhance the visibility and color realism of underwater images, so that underwater images are processed like images taken in the air.

How to capture clear images in the underwater environment is an important topic in ocean engineering. The effectiveness of underwater navigation monitoring, environmental assessment and mineral exploration applications are all based on the quality of underwater photography. Faced with many challenges, the most important reason stems from the atomization effect caused by dispersion and the color shift phenomenon caused by the inconsistent energy attenuation of each wavelength when the light propagates in water.

It is known that underwater image enhancement technology only corrects the atomization effect caused by dispersion or the color shift distortion caused by the inconsistent energy attenuation of each wavelength. There is no solution to overcome the color shift and dispersion phenomenon at the same time; And the progress is more clear, first analyze and discuss the patents and published journal articles related to the existing underwater image enhancement.

First, remove the atomization effect caused by dispersion:

1. W. Hou and A. Weidemann, "Automated Underwater Image Restoration Via Denoised Deconvolution," US 20090252430A1, 2009. This method is a combination of point spread functions (PSF) and modulation transfer function (MTF) to enhance underwater image contrast to reduce the atomization effect, although it can improve the contrast and image visibility of underwater scenes, but The color shift phenomenon caused by the inconsistent energy attenuation of each wavelength still exists in the image.

2. Y. Y. Schechner and N. Karpel, "Enhanced underwater imaging," US 7,804,518,2010. The method uses the polarization and polarization characteristics of the polarizer to estimate the distance from the object to the camera, and compensates for the attenuation of the image visibility by the defogging algorithm according to the physical characteristics of the underwater image imaging, thereby achieving the effect of image enhancement, but at least Two or more images with different polarization angles can estimate the distance from the object to the camera. The characteristics of the invention can be enhanced only by a single image, and the image depolarization algorithm cannot eliminate the image color deviation phenomenon. The color shift phenomenon still exists in the image after processing.

3. V. K. Asari and L. Tao, "Visibility improvement in color video stream," US 7,428,333, 2008. The method combines the method of enhancing brightness and contrast, firstly separating the underwater image into a chroma map and a brightness map, and then performing nonlinear enhancement on the luminance map, and finally synthesizing the chroma map and the luminance map into an output image, although Enhance the contrast of the image and enhance the details of the shadow, but it does not help the color shift of the image.

Second, the improvement of image color shift phenomenon:

Yamashita et al. consider the degree of light attenuation to estimate underwater environmental parameters and perform energy compensation to correct image color shift phenomenon ( Intl. Conf. Robotics and Automation, 2007 ), and Iqbal et al. use histograms to equalize RGB and HSI color spaces. Color brightness distribution balance ( Intl. Journal of Computer Science , 2007) and Vasilescu et al. Distance-based dynamic hybrid illumination technology compensates for differences in wavelength loss by using a controllable multi-color source to estimate the attenuation of color light at each wavelength The degree ( Proc. of Robotics: Science and Systems , 2010), although the above method can improve the degree of color balance, but can not remove the fog caused by dispersion.

This system considers that the atomization effect, color shift distortion and artificial light source illumination effect of underwater image are not seen in the prior art, so the invention is novel, progressive, practical and industrial.

In view of the fact that the single-function underwater image enhancement technology cannot meet the requirements of marine engineering for underwater image quality, the known underwater image enhancement algorithm does not consider the influence of artificial light source on the brightness of the sensed image. After multi-party evaluation and research, develop a set of underwater image enhancement algorithms that can simultaneously correct color shift and dispersion and remove the effects of artificial light sources. Only a single input image can effectively improve image visibility and color realism. Underwater shooting images and videos have achieved good visual effects, showing the original color, clarity and detail fidelity in the air.

The underwater image enhancement system of the present invention comprises a data acquisition device, a data transmission interface, a data processing device and an image display device, wherein the enhancement processing performed by the data processing device is combined with defogging and wavelength energy compensation. And consider the underwater image enhancement algorithm illuminated by artificial light source, which can strengthen the underwater image affected by atomization, color shift and artificial light source to restore underwater image visibility and brightness, color realism, and make underwater image It is like the original color, clarity and detail fidelity in the air. Firstly, the static underwater image or the dynamic underwater image sequence is captured by the data acquisition device, and temporarily stored in a data storage device, and the static underwater image or the dynamic underwater image sequence is captured by the data transmission interface, and the data is stored. The device is transmitted to the data processing device for enhancement processing, and finally the processed underwater image is displayed on the image display device for the user to observe.

In order to achieve the above objects of the present invention, the following flow chart (second diagram) of the underwater image enhancement algorithm combined with defogging and wavelength energy compensation and considering artificial light source illumination according to the present invention will be respectively described according to each component:

(I. Introduction

The underwater image intensification system of the present invention simultaneously considers the underwater depth D to D + R , the object-to-camera distance d(x) and the artificial light source L , and overcomes and solves the dispersion of the underwater image one by one. Color shift phenomenon to enhance the clarity and detail fidelity of underwater images.

The underwater image enhancement system performs inverse compensation by the underwater imaging model. As shown in the first figure, the underwater image can be expressed as uniform natural light A (101) from the air through the underwater depth D(x) ( 104) attenuation to form underwater ambient light W(x) , where D<D(x)<D + R , the energy of natural light A and underwater ambient light W(x) are ( x ) and ( x ), their relationship to each other can be expressed as a wavelength energy attenuation model:

Where λ represents the wavelength of each color of light in the light, and Nrer(λ) is the normalized energy residual ratio:

After the underwater ambient light W(x) attenuated by water depth is irradiated to a point x in the shooting scene, the reflected light travels through the distance d(x) (105) to the camera (107) to become the underwater image I _λ ( x ), λ { red , green , blue }. When the reflected light of the object passes through the distance d(x) , it will suffer both dispersion and color shift distortion. The color shift phenomenon not only exists in the attenuation of the underwater depth D(x) , but also occurs in the object reflected light at the distance d(x) . Propagation, and the phenomenon of dispersion is the reflection of light from the object to the camera. Part of the reflected light hits the suspended particles in the water (108), causing absorption and scattering, so that the reflected light that should travel straight to the camera collides with the suspended particles. scattering. In an environment without black body radiation, the scattering phenomenon usually spreads into multiple scattering, which causes the scattered light to form a uniform background light B _λ . Therefore, the underwater blurred image of the scattering process can be based on the principle of "fuzzy image imaging" (R. Fattal , "Single Image Dehazing," Intl. Conf. on Computer Graphics and Interactive Technique , No. 72, Pages 1-9, 2008.) is expressed as the weighted sum of the direct transmitted light and the background light scattered by the suspended particles, as in the formula 3 shows:

Where the directly conducting object reflects light ( x ) ‧ Nrer ( λ ) ^{D ( x )} ‧ ρ _λ ( x ) is the attenuation of natural light A through the depth D (x) , and then irradiated to the object in the shooting scene, and then through the surface reflectivity ρ _λ ( x ) The reflection is formed.

In addition, in the underwater environment, in order to overcome the lack of brightness of the ambient light source, the artificial light source L is added to assist the shooting. Therefore, the compensation for the difference in energy attenuation caused by the underwater depth D(x) and the defogging of the underwater image must be removed. The brightness contributed by the artificial light source avoids excessive compensation. Assuming that the artificial light source is a point light source placed beside the camera , the energy emitted by the artificial light source Before the object is illuminated, it needs to be attenuated by the distance d(x) , so the reflected light of the object can be corrected to the residual energy of the artificial light source attenuated from the object to the camera distance d(x) . ‧ Nrer ( λ ) ^{d ( x )} and underwater ambient light attenuated by water depth D (x) ( x )‧ Nrer ( λ ) ^{D ( x ) is} added together, and is irradiated to the subject together and reflected by the surface reflectance ρ _λ ( x ), so Equation 3 can be further corrected as:

Formula 4 is a complete mathematical expression of the underwater image imaging model proposed by the present invention, which corresponds to the first figure. The underwater image enhancement system of the present invention simultaneously considers the atomization effect, wavelength attenuation and artificial light source, etc. The underwater image I _λ ( x ) affected by atomization and color shift can effectively eliminate the influence of artificial light source and remove the color shift and dispersion caused by the distance d(x) , and finally compensate the underwater depth D(x) . Color shift phenomenon. Therefore, it will be introduced in the following sections: 1. Estimate the distance from the object to the camera d(x) (210), 2. Split the foreground and background of the underwater image (220), 3. Determine whether there is an artificial light source in the underwater shooting scene. (230), 4. Detect and remove the brightness of the artificial light source (240), 5. remove the color deviation and dispersion caused by the distance d (x) (250), 6. estimate the underwater scene of the shooting scene Depth D(x) (260) Finally 7. Compensate for the color shift phenomenon caused by the underwater depth D(x) of the shooting scene (270), so that the underwater captured image has a good visual effect.

(2) Estimating the distance from the object to the camera d(x) (210)

A method commonly used to estimate the distance from an object to the camera in an underwater image is generally estimated by parallax through at least two underwater images, but there is still only a single underwater image for distance estimation. The method includes a Dark-Channel estimation method, a vanishing point estimation method or a DFD (Depth-from-defocus) estimation method. However, in a foggy environment, the concentration of the fog will increase with the increase of the distance. Therefore, the fog concentration information in the scene can be used to predict the distance from the object to the camera d(x) in the shooting scene. The invention is based on the Dark-Channel proposed by He. The estimation method is an example (K. He, J. Sun and X. Tang, "Single image haze removal using Dark Channel Prior," Proc. of IEEE CVPR , vol. 1, pp. 1956-1963, 2009) with Laplacian The matrix matting technique can effectively estimate and optimize the distance from the object to the camera. The Dark-Channel estimation method indicates that after subtracting the background portion of the image from the underwater image without fog, the local area formed by any point x of the remaining foreground portion is Ω. In (x), at least one pixel has a luminance value that is extremely low and close to zero. The main reasons for this low luminance value are:

1. Shadow: Bio-shadow, plankton, plant or seabed rock.

2. The object itself is rich in color: green plants, yellow sand beaches, colorful rocks and minerals, etc. The lack of any color in the color space will result in a very low value when processed by the Dark Channel.

3. Darker matter: black creatures and rock surfaces.

The Dark-Channel estimation method has a low reflectivity ρ _λ ( x ) from a color channel on the surface of the object. If the local region Ω(x) in the foreground portion of the image does not have a lower luminance value after the Dark Channel operation, It means that there is fog in this local area, so the Dark Channel can evaluate the fog concentration distribution range in the shooting scene, and further provide the object-to-camera distance d(x) , and the distance d for each pixel in the underwater image ( x) After the estimation, the object to camera distance depth map (302) can be obtained. However, Dark Channel is a block-based operation method, so the depth map generated is less accurate and is prone to mosaic distortion. Therefore, it is necessary to optimize the object estimated by Dark Channel to the camera distance depth map with Laplacian matrix matting technology. The third graph (303) shows the optimized depth map, and the mosaic phenomenon has been greatly improved compared with the third graph (302).

(III) Dividing the underwater scene foreground and background (220)

After obtaining the object-to-camera distance depth map (303), the image foreground and background segmentation can be further performed with the object-to-camera distance depth map as a reference, and the segmentation formula is as follows:

Where d(x) represents the distance from the object to the camera, and σ is an adjustable parameter. After the underwater image front view and the background segmentation map are initially obtained, the connected component is also used to filter the background area with a small area to reduce the simple use. The misjudgment caused by the division of the parameter σ by the dichotomy.

(4) Judging whether there is an artificial light source in the underwater shooting scene (230)

Whether there is an artificial light source in the underwater image shooting scene, mainly based on the difference between the foreground image and the background average brightness of the underwater image as the basis for judging whether the underwater light source is used for the auxiliary illumination, and the water without the artificial light source In the lower image, the background portion is not directly reflected by the object, but is directly transmitted by the underwater ambient light W(x) , and thus belongs to the higher brightness portion of the underwater image. If the average brightness of the underwater image is higher than the background average brightness, it means that there is an artificial light source in the underwater image shooting scene. Otherwise, the artificial light source is not used when the underwater image is captured.

(5) Detecting and removing the brightness of the artificial light source (240)

If there is an artificial light source in the underwater image, the object reflected light J _λ ( x ) can be expressed as the sum of the energy of the artificial light source and the underwater background light illuminating the surface of the object, and reflected by the surface reflectance ρ _λ ( x ) Measure the energy, so reflect the light on the object Artificial light source It can be a point, line or planar light source, emitting a white light source with the same energy of red, green and blue wavelengths or a colored light source with a specific wavelength intensity greater than the wavelength intensity. The invention is a white light source with equal energy of each wavelength. The radiation is performed in a point-like spherical manner as the underwater image is inversely proportional to the square of the distance from the object to the camera (the closer the object is to the camera, the more the artificial light source is affected, and vice versa), at the same distance. All the objects on the object are illuminated by the artificial light source with the same brightness, so this characteristic can be used to solve the energy of the corresponding artificial light source at all pixels of the same distance d(x) . And the surface reflectance ρ _λ ( x ) of each wavelength.

Reflecting light from objects Solution ρ _red ( x ), ρ _green ( x ), and ρ _blue ( x ) are over-determined equations, so they need to be solved by the following least squares difference method.

If you get artificial light source illumination energy And after the surface reflectance ρ _λ ( x ) of each wavelength (see Figure 5), the artificial light source can be further removed from the underwater image according to the following formula, as shown in the third figure (305).

If the illumination energy of the artificial light source is not removed, the difference in energy attenuation caused by the subsequent compensation of the underwater depth D(x) and the defogging of the underwater image may cause excessive image compensation, as shown in the fourth figure (402). Shown.

(6) Removal distance d(x) Caused by color deviation and dispersion (250) To remove the energy from the artificial light source ‧ After Nrer ( λ ) ^{d ( x )} , the atomization term (1- Nrer ( λ ) ^{d ( x )} ) ‧ B _λ formed by scattering can be removed and expressed as follows:

The underwater image with the atomization effect removed still has the color shift phenomenon caused by the inconsistent wavelength energy attenuation, so it is necessary to compensate for the difference in energy attenuation between the wavelengths. The equation 8 is equally divided by the wavelength energy attenuation ratio Nrer ( λ ) ^{d ( x )} to correct the color shift phenomenon, and the object reflected light J _λ after removing the color shift and dispersion caused by the distance d(x) x ) can be expressed as:

The third picture (306) is the result of the color shift and dispersion caused by the removal of the distance d(x) . It can be found that there is still a blue-shifted color shift in the image because the natural light A in the air penetrates the water depth. When D(x) reaches the underwater image capturing scene, the underwater ambient light W(x) is caused by the color shift phenomenon due to the inconsistent energy attenuation of each wavelength. Therefore, to obtain images without the effects of fogging and color shifting ( x ) ‧ ρ _λ ( x ), also estimated in the only unknown variable in Equation 9: underwater depth D(x) .

(7) Estimating the underwater depth of the shooting scene D(x) (260)

Assuming that the natural light A incident from the air is white light with uniform energy at each wavelength, the energy at each wavelength of one point x can be expressed as After entering the depth D(x) of the water, the incident natural light diminishes to form the underwater ambient light W(x) , and the energy of each wavelength is . It can be obtained from Equation 1 that the incident natural light A has the following relationship with the underwater ambient light W(x) :

Therefore, the underwater depth D(x) of the shooting scene can be estimated based on the remaining energy ratio of each wavelength of the underwater ambient light.

First obtain the underwater ambient light W(x) at any point x , and then solve it by the least square method, so that the light is incident from the air. The energy remaining after attenuation by depth k and the energy of each wavelength of underwater ambient light Minimum error between each other:

The depth of the underwater shooting scene extends from D to D + R. If the energy difference compensation of each wavelength is performed with only a single underwater depth D for the entire image, the color will remain under the image due to the depth coverage range R. The partial phenomenon, as shown in the third figure (307), therefore, in order to obtain a consistent color distortion correction result, the underwater depth D(x) in each underwater image must be completely estimated and compensated. However, the underwater ambient light W(x) represents the component that is not reflected by the surface reflectance ρ _λ ( x ), which is the background portion of the underwater image. If the underwater depth is estimated based on the residual energy of the underwater ambient light, In order to overcome this problem, the present invention is solved by linear interpolation, so that the underwater depth D(x) of any point x in the underwater image can be estimated.

Let the background pixel at the top of the underwater image correspond to the underwater depth of the shooting scene as D , and the bottom pixel is D + R. R represents the depth coverage in the image scene, where the underwater depth D and D + R can be estimated by Equation 10, and the underwater depth D(x) corresponding to any point x in the scene is between D and D + R , so the underwater depth D is everywhere in the underwater image ( x) can be obtained by linear interpolation from the topmost and bottommost background pixel points. Suppose of any pixel x, the top and the bottom of the background image pixels are located on a _x, b and c of scanning lines, any of the water depth D (x) x a pixel can be represented by the formula of the linear The interpolation method is derived:

(8) Compensating for the underwater depth of the shooting scene D(x) Caused by color shift (270)

After obtaining the underwater depth D(x) of any point, the color shift phenomenon caused by the natural light A propagating along the underwater depth D(x) can be corrected according to the formula 12:

After compensating for the difference in energy attenuation of each wavelength caused by the underwater depth D(x) at any point, the result of the third graph (308) can be obtained, and the invention of the present invention can be found from the third graph (308) and the fourth graph (403). The underwater image enhancement system can effectively restore the color of the image and remove the effect of the atomization effect and the artificial light source, so that the color shift and the atomized image in the original water are enhanced to be like the color tone and clarity in the air.

101. . . Natural light in the air

102. . . The shortest distance of underwater depth in the scene

103. . . Artificial light source for auxiliary shooting next to the camera

104. . . Shoot a point from the scene x to the surface of the water

105. . . Shoot a point x to the distance of the camera in the scene

106. . . Underwater depth coverage of the scene

107. . . Image capture camera

108. . . Contains suspended particles such as sand, minerals, and plankton in water

210. . . Estimate object to camera distance

215. . . Object to camera distance depth map

220. . . Front view and background after splitting the image

230. . . Determine whether there is artificial light source illumination in the shooting scene

240. . . Remove the energy from the artificial light source

250. . . Remove color deviation and dispersion caused by distance d(x)

260. . . Estimate the underwater depth D(x) of the shooting scene

270. . . Compensate for the color shift caused by the underwater depth D(x) of the shooting scene

301. . . Underwater image taken in real underwater scene (1)

302. . . Object-to-camera distance depth map estimated by Dark-channel

303. . . Object-to-camera distance depth map optimized by Laplacian matrix matting

304. . . Underwater image taken in real underwater scenes (2)

305. . . Underwater image after removing artificial light source

306. . . Remove the color cast and disperse image caused by the distance d(x)

307. . . Compensate the color shift result of the entire image only with the underwater depth D

308. . . Compensation for the color shift result of depth D(x) at each point of the underwater

401. . . Underwater image taken in real underwater scenes (2)

402. . . Over-compensation of underwater images

403. . . Correct result of underwater image compensation

501. . . Distribution map of images illuminated by artificial light sources

502. . . Red wavelength object reflectivity map

503. . . Green wavelength object reflectivity map

504. . . Blue wavelength object reflectivity map

Figure 1 is an imaging model of underwater images.

Figure 2 is a text system flow of the underwater enhanced system of the present invention

Figure 3 is the image system flow of the underwater enhanced system of the present invention

Figure 4 shows the effect of the artificial light source in the image.

Figure 5 shows the distribution of surface reflectance of artificial light sources and objects.

101. . . Natural light in the air

102. . . The shortest distance of underwater depth in the scene

103. . . Artificial light source for auxiliary shooting next to the camera

104. . . Shoot a point from the scene x to the surface of the water

105. . . Shoot a point x to the distance of the camera in the scene

106. . . Underwater depth coverage of the scene

107. . . Image capture camera

108. . . Contains suspended particles such as sand, minerals, and plankton in water

Claims (12)

An underwater image enhancement system for enhancing underwater images affected by fog, color cast and artificial light sources, the underwater image enhancement system comprising: - a data capture device for capturing static underwater images or dynamics The underwater image sequence is temporarily stored in a data storage device; the data transmission interface is configured to transmit the captured underwater underwater image or the dynamic underwater image sequence to the data processing device; and the data processing device is configured to receive The image processing device displays the processed image for viewing by the user; wherein the enhanced processing performed by the data processing device is combined with defogging and Wavelength energy compensation and considering the underwater image enhancement algorithm of artificial light source illumination can enhance the underwater image affected by atomization, color shift and artificial light source to restore underwater image visibility and brightness and color realism. The underwater image enhancement system according to claim 1, wherein the underwater image enhancement algorithm combined with defogging and wavelength energy compensation and considering artificial light source illumination further comprises: - the distance from the object to the camera in the underwater environment Estimation step; - estimation step of underwater depth of shooting scene; - estimation step of irradiation energy of artificial light source of underwater image; - removal step of irradiation energy of artificial light source of underwater image; - image defogging step; - wavelength energy Compensation step. The underwater image intensification system of claim 2, wherein the estimating step of the object to the camera distance in the underwater environment is based on a single underwater image to estimate the distance from the object to the camera A depth map corresponding to the distance from the object to the camera in the underwater environment is generated. The underwater image enhancement system described in claim 3, wherein the method for estimating the distance by using a single underwater image may be a Dark Channel estimation method, a vanishing point estimation method or a DFD (Depth-From-Defocus) method. Estimation method. The underwater image intensification system of claim 2, wherein the estimating step of the underwater depth of the shooting scene is calculated according to the degree of energy attenuation of light propagation in the water, minimizing the energy and water after being attenuated from the incident light of the air. The ambient light energy is different from each other to estimate the underwater depth of the scene being photographed. The underwater image intensification system of claim 3, wherein the estimating step of the underwater depth of the shooting scene is based on the depth map of the object to the camera in the corresponding underwater environment for underwater image foreground and background segmentation. After deriving the underwater depths of the most vertex pixels and the bottom pixels of the background portion of the underwater image respectively, the underwater depths of the remaining points in the scene are calculated by linear interpolation. The underwater image intensification system of claim 2, wherein in the estimating step of the underwater image artificial light source, the artificial light source may be a point, line or planar light source, and emit red, green and A white light source with a uniform blue wavelength energy or a light source with a specific wavelength intensity. The underwater image intensification system according to Item 7 of the present invention, wherein the difference between the foreground image and the background average brightness of the underwater image is used as a basis for determining whether to use the human illumination source auxiliary illumination in the underwater image, if the underwater image is stored When the artificial light source is irradiated, the minimum square error between the reflected light of the object and the incident light (ie, the artificial light source plus the underwater ambient light) is calculated, and the brightness contributed by the artificial light source and the surface reflectance distribution information of the object are respectively derived. The underwater image intensification system of claim 8, wherein the step of removing the illumination energy of the underwater image artificial light source is performed after obtaining the information of the brightness and the surface reflectance distribution of the artificial light source, and further the artificial light source. The energy contributed is removed from the underwater image. The underwater image intensification system of claim 3, wherein the image defogging step is performed after obtaining a depth map of the object to the camera in the corresponding underwater environment, and further adopting a defogging algorithm to implement the underwater environment. The atomization effect generated by the object to the camera distance is removed from the underwater image. The underwater image intensification system of claim 10, wherein after the image defogging step, further comprising a wavelength energy compensation step for attenuating the energy of each wavelength in the distance from the object to the camera in the underwater environment The difference is compensated. The underwater image intensification system of claim 2, wherein the wavelength energy compensation step is performed after obtaining the underwater depth information and the depth range information of the shooting scene from the water surface, and respectively, the air light from the water surface to the scene. The difference in attenuation of the wavelength energy is compensated.