CN113592738A - Underwater distorted image restoration method - Google Patents
Underwater distorted image restoration method Download PDFInfo
- Publication number
- CN113592738A CN113592738A CN202110855868.XA CN202110855868A CN113592738A CN 113592738 A CN113592738 A CN 113592738A CN 202110855868 A CN202110855868 A CN 202110855868A CN 113592738 A CN113592738 A CN 113592738A
- Authority
- CN
- China
- Prior art keywords
- image
- sequence
- distortion
- underwater
- distorted
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 35
- 239000013598 vector Substances 0.000 claims abstract description 72
- 230000000737 periodic effect Effects 0.000 claims abstract description 45
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims abstract description 29
- 230000000694 effects Effects 0.000 claims description 21
- 238000000354 decomposition reaction Methods 0.000 claims description 12
- 238000006073 displacement reaction Methods 0.000 claims description 9
- 238000012937 correction Methods 0.000 claims description 7
- 230000004927 fusion Effects 0.000 claims description 7
- 238000010276 construction Methods 0.000 claims description 6
- 238000001514 detection method Methods 0.000 claims description 6
- 238000005516 engineering process Methods 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 claims description 6
- 238000005259 measurement Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000000605 extraction Methods 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 3
- 238000000638 solvent extraction Methods 0.000 claims description 3
- 238000003384 imaging method Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000002427 irreversible effect Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000012804 iterative process Methods 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/80—Geometric correction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
Abstract
The invention discloses an underwater distorted image restoration method, which comprises the following steps of S1, reading a distorted image; s2, selecting the salient feature points; s3, tracking the salient feature points; s4, estimating a motion vector field; s5, preliminary restoration of periodic distortion; s6, fusing lucky blocks; s7, blind deconvolution deblurring; and S8, iteratively registering the non-rigid images. The invention has the following advantages: and (3) proposing an assumption of the water surface fluctuation type and proposing a corresponding two-stage restoration algorithm, thereby realizing effective restoration on the distorted underwater image.
Description
Technical Field
The invention belongs to the technical field of underwater image processing, and particularly relates to an underwater distorted image restoration method.
Background
In recent years, with the development of underwater resources and the popularization of underwater operations, underwater images are widely used in many fields such as marine information acquisition, underwater operation monitoring, underwater ecology monitoring, and the like. Then, refraction phenomenon occurs because light undergoes medium change of air-water in an imaging path, and meanwhile, the water surface under a real environment often has severe fluctuation, so that the caused refractive index change causes serious distortion and degradation of an underwater image. In addition, the underwater image is in a low-contrast and high-noise state due to the motion blur phenomenon caused by medium scattering and water surface fluctuation caused by the existence of suspended particles in water, and accurate acquisition of underwater information and subsequent information processing are seriously influenced. How to carry out effectual restoration to distortion image under water, correct the distortion, improve the whole definition of image and have important meaning.
At present, the restoration method for underwater distorted images mostly focuses on the improvement of the water surface ripple estimation, the lucky block fusion and the image registration technology. Although these algorithms can correct geometric distortion in an image to some extent, they often fail to achieve satisfactory results in terms of distortion correction. In addition, the image registration technology with relatively outstanding restoration effect limits the application of the underwater image restoration technology in the actual environment due to the problems of large calculation amount and long time consumption of the iterative process.
Currently, there are scholars such as Oreifej et al, which introduce gaussian blur into distorted sequence images before registration with sequence mean as a reference image, and the theoretical support of this strategy is: when the reference image is at the same blur level as the sequence image, the registration will be directed to a relatively sharp portion. However, artificially introducing blur into the sequence images may cause irreversible image information loss, thereby limiting the effectiveness of registration.
Disclosure of Invention
The invention aims to provide an underwater distorted image restoration method aiming at the defects of the prior art, and provides an assumption of a water surface fluctuation type and a corresponding two-stage restoration algorithm, so that the distorted underwater image is effectively restored.
In order to solve the technical problems, the technical scheme adopted by the invention is as follows: an underwater distorted image restoration method comprises the following steps,
s1, reading the distorted image, reading the underwater distorted video according to the characteristics of the fluctuating water surface under the natural condition, and decomposing the distorted video into an image sequence frame by frame;
s2, selecting the salient feature points, and detecting the salient feature points in the first frame image of the video;
s3, tracking the salient feature points, and tracking the selected feature points in the subsequent image sequence to obtain the motion vectors of the feature points;
s4, estimating the motion vector field, and solving the motion vector field of the whole image by the motion vector of the characteristic point based on the compressed sensing theory;
s5, performing preliminary restoration on the periodic distortion, and correcting the periodic distortion in each frame of image according to the motion vector field obtained in the step S4 to realize the preliminary restoration;
s6, fusing lucky blocks, dividing the image sequence after the initial restoration into image blocks, selecting the image blocks with the first 50% of small distortion degree by taking the structural similarity as an index, and fusing the image blocks into a single image;
s7, blind deconvolution deblurring, and a Richardson-Lucy blind deconvolution algorithm is used for removing Gaussian blur in the fused image and reconstructing a clearer reference image;
s8, performing iterative registration on the non-rigid image, registering the image sequence and the reconstructed reference image by using a non-rigid registration algorithm based on a B spline to achieve the effect of eliminating residual random distortion, and repeating the steps S6, S7 and S8 when the iterative termination condition is not met; when the image sequence is registered with the reconstructed reference image in the step S8 and the iteration termination condition is satisfied, outputting the restored undistorted image sequence;
in step S1, the water surface fluctuation is assumed to be a superposition of periodic fluctuation and random fluctuation according to the characteristics of the fluctuating water surface under natural conditions;
the image restoration method aiming at the periodic distortion caused by the periodic fluctuation comprises the following steps: A. reading of distorted images, reading of distorted underwater video and decomposition thereof frame by frame into a sequence of distorted images Vd＝{Id1，Id2，...，Idn}；
B. Selecting the salient feature points, subjecting a motion vector field caused by periodic fluctuation to sparse decomposition, namely solving the motion vector field of the whole image through motion vectors of a few points, extracting the salient feature points on the first frame image of the video by using common feature point extraction algorithms such as SURF algorithm, FAST algorithm, Harris corner detection algorithm and BRISK algorithm, and obtaining a group of salient feature point sets by taking and collecting the feature points extracted by all the algorithms
Wherein (x)i，yi) M is the number of the feature points for the selected feature point coordinates;
C. tracking the salient feature points, and tracking the feature points selected in the step A by utilizing a Kanada-Lucas-Tomasi algorithm
Obtaining the motion trail of a group of characteristic points
Wherein n is the frame number of the sequence image, and i is the serial number of the characteristic point;
D. estimating the motion vector field, and after finishing the tracking part, according to the motion track p of the characteristic pointiCalculating the real position of the feature point, and according to the Cox-Munk rule, when the camera observes a point underwater for a long time, the captured point fluctuates by taking the real position as the center, so that the center of the motion track is taken as the real position of the feature point, and the specific formula is as follows:
wherein, the subscript 0 represents the original undistorted image, i.e. the true position of each feature point, and the displacement of each feature point is calculated according to the estimation of the true position of the feature point
Since the 3D motion vector field is subject to sparse decomposition in the discrete Fourier domain, the set of displacements of feature points
Can be regarded as sparse samples in the space-time domain, and displacement is collected
Splicing into a measurement vector upsilon consisting of nM elements, and constructing a solving model based on a compressive sensing theory:
υ＝ΦΨS+η
the purpose of compressed sensing is to restore original signals from sparsely sampled signal points, namely, to restore a sparse coefficient S of the decomposition of the whole motion vector field in a Fourier domain through motion vectors of a small number of characteristic points, and further to obtain the motion vectors of the whole image at all position points;
E. distortion correction is carried out on all sequence images based on the whole motion vector field of the images so as to achieve the effect of eliminating periodic distortion and further obtain an image sequence V only containing random local distortions＝{Is1，Is2，...Isn}；
Aiming at the random distortion caused by random fluctuation, the restoration of an underwater distorted image is realized by adopting a method of reconstructing a high-quality reference image and then carrying out iterative registration, and the method specifically comprises the following steps:
F. image sequence V from which periodic distortion is to be removedsPartitioning into image Block sequences { { Bk}(i，j)And selecting the first 50% image block with smaller distortion degree as a lucky block { { L } by taking the structural similarity SSIM as an indexk}(i，j)The lucky block picked from each position is { { L { (L) }k}(i，j)The fusion takes a single image block { M }(i，j)And splicing the images into a whole image, wherein the SSIM has the following calculation formula:
H. eliminating Gaussian blur in the fused image by using a Richardson-Lucy blind deconvolution algorithm to obtain a reference image R with higher definition, wherein the specific formula is as follows:
I. registering the image sequence with the reference image by using a B-spline-based non-rigid image iterative registration technology, and repeating the steps S6, S7 and S8 by taking the registered image sequence as a new input when the registration result does not meet the iterative termination condition; when the iteration termination condition is met, outputting a restoration result to obtain a distortion-free image sequence Vf＝{If1，If2，…，IfnThe specific formula of non-rigid image registration is:
the invention is further improved in that: aiming at periodic distortion caused by periodic fluctuation, the image is sparsely represented in a Fourier transform domain, and a motion vector field of the whole image is estimated by motion vectors of a small number of characteristic points by utilizing a compressed sensing theory, wherein the method comprises the following steps: detecting feature points in the first frame image of the video by utilizing an SURF algorithm, a FAST algorithm, a Harris corner detection algorithm and a BRISK algorithm, and taking a union set as a finally selected feature point set; tracking the feature points by using a Kanada-Lucas-Tomasi algorithm to obtain the motion vectors of the feature point set
And estimating a motion vector field of the whole image by using a compressed sensing theory for eliminating the periodic distortion.
The invention is further improved in that: in the compressed sensing theory structure solving model, v is a measurement vector constructed by the motion vector of the characteristic point.
The invention is further improved in that: in the compressed sensing theory construction solution model, phi is an observation matrix and is used for representing the position of a selected characteristic point in the whole image.
The invention is further improved in that: in the compressive sensing theory construction solving model, psi and S are sparse coefficients of a Fourier transform basis matrix and sparse representation of an overall motion vector field in a Fourier domain respectively.
The invention is further improved in that: in the compressed sensing theory construction solution model, eta is a noise component.
The invention is further improved in that: in step F, k is the frame number of the image, (i, j) represents the position of the image block in the whole image, and muI，μM，σIM，σI，σMSequentially representing the local mean, covariance and variance of a certain image block and the mean image of the current block sequence, c1，c2Is a constant.
The invention is further improved in that: in step H, g is a distorted noise image, f is an original undistorted image, H is a point spread function,
which represents a convolution operation, the operation of the convolution,
representing the correlation, k is the iteration number of the blind deconvolution, and γ is the optimization function.
The invention is further improved in that: in the step I, the step I is carried out,
b is a function based on B splines and is specifically expressed as
B0(t)＝(1-t)3/6，B1(t)＝(3t3-6t2+4)/6，B2(t)＝(-3t3+3t2+3t +1)/6 and B3(t)＝t3/6。
The invention is further improved in that: the iteration termination condition is as follows:
k is the frame number of the image sequence, omega is the image domain, IkM is the mean value of the current sequence image, and h, w and n are the height, width and frame number of the image respectively.
The invention has the following beneficial effects:
the invention assumes the water surface fluctuation to be a superposition state of periodic fluctuation and random fluctuation according to the underwater imaging characteristics and the fluctuation characteristics of the water surface. For distortion caused by periodic fluctuation, based on a compressed sensing theory, solving a motion vector field of the whole image by using motion vectors of salient feature points in the image so as to achieve the effect of eliminating the periodic distortion; for random local distortion, a new registration strategy is provided, namely, a high-quality reference image is obtained through processing of a lucky block fusion algorithm and a blind deconvolution algorithm, then a sequence image is registered with the reference image, and an image sequence without distortion is restored after multiple iterations. The two-stage underwater image restoration algorithm provided by the invention can realize more stable and more obvious distortion correction effect aiming at the distortion characteristics caused by different types of fluctuation. Meanwhile, as the periodic distortion in the image distortion occupies a main position, the overall effect of the image sequence after the first stage of restoration is obviously improved, and a reference image with better quality is reconstructed before registration, the running speed of the registration algorithm of the second stage is obviously accelerated, and the iteration times required for achieving the established effect are also obviously reduced. Compared with the iterative registration algorithm of the second stage, the calculation time of compressed sensing can be ignored, so that the method realizes great improvement in image restoration effect, algorithm robustness and algorithm operation time, and has very good practical application value.
Drawings
The following examples are presented to enable one of ordinary skill in the art to more fully understand the present invention and are not intended to limit the scope of the embodiments described herein.
FIG. 1 is a flow chart of an underwater distorted image restoration method according to the present invention.
FIG. 2 is a comparison of the reconstruction results in underwater image data of the present invention with other image reconstruction algorithms; FIG. (a) is a warped image of a frame in an underwater image dataset; figure (b) is the sequence mean image after restoration for the iterative image registration algorithm proposed by Oreifej. And (c) is a sequence mean image after the underwater image restoration algorithm provided by the invention restores.
Fig. 3 is a comparison of the number of iterations required for the present invention and the Oreifej iterative registration algorithm to meet the iteration termination condition.
Detailed Description
The following description of the embodiments of the present invention is provided for illustrative purposes, and other advantages and effects of the present invention will become apparent to those skilled in the art from the present disclosure.
Fig. 1 shows an underwater distorted image restoration method, which includes the following steps:
s1, reading the distorted image, reading the underwater distorted video according to the characteristics of the fluctuating water surface under the natural condition, and decomposing the distorted video into an image sequence frame by frame;
s2, selecting the salient feature points, and detecting the salient feature points in the first frame image of the video;
s3, tracking the salient feature points, and tracking the selected feature points in the subsequent image sequence to obtain the motion vectors of the feature points;
s4, estimating the motion vector field, and solving the motion vector field of the whole image by the motion vector of the characteristic point based on the compressed sensing theory;
s5, performing preliminary restoration on the periodic distortion, and correcting the periodic distortion in each frame of image according to the motion vector field obtained in the step S4 to realize the preliminary restoration;
s6, fusing lucky blocks, dividing the image sequence after the initial restoration into image blocks, selecting the image blocks with the first 50% of small distortion degree by taking the structural similarity as an index, and fusing the image blocks into a single image;
s7, blind deconvolution deblurring, and a Richardson-Lucy blind deconvolution algorithm is used for removing Gaussian blur in the fused image and reconstructing a clearer reference image;
s8, performing iterative registration on the non-rigid image, registering the image sequence and the reconstructed reference image by using a non-rigid registration algorithm based on a B spline to achieve the effect of eliminating residual random distortion, and repeating the steps S6, S7 and S8 when the iterative termination condition is not met; when the image sequence is registered with the reconstructed reference image in step S8 and the iteration termination condition is satisfied, the reconstructed undistorted image sequence is output.
The invention assumes the water surface fluctuation to be a superposition state of periodic fluctuation and random fluctuation according to the underwater imaging characteristics and the fluctuation characteristics of the water surface. For distortion caused by periodic fluctuation, based on a compressed sensing theory, solving a motion vector field of the whole image by using motion vectors of salient feature points in the image so as to achieve the effect of eliminating the periodic distortion; for random local distortion, a new registration strategy is provided, namely, a high-quality reference image is obtained through processing of a lucky block fusion algorithm and a blind deconvolution algorithm, then a sequence image is registered with the reference image, and an image sequence without distortion is restored after multiple iterations. The two-stage underwater image restoration algorithm provided by the invention can realize more stable and more obvious distortion correction effect aiming at the distortion characteristics caused by different types of fluctuation. Meanwhile, as the periodic distortion in the image distortion occupies a main position, the overall effect of the image sequence after the first stage of restoration is obviously improved, and a reference image with better quality is reconstructed before registration, the running speed of the registration algorithm of the second stage is obviously accelerated, and the iteration times required for achieving the established effect are also obviously reduced. Compared with the iterative registration algorithm of the second stage, the calculation time of compressed sensing can be ignored, so that the method realizes great improvement in image restoration effect, algorithm robustness and algorithm operation time, and has very good practical application value.
It should be noted that the first stage is a proposed correction scheme for distortion caused by periodic fluctuation, the second stage is for random distortion caused by random fluctuation, and the iterative registration method of the second stage corrects for local distortion remaining in the first stage.
Further, the characteristics of the fluctuating water surface in the natural condition in step S1 are such that the water surface fluctuation is assumed to be a superposition of the periodic fluctuation and the random fluctuation.
In the application, under the environment that water quality is clear, the refraction phenomenon caused by water surface fluctuation is a main cause of geometric distortion of an underwater image. The characteristics of the water surface fluctuation directly determine the characteristics of distortion in the underwater image. Under natural conditions, the water surface which continuously fluctuates has space-time smoothness and periodicity in time and space, and under the influence of the periodic fluctuation, motion vectors of all points on an image have sparse representation in a 3D Fourier domain. However, the water surface under natural conditions cannot avoid turbulence caused by local disturbance or environmental change, and the disturbance and local turbulence destroy the periodicity of the water surface fluctuation and introduce random distortion at local positions of the image. Therefore, it is of practical significance to assume the water surface under natural conditions as a mutual superposition of periodic and random fluctuations.
Furthermore, aiming at the periodic distortion caused by the periodic fluctuation, the image is sparsely represented in a Fourier transform domain, and a motion vector field of the whole image is estimated by motion vectors of a small number of characteristic points by utilizing a compressed sensing theory, wherein the method comprises the following steps: detecting feature points in the first frame image of the video by utilizing an SURF algorithm, a FAST algorithm, a Harris corner detection algorithm and a BRISK algorithm, and taking a union set as a finally selected feature point set; tracking the feature points by using a Kanada-Lucas-Tomasi algorithm to obtain the motion vectors of the feature point set
And estimating a motion vector field of the whole image by using a compressed sensing theory for eliminating the periodic distortion.
Further, for image restoration of periodic distortion caused by periodic fluctuation, the specific method comprises the following steps:
A. reading of distorted images, reading of distorted underwater video and decomposition thereof frame by frame into a sequence of distorted images Vd＝{Id1，Id2，...Idn}；
B. Selecting the salient feature points, subjecting a motion vector field caused by periodic fluctuation to sparse decomposition, namely solving the motion vector field of the whole image through motion vectors of a few points, extracting the salient feature points on the first frame image of the video by using common feature point extraction algorithms such as SURF algorithm, FAST algorithm, Harris corner detection algorithm and BRISK algorithm, and obtaining a group of salient feature point sets by taking and collecting the feature points extracted by all the algorithms
Wherein (x)i，yi) M is the number of the feature points for the selected feature point coordinates;
C. tracking the salient feature points, and tracking the feature points selected in the step A by utilizing a Kanada-Lucas-Tomasi algorithm
Obtaining the motion trail of a group of characteristic points
Wherein n is the frame number of the sequence image, and i is the serial number of the characteristic point;
D. estimating the motion vector field, and after finishing the tracking part, according to the motion track p of the characteristic pointiThe true positions of the feature points are calculated. According to the Cox-Munk rule, when a camera observes a point underwater for a long time, the captured point fluctuates by taking the real position as the center, so that the center of the motion track is taken as the real position of the feature point, and the specific formula is as follows:
wherein the subscript 0 represents the original undistorted image, i.e., eachThe real positions of the feature points, and the displacement of each feature point is calculated according to the estimation of the real positions of the feature points
Since the 3D motion vector field is subject to sparse decomposition in the discrete Fourier domain, the set of displacements of feature points
Can be regarded as sparse samples in the space-time domain, and displacement is collected
Splicing into a measurement vector upsilon consisting of nM elements, and constructing a solving model based on a compressive sensing theory:
υ＝ΦΨS+η
the purpose of compressed sensing is to restore original signals from sparsely sampled signal points, namely, to restore a sparse coefficient S of the decomposition of the whole motion vector field in a Fourier domain through motion vectors of a small number of characteristic points, and further to obtain the motion vectors of the whole image at all position points;
E. distortion correction is carried out on all sequence images based on the whole motion vector field of the images so as to achieve the effect of eliminating periodic distortion and further obtain an image sequence V only containing random local distortions＝{Is1，Is2，...Isn}。
Further, v is a measurement vector constructed by the motion vector of the feature point.
Further, Φ is an observation matrix for representing the positions selected as the feature points in the whole image.
Further, Ψ and S are sparse coefficients for sparse representation of the fourier transform basis matrix and the overall motion vector field in the fourier domain, respectively.
Further, η is a noise component.
In the application, because the water surface fluctuation under natural conditions is difficult to avoid the occurrence of random disturbance, the iterative registration method at the second stage corrects the residual local distortion at the first stage, the difference of registration strategies directly influences the effect and speed of iterative registration, previous scholars such as Oreifej and the like take a sequence mean value as a reference image, and Gaussian blur is introduced into a distorted sequence image before registration, and the theoretical support of the strategy lies in that: when the reference image is at the same blur level as the sequence image, the registration will be directed to a relatively sharp portion. However, artificially introducing blur into the sequence images may cause irreversible image information loss, thereby limiting the effectiveness of registration. Therefore, the invention provides another image registration strategy, namely reconstructing a reference image with higher quality and clearer quality, and then registering the sequence image with the reference image. As shown in fig. 2 and fig. 3, since the overall sharpness of the reconstructed reference image is higher, and each point of the image is closer to its true position, the number of iterations required is significantly reduced, and meanwhile, the registration is guided to a relatively sharp part of the reference image, which also has the effect of speeding up the registration. Furthermore, past scholars have demonstrated that: in the underwater distorted image sequence, the positions of all points are in Gaussian distribution by taking the real positions as the centers, and the average value image obtained by averaging in the image sequence is in Gaussian blur as a whole. Therefore, on the premise of the known fuzzy form, the blind deconvolution algorithm can realize a very obvious deblurring effect, and further improve the definition degree of the reference image.
Aiming at the random distortion caused by random fluctuation, the restoration of an underwater distorted image is realized by adopting a method of reconstructing a high-quality reference image and then carrying out iterative registration, and the method specifically comprises the following steps:
F. lucky block fusion, namely eliminating the image sequence V after periodic distortionsPartitioning into image Block sequences { { Bk}(i，j)And selecting the first 50% image block with smaller distortion degree as a lucky block { { L } by taking the structural similarity SSIM as an indexk}(i，j)The lucky block picked from each position is { { L { (L) }k}(i，j)The fusion takes a single image block { M }(i，j)And splicing the images into a whole image, SSIThe formula for M is:
H. blind deconvolution deblurring, and a Richardson-Lucy blind deconvolution algorithm is used for eliminating Gaussian blur in the fused image to obtain a reference image R with higher definition, wherein the specific formula is as follows:
I. performing non-rigid image iterative registration, registering the image sequence with a reference image by using a non-rigid image iterative registration technology based on a B spline, and repeating the steps S6, S7 and S8 by taking the registered image sequence as new input when a registration result does not meet an iteration termination condition; when the iteration termination condition is met, outputting a restoration result to obtain a distortion-free image sequence Vf＝{If1，If2，…IfnThe specific formula of non-rigid image registration is:
further, k is the frame number of the image, and (i, j) represents the position of the image block in the whole image.
Further, in step F, the μI，μM，σIM，σI，σMSequentially representing the local mean, covariance and variance of a certain image block and the mean image of the current block sequence, c1，c2 is a constant.
Further, in step H, g is a distorted noise image, f is an original undistorted image, H is a point spread function,
which represents a convolution operation, the operation of the convolution,
and expressing the correlation, wherein k is the iteration number of blind deconvolution, and gamma is an optimization function.
Further, in the step I,
further, B is a function based on B spline, and is specifically expressed as
B0(t)＝(1-t)3/6，B1(t)＝(3t3-6t2+4)/6，B2(t)＝(-3t3+3t2+3t +1)/6 and B3(t)＝t3/6。
Further, the iteration termination condition is as follows:
the invention is further improved in that: k is the frame number of the image sequence, Ω is the image field, IkM is the mean value of the current sequence image, and h, w and n are the height, width and frame number of the image respectively.
The above embodiments are only preferred embodiments of the present invention, and are not intended to limit the technical solutions of the present invention, so long as the technical solutions can be realized on the basis of the above embodiments without creative efforts, which should be considered to fall within the protection scope of the patent of the present invention.
Claims (10)
1. An underwater distorted image restoration method is characterized by comprising the following steps,
s1, reading the distorted image, reading the underwater distorted video according to the characteristics of the fluctuating water surface under the natural condition, and decomposing the distorted video into an image sequence frame by frame;
s2, selecting the salient feature points, and detecting the salient feature points in the first frame image of the video;
s3, tracking the salient feature points, and tracking the selected feature points in the subsequent image sequence to obtain the motion vectors of the feature points;
s4, estimating the motion vector field, and solving the motion vector field of the whole image by the motion vector of the characteristic point based on the compressed sensing theory;
s5, performing preliminary restoration on the periodic distortion, and correcting the periodic distortion in each frame of image according to the motion vector field obtained in the step S4 to realize the preliminary restoration;
s6, fusing lucky blocks, dividing the image sequence after the initial restoration into image blocks, selecting the image blocks with the first 50% of small distortion degree by taking the structural similarity as an index, and fusing the image blocks into a single image;
s7, blind deconvolution deblurring, and a Richardson-Lucy blind deconvolution algorithm is used for removing Gaussian blur in the fused image and reconstructing a clearer reference image;
s8, performing iterative registration on the non-rigid image, registering the image sequence and the reconstructed reference image by using a non-rigid registration algorithm based on a B spline to achieve the effect of eliminating residual random distortion, and repeating the steps S6, S7 and S8 when the iterative termination condition is not met; when the image sequence is registered with the reconstructed reference image in the step S8 and the iteration termination condition is satisfied, outputting the restored undistorted image sequence;
in step S1, the water surface fluctuation is assumed to be a superposition of periodic fluctuation and random fluctuation according to the characteristics of the fluctuating water surface under natural conditions;
the image restoration method aiming at the periodic distortion caused by the periodic fluctuation comprises the following steps: A. reading of distorted images, reading of distorted underwater video and decomposition thereof frame by frame into a sequence of distorted images Vd＝{Id1，Id2，...Idn}；
B. The selection of the salient feature points, the motion vector field caused by the periodic fluctuation obeys the sparse decomposition, namely the motion vector field of the whole image can be solved by the motion vectors of a few points, and common feature point extraction algorithms such as SURF algorithm, FAST algorithm, Harris angular point detection algorithm and BRISK algorithm are utilizedExtracting the salient feature points from the first frame image of the video, and collecting the feature points extracted by all algorithms to obtain a group of salient feature point sets
Wherein (x)i，yi) M is the number of the feature points for the selected feature point coordinates;
C. tracking the salient feature points, and tracking the feature points selected in the step A by utilizing a Kanada-Lucas-Tomasi algorithm
Obtaining the motion trail of a group of characteristic points
Wherein n is the frame number of the sequence image, and i is the serial number of the characteristic point;
D. estimating the motion vector field, and after finishing the tracking part, according to the motion track p of the characteristic pointiCalculating the real position of the feature point, and according to the Cox-Munk rule, when the camera observes a point underwater for a long time, the captured point fluctuates by taking the real position as the center, so that the center of the motion track is taken as the real position of the feature point, and the specific formula is as follows:
wherein, the subscript 0 represents the original undistorted image, i.e. the true position of each feature point, and the displacement of each feature point is calculated according to the estimation of the true position of the feature point
Since the 3D motion vector field is subject to sparse decomposition in the discrete Fourier domain, the set of displacements of feature points
Can be regarded as sparse samples in the space-time domain, and displacement is collected
Splicing into a measurement vector upsilon consisting of nM elements, and constructing a solving model based on a compressive sensing theory:
υ＝ΦΨS+η
the purpose of compressed sensing is to restore original signals from sparsely sampled signal points, namely, to restore a sparse coefficient S of the decomposition of the whole motion vector field in a Fourier domain through motion vectors of a small number of characteristic points, and further to obtain the motion vectors of the whole image at all position points;
E. distortion correction is carried out on all sequence images based on the whole motion vector field of the images so as to achieve the effect of eliminating periodic distortion and further obtain an image sequence V only containing random local distortions＝{Is1，Is2，...Isn}；
Aiming at the random distortion caused by random fluctuation, the restoration of an underwater distorted image is realized by adopting a method of reconstructing a high-quality reference image and then carrying out iterative registration, and the method specifically comprises the following steps:
F. image sequence V from which periodic distortion is to be removedsPartitioning into image Block sequences { { Bk}(i，j)And selecting the first 50% image block with smaller distortion degree as a lucky block { { L } by taking the structural similarity SSIM as an indexk}(i，j)The lucky block picked from each position is { { L { (L) }k}(i，j)The fusion takes a single image block { M }(i，j)And splicing the images into a whole image, wherein the SSIM has the following calculation formula:
H. eliminating Gaussian blur in the fused image by using a Richardson-Lucy blind deconvolution algorithm to obtain a reference image R with higher definition, wherein the specific formula is as follows:
I. registering the image sequence with the reference image by using a B-spline-based non-rigid image iterative registration technology, and repeating the steps S6, S7 and S8 by taking the registered image sequence as a new input when the registration result does not meet the iterative termination condition; when the iteration termination condition is met, outputting a restoration result to obtain a distortion-free image sequence Vf＝{If1，If2，...IfnThe specific formula of non-rigid image registration is:
2. the method for restoring the underwater distorted image according to claim 1, wherein the image is sparsely represented in a Fourier transform domain aiming at the periodic distortion caused by the periodic fluctuation, and a motion vector field of the whole image is estimated from motion vectors of a small number of feature points by using a compressed sensing theory, and the method comprises the following steps: detecting feature points in the first frame image of the video by utilizing an SURF algorithm, a FAST algorithm, a Harris corner detection algorithm and a BRISK algorithm, and taking a union set as a finally selected feature point set; tracking the feature points by using a Kanada-Lucas-Tomasi algorithm to obtain the motion vectors of the feature point set
And estimating a motion vector field of the whole image by using a compressed sensing theory for eliminating the periodic distortion.
3. The method for restoring the underwater distorted image as claimed in claim 2, wherein in the compressive sensing theory, v is a measurement vector constructed by a motion vector of a characteristic point in a solution model.
4. The method as claimed in claim 3, wherein in the compressed sensing theory construction solution model, Φ is an observation matrix used for representing the position of the selected feature point in the whole image.
5. The method for restoring the underwater distorted image as claimed in claim 4, wherein in the compressive sensing theory construction solution model, Ψ and S are sparse coefficients of a Fourier transform basis matrix and a sparse representation of an overall motion vector field in a Fourier domain respectively.
6. The method for restoring the underwater distorted image as claimed in claim 5, wherein η is a noise component in the compressive sensing theory construction solution model.
7. The method for restoring the underwater distorted image as claimed in claim 1, wherein in step F, k is the frame number of the image, (i, j) represents the position of the image block in the whole image, and μI，μM，σIM，σI，σMSequentially representing the local mean, covariance and variance of a certain image block and the mean image of the current block sequence, c1，c2Is a constant.
8. The method for underwater distorted image restoration according to claim 7, wherein in step H, g is a distorted noise image, f is an original undistorted image, H is a point spread function,
which represents a convolution operation, the operation of the convolution,
representing the correlation, k is the iteration number of the blind deconvolution, and γ is the optimization function.
9. The method for underwater distorted image restoration according to claim 8, wherein in the step I,
b is a function based on B splines and is specifically expressed as
B0(t)＝(1-t)3/6，B1(t)＝(3t3-6t2+4)/6，B2(t)＝(-3t3+3t2+3t+1)/6
And B3(t)＝t3/6。
10. A method for underwater distorted image restoration according to claim 9, wherein the iteration termination condition is:
k is the frame number of the image sequence, omega is the image domain, IkM is the mean value of the current sequence image, and h, w and n are the height, width and frame number of the image respectively.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110855868.XA CN113592738B (en) | 2021-07-28 | 2021-07-28 | Underwater distorted image restoration method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110855868.XA CN113592738B (en) | 2021-07-28 | 2021-07-28 | Underwater distorted image restoration method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113592738A true CN113592738A (en) | 2021-11-02 |
| CN113592738B CN113592738B (en) | 2024-04-23 |
Family
ID=78251166
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110855868.XA Active CN113592738B (en) | 2021-07-28 | 2021-07-28 | Underwater distorted image restoration method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113592738B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114022385A (en) * | 2021-11-07 | 2022-02-08 | 桂林电子科技大学 | Image restoration method based on local surface fitting |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111696049A (en) * | 2020-05-07 | 2020-09-22 | 中国海洋大学 | Deep learning-based underwater distorted image reconstruction method |
| KR20210084806A (en) * | 2019-12-27 | 2021-07-08 | 주식회사 비즈모델라인 | Method for Restoring Distortion Video Based on Iris Recognition |
-
2021
- 2021-07-28 CN CN202110855868.XA patent/CN113592738B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20210084806A (en) * | 2019-12-27 | 2021-07-08 | 주식회사 비즈모델라인 | Method for Restoring Distortion Video Based on Iris Recognition |
| CN111696049A (en) * | 2020-05-07 | 2020-09-22 | 中国海洋大学 | Deep learning-based underwater distorted image reconstruction method |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114022385A (en) * | 2021-11-07 | 2022-02-08 | 桂林电子科技大学 | Image restoration method based on local surface fitting |
| CN114022385B (en) * | 2021-11-07 | 2024-03-26 | 桂林电子科技大学 | Image restoration method based on local surface fitting |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113592738B (en) | 2024-04-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106920220B (en) | The turbulent flow method for blindly restoring image optimized based on dark primary and alternating direction multipliers method | |
| CN112801900B (en) | Video blurring removal method for generating countermeasure network based on bidirectional circular convolution | |
| CN110675347B (en) | Image blind restoration method based on group sparse representation | |
| CN104103050B (en) | A kind of real video restored method based on local policy | |
| CN109345474A (en) | Image motion based on gradient field and deep learning obscures blind minimizing technology | |
| CN111489303A (en) | Maritime affairs image enhancement method under low-illumination environment | |
| CN102208100A (en) | Total-variation (TV) regularized image blind restoration method based on Split Bregman iteration | |
| CN110796616B (en) | Turbulence degradation image recovery method based on norm constraint and self-adaptive weighted gradient | |
| CN107610069B (en) | DVS visual video denoising method based on shared K-SVD dictionary | |
| CN109146797B (en) | A kind of impulsive noise ancient book image inpainting method sparse based on Lp pseudonorm and overlapping group | |
| CN106251297A (en) | A kind of estimation based on multiple image fuzzy core the rebuilding blind super-resolution algorithm of improvement | |
| CN114331886B (en) | Image deblurring method based on depth features | |
| CN102236887A (en) | Motion-blurred image restoration method based on rotary difference and weighted total variation | |
| CN113592738B (en) | Underwater distorted image restoration method | |
| Wang et al. | RT-Deblur: Real-time image deblurring for object detection | |
| CN111899196A (en) | Blade defect motion blurred image restoration method based on classical restoration algorithm | |
| CN116523790A (en) | SAR image denoising optimization method, system and storage medium | |
| CN116664414A (en) | Unified image defogging and denoising method based on unsupervised learning | |
| CN112862729B (en) | Remote sensing image denoising method based on characteristic curve guidance | |
| CN112767261B (en) | Non-local denoising frame for color image and video based on generalized non-convex tensor robust principal component analysis model | |
| CN110136164A (en) | Method based on online transitting probability, low-rank sparse matrix decomposition removal dynamic background | |
| Chen et al. | A deep motion deblurring network using channel adaptive residual module | |
| Zeng et al. | Single image motion deblurring based on modified denseNet | |
| Jian et al. | Computer image recognition and recovery method for distorted underwater images by structural light | |
| CN113538290A (en) | Agricultural aerial image processing method and system based on artificial intelligence |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |