CN107909548A - A kind of video and removes rain method based on noise modeling - Google Patents
A kind of video and removes rain method based on noise modeling Download PDFInfo
- Publication number
- CN107909548A CN107909548A CN201710992669.7A CN201710992669A CN107909548A CN 107909548 A CN107909548 A CN 107909548A CN 201710992669 A CN201710992669 A CN 201710992669A CN 107909548 A CN107909548 A CN 107909548A
- Authority
- CN
- China
- Prior art keywords
- rain
- video
- model
- foreground
- noise
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 58
- 238000001514 detection method Methods 0.000 claims abstract description 11
- 239000011159 matrix material Substances 0.000 claims description 23
- 230000006870 function Effects 0.000 claims description 22
- 238000005457 optimization Methods 0.000 claims description 13
- 238000000354 decomposition reaction Methods 0.000 claims description 10
- 238000013179 statistical model Methods 0.000 claims description 10
- 238000007476 Maximum Likelihood Methods 0.000 claims description 9
- 239000000203 mixture Substances 0.000 claims description 6
- 239000013598 vector Substances 0.000 claims description 6
- 230000003068 static effect Effects 0.000 claims description 5
- 230000003190 augmentative effect Effects 0.000 claims description 4
- 230000000295 complement effect Effects 0.000 claims description 4
- 238000002156 mixing Methods 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 abstract description 4
- 238000004458 analytical method Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Probability & Statistics with Applications (AREA)
- Image Processing (AREA)
Abstract
A kind of video and removes rain method based on noise modeling, under the hypothesis of low-rank background, estimation is carried out at the same time to the rain bar noise contribution in video and mobile prospect.First, the video data containing rain noise, initialization model are obtained；Rain map generalization model is established according to the characteristics of rain noise and video foreground；The rain bar that the raindrop of the architectural characteristic being imaged in video according to rain --- movement are formed on each fritter in the picture has consistent directionality, establishes the fritter prior distribution on rain bar；According to video foreground it is openness the characteristics of establish moving Object Detection model；It is to remove model to rain in maximal possibility estimation frame by model conversation；Using video containing rain and rain model is removed, obtains rain video and other statistical variables, rain video is removed in output.It is contemplated that establishing the high-quality video based on rain figure generating principle and rain bar noise structure characteristic removes rain model, and then more accurately so that video and removes rain technology can be widely applied to the complexity containing the prospect of movement and rain in scene.
Description
Technical Field
The invention relates to a video image processing technology for outdoor shooting images, in particular to a video rain removing method based on noise modeling.
Background
Because the shooting quality of the outdoor shooting system is often affected by severe weather (such as rain, snow and fog), details of a shot video or image are damaged, textures are blurred, and a background part is shielded by high-brightness raindrops and raindrops, so that the shot image cannot be used for further processing operations, such as feature extraction, target identification and the like. Therefore, video image rain and snow removal is a technology that has been recently developed in the field of computer vision. On the premise of keeping the details of the video image, the rain removing technology preprocesses the damaged video image, and recovers the quality of the affected video image to the maximum extent, thereby allowing a computer vision algorithm to further analyze the video image.
Rain removal techniques can be summarized in two broad categories, frequency domain information based analysis methods and time domain information based analysis methods. Transforming the image to a frequency domain space based on an analysis method of frequency domain information, and removing high-frequency components in the image which is taken as rain; the analysis method based on the time domain information mainly utilizes several types of characteristics of rain in the image, such as brightness characteristics, shape characteristics, color characteristics, space characteristics and the like.
In the video rain removing method, a method analyzes the distribution rule of rain in adjacent frames of an image based on the color characteristic of raindrops, and because the raindrops have the effect of improving the background brightness, the method mainly utilizes the pixel difference between the adjacent frames to judge whether the pixel point is covered by the rain, but the method cannot be applied to heavy rain scenes and scenes with moving prospects. Since rain and moving objects have similar edge characteristics, distinguishing raindrops from moving prospects has been a key to the problem of rain removal. To overcome such difficulties, a number of rain removal methods have been proposed in succession: one is a probability-based rain removing method, which distinguishes raindrops and moving objects by comparing the rule of the fluctuation of pixel values caused by raindrops and the rule of the fluctuation of pixel values caused by moving objects; one is a method for modeling the raindrops based on structural characteristics, such as dynamic characteristic modeling of forming the raindrops according to raindrop movement described by a random field, and constrained modeling according to an optical model of the raindrops and the sizes and direction angles of the raindrops; one method is to perform a raindrop initial detection first, and then perform post-processing (such as image-guided filtering, SVM feature classification, etc.) on the initial detection result to distinguish raindrops from moving prospects.
The prior art generally considers rain as a deterministic object, and realizes detection and separation in a video by means of characterizing typical features and constructing the deterministic object or learning the discriminative information of the deterministic object different from non-rain images. On one hand, the method focuses on the structural information depiction of rain in the video, and does not fully consider other information in the video, such as the prior structural knowledge of a foreground object and a background scene, so that the complementary action of the beneficial structural knowledge of the non-rain part of the video on the problem of rain removal is not utilized; on the other hand, in order to obtain the special structure information of rain, many video rain removing methods (especially the more recent methods) need to externally construct a rain/rain-free labeled discrimination database to learn the structure of rain, and this information is often difficult to obtain for the rain-carrying video with a specific structure in practice.
Disclosure of Invention
The invention aims to provide a video rain removing method based on noise modeling.
In order to achieve the purpose, the invention adopts the technical scheme that:
step S1: obtaining the original video, namely the rain video D epsilon R mn×T Wherein m, n represents the length and width of the video, and T represents the number of video frames), initializing model variables and parameters;
step S2: establishing a statistical model generated by rain strips according to the foreground, the background and the rain-containing noise of the original video;
and step S3: based on a low-rank matrix decomposition method, according to the distribution rule and the directional characteristic of the rain strips on the small blocks in a statistical model generated by the rain strips, a maximum likelihood estimation method is utilized to construct a small block prior rain removal model under the condition of no moving foreground;
and step S4: constructing a moving object detection model according to the structural characteristics of the video foreground support;
step S5: combining the steps S3 and S4, constructing a comprehensive model for alternately optimizing the moving foreground and the rain strip, and establishing a rain removing algorithm based on noise modeling under the moving foreground;
step S6: and (4) taking the original rain video obtained in the step (S1) as input, and applying the rain removing algorithm based on the noise modeling under the moving foreground in the step (S5) to obtain the rain removing video and other statistical variables.
In the step S2, a statistical model generated by the rain strip is established according to the foreground, the background and the rain noise of the video:
D＝H⊙D+H ⊥ ⊙D
f(H ⊥ ⊙D)＝f(UV T )+E
wherein D is the input video, H is the same as (0,1) } mn×T For supporting the moving foreground of the video, the moving foreground is expanded along the length and the width of the video, and the equivalent tensor expression H epsilon {0,1} is correspondingly obtained m×n×T It is defined as:
that is, H takes a value of 1 at a pixel point with a moving foreground in the video, takes a value of 0 at other places, and records H ⊥ Represents the orthogonal complement of H, i.e.: h + H ⊥ ＝1，H ⊥ D represents a portion of the original video without moving foreground;
u, V is a low rank decomposition of video background, i.e. U is in the form of R mn×r ,V∈R T×4 Is a low rank matrix r of rank r&Min (mn, T), which is the hadamard product operator, means that the corresponding elements of the matrix are multiplied one by one, i.e.: h | _ D is still a matrix of the same size as H and D, each element of which is H, and the corresponding element of D is multiplied to obtain E is the noise of the rainbars on each frame of video after being cut into small blocks, E is the noise of the rainbars on each frame of video after being cut into small blocks i The ith column is E and represents a rain strip on the ith small block, and the f is a cutting operator which has the functions of selecting small blocks in each frame picture of the original video according to the specified interval and size, drawing each small block into column vectors and splicing the column vectors into a matrix;representing the distribution satisfied by the rain strip noise on the ith patch, according to the directional characteristic of the rain strip, a high-dimensional Gaussian mixture distribution is assumed here, and the form of K Gaussian mixture components is as follows:
wherein the mean vector of the k-th Gaussian mixture component is mu k The covariance matrix is sigma k In a mixing ratio of pi k
In the step S3, based on a low-rank matrix decomposition method, a small block prior rain removal model under a no-movement foreground condition is constructed by using a distribution rule and a directional characteristic of a rain strip on a small block and a maximum likelihood estimation method, and is expressed as a probability form:
the likelihood function of the source data is
Taking its logarithmic form:
wherein N (f (H) ⊥ ⊙(D-UV T )) n |μ k ,∑ k ) Defined by formula (1);
the small block prior rain removal model under the no-movement prospect converted by the statistical model is an optimization problem as follows:
in the step S4, according to the structural characteristics of the video foreground support, the following moving object detection models are constructed for distinguishing the video foreground target:
where D is the input video, B is the background part in the video, H is the video foreground support, l (·,) is a loss function used to measure the similarity between two segments of video, H | _ D is the video moving foreground part.
In step S5, a foreground detection model with a foreground support and a static background rain removing method under a maximum likelihood frame are introduced, and a rain removing model based on noise modeling is an optimized model as follows:
in the step S6, an EM algorithm is adopted to solve the rain removal model formula (3) based on the noise modeling in the step S5, and the specific steps comprise:
s4.1) E step: likelihood probability of updating model hidden variables
Introducing an implicit variable z nk ∈{0,1}，z nk Is an indicator variable for judging whether the noise of the rain strip on the nth small block belongs to the kth mixed Gaussian class or not, and has
Updating the nth tile E n Probability of belonging to the kth mixed gaussian:
s4.2) M step: maximizing the expectation of the full data likelihood function with respect to the hidden variable, i.e., minimizing the reciprocal of the expectation, has the objective function of:
where θ represents the parameter set: theta = [ pi, mu, sigma ]]；θ old Representing the value of theta updated in the last iteration,
for the convenience of solving, a cut variable L is introduced, and the above optimization problem is equivalent to:
s.t.L＝f(H⊙D+H ⊥ οUV T )。
in the step S6, the small block prior rain removal model formula (3) in the step S3 is solved by adopting an EM algorithm, and the objective function in the step M is solved by adopting an alternating direction multiplier method, and the specific steps comprise:
s4.2.1) gives the augmented Lagrangian function of equation (4)
Wherein Λ is a multiplier and μ is a number greater than 0;
s4.2.2) establishes the iteration format and termination condition of the alternative direction multiplier method:
μ k+1 ＝ρμ k (11)
where p is a positive constant greater than 1, typically set to p =1.05,
the iteration termination condition is as follows:
s4.2.3) solving the problems (5), (6), (7), (8) and (9) to give a specific iterative formula;
s4.2.4) sets the initial value of the iteration to: h 0 ＝0，U 0 ,V 0 Generated by the well-known singular value decomposition method acting on D, with an initial Gaussian mean value set to 0 and an initial covariance matrixIs obtained by symmetrical orthogonalization of a random matrix;
s4.2.5) performs the iterative operations of (5) - (11) until the iteration satisfies the termination condition (the termination condition is that the likelihood function descent rate is less than the threshold or the iteration number reaches the upper limit).
The equation (5) solves the following problem:
since the elements in H consist of 0,1, the above optimization problem is equivalent to the following convex optimization problem:
the problem is regarded as a first-order binary Markov random field problem, and H is solved by a graph cut algorithm;
the equation (6) is to solve the following problem:
the optimization problem can be divided, L can be solved as follows:
the equation (7) solves the following problem:
equivalent to solving:
is equivalent to:
this is a weighted two-norm problem, where f -1 Representing an operator for reducing the small blocks into the video, wherein W is a weight tensor which has the same size as the input video, and the value of each point in W is equal to the number of the small blocks overlapped at the point in the original video;
rewrite equation (12) as:
u, V can be iteratively solved to approximate the optimum:
is a weighted two-norm problem in which U t ,V t Solving the U, V value obtained by the iterative computation of the t step according to the rows as follows:
the equation (8) solves the following problem:
the solution is as follows:
wherein:
on one hand, the random dynamic structure of rain is effectively encoded based on a noise modeling principle, and on the other hand, the sparse and blocky structure characteristics of a foreground target in a video with rain and the low-rank characteristic of a background scene are fully utilized to form an effective complementary effect on extracting the rain information of the video, so that effective video rain removal is realized. In particular, the invention utilizes the mode of pertinently modeling different objects of the video to eliminate the dependence of the video rain removing method on the image data set with rain or without rain in advance, so that the video rain removing method can complete effective video rain removing effect under the unsupervised condition.
Compared with the traditional rain removing method, the method has more universality on rain modeling. The rain and the moving foreground are optimized in the model at the same time, so that the learning accuracy of the rain and the moving foreground is promoted, the component learning of the rain is more accurate, and the edge information of the moving object is better reserved.
Drawings
The invention is further illustrated by means of the attached drawings, the contents of which are not in any way limitative of the invention.
FIG. 1 is a flow chart of the present invention.
Fig. 2 shows (a) the real raining video data used in example 1, (b) the raining video restored by the novel video raining method based on noise modeling, in the same frame of the video, and the image in the lower right red frame is the result of enlarging the red frame mark part in the original image by two times.
Fig. 3 is a rain map obtained using the novel video rain removal method based on noise modeling in example 1.
Fig. 4 (a), (b), and (c) show three layers of small mixed gaussian components extracted from the obtained rain map.
Fig. 5 (a), (b), and (c) are covariance matrices corresponding to three gaussian components in a rain map. It can be seen that the covariance matrix of each component is interpretable: the first component corresponds to the relatively sparse raindrops farther from the lens in the original video; the second component corresponds to the dense rain strips closer to the lens in the original video, and the covariance matrix of the third component is close to the diagonal matrix, i.e., the correlation of each point in the component is not strong, and corresponds to the camera noise in the original video.
Fig. 6 is (a) the artificially rained video data used in example 2, and (b) the rained video recovered using a novel video raining method based on noise modeling, in the same frame of video. The rain chart of the artificial addition is from a rain video with a black background shot by a static camera under a real scene, the rain in the rain section has hierarchy, and the size of the rain strip is different according to the distance from the lens.
Fig. 7 is a rain map obtained using the novel video rain removal method based on noise modeling in example 2.
Fig. 8 is a moving foreground plot obtained using the novel video de-raining method based on noise modeling in example 2.
Fig. 9 is a background plot obtained using the novel video rain removal method based on noise modeling in example 2.
Detailed Description
The invention is further described with reference to the following examples.
Example 1
As the experimental object of the present invention, the raining video data as shown in fig. 2 (a), which is a real raining video without moving objects photographed in a static scene, is used. The video data size is 288 × 368 × 171, the mixed fraction of the small block mixed gaussians is taken as 3, the maximum iteration step number is 40, the gaussians block size is 2 × 2, and the background rank is 2.
Referring to fig. 1, the process is as follows:
s1, reading an original video, and initializing each statistical variable and parameter of a model;
s2, establishing a statistical model generated by rain strips according to the characteristics of the foreground, the background and the rain noise of the video;
D＝H⊙D+H ⊥ ⊙D
f(H ⊥ ⊙D)＝f(H ⊥ ⊙UV T )+E
wherein D is input video data, H is a support of a video foreground, E is noise containing rain, the ith column represents the noise on the ith small block in the video after being converted into the small block, and the high-dimensional Gaussian distribution formula is defined by (1).
And step S3: based on a low-rank matrix decomposition method, a small block prior rain removal model under the condition of no moving foreground is constructed by utilizing the distribution rule and the structural characteristics of rain strips on small blocks and a maximum likelihood estimation method;
and step S4: since there are no moving objects in the input video used in this example, the support H can be made zero.
Step S5: and combining the steps S3 and S4 to construct the following raining model:
f(D)＝f(UV T )+E
the idea of maximum likelihood is utilized, which is equivalent to the following optimization model:
step S6: based on the video data input in step S1 and the model parameters set therein, the rain removal model in step S5 is solved to obtain the processing result of fig. 2 (b).
For the convenience of solving, a cut variable L is introduced, and the above optimization problem is equivalent to:
s.t.L＝f(UV T )
the augmented Lagrangian function is:
wherein Λ is a multiplier and μ is a number greater than 0;
the solution process uses the following iterative format:
μ k+l ＝ρμ k (11)
where ρ is a positive constant greater than 1, typically set to ρ =1.05, iteration details are given below:
A. let H =0 in this example, so H is not updated here.
B. (6) The following problem is solved by the equation:
solving the problem with an explicit solution as follows:
l is often too large because of the column count of L n The calculation of (2) is time-consuming, and can be accelerated by the following approximate solution of the FISTA algorithm:
the objective function of FISTA is:
wherein L is (i) The value of L obtained in the previous step. The solution to the FISTA objective function is:
C. (7) The following problem is solved:
this problem is equivalent to:
iterative solution is performed on U, V:
this is a weighted two-norm problem whose solution is:
D. (8) The following problem is solved by the equation:
the solution is as follows:
when the iteration reaches the termination condition, the rainless part, namely the background UV is obtained T (FIG. 2 (b))), the rain layer is D-UV T (FIG. 3). The three Gaussian components are weighted by gamma block by block nk The three rain layers are extracted and combined to obtain the corresponding three rain layers (fig. 4 (a), (b), (c)), and the covariance matrices thereof correspond to fig. 5 (a), (b), (c).
Example 2
As the experimental subject of the present invention, the raining video data as shown in fig. 6 (a)) is used, which is a real raining video with a moving foreground (person, car) photographed in a static scene. The video data size is 240 × 360 × 119, the blending score of the small block blended gaussians is 3, the maximum iteration step number is 40, and the gaussians block size is 2 × 2.
S1, reading an original video, and initializing each statistical variable and parameter of a model;
s2, establishing a statistical model generated by rain strips according to the characteristics of the foreground, the background and the rain noise of the video;
D＝H⊙D+H ⊥ ⊙D
f(H ⊥ ⊙D)＝f(H ⊥ ⊙UV T )+E
wherein D is input video data, H is a support of a video foreground, E is noise containing rain, the ith column represents the noise on the ith small block in the video after being converted into the small block, and the high-dimensional Gaussian distribution formula is defined by (1).
And step S3: based on a low-rank matrix decomposition method, a small block prior rain removal model under the condition of no moving foreground is constructed by utilizing the distribution rule and the structural characteristics of rain strips on small blocks and a maximum likelihood estimation method;
and step S4: constructing a moving object detection model according to the structural characteristics of the video foreground support;
step S5: combining the steps S3 and S4, constructing a comprehensive model for alternately optimizing the moving foreground and the rain strip, and establishing a small block prior rain removal model under the moving foreground;
step S6: solving the rain removing model in the step S5 based on the video data input in the step S1 and the model parameters set by the video data;
for the convenience of solving, a cut variable L is introduced, and the above optimization problem is equivalent to:
s.t.L＝f(H⊙D+H ⊥ ⊙UV T )
the augmented Lagrangian function is:
wherein Λ is a multiplier and μ is a number greater than 0;
the solving process adopts the following iterative format:
μ k+1 ＝ρμ k (11)
where ρ is a positive constant greater than 1, typically set to ρ =1.05, iteration details are given below:
A. (5) The following problem is solved by the equation:
the problem can be solved using a graph cut algorithm package.
B. (6) The following problem is solved:
can be solved according to the following steps:
l is often too large because of the column count of L n The calculation of (A) is more time-consuming, and the following FISTA algorithm can be used for approximate solution
And (3) solving and accelerating:
the objective function of FISTA is:
wherein L is (i) The value of L obtained in the previous step. The solution to the FISTA objective function is:
C. (7) The following problem is solved by the equation:
this problem is equivalent to:
iterative solution is performed on U, V:
this is a weighted two-norm problem whose solution is:
D. (8) The following problem is solved:
the solution is as follows:
when the iteration reaches the end condition, a no-rain part is obtained as H ^ D + H ⊥ ⊙UV T (FIG. 6 (b))), the rain layer is D-H [ < D-H > ] ⊥ ⊙UV T (FIG. 7), the moving object layer is H [ < D > (FIG. 8), the background layer is H ⊥ ⊙UV T (FIG. 9)).
Claims (8)
1. A video rain removing method based on noise modeling is characterized by comprising the following steps:
step S1: obtaining the original video, namely the rain video D belongs to R mn×T Wherein m and n represent the length and width of a video, and T represents the number of video frames), initializing model variables and parameters;
step S2: establishing a statistical model generated by rain strips according to the foreground, the background and the rain-containing noise of the original video;
and step S3: based on a low-rank matrix decomposition method, according to the distribution rule and the directional characteristic of the rain strips on the small blocks in a statistical model generated by the rain strips, a maximum likelihood estimation method is utilized to construct a small block prior rain removal model under the condition of no moving foreground;
and step S4: constructing a moving object detection model according to the structural characteristics of the video foreground support;
step S5: combining the steps S3 and S4, constructing a comprehensive model for alternately optimizing the moving foreground and the rain strip, and establishing a rain removing algorithm based on noise modeling under the moving foreground;
step S6: and (4) taking the original rain video obtained in the step (S1) as input, and applying the rain removing algorithm based on the noise modeling under the moving foreground in the step (S5) to obtain the rain removing video and other statistical variables.
2. The video rain removal method based on noise modeling according to claim 1, characterized in that: in the step S2, a statistical model generated by the rain strip is established according to the foreground, the background and the rain noise of the video:
D＝H⊙D+H ⊥ ⊙D
f(H ⊥ ⊙D)＝f(UV T )+E
wherein D is the input video, H is the same as (0,1) } mn×T For the moving foreground support of the video, the moving foreground support is expanded along the length and the width of the video, and equivalent tensor expression of the moving foreground support is correspondingly obtainedIt is defined as:
that is, the value of H is 1 at the pixel point with moving foreground in the video, the values of other places are 0, and H is recorded ⊥ Represents the orthogonal complement of H, i.e.: h + H ⊥ ＝1，H ⊥ D represents a portion of the original video without moving foreground;
u, V is a low rank decomposition of video background, i.e. U is in the form of R mn×r ,V∈R T×r Is a low rank matrix r of rank r&Min (mn, T), which is the hadamard product operator, means that the corresponding elements of the matrix are multiplied one by one, i.e.: h | _ D is still a matrix of the same size as H and D, each element of which is H, and the corresponding element of D is multiplied to obtain E is the noise of the rainbars on each frame of video after being cut into small blocks, E is the noise of the rainbars on each frame of video after being cut into small blocks i The ith column is E and represents a rain strip on the ith small block, and the f is a cutting operator which has the functions of selecting small blocks in each frame picture of the original video according to the specified interval and size, drawing each small block into column vectors and splicing the column vectors into a matrix;representing the distribution satisfied by the rain strip noise on the ith patch, according to the directional characteristic of the rain strip, a high-dimensional Gaussian mixture distribution is assumed here, and the form of K Gaussian mixture components is as follows:
wherein the mean vector of the k-th Gaussian mixture component is mu k The covariance matrix is ∑ k In a mixing ratio of
3. The video rain removal method based on noise modeling according to claim 1, wherein: in the step S3, based on a low-rank matrix decomposition method, a small block prior rain removal model under a no-movement foreground condition is constructed by using a distribution rule and a directional characteristic of a rain strip on a small block and a maximum likelihood estimation method, and is expressed as a probability form:
the likelihood function of the source data is
Taking its logarithmic form:
log P(f(H ⊥ ⊙D)|U,V,∑)
wherein V (f (H) ⊥ ⊙(D-UV T )) n |μ k ,Σ k ) Is defined by formula (1);
the small block prior rain removal model under the no-movement prospect converted by the statistical model is an optimization problem as follows:
4. the video rain removal method based on noise modeling according to claim 1, wherein: in the step S4, according to the structural characteristics of the video foreground support, the following moving object detection models are constructed for distinguishing the video foreground target:
where D is the input video, B is the background portion of the video, H is the video foreground support, l (·,) is a loss function to measure the similarity between two segments of video, H · D is the moving foreground portion of the video.
5. The video rain removal method based on noise modeling according to claim 1, wherein: in step S5, a foreground detection model with a foreground support and a static background rain removing method under a maximum likelihood frame are introduced, and a rain removing model based on noise modeling is an optimized model as follows:
6. the video rain removal method based on noise modeling according to claim 1, characterized in that: in the step S6, an EM algorithm is adopted to solve the rain removal model formula (3) based on the noise modeling in the step S5, and the specific steps comprise:
s4.1) E step: likelihood probability of updating model hidden variables
Introducing an implicit variable z nk ∈{0,1}，z nk Is an indicator variable for judging whether the noise of the rain strip on the nth small block belongs to the kth mixed Gaussian class or not, and has
Updating the nth tile E n Probability of belonging to the kth mixed gaussian:
s4.2) M step: maximizing the expectation of the full data likelihood function with respect to the hidden variable, i.e., minimizing the reciprocal of the expectation, has the objective function of:
where θ represents the parameter set: theta = [ pi, mu, sigma ]]；θ old Representing the value of theta updated in the last iteration,
for the convenience of solving, a cut variable L is introduced, and the above optimization problem is equivalent to:
s.t.
7. the video rain removal method based on noise modeling according to claim 6, wherein: in the step S6, the small block prior rain removal model formula (3) in the step S3 is solved by adopting an EM algorithm, and the objective function in the step M is solved by adopting an alternating direction multiplier method, and the specific steps comprise:
s4.2.1) gives the augmented Lagrangian function of equation (4)
Wherein Λ is a multiplier and μ is a number greater than 0;
s4.2.2) establishes the iteration format and termination condition of the alternative direction multiplier method:
μ k+1 ＝ρμ k (11)
where p is a positive constant greater than 1, typically set to p =1.05,
the iteration termination condition is as follows:
s4.2.3) solving the problems (5), (6), (7), (8) and (9) to give a specific iterative formula;
s4.2.4) sets the initial value of the iteration to: h 0 ＝0，U 0 ,V 0 Generated by the well-known singular value decomposition method acting on D, with an initial Gaussian mean value set to 0 and an initial covariance matrixIs obtained by symmetrical orthogonalization of a random matrix;
s4.2.5) performing the iterative operations of (5) - (11) until the iteration meets the termination condition, i.e. the likelihood function descent rate is less than the threshold or the iteration number reaches the upper limit.
8. The video rain removal method based on noise modeling according to claim 7, wherein: the equation (5) solves the following problem:
since the elements in H consist of 0,1, the above optimization problem is equivalent to the following convex optimization problem:
the problem is regarded as a first-order binary Markov random field problem, and H is solved by a graph cut algorithm;
the equation (6) solves the following problem:
the optimization problem can be divided, L can be solved as follows:
the equation (7) solves the following problem:
equivalent to solving:
equivalent to:
this is a weighted two-norm problem, where f -1 Representing an operator for reducing the small blocks into the video, wherein W is a weight tensor which has the same size as the input video, and the value of each point in W is equal to the number of the small blocks overlapped at the point in the original video;
rewrite equation (12) as:
u, V the approximate optimum can be iteratively solved:
is a weighted two-norm problem in which U t ,V t Solving the U, V value obtained by the iterative computation of the t step according to the rows as follows:
the equation (8) solves the following problem:
the solution is as follows:
∑ k
wherein:
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710319380 | 2017-05-09 | ||
| CN2017103193809 | 2017-05-09 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107909548A true CN107909548A (en) | 2018-04-13 |
| CN107909548B CN107909548B (en) | 2020-05-15 |
Family
ID=61841552
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710992669.7A Active CN107909548B (en) | 2017-05-09 | 2017-10-23 | Video rain removing method based on noise modeling |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107909548B (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108510013A (en) * | 2018-07-02 | 2018-09-07 | 电子科技大学 | The steady tensor principal component analytical method of improvement based on low-rank kernel matrix |
| CN108900841A (en) * | 2018-07-10 | 2018-11-27 | 中国科学技术大学 | Method for video coding based on image rain removing algorithm |
| CN109150775A (en) * | 2018-08-14 | 2019-01-04 | 西安交通大学 | A kind of online channel state information estimation method of the robustness of environment adaptive noise dynamic change |
| CN109636738A (en) * | 2018-11-09 | 2019-04-16 | 温州医科大学 | The single image rain noise minimizing technology and device of double fidelity term canonical models based on wavelet transformation |
| CN109859119A (en) * | 2019-01-07 | 2019-06-07 | 南京邮电大学 | A kind of video image rain removing method restored based on adaptive low-rank tensor |
| CN110610152A (en) * | 2019-09-10 | 2019-12-24 | 西安电子科技大学 | Multispectral cloud detection method based on discriminative feature learning unsupervised network |
| CN111738932A (en) * | 2020-05-13 | 2020-10-02 | 合肥师范学院 | Automatic rain removing method for photographed image of vehicle-mounted camera |
| CN111879293A (en) * | 2020-07-31 | 2020-11-03 | 国家海洋技术中心 | Device and method for in-situ measurement of noise characteristics of rainfall on sea |
| CN113538297A (en) * | 2021-08-27 | 2021-10-22 | 四川大学 | Image rain removing method based on gradient priori knowledge and N-S equation |
| CN116012263A (en) * | 2023-03-27 | 2023-04-25 | 四川工程职业技术学院 | Image noise removing method and device, storage medium and electronic equipment |
| CN117152000A (en) * | 2023-08-08 | 2023-12-01 | 华中科技大学 | Rainy day image-clear background paired data set manufacturing method and device and application thereof |
| CN117455809A (en) * | 2023-10-24 | 2024-01-26 | 武汉大学 | Image mixed rain removing method and system based on depth guiding diffusion model |
| CN118014892A (en) * | 2024-04-08 | 2024-05-10 | 北京航空航天大学 | Single image rain removal model construction method and rain removal method based on frequency domain comparison regularization |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130236116A1 (en) * | 2012-03-08 | 2013-09-12 | Industrial Technology Research Institute | Method and apparatus for single-image-based rain streak removal |
| CN103337061A (en) * | 2013-07-18 | 2013-10-02 | 厦门大学 | Rain and snow removing method for image based on multiple guided filtering |
| CN103700070A (en) * | 2013-12-12 | 2014-04-02 | 中国科学院深圳先进技术研究院 | Video raindrop-removing algorithm based on rain-tendency scale |
| CN103714517A (en) * | 2013-12-12 | 2014-04-09 | 中国科学院深圳先进技术研究院 | Video rain removing method |
| CN106067163A (en) * | 2016-05-24 | 2016-11-02 | 中国科学院深圳先进技术研究院 | A kind of image rain removing method based on wavelet analysis and system |
-
2017
- 2017-10-23 CN CN201710992669.7A patent/CN107909548B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130236116A1 (en) * | 2012-03-08 | 2013-09-12 | Industrial Technology Research Institute | Method and apparatus for single-image-based rain streak removal |
| CN103337061A (en) * | 2013-07-18 | 2013-10-02 | 厦门大学 | Rain and snow removing method for image based on multiple guided filtering |
| CN103700070A (en) * | 2013-12-12 | 2014-04-02 | 中国科学院深圳先进技术研究院 | Video raindrop-removing algorithm based on rain-tendency scale |
| CN103714517A (en) * | 2013-12-12 | 2014-04-09 | 中国科学院深圳先进技术研究院 | Video rain removing method |
| CN106067163A (en) * | 2016-05-24 | 2016-11-02 | 中国科学院深圳先进技术研究院 | A kind of image rain removing method based on wavelet analysis and system |
Non-Patent Citations (5)
| Title |
|---|
| N. BREWER: "Using the shape characteristics of rain to identify and remove rain from video", 《IN JOINT IAPR INTERNATIONAL WORKSHOPS ON STATISTICAL TECHNIQUES IN PATTERN RECOGNITION (SPR) AND STRUCTURAL AND SYNTACTIC PATTERN RECOGNITION (SSPR)》 * |
| XIANGYONG CAO: "Low-Rank Matrix Factorization under General Mixture Noise Distributions", 《2015 IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV)》 * |
| Y-L: "A generalized low-rank appearance model for spatio-temporally correlated rain streaks", 《IN PROCEEDINGS OF THE IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION》 * |
| 江齐: "基于U-GAN神经网络的视频图像去雨雾技术", 《图形图像》 * |
| 董蓉: "一种视频雨滴检测与消除的方法", 《自动化学报》 * |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108510013A (en) * | 2018-07-02 | 2018-09-07 | 电子科技大学 | The steady tensor principal component analytical method of improvement based on low-rank kernel matrix |
| CN108510013B (en) * | 2018-07-02 | 2020-05-12 | 电子科技大学 | Background modeling method for improving robust tensor principal component analysis based on low-rank core matrix |
| CN108900841A (en) * | 2018-07-10 | 2018-11-27 | 中国科学技术大学 | Method for video coding based on image rain removing algorithm |
| CN109150775A (en) * | 2018-08-14 | 2019-01-04 | 西安交通大学 | A kind of online channel state information estimation method of the robustness of environment adaptive noise dynamic change |
| CN109636738A (en) * | 2018-11-09 | 2019-04-16 | 温州医科大学 | The single image rain noise minimizing technology and device of double fidelity term canonical models based on wavelet transformation |
| CN109636738B (en) * | 2018-11-09 | 2019-10-01 | 温州医科大学 | The single image rain noise minimizing technology and device of double fidelity term canonical models based on wavelet transformation |
| CN109859119A (en) * | 2019-01-07 | 2019-06-07 | 南京邮电大学 | A kind of video image rain removing method restored based on adaptive low-rank tensor |
| CN109859119B (en) * | 2019-01-07 | 2022-08-02 | 南京邮电大学 | Video image rain removing method based on self-adaptive low-rank tensor recovery |
| CN110610152A (en) * | 2019-09-10 | 2019-12-24 | 西安电子科技大学 | Multispectral cloud detection method based on discriminative feature learning unsupervised network |
| CN110610152B (en) * | 2019-09-10 | 2022-03-22 | 西安电子科技大学 | Multispectral cloud detection method based on discriminative feature learning unsupervised network |
| CN111738932A (en) * | 2020-05-13 | 2020-10-02 | 合肥师范学院 | Automatic rain removing method for photographed image of vehicle-mounted camera |
| CN111879293A (en) * | 2020-07-31 | 2020-11-03 | 国家海洋技术中心 | Device and method for in-situ measurement of noise characteristics of rainfall on sea |
| CN111879293B (en) * | 2020-07-31 | 2023-02-10 | 国家海洋技术中心 | Device and method for in-situ measurement of noise characteristics of rainfall on sea |
| CN113538297A (en) * | 2021-08-27 | 2021-10-22 | 四川大学 | Image rain removing method based on gradient priori knowledge and N-S equation |
| CN116012263A (en) * | 2023-03-27 | 2023-04-25 | 四川工程职业技术学院 | Image noise removing method and device, storage medium and electronic equipment |
| CN117152000A (en) * | 2023-08-08 | 2023-12-01 | 华中科技大学 | Rainy day image-clear background paired data set manufacturing method and device and application thereof |
| CN117152000B (en) * | 2023-08-08 | 2024-05-14 | 华中科技大学 | Rainy day image-clear background paired data set manufacturing method and device and application thereof |
| CN117455809A (en) * | 2023-10-24 | 2024-01-26 | 武汉大学 | Image mixed rain removing method and system based on depth guiding diffusion model |
| CN117455809B (en) * | 2023-10-24 | 2024-05-24 | 武汉大学 | Image mixed rain removing method and system based on depth guiding diffusion model |
| CN118014892A (en) * | 2024-04-08 | 2024-05-10 | 北京航空航天大学 | Single image rain removal model construction method and rain removal method based on frequency domain comparison regularization |
| CN118014892B (en) * | 2024-04-08 | 2024-06-25 | 北京航空航天大学 | Single image rain removal model construction method and rain removal method based on frequency domain comparison regularization |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107909548B (en) | 2020-05-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107909548B (en) | Video rain removing method based on noise modeling | |
| CN108520501B (en) | Video rain and snow removing method based on multi-scale convolution sparse coding | |
| Bahnsen et al. | Rain removal in traffic surveillance: Does it matter? | |
| CN111639564B (en) | Video pedestrian re-identification method based on multi-attention heterogeneous network | |
| CN109685045B (en) | Moving target video tracking method and system | |
| CN106447674B (en) | Background removing method for video | |
| CN110310238B (en) | Single image rain removing method based on compression award and punishment neural network reusing original information | |
| CN106097256B (en) | A kind of video image fuzziness detection method based on Image Blind deblurring | |
| CN102637298A (en) | Color image segmentation method based on Gaussian mixture model and support vector machine | |
| CN110415260B (en) | Smoke image segmentation and identification method based on dictionary and BP neural network | |
| CN109215053A (en) | Moving vehicle detection method containing halted state in a kind of unmanned plane video | |
| CN114693924A (en) | Road scene semantic segmentation method based on multi-model fusion | |
| CN110111267A (en) | A kind of single image based on optimization algorithm combination residual error network removes rain method | |
| CN111815528A (en) | Bad weather image classification enhancement method based on convolution model and feature fusion | |
| Chen et al. | Visual depth guided image rain streaks removal via sparse coding | |
| CN111160282B (en) | Traffic light detection method based on binary Yolov3 network | |
| CN111160274B (en) | Pedestrian detection method based on binaryzation fast RCNN (radar cross-correlation neural network) | |
| CN117132503A (en) | Method, system, equipment and storage medium for repairing local highlight region of image | |
| CN114926348B (en) | Device and method for removing low-illumination video noise | |
| Li et al. | Advanced multiple linear regression based dark channel prior applied on dehazing image and generating synthetic haze | |
| Shivakumar et al. | Remote sensing and natural image dehazing using DCP based IDERS framework | |
| Joy et al. | Deblurring Of License Plate Image Using Blur Kernel Estimation | |
| Iqbal et al. | A comprehensive analysis on single image-based deraining using generative adversarial network | |
| Haque et al. | Grayscale portrait colorization using cnns and pretrained vgg-face descriptor | |
| Leroy et al. | Superrare: an object-oriented saliency algorithm based on superpixels rarity |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |