CN104463812A - Method for repairing video image disturbed by raindrops in shooting process - Google Patents

Abstract

The invention provides a method for repairing a video image disturbed by raindrops in the shooting process. The method comprises the steps of A, a frame of image based on a video of RGB color space is received; B, the received frame of image is converted into an image based on HSL color space; C, on the basis of tone components and saturation components, a pure raindrop portion and a raindrop and motion object overlapping portion in the converted image are detected; D, raindrop removing is carried out on the pure raindrop portion and the raindrop and motion object overlapping portion on the basis of luminance components; E, the image obtained after raindrop removing is converted into the image based on the RGB color space and output. According to the method, on the premise of high robustness, the precision and efficiency of video image repairing can be improved.

Description

Repair the method for the video image being subject to raindrop interference when taking

Technical field

The present invention relates to technical field of image processing, more particularly, relating to a kind of reparation is subject to the video image of raindrop interference method when taking.

Background technology

Rain has a great impact image imaging, can cause image image blur, information is capped, such that the sharpness of video image declines, the digitized processing hydraulic performance decline of video image.Therefore, the further process that repair process is conducive to image is carried out to the video image being subject to raindrop interference during shooting, comprise the performance improved based on technology such as the target detection of image, identification, tracking, segmentation and monitoring.Video image goes rain technology to have a wide range of applications in fields such as modern military, traffic and security monitorings.

Existingly remove rain algorithm for video image, comparatively ripe on the video image being applied in static scene, but the video image being applied in dynamic scene cannot reach desirable Detection results.

In addition, in real time repair process is carried out to the video image being subject to raindrop interference during shooting and have very large demand in the occasion such as automated navigation system, safety monitoring system.Often need in these application scenarios to obtain result in time, feed back to user, the delayed of Video processing likely causes user to do the judgement made mistake.And the existing rain algorithm that goes for video image causes efficiency lower due to the complexity that it calculates, real-time need to improve.

Summary of the invention

Exemplary embodiment of the present invention is to provide a kind of reparation to be subject to the method for the video image of raindrop interference when taking, and repairs precision and the undesirable problem of remediation efficiency to overcome in prior art.

The invention provides a kind of reparation is subject to the video image of raindrop interference method when taking, it is characterized in that, comprise: (A) receives the two field picture based on the video of rgb color space; (B) two field picture received is converted to image based on HSL color space; (C) based on the pure raindrop part in the image after chrominance component and the conversion of saturation degree component detection and raindrop and moving object lap; (D) respectively raindrop removal is carried out to pure raindrop part and raindrop and moving object lap based on luminance component; (E) image after removal raindrop is converted to the image based on rgb color space and exports.

Alternatively, step (C) comprising: (C1) is based on the moving object part in the image after conversion described in chrominance component and saturation degree component detection; (C2) according to a described two field picture and the gray scale difference of a two field picture, the optical characteristics of raindrop and the chromatic characteristic adjacent with a described two field picture detect in the image after described conversion by raindrop pollution part; (C3) described moving object part and the described lap by raindrop pollution part are defined as raindrop and moving object lap, and are defined as pure raindrop part by described by the part in raindrop pollution part except partly overlapping with described moving object.

Alternatively, step (C1) comprising: (C11) is based on the edge of the moving object in the image after conversion described in chrominance component and saturation degree component detection; (C12) according to color character, the pixel in the image after described conversion is carried out cluster, so that the Iamge Segmentation after described conversion is become multiple pieces; (C13) pixel of the block inside belonging to the edge of the moving object detected is labeled as the pixel belonging to moving object part.

Alternatively, step (C11) comprising: when the metric function value of the pixel in the image after described conversion is greater than predetermined threshold, determines that this pixel is positioned on the edge of moving object, and wherein, metric function F (x, y) is:

$F(x,y)=\frac{1}{2}\left(\right|H_r(x,y)-H_b(x,y)|+|S_r(x,y)-S_b(x,y)\left|\right)\×\left|\right|{\stackrel{r}{I}}_r(x,y)-{\stackrel{r}{I}}_b(x,y)\left|\right|$

Wherein, H _r(x, y) indicates the tone value of pixel (x, y), H _b(x, y) indicates the background colour tone pitch of pixel (x, y), S _r(x, y) indicates the intensity value of pixel (x, y), S _b(x, y) indicates the background intensity value of pixel (x, y), the gray-scale value of instruction pixel (x, y), the background gray levels of instruction pixel (x, y).

Alternatively, step (D) comprising: replaced with by the brightness value of each pixel in pure raindrop part: the weighted mean value of the brightness value of the corresponding pixel points of front M two field picture to rear M two field picture of a described two field picture, wherein, M be greater than 0 integer; The brightness value of each pixel in raindrop and moving object lap is replaced with: the mean value of the weighted mean value of the brightness value of the corresponding pixel points in each two field picture in the previous frame image of a described two field picture to a rear two field picture and the neighbor pixel of corresponding pixel points.

Alternatively, M=3, wherein, the brightness value of the pixel (x, y) in pure raindrop part calculates replacement values by following formula:

$L(x,y,N)=\frac{\underset{t=N-3}{\overset{N+3}{\mathrm{\Σ}}}{F}_b\left(t\right)L(x,y,t)}{\underset{t=N-3}{\overset{N+3}{\mathrm{\Σ}}}{F}_b\left(t\right)}$

Wherein, N is the frame number of a described two field picture, and L (x, y, t) indicates the brightness value of the pixel (x, y) of t two field picture, F _bt () is weighting coefficient matrix, F _b(t) [1,2,4,0,4,2,1].

Alternatively, the brightness value of the pixel (x, y) in raindrop and moving object lap calculates replacement values by following formula:

$L(x,y,N)=\frac{\underset{t=N-1}{\overset{N+1}{\mathrm{\Σ}}}\underset{(x,y)\∈V}{\mathrm{\Σ}}{F}_m(x,y,t)L(x,y,t)}{\underset{t=N-1}{\overset{N+1}{\mathrm{\Σ}}}\underset{(x,y)\∈V}{\mathrm{\Σ}}{F}_m(x,y,t)}$

Wherein, N is the frame number of a described two field picture, L (x, y, t) indicate the brightness value of the pixel (x, y) of t two field picture, V indicate the previous frame image of a described two field picture in a rear two field picture with pixel (x, the territory that the neighbor pixel of y) corresponding pixel and the pixel of correspondence is formed, F _m(x, y, t) is weighting coefficient matrix,

$F_m(x,y,t)=\left(\begin{array}{ccc}1& 2& 1\\ 2& 0& 2\\ 1& 2& 1\end{array}\right).$

Repair the method for the video image being subject to raindrop interference when taking according to an exemplary embodiment of the present invention, the HSL color space based on video image and the identification to moving target realize under the prerequisite of high robust, improve restored video image precision and efficiency.

Part in ensuing description is set forth general plotting of the present invention other in and/or advantage, some will be clearly by describing, or can learn through the enforcement of general plotting of the present invention.

Accompanying drawing explanation

Fig. 1 illustrates the process flow diagram of the method for repairing the video image being subject to raindrop interference when taking according to an exemplary embodiment of the present invention;

Fig. 2 illustrates the process flow diagram of the method detecting moving object part according to an exemplary embodiment of the present invention.

Embodiment

Now will in detail with reference to embodiments of the invention, the example of described embodiment is shown in the drawings, and wherein, identical label refers to identical parts all the time.Below by referring to accompanying drawing, described embodiment will be described, to explain the present invention.

Fig. 1 illustrates the process flow diagram of the method for repairing the video image being subject to raindrop interference when taking according to an exemplary embodiment of the present invention.

With reference to Fig. 1, in step S10, receive a two field picture of the video based on rgb color space.

In step S20, convert the two field picture received to image based on HSL color space.

Particularly, convert the RGB image of the redness based on rgb color space received (R) component, green (G) component and blue (B) representation in components to image based on tone (H) component in HSL color space, saturation degree (S) component and brightness (L) representation in components, to carry out detection and the removal of raindrop under HSL color space.

Image based on rgb color space converted to the image based on HSL color space by following formula:

$L=\frac{1}{2}(\max +\min )---\left(3\right)$

Wherein, max indicates the maximal value in R, G, B component, and min indicates the minimum value in R, G, B component.

Relevant with the light characteristic of raindrop owing to removing raindrop, have nothing to do with the tonal properties of raindrop and saturation degree characteristic, therefore, convert RGB image to image based on HSL color space, just only can carry out raindrop removal for the luminance component of raindrop, and the detection of moving target can be carried out based on chrominance component and saturation degree component, thus greatly reduce the complexity of algorithm, improve counting yield.

In step S30, based on the pure raindrop part in the image after chrominance component and the conversion of saturation degree component detection and raindrop and moving object lap.

In one example, can first based on the moving object part in the image after chrominance component and the conversion of saturation degree component detection.Here, various applicable method can be used based on the moving object part in the image after chrominance component and the conversion of saturation degree component detection.Preferably, the method by the detection moving object part shown in Fig. 2 realizes.

Then, according to a described two field picture and the gray scale difference of a two field picture, the optical characteristics of raindrop and the chromatic characteristic adjacent with a described two field picture detect in the image after described conversion by raindrop pollution part.

In one example, first can try to achieve the gray scale difference of each pixel in two continuous frames video image, when the gray scale difference of a pixel is greater than difference threshold, then determine that this pixel is the pixel polluted by raindrop of candidate.Here, choosing of difference threshold size will make the change of the gray-scale value of all pixels polluted by raindrop to be detected.Such as, difference threshold size can be taken as 3/255.

Then, further screen based on the optics of raindrop and the chromatic characteristic pixel polluted by raindrop to candidate, obtain the pixel polluted by raindrop, thus determine by raindrop pollution part.These characteristics comprise: strength fluctuation scope α, the changing value etc. of largest connected region area β and rgb color component.Such as, by the value of α between 3/255-30/255, the value of β is between 30-50 pixel, and Δ R, Δ G and the approximately equalised pixel of Δ B are defined as the pixel polluted by raindrop, otherwise is defined as the non-pixel polluted by raindrop.Here, the value of α and β can according to the size of raindrop, and the size of frame of video and shooting focal length etc. are arranged.

Described moving object part and the described lap by raindrop pollution part are defined as raindrop and moving object lap, pollute the part except partly overlapping by raindrop be defined as pure moving object part except with described in described moving object part, and be defined as pure raindrop part by described by the part in raindrop pollution part except partly overlapping with described moving object, thus detect pure raindrop part, raindrop and moving object lap, pure moving object part.

In other words, in conjunction with moving object part and obtain them by raindrop pollution part common factor as raindrop and moving object lap, the part of moving object part except this common factor is pure moving object part, and polluting the part of part except this common factor by raindrop is pure raindrop part.

Should be appreciated that, other methods be applicable to also can be used based on the pure raindrop part in the image after chrominance component and the conversion of saturation degree component detection and raindrop and moving object lap.

In step S40, respectively raindrop removal is carried out to pure raindrop part and raindrop and moving object lap based on luminance component.

Because pure moving object part is not subject to the pollution of raindrop, brightness value remains unchanged, and therefore, only need carry out raindrop to pure raindrop part and raindrop and moving object lap respectively based on luminance component and remove.

In one example, the brightness value of each pixel in pure raindrop part can be replaced with: the weighted mean value of the brightness value of the corresponding pixel points in the front M two field picture of a described two field picture to rear M two field picture, wherein, M be greater than 0 integer.That is, the brightness value of this pixel in pure raindrop part is replaced by the weighted mean value of the brightness value of this pixel in the front and back M frame in time domain space.

The brightness value of each pixel in raindrop and moving object lap is replaced with: the mean value of the weighted mean value of the brightness value of the corresponding pixel points in each two field picture in the previous frame image of a described two field picture to a rear two field picture and the neighbor pixel of corresponding pixel points.For the brightness value of the pixel in raindrop and moving object lap, because the relativity of time domain between frame is also little, spatial coherence is larger on the contrary, so each frame is averaged before and after time domain can only be chosen, and carry out the weighted mean of the neighbor pixel of this pixel and this pixel respectively in each frame, thus comprehensive temporal correlation carries out the removal of raindrop.

Such as, M=3, wherein, the brightness value of the pixel (x, y) in pure raindrop part calculates replacement values by following formula:

$L(x,y,N)=\frac{\underset{t=N-3}{\overset{N+3}{\mathrm{\Σ}}}{F}_b\left(t\right)L(x,y,t)}{\underset{t=N-3}{\overset{N+3}{\mathrm{\Σ}}}{F}_b\left(t\right)}---\left(4\right)$

Wherein, N is the frame number of a described two field picture, and L (x, y, t) indicates the brightness value of the pixel (x, y) of t two field picture, F _bt () is weighting coefficient matrix, F _b(t) [1,2,4,0,4,2,1].

The brightness value of the pixel (x, y) in raindrop and moving object lap calculates replacement values by following formula:

$L(x,y,N)=\frac{\underset{t=N-1}{\overset{N+1}{\mathrm{\Σ}}}\underset{(x,y)\∈V}{\mathrm{\Σ}}{F}_m(x,y,t)L(x,y,t)}{\underset{t=N-1}{\overset{N+1}{\mathrm{\Σ}}}\underset{(x,y)\∈V}{\mathrm{\Σ}}{F}_m(x,y,t)}---\left(5\right)$

Wherein, N is the frame number of a described two field picture, L (x, y, t) indicate the brightness value of the pixel (x, y) of t two field picture, V indicate the previous frame image of a described two field picture in a rear two field picture with pixel (x, the territory that the neighbor pixel of y) corresponding pixel and the pixel of correspondence is formed, F _m(x, y, t) is weighting coefficient matrix, $F_m(x,y,t)=\left(\begin{array}{ccc}1& 2& 1\\ 2& 0& 2\\ 1& 2& 1\end{array}\right).$

In step S50, the image after removing raindrop is converted to the image based on rgb color space and exports.That is, the HSL image completing rain process is converted again to RGB image to export.

Fig. 2 illustrates the process flow diagram of the method detecting moving object part according to an exemplary embodiment of the present invention.Can perform when performing step S30.

As shown in Figure 2, in step S301, based on the edge of the moving object in the image after conversion described in chrominance component and saturation degree component detection.

Because raindrop falling speed is very fast, under normal exposure speed, image does not observe spherical raindrop substantially, but the rain line that raindrop are formed due to rapid movement.Under physical environment, formula (6) can describe the physics imaging process of raindrop, and can describe quantitatively raindrop fall time produce fuzzy:

I _r(x,y)αI _E(x,y)+(1-α)I _b(x,y) (6)

Wherein, I _r(x, y) indicates the gray-scale value of pixel (x, y), I _e(x, y) instruction, in time shutter T, supposes the equivalent desired gray level value that raindrop cover pixel (x, y) always and formed, I _b(x, y) indicates the background gray levels of pixel (x, y), that is, gray-scale value when not polluted by raindrop, α τ/T, represents the ratio that raindrop fell through the time needed for pixel (x, y) and time shutter.

Available light is mixed by the light of different frequency, and the optical model of therefore raindrop imaging is still set up in the passage based on arbitrary color component.That is, in formula (6), each variable is still set up after the representation in components of R, G, B tri-passages with it.By three component vector representations be:

${\stackrel{r}{I}}_r(x,y)=\mathrm{\α}{\stackrel{r}{I}}_E(x,y)+(1-\mathrm{\α}){\stackrel{r}{I}}_b(x,y)---\left(7\right)$

$\left\{\begin{array}{c}{R}_r(x,y)=\mathrm{\α}{I}_E(x,y)+(1-\mathrm{\α})R_b(x,y)\\ G_r(x,y)=\mathrm{\α}{I}_E(x,y)+(1-\mathrm{\α})G_b(x,y)\\ B_r(x,y)=\mathrm{\α}{I}_E(x,y)+(1-\mathrm{\α})B_b(x,y)\end{array}\right.---\left(8\right)$

Wherein, R _r(x, y) indicates the R component value of pixel (x, y), G _r(x, y) indicates the G component value of pixel (x, y), B _r(x, y) indicates the B component value of pixel (x, y), R _b(x, y) indicates the background R component value (that is, R component value when not polluted by raindrop) of pixel (x, y), G _b(x, y) indicates the background G component value (that is, G component value when not polluted by raindrop) of pixel (x, y), B _b(x, y) indicates the background B component value (that is, B component value when not polluted by raindrop) of pixel (x, y).

Because raindrop falling speed is very fast, so α τ/T levels off to zero, α/(1-α) and levels off to zero, combined from formula (1), formula (2), formula (3) and formula (8), by in the pixel that raindrop pollute, H _r-H _band S _r-S _bconvergence zero, it is less that the tone value (that is, background colour tone pitch) when polluted by raindrop by the tone value of the pixel of raindrop pollution and intensity value and this pixel and intensity value (that is, background intensity value) compare change.And for the pixel on the edge of moving object, can there is obvious change in tone value and intensity value.

Because tone value change compared with background colour tone pitch of the pixel polluted by raindrop is less, and the tone value of pixel on the edge of moving object changes greatly.But due to the impact of the blurring effect by different video quality and the formation of distant view misty rain, tone value and the background colour tone pitch of the pixel polluted by raindrop accurately cannot be obtained, therefore single use tone value cannot detect the edge of moving object exactly, needs in conjunction with intensity value and gray-scale value to construct metric function.

Therefore, can construct metric function F (x, y) is:

$F(x,y)=\frac{1}{2}\left(\right|H_r(x,y)-H_b(x,y)|+|S_r(x,y)-S_b(x,y)\left|\right)\×\left|\right|{\stackrel{r}{I}}_r(x,y)-{\stackrel{r}{I}}_b(x,y)\left|\right|---\left(9\right)$

Wherein, H _r(x, y) indicates the tone value of pixel (x, y), H _b(x, y) indicates the background colour tone pitch of pixel (x, y), S _r(x, y) indicates the intensity value of pixel (x, y), S _b(x, y) indicates the background intensity value of pixel (x, y), the gray-scale value of instruction pixel (x, y), the background gray levels of instruction pixel (x, y).

Should be appreciated that, in the consecutive frame that the background colour tone pitch of pixel (x, y) is not polluted by raindrop by this pixel, the tone value of this pixel obtains, and the background intensity value of pixel and background gray levels also obtain by corresponding mode.

Due to the chromatic characteristic of raindrop, the metric function value of pixel polluted by raindrop levels off to zero, and pixel on the edge of moving object all obvious change can occur due to tone value and intensity value, and therefore, metric function value is a larger value.Therefore, a threshold value can be set to the pixel on the edge filtering out moving object.Namely, when the metric function value of the pixel in the image after described conversion is greater than predetermined threshold, determine that this pixel is positioned on the edge of moving object.

In step S302, according to color character, the pixel in the image after described conversion is carried out cluster, so that the Iamge Segmentation after described conversion is become multiple pieces.

In step S303, the pixel of the block inside belonging to the edge of the moving object detected is labeled as the pixel belonging to moving object part.That is, color clustering image partition method is adopted to determine moving object part.

In addition, the said method according to exemplary embodiment of the present invention may be implemented as computer program, thus when running this program, realizes said method.

Repair the method for the video image being subject to raindrop interference when taking according to an exemplary embodiment of the present invention, the HSL color space based on video image and the identification to moving target realize under the prerequisite of high robust, improve restored video image precision and efficiency.

Although show and described exemplary embodiments more of the present invention, but those skilled in the art should understand that, when not departing from by the principle of the present invention of claim and its scope of equivalents thereof and spirit, can modify to these embodiments.

Claims (7)

1. repairing a method for the video image being subject to raindrop interference when taking, it is characterized in that, comprise:

(A) two field picture based on the video of rgb color space is received;

(B) two field picture received is converted to image based on HSL color space;

(C) based on the pure raindrop part in the image after chrominance component and the conversion of saturation degree component detection and raindrop and moving object lap;

(D) respectively raindrop removal is carried out to pure raindrop part and raindrop and moving object lap based on luminance component;

(E) image after removal raindrop is converted to the image based on rgb color space and exports.

2. the method for claim 1, is characterized in that, step (C) comprising:

(C1) based on the moving object part in the image after conversion described in chrominance component and saturation degree component detection;

(C2) according to a described two field picture and the gray scale difference of a two field picture, the optical characteristics of raindrop and the chromatic characteristic adjacent with a described two field picture detect in the image after described conversion by raindrop pollution part;

(C3) described moving object part and the described lap by raindrop pollution part are defined as raindrop and moving object lap, pollute the part except partly overlapping by raindrop be defined as pure moving object part except with described in described moving object part, and be defined as pure raindrop part by described by the part in raindrop pollution part except partly overlapping with described moving object.

3. method as claimed in claim 2, it is characterized in that, step (C1) comprising:

(C11) based on the edge of the moving object in the image after conversion described in chrominance component and saturation degree component detection;

(C12) according to color character, the pixel in the image after described conversion is carried out cluster, so that the Iamge Segmentation after described conversion is become multiple pieces;

(C13) pixel of the block inside belonging to the edge of the moving object detected is labeled as the pixel belonging to moving object part.

4. method as claimed in claim 3, it is characterized in that, step (C11) comprising:

When the metric function value of the pixel in the image after described conversion is greater than predetermined threshold, determine that this pixel is positioned on the edge of moving object,

Wherein, metric function F (x, y) is:

$F(x,y)=\frac{1}{2}\left(\right|H_r(x,y)-H_b(x,y)|+|(S_r(x,y)-S_b(x,y)|)\×\left|\right|{\stackrel{r}{I}}_r(x,y)-{\stackrel{r}{I}}_b(x,y)\left|\right|$

Wherein, H _r(x, y) indicates the tone value of pixel (x, y), H _b(x, y) indicates the background colour tone pitch of pixel (x, y), S _r(x, y) indicates the intensity value of pixel (x, y), S _b(x, y) indicates the background intensity value of pixel (x, y), the gray-scale value of instruction pixel (x, y), the background gray levels of instruction pixel (x, y).

5. the method for claim 1, is characterized in that, step (D) comprising:

The brightness value of each pixel in pure raindrop part is replaced with: the weighted mean value of the brightness value of the corresponding pixel points in the front M two field picture of a described two field picture to rear M two field picture, wherein, M be greater than 0 integer;

The brightness value of each pixel in raindrop and moving object lap is replaced with: the mean value of the weighted mean value of the brightness value of the corresponding pixel points in each two field picture in the previous frame image of a described two field picture to a rear two field picture and the neighbor pixel of corresponding pixel points.

6. method as claimed in claim 5, is characterized in that, M=3,

Wherein, the brightness value of the pixel (x, y) in pure raindrop part calculates replacement values by following formula:

$L(x,y,N)=\frac{\underset{t=N-3}{\overset{N+3}{\mathrm{\Σ}}}{F}_b\left(t\right)L(x,y,t)}{\underset{t=N-3}{\overset{N+3}{\mathrm{\Σ}}}{F}_b\left(t\right)}$

Wherein, N is the frame number of a described two field picture, and L (x, y, t) indicates the brightness value of the pixel (x, y) of t two field picture, F _bt () is weighting coefficient matrix, F _b(t) [1,2,4,0,4,2,1].

7. method as claimed in claim 5, is characterized in that, the brightness value of the pixel (x, y) in raindrop and moving object lap calculates replacement values by following formula:

$L(x,y,N)=\frac{\underset{t=N-1}{\overset{N+1}{\mathrm{\Σ}}}\underset{(x,y)\∈V}{\mathrm{\Σ}}{F}_m(x,y,t)L(x,y,t)}{\underset{t=N-1}{\overset{N+1}{\mathrm{\Σ}}}\underset{(x,y)\∈V}{\mathrm{\Σ}}{F}_m(x,y,t)}$

Wherein, N is the frame number of a described two field picture, L (x, y, t) indicate the brightness value of the pixel (x, y) of t two field picture, V indicate the previous frame image of a described two field picture in a rear two field picture with pixel (x, the territory that the neighbor pixel of y) corresponding pixel and the pixel of correspondence is formed, F _m(x, y, t) is weighting coefficient matrix, $F_m(x,y,t)=\left(\begin{array}{ccc}1& 2& 1\\ 2& 0& 2\\ 1& 2& 1\end{array}\right).$