CN101895769A - Sparse representation based method and device for concealing video errors - Google Patents

Abstract

The invention provides a sparse representation based method for concealing video errors. The method comprises the following steps: detecting and acquiring broken pixel regions and complete pixel regions in a current frame; acquiring the motion vector of the pixels in the complete pixel regions in the current frame relatively to a reference frame; performing macro block matching on the current frame and the reference frame according to the motion vector to generate a residual image of the complete pixel region after motion compensation; constructing an image sparsity target function according to the residual image; and solving the image sparsity target function, and filling the broken pixel regions according to the solving result. The method reconstructs the image region occurring errors by directly utilizing the image sparse representation, and has the advantages of simple and easy use and excellent performance.

Description

Video error concealing method and device based on rarefaction representation

Technical field

The present invention relates to technical field of image processing, particularly a kind of video error concealing method and device based on rarefaction representation.

Background technology

Move killer's level of (3G) network as the Internet and the third generation and use, the real-time video communication service has become operator, equipment vendor and service provider institute Focal Point of Common Attention.Its core is how to provide high-quality video transmission quality for the user.But the bottom bearer network has time variation, has caused bigger uncertainty for the video flowing high efficiency of transmission.Particularly, the network congestion of burst or channel fading will cause the decode error of receiving terminal video flowing, further cause reconstructed image serious distortion to occur.In order to cover the vision distortion of this type of video image, error concealment algorithm is arisen at the historic moment, and makes indispensable decoding reprocessing instrument in a kind of Streaming Media application.

The essence of error concealment is farthest to recover the image-region of decoding and makeing mistakes according to the correct various video informations (as texture, coding mode and motion vector etc.) that receive.At present, the error concealment algorithm of academia and industrial quarters design is mainly based on two class thinkings: (1) time domain alternative strategy, promptly directly adopt the corresponding macro block (reference macroblock of same position, perhaps current macro motion vector reference macroblock pointed) in the last decoded picture to substitute current damage macro block; (2) spatial domain is smoothly tactful, promptly utilizes neighboring pixel information interpolation to go out the pixel value of current damage macro block.Though first kind thinking has been utilized the correlation on the frame of video time domain, to the not well utilization of spatial domain texture information of failure area.The second class thinking then often is not suitable for the situation of continuous information packet loss.Because the macro block that is positioned at around the failure area is lost in a large number, just mean that also the information that can offer the spatial concealment strategy is quite limited, so the picture quality of energy reconstruct will be very poor.

Summary of the invention

Purpose of the present invention is intended to solve above-mentioned technological deficiency at least, the invention provides a kind of video error concealing method and device based on rarefaction representation.

For achieving the above object, one aspect of the present invention proposes a kind of video error concealing method based on rarefaction representation, may further comprise the steps: detect and obtain damaged pixel district in the present frame and intact pixel region; Obtain the motion vector of the interior pixel of pixel region intact in the described present frame with respect to reference frame; According to described motion vector described present frame and described reference frame are carried out the residual image of macroblock match after with the motion compensation that generates described intact pixel region; According to described residual image design of graphics as the degree of rarefication target function; With described image sparse degree target function is found the solution, and fill described damaged pixel district according to solving result.

The present invention has also proposed a kind of concealing video errors device based on rarefaction representation on the other hand, comprising: macro block detects and acquisition module, is used for detecting and obtaining the damaged pixel district of present frame and intact pixel region; Motion vector detects and obtains module, is used for obtaining the motion vector of the interior pixel of the intact pixel region of described present frame with respect to reference frame; The residual image generation module is used for according to described motion vector described present frame and described reference frame being carried out the residual image of macroblock match after with the motion compensation that generates described intact pixel region; The degree of rarefication target function makes up module, and the described residual image design of graphics that is used for generating according to described residual image generation module is as the degree of rarefication target function; And packing module, be used for the image sparse degree target function that described degree of rarefication target function structure module generates is found the solution, and fill described damaged pixel district according to solving result.

The image-region that the present invention has directly utilized the image sparse characterization to come reconstruct to make a mistake is simple and easy to usefulness, excellent performance, and the present invention unite utilized between frame of video and frame in correlation properties, the video that can damage big zone effectively recover.In addition, the present invention utilizes the spatial domain and the relativity of time domain of frame of video, and the degree of rarefication representative function of combined optimization spatial domain image and time domain residual image, thereby guarantees that excellent error concealment performance can both be arranged under abominable status transmission.Therefore, the present invention has not only solved the problem that video quality declined to a great extent when the user side receiving video data was made mistakes, but also can overcome the deficiencies in the prior art.

Aspect that the present invention adds and advantage part in the following description provide, and part will become obviously from the following description, or recognize by practice of the present invention.

Description of drawings

Above-mentioned and/or additional aspect of the present invention and advantage are from obviously and easily understanding becoming the description of embodiment below in conjunction with accompanying drawing, wherein:

Fig. 1 is the flow chart based on the video error concealing method of rarefaction representation of the embodiment of the invention;

Fig. 2 is the schematic diagram of the residual image after the structure motion compensation of the embodiment of the invention;

Fig. 3 is the concealing video errors structure drawing of device based on rarefaction representation of the embodiment of the invention.

Embodiment

Describe embodiments of the invention below in detail, the example of described embodiment is shown in the drawings, and wherein identical from start to finish or similar label is represented identical or similar elements or the element with identical or similar functions.Below by the embodiment that is described with reference to the drawings is exemplary, only is used to explain the present invention, and can not be interpreted as limitation of the present invention.

As shown in Figure 1, be the flow chart based on the video error concealing method of rarefaction representation of the embodiment of the invention.As shown in Figure 2, be the schematic diagram of the residual image after the structure motion compensation of the embodiment of the invention, wherein, F _cBe present frame, F _rBe F _cThe corresponding reference frame.If

With

Be respectively present frame F _cWith reference frame F _rMiddle location of pixels The pixel value at place,

Be present frame F _cMiddle pixel

Pairing motion vector.This method may further comprise the steps:

Step S101 detects and obtains present frame F _cThe middle pixel region that damages (being the gray scale square among Fig. 2) and intact pixel region D _S(being the white square among Fig. 2).

Step S 102, obtain intact pixel region D in the present frame _SInterior pixel

Motion vector with respect to reference frame

Step S103, according to described motion vector to present frame F _cWith reference frame F _rCarry out macroblock match to generate described intact pixel region D _SMotion compensation after residual image:

$F_{\mathrm{\Δ}}(\stackrel{\→}{x}\∈D_S)=F_r(\stackrel{\→}{x}+\stackrel{\→}{v}\left(\stackrel{\→}{x}\right);\stackrel{\→}{x}\∈D_S)-F_c(\stackrel{\→}{x}\∈D_S).$

Step S104, according to described residual image design of graphics as the degree of rarefication target function.If ψ is reverse two-dimension discrete cosine transform (DCT) base, θ _cBe present frame F _cTwo dimensional DCT coefficients (rarefaction representation of spatial domain image), θ _ΔBe residual frame F _ΔTwo dimensional DCT coefficients (rarefaction representation of residual image after the time domain motion compensation).In one embodiment of the invention, weight coefficient λ=0.6 that this two classes degree of rarefication is represented for example can be set, the target function that this weighting degree of rarefication is optimized is as follows:

min‖Vec[θ _c]‖ ₁+0.6×‖Vec[θ _Δ]‖ ₁

$s.t.\left\{\begin{array}{c}{F}_c(\stackrel{\→}{x}\∈D_S)=S\left[{\mathrm{\ψ\θ}}_c\right]\\ F_{\mathrm{\Δ}}(\stackrel{\→}{x}\∈D_S)=S\left[{\mathrm{\ψ\θ}}_{\mathrm{\Δ}}\right]\\ S^{*}\left[{\mathrm{\ψ\θ}}_c\right]-F_r(\stackrel{\→}{x}\∈D_S^{*})=S^{*}\left[{\mathrm{\ψ\θ}}_{\mathrm{\Δ}}\right]\end{array}\right.$

Wherein, ‖ ‖ ₁Single order norm operator, Vec[are asked in expression] representing matrix vectorization operator, S[] be to D _SThe sampling operation symbol in zone, S ^*[] is right

The sampling operation symbol in zone, s.t. represents utmost point constraints.

Step S105 treats the image sparse degree target function of constraint and finds the solution.In one embodiment of the invention, can utilize method of Lagrange multipliers that the target function that minimizes of above-mentioned belt restraining is converted into unconfined minimization problem, promptly

$\begin{array}{c}\underset{\{{\mathrm{\θ}}_c,{\mathrm{\θ}}_{\mathrm{\Δ}},\mathrm{\β}\}}{\min }J({\mathrm{\θ}}_c;{\mathrm{\θ}}_{\mathrm{\Δ}};\mathrm{\β})={\left|\right|\mathrm{Vec}\left[{\mathrm{\θ}}_c\right]\left|\right|}_1+0.6\×{\left|\right|\mathrm{Vec}\left[{\mathrm{\θ}}_{\mathrm{\Δ}}\right]\left|\right|}_1+\mathrm{\β}\×\{F_c(\stackrel{\→}{x}\∈D_S)-S[{\mathrm{\ψ\θ}}_c\left]\right\}\\ +\mathrm{\β}\×\{F_{\mathrm{\Δ}}(\stackrel{\→}{x}\∈D_S)-S[{\mathrm{\ψ\θ}}_{\mathrm{\Δ}}\left]\right\}+\mathrm{\β}\×\left\{S^{*}\right[{\mathrm{\ψ\θ}}_c]-F_r(\stackrel{\→}{x}\∈D_S^{*})-S^{*}[{\mathrm{\ψ\θ}}_{\mathrm{\Δ}}\left]\right\}\end{array}$

Wherein, β is a Lagrange multiplier.

To above-mentioned no constrained minimization problem, adopt iteration threshold algorithm (FISTA) to obtain optimal solution

With

Concrete iterative process can be referring to document: (A.Beck, and M.Teboulle, " A fastiterative shrinkage-thresholding algorithm for linear inverse problems; " SIAMJournal on Imaging Sciences, vol.2, no.1, pp.183-202,2009.)

Step S106 calculates

Reverse two-dimensional dct transform And taking-up failure area corresponding pixel value And the pixel region of filling up damage with this result

As shown in Figure 3, be the concealing video errors structure drawing of device based on rarefaction representation of the embodiment of the invention.This device 100 comprises that macro block detects and acquisition module 110, motion vector detect acquisition module 120, residual image generation module 130, degree of rarefication target function structure module 140 and packing module 150.Macro block detects and acquisition module 110 is used for detecting and obtaining the damaged pixel district of present frame and intact pixel region.Motion vector detects acquisition module 120 and is used for obtaining the motion vector of the interior pixel of the intact pixel region of described present frame with respect to reference frame.Residual image generation module 130 is used for according to motion vector present frame and described reference frame being carried out the residual image of macroblock match after with the motion compensation that generates described intact pixel region.The described residual image design of graphics that degree of rarefication target function structure module 140 is used for generating according to residual image generation module 130 is as the degree of rarefication target function.Packing module 150 is used for the image sparse degree target function that degree of rarefication target function structure module 140 generates is found the solution, and fills described damaged pixel district according to solving result.

In one embodiment of the invention, packing module 150 comprises calculating sub module 151 and fills submodule 152.Calculating sub module 151 is used to utilize method of Lagrange multipliers and iteration threshold algorithm to obtain optimal solution

With

And calculate

Reverse two-dimensional dct transform

Fill submodule 152 and be used to take out the failure area corresponding pixel value

And fill up the pixel region of damage with this result

The image-region that the present invention has directly utilized the image sparse characterization to come reconstruct to make a mistake is simple and easy to usefulness, excellent performance, the present invention unite utilized between frame of video and frame in correlation properties, the video that can damage big zone effectively recover.In addition, the present invention utilizes the spatial domain and the relativity of time domain of frame of video, and the degree of rarefication representative function of combined optimization spatial domain image and time domain residual image, thereby guarantees that excellent error concealment performance can both be arranged under abominable status transmission.Therefore, the present invention has not only solved the problem that video quality declined to a great extent when the user side receiving video data was made mistakes, but also can overcome the deficiencies in the prior art.

Although illustrated and described embodiments of the invention, for the ordinary skill in the art, be appreciated that without departing from the principles and spirit of the present invention and can carry out multiple variation, modification, replacement and modification that scope of the present invention is by claims and be equal to and limit to these embodiment.

Claims (8)

1. the video error concealing method based on rarefaction representation is characterized in that, may further comprise the steps:

Detect and obtain damaged pixel district in the present frame and intact pixel region;

Obtain the motion vector of the interior pixel of pixel region intact in the described present frame with respect to reference frame;

According to described motion vector described present frame and described reference frame are carried out the residual image of macroblock match after with the motion compensation that generates described intact pixel region;

According to described residual image design of graphics as the degree of rarefication target function; With

Described image sparse degree target function is found the solution, and fill described damaged pixel district according to solving result.

2. the video error concealing method based on rarefaction representation as claimed in claim 1 is characterized in that, described residual image is:

Wherein, D _SBe intact pixel region,

Be present frame F _cThe pixel value function, Be F _cCorresponding reference frame F _rPixel value, wherein,

The coordinate of remarked pixel in frame of video, Be present frame F _cMiddle pixel

Pairing motion vector.

3. the video error concealing method based on rarefaction representation as claimed in claim 2 is characterized in that, described image sparse degree target function is:

min‖Vec[θ _c]‖ ₁+λ‖Vec[θ _Δ]‖ ₁

Wherein, ψ is reverse two-dimension discrete cosine transform DCT base, θ _cBe present frame F _cTwo dimensional DCT coefficients, θ _ΔBe reference frame F _ΔTwo dimensional DCT coefficients, ‖ ‖ ₁Single order norm operator is asked in expression, and λ ∈ [0,1] is predefined weight coefficient, Vec[] representing matrix vectorization operator, S[] be to D _SThe sampling operation symbol in zone, S ^*[] is right The sampling operation symbol in zone, s.t. represents constraints.

4. the video error concealing method based on rarefaction representation as claimed in claim 3 is characterized in that, described image sparse degree target function is found the solution, and fills described damaged pixel district according to solving result and further comprise:

Utilize method of Lagrange multipliers and iteration threshold algorithm to obtain optimal solution

With

With

Calculate Reverse two-dimensional dct transform

And taking-up failure area corresponding pixel value

And the pixel region of filling up damage with this result

5. the concealing video errors device based on rarefaction representation is characterized in that, comprising:

Macro block detects and acquisition module, is used for detecting and obtaining the damaged pixel district of present frame and intact pixel region;

Motion vector detects and obtains module, is used for obtaining the motion vector of the interior pixel of the intact pixel region of described present frame with respect to reference frame;

The residual image generation module is used for according to described motion vector described present frame and described reference frame being carried out the residual image of macroblock match after with the motion compensation that generates described intact pixel region;

The degree of rarefication target function makes up module, and the described residual image design of graphics that is used for generating according to described residual image generation module is as the degree of rarefication target function; With

Packing module is used for the image sparse degree target function that described degree of rarefication target function structure module generates is found the solution, and fills described damaged pixel district according to solving result.

6. the concealing video errors device based on rarefaction representation as claimed in claim 5 is characterized in that described residual image is:

Wherein, D _SBe intact pixel region,

Be present frame F _cThe pixel value function,

Be F _cCorresponding reference frame F _rPixel value, wherein,

The coordinate of remarked pixel in frame of video,

Be present frame F _cMiddle pixel Pairing motion vector.

7. the concealing video errors device based on rarefaction representation as claimed in claim 6 is characterized in that, described image sparse degree target function is:

min‖Vec[θ _c]‖ ₁+λ‖Vec[θ _Δ]‖ ₁

Wherein, ψ is reverse two-dimension discrete cosine transform DCT base, θ _cBe present frame F _cTwo dimensional DCT coefficients, θ _ΔBe reference frame f _ΔTwo dimensional DCT coefficients, ‖ ‖ ₁Single order norm operator is asked in expression, and λ ∈ [0,1] is predefined weight coefficient, Vec[] representing matrix vectorization operator, S[] be to D _SThe sampling operation symbol in zone, S ^*[] is right

The sampling operation symbol in zone, s.t. represents constraints.

8. the concealing video errors device based on rarefaction representation as claimed in claim 7 is characterized in that described packing module comprises:

Calculating sub module is used to utilize method of Lagrange multipliers and iteration threshold algorithm to obtain optimal solution

With

And calculate

Reverse two-dimensional dct transform With

Fill submodule, be used to take out the failure area corresponding pixel value And fill up the pixel region of damage with this result

Images