CN105894485A - Adaptive video reconstruction method based on signal correlation - Google Patents

Abstract

The invention discloses an adaptive video reconstruction method based on signal correlation. The adaptive video reconstruction method is characterized in that during a high time resolution video reconstruction process based on compressed sensing, considering the situation that the velocity of movement for each moving object in a video image is not consistent, aiming at matching, tracking and other reconstruction methods based on a dictionary (a sparse domain), firstly during the process of constructing dictionaries, dividing a training sample into a plurality of sample sets according to difference of the amount of movement during the process of constructing the dictionaries, and respectively training the sample sets so as to obtain dictionaries corresponding to different amount of movement; secondly, during the video reconstruction step, performing blocking reconstruction without superposition on the observed images requiring reconstruction, then calculating the correlation coefficients between each frame of image blocks, and determining the local image amount of movement according to the magnitude of the correlation coefficients; and at last, according to the magnitude of the amount of movement, performing targeted selection of a dictionary for image reconstruction. The adaptive video reconstruction method based on signal correlation can distinguish the amount of movement for each object in the video during the process of reconstructing video signals, and can perform targeted image reconstruction according to the movement information, thus improving the video reconstruction effect and reducing the reconstruction time at the same time.

Description

A kind of adaptive video method for reconstructing based on signal correlation

Technical field

The invention belongs to image processing field, when particularly relating to a kind of self adaptation height based on signal correlation Between resolution video method for reconstructing.

Background technology

High time resolution video reconstruction technology based on compressed sensing, is by pixel is carried out single pixel Code exposure obtains encoded observed image, and observed image is rebuild acquisition one by recycling algorithm for reconstructing Series of video sequences image, is i.e. obtained the temporal resolution Extended Technology of 3 D video by two dimensional image.By It is to carry out restoration and reconstruction to less than the sampled signal of nyquist sampling rate in compressed sensing, therefore rebuilds letter Number levels of precision and reconstruction speed be emphasis of concern.Algorithm for reconstructing generally can be divided into based on l₁ Norm minimum method, iteration method, matching pursuit algorithm, convex rule method, weight based on bayesian theory Construction methods etc., wherein matching pursuit algorithm and part iteration method think that signal to be restored is at certain dictionary Or the expression coefficient under sparse territory is sparse, thus by estimating that sparse coefficient comes reconstruction signal, dictionary Generally use DCT base, wavelet basis etc., in order to preferably be rebuild effect, it is possible to regard known to utilization Frequently dictionary is trained by signal.

Actual video often has the object of multiple different motion speed, the figure of the same area difference interframe Similarity is had, if this region is without motion, then each frame signal dependency is equal to 1, the completeest between image signal Exactly the same, if motion is the biggest, signal correction is sparse the least.Under some specific coded systems (as Each pixel exposure time is identical), in observed image, static background part is clearly, it is not necessary to rebuild or Need not use the dictionary of training.Meanwhile, majority of case people more concerned be the fortune in video Dynamic region, if therefore rebuild image Zone Full according to same way, not only loses time, And the movable information of object cannot be obtained.Traditional self adaptation method for reconstructing generally is directed to based on pattra leaves The method for reconstructing of this theory or the observed image of specific coding is carried out range searching.

Summary of the invention

It is an object of the invention to during high time resolution video reconstruction based on compressed sensing, adaptive The quantity of motion size of each object in video should be estimated in ground, and rebuild image according to movable information pointedly, Thus reduce reconstruction time while improving reconstruction video effect.

It is an object of the invention to be achieved through the following technical solutions: a kind of based on signal correlation from Adaptive video method for reconstructing, the method comprises the following steps:

(1) sample classification, specifically:

(1.1) optical flow method is utilized to calculate the motion vector figure of consecutive frame image in Sample video；

(1.2) Sample video is carried out stochastical sampling, sample block size be 8 × 8 × T, T be video frame number；

(1.3) the mean motion amount of each sample block is calculated according to the motion vector figure in (1.1), and according to fortune Sample block is classified by momentum, obtains the sample set of Activity, wherein l is sample set Number, N is number of samples.

(2) dictionary training, specifically:

(2.1) utilize K-SVD algorithm that the sample set of Activity is trained respectively, obtain corresponding to The complete dictionary Ψ of Activity^l；

(2.2) dictionary of training in (2.1) is merged into a dictionary Ψ, Ψ=[Ψ¹,Ψ²,…]。

(3) preliminary piecemeal is rebuild, specifically；

(3.1) observed image is divided into the image block of not superposition, and tile size is 8 × 8；

(3.2) three-dimensional complete dictionary Ψ is built₀, wherein space dimension is the Two-dimensional Cosine base of 64 × 64, time dimension One-dimensional wavelet basis for T × T；

(3.3) OMP algorithm and dictionary Ψ are utilized₀Each image block is rebuild, obtains rebuilding video block.

(4) moving region classification, specifically:

(4.1) correlation coefficient between each two field picture block in the video block of preliminary reconstruction is calculated；

(4.2) correlation coefficient is taken the correlation coefficient as correspondence image block region after average, thus obtains whole The correlation coefficient figure of width image；

(4.3) according to Threshold segmentation correlation coefficient figure, motion profile is obtained.

(5) video signal reconstructed, specifically:

(5.1) image block each element in corresponding region in (4.3) gained motion profile when rebuilding is calculated Number, and it is multiplied by weight coefficient respectively, select the zones of different of training dictionary Ψ for weight according to individual numerical value Build；

(5.2) utilize the dictionary region selected in OMP algorithm and (5.1) that block is had superposition Block is rebuild, the most mobile 1 pixel.

Further, in step 1.3, described number of samples N=40000.

Further, in step 1.3, described Activity is [0,0.1], [0.1,1.5], [1.5,3] three Region.

Further, in step 4.3, described empirical value is 0.85 and 0.95, and will figure according to threshold value As pixel value is divided into 0,0.5 and 1.

Further, in step 5.1, described weight coefficient is respectively 2,2,1.

Further, in step 5.1, described selection rule is, region interior element 1 number selects the most at most Ψ¹, element 0.5 number selects Ψ the most at most², element 0 number selects Ψ the most at most³。

Beneficial effects of the present invention: during high time resolution video reconstruction based on compressed sensing, In view of the situation that moving object movement velocity each in video image is inconsistent, first pass through sample classification instruction The mode practiced, obtains the dictionary of corresponding different motion speed, makes dictionary more specific aim；Secondly, regarding Frequently, during signal reconstruction, the dependency of preliminary reconstruction video signal is utilized to judge the quantity of motion of institute's reconstruction regions, And then acquisition motion profile, owing to the block that preliminary reconstruction is non-superimposed is rebuild, and use simple Complete dictionary, one of percentage when therefore the preliminary reconstruction time only used complete dictionary perfect reconstruction； Select dictionary region to rebuild video more targetedly according to judged result, the same of reconstruction quality can improved Time reduce reconstruction time.

Accompanying drawing explanation

Fig. 1 is the inventive method schematic diagram.

Fig. 2 (a) is example Sample video.

Fig. 2 (b) is that example Sample video is by optical flow method calculated interframe movement vectogram.

Fig. 3 is that preliminary piecemeal rebuilds schematic diagram.

Fig. 4 is high time resolution video reconstruction process schematic.

Fig. 5 is correlation coefficient figure.

Fig. 6 motion profile.

Fig. 7 is according to motion profile adaptively selected dictionary schematic diagram.

Observed image when Fig. 8 (a) is to wait long exposure mode.

Fig. 8 (b) is observed image shown in Fig. 8 (a) by OMP algorithm, uses common complete dictionary to rebuild video The 5th frame in (8 frame).

Fig. 8 (c) is that observed image shown in Fig. 8 (a) is rebuild the 5th frame in video (8 frame) by the inventive method.

Detailed description of the invention

Below in conjunction with accompanying drawing, the invention will be further described.

A kind of based on signal correlation the self adaptation method for reconstructing that the present invention provides, mainly includes that sample divides Several steps such as class, dictionary training, the reconstruction of preliminary piecemeal, quantity of motion estimation and video reconstruction, such as Fig. 1 Shown in.

Step 1. sample classification

1-1 utilizes optical flow method to calculate the motion vector figure of consecutive frame image in Sample video, as shown in Figure 2, For the video of T frame, then having T-1 to open vectogram, in motion vector figure, the quantity of motion size of each point is for being somebody's turn to do The mould of point vector, takes T=8 here；

1-2 carries out stochastical sampling to Sample video, and sample block size is 8 × 8 × T；

1-3 calculates the mean motion amount of each sample block according to the motion vector figure in 1-1:

$d=\left\{\underset{i=1}{\overset{7}{\mathrm{\Σ}}}\underset{j=1}{\overset{32}{\mathrm{\Σ}}}{B}_{ij}\right\}/\[32\·(T-1)\]---\left(1\right)$

Wherein B_i∈R^1×64It it is the motion value in i-th motion vector figure in corresponding sample area (8 × 8) The vector obtained according to descending order, takes B here_iThe average of first 32 as sample area i-th Open the quantity of motion on vectogram.According to quantity of motion, sample block is classified, obtain the sample of corresponding Activity This collectionWherein l=1,2,3 is sample set number, and N is number of samples.To multitude of video Find after classifying, in most videos, the interframe movement amount of moving object within 3 pixels, The most here quantity of motion being divided into [0,0.1], [0.1,1.5], [1.5,3] three regions, number of samples is 40000.

Step 2. dictionary training

2-1 utilizes K-SVD algorithm to be trained the sample set of Activity respectively, obtain corresponding to The complete dictionary Ψ of Activity^l；

The mathematical model of signal sparse resolution theory is: a given set Ψ={ ψ_k, k=1,2 ..., K}, Wherein Ψ is dictionary, each element ψ in Ψ_kIt is referred to as dictionary atom.For any given signal X, Can be broken down into the linear combination of each atom under dictionary:

$X=\underset{k=1}{\overset{K}{\mathrm{\Σ}}}{\mathrm{\α}}_k{\mathrm{\ψ}}_k---\left(2\right)$

Wherein α is rarefaction representation coefficient.

The purpose of dictionary training is the basic function the selecting approximating spline notebook data as far as possible atom as dictionary. KSVD dictionary learning algorithm is a kind of iterative algorithm, by the way of updating the most by column, Realize the whole updating of dictionary.Definitions collection C=[c₁,c₂,…,c_K], when C gives timing, sample signal Y={y₁,y₂,…,y_NCan use its nearest code word to represent, i.e. y_i=C α_ij, wherein α_ijIt it is a sparse base In vector, this vector is only 1 at jth item, and its remainder is all 0.J is obtained by formula (3):

$\∀k\≠j\left|\right|y_i-{\mathrm{C\α}}_j|{|}_2^2\≤||y_i-{\mathrm{C\α}}_k|{|}_2^2---\left(3\right)$

This is considered as a kind of limiting case of rarefaction representation: sparse coefficient only has one, and is necessary for 1. Global error can be expressed as:

$Error=\underset{i=1}{\overset{K}{\mathrm{\Σ}}}{e}_i^2=\left|\right|Y-CA|{|}_F^2---\left(4\right)$

Optimal coded set is found to represent training sample in nearest-neighbor by solving formula (5):

$\underset{C,A}{min}\left\{\right||Y-CA|{|}_F^2\}---\left(5\right)$

Solve and mainly include two processes, first with K-means cluster by training sample Y with closeDegree be foundation, be divided into K group

$R_k^{(J-1)}=\left\{i\right|\&ForAll;l\&NotEqual;k,\left|\right|y_i-c_k^{(J-1)}|{|}_2<||y_i-c_l^{(J-1)}|{|}_2\}---\left(6\right)$

Then to C^(J-1)In every string update according to formula (7), and make J=J+1.Repeat the above steps is straight To convergence.

$c_k^{\left(J\right)}=\frac{1}{\left|R_k\right|}\underset{i\&Element;R_k^{(J-1)}}{\mathrm{\&Sigma;}}{y}_i---\left(7\right)$

Finally give the dictionary Ψ={ ψ after training_k, k=1,2 ..., K},

The preliminary piecemeal of step 3. is rebuild

Observed image is divided into the image block of not superposition by 3-1, and tile size is 8 × 8；

3-2 builds three-dimensional complete dictionary Ψ₀∈R^512×512, wherein space dimension is the Two-dimensional Cosine base of 64 × 64, Time dimension is the one-dimensional wavelet basis of 8 × 8:

${\mathrm{\&Psi;}}_0={\mathrm{\&Psi;}}_{dwt}\&CircleTimes;{\mathrm{\&Psi;}}_{dct}\&CircleTimes;{\mathrm{\&Psi;}}_{dct}---\left(8\right)$

Wherein Ψ_dwt, Ψ_dctIt is respectively one-dimensional wavelet basis and one-dimensional cosine basis,For Kronecker product.Due to Ψ₀ For complete (nonredundancy) base, and image is that non-superimposed is rebuild, and therefore the time of this process of reconstruction is the shortest, Only about the 1/100 of perfect reconstruction time.

3-3 utilizes OMP algorithm and dictionary Ψ₀Each image block is rebuild, obtains rebuilding video block, as attached Shown in Fig. 3.As shown in Figure 4, setting video signal is three dimensions to high time resolution video reconstruction process According to body E, (x, y, t), (x, y t) are each pixel sampling function on whole time of exposure to S (S (and x, y, t) ∈ 0,1}), then the observed image I of acquisition (x, y) is expressed as:

$I(x,y)=\underset{t=1}{\overset{N}{\mathrm{\&Sigma;}}}S(x,y,t)\&CenterDot;E(x,y,t)---\left(9\right)$

Wherein S (x, y, t) known.Formula (12) can write matrix form I=SE, wherein I (observation signal) and E (video signal) is respectively E, and (x, y, t) with S (x, y, vector form t).Owing to observation signal will be far fewer than regarding Frequently signal, therefore the equation is a underdetermined equation.According to compressive sensing theory, the reconstruct of video signal is asked Topic is represented by:

$\hat{E}=\arg \underset{E}{min}\left|\right|I-SE|{|}_2^2---\left(10\right)$

Wherein E can be write again as the rarefaction representation of a certain dictionary Ψ, i.e. E=Ψ θ, and wherein θ is sparse system Number, can be solved by algorithm for reconstructing.

Classify in step 4. moving region, specifically:

4-1 calculates the correlation coefficient in the video block of preliminary reconstruction between each two field picture block:

$\begin{array}{c}r=\frac{Cov(X_n,X_{n+1})}{D\left(X_n\right)D\left(X_{n+1}\right)}\\ =\frac{\underset{i=1}{\overset{64}{\mathrm{\&Sigma;}}}(x_{ni}-{\stackrel{\&OverBar;}{x}}_n)(x_{(n+1)i}-{\stackrel{\&OverBar;}{x}}_{n+1})}{\sqrt{\underset{i=1}{\overset{64}{\mathrm{\&Sigma;}}}{(x_{ni}-{\stackrel{\&OverBar;}{x}}_n)}^2\&CenterDot;\underset{i=1}{\overset{64}{\mathrm{\&Sigma;}}}{(x_{(n+1)i}-{\stackrel{\&OverBar;}{x}}_{n+1})}^2}}\end{array}---\left(11\right)$

Wherein X_n=[x_n1,…,x_n64]^TRebuilding image block signal for n-th frame, correlation coefficient absolute value is closer to 1 The most relevant, uncorrelated closer to 0.

The absolute value of each for video block interframe correlation coefficient is taken the phase relation after average as corresponding region by 4-2 Number, thus obtain the correlation coefficient figure of entire image, as shown in Figure 5；

4-3 empirically Threshold segmentation correlation coefficient figure, obtains motion profile, as shown in Figure 6, and this In in corresponding step 1-3 the empirical value of class interval be 0.85 and 0.95, the order element less than 0.85 is 0, the element between [0.85,0.95] is 0.5, and the element more than 0.95 is 1.

Step 5. video signal reconstructed:

5-1 in step 4-3 gained moving region scattergram, selects the difference of training dictionary Ψ according to image block Region is used for rebuilding: calculate rebuild image block in the scattergram of 4-3 gained moving region in corresponding region 0, The number of 0.5 and 1, and it is multiplied by weight coefficient respectively, region interior element 1 number selects Ψ the most at most¹, Element 0.5 number selects Ψ the most at most², element 0 number selects Ψ the most at most³.As shown in Figure 7, weight Build block on motion profile in overlay area the number of 0,0.5,1 be respectively 20,12,32, due to More concerned with moving region (i.e. 0,1 corresponding region), the number greater weight of 0 and 1 can be given, here Using weight is 2, and therefore final number is respectively 40,24 and 32, selects dictionary when this image block is rebuild Ψ³。

5-2 utilizes OMP algorithm and training dictionary Ψ to have the block of superposition to rebuild, every time to block Mobile 1 pixel.

The inventive method can make reconstruction time reduce by more than 50, and improves reconstructed image quality simultaneously, as attached Shown in Fig. 8, it can be seen that the inventive method can preferably reconstruct moving region.

Claims (6)

1. an adaptive video method for reconstructing based on signal correlation, it is characterised in that the method bag Include following steps:

(1) sample classification, specifically:

(1.1) optical flow method is utilized to calculate the motion vector figure of consecutive frame image in Sample video；

(1.2) Sample video is carried out stochastical sampling, sample block size be 8 × 8 × T, T be video frame number；

(1.3) the mean motion amount of each sample block is calculated according to the motion vector figure in (1.1), and according to fortune Sample block is classified by momentum, obtains the sample set of ActivityWherein l is sample set Number, N is number of samples.

(2) dictionary training, specifically:

(2.1) utilize K-SVD algorithm that the sample set of Activity is trained respectively, obtain corresponding to The complete dictionary Ψ of Activity^l；

(2.2) dictionary of training in (2.1) is merged into a dictionary Ψ, Ψ=[Ψ¹,Ψ²,…]。

(3) preliminary piecemeal is rebuild, specifically；

(3.1) observed image is divided into the image block of not superposition, and tile size is 8 × 8；

(3.2) three-dimensional complete dictionary Ψ is built₀, wherein space dimension is the Two-dimensional Cosine base of 64 × 64, time dimension One-dimensional wavelet basis for T × T；

(3.3) OMP algorithm and dictionary Ψ are utilized₀Each image block is rebuild, obtains rebuilding video block.

(4) moving region classification, specifically:

(4.1) correlation coefficient between each two field picture block in the video block of preliminary reconstruction is calculated；

(4.2) correlation coefficient is taken the correlation coefficient as correspondence image block region after average, thus obtains whole The correlation coefficient figure of width image；

(4.3) according to Threshold segmentation correlation coefficient figure, motion profile is obtained.

(5) video signal reconstructed, specifically:

(5.1) image block each element in corresponding region in (4.3) gained motion profile when rebuilding is calculated Number, and it is multiplied by weight coefficient respectively, select the zones of different of training dictionary Ψ for weight according to individual numerical value Build；

(5.2) utilize the dictionary region selected in OMP algorithm and (5.1) that block is had superposition Block is rebuild, the most mobile 1 pixel.

A kind of adaptive video method for reconstructing, it is characterised in that step 1.3 In, described number of samples N=40000.

A kind of adaptive video method for reconstructing, it is characterised in that step 1.3 In, described Activity is [0,0.1], [0.1,1.5], [1.5,3] three regions.

A kind of adaptive video method for reconstructing, it is characterised in that step 4.3 In, described threshold value is 0.85 and 0.95, by threshold value, pixel value is divided into 0,0.5 and 1.

A kind of adaptive video method for reconstructing, it is characterised in that step 5.1 In, described weight is respectively 2,2,1.

A kind of adaptive video method for reconstructing, it is characterised in that step 5.1 In, described selection rule is, region interior element 1 number selects Ψ the most at most¹, element 0.5 number is most Then select Ψ², element 0 number selects Ψ the most at most³。