CN105160664A - Low-rank model based compressed sensing video reconstruction method - Google Patents

Abstract

The invention discloses a compressed sensing video reconstruction method based on a low-rank model, which mainly solves the problems of inaccurate and low robustness of the compressed sensing video reconstruction. The implementation process is: (1) receiving measurement data; (2) initializing A collection of single-frame covariance matrices; (3) Construct an initial reconstructed video based on piecewise linear estimation of joint sparse and Gaussian distributions; (4) Search for similar blocks for each image block using the inter- and intra-frame correlation of the video; ( 5) Solving the low-rank structure of the video; (6) updating the reconstructed video; (7) judging whether the iteration stop condition is reached; (8) outputting the reconstructed video. Compared with the existing video reconstruction technology, the invention has the advantages of high reconstructed image quality and good robustness, and can be used for natural scene video reconstruction.

Description

Compressed Sensing Video Reconstruction Method Based on Low-rank Model

technical field

The invention belongs to the technical field of image processing, and further relates to a video reconstruction method based on a compressed sensing system in the technical field of video coding. The invention excavates the similarity between frames and frames of the video sequence, reconstructs the video based on the low-rank model, and can be used to reconstruct the video sequence of the natural image.

Background technique

In recent years, a new data theory compressive sensing CS has emerged in the field of signal processing. This theory realizes compression while collecting data, breaks through the limitations of the traditional Nyquist sampling theorem, and brings new advantages to data collection technology. The revolutionary changes make the theory have broad application prospects in compressed imaging systems, military cryptography, wireless sensing and other fields. Compressed sensing theory mainly includes three aspects: sparse representation of signal, observation of signal and reconstruction of signal. Among them, designing a fast and effective reconstruction algorithm is an important part of successfully promoting and applying CS theory to actual data models and acquisition systems.

High-speed cameras play an important role in capturing fast motion in applications ranging from science to sports, but measuring high-speed video presents a challenge to camera design. Compressed sensing can capture high frame rate video information through low frame rate compression measurement, so compressed sensing is used to capture high-speed video information, thereby reducing the difficulty of high-speed camera design.

In their paper "SolvingInverseProblemsWithPiecewiseLinearEstimators: FromGaussianMixtureModelstoStructuredSparsity" ("IEEE Transactions on Image Processing", 2012, 21(5): 2481-2499), GuoshenYu et al. proposed a method for solving image inverse problems using piecewise linear estimation. This method models the problem as a Gaussian mixture model, that is, an image patch obeys one of multiple multivariate Gaussian distributions, and it achieves better results on image inverse problems including image reconstruction. The disadvantage of this method is that this method is aimed at two-dimensional images and cannot be directly used for the reconstruction of video sequences, and it is not accurate to use statistical methods to capture the similarity between image blocks.

Janbo Yang et al. proposed a method based on the Gaussian mixture model in their paper "VideoCompressiveSensingUsingGaussianMixtureModels" ("IEEETransaction on Image Processing A Publication of the IEEE Signal Processing Society", 2014, 23). This method reconstructs the time-compressed video sequence by establishing a Gaussian mixture model for the spatio-temporal video block, and obtains a better reconstruction effect, but the method still has the disadvantage that the reconstruction method only establishes a The Gaussian mixture model does not capture the correlation between video frames and frames, which leads to inaccurate video sequences reconstructed by this method.

Contents of the invention

The object of the present invention is to propose a method for compressive sensing video reconstruction based on a low-rank model to improve the correlation between video frames and inter-frames in the spatio-temporal video compression sensing reconstruction technology in the prior art. The quality of the reconstructed image.

The technical idea of realizing the purpose of the present invention is: using the correlation between video frames, that is, the same position of different frames of video has similarity, and the image blocks of the same position in different frames of modeling have the same Gaussian distribution, so as to obtain the initial reconstructed video Sequence; establish a low-rank model for all similar image blocks between frames and within a frame, and iteratively optimize and solve the low-rank structure of the video and the reconstructed video sequence, realizing high-quality temporal video compression perception reconstruction.

The concrete steps that realize the object of the present invention are as follows:

(1) Receive measurement data:

(1a) The compressed sensing sender observes the video data, and uses a random mask observation matrix for each H frame of video data to make one observation result to form a frame of measurement data, and sends the measurement data and random mask observation matrix, where , H represents a positive integer ranging from 1 to 20;

(1b) The receiver receives the measurement data and random mask observation matrix sent by the sender;

(2) Initialize the single frame covariance matrix set:

(2a) Generate 18 artificial black-and-white images, the size of each artificial black-and-white image is 65×65 pixels, and each artificial black-and-white image represents a direction;

(2b) Using a window size of 8×8 pixels, with a step size of 1 pixel, slide the window on the artificial black and white image in each direction to select all blocks with a size of 8×8 pixels, and obtain the direction blocks in each direction gather;

(2c) Perform principal component analysis (PCA) decomposition on the set of direction blocks in each direction to obtain the PCA orthogonal basis and eigenvalue matrix of principal component analysis, and retain the first 8 largest eigenvalues and corresponding principal component positive values in each direction. Intersect basis to get the corresponding eigenvalue matrix and direction basis;

(2d) Calculate the single-frame covariance matrix in the direction represented by each artificial black-and-white image to obtain a single-frame covariance matrix set;

(3) Construct the initial reconstructed video based on the piecewise linear estimation of the joint sparse and Gaussian distribution:

(3a) For the direction represented by each artificial black-and-white image, place the H single-frame covariance matrices in the direction on the diagonal of the matrix, and construct the direction represented by each artificial black-and-white image for reconstructing 3D video The joint sparse video covariance matrix of the data, where the video covariance matrix in the k-th direction is as follows:

in, Represent the video covariance matrix of the kth direction, P _k represents the single frame covariance matrix of the kth direction, k represents the direction number represented by the artificial black and white image, k=1,2,...,18;

(3b) The M×N×H-dimensional reconstructed video is initially a zero matrix, and each frame of the initial reconstructed video is divided into n×n-dimensional S image blocks with a step size of p, and the blocks are reserved The position of each frame is composed of image blocks at the same position as video blocks, and a set of video blocks {x ₁ ,...,x _l ,...,x _S } is obtained, where, x _l represents the lth video block, Represents the l-th image block with size n×n dimensions of the t-th frame of the reconstructed video, T represents the transpose operation, for the l-th video block in the direction k represented by the artificial black-and-white image Obedience mean is 0, covariance matrix is Gaussian distribution, Represents the video covariance matrix in the k-th direction, M, N, H represent the size of the first dimension, the second dimension, and the third dimension in the reconstructed video, respectively, p, n represent the smallest dimension of M and N, respectively is a positive integer of the value of , k represents the direction number represented by the artificial black-and-white image, k=1,2,...,18, l=1,2,...,S, S represents the number of video blocks, that is, each The number of image blocks divided into one frame;

(3c) Calculate the estimated value of the video block in the direction represented by the artificial black and white image according to the following formula:

${\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; =</span>{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; P</span>}}}_{\mathrm{<span\; class="notranslate">\; k</span>}}{{\mathrm{<span\; class="notranslate">\; \&Phi;</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}}^{\mathrm{<span\; class="notranslate">\; T</span>}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; \&Phi;</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; P</span>}}}_{\mathrm{<span\; class="notranslate">\; k</span>}}{{\mathrm{<span\; class="notranslate">\; \&Phi;</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}}^{\mathrm{<span\; class="notranslate">\; T</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; d</span>}}<span\; class="notranslate">\; )</span>}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}$

in, Indicates the estimated value of the lth video block in the direction k represented by the artificial black and white image, Indicates the k-th video covariance matrix, Φ _l represents the observation matrix of the l-th video block taken from the random mask observation matrix, y _l represents the measurement data of the l-th video block taken from the measurement data, and the value of σ The range is 0 to 1, I _d represents the d-dimensional unit matrix, T represents the transpose operation, ^-1 represents the matrix inversion, k represents the direction number represented by the artificial black and white image, k=1,2,...,18 , l represents the numbering of video blocks, l=1,2,...,S, S represents the number of video blocks, i.e. the number of image blocks divided by each frame;

(3d) According to the following formula, calculate the optimal direction of the direction represented by the artificial black and white image:

${\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; k</span>}}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\underset{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; \&Phi;</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; T</span>}}{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; P</span>}}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; l</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; |</span>{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; P</span>}}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; )</span>$

in, Indicates the optimal direction of the lth video block in the direction represented by the artificial black and white image, Indicates to return the value of k when the objective function value is minimized, Φ _l represents the observation matrix of the lth video block taken from the random mask observation matrix, Indicates the estimated value of the l-th video block in the direction k represented by the artificial black-and-white image, y _l represents the measurement data of the l-th video block taken out from the measurement data, Indicates the kth video covariance matrix, the value of σ ranges from 0 to 1, ‖·‖ ² represents the square of the norm, |·| represents the value of the determinant, T represents the transpose operation, · ^-1 represents the matrix calculation Inverse, k represents the direction number represented by the artificial black and white image, k=1,2,...,18, l represents the number of video blocks, l=1,2,...,S, S represents the number of video blocks , that is, the number of image blocks divided by each frame;

(3e) the estimated value of each video block on its optimal direction, according to step (3b) the position of the sub-block that is reserved when dividing the video block, is combined into initial reconstructed video;

(3f) Set the number of iterations s and the maximum number of external iterations U, make the current number of iterations s 1, and use the initial reconstructed video as the reconstructed video of the first iteration, where U represents a positive integer;

(4) Use the correlation between video frames and frames to search for similar blocks of each image block:

(4a) Divide each frame of the reconstructed video into n×n-dimensional blocks with a step size of p, and form blocks of all frames into a two-dimensional image block set G ¹ , where p, n represent less than or equal to M, A positive integer of the minimum value in N dimensions, M and N represent the size of the first dimension and the second dimension of the reconstructed video;

(4b) Divide each frame of the reconstructed video into n×n-dimensional blocks with a step size of 1, form blocks of all frames into a two-dimensional image block set G ² , and record the index of the video block in G ² , Wherein, n represents a positive integer less than or equal to the smallest value in M and N dimensions, and M and N represent the size of the first dimension and the second dimension of the reconstructed video;

(4c) For each block in G ¹ , take out all blocks in the Z×Z×H window around the block from G ² , and record them as the neighboring blocks of the block, where Z represents the first dimension of the window and the second Dimension size, H represents the size of the third dimension of the window, G ¹ represents a two-dimensional image block set divided into blocks with a step size of p, G ² represents a two-dimensional image block set divided into blocks with a step size of 1;

(4d) Calculate the Euclidean distance between each block in the two-dimensional video block set G ¹ and its neighbor blocks, sort according to the Euclidean distance from small to large, select the first Q blocks as similar blocks of the corresponding block, and record the similar blocks of each block Index in G2, where Q represents a positive integer less ^than half the number of adjacent blocks;

(5) Solve the low-rank structure of the video according to the following formula:

${{\mathrm{<span\; class="notranslate">\; L</span>}}_{{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}}}^{\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; L</span>}}_{{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; R</span>}}}_{{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; f</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{<span\; class="notranslate">\; *</span>}$

in, Represents the low-rank structure of the lth image block in the tth frame of the s+1th iteration, Represents the low-rank structure of the image block when the objective function value is minimized the value of Indicates the extraction transformation for extracting all similar blocks of the l-th image block in the t-th frame, X ^s represents the reconstructed video of the s-th iteration, ${\tilde{R}}_{l_t}\left(x^{\mathrm{the\; s}}\right)=\left(R_{l_t^1}\right(x^{\mathrm{the\; s}}),...,R_{l_t^q}(x^{\mathrm{the\; s}}),...R_{l_t^Q}(x^{\mathrm{the\; s}}\left)\right),$ Represents the extraction transformation of all similar blocks of the l-th image block in the t-th frame of the s-th iteratively reconstructed video X ^s , Represents the extraction matrix of the qth similar block that extracts the lth image block of the tth frame, Represents the extraction matrix of the qth similar block of the tth frame l image block of the reconstructed video X ^s of the sth iteration, the value of λ is 0.75, Indicates the square operation of the Frobenius norm of Frobenius, ‖·‖ _* indicates the nuclear norm operation, H indicates the size of the third dimension of the reconstructed video, t indicates the number of the video frame, that is, the number of the image block of the tth frame , t=1,2,...,H, l represents the numbering of video blocks, l=1,2,...,S, S represents the number of video blocks, i.e. the number of image blocks divided by each frame, q=1,2,...,Q, Q represents the number of similar blocks;

(6) Update and reconstruct the video according to the following formula:

${\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; x</span>}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&Phi;</span>}\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&eta;</span>}\underset{\mathrm{<span\; class="notranslate">\; t</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}\underset{\mathrm{<span\; class="notranslate">\; l</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; R</span>}}}_{{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}}^{\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; f</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}$

Among them, X ^s+1 represents the reconstructed video of the s+1th iteration, Indicates to take the value of the reconstructed video X when the value of the objective function is the smallest, y represents the one-dimensional vector drawn from the measurement data, Φ represents the video observation matrix generated by the random mask observation matrix, Indicates the extraction transformation of all similar blocks of the l-th image block in the t-th frame, X represents the reconstructed video, ${\tilde{R}}_{l_t}\left(x\right)=\left(R_{l_t^1}\right(x),...,R_{l_t^q}(x),...R_{l_t^Q}(x\left)\right),{\tilde{R}}_{l_t}\left(x\right)$ Represents the extraction transformation of all similar blocks of the l-th image block in the t-th frame of the reconstructed video X, Represents the extraction matrix of the qth similar block that extracts the lth image block of the tth frame, Represents the extraction matrix for extracting the qth similar block of the tth frame l image block of the reconstructed video X, Represents the low-rank structure of the lth image block in the tth frame of the s+1th iteration, the value of η is 1, and Σ represents the summation operation, Indicates the square operation of the 2 norm, Indicates the square operation of the Frobenius norm of Frobenius, H indicates the size of the third dimension of the reconstructed video, t indicates the number of the video frame, that is, the number of the image block of the tth frame, t=1,2,... , H, l represents the numbering of video blocks, l=1,2,...,S, S represents the number of video blocks, that is, the number of image blocks divided by each frame, q=1,2,..., Q, Q represents the number of similar blocks;

(7) Determine whether the current iteration number is greater than the maximum external iteration number, if so, execute step (8), otherwise add 1 to the current iteration number, and execute step (4);

(8) Output reconstructed video.

Compared with the prior art, the present invention has the following advantages:

First, the present invention utilizes the method of constructing the initial reconstructed video based on the piecewise linear estimation of the joint sparse and Gaussian distribution, which overcomes the fact that the piecewise linear reconstruction method in the prior art cannot be directly used for the reconstruction of the video sequence, making this The invention has the advantage of improving the robustness of the video reconstruction method.

Second, the present invention utilizes the correlation between video frames and intra-frames to search for similar blocks of each image block, solves the low-rank structure of the video, and solves the reconstructed video through iterative optimization, which overcomes the problem of Gaussian mixture model in the prior art. The method does not utilize the disadvantages of correlation within and between video frames, so that the present invention has the advantage of improving the accuracy of reconstructed video.

Description of drawings

Fig. 1 is a flow chart of the present invention;

Fig. 2 is the vehicle video reconstruction result figure when the present invention and prior art are H=8;

Fig. 3 is a broken line diagram of peak signal-to-noise ratio PSNR of vehicle video reconstructed by the method of the present invention and the prior art.

detailed description

The present invention will be further described below in conjunction with the accompanying drawings.

With reference to accompanying drawing 1, concrete steps of the present invention are as follows.

Step 1, receiving measurement data.

The compressed sensing sender observes the video data, and uses a random mask observation matrix for each H frame of video data to make one observation result to form a frame of measurement data, and sends the measurement data and random mask observation matrix, where H represents A positive integer with a value ranging from 1 to 20.

The receiver receives the measurement data and the random mask observation matrix sent by the sender.

Raw video data in M×N×H dimensions Use the M×N×H-dimensional random mask observation matrix A for observation, and obtain a frame of M×N-dimensional measurement data Y: $Y_{i,j}=\&lsqb;A_{i,j,1},A_{i,j,2},...,A_{i,j,t},...,A_{i,j,h}\&rsqb;{\&lsqb;{\hat{x}}_{i,j,1},{\hat{x}}_{i,j,2},...,{\hat{x}}_{i,j,t},...,{\hat{x}}_{i,j,h}\&rsqb;}^T,$ · ^T means to do the transpose operation of the vector, Y _{i, j} means the elements of the i-th row j column of the measurement data Y, where i=1,2,...,M, j=1,2,..., N, Represents raw video data Elements in the i-th row and j-column on the t-th frame, where t=1,2,...,H, A _i,j,n represent the elements of the i-th row and j-column on the t-th frame of the random mask observation matrix A , A _{i, j, n} are randomly 0 or 1 with probability r.

Step 2, initialize the single frame covariance matrix set.

Generate 18 artificial black-and-white images, the size of each artificial black-and-white image is 65×65 pixels, and each artificial black-and-white image represents a direction.

Using a window size of 8×8 pixels, with a step size of 1 pixel, slide the window to select all blocks with a size of 8×8 pixels on the artificial black and white image in each direction, and obtain a set of direction blocks in each direction.

Perform principal component analysis (PCA) decomposition on the set of direction blocks in each direction to obtain the PCA orthogonal basis and eigenvalue matrix of principal component analysis, and retain the first 8 largest eigenvalues and corresponding principal component orthogonal basis in each direction. The corresponding eigenvalue matrix and direction basis are obtained.

Calculate the single-frame covariance matrix in the direction represented by each artificial black-and-white image to obtain a single-frame covariance matrix set.

According to the following formula, calculate the single-frame covariance matrix in the direction represented by each black and white image:

P _k = B _k D _k B _k ^T ,

Among them, P _k represents the single-frame covariance matrix in the k-th direction, B _k represents the direction basis in the k-th direction, D _k represents the eigenvalue matrix in the k-th direction, T represents the transpose operation, and k represents Direction number, k=1,2,...,18.

The steps of principal component analysis PCA decomposition are as follows:

Select a direction from all the directions of the artificial black and white image, and calculate the covariance matrix of the direction block set of the selected direction according to the following formula:

P＝E[f _i f _i ^T ]

Among them, P represents the covariance matrix of the direction block set of the selected direction, E represents the mathematical expectation, f _i represents the ith block in the direction block set of the selected direction, and T represents the transpose operation.

According to the following formula, the covariance matrix is diagonalized to obtain the PCA orthogonal basis and eigenvalue matrix of principal component analysis:

P = BDB ^T

Among them, P represents the covariance matrix of the direction block set of the selected direction, B represents the PCA orthogonal basis of the selected direction, D represents the eigenvalue matrix of the direction, and T represents the transpose operation.

Step 3. Construct an initial reconstructed video based on piecewise linear estimation of joint sparse and Gaussian distributions.

For the direction represented by each artificial black-and-white image, put the H single-frame covariance matrices in the direction on the diagonal of the matrix, and jointly sparsely represent the three-dimensional video data representation of the direction represented by each artificial black-and-white image Video covariance matrix, wherein, the video covariance matrix of the kth direction is as follows:

in, Represents the video covariance matrix of the k-th direction, P _k represents the single-frame covariance matrix of the k-th direction, k represents the direction number represented by the artificial black and white image, k=1,2,...,18.

The M×N×H-dimensional reconstructed video is initially a zero matrix, and each frame of the initial reconstructed video is divided into n×n-dimensional S image blocks with a step size of p, and the position of the block is reserved. The image blocks at the same position in each frame are composed of video blocks to obtain a set of video blocks {x ₁ ,...,x _l ,...,x _S }, where, x _l represents the lth video block, Represents the l-th image block with size n×n dimensions of the t-th frame of the reconstructed video, T represents the transpose operation, for the l-th video block in the direction k represented by the artificial black-and-white image Obedience mean is 0, covariance matrix is Gaussian distribution, Represents the video covariance matrix in the k-th direction, M, N, H represent the size of the first dimension, the second dimension, and the third dimension in the reconstructed video, respectively, p, n represent the smallest dimension of M and N, respectively is a positive integer of the value of , k represents the direction number represented by the artificial black-and-white image, k=1,2,...,18, l=1,2,...,S, S represents the number of video blocks, that is, each The number of image blocks divided into one frame.

Using piecewise linear estimation to calculate the optimal direction of the video block in the direction represented by the artificial black and white image, the steps are as follows:

Calculate the estimated value of the video block in the direction represented by the artificial black and white image according to the following formula:

${\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; =</span>{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; P</span>}}}_{\mathrm{<span\; class="notranslate">\; k</span>}}{{\mathrm{<span\; class="notranslate">\; \&Phi;</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}}^{\mathrm{<span\; class="notranslate">\; T</span>}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; \&Phi;</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; P</span>}}}_{\mathrm{<span\; class="notranslate">\; k</span>}}{{\mathrm{<span\; class="notranslate">\; \&Phi;</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}}^{\mathrm{<span\; class="notranslate">\; T</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; d</span>}}<span\; class="notranslate">\; )</span>}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}$

in, Indicates the estimated value of the lth video block in the direction k represented by the artificial black and white image, Indicates the k-th video covariance matrix, Φ _l represents the observation matrix of the l-th video block taken from the random mask observation matrix, y _l represents the measurement data of the l-th video block taken from the measurement data, and the value of σ The range is 0 to 1, I _d represents the d-dimensional unit matrix, T represents the transpose operation, ^-1 represents the matrix inversion, k represents the direction number represented by the artificial black and white image, k=1,2,...,18 , l represents the serial number of the video block, l=1, 2,..., S, S represents the number of video blocks, that is, the number of image blocks divided into each frame.

Calculate the optimal direction of the direction represented by the artificial black and white image according to the following formula:

${\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; k</span>}}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\underset{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; \&Phi;</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; T</span>}}{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; P</span>}}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; l</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; |</span>{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; P</span>}}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; )</span>$

in, Indicates the optimal direction of the lth video block in the direction represented by the artificial black and white image, Indicates to return the value of k when the objective function value is minimized, Φ _l represents the observation matrix of the lth video block taken from the random mask observation matrix, Indicates the estimated value of the l-th video block in the direction k represented by the artificial black-and-white image, y _l represents the measurement data of the l-th video block taken out from the measurement data, Indicates the kth video covariance matrix, the value of σ ranges from 0 to 1, ‖·‖ ² represents the square of the norm, |·| represents the value of the determinant, T represents the transpose operation, · ^-1 represents the matrix calculation Inverse, k represents the direction number represented by the artificial black and white image, k=1,2,...,18, l represents the number of video blocks, l=1,2,...,S, S represents the number of video blocks , that is, the number of image blocks divided by each frame.

The estimated value of each video block in its optimal direction is combined into an initial reconstructed video according to the reserved block position when dividing the video block.

Set the number of iterations s and the maximum number of external iterations U, set the current number of iterations s to 1, and use the initial reconstructed video as the reconstructed video of the first iteration, where U represents a positive integer.

Step 4, using the inter-frame and intra-frame correlation of the video to search for similar blocks of each video block.

Divide each frame of the reconstructed video into n×n-dimensional blocks with a step size of p, and form blocks of all frames into a two-dimensional image block set G ¹ , where p and n represent dimensions less than or equal to M and N A positive integer with the smallest value in , M and N represent the size of the first dimension and the second dimension of the reconstructed video.

Divide each frame of the reconstructed video into n×n-dimensional blocks with a step size of 1, form blocks of all frames into a two-dimensional image block set G ² , and record the index of the video block in G ² , where n Represents a positive integer less than or equal to the smallest value among M and N dimensions, M and N represent the size of the first and second dimensions of the reconstructed video.

For each block in ^G1 , take out all the blocks in the Z×Z×H window around the block from G2, and record it as the adjacent block of the block, where Z represents the size of the first and ^second dimensions of the window , H represents the size of the third dimension of the window, G ¹ represents a set of two-dimensional image blocks divided into blocks with a step size of p, and G ² represents a set of two-dimensional image blocks divided into blocks with a step size of 1.

Calculate the Euclidean distance between each block in the two-dimensional video block set G ¹ and its neighbor blocks, sort according to the Euclidean distance from small to large, select the first Q blocks as similar blocks of the corresponding block, and record the similar blocks of each block in G ² The index in , where Q represents a positive integer less than half the number of adjacent blocks.

^The index of similar blocks in G2 for each block is denoted as in, is called the similarity block index of the lth block in the tth frame, Represents the index value of the qth block similar to this block, q=1,2,...,Q, t represents the number of the video frame, t=1,2,...,H, l represents the number of the video block Numbering, l=1,2,...,S, H represents the size of the third dimension of the reconstructed video, S represents the number of video blocks, that is, the number of image blocks divided by each frame, and Q represents the number of blocks less than one adjacent block half positive integer.

Step 5, solve the low-rank structure of the video according to the following formula.

${{\mathrm{<span\; class="notranslate">\; L</span>}}_{{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}}}^{\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; L</span>}}_{{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; R</span>}}}_{{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; f</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{<span\; class="notranslate">\; *</span>}$

in, Represents the low-rank structure of the lth image block in the tth frame of the s+1th iteration, Represents the low-rank structure of the image block when the objective function value is minimized the value of Indicates the extraction transformation for extracting all similar blocks of the l-th image block in the t-th frame, X ^s represents the reconstructed video of the s-th iteration, ${\tilde{R}}_{l_t}\left(x^{\mathrm{the\; s}}\right)=(R_{l_t^1}\left(x^{\mathrm{the\; s}}\right),...,R_{l_t^q}\left(x^{\mathrm{the\; s}}\right),...R_{l_t^Q}\left(x^{\mathrm{the\; s}}\right)),$ Represents the extraction transformation of all similar blocks of the l-th image block in the t-th frame of the s-th iteratively reconstructed video X ^s , Represents the extraction matrix of the qth similar block that extracts the lth image block of the tth frame, Represents the extraction matrix of the qth similar block of the tth frame l image block of the reconstructed video X ^s of the sth iteration, the value of λ is 0.75, Indicates the square operation of the Frobenius norm of Frobenius, ‖·‖ _* indicates the nuclear norm operation, H indicates the size of the third dimension of the reconstructed video, t indicates the number of the video frame, that is, the number of the image block of the tth frame , t=1,2,...,H, l represents the numbering of video blocks, l=1,2,...,S, S represents the number of video blocks, i.e. the number of image blocks divided by each frame, q=1,2,...,Q, Q represents the number of similar blocks,

right Perform SVD decomposition: Among them, U represents the left unitary matrix, V represents the right unitary matrix, and Λ singular value matrix.

Soft thresholding is performed on the singular value matrix to obtain ${\widetilde{\mathrm{\&Lambda;}}}_{ij}=\left\{\begin{array}{cc}{\mathrm{\&Lambda;}}_{ij}-\mathrm{\&lambda;}& {\mathrm{\&Lambda;}}_{ij}>\mathrm{\&lambda;}\\ 0& {\mathrm{\&Lambda;}}_{ij}\&le;\mathrm{\&lambda;}\end{array}\right.,$ representation matrix The element in row i and column j of .

Compute the low-rank structure at iteration s+1:

Step 6: Update and reconstruct the video according to the following formula.

${\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; x</span>}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&Phi;</span>}\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&eta;</span>}\underset{\mathrm{<span\; class="notranslate">\; t</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}\underset{\mathrm{<span\; class="notranslate">\; l</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; R</span>}}}_{{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}}^{\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; f</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}$

Among them, X ^s+1 represents the reconstructed video of the s+1th iteration, Indicates to take the value of the reconstructed video X when the value of the objective function is the smallest, y represents the one-dimensional vector drawn from the measurement data, Φ represents the video observation matrix generated by the random mask observation matrix, Indicates the extraction transformation of all similar blocks of the l-th image block in the t-th frame, X represents the reconstructed video, ${\tilde{R}}_{l_t}\left(x\right)=\left(R_{l_t^1}\right(x),...,R_{l_t^q}(x),...R_{l_t^Q}(x\left)\right),$ Represents the extraction transformation of all similar blocks of the l-th image block in the t-th frame of the reconstructed video X, Represents the extraction matrix of the qth similar block that extracts the lth image block of the tth frame, Represents the extraction matrix for extracting the qth similar block of the tth frame l image block of the reconstructed video X, Represents the low-rank structure of the lth image block in the tth frame of the s+1th iteration, the value of η is 1, and Σ represents the summation operation, Indicates the square operation of the 2 norm, Indicates the square operation of the Frobenius norm of Frobenius, H indicates the size of the third dimension of the reconstructed video, t indicates the number of the video frame, that is, the number of the image block of the tth frame, t=1,2,... , H, l represents the numbering of video blocks, l=1,2,...,S, S represents the number of video blocks, that is, the number of image blocks divided by each frame, q=1,2,..., Q, Q represents the number of similar blocks.

The solution for the reconstructed video at iteration s+1: $x^{\mathrm{the\; s}+1}={({\mathrm{\&Phi;}}^T\mathrm{\&Phi;}+\mathrm{\&eta;}\underset{t}{\&Sigma;}\underset{l}{\&Sigma;}{\tilde{R}}_{l_t}^T{\tilde{R}}_{l_t})}^{-1}({\mathrm{\&Phi;}}^T\mathrm{the\; y}+\mathrm{\&eta;}\underset{t}{\&Sigma;}\underset{l}{\&Sigma;}{\tilde{R}}_{l_t}^T{L}_{l_t}^{\mathrm{the\; s}+1})$

in, ${\tilde{R}}_{l_t}^T{\tilde{R}}_{l_t}=\underset{l_t^q\&Element;{\mathrm{E.}}_{l_t}}{\&Sigma;}{R}_{l_t^q}^T{R}_{l_t^q},{\tilde{R}}_{l_t}=(R_{l_t^1},...,R_{l_t^q},...R_{l_t^Q}).$

Step 7, judge whether the current number of iterations is greater than the maximum number of external iterations, if so, go to step 8, otherwise, add 1 to the current number of iterations, go to step 4.

Step 8, output the reconstructed video.

The present invention will be further described below in conjunction with the simulation diagram.

1. Simulation conditions:

The operating system of the simulation experiment of the present invention is CPUIntel(R) Core(TM) i5-3470 3.20GHz, 32-bit windows7 operating system, the simulation software adopts MatlabR2011b, and the simulation parameters are set as follows.

What the simulation of the present invention selects is 256 * 256 96 frames of traffic vehicle pictures, every H=8 frames of video data are observed once, and the element A _{i in the random mask observation matrix A, j, n} is randomly 1 with probability r=0.5 , block size n×n=8×8, step size p=4, search similar block window size Z×Z×H=50×50×8, number of similar blocks Q=100, maximum number of iterations U=10, where Given parameters λ=0.75, η=1.

2. Simulation content and result analysis:

Under the above simulation conditions, three different comparison methods are used to reconstruct the pictures of 96 frames of traffic vehicles:

Comparative method 1 is the Gaussian mixture model method of the prior art, and reconstructs the pictures of 96 frames of traffic vehicles;

Contrast method 2 is to adopt steps 1, 2, and 3 of the present invention to construct the initial reconstructed video, reconstruct the pictures of 96 frames of traffic vehicles, and use the initial reconstructed video as the reconstruction result;

Comparative method 3 is to use the Gaussian mixture model method in the prior art to construct the initial reconstructed video, instead of step 3 of the present invention, to reconstruct 96 frames of pictures of traffic vehicles.

The reconstructed visual effects of the three comparison methods and the method of the present invention are shown in Figure 2, wherein Figure 2(a) is the original image of the first frame of the 96-frame traffic vehicle video, and Figure 2(b) is the first frame of the 96-frame traffic vehicle video 2 frames of the original image, Figure 2(c) is the original image of the third frame of the 96-frame traffic vehicle video, Figure 2(d) is the original image of the third frame of the 96-frame traffic vehicle video, Figure 2(e) is for comparison Method 1 reconstructs the first frame of the traffic vehicle video. Figure 2(f) is the reconstruction result of the second frame of the traffic vehicle video using the comparative method 1. Figure 2(g) is the reconstruction result Figure 2(h) is the reconstruction result picture of the fourth frame of traffic vehicle video reconstructed by comparison method 1, Figure 2(i ) is the reconstruction result diagram of the first frame of the traffic vehicle video reconstructed by the comparison method 2, and Fig. 2(j) is the reconstruction result diagram of the second frame of the traffic vehicle video reconstruction by the comparison method 2, Fig. 2 (k) is the reconstruction result diagram of the third frame of the traffic vehicle video reconstructed by the comparison method 2, and Fig. 2(l) is the reconstruction result diagram of the fourth frame of the traffic vehicle video reconstruction by the comparison method 2, Figure 2(m) is the reconstruction result of the reconstruction of the first frame of the traffic vehicle video using the comparison method 3, and Figure 2(n) is the reconstruction result of the reconstruction of the second frame of the traffic vehicle video using the comparison method 3 Figure 2(o) is the reconstruction result of the reconstruction of the third frame of the traffic vehicle video using the comparison method 3, and Figure 2(p) is the reconstruction of the fourth frame of the traffic vehicle video using the comparison method 3 Fig. 2 (q) is the reconstructed result map of the first frame of traffic vehicle video with the method of the present invention, and Fig. 2 (r) is the reconstruction of the second frame of traffic vehicle video with the method of the present invention Fig. 2 (s) is a reconstruction result diagram of reconstructing the 3rd frame of the traffic vehicle video with the method of the present invention, and Fig. 2 (t) is a reconstruction result map of the 4th frame of the traffic vehicle video with the method of the present invention Refactored refactored result graph.

It can be seen from the reconstructed image that the noise near the edge of the reconstructed image of the present invention is obviously less, and the visual effect of the reconstructed image is better than the other three comparison algorithms.

The first eight frames of the traffic vehicle picture are reconstructed, and the peak signal-to-noise ratio (PSNR) values of the present invention and the three comparison methods are shown in Table 1.

It can be seen from Table 1 that the PSNR value of the reconstructed video in the present invention is higher than that of the three comparison methods, indicating that the quality of the reconstructed video is good.

Table 1 Reconstruction results of the first eight frames of traffic vehicle pictures using different methods

Fig. 3 is the broken line graph of the peak signal-to-noise ratio (PSNR) value that reconstructs with three kinds of comparison methods and the method of the present invention to 96 frame pictures, abscissa among Fig. 3 represents the video frame of traffic vehicle, and ordinate represents peak signal-to-noise ratio PSNR (dB) value, where the dashed line with an asterisk marks the polyline of the PSNR value of the video reconstructed by the comparison method 1, and the PSNR value of the video reconstructed by the comparison method 2 is marked by a solid line with an asterisk The polyline of the PSNR value reconstructed by the comparison method 3 is marked by a dotted line with a circle, and the polyline of the PSNR value of the video reconstructed by the present invention is marked by a solid line with a circle.

It can be seen from FIG. 3 that the PSNR value of the reconstruction result map of each frame obtained by the method of the present invention is obviously higher than that of other methods.

To sum up, the present invention can obtain clear reconstructed video well, and compared with other existing reconstruction methods, the present invention improves the reconstruction quality of video reconstruction.

Claims (4)

1. A compressed sensing video reconstruction method based on a low-rank model comprises the following steps:

(1) receiving measurement data:

(1a) a compressed sensing sender observes video data, and a result of one-time observation of every H frames of video data by using a random mask observation matrix is formed to form a frame of measurement data, and the measurement data and the random mask observation matrix are sent, wherein H represents a positive integer with the value range of 1 to 20;

(1b) the receiver receives the measurement data and the random mask observation matrix sent by the sender;

(2) initializing a single-frame covariance matrix set:

(2a) generating 18 artificial black-and-white images, wherein the size of each artificial black-and-white image is 65 multiplied by 65 pixels, and each artificial black-and-white image represents one direction;

(2b) adopting 8 × 8 pixels in window size, and respectively sliding a window on the artificial black-and-white image in each direction to select all blocks with the size of 8 × 8 pixels in a step length of 1 pixel to obtain a direction block set in each direction;

(2c) respectively carrying out Principal Component Analysis (PCA) decomposition on the direction block set of each direction to obtain Principal Component Analysis (PCA) orthogonal bases and eigenvalue matrixes, and reserving the first 8 maximum eigenvalues and corresponding principal component orthogonal bases in each direction to obtain corresponding eigenvalue matrixes and direction bases;

(2d) calculating a single-frame covariance matrix in the direction represented by each artificial black-and-white image to obtain a single-frame covariance matrix set;

(3) constructing an initial reconstruction video based on joint sparse and Gaussian distributed piecewise linear estimation:

(3a) for the direction represented by each artificial black-and-white image, putting H single-frame covariance matrixes in the direction on a diagonal line of the matrixes, and constructing a joint sparse video covariance matrix for reconstructing three-dimensional video data in the direction represented by each artificial black-and-white image, wherein the video covariance matrix in the k direction is as follows:

wherein,video covariance matrix, P, representing the k-th direction_kA single-frame covariance matrix representing a k-th direction, wherein k represents a direction number represented by the artificial black-and-white image, and k is 1, 2., 18;

(3b) initializing M × N × H-dimensional reconstructed video as a zero matrix, dividing each frame of the initial reconstructed video into N × N-dimensional S image blocks by a step size of p, andreserving the position of the block, and forming the image blocks at the same position of each frame into a video block to obtain a video block set { x }₁,...,x_l,...,x_SAnd (c) the step of (c) in which,x_lwhich represents the l-th video block,representing the first image block with size of n × n of the T-th frame of the reconstructed video, and T representing the transposition operation, for the first video block in the direction k represented by the artificial black-and-white imageObey a mean of 0 and a covariance matrix ofThe distribution of the gaussian component of (a) is,a video covariance matrix representing a k-th direction, M, N, H respectively represents the sizes of a first dimension, a second dimension and a third dimension in a reconstructed video, p and n respectively represent positive integers less than or equal to the minimum value in the M, N dimension, k represents a direction number represented by an artificial black-and-white image, k is 1,2,.., 18, l is 1,2,.., S represents the number of video blocks, i.e., the number of image blocks divided per frame;

(3c) the estimated value of the video block in the direction represented in the artificial black-and-white image is calculated according to the following formula:

<math> <mrow> <msubsup> <mi>x</mi> <mi>l</mi> <mi>k</mi> </msubsup> <mo>=</mo> <msub> <mover> <mi>P</mi> <mo>&OverBar;</mo> </mover> <mi>k</mi> </msub> <msup> <msub> <mi>&Phi;</mi> <mi>l</mi> </msub> <mi>T</mi> </msup> <msup> <mrow> <mo>(</mo> <msub> <mi>&Phi;</mi> <mi>l</mi> </msub> <msub> <mover> <mi>P</mi> <mo>&OverBar;</mo> </mover> <mi>k</mi> </msub> <msup> <msub> <mi>&Phi;</mi> <mi>l</mi> </msub> <mi>T</mi> </msup> <mo>+</mo> <msup> <mi>&sigma;</mi> <mn>2</mn> </msup> <msub> <mi>I</mi> <mi>d</mi> </msub> <mo>)</mo> </mrow> <mrow> <mo>-</mo> <mn>1</mn> </mrow> </msup> <msub> <mi>y</mi> <mi>l</mi> </msub> </mrow> </math>

wherein,representing the estimate of the l-th video block in the direction k represented by the artificial black-and-white picture,represents the kth video covariance matrix, Φ_lAn observation matrix, y, representing the l-th video block taken from the random mask observation matrix_lRepresenting the measurement data of the l-th video block taken from the measurement data, sigma is taken to be in the range of 0 to 1, I_dRepresents a d-dimensional identity matrix, T represents a transpose operation, ·^-1The matrix inversion is represented, k represents a direction number represented by an artificial black-and-white image, k is 1,2,.., 18, l represents a number of video blocks, and l is 1, 2.., S represents the number of video blocks, i.e., the number of image blocks divided per frame;

(3d) calculating the optimal direction of the direction represented by the artificial black-and-white image according to the following formula:

<math> <mrow> <msub> <mover> <mi>k</mi> <mo>~</mo> </mover> <mi>l</mi> </msub> <mo>=</mo> <mi>arg</mi> <mi> </mi> <munder> <mrow> <mi>m</mi> <mi>i</mi> <mi>n</mi> </mrow> <mi>k</mi> </munder> <mrow> <mo>(</mo> <mo>|</mo> <mo>|</mo> <msub> <mi>&Phi;</mi> <mi>l</mi> </msub> <msubsup> <mi>x</mi> <mi>l</mi> <mi>k</mi> </msubsup> <mo>-</mo> <msub> <mi>y</mi> <mi>l</mi> </msub> <mo>|</mo> <msup> <mo>|</mo> <mn>2</mn> </msup> <mo>+</mo> <msup> <mi>&sigma;</mi> <mn>2</mn> </msup> <msup> <mrow> <mo>(</mo> <msubsup> <mi>x</mi> <mi>l</mi> <mi>k</mi> </msubsup> <mo>)</mo> </mrow> <mi>T</mi> </msup> <msubsup> <mover> <mi>P</mi> <mo>&OverBar;</mo> </mover> <mi>k</mi> <mrow> <mo>-</mo> <mn>1</mn> </mrow> </msubsup> <mo>(</mo> <msubsup> <mi>x</mi> <mi>l</mi> <mi>k</mi> </msubsup> <mo>)</mo> </mrow> <mo>+</mo> <msup> <mi>&sigma;</mi> <mn>2</mn> </msup> <mi>l</mi> <mi>o</mi> <mi>g</mi> <mo>|</mo> <msub> <mover> <mi>P</mi> <mo>&OverBar;</mo> </mover> <mi>k</mi> </msub> <mo>|</mo> <mo>)</mo> </mrow> </math>

wherein,represents the optimal direction of the ith video block in the direction represented by the artificial black-and-white picture,denotes the value of k, phi, when returning the objective function value to the minimum_lAn observation matrix representing the ith video block taken from the random mask observation matrix,represents the estimated value of the l-th video block in the direction k represented by the artificial black-and-white picture, y_lMeasurement data representing the ith video block taken from the measurement data,representing the kth video covariance matrix, the value range of sigma is 0 to 1, | · |. the luminance of the calculation is zero²Represents the square of the norm, | - | represents the value of the determinant, T represents the transpose operation, ·^-1The matrix inversion is represented, k represents a direction number represented by an artificial black-and-white image, k is 1,2,.., 18, l represents a number of video blocks, and l is 1, 2.., S represents the number of video blocks, i.e., the number of image blocks divided per frame;

(3e) combining the estimated value of each video block in the optimal direction into an initial reconstructed video according to the positions of the blocks reserved when the video blocks are divided in the step (3 b);

(3f) setting iteration times s and a maximum external iteration time U, setting the current iteration time s to be 1, and taking an initial reconstructed video as a reconstructed video of a first iteration, wherein U represents a positive integer;

(4) searching for similar blocks of each image block using correlation between video frames and intra frames:

(4a) dividing each frame of the reconstructed video into n multiplied by n dimensional blocks according to the size of step length p, and forming the blocks of all the frames into a two-dimensional image block set G¹Wherein p and n represent positive integers less than or equal to the minimum value in the M, N dimensions, and M, N represents the sizes of the first dimension and the second dimension of the reconstructed video;

(4b) dividing each frame of the reconstructed video into n x n dimensional blocks with step length of 1, and forming the blocks of all the frames into a two-dimensional image block set G²And recording the video block in G²Wherein n represents a positive integer less than or equal to the smallest value among the M, N dimensions, and M, N represents the sizes of the first and second dimensions of the reconstructed video;

(4c) for G¹From G to G²Take out all the Z XZ XH windows around the blockInner block, denoted as the block's neighbor, where Z represents the size of the window's first and second dimensions, H represents the size of the window's third dimension, G¹Representing a set of two-dimensional image blocks divided into blocks by the size of a step p, G²Representing a set of two-dimensional image blocks partitioned into blocks with a step size of 1;

(4d) computing a set of two-dimensional image blocks G¹The Euclidean distance between each block and its adjacent blocks is sorted from small to large according to the Euclidean distance, the first Q blocks are selected as the similar blocks of the corresponding blocks, and the similar blocks of each block are recorded in G²Wherein Q represents a positive integer less than half the number of neighbor blocks;

(5) and solving the low-rank structure of the video according to the following formula:

<math> <mrow> <msup> <msub> <mi>L</mi> <msub> <mi>l</mi> <mi>t</mi> </msub> </msub> <mrow> <mi>s</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> <mo>=</mo> <munder> <mrow> <mi>arg</mi> <mi> </mi> <mi>m</mi> <mi>i</mi> <mi>n</mi> </mrow> <msub> <mi>L</mi> <msub> <mi>l</mi> <mi>t</mi> </msub> </msub> </munder> <mo>|</mo> <mo>|</mo> <msub> <mover> <mi>R</mi> <mo>~</mo> </mover> <msub> <mi>l</mi> <mi>t</mi> </msub> </msub> <mrow> <mo>(</mo> <msup> <mi>X</mi> <mi>s</mi> </msup> <mo>)</mo> </mrow> <mo>-</mo> <msub> <mi>L</mi> <msub> <mi>l</mi> <mi>t</mi> </msub> </msub> <mo>|</mo> <msubsup> <mo>|</mo> <mi>F</mi> <mn>2</mn> </msubsup> <mo>+</mo> <mi>&lambda;</mi> <mo>|</mo> <mo>|</mo> <msub> <mi>L</mi> <msub> <mi>l</mi> <mi>t</mi> </msub> </msub> <mo>|</mo> <msub> <mo>|</mo> <mo>*</mo> </msub> </mrow> </math>

wherein,a low rank structure representing the ith image block of the t frame for the (s + 1) th iteration,representing a low-rank structure of an image block when the value of an objective function is minimalThe value of (a) is,representing an extraction transformation, X, of all similar blocks from the ith image block of the t frame^sRepresenting the reconstructed video for the s-th iteration, ${\tilde{R}}_{l_t}\left(X^s\right)=\left(R_{l_t^1}\right(X^s),...,R_{l_t^q}(X^s),...R_{l_t^Q}(X^s\left)\right),$ representing the extraction of the s-th iterative reconstructed video X^sThe extracted transforms of all similar blocks of the ith image block of the tth frame,an extraction matrix representing the q-th similar block for extracting the l-th image block of the t-th frame,representing reconstructed video X from the s-th iteration of extraction^sThe extraction matrix of the q-th similar block of the ith image block of the t frame is 0.75,represents the squaring operation of Frobenius norm, | | ·| luminance_*Representing a kernel norm operation, H representing the size of a third dimension of the reconstructed video, t representing the number of a video frame, i.e., the number of image blocks of the t-th frame, t being 1,2,.., H, l representing the number of video blocks, l being 1,2,.., S representing the number of video blocks, i.e., the number of divided image blocks per frame, Q being 1,2,.., Q representing the number of similar blocks;

(6) the reconstructed video is updated as follows:

<math> <mrow> <msup> <mi>X</mi> <mrow> <mi>s</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> <mo>=</mo> <munder> <mrow> <mi>arg</mi> <mi> </mi> <mi>m</mi> <mi>i</mi> <mi>n</mi> </mrow> <mi>X</mi> </munder> <mo>|</mo> <mo>|</mo> <mi>y</mi> <mo>-</mo> <mi>&Phi;</mi> <mi>X</mi> <mo>|</mo> <msubsup> <mo>|</mo> <mn>2</mn> <mn>2</mn> </msubsup> <mo>+</mo> <mi>&eta;</mi> <munder> <mo>&Sigma;</mo> <mi>t</mi> </munder> <munder> <mo>&Sigma;</mo> <mi>l</mi> </munder> <mo>|</mo> <mo>|</mo> <msub> <mover> <mi>R</mi> <mo>~</mo> </mover> <msub> <mi>l</mi> <mi>t</mi> </msub> </msub> <mrow> <mo>(</mo> <mi>X</mi> <mo>)</mo> </mrow> <mo>-</mo> <msubsup> <mi>L</mi> <msub> <mi>l</mi> <mi>t</mi> </msub> <mrow> <mi>s</mi> <mo>+</mo> <mn>1</mn> </mrow> </msubsup> <mo>|</mo> <msubsup> <mo>|</mo> <mi>F</mi> <mn>2</mn> </msubsup> </mrow> </math>

wherein, X^s+1Representing the reconstructed video for the s +1 th iteration,representing taking the value of X for the reconstructed video when the value of the objective function is minimal, y represents the one-dimensional vector drawn from the measurement data, Φ represents the video observation matrix generated from the random mask observation matrix,representing the extraction transform of all similar blocks of the ith image block of the tth frame, X representing the reconstructed video, ${\tilde{R}}_{l_t}\left(X\right)=\left(R_{l_t^1}\right(X),...,R_{l_t^q}(X),...R_{l_t^Q}(X\left)\right),$ representing an extraction transformation to extract all similar blocks of the ith image block of the t frame of reconstructed video X,an extraction matrix representing the q-th similar block for extracting the l-th image block of the t-th frame,an extraction matrix representing the extraction of the q-th similar block of the i-th image block of the t-th frame of the reconstructed video X,the low rank structure of the ith image block of the t frame of the (s + 1) th iteration is represented, the value of eta is 1, sigma represents the summation operation,which means that a 2-norm squaring operation is performed,expressing a squaring operation of a Frobenius norm, H expressing the size of a third dimension of a reconstructed video, t expressing the number of a video frame, namely the number of image blocks of the t-th frame, t being 1,2, wherein H, l expressing the number of the video blocks, l being 1,2, wherein S, S expressing the number of the video blocks, namely the number of image blocks divided by each frame, and Q being 1,2, wherein Q expresses the number of similar blocks;

(7) judging whether the current iteration times are larger than the maximum external iteration times, if so, executing the step (8), otherwise, adding 1 to the current iteration times, and executing the step (4);

(8) and outputting the reconstructed video.

2. The low-rank model-based compressed perceptual video reconstruction method of claim 1, wherein: and (3) in the step (2a), the boundary of the black area and the white area of each artificial black-white image of the artificial black-white images passes through the center coordinates (33,33), and the angles of the 18 artificial black-white image boundaries are uniformly sampled from 0 to 180 degrees.

3. The low-rank model-based compressed perceptual video reconstruction method of claim 1, wherein: the principal component analysis PCA decomposition steps in step (2c) are as follows:

step 1, selecting one direction from all directions of the artificial black-and-white image, and solving a covariance matrix of a direction block set in the selected direction according to the following formula:

<math> <mrow> <mi>P</mi> <mo>=</mo> <mi>E</mi> <mrow> <mo>&lsqb;</mo> <mrow> <msub> <mi>f</mi> <mi>i</mi> </msub> <msup> <msub> <mi>f</mi> <mi>i</mi> </msub> <mi>T</mi> </msup> </mrow> <mo>&rsqb;</mo> </mrow> </mrow> </math>

where P represents the covariance matrix of the set of directional blocks for the selected direction, E represents the mathematical expectation, f_iAn ith block in the direction block set representing the selected direction, T representing a transpose operation;

and step 2, diagonalizing the covariance matrix according to the following formula to obtain a Principal Component Analysis (PCA) orthogonal basis and an eigenvalue matrix:

P＝BDB^T

wherein, P represents the covariance matrix of the direction block set of the selected direction, B represents the principal component analysis PCA orthogonal basis of the selected direction, D represents the eigenvalue matrix of the direction, and T represents the transposition operation.

4. The low-rank model-based compressed perceptual video reconstruction method of claim 1, wherein: the formula for calculating the single-frame covariance matrix in the direction represented by each artificial black-and-white image in step (2d) is as follows:

P_k＝B_kD_kB_k ^T

wherein, P_kRepresents the covariance matrix of the single frame in the k-th direction, B_kDenotes a direction base in the k-th direction, D_kAnd c, representing a characteristic value matrix in the k-th direction, T representing a transposition operation, k representing a direction number represented by the artificial black-and-white image, and k being 1, 2.