CN108881911B - Foreground and background recovery method for compressed sensing rear video data stream - Google Patents

Foreground and background recovery method for compressed sensing rear video data stream Download PDF

Info

Publication number
CN108881911B
CN108881911B CN201810667783.7A CN201810667783A CN108881911B CN 108881911 B CN108881911 B CN 108881911B CN 201810667783 A CN201810667783 A CN 201810667783A CN 108881911 B CN108881911 B CN 108881911B
Authority
CN
China
Prior art keywords
matrix
estimator
estimation
low
foreground
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201810667783.7A
Other languages
Chinese (zh)
Other versions
CN108881911A (en
Inventor
袁晓军
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sichuan Chuangshu Intelligent Technology Co ltd
Original Assignee
University of Electronic Science and Technology of China
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by University of Electronic Science and Technology of China filed Critical University of Electronic Science and Technology of China
Priority to CN201810667783.7A priority Critical patent/CN108881911B/en
Publication of CN108881911A publication Critical patent/CN108881911A/en
Application granted granted Critical
Publication of CN108881911B publication Critical patent/CN108881911B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

The invention belongs to the technical field of video information processing and image processing, and particularly relates to a foreground and background recovery method for compressed sensing of a rear video data stream. The method adopts an iterative recovery mode, adopts a linear estimator to simultaneously estimate a low-rank matrix and a sparse matrix to respectively obtain the estimation values and the estimation errors of the two matrices, further obtains the estimation value of the sparse matrix and the estimation value of the low-rank matrix by respectively adopting the sparse matrix estimator and the low-rank matrix estimator, then feeds back the obtained estimation values to the linear estimator, and iteratively estimates until the output converges, thereby respectively obtaining the recovered foreground and the recovered background. The method has the beneficial effect of solving the problem of separation of the foreground and the background of the video data stream obtained after the compressed sensing.

Description

Foreground and background recovery method for compressed sensing rear video data stream
Technical Field
The invention belongs to the technical field of video information processing and image processing, and particularly relates to a foreground and background recovery method for compressed sensing of a rear video data stream.
Background
Industrial surveillance cameras, cameras for mobile devices, motion cameras, and the like are producing large amounts of image and video data each day. Due to considerations such as device cost, mobility requirements, size limitations, battery capacity, etc., these image and video sensing devices need to compress and store or transmit data after acquiring image and video data. This approach of sensing first and compressing data requires energy consumption in both sensing and compressing data steps, while occupying more storage space. The proposed technique of compressed sensing solves this problem. The compressed sensing utilizes the sparse structure property of the measurement signal, and the combination of the sensing and the compression of the information is achieved. The signals obtained by the compressive sensing technology need to be decompressed by some existing compressive sensing algorithms to restore the original signals.
The proposed compressed sensing technology has not been applicable on a large scale due to several problems: 1. most of actually measured signals are non-sparse, 2, the measured signals are easily interfered by burst noise, and 3, the popularization of compressed sensing sensor hardware is realized. The solution of the last problem requires the development of the hardware industry, while the first two problems can be solved by better algorithmic and technical methods.
In addition to the sparsity concept introduced at the beginning by the compressed sensing technology, the low rank property is also an important data property. Many signals in nature can be modeled with low rank models, such as video streams. Since there is a great correlation between each frame of video image, the video stream can be modeled as low-rank matrix data. The difference between foreground and background results from the difference between video frames. The variation of foreground objects makes low rank modeling of the entire data inaccurate. Since the variation of the foreground is generally more concentrated and small relative to the amount of background data, it can be modeled as a sparse signal. Based on such an idea, robust principal component analysis is proposed and applied to the separation of low rank and sparse signals.
Due to the significant correlation and difference between each frame of image of the video data, the video stream data can be modeled as the superimposed data of the address matrix and the sparse matrix. Based on the method, for uncompressed video stream data, the foreground and the background of the video are effectively recovered based on a robust principal component analysis method. But for sensors based on compressed sensing technology, this approach loses its effectiveness due to the compressed measurement.
Disclosure of Invention
The present invention is directed to the above problem, and provides a method for solving the problem of separating foreground and background of a video data stream obtained after compressed sensing.
The technical scheme adopted by the invention is as follows:
a foreground and background restoration method for a compressed perceptual post-video data stream, comprising the steps of:
s1, initialization:
uncompressed video stream data is modeled as a mixture of low rank and sparse matrices, namely:
X＝L+S
wherein L represents a low-rank matrix, i.e. the background in the video stream, S represents a sparse matrix, i.e. the foreground image changed for each frame, and X is the complete video stream data;
after the compression operation, the data observed by the sensor is y ═ F (x) + n, where F is the compression operation, n is the measurement noise, obeys a mean of 0, and has a variance σ2(ii) a gaussian distribution of;
method for recovering front and back backgrounds based on information transmission method
S2, estimating the low-rank matrix and the sparse matrix simultaneously by using the linear estimator A, and respectively obtaining the estimation values and the estimation errors of the two matrices, wherein the estimation values and the estimation errors are specifically as follows:
the estimated values for the sparse matrix are obtained as:
Figure GDA0002486141220000021
wherein the content of the first and second substances,
Figure GDA0002486141220000022
is the linear estimator estimate of S, M is the dimension of the output after operation of the y operator a, N is the product of the number of rows and columns of the matrix X,
Figure GDA0002486141220000023
is an estimate of L by the linear estimator;
the estimation error of the sparse matrix is:
Figure GDA0002486141220000024
wherein the content of the first and second substances,
Figure GDA0002486141220000031
is the estimation error of S input to the linear estimator a,
Figure GDA0002486141220000032
is the estimate error of L input to the linear estimator a;
the estimated values for the low rank matrix are obtained as:
Figure GDA0002486141220000033
the estimation error of the low rank matrix is:
Figure GDA0002486141220000034
wherein the content of the first and second substances,
Figure GDA0002486141220000035
s3, according to the result of the step S2, a sparse matrix estimator is adopted to further obtain an estimated value of the sparse matrix (different sparse matrix estimators may be selected, such as a soft threshold denoiser, a SURE-L ET estimator, etc.):
Figure GDA0002486141220000036
wherein c isBAnd αBIs a linear combination coefficient used to make the correlation of the input and output estimation errors 0 while minimizing the output estimation error of the module:
Figure GDA0002486141220000037
wherein the content of the first and second substances,<A,B>＝tr(ATB)，
Figure GDA0002486141220000038
DSis the estimation operation of the sparse matrix estimator on S;
and further obtaining an estimated value of the low-rank matrix by using a low-rank matrix estimator (the low-rank matrix estimator which can be selected comprises an optimal rank r estimator, a soft threshold low-rank matrix estimator, a hard threshold estimator and the like):
Figure GDA0002486141220000039
wherein the content of the first and second substances,
Figure GDA00024861412200000310
for the low rank matrix estimator C to estimate L,
Figure GDA00024861412200000311
Figure GDA0002486141220000041
DLis the estimation operation of the low rank matrix estimator pair L;
and S4, feeding the result of the step S3 back to the step S2, and performing iterative estimation until the output converges to respectively obtain the restored foreground and background.
The estimation error output in step S3 can be estimated by different estimation methods, and the specific estimation method may be different according to different usage scenarios. Meanwhile, in order to ensure the robustness of the method, the estimator involved in step S2 may be damming, i.e. the variation speed is slowed down, so that the whole method is more robust and reliable
The method has the beneficial effect of solving the problem of separation of the foreground and the background of the video data stream obtained after the compressed sensing.
Drawings
FIG. 1 is a block diagram of a context separation method of the present invention;
FIG. 2 is a block diagram of two frames of images extracted from a video segment, the picture pixel size being 240px × 320 px;
FIG. 3 is a video background extracted using the method of the present invention corresponding to the two frames of images in FIG. 2;
fig. 4 is a video foreground extracted by the method of the present invention corresponding to the two frames of images in fig. 2.
Detailed Description
The method of the present invention will be further described with reference to the accompanying drawings and examples.
Examples
Using a segment of surveillance video from a laboratory surveillance video for example, the video includes 500 frames of images, each frame of image has a size of 240 × 320 pixels, the whole video is compressed first, the compression uses a randomly selected discrete cosine transform operation + a random positive and negative phase operation, the compression ratio of the video is 15%.
Firstly, initializing a matrix of a front background and a back background to be expressed as an all-0 matrix with corresponding sizes, and simultaneously initializing an estimation error to be
Figure GDA0002486141220000042
The whole iterative recovery process is carried out with reference to fig. 1, the estimated matrix is transmitted from the module a to the module B and the module C, and then the input of the module B and the module C is transmitted to the module a, and the whole process is repeated until the estimated matrix is converged.
The SURE-L ET estimator is used in a module B to estimate a sparse matrix, and the optimal rank r estimator is used in a module C to estimate a low rank matrix, the final recovery effect is shown in figures 3 and 4, the recovered video background image is shown in figure 3, and the corresponding foreground image is shown in figure 4.

Claims (1)

1. A foreground and background restoration method for a compressed perceptual post-video data stream, comprising the steps of:
s1, initialization:
for uncompressed perceptual video stream data, it is modeled as a mixture of low rank matrices and sparse matrices, i.e.:
X＝L+S
wherein L represents a low-rank matrix, i.e. the background in the video stream, S represents a sparse matrix, i.e. the foreground image changed for each frame, and X is the complete video stream data;
after the compressive sensing operation, the data observed by the sensor is y ═ F (x) + n, where F is the compressive sensing operator, n is the measurement noise, obeys a mean of 0, and has a variance of σ2(ii) a gaussian distribution of;
s2, estimating the low-rank matrix and the sparse matrix simultaneously by using the linear estimator A, and respectively obtaining the estimation values and the estimation errors of the two matrices, wherein the estimation values and the estimation errors are specifically as follows:
the estimated values for the sparse matrix are obtained as:
Figure FDA0002486141210000011
wherein the content of the first and second substances,
Figure FDA0002486141210000012
is an estimate of S by the linear estimator, M is the dimension of y, N is the product of the number of rows and columns of the matrix X,
Figure FDA0002486141210000013
is an estimate of L by the linear estimator;
the estimation error of the sparse matrix is:
Figure FDA0002486141210000014
wherein the content of the first and second substances,
Figure FDA0002486141210000015
is the estimation error of S input to the linear estimator a,
Figure FDA0002486141210000016
Figure FDA0002486141210000017
is the estimate error of L input to the linear estimator a;
the estimated values for the low rank matrix are obtained as:
Figure FDA0002486141210000018
the estimation error of the low rank matrix is:
Figure FDA0002486141210000019
wherein the content of the first and second substances,
Figure FDA0002486141210000021
s3, according to the result of the step S2, a sparse matrix estimator B is adopted to further obtain a sparse matrix estimation value:
Figure FDA0002486141210000022
wherein c isBAnd αBIs a linear combination coefficient of the two or more,
Figure FDA0002486141210000023
for the sparse matrix estimator estimation of S:
Figure FDA0002486141210000024
wherein the content of the first and second substances,<A,B>＝tr(ATB)，
Figure FDA0002486141210000025
and further obtaining a low-rank matrix estimation value by adopting a low-rank matrix estimator C:
Figure FDA0002486141210000026
wherein the content of the first and second substances,
Figure FDA0002486141210000027
for the low rank matrix estimator C to estimate L,
Figure FDA0002486141210000028
Figure FDA0002486141210000029
and S4, feeding the result of the step S3 back to the linear estimator of the step S2, and performing iterative estimation until the output converges to respectively obtain the restored foreground and background.
CN201810667783.7A 2018-06-26 2018-06-26 Foreground and background recovery method for compressed sensing rear video data stream Active CN108881911B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810667783.7A CN108881911B (en) 2018-06-26 2018-06-26 Foreground and background recovery method for compressed sensing rear video data stream

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810667783.7A CN108881911B (en) 2018-06-26 2018-06-26 Foreground and background recovery method for compressed sensing rear video data stream

Publications (2)

Publication Number Publication Date
CN108881911A CN108881911A (en) 2018-11-23
CN108881911B true CN108881911B (en) 2020-07-10

Family

ID=64295673

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810667783.7A Active CN108881911B (en) 2018-06-26 2018-06-26 Foreground and background recovery method for compressed sensing rear video data stream

Country Status (1)

Country Link
CN (1) CN108881911B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113382247B (en) * 2021-06-09 2022-10-18 西安电子科技大学 Video compression sensing system and method based on interval observation, equipment and storage medium

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105160664A (en) * 2015-08-24 2015-12-16 西安电子科技大学 Low-rank model based compressed sensing video reconstruction method
US9224210B2 (en) * 2013-02-06 2015-12-29 University Of Virginia Patent Foundation Systems and methods for accelerated dynamic magnetic resonance imaging

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8433148B2 (en) * 2011-03-31 2013-04-30 Mitsubishi Electric Research Laboratories, Inc. Method for compressing textured images

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9224210B2 (en) * 2013-02-06 2015-12-29 University Of Virginia Patent Foundation Systems and methods for accelerated dynamic magnetic resonance imaging
CN105160664A (en) * 2015-08-24 2015-12-16 西安电子科技大学 Low-rank model based compressed sensing video reconstruction method

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
On the Performance of Turbo Signal Recovery with Partial DFT Sensing Matrices;Junjie Ma, Xiaojun Yuan，Li Ping;《IEEE SIGNAL PROCESSING LETTERS》;20151010;全文 *
Turbo Compressed Sensing with Partial DFT Sensing Matrix;Junjie Ma, Xiaojun Yuan，Li Ping;《IEEE SIGNAL PROCESSING LETTERS》;20150228;第22卷(第2期);全文 *

Also Published As

Publication number Publication date
CN108881911A (en) 2018-11-23

Similar Documents

Publication Publication Date Title
Ji et al. Robust video restoration by joint sparse and low rank matrix approximation
Chatterjee et al. Efficient and robust large-scale rotation averaging
US9736455B2 (en) Method and apparatus for downscaling depth data for view plus depth data compression
Dong et al. Wavelet frame based blind image inpainting
Wen et al. Joint adaptive sparsity and low-rankness on the fly: An online tensor reconstruction scheme for video denoising
US9654752B2 (en) Method and apparatus for compressive acquisition and recovery of dynamic imagery
CN105451019B (en) A kind of image compression transmission method towards wireless video sensor network
US10412462B2 (en) Video frame rate conversion using streamed metadata
CN104284059A (en) Apparatus and method for stabilizing image
JP2008529414A5 (en)
JP2008521347A5 (en)
CN102148986B (en) Method for encoding progressive image based on adaptive block compressed sensing
Chen et al. Robust Kronecker product video denoising based on fractional-order total variation model
CN108881911B (en) Foreground and background recovery method for compressed sensing rear video data stream
CN106559670A (en) A kind of improved piecemeal video compress perception algorithm
US8478062B2 (en) Reducing signal-dependent noise in digital cameras
Ji et al. Wavelet frame based image restoration with missing/damaged pixels
CN111279392A (en) Cluster density calculation device, cluster density calculation method, and cluster density calculation program
CN116708807A (en) Compression reconstruction method and compression reconstruction device for monitoring video
Sankaran et al. Non local image restoration using iterative method
CN105574831A (en) Image denoising method through corrected image block expected logarithmic likelihood estimation
Choudhury et al. Low bit-rate compression of video and light-field data using coded snapshots and learned dictionaries
CN109214975A (en) A kind of two-dimentional gradually orthogonal matching pursuit method restored based on two-dimentional sparse signal
Pathak et al. Feature enhancing image inpainting through adaptive variation of sparse coefficients
CN114612305A (en) Event-driven video super-resolution method based on stereogram modeling

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right

Effective date of registration: 20211129

Address after: 628000 6th floor, Liangjiang Jiayuan office building, west side of Zhengzhou road, Nanhe office, Lizhou District, Guangyuan City, Sichuan Province

Patentee after: Sichuan Shannan Technology Co.,Ltd.

Address before: 611731, No. 2006, West Avenue, hi tech West District, Sichuan, Chengdu

Patentee before: University of Electronic Science and Technology of China

TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20211228

Address after: No. 708, 709, 710, 7th floor, building 1, No. 7, Jiuxing Avenue, high tech Zone, Chengdu, Sichuan 610000

Patentee after: Sichuan ChuangShu Intelligent Technology Co.,Ltd.

Address before: 628000 6th floor, Liangjiang Jiayuan office building, west side of Zhengzhou road, Nanhe office, Lizhou District, Guangyuan City, Sichuan Province

Patentee before: Sichuan Shannan Technology Co.,Ltd.

TR01 Transfer of patent right