CN112884644A - Multi-scale coding aperture spectrum time compressed sensing imaging method - Google Patents

Multi-scale coding aperture spectrum time compressed sensing imaging method Download PDF

Info

Publication number
CN112884644A
CN112884644A CN202110037205.7A CN202110037205A CN112884644A CN 112884644 A CN112884644 A CN 112884644A CN 202110037205 A CN202110037205 A CN 202110037205A CN 112884644 A CN112884644 A CN 112884644A
Authority
CN
China
Prior art keywords
image
image sequence
frame
matrix
multispectral
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110037205.7A
Other languages
Chinese (zh)
Other versions
CN112884644B (en
Inventor
张廷华
孙华燕
樊桂花
李迎春
张怀利
李春阳
田磊源
郭惠超
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Peoples Liberation Army Strategic Support Force Aerospace Engineering University
Original Assignee
Peoples Liberation Army Strategic Support Force Aerospace Engineering University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Peoples Liberation Army Strategic Support Force Aerospace Engineering University filed Critical Peoples Liberation Army Strategic Support Force Aerospace Engineering University
Priority to CN202110037205.7A priority Critical patent/CN112884644B/en
Publication of CN112884644A publication Critical patent/CN112884644A/en
Application granted granted Critical
Publication of CN112884644B publication Critical patent/CN112884644B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Color Television Image Signal Generators (AREA)

Abstract

The invention discloses a multi-scale coding aperture spectrum time compression perception imaging method, which utilizes the 0 and 1 control states of a digital micromirror array (DMD) to obtain two paths of aperture coding coded images with complementary coding modes of a moving target scene, wherein one path adopts a time-varying amplitude modulation mode to realize time compression, and a multi-frame sequence image is resolved through a single-frame aliasing image of single exposure integration; the other path adopts time-varying amplitude modulation and a dispersion grating to realize the resolution of the single-frame spectrum aliasing image of single exposure integration into a multi-spectral sequence image; then fusing the time compression reconstruction image sequence and the multispectral image sequence to realize space-time-spectrum combined compression sensing; the method of the invention realizes CACTI imaging to obtain the reference frame of the video sequence according to the large-scale observation matrix, and can realize rapid registration with the CASSI imaging channel.

Description

Multi-scale coding aperture spectrum time compressed sensing imaging method
Technical Field
The invention belongs to the technical field of light field modulation and computational imaging, and particularly relates to a multi-scale coding aperture spectral time compression sensing imaging method.
Background
The method for realizing the spatial target multi-dimensional light field joint acquisition and super-resolution reconstruction by utilizing the light field modulation and compressed sensing principle mainly comprises the following steps: CS-MUVI, CS-CAKEI, CASSI, and CACTI.
The CS-MUVI imaging method is a space-time super-resolution imaging method based on a single-pixel camera, and a dual-scale coding observation matrix is adopted in the method, so that a high-resolution video sequence can be recovered under the conditions that the target movement speed is low and a certain observation frequency is met. The method utilizes the unit detector to realize the acquisition of the array image, greatly reduces the hardware cost of the system, but the image reconstruction resolution depends on the motion estimation precision of a target scene, has long observation time, and is only effective for the observation of a static or low-speed moving target.
The CS-CAKEI imaging method is a space-time super-resolution imaging technology combining coding exposure and a compressed coding aperture imaging technology, realizes time and space super-resolution imaging at the same time, has higher requirements on hardware and a reconstruction algorithm, and needs to strictly calibrate the system.
The CACT imaging method can realize time super-resolution imaging, the hardware realization cost is low, and due to the adoption of a motion coding aperture mode, an optical mask and a motion controller are required to be added to the system, so that the spatial resolution and the imaging contrast of the imaging system are reduced, and the complexity of the system is increased.
The CASSI utilizes a coded aperture and one or more dispersive elements to modulate the light field of a scene, the coded aperture enables a detector to capture multi-channel projections of a data cube corresponding to different spectral images, and the method provides a mechanism for acquiring a three-dimensional data cube through a single two-dimensional measurement. The method can only improve spectral resolution, and requires a complex reconstruction algorithm to invert spectral data from aliased data.
In summary, the following problems mainly exist in the current method for realizing multi-dimensional light field joint acquisition and super-resolution reconstruction based on the compressed sensing principle: firstly, the dimensionality of joint acquisition and compressed sensing is limited, and usually, joint compressed sensing of two dimensionalities, namely time-space, time-space and time-spectrum, can be realized only; secondly, the improvement of multi-dimension combined compressed sensing reconstruction information is limited, and according to the compressed sensing imaging principle, the existing measuring method mainly sacrifices information of one dimension to exchange information of another dimension, so that the simultaneous improvement of multi-dimension resolution is difficult to realize; thirdly, the system is complex in realization structure, and due to the introduction of the light field regulation and control device, the luminous flux and the imaging contrast of the system are generally influenced, so that the function expansion and the application are difficult to carry out.
Disclosure of Invention
In view of the above, the present invention provides a multi-scale coded aperture spectral time compressed sensing imaging method, in which a digital micromirror array (DMD) is used as a multi-scale coded aperture modulator, and a time compressed sensing image sequence and a multi-spectral compressed sensing image sequence are subjected to space-time registration and fusion reconstruction through a common coded aperture, so as to implement compressed sensing and super-resolution reconstruction of time, space and spectrum of a moving scene, and an algorithm can improve the acquisition capability of instantaneous optical characteristic data of a moving target scene, and is suitable for performance improvement of spatial target monitoring and live recording equipment.
A multi-scale coded aperture spectrum time compressed sensing imaging method comprises the following steps:
step 1, randomly extracting N from Hadamard with matrix elements { +1, -1}FLines, forming a matrix from each line, and upsampling to obtain NFN corresponding to the number of DMD units1×M1A coding matrix of dimensions; wherein N isFThe number of images that can be reconstructed for a single exposure image;
step 2, reconstructing the single-exposure multispectral compressed sensing image, which specifically comprises the following steps:
controlling the coding of the DMD by the coding matrix; the DMD receives a light beam of a target scene, and the light beam is divided into two beams, wherein one beam enters the wide spectrum camera after being subjected to grating dispersion; the wide-spectrum camera obtains a spectral image containing visible light and near infrared; the other beam of light enters the visible light camera;
the DMD passes through N in one exposure period of the wide-spectrum cameraFControlling the encoding matrix, wherein light beams form multispectral aliasing images with different wavelengths on the surface of a wide-spectrum camera detector; for the multispectral aliasing image, reconstructing a multispectral image sequence of a primary exposure period by adopting an alternative direction multiplier total variation regularization algorithm; after the wide spectrum camera is exposed for a set number of times, a multispectral image sequence S is obtainedo；
And 3, reconstructing a single-exposure aperture coding time compressed image sequence, specifically:
DMD Via NFControlling the coding matrix, and completing one-time exposure by a visible light camera to obtain an aliased full-color image; obtaining a full-color image sequence by an optimized inversion algorithm by using the aliased full-color image; the visible light camera obtains a full-color image sequence S after exposure for a set number of timesr；
Step 4, finding out a multispectral image sequence S in the same exposure periodoIn the exposure periodo(i)，i∈[1,…,NF]And a full-color image sequence SrIn the exposure periodr(t)，t∈[1,…,NF]Estimating the transfer function between the different wavelength spectral image and the panchromatic image:
Figure RE-GDA0002994040640000021
the image sequence Sr(t) the ith wavelength λ corresponding to the t-th frame imageiThe following subframe images are:
So(λi,t)＝Sr(t)V(λi,t)，i∈[1,…,NF]
is pressed onObtaining a sub-frame image corresponding to each frame of full-color image by the surface calculation method, and obtaining MF×NF×NFA frame multispectral image sequence; mFSetting the exposure times;
step 5, for M obtained in step 4F×NF×NFAnd respectively carrying out super-resolution reconstruction on the frame multispectral image sequence to obtain a super-resolution image sequence.
Further, after step 3 is executed and before step 4 is executed, the multispectral image sequence S is first executedoAnd a full-color image sequence SrThe spatio-temporal registration, i.e. the alignment of the images with respect to the same DMD exposure instant, is performed.
Preferably, the space-time registration method is as follows:
in step 1, arbitrarily decimating N in Hadamard with matrix elements { +1, -1}FForming a matrix by each row to be used as a large-scale coding matrix; the dimension of the matrix being less than N1×M11/K of dimension; k is an integer greater than 1; when step 2 and step 3 are executed, the large-scale coding matrix is also used as a coding matrix of the DMD to control the DMD, and multispectral image sequences under the large-scale coding matrix are respectively obtained
Figure RE-GDA0002994040640000032
And full color image sequence
Figure RE-GDA0002994040640000033
Using a sequence of multispectral images
Figure RE-GDA0002994040640000034
And full color image sequence
Figure RE-GDA0002994040640000035
The spatiotemporal registration is achieved.
Preferably, the spatiotemporal registration is achieved using a posterior probability based approach.
Preferably, the step 5 adopts a maximum posterior probability algorithm for reconstruction, and the specific method is as follows:
Figure RE-GDA0002994040640000031
wherein, I is a full-color image corresponding to the current image to be reconstructed; μ denotes a regular coefficient; j. the design is a squarejFor a current frame image in an image sequence to be reconstructed, JiFor the current frame J in the image sequencejFront and rear adjacent images; d is a down-sampling matrix; k is based on the current frame image JjA fuzzy kernel estimated by an IRLS algorithm; (lambdajT) and V (lambda)iI) is a spectral transfer function; ^ represents the gradient operator; mijRepresents the current frame JjAnd adjacent frame JiFlow of light between; beta is a weight factor;
reconstructing frame by frame to obtain MF×NF×NFSuper-resolution image sequence I of frames*。
Preferably, the down-sampling factor of the down-sampling matrix D is 2.
The invention has the following beneficial effects:
(1) the invention provides a multi-scale coding aperture spectrum time compression perception imaging method, which utilizes the 0 and 1 control states of a digital micromirror array (DMD) to obtain two paths of aperture coding coded images with complementary coding modes of a moving target scene, wherein one path adopts a time-varying amplitude modulation mode to realize time compression, and a multi-frame sequence image is resolved through a single-frame aliasing image of single exposure integration; the other path adopts time-varying amplitude modulation and a dispersion grating to realize the resolution of the single-frame spectrum aliasing image of single exposure integration into a multi-spectral sequence image; and then fusing the time compression reconstruction image sequence and the multispectral image sequence to realize space-time-spectrum combined compression sensing.
(2) The invention provides a method for realizing space-time registration of different compressed sensing systems based on multi-scale coded apertures, which realizes CACT imaging to obtain a reference frame of a video sequence according to a large-scale observation matrix and can realize rapid registration with a CASSI imaging channel.
(3) The invention provides a CACTI and CASSI fusion method, which takes a CACTI image sequence reconstructed by a large-scale observation matrix as a video reference frame, takes a video sequence reconstructed by a small-scale observation matrix as a subframe between reference frames, and obtains a spectral efficiency conversion function between different wavelengths by using a multispectral image obtained by CASSI, and is applied to the subframe, thereby finally obtaining a multispectral and panchromatic image sequence with improved time resolution, and improving the efficiency of a system for obtaining target optical characteristics.
Drawings
FIG. 1 is a block diagram of a multi-scale coded aperture spectral temporal compressed sensing imaging system.
FIG. 2 is a flow chart of a multi-scale coded aperture spectral temporal compression perception fusion algorithm.
Detailed Description
The invention is described in detail below by way of example with reference to the accompanying drawings.
The method has the basic principle that according to the spatial light modulation principle of the DMD, the CACTI imaging system and the CASSI imaging system are combined in a multi-scale coding aperture mode, the space-time registration of two reconstructed image sequences is realized through multi-scale coding, and finally the multi-dimensional light field combined compressed sensing and reconstruction of a target motion scene are realized. An imaging system to which the method is applicable is shown in fig. 1. The key steps of the imaging method are as follows: firstly, a multi-scale observation matrix is utilized to control a DMD to realize aperture coding and light splitting on a motion scene, and the coding modes of two paths of split imaging light beams are complementary to each other by 0-1; the split imaging light beam is subjected to single integral aliasing through a camera, and reconstruction is realized by using a GAP algorithm to obtain an image sequence with improved time resolution; the other path of imaging light beam is subjected to diffraction grating dispersion and then integrated on the sensor surface of the wide spectrum camera to form a spectrum aliasing image, and a multispectral image sequence can be quickly reconstructed by adopting an ADMM algorithm; secondly, by utilizing the complementary characteristics of the coding modes of the two imaging light beams, the space-time registration between the reference frames can be realized, the spectrum conversion function between the full-color image and the multispectral image is solved, and then the multispectral image sequence corresponding to each sub-frame of the CACTI is estimated; and finally, improving the spatial resolution of the image sequence by using a Bayes super-resolution reconstruction algorithm, wherein the basic algorithm flow is shown in figure 2, and the multi-scale coding aperture spectrum time compression sensing imaging method comprises the following specific steps:
step 1, multi-scale coding aperture coding and light splitting based on DMD:
first, according to the number of cells being N1×M1The physical characteristics (on state, flat state and off state) of the DMD coding element are selected from Hadamard with matrix elements of { +1, -1} and matrix dimension of (N)2×M2)×(N2×M2) Wherein
Figure RE-GDA0002994040640000051
K is an integer greater than 0. Extract any row 1X (N) of the Hadamard matrix2×M2) Dimension vector, converted into N2×M2Dimension matrix to obtain large scale coding matrix; carrying out K times of upsampling on the large-scale coding matrix to obtain N consistent with the number of DMD units1×M1A dimension matrix; then, N with the element { +1, -1} is added1×M1The random matrix of dimension and the Hadamard matrix after up-sampling are subjected to Hadamard product, thereby obtaining N1×M1And (5) dimension small-scale coding matrix. Extracting N in total according to the above processFGet N1×M1×NFDimension small scale coding matrix ThAnd N2×M2×NFDimension large scale coding matrix TlIn which N isFThe number of images that can be reconstructed for a single exposure image of the system, i.e. the temporal resolution is raised by a multiple.
The deflection angles of the DMD micro-mirror are +12 degrees, 0 degree and-12 degrees, which respectively correspond to three stable working states of the DMD, namely an on state, a flat state and an off state, wherein the on state and the off state correspond to +12 degrees and-12 degrees, and N is used1×M1When the small-scale coding matrix controls the DMD device, the DMD enables images with the gray values of +1 to-1 to form two imaging light beams deflected by +12 degrees and-12 degrees according to an on state and an off state, and coding and light splitting are completed simultaneously. The DMD changes N according to the small-scale coding matrix within the one-time exposure time of the back-end imaging cameraFNext, the process is carried out.
Step 2, reconstructing a single-exposure multispectral compressed sensing image:
one path of the light split is imagedAfter the light beam is subjected to grating dispersion, a multispectral aliasing image is formed on the surface of the wide-spectrum camera detector. Assuming a target scene spatial energy spectral density at the coded aperture of S0(x, y, λ), the spectral density of energy arriving at the detector plane after the light field passes through the coded aperture is:
I(x,y)＝∫∫S0(x,y,λ)Φ(x,y，λ)dλ (1)
wherein a spectral density filter function Φ (x, y, λ) is defined as T (x- α (λ - λ)c) Y), T (x, y) is the coding mode of the coded aperture, α is the linear dispersion coefficient of the dispersion grating, λcFor the central wavelength of a dispersive grating, λ ═ λc+ Δ/α, Δ is the physical size of the detector pixel at the back end.
The back end detector integrates the changing scene over the exposure time, the energy of the pixel integration at detector (m, n) position is:
Figure RE-GDA0002994040640000052
Figure RE-GDA0002994040640000053
as a function of discrete sampling of the sensor.
The above process is represented in the form of a linear matrix
g＝Φf+n1 (3)
Wherein f is a matrix form of a target motion scene spectral image, and n1Is a matrix form of system imaging noise, phi is a matrix form of a spectral density filter function, and the expression is
Figure RE-GDA0002994040640000061
Figure RE-GDA0002994040640000062
T is controlN of DMD1×M1×NFDimensional small-scale coding matrix, Ti,j,k,1≤i≤N1,1≤j≤M1,1≤k≤NFAre matrix elements. The aliased image g observed by the detector, respectively, encodes the matrix T in large scalelAnd a small-scale coding matrix ThFor priori knowledge, a low-resolution image is reconstructed by adopting an alternating direction multiplier total variation regularization algorithm
Figure RE-GDA0002994040640000063
And high resolution images
Figure RE-GDA0002994040640000064
Camera performing MFObtaining M by secondary shootingFPerforming the above reconstruction process to obtain MF×NFHigh resolution image sequence of frames
Figure RE-GDA0002994040640000065
And low resolution image sequence
Figure RE-GDA0002994040640000066
And 3, reconstructing a single-exposure aperture coding time compressed image sequence:
after light splitting, the other path of imaging light beam forms an aliasing observation image in the motion scene modulated by the DMD in an integration period of the rear-end imaging detection device, the aliasing observation image comprises a multi-frame observation value of the motion scene in the integration period, and a sequence image with a higher frame frequency can be obtained from a single-exposure observation image of the detector through an optimization inversion algorithm. Similar to the step 2 imaging process, the above process is represented in the form of a linear matrix
y＝Hx+n2 (6)
Where x is the matrix form of the target motion scene image, n2For system imaging noise, H is a filter function corresponding to a time-varying code modulation aperture, and the expression is
Figure RE-GDA0002994040640000067
Figure RE-GDA0002994040640000068
Where M and T are complementary coding matrices. The aliased image y observed by the detector, respectively, encodes the matrix M in large scalelAnd a small-scale coding matrix MhTo a priori knowledge, MlAnd MhRespectively with TlAnd ThComplementation, using Generalized Alternative Projection (GAP) algorithm to reconstruct high resolution image
Figure RE-GDA0002994040640000069
And low resolution images
Figure RE-GDA00029940406400000610
Camera performing MFObtaining M by secondary shootingFPerforming the above reconstruction process to obtain MF×NFHigh resolution image sequence of frames
Figure RE-GDA00029940406400000611
And low resolution image sequence
Figure RE-GDA00029940406400000612
And 4, the space-time registration of the image sequence and the solving of a spectrum transfer function:
obtaining two image sequences respectively by step 2 and step 3
Figure RE-GDA00029940406400000613
And
Figure RE-GDA00029940406400000614
and simultaneously DMD modulation images in the two image sequences are taken as reference frames, and in order to accelerate the space-time registration speed, images resolved by a large-scale coding matrix are taken as the reference frames. The invention adopts a posterior probability-based method to realize space-time registration, and the specific algorithm is as follows:
is provided with
Figure RE-GDA00029940406400000615
Representing the parameters to be estimated in a spatio-temporal registration model of two image sequences,
Figure RE-GDA00029940406400000616
is a time mapping parameter, where xi∈{1,…,nrN is the sequence number of the frame in the reference sequence corresponding to the ith frame in the observation sequence, noAnd nrNumber of observation sequence and reference sequence frames, respectively, and nr＝n0＝MF×NF；
Figure RE-GDA0002994040640000071
For spatial mapping parameters, the corresponding frame (i, x) is representedi) The spatial translation relationship between them.
The estimation of the joint spatio-temporal registration parameters Θ is:
Figure RE-GDA0002994040640000072
in the formula, the first term E (S)o(ii) a Θ) represents the registration parameters at a given spatio-temporal location
Figure RE-GDA0002994040640000073
And
Figure RE-GDA0002994040640000074
similarity between image sequences. Second item
Figure RE-GDA0002994040640000075
Representing a time registration regular term of continuous frames, and being used for punishing a frame corresponding relation which does not meet continuity constraint of a target moving direction, and adopting a normalized motion vector field integral value in an adjacent frame; item III
Figure RE-GDA0002994040640000076
Regularized to spatial registrationAnd the term is used for controlling the smoothness of spatial transformation between continuous frames and adopting the gray level correlation values of corresponding frames in different image sequences.
After the registration between the two sequences, obtaining a full-color image and a multispectral image of the reference frame image, in order to obtain SoAnd (3) a multispectral image corresponding to the subframe image, and a conversion function between the spectral image with different wavelengths and the panchromatic image is estimated by the reference frame image:
Figure RE-GDA0002994040640000077
the multispectral image of the sub-frame image is:
Figure RE-GDA0002994040640000078
obtaining M according to the above calculation procedureF×NF×NFIs/are as follows
Figure RE-GDA0002994040640000079
A sequence of multispectral images.
And 5, image sequence super-resolution reconstruction:
the method can obtain an image sequence with improved time and spectral resolution by a multi-scale aperture coding mode, adopts a maximum posterior probability algorithm to carry out super-resolution reconstruction on the multispectral image sequence, and can effectively improve the spatial resolution of the reconstructed image sequence by simultaneously utilizing non-redundant information between multi-frame panchromatic and multispectral images obtained by single exposure due to the introduction of a spectral conversion function in the algorithm, thereby realizing the improvement of the time, the space and the spectral resolution of the imaging system during single exposure imaging.
And (4) performing super-resolution reconstruction on the image sequence subjected to the space-time registration in the step (4) by adopting a maximum posterior probability algorithm to the high-resolution image sequence reconstructed by adopting the small-scale coding matrix:
Figure RE-GDA00029940406400000710
wherein, I is a full-color image corresponding to the current image to be reconstructed; μ denotes a regular coefficient; j. the design is a squarejIs composed of
Figure RE-GDA00029940406400000711
Current frame image in a multispectral image sequence, Ji,i∈[-N,N]For the current frame J in the image sequencejThe front and back adjacent images generally take front and back 2 frames; d is a down-sampling matrix, and the down-sampling factor is 2; k is based on the current frame image JjA fuzzy kernel estimated by an IRLS algorithm; (lambdajT) and V (lambda)iI) is a spectral transfer function; ^ represents the gradient operator; mijRepresents the current frame JjAnd adjacent frame JiFlow of light between; beta is a weight factor, and beta is more than 0 and less than or equal to 1. The above estimation is performed frame by frame to obtain MF×NF×NFA super-resolution image sequence I (λ, t) of frames.
In summary, the above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (6)

1. A multi-scale coded aperture spectrum time compressed sensing imaging method is characterized by comprising the following steps:
step 1, randomly extracting N from Hadamard with matrix elements { +1, -1}FLines, forming a matrix from each line, and upsampling to obtain NFN corresponding to the number of DMD units1×M1A coding matrix of dimensions; wherein N isFThe number of images that can be reconstructed for a single exposure image;
step 2, reconstructing the single-exposure multispectral compressed sensing image, which specifically comprises the following steps:
controlling the coding of the DMD by the coding matrix; the DMD receives a light beam of a target scene, and the light beam is divided into two beams, wherein one beam enters the wide spectrum camera after being subjected to grating dispersion; the wide-spectrum camera obtains a spectral image containing visible light and near infrared; the other beam of light enters the visible light camera;
the DMD passes through N in one exposure period of the wide-spectrum cameraFControlling the encoding matrix, wherein light beams form multispectral aliasing images with different wavelengths on the surface of a wide-spectrum camera detector; for the multispectral aliasing image, reconstructing a multispectral image sequence of a primary exposure period by adopting an alternative direction multiplier total variation regularization algorithm; after the wide spectrum camera is exposed for a set number of times, a multispectral image sequence S is obtainedo；
And 3, reconstructing a single-exposure aperture coding time compressed image sequence, specifically:
DMD Via NFControlling the coding matrix, and completing one-time exposure by a visible light camera to obtain an aliased full-color image; obtaining a full-color image sequence by an optimized inversion algorithm by using the aliased full-color image; the visible light camera obtains a full-color image sequence S after exposure for a set number of timesr；
Step 4, finding out a multispectral image sequence S in the same exposure periodoIn the exposure periodo(i)，i∈[1,…,NF]And a full-color image sequence SrIn the exposure periodr(t)，t∈[1,…,NF]Estimating the transfer function between the different wavelength spectral image and the panchromatic image:
Figure FDA0002893692360000011
the image sequence Sr(t) the ith wavelength λ corresponding to the t-th frame imageiThe following subframe images are:
So(λi,t)＝Sr(t)V(λi,t)，i∈[1,…,NF]
obtaining the sub-frame image corresponding to each frame of full-color image according to the above calculation method, and then obtaining MF×NF×NFA frame multispectral image sequence; mFSetting the exposure times;
step 5, for M obtained in step 4F×NF×NFAnd respectively carrying out super-resolution reconstruction on the frame multispectral image sequence to obtain a super-resolution image sequence.
2. The method as claimed in claim 1, wherein the multispectral image sequence S is obtained before step 3 and step 4 are performedoAnd a full-color image sequence SrThe spatio-temporal registration, i.e. the alignment of the images with respect to the same DMD exposure instant, is performed.
3. The multi-scale coded aperture spectral temporal compressed sensing imaging method of claim 2, wherein the spatio-temporal registration method is as follows:
in step 1, arbitrarily decimating N in Hadamard with matrix elements { +1, -1}FForming a matrix by each row to be used as a large-scale coding matrix; the dimension of the matrix being less than N1×M11/K of dimension; k is an integer greater than 1; when step 2 and step 3 are executed, the large-scale coding matrix is also used as a coding matrix of the DMD to control the DMD, and multispectral image sequences under the large-scale coding matrix are respectively obtained
Figure FDA0002893692360000022
And full color image sequence
Figure FDA0002893692360000023
Using a sequence of multispectral images
Figure FDA0002893692360000024
And full color image sequence
Figure FDA0002893692360000025
The spatiotemporal registration is achieved.
4. The method as claimed in claim 2 or 3, wherein the spatio-temporal registration is achieved by a posterior probability based method.
5. The multi-scale coded aperture spectral temporal compressed sensing imaging method according to claim 1, wherein the step 5 adopts a maximum a posteriori probability algorithm for reconstruction, and the specific method is as follows:
Figure FDA0002893692360000021
wherein, I is a full-color image corresponding to the current image to be reconstructed; μ denotes a regular coefficient; j. the design is a squarejFor a current frame image in an image sequence to be reconstructed, JiFor the current frame J in the image sequencejFront and rear adjacent images; d is a down-sampling matrix; k is based on the current frame image JjA fuzzy kernel estimated by an IRLS algorithm; (lambdajT) and V (lambda)iI) is a spectral transfer function; ^ represents the gradient operator; mijRepresents the current frame JjAnd adjacent frame JiFlow of light between; beta is a weight factor;
reconstructing frame by frame to obtain MF×NF×NFSuper-resolution image sequence I of frames*。
6. The method as claimed in claim 1, wherein the down-sampling factor of the down-sampling matrix D is 2.
CN202110037205.7A 2021-01-12 2021-01-12 Multi-scale coding aperture spectrum time compressed sensing imaging method Active CN112884644B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110037205.7A CN112884644B (en) 2021-01-12 2021-01-12 Multi-scale coding aperture spectrum time compressed sensing imaging method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110037205.7A CN112884644B (en) 2021-01-12 2021-01-12 Multi-scale coding aperture spectrum time compressed sensing imaging method

Publications (2)

Publication Number Publication Date
CN112884644A true CN112884644A (en) 2021-06-01
CN112884644B CN112884644B (en) 2022-11-15

Family

ID=76044452

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110037205.7A Active CN112884644B (en) 2021-01-12 2021-01-12 Multi-scale coding aperture spectrum time compressed sensing imaging method

Country Status (1)

Country Link
CN (1) CN112884644B (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106679807A (en) * 2016-11-01 2017-05-17 北京理工大学 Image compression and reconstruction method based on LCTF (liquid crystal tunable filter) hyperspectral imaging system
CN109102551A (en) * 2018-07-11 2018-12-28 中国人民解放军战略支援部队航天工程大学 A kind of time compressed sensing reconstructing method based on ray tracing

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106679807A (en) * 2016-11-01 2017-05-17 北京理工大学 Image compression and reconstruction method based on LCTF (liquid crystal tunable filter) hyperspectral imaging system
CN109102551A (en) * 2018-07-11 2018-12-28 中国人民解放军战略支援部队航天工程大学 A kind of time compressed sensing reconstructing method based on ray tracing

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
刘铭鑫等: "压缩感知光谱成像技术的编码孔径与探测器匹配优化", 《中国光学》 *

Also Published As

Publication number Publication date
CN112884644B (en) 2022-11-15

Similar Documents

Publication Publication Date Title
Cao et al. Computational snapshot multispectral cameras: Toward dynamic capture of the spectral world
Wu et al. Dense deep unfolding network with 3D-CNN prior for snapshot compressive imaging
JP5726057B2 (en) Camera and method for acquiring a sequence of frames of a scene as video
Monno et al. Multispectral demosaicking with novel guide image generation and residual interpolation
CN109146787B (en) Real-time reconstruction method of dual-camera spectral imaging system based on interpolation
CN111986084A (en) Multi-camera low-illumination image quality enhancement method based on multi-task fusion
CN110443865B (en) Multispectral imaging method and device based on RGB camera and depth neural network
JP2013108788A (en) Multispectral image information acquisition device and multispectral image information acquisition method
CN110650340A (en) Space-time multiplexing compressed video imaging method
CN116245726A (en) Compressed sensing polarization super-resolution imaging method based on deep learning framework
US10944923B2 (en) Code division compression for array cameras
CN114757842A (en) Color calculation ghost imaging method based on plug-and-play generalized alternative projection algorithm
Ding et al. Uncertainty-based thin cloud removal network via conditional variational autoencoders
Li et al. Hsvcnn: Cnn-based hyperspectral reconstruction from rgb videos
Saragadam et al. SASSI—super-pixelated adaptive spatio-spectral imaging
Ibrahim et al. 3DRRDB: Super resolution of multiple remote sensing images using 3D residual in residual dense blocks
KR20230118938A (en) optical method
CN112884644B (en) Multi-scale coding aperture spectrum time compressed sensing imaging method
CN111640069B (en) Compressive imaging method, system and device based on light sensing network and phase compensation
Li et al. Physics-based efficient full projector compensation using only natural images
CN113205453A (en) Hyperspectral fusion method based on space-spectrum total variation regularization
Li et al. Spec-nerf: Multi-spectral neural radiance fields
US20220405945A1 (en) Method, Device And Non-Transitory Computer-Readable Storage Medium For Processing A Sequence Of Top View Image Frames
CN103986936A (en) Video compression collecting system and method
Zhao et al. PPI Edge Infused Spatial-Spectral Adaptive Residual Network for Multispectral Filter Array Image Demosaicing

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant