CN113034414A - Image reconstruction method, system, device and storage medium - Google Patents

Image reconstruction method, system, device and storage medium Download PDF

Info

Publication number
CN113034414A
CN113034414A CN202110304115.XA CN202110304115A CN113034414A CN 113034414 A CN113034414 A CN 113034414A CN 202110304115 A CN202110304115 A CN 202110304115A CN 113034414 A CN113034414 A CN 113034414A
Authority
CN
China
Prior art keywords
image
convolution
sparse vector
alpha
output
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110304115.XA
Other languages
Chinese (zh)
Other versions
CN113034414B (en
Inventor
戴文睿
孙佳琦
李成林
邹君妮
熊红凯
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Jiaotong University
Original Assignee
Shanghai Jiaotong University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Jiaotong University filed Critical Shanghai Jiaotong University
Priority to CN202110304115.XA priority Critical patent/CN113034414B/en
Publication of CN113034414A publication Critical patent/CN113034414A/en
Application granted granted Critical
Publication of CN113034414B publication Critical patent/CN113034414B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Abstract

The invention provides an image reconstruction method and system, comprising the following steps: decomposing an input image into a group of convolution dictionaries and a sum of sparse vector convolutions, and establishing a compressed sensing optimization problem under the constraint of sparse vector zero norm; and constructing a differentiable optimization deep neural network based on the compressed sensing optimization problem, taking the measurement signal as the input of the deep neural network, solving a convolution dictionary and a sparse vector from a sampling matrix and the measurement signal, and reconstructing an image. A corresponding image reconstruction apparatus and storage medium are also provided. The compressed sensing image reconstruction model is constructed based on the convolution sparse coding, so that the redundancy of dictionary representation is reduced; the method is simultaneously suitable for random matrix sampling and 0/1 sparse matrix sampling modes, and can be respectively suitable for natural image compressed sensing reconstruction and magnetic resonance imaging reconstruction; has higher reconstruction precision.

Description

Image reconstruction method, system, device and storage medium
Technical Field
The invention relates to the field of image processing, in particular to an image reconstruction method, a system, a device and a storage medium based on a differentiable optimization deep neural network design, which can be applied to compressed sensing reconstruction of natural images and Magnetic Resonance Images (MRI).
Background
With the development of media technology, massive image data faces huge challenges in real-time transmission and storage. The proposal of the compressed sensing technology opens up a new idea in theory for solving the problems effectively. The compressive sensing theory considers that if a signal has sparsity under a certain transformation base, an original signal can be projected from a high-dimensional space to a low-dimensional space by using an undersampling matrix meeting certain characteristics to obtain an observed signal, and then the original signal is reconstructed at high probability from the observed signal far shorter than the signal length by solving an optimization problem with norm constraint.
For the above compressed sensing model, different norm constraints represent different reconstruction algorithms and reconstruction performance. If it adopts l0Norm, usually reconstructed by Orthogonal Matching Pursuit (OMP) algorithm. Although the norm satisfies the original idea of a compressed sensing model, namely finding a solution of an optimization problem with the minimum sparsity, the solution is often not accurate because it is an NP-Hard problem. Therefore, the students propose to utilize1Norm substitution of l0Norm makes it a convex optimization problem and proposes a series of iterative based reconstruction algorithms: an iterative threshold shrinkage algorithm ISTA, an approximate information transfer algorithm AMP, an alternating direction multiplier algorithm ADMM, and the like. Although the problems are further solved by the proposed algorithms, the algorithms are all solution algorithms based on an optimization theory, and still have the problems of limited reconstruction precision, complex iteration process, low convergence speed and the like.
With the development of deep learning techniques in recent years, researchers have proposed compressed sensing reconstruction methods Recon-Net and DR2-Net based on convolutional neural networks. The neural network-based method takes an observation signal as an input, and learns the direct mapping from the observation signal to an original signal by means of the strong learning ability of the neural network, but the method lacks interpretability and lacks theoretical guidance when a network is actually designed.
Through the literature search of the prior art, it is found that in the 'IEEE Conference on Neural Information Processing Systems (NIPS)' Conference in 2016, "Deep ADMM-Net for compressive sensing MRI", the calculation process of the alternating direction multiplier iterative algorithm of the optimization problem of compressed sensing reconstruction is abstracted into a Deep Neural network, i.e., the alternating direction multiplier Deep Neural network, and the original signal is reconstructed. The method combines a neural network and an optimization algorithm for the first time, but a measurement matrix used by the method is an 0/1 sparse matrix, and the method cannot be applied to the problem of compressed sensing reconstruction of natural images of random measurement matrices. At the same time, l for ADMM-Net1Norm instead of l0Norm, approximating the compressed sensing reconstruction problem to a convex optimization problem, which cannot directly solve for l0Norm constraint problem.
Disclosure of Invention
In view of the above-mentioned shortcomings in the prior art, the present invention provides an image reconstruction method, system, device and storage medium based on the design of a differentiable optimized deep neural network.
According to an aspect of the present invention, there is provided an image reconstruction method including: decomposing an input image into a group of convolution dictionaries and a sum of sparse vector convolutions, and establishing a compressed sensing optimization problem under the constraint of sparse vector zero norm; and constructing a differentiable optimization deep neural network based on the compressed sensing optimization problem, taking the measurement signal y as the input of the deep neural network, solving a convolution dictionary and a sparse vector from a sampling matrix and the measurement signal, and reconstructing an image.
Preferably, the establishing a compressed sensing optimization problem and the constructing a differentiable optimization deep neural network include:
decomposing an input image into a set of convolutional dictionaries dmAnd sparse vector convolution alphamSum ofmdm*αmAnd under the constraint of the sparse vector zero norm, establishing a compressed sensing optimization problem:
Figure BDA0002987424630000021
whereinPhi is a sampling matrix, y is a measurement signal, and lambda is a parameter of a constraint regular term;
will convolution dictionary dmToplitz expansion, splicing each convolution dictionary d along the row directionmObtaining a redundant dictionary D by using a Topriz-unfolded matrix, and splicing all sparse vectors alpha along the column directionmObtaining a total sparse vector alpha, i.e. the product of the redundant dictionary D and the total sparse vector alpha is equal to each convolution dictionary DmAnd a sparse vector alphamThe sum of the convolutions, expressed as Σmdm*αmD α; optimizing a compressed sensing problem
Figure BDA0002987424630000022
Conversion to solve the equivalent optimization problem
Figure BDA0002987424630000023
Decomposing the equivalent optimization problem into sub-optimization problems to obtain closed solutions of the sub-optimization problems in the t iteration, wherein the closed solutions comprise:
sparse vector solving: alpha is alphat＝[I+ρDTD]-1[ρDt(xt-1-vt-1)+(zt-1-ut-1)](ii) a Wherein alpha istIs a sparse vector, I is an identity matrix, ρ is a learnable parameter, xtTo reconstruct the signal, vtIs a dual variable with respect to the reconstructed signal, ztAs an auxiliary variable, utIs a dual variable with respect to sparse signals;
solving auxiliary variables: derived from the near-end mapping
Figure BDA0002987424630000028
Where ρ is2Is a learnable threshold parameter;
updating a reconstructed signal:
Figure BDA0002987424630000024
where ρ is1Is a learnable parameter;
updating dual variables: u. oft＝ut-1+αt-zt，vt＝vt-1+Dαt-xt；
Initializing a reconstructed signal x0Auxiliary variable z0And dual variable u0And v0；
Constructing a differentiable optimization deep neural network formed by T blocks according to the closed solution expressions of the iterative subproblems of sparse vector solution, auxiliary variable solution, reconstruction signal update and dual variable update, and respectively corresponding to the Tth (1) to Tth (T) iterations of the subproblems; outputs x of t-1 th blockt-1、zt-1、ut-1And vt-1As input to the t-th block, the solution process of the sub-optimization problem is repeated, and the corresponding output x is updated and obtainedt、zt、utAnd vt，t＝1,…,T；
Learning the deep neural network by using a training data set and adopting a reverse gradient propagation algorithm; defining a loss function of the deep neural network as a mean square error of an image reconstruction error in a training data set
Figure BDA0002987424630000031
Figure BDA0002987424630000032
Where N is the number of images in the training dataset, xiFor the ith image in the training dataset,
Figure BDA0002987424630000033
for the deep neural network according to the ith image xiMeasurement signal y ofiThe output obtained.
Preferably, the output x of the t-1 th blockt-1、zt-1、ut-1And vt-1As input to the t-th block, the solution process of the sub-optimization problem is repeated, and the corresponding output x is updated and obtainedt、zt、utAnd vtThe method comprises the following steps:
sparse vector solving: outputs x of t-1 th blockt-1、zt-1、ut-1And vt-1Convolutional layer through the deep neural network
Figure BDA0002987424630000034
A sum series approximation submodule for solving and outputting a sparse vector alphat；
Solving auxiliary variables: according to the sparse vector alphatCalculating alphat+ut-1And input to a Hard threshold function HardθImplement near-end mapping when alphat+ut-1When the value is more than or equal to theta, the auxiliary variable z is output by the hard threshold functiont＝αt+ut-1Otherwise, output zt0, where the threshold θ is λ/ρ2Is a learnable parameter;
updating a reconstructed signal: the sparse vector alpha istBy convolutional layers in the deep neural network
Figure BDA0002987424630000035
Corresponding transposed convolution layer N1The resulting output is used as D alpha in the reconstruction signal updatetUpdate and output xtWherein, in the step (A),
Figure BDA0002987424630000036
and N1The coefficients of (a) and (b) are kept consistent;
updating dual variables: using xt,zt，αt，ut-1,vt-1Updating and outputting dual variable utAnd vtWherein, D αtA value of alphatBy transposing the convolution layer N1And (4) obtaining.
Preferably, the output x of the t-1 th blockt-1、zt-1、ut-1And vt-1Convolutional layer through the deep neural network
Figure BDA0002987424630000037
A sum series approximation submodule for solving and outputting a sparse vector alphatThe method comprises the following steps:
calculating rho (x)t-1-vt-1) And input intoConvolution kernel height of H1A convolution kernel width of W1The number of input channels is CI, the number of output channels is CO1Of
Figure BDA0002987424630000038
Laminating the layers
Figure BDA0002987424630000039
Is output and zt-1-ut-1Summing, wherein the convolutional layers
Figure BDA00029874246300000310
Input channel number CI and input signal xt-1The number of channels is the same;
outputting a sparse vector alpha through the series approximation submoduletWherein, the number of input and output channels of the series approximation submodule is the convolution layer
Figure BDA0002987424630000041
Number of output channels CO1。
Preferably, the series approximation submodule includes a layer jump connection unit and a plurality of branch combination units cascaded with series approximation units M with different numbers; the number of branches of the series approximation unit M is a preset positive integer, when the number of branches is K, each branch combination unit is respectively cascaded with 1 to K series approximation units M, and the output of the branch combination units passes through a coefficient c1,…,cKWeighted combination and addition with the input of the series approximation submodule transmitted by the layer jump connecting unit to obtain a sparse vector alphat；
The series approximation unit M includes a transposed convolutional layer N2A linear rectification function layer and a corresponding transpose convolution layer N2Of
Figure BDA0002987424630000042
Wherein the convolutional layer
Figure BDA0002987424630000043
The number of output channels is CO1Convolution kernel height of H2Convolution kernel width of W2The number of input channels is CO2(ii) a The transposed convolution layer N2Has a convolution kernel height of H2Convolution kernel width of W2The number of input channels is CO1The number of output channels is CO2Convolution kernel coefficients and convolution layers
Figure BDA0002987424630000044
The same;
convolution layer used in reconstruction signal updating and dual variable updating
Figure BDA0002987424630000045
Corresponding transposed convolution layer N1With convolution kernel height H1Convolution kernel width of W1The number of input channels is CO1The number of output channels is CI, the convolution kernel coefficient and convolution layer
Figure BDA0002987424630000046
The same is true.
Preferably, the method for constructing the training data set includes:
randomly selecting a fixed-size image block from each training image, taking a random matrix as a sampling matrix to multiply the image block for sampling to obtain a measurement signal, taking each pair of original image block and measurement signal as a training sample pair, and establishing a training data set.
Preferably, the random matrix is 0/1 sparse matrix, the 0/1 sparse matrix is multiplied element by element with the whole image or the discrete fourier transform coefficient of the image to obtain measurement signals, and each pair of original image and measurement signal is used as a training sample pair to establish a training data set.
According to another aspect of the present invention, there is provided an image reconstruction system including:
an image decomposition module that decomposes an input image into a set of convolved dictionaries and a sum of sparse vector convolutions;
the problem optimization module is used for establishing a compressed sensing optimization problem under the constraint of zero norm of sparse vector;
and the image reconstruction module is used for constructing a differentiable optimization deep neural network according to the compressed sensing optimization problem, solving a convolution dictionary and a sparse vector from a sampling matrix and a measurement signal and reconstructing an image.
According to a third aspect of the present invention, there is provided an image reconstruction apparatus comprising a memory, a processor and a computer program stored on the memory and executable on the processor; the processor may be adapted to perform the image reconstruction method of any of the above mentioned items or to implement the image reconstruction system of the above mentioned items when executing the computer program.
According to a fourth aspect of the present invention, there is provided a computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, is adapted to carry out the image reconstruction method of any one of the above, or to carry out the image reconstruction system of the above.
Due to the adoption of the technical scheme, compared with the prior art, the invention has the following beneficial effects:
the image reconstruction method, the system, the device and the storage medium provided by the invention adopt the compressed sensing image reconstruction model constructed based on the convolution sparse coding, and reduce the redundancy of dictionary representation.
The image reconstruction method, the system, the device and the storage medium provided by the invention are simultaneously suitable for random matrix sampling and 0/1 sparse matrix sampling modes, and can be respectively suitable for natural image compressed sensing reconstruction and Magnetic Resonance Imaging (MRI) reconstruction.
The image reconstruction method, the system, the device and the storage medium provided by the invention directly solve the problem of the l0Norm constrained compressed sensing reconstruction.
The image reconstruction method, the system, the device and the storage medium provided by the invention adopt an optimization algorithm of a compressed sensing signal reconstruction model based on convolution sparse coding, namely, an iterative process of an alternative direction multiplier method is modeled into a deep neural network, and further, network parameters are trained through a large amount of training data to obtain the optimal model parameters for high-precision compressed sensing image reconstruction. And finally, reconstructing a high-quality original image from the measurement signal by applying the trained network.
The image reconstruction method, the system, the device and the storage medium have higher reconstruction precision.
The image reconstruction method, the system, the device and the storage medium provided by the invention provide a novel image reconstruction technology for multi-dimensional signals such as image videos and the like.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
fig. 1 is a flowchart of an image reconstruction method according to an embodiment of the present invention;
FIG. 2 is a flow chart of an image reconstruction method according to a preferred embodiment of the present invention;
FIG. 3 is a flow chart of the construction of a differentiably optimized deep neural network in a preferred embodiment of the present invention;
fig. 4 is a schematic diagram of a design method of the tth block (T ═ 1, …, T) of the deep neural network in a preferred embodiment of the present invention;
FIG. 5 is a diagram illustrating a design method of a sparse vector solution process (S131) in the t-th block of the deep neural network according to a preferred embodiment of the present invention;
FIG. 6 is a schematic structural diagram of a series approximation unit M of a series approximation submodule (S1312) in the sparse vector solving process (S131) of the deep neural network according to a preferred embodiment of the present invention;
FIG. 7 is a flow chart of an image reconstruction process in a preferred embodiment of the present invention;
fig. 8 is a schematic composition diagram of an image reconstruction system according to an embodiment of the present invention.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that variations and modifications can be made by persons skilled in the art without departing from the spirit of the invention. All falling within the scope of the present invention. Portions not described in detail below may be implemented using conventional techniques.
An embodiment of the invention provides an image reconstruction method, which is based on the design of a differentiable optimized deep neural network, combines the theoretical explanation of the traditional iterative reconstruction method and the powerful learning capacity of the neural network, effectively improves the image reconstruction quality, and can be simultaneously applied to the natural image compressive sensing reconstruction based on random Gaussian measurement matrix sampling and the magnetic resonance image compressive sensing reconstruction based on 0/1 sparse measurement matrix sampling.
Fig. 1 is a flowchart of an image reconstruction method according to this embodiment.
As shown in fig. 1, the image reconstruction method provided by this embodiment may include the following steps:
s100, decomposing an input image into a group of convolution dictionaries and a sum of sparse vector convolutions, solving an optimization problem of zero norm constraint, and realizing a compressed sensing optimization problem from a measurement signal; constructing a differentiable optimization deep neural network based on a compressed sensing optimization problem;
and S200, taking the measurement signal as the input of the deep neural network, solving the convolution dictionary and the sparse vector from the sampling matrix and the measurement signal, and reconstructing the image.
In S100 of this embodiment, as a preferred embodiment, establishing a compressed sensing optimization problem and constructing a differentiably optimized deep neural network may include the following steps:
s101, decomposing an input image into a group of convolution dictionaries dmAnd sparse vector convolution alphamSum ofmdm*αmAnd under the constraint of the sparse vector zero norm, establishing a compressed sensing optimization problem:
Figure BDA0002987424630000061
Figure BDA0002987424630000062
wherein phi is a sampling matrix, y is a measurement signal, and lambda is a parameter of a constraint regular term; will convolution dictionary dmToplitz expansion, splicing each convolution dictionary d along the row directionmObtaining a redundant dictionary D by using a Topriz-unfolded matrix, and splicing all sparse vectors alpha along the column directionmObtaining a total sparse vector alpha, i.e. the product of the redundant dictionary D and the total sparse vector alpha is equal to each convolution dictionary DmAnd a sparse vector alphamThe sum of the convolutions, expressed mathematically as Σmdm*αmD α. Will be the original problem
Figure BDA0002987424630000063
Conversion to solve the equivalent optimization problem
Figure BDA0002987424630000064
Figure BDA0002987424630000065
Decomposing the equivalent optimization problem into sub-optimization problems to obtain closed solutions of the sub-optimization problems in the t-th iteration, wherein the closed solutions can comprise the following steps:
s1011, sparse vector solving: alpha is alphat＝[I+ρDTD]-1[ρDT(xt-1-vt-1)+(zt-1-ut-1)](ii) a Wherein alpha istIs a sparse vector, I is an identity matrix, ρ is a learnable parameter, xtTo reconstruct the signal, vtIs a dual variable with respect to the reconstructed signal, ztAs an auxiliary variable, utIs a dual variable with respect to a sparse vector;
s1012, solving auxiliary variables: derived from the near-end mapping
Figure BDA0002987424630000071
Where ρ is2Is a learnable threshold parameter;
s1013, reconstruction signal update:
Figure BDA0002987424630000072
where ρ is1Is a learnable parameter;
s1014, updating the dual variable: u. oft＝ut-1+αt-zt，vt＝vt-1+Dαt-xt；
S102, initializing a reconstruction signal x0Auxiliary variable z0And dual variable u0And v0；
S103, constructing a differentiable optimization deep neural network formed by T blocks according to the iterative updating formula expression in S011-S1014, and respectively corresponding to the sub-optimization problem, wherein the T is 1, and the T is T iterations; outputs x of t-1 th blockt-1、zt-1、ut-1And vt-1As input to the t-th block, the solution process of the sub-optimization problem is repeated, and the corresponding output x is updated and obtainedt、zt、utAnd vt，t＝1,…,T；
S104, learning a deep neural network by using a training data set and adopting a reverse gradient propagation algorithm; defining a loss function of a deep neural network as a mean square error of an image reconstruction error in a training data set
Figure BDA0002987424630000073
Where N is the number of images in the training dataset, xiFor the ith image in the training dataset,
Figure BDA0002987424630000076
from the ith image x for a deep neural networkiMeasurement signal y ofiThe output obtained.
In S103 of this embodiment, as a preferred embodiment, the output x of the t-1 th block is processedt-1、zt-1、ut-1And vt-1As input to the t-th block, the solution process of the sub-optimization problem is repeated, and the corresponding output x is updated and obtainedt、zt、utAnd vtThe method can comprise the following steps:
s1031, sparse vector solving: outputs x of t-1 th blockt-1、zt-1、ut-1And vt-1Convolutional layer through deep neural network
Figure BDA0002987424630000077
A sum series approximation submodule for solving and outputting a sparse vector alphat；
S1032, solving auxiliary variables: from the output sparse vector alphatCalculating alphat+ut-1And input to a Hard threshold function HardθImplement near-end mapping when alphat+ut-1When the value is more than or equal to theta, the auxiliary variable z is output by the hard threshold functiont＝αt+ut-1Otherwise, output zt0, where the threshold θ is λ/ρ2Is a learnable parameter;
s1033, reconstruction signal updating: will sparsely vector alphatBy convolutional layers in deep neural networks
Figure BDA0002987424630000074
Corresponding transposed convolution layer N1The resulting output is used as D alpha in the reconstruction signal updatetUpdate and output xtWherein, in the step (A),
Figure BDA0002987424630000075
and N1The coefficients of (a) and (b) are kept consistent;
s1034, updating dual variables: using xt,zt，αt，ut-1,vt-1Updating and outputting dual variable utAnd vtWherein, D αtA value of alphatBy transposing the convolution layer N1And (4) obtaining.
In S1031 of this embodiment, as a preferred embodiment, the output x of the t-1 th block is outputt-1、zt-1、ut-1And vt -1Convolutional layer through deep neural network
Figure BDA0002987424630000081
A sum series approximation submodule for solving and outputting a sparse vector alphatThe method can comprise the following steps:
s10311, calculating rho (x)t-1-vt-1) And input to convolution kernel with height of H1A convolution kernel width of W1The number of input channels is CI, the number of output channels is CO1Of
Figure BDA0002987424630000082
Laminating the layers
Figure BDA0002987424630000083
Is output and zt-1-ut-1Adding and summing, wherein the convolution layers
Figure BDA0002987424630000084
Input channel number CI and input signal xt-1The number of channels is the same;
s10312, outputting a sparse vector alpha through a series approximation submoduletWherein, the number of input and output channels of the series approximation submodule is the convolution layer
Figure BDA0002987424630000085
Number of output channels CO1。
In S1031 of this embodiment, as a preferred embodiment, the series approximation submodule includes a layer jump connection unit and a plurality of branch combination units that cascade different number series approximation units M; the number of branches of the series approximation unit M is a preset positive integer, when the number of branches is K, each branch combination unit is respectively cascaded with 1 to K series approximation units M, and the output of the branch combination units passes through a coefficient c1,…,cKWeighted combination and addition with the input of the series approximation submodule transmitted by the layer jump connecting unit to obtain a sparse vector alphat。
In S1031 of this embodiment, as a preferred embodiment, the series approximation unit M includes a transposed convolutional layer N2A linear rectification function layer and a corresponding transpose convolution layer N2Of
Figure BDA0002987424630000086
Wherein the convolution layer
Figure BDA0002987424630000087
The number of output channels is CO1Convolution kernel height of H2Convolution kernel width of W2The number of input channels is CO2(ii) a Transposed convolution layer N2Has a convolution kernel height of H2Convolution kernel width of W2The number of input channels is CO1The number of output channels is CO2Convolution kernel coefficients and convolution layers
Figure BDA0002987424630000088
The same is true.
In S1031 of this embodiment, as a preferred embodiment, the convolution layers used in the signal update and the dual variable update are reconstructed
Figure BDA0002987424630000089
Corresponding transposed convolution layer N1With convolution kernel height H1Convolution kernel width of W1The number of input channels is CO1The number of output channels is CI, the convolution kernel coefficient and convolution layer
Figure BDA00029874246300000810
The same is true.
In S104 of this embodiment, as a preferred embodiment, the method for constructing the training data set may include the following steps:
randomly selecting a fixed-size image block from each training image, taking a random matrix as a sampling matrix to multiply the image block for sampling to obtain a measurement signal, taking each pair of original image block and measurement signal as a training sample pair, and establishing a training data set.
In S104 of this embodiment, as a preferred embodiment, the random matrix is 0/1 sparse matrix, the 0/1 sparse matrix is multiplied by the whole image or the discrete fourier transform coefficient of the image element by element to obtain measurement signals, and each pair of the original image and the measurement signal is used as a training sample pair to establish a training data set.
Fig. 2 is a flowchart of an image reconstruction method according to a preferred embodiment of the present invention.
As shown in fig. 2, the image reconstruction method provided by the preferred embodiment may include the following steps:
s1, decomposing the input image x into a set of convolution dictionaries dmAnd a sparse vector alphamThe sum of the convolutions: sigmamdm*αm(ii) a Establishing a compressed sensing optimization problem under the constraint of zero norm of sparse vector
Figure BDA0002987424630000091
Figure BDA0002987424630000092
Wherein phi is a sampling matrix, y is a measurement signal, lambda is a parameter of a constraint regular term, and alpha is a sparse vector; constructing a differentiable optimization deep neural network based on a compressed sensing optimization problem;
s2, taking the measurement signal y as the input of the deep neural network, and solving the convolution dictionary d from the sampling matrix phi and the measurement signal ymAnd a sparse vector alphamAnd reconstructing the image.
As a preferred embodiment, in S1, C convolution dictionaries d with length of i +1 are usedmHas a size of (j +1) × (j +1) Topritz matrix DmSplicing to obtain a redundant dictionary D with the size of (j +1) multiplied by C (j + 1); c sparse vectors alpha with the length of j +1mAnd splicing according to columns to obtain a sparse vector alpha.
Figure BDA0002987424630000093
Figure BDA0002987424630000094
Here, di,jIs a convolution dictionary diThe j element of (a)i,jIs a sparse vector alphaiThe jth element of (1).
Due to sigmamdm*αm＝DαReplacing C convolutional dictionaries D with D alphamAnd a sparse vector alphamOf convolutions, i.e. of
Figure BDA0002987424630000101
Constructing a compressed sensing image reconstruction problem:
Figure BDA0002987424630000102
and the image reconstruction efficiency is enhanced, and the convergence rate is improved.
As shown in fig. 3, as a preferred embodiment, the construction process of the differentiable deep neural network may include the following steps:
s11, optimizing problem decomposition: solving compressed sensing reconstruction optimization problem by using alternative direction multiplier algorithm
Figure BDA0002987424630000103
Figure BDA0002987424630000104
And obtaining a sub-optimization problem, and giving a closed-form solution of the sub-optimization problem in the t iteration. In this process, as a preferred embodiment, the following steps may be included:
s111, sparse vector solving: alpha is alphat＝[I+ρDTD]-1[ρDT(xt-1-vt-1)+(zt-1-ut-1)]。
S112, solving auxiliary variables: derived from the near-end mapping
Figure BDA0002987424630000105
S113, reconstruction signal updating:
Figure BDA0002987424630000106
s114, dual variable updating: u. oft＝ut-1+αt-zt，vt＝vt-1+Dαt-xt
S12, initialization: initializing a reconstructed signal x0Auxiliary variable z0And dual variable u0And v0。
S13, deep neural network structure design: and designing the deep neural network based on linear rectification function (ReLU), convolution and transposition convolution operations according to the closed solution expression given by S11. The deep neural network is composed of T blocks (blocks), and T equals 1 to T equals T iterations corresponding to S11. For the T-th block (T is 1, …, T), the output x of the T-1-th block is processedt-1,zt-1,ut-1,vt-1As an input (t is an input of 1 block as an initial value x0,z0,u0,v0) Updating and obtaining the corresponding output xt,zt,ut,vt。
S131: sparse vector solving: will input xt-1,zt-1,ut-1,vt-1By convolutional layers
Figure BDA0002987424630000107
A sum series approximation submodule for solving and outputting a sparse vector alphat。
As shown in fig. 4, in this process, as a preferred embodiment, the following steps may be included:
s1311: calculating rho (x)t-1-vt-1) And input to convolution kernel with height of H1Convolution kernel width of W1The number of input channels is CI, and the number of output channels is CO1Of
Figure BDA0002987424630000108
Laminating the layers
Figure BDA0002987424630000109
Is output and zt-1-ut-1And (4) adding and summing. Convolutional layer
Figure BDA00029874246300001010
Input channel number CI and input signal xt-1Has the same number of channels H1，W1And CO1Common convolutional layer parameters may be employed;
s1312: the sum obtained in S1311An over-series approximation submodule for outputting a sparse vector alphat. The number of input and output channels of the series approximation submodule being convolutional layers
Figure BDA00029874246300001011
Number of output channels CO1。
S132: auxiliary variables solving sub-network: solving sparse vector alpha output by sub-network according to sparse vectortCalculating alphat+ut-1And input to a Hard threshold function HardθImplement near-end mapping when alphat+ut-1Hard threshold function output auxiliary variable z at not less than thetat＝αt+ut-1Otherwise, output zt0, where the threshold θ is λ/ρ2Are learnable parameters.
S133: reconstruction signal update sub-network: will sparsely vector alphatBy corresponding to the convolution layer in S131
Figure BDA0002987424630000111
Corresponding transposed convolution layer N1(
Figure BDA0002987424630000112
And N1The coefficients of (D) are kept consistent), the resultant output is regarded as D α in S113tAccording to S113, x is updated and outputt。
S134: dual variables update the subnet: direct utilization of x according to S114t,zt，αt，ut-1,vt-1Updating and outputting dual variable utAnd vtIn which D αtA value of alphatBy transposing the convolution layer N1And (4) obtaining.
S14, learning learnable parameters in each constructed sub-network by using a reverse gradient propagation algorithm according to the training data set. The loss function of the network learning is defined as the mean square error of the image reconstruction error in the training data set
Figure BDA0002987424630000113
Figure BDA0002987424630000114
Where N is the number of images in the training dataset, xiIs the ith image in the training data set,
Figure BDA0002987424630000115
is S13 designed network according to xiMeasurement signal y ofiThe output obtained.
As shown in fig. 5, as a preferred embodiment, the series approximation submodule in S1312 is formed by a jump layer connection and a plurality of branch combinations of cascade-connected series approximation units M with different numbers. The number of the branches of the approximation unit is a preset positive integer, when the number of the branches is K, each branch is respectively cascaded with 1 to K series approximation units M, and the output of the series approximation units M passes through a coefficient c1,…,cKWeighted combination and series approximation submodule input addition transmitted by layer jump connection to obtain sparse vector alphat。
As shown in FIG. 6, further, the series approximation unit M is composed of a transposed convolution layer N2A linear rectifying function (ReLU) layer and a corresponding N2Of
Figure BDA0002987424630000116
And (4) forming. Here, the convolutional layer
Figure BDA0002987424630000117
The number of output channels is CO1Height of convolution kernel H2Width of convolution kernel W2And number of input channels CO2Conventional convolutional layer parameters may be used. Transposed convolution layer N2Height H of convolution kernel2Width of convolution kernel W2And convolution kernel coefficients and
Figure BDA0002987424630000118
same, but with the number of input channels CO1The number of output channels is CO2。
As a preferred embodiment, the blocks in S13 may be multiplexed to include convolutional layers
Figure BDA0002987424630000119
And
Figure BDA00029874246300001110
and the transposed convolution layer N1And N2Parameters including the convolution kernel coefficient reduce the training complexity and reduce the parameter quantity of the network model;
as a preferred embodiment, different parameters may be trained for each block in S13, so as to improve the image reconstruction performance.
As a preferred embodiment, as shown in fig. 7, in S14, a training data set of the image and its measurement signal is constructed, which may be as follows:
randomly selecting a fixed-size image block from each training image, taking a random matrix as a sampling matrix to multiply the image block for sampling to obtain a measurement signal, taking each pair of original image block and measurement signal as a training sample pair, and establishing a training data set.
Further, an 0/1 sparse matrix is used as a sampling matrix, the 0/1 sparse matrix is multiplied element by element with the whole image or the discrete Fourier transform coefficient of the image to obtain measurement signals, each pair of original image and measurement signal is used as a training sample pair, and a training data set is established.
Further, according to the training data set and the construction method of the deep neural network, network parameters are obtained through training.
Further, for a given measurement signal, as an input to the deep neural network, a reconstructed image is obtained at the output.
The image reconstruction method provided by the above embodiment of the present invention can be applied to natural image compressed sensing reconstruction in which a random matrix is used as a sampling matrix, and Magnetic Resonance Imaging (MRI) reconstruction in which an 0/1 sparse matrix is used as a sampling matrix. The following describes the technical solution provided by the above embodiment of the present invention in further detail with two specific application examples according to the above two image reconstruction requirements.
Example of specific application 1, Magnetic Resonance Imaging (MRI) image reconstruction
In this specific application example, three main steps of the image reconstruction system based on the design of the differentiable optimization deep neural network are included:
step 1, constructing a training data set of an image and a measurement signal thereof aiming at a magnetic resonance imaging problem. An 0/1 sparse matrix is used as a sampling matrix, the 0/1 sparse matrix phi and the whole image x or the discrete Fourier transform coefficient of the image are multiplied element by element to obtain a measurement signal y, each pair of original image and measurement signal is used as a training sample pair to establish a training set, and the trainable deep neural network is used for Magnetic Resonance Imaging (MRI) reconstruction. According to the existing original pictures, sparse sampling is carried out to obtain corresponding observation signals, each pair of original pictures and observation signals are used as a training sample pair (x, u), and a training data set gamma is established.
According to the training data set gamma, carrying out structural design according to the design method based on the differentiable optimization deep neural network shown in figure 3, and training to obtain the parameters of the deep neural network.
The method specifically comprises the following steps:
s11, optimizing problem decomposition: solving compressed sensing reconstruction optimization problem by using alternative direction multiplier algorithm
Figure BDA0002987424630000121
Figure BDA0002987424630000122
And obtaining a sub-optimization problem, and giving a closed-form solution of the sub-optimization problem in the t iteration. The system specifically comprises the following modules:
s111, a sparse vector solving module: alpha is alphat＝[I+ρDTD]-1[ρDT(xt-1-vt-1)+(zt-1-ut-1)]。
S112, an auxiliary variable solving module: derived from the near-end mapping
Figure BDA0002987424630000123
S113, a reconstruction signal updating module:
Figure BDA0002987424630000124
s114, a dual variable updating module: u. oft＝ut-1+αt-zt，vt＝vt-1+Dαt-xt
S12, initialization: initializing a reconstructed signal x0Auxiliary variable z0And dual variable u0And v0。
S13, deep neural network structure design: and designing the deep neural network based on linear rectification function (ReLU), convolution and transposition convolution operations according to the closed solution expression given by S11. The deep neural network is composed of T blocks (blocks), and T equals 1 to T equals T iterations corresponding to S11. For the T-th block (T is 1, …, T), the output x of the T-1-th block is processedt-1,zt-1,ut-1,vt-1As an input (t is an input of 1 block as an initial value x0,z0,u0,v0) Updating and obtaining the corresponding output xt,zt,ut,vtAs shown in fig. 3, the following sub-network structures are specifically included:
s131: sparse vector solving subnetwork: according to FIG. 4, input xt-1,zt-1,ut-1,vt-1By convolutional layers
Figure BDA0002987424630000131
A sum series approximation submodule for solving and outputting a sparse vector alphat(ii) a The method specifically comprises the following calculation steps:
s1311: calculating rho (x)t-1-vt-1) And input to convolution kernel with height of H1Convolution kernel width of W1The number of input channels is CI, and the number of output channels is CO1Of
Figure BDA0002987424630000132
Laminating the layers
Figure BDA0002987424630000133
Is output and zt-1-ut-1And (4) adding and summing. Convolutional layer
Figure BDA0002987424630000134
Input channel number CI and input signal xt-1Has the same number of channels H1，W1And CO1Common convolutional layer parameters may be employed;
s1312: the sum obtained by S1311 is processed by a series approximation submodule to output a sparse vector alphat. The number of input and output channels of the series approximation submodule being convolutional layers
Figure BDA0002987424630000135
Number of output channels CO1。
The series approximation submodule is formed by one jump layer connection and a plurality of branches of cascade different number series approximation units M. The number of the branches of the approximation unit is a preset positive integer, when the number of the branches is K, each branch is respectively cascaded with 1 to K series approximation units M, and the output of the series approximation units M passes through a coefficient c1,…,cKWeighted combination and series approximation submodule input addition transmitted by layer jump connection to obtain sparse vector alphat. As shown in FIG. 5, a series approximation unit M is designed, which is composed of a transposed convolutional layer N2A linear rectifying function (ReLU) layer and a corresponding N2Of
Figure BDA0002987424630000136
And (4) forming. Here, the convolutional layer
Figure BDA0002987424630000137
The number of output channels is CO1Height of convolution kernel H2Width of convolution kernel W2And number of input channels CO2Conventional convolutional layer parameters may be used. Transposed convolution layer N2Height H of convolution kernel2Width of convolution kernel W2And convolution kernel coefficients and
Figure BDA0002987424630000138
same, but with the number of input channels CO1The number of output channels is CO2。
S132: auxiliary variables solving sub-network: solving sparse vector alpha output by sub-network according to sparse vectortCalculating alphat+ut-1And input to a Hard threshold function HardθImplement near-end mapping when alphat+ut-1Hard threshold function output auxiliary variable z at not less than thetat＝αt+ut-1Otherwise, output zt0, where the threshold θ is λ/ρ2Are learnable parameters.
S133: reconstruction signal update sub-network: will sparsely vector alphatBy corresponding to the convolution layer in S131
Figure BDA0002987424630000139
Corresponding transposed convolution layer N1(
Figure BDA00029874246300001310
And N1The coefficients of (D) are kept consistent), the resultant output is regarded as D α in S113tAccording to S113, x is updated and outputt。
S134: dual variables update the subnet: direct utilization of x according to S114t,zt，αt，ut-1,vt-1Updating and outputting dual variable utAnd vtIn which D αtA value of alphatBy transposing the convolution layer N1And (4) obtaining.
S14, learning learnable parameters in each constructed sub-network by using a reverse gradient propagation algorithm according to the training data set. The loss function of the network learning is defined as the mean square error of the image reconstruction error in the training data set
Figure BDA00029874246300001311
Figure BDA0002987424630000141
Where N is the number of images in the training dataset, xiIs the ith image in the training data set,
Figure BDA0002987424630000142
is S13 designed network according to xiMeasurement signal y ofiThe output obtained.
And 2, inputting the observation signal to be tested into the network model trained in the step 1 to obtain a corresponding reconstructed image.
In numerical experiments, 150 256 × 256 brain Magnetic Resonance Imaging (MRI) images were randomly selected, and 50 images were arbitrarily selected as test data and 100 images were selected as training data. And taking a radial 0/1 sparse matrix as a sampling matrix, multiplying the sampling matrix by the whole image element by element to obtain a measurement signal, and taking each pair of original image and measurement signal as a training sample pair to establish a training set. The sampling rates are respectively 10%, 20%, 30%, 40% and 50%, that is to say, 10%, 20%, 30%, 40% and 50% of the number of the nonzero elements exist in the radial 0/1 sparse matrix. According to the compressive sensing reconstruction method based on the design of the differentiable optimization deep neural network, provided by the invention, a network structure is designed, and the five sampling rates are respectively subjected to network training to obtain the parameters of the deep neural network under the five sampling rates. And taking the radial 0/1 sparse matrix as a sampling matrix, multiplying the sampling matrix by the whole test image element by element to obtain a measurement signal, inputting the measurement signal into the trained deep neural network, and obtaining the output of the network, namely the original image needing to be reconstructed. And calculating the peak signal-to-noise ratio (PSNR) of the original test image and a reconstructed image output by the deep neural network, and evaluating the image reconstruction performance based on the design of the differentiable optimization deep neural network provided by the invention by taking the peak signal-to-noise ratio (PSNR) as a performance evaluation criterion.
This example compares the labeling method for magnetic resonance image reconstruction based on the Neural network developed by ADMM, the reconstruction method proposed by Yang Y, Sun J and Li H in "IEEE Conference on Neural Information Processing Systems (NIPs)" Conference in 2016, "Deep ADMM-Net for compressing MRI", and the reconstruction method proposed by Xie, Xingyu in "International Conference on Machine Learning (ICML)" Conference in 2016, "differential linear ADMM". In addition, this embodiment compares the labeling method of the compressed sensing reconstruction problem based on convolutional sparse coding, which is proposed by Heide, F and Heidrich, W et al in "Fast and flexible conditional space coding" published in "IEEE Conference on Computer Vision and Pattern Recognition (CVPR)" in 2015. Specific comparative experimental results are shown in table 1, and it can be seen that our method can achieve better reconstitution performance.
Table 1: FFCSC, ADMM-Net, DLADMM and methods of the present invention compare performance on brain Magnetic Resonance Imaging (MRI) datasets with radial 0/1 sparse matrix sampling
Figure BDA0002987424630000143
Figure BDA0002987424630000151
Specific application example 2, natural image compressed sensing reconstruction with random matrix as sampling matrix
In this specific application example, three main steps of the image reconstruction system based on the design of the differentiable optimization deep neural network are included:
step 1, constructing a training data set of an image and a measuring signal thereof aiming at a random matrix sampling compressed sensing reconstruction problem. The specific method is that image blocks with fixed size (33 multiplied by 33) and no overlap are randomly selected from each training image, and a random Gaussian matrix is used as a sampling matrix to be multiplied by the image blocks for sampling, so that a measurement signal is obtained. Because the pictures in the data set are different in size, such as 256 × 256 in some pictures and 500 × 500 in some pictures, there are blocks less than 33 × 33 in the edge of the image, and zero padding needs to be performed on the edge of the image. In an experiment, specifically, 33 × 33 image blocks are arranged in a row of column vectors of 1089 dimensions, and if the sampling rate is 0.01, the size of the sampling matrix Φ is 10 × 1089. And multiplying the column vectors formed by the image blocks by the sampling matrix phi to perform sampling to obtain a 10-dimensional measurement vector.
And taking each pair of original image blocks and measurement signals as a training sample pair, establishing a training data set gamma, and using the trainable deep neural network for random matrix sampling compressed sensing image reconstruction.
According to the training data set gamma, carrying out structural design according to the design method based on the differentiable optimization deep neural network shown in figure 3, and training to obtain the parameters of the deep neural network.
The method specifically comprises the following steps:
s11, optimizing problem decomposition: solving compressed sensing reconstruction optimization problem by using alternative direction multiplier algorithm
Figure BDA0002987424630000152
Figure BDA0002987424630000153
And obtaining a sub-optimization problem, and giving a closed-form solution of the sub-optimization problem in the t iteration. The system specifically comprises the following modules:
s111, a sparse vector solving module: alpha is alphat＝[I+ρDTD]-1[ρDT(xt-1-vt-1)+(zt-1-ut-1)]。
S112, an auxiliary variable solving module: derived from the near-end mapping
Figure BDA0002987424630000154
S113, a reconstruction signal updating module:
Figure BDA0002987424630000155
s114, a dual variable updating module: u. oft＝ut-1+αt-zt，vt＝vt-1+Dαt-xt
S12, initialization: initializing a reconstructed signal x0Auxiliary variable z0And dual variable u0And v0。
S13, deep neural network structure design: according to the supply of S11The obtained closed-form solution expression is based on linear rectification function (ReLU), convolution and transposition convolution operation, and a deep neural network is designed. The deep neural network is composed of T blocks (blocks), and T equals 1 to T equals T iterations corresponding to S11. For the T-th block (T is 1, …, T), the output x of the T-1-th block is processedt-1,zt-1,ut-1,vt-1As an input (t is an input of 1 block as an initial value x0,z0,u0,v0) Updating and obtaining the corresponding output xt,zt,ut,vtAs shown in fig. 3, the following sub-network structures are specifically included:
s131: sparse vector solving subnetwork: according to FIG. 4, input xt-1,zt-1,ut-1,vt-1By convolutional layers
Figure BDA00029874246300001610
A sum series approximation submodule for solving and outputting a sparse vector alphat(ii) a The method specifically comprises the following calculation steps:
s1311: calculating rho (x)t-1-vt-1) And input to convolution kernel with height of H1Convolution kernel width of W1The number of input channels is CI, and the number of output channels is CO1Of
Figure BDA0002987424630000161
Laminating the layers
Figure BDA0002987424630000162
Is output and zt-1-ut-1And (4) adding and summing. Convolutional layer
Figure BDA0002987424630000163
Input channel number CI and input signal xt-1Has the same number of channels H1，W1And CO1Common convolutional layer parameters may be employed;
s1312: the sum obtained by S1311 is processed by a series approximation submodule to output a sparse vector alphat. The number of input and output channels of the series approximation submodule being convolutional layers
Figure BDA0002987424630000164
Number of output channels CO1。
The series approximation submodule is formed by one jump layer connection and a plurality of branches of cascade different number series approximation units M. The number of the branches of the approximation unit is a preset positive integer, when the number of the branches is K, each branch is respectively cascaded with 1 to K series approximation units M, and the output of the series approximation units M passes through a coefficient c1,…,cKWeighted combination and series approximation submodule input addition transmitted by layer jump connection to obtain sparse vector alphat. As shown in FIG. 5, a series approximation unit M is designed, which is composed of a transposed convolutional layer N2A linear rectifying function (ReLU) layer and a corresponding N2Of
Figure BDA0002987424630000165
And (4) forming. Here, the convolutional layer
Figure BDA0002987424630000166
The number of output channels is CO1Height of convolution kernel H2Width of convolution kernel W2And number of input channels CO2Conventional convolutional layer parameters may be used. Transposed convolution layer N2Height H of convolution kernel2Width of convolution kernel W2And convolution kernel coefficients and
Figure BDA0002987424630000167
same, but with the number of input channels CO1The number of output channels is CO2。
S132: auxiliary variables solving sub-network: solving sparse vector alpha output by sub-network according to sparse vectortCalculating alphat+ut-1And input to a Hard threshold function HardθImplement near-end mapping when alphat+ut-1Hard threshold function output auxiliary variable z at not less than thetat＝αt+ut-1Otherwise, output zt0, where the threshold θ is λ/ρ2Are learnable parameters.
S133: reconstruction signal update sub-network: will sparsely vector alphatBy corresponding to the convolution layer in S131
Figure BDA0002987424630000168
Corresponding transposed convolution layer N1(
Figure BDA0002987424630000169
And N1The coefficients of (D) are kept consistent), the resultant output is regarded as D α in S113tAccording to S113, x is updated and outputt。
S134: dual variables update the subnet: direct utilization of x according to S114t,zt，αt，ut-1,vt-1Updating and outputting dual variable utAnd vtIn which D αtA value of alphatBy transposing the convolution layer N1And (4) obtaining.
S14, learning learnable parameters in each constructed sub-network by using a reverse gradient propagation algorithm according to the training data set. The loss function of the network learning is defined as the mean square error of the image reconstruction error in the training data set
Figure BDA0002987424630000171
Figure BDA0002987424630000172
Where N is the number of images in the training dataset, xiIs the ith image in the training data set,
Figure BDA0002987424630000173
is S13 designed network according to xiMeasurement signal y ofiThe output obtained.
And 2, inputting a measurement signal of the image to be tested into the network model trained in the step 1 to obtain a corresponding 33 x 33 image reconstruction block, rearranging the image block according to the index and the original row and column values, and removing the zero filling part if the image has zero filling operation in the step 1 to finally obtain a reconstructed image.
In numerical experiments, we randomly drawn 91 pictures from the ImageNet dataset as training data. 88912 image blocks of 33 x 33 size were randomly cut out of the 91 pictures. A random Gaussian matrix is used as a sampling matrix and multiplied by a plurality of image blocks with the size of 33 multiplied by 33 cut from the whole image to obtain measurement signals, and each pair of original image blocks and measurement signals are used as a training sample pair to establish a training set. The sampling rate is 1%, 4% and 10% respectively, that is to say, the random gaussian matrix has 1%, 4% and 10% of the number of non-zero elements respectively. According to the design method of the depth neural network based on the differential optimization for the image reconstruction, provided by the invention, a network structure is designed, and the three sampling rates are respectively subjected to network training to obtain the depth neural network parameters under the three sampling rates. In this embodiment, the common test image sets Set11 and BSD68 are selected as the test image sets. The method comprises the steps of using a random Gaussian matrix as a sampling matrix, multiplying the sampling matrix by a plurality of 33 x 33 image blocks divided from a whole test image to obtain a measurement signal, inputting the measurement signal into a trained deep neural network to obtain the output of the network, namely the original image blocks needing to be reconstructed, rearranging the image blocks according to original row and column values according to an index, and removing zero padding parts if the image has zero padding operation in step 1 to finally obtain a reconstructed image. And calculating the peak signal-to-noise ratio (PSNR) of the original test image and a reconstructed image output by the deep neural network, and evaluating the image reconstruction performance based on the design of the differentiable optimization deep neural network provided by the invention by taking the peak signal-to-noise ratio (PSNR) as a performance evaluation criterion.
It is noted that ADMM-Net, DLADMM and FFCSC are only suitable for 0/1 sparse matrix sampling in embodiment one, and cannot be applied to compressed sensing signal reconstruction in random matrix sampling mode.
This embodiment compares the labeling method of compressed sensing Reconstruction of natural Images based on neural network, and the Reconstruction method proposed by K.Kulkarni, S.Lohit and P.Turaga et al in the Conference "IEEE Conference on Vision and Pattern registration (CVPR)" in 2016, "Reconnet: Non-Iterative Reconstruction of Images from Computer compensated measures", and the Reconstruction method proposed by Zhang J and Ghang B in the Conference "IEEE Conference on Vision and Pattern Registration (PR)" in 2018, "ISTA-Net: inter-predictive-temporal-estimated word Reconstruction" are proposed. The specific comparison experiment results are shown in table 2, and it can be seen that the method is greatly improved compared with Reconnet under the condition of low sampling rate, and has equivalent effect to ISTA-Net.
TABLE 2 Reconnet, ISTA-Net and the method of the present invention compare the performance of the samples taken with random Gaussian measurement matrices at Set11 and Set68
Figure BDA0002987424630000181
An embodiment of the present invention provides an image reconstruction system, as shown in fig. 8, which may include: the system comprises an image decomposition module, a problem optimization module and an image reconstruction module; wherein:
an image decomposition module that decomposes an input image into a set of convolved dictionaries and a sum of sparse vector convolutions;
the problem optimization module is used for establishing a compressed sensing optimization problem under the constraint of zero norm of sparse vector;
and the image reconstruction module is used for constructing a differentiable optimization deep neural network according to the compressed sensing optimization problem, solving a convolution dictionary and a sparse vector from the sampling matrix and the measurement signal and reconstructing an image.
Further, the measurement signal y is used as an input of the deep neural network in the image reconstruction module.
An embodiment of the present invention provides an image reconstruction apparatus, including a memory, a processor, and a computer program stored on the memory and executable on the processor; the processor may be adapted to perform the image reconstruction method of any of the above embodiments, or to implement the image reconstruction system described above, when executing the computer program.
Optionally, a memory for storing a program; a Memory, which may include a volatile Memory (RAM), such as a Random Access Memory (SRAM), a Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), and the like; the memory may also comprise a non-volatile memory, such as a flash memory. The memories are used to store computer programs (e.g., applications, functional modules, etc. that implement the above-described methods), computer instructions, etc., which may be stored in partition in the memory or memories. And the computer programs, computer instructions, data, etc. described above may be invoked by a processor.
The computer programs, computer instructions, etc. described above may be stored in one or more memories in a partitioned manner. And the computer programs, computer instructions, data, etc. described above may be invoked by a processor.
A processor for executing the computer program stored in the memory to implement the steps of the method according to the above embodiments. Reference may be made in particular to the description relating to the preceding method embodiment.
The processor and the memory may be separate structures or may be an integrated structure integrated together. When the processor and the memory are separate structures, the memory, the processor may be coupled by a bus.
An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, is operable to perform the image reconstruction method of any one of the above embodiments or to implement the image reconstruction system of any one of the above embodiments.
According to the image reconstruction method, the image reconstruction system, the image reconstruction device and the storage medium, the input image is decomposed into a set of convolution dictionaries and the convolution sum of sparse vectors, the optimization problem of sparse vector zero norm constraint is solved, a closed solution expression of sub-optimization problems is formed, a depth neural network capable of being optimized differentially is designed, and efficient compressed sensing image reconstruction is achieved. The method mainly comprises the steps of problem decomposition optimization, network parameter initialization, deep neural network structure design and deep neural network parameter training. A plurality of pairs of measuring signals under low sampling rate and corresponding original images are used as training data sets to train model parameters of the designed deep neural network, so that the output image of the deep neural network is as close to the original image as possible when the measuring signals under low sampling rate are input; the method can be applied to compressed sensing image reconstruction in two sampling modes of random matrix sampling and 0/1 sparse matrix sampling, the image reconstruction efficiency is enhanced, and the convergence rate is improved.
It should be noted that, the steps in the method provided by the present invention may be implemented by using corresponding modules, devices, units, and the like in the system, and those skilled in the art may implement the composition of the system by referring to the technical solution of the method, that is, the embodiment in the method may be understood as a preferred example for constructing the system, and will not be described herein again.
Those skilled in the art will appreciate that, in addition to implementing the system and its various devices provided by the present invention in purely computer readable program code means, the method steps can be fully programmed to implement the same functions by implementing the system and its various devices in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Therefore, the system and various devices thereof provided by the present invention can be regarded as a hardware component, and the devices included in the system and various devices thereof for realizing various functions can also be regarded as structures in the hardware component; means for performing the functions may also be regarded as structures within both software modules and hardware components for performing the methods.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes and modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention.

Claims (10)

1. An image reconstruction method, comprising:
decomposing an input image into a group of convolution dictionaries and a sum of sparse vector convolutions, and establishing a compressed sensing optimization problem under the constraint of sparse vector zero norm;
and constructing a differentiable optimization deep neural network based on the compressed sensing optimization problem, taking the measurement signal y as the input of the deep neural network, solving a convolution dictionary and a sparse vector from a sampling matrix and the measurement signal, and reconstructing an image.
2. The image reconstruction method according to claim 1, wherein the establishing of the compressed sensing optimization problem and the constructing of the differentiably optimized deep neural network comprise:
decomposing an input image into a set of convolutional dictionaries dmAnd sparse vector convolution alphamSum ofmdm*αmAnd under the constraint of the sparse vector zero norm, establishing a compressed sensing optimization problem:
Figure FDA0002987424620000011
wherein phi is a sampling matrix, y is a measurement signal, and lambda is a parameter of a constraint regular term;
will convolution dictionary dmToplitz expansion, splicing each convolution dictionary d along the row directionmObtaining a redundant dictionary D by using a Topriz-unfolded matrix, and splicing all sparse vectors alpha along the column directionmObtaining a total sparse vector alpha, i.e. the product of the redundant dictionary D and the total sparse vector alpha is equal to each convolution dictionary DmAnd a sparse vector alphamThe sum of the convolutions, expressed mathematically as Σmdm*αmD α; optimizing a compressed sensing problem
Figure FDA0002987424620000012
Conversion to solve the equivalent optimization problem
Figure FDA0002987424620000013
Decomposing the equivalent optimization problem into sub-optimization problems to obtain the sub-optimization problem on the second levelClosed-form solutions in t iterations include:
sparse vector solving: alpha is alphat＝[I+ρDTD]-1[ρDT(xt-1-vt-1)+(zt-1-ut-1)](ii) a Wherein alpha istIs a sparse vector, I is an identity matrix, ρ is a learnable parameter, xtTo reconstruct the signal, vtIs a dual variable with respect to the reconstructed signal, ztAs an auxiliary variable, utIs a dual variable with respect to a sparse vector;
solving auxiliary variables: derived from the near-end mapping
Figure FDA0002987424620000014
Where ρ is2Is a learnable threshold parameter;
updating a reconstructed signal:
Figure FDA0002987424620000015
where ρ is1Is a learnable parameter;
updating dual variables: u. oft＝ut-1+αt-zt，vt＝vt-1+Dαt-xt；
Initializing a reconstructed signal x0Auxiliary variable z0And dual variable u0And v0；
Constructing a differentiable optimization deep neural network formed by T blocks according to closed solution expressions of iterative subproblems of sparse vector solution, auxiliary variable solution, reconstruction signal update and dual variable update, wherein the T is 1 to T is T times of iteration corresponding to the subproblem; outputs x of t-1 th blockt-1、zt-1、ut-1And vt-1As input to the t-th block, the solution process of the sub-optimization problem is repeated, and the corresponding output x is updated and obtainedt、zt、utAnd vt，t＝1,…,T；
Learning the deep neural network by using a training data set and adopting a reverse gradient propagation algorithm; applying a loss function of the deep neural networkMean square error defined as the error of image reconstruction in a training data set
Figure FDA0002987424620000021
Figure FDA0002987424620000022
Where N is the number of images in the training dataset, xiFor the ith image in the training dataset,
Figure FDA0002987424620000023
for the deep neural network according to the ith image xiMeasurement signal y ofiThe output obtained.
3. The image reconstruction method according to claim 2, wherein the output x of the t-1 th block is outputt-1、zt -1、ut-1And vt-1As input to the t-th block, the solution process of the sub-optimization problem is repeated, and the corresponding output x is updated and obtainedt、zt、utAnd vtThe method comprises the following steps:
sparse vector solving: outputs x of t-1 th blockt-1、zt-1、ut-1And vt-1Convolutional layer through the deep neural network
Figure FDA0002987424620000024
A sum series approximation submodule for solving and outputting a sparse vector alphat；
Solving auxiliary variables: according to the sparse vector alphatCalculating alphat+ut-1And input to a Hard threshold function HardθImplement near-end mapping when alphat+ut-1When the value is more than or equal to theta, the auxiliary variable z is output by the hard threshold functiont＝αt+ut-1Otherwise, output zt0, where the threshold θ is λ/ρ2Is a learnable parameter;
updating a reconstructed signal: the sparse vector alpha istBy passingConvolutional layer in the deep neural network
Figure FDA0002987424620000025
Corresponding transposed convolution layer N1The resulting output is used as D alpha in the reconstruction signal updatetUpdate and output xtWherein, in the step (A),
Figure FDA0002987424620000026
and N1The coefficients of (a) and (b) are kept consistent;
updating dual variables: using xt,zt，αt，ut-1,vt-1Updating and outputting dual variable utAnd vtWherein, D αtA value of alphatBy transposing the convolution layer N1And (4) obtaining.
4. The image reconstruction method according to claim 3, wherein the output x of the t-1 block is outputt-1、zt -1、ut-1And vt-1Convolutional layer through the deep neural network
Figure FDA0002987424620000027
A sum series approximation submodule for solving and outputting a sparse vector alphatThe method comprises the following steps:
calculating rho (x)t-1-vt-1) And input to convolution kernel with height of H1A convolution kernel width of W1The number of input channels is CI, the number of output channels is CO1Of
Figure FDA0002987424620000028
Laminating the layers
Figure FDA0002987424620000029
Is output and zt-1-ut-1Summing, wherein the convolutional layers
Figure FDA00029874246200000210
Input channel number CI and input signal xt-1The number of channels is the same;
outputting a sparse vector alpha through the series approximation submoduletWherein, the number of input and output channels of the series approximation submodule is the convolution layer
Figure FDA00029874246200000211
Number of output channels CO1。
5. The image reconstruction method according to claim 4, wherein the series approximation submodule includes a layer jump connection unit and a plurality of branch combination units cascade-connected different number series approximation units M; the number of branches of the series approximation unit M is a preset positive integer, when the number of branches is K, each branch combination unit is respectively cascaded with 1 to K series approximation units M, and the output of the branch combination units passes through a coefficient c1,…,cKWeighted combination and addition with the input of the series approximation submodule transmitted by the layer jump connecting unit to obtain a sparse vector alphat；
The series approximation unit M includes a transposed convolutional layer N2A linear rectification function layer and a corresponding transpose convolution layer N2Of
Figure FDA0002987424620000031
Wherein the convolutional layer
Figure FDA0002987424620000032
The number of output channels is CO1Convolution kernel height of H2Convolution kernel width of W2The number of input channels is CO2(ii) a The transposed convolution layer N2Has a convolution kernel height of H2Convolution kernel width of W2The number of input channels is CO1The number of output channels is CO2Convolution kernel coefficients and convolution layers
Figure FDA0002987424620000033
The same;
convolution layer used in reconstruction signal updating and dual variable updating
Figure FDA0002987424620000034
Corresponding transposed convolution layer N1With convolution kernel height H1Convolution kernel width of W1The number of input channels is CO1The number of output channels is CI, the convolution kernel coefficient and convolution layer
Figure FDA0002987424620000035
The same is true.
6. The image reconstruction method according to claim 2, wherein the training data set is constructed by a method comprising:
randomly selecting a fixed-size image block from each training image, taking a random matrix as a sampling matrix to multiply the image block for sampling to obtain a measurement signal, taking each pair of original image block and measurement signal as a training sample pair, and establishing a training data set.
7. The image reconstruction method of claim 6, wherein the random matrix is 0/1 sparse matrix, the 0/1 sparse matrix is multiplied element by element with the whole image or the discrete Fourier transform coefficient of the image to obtain the measurement signal, and each pair of the original image and the measurement signal is used as a training sample pair to establish the training data set.
8. An image reconstruction system, comprising:
an image decomposition module that decomposes an input image into a set of convolved dictionaries and a sum of sparse vector convolutions;
the problem optimization module is used for establishing a compressed sensing optimization problem under the constraint of zero norm of sparse vector;
and the image reconstruction module is used for constructing a differentiable optimization deep neural network according to the compressed sensing optimization problem, solving a convolution dictionary and a sparse vector from a sampling matrix and a measurement signal and reconstructing an image.
9. An image reconstruction apparatus comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor is operable to execute the image reconstruction method of any one of claims 1 to 7 or to implement the image reconstruction system of claim 8 when executing the program.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, is adapted to carry out the image reconstruction method of any one of claims 1 to 7 or to carry out the image reconstruction system of claim 8.
CN202110304115.XA 2021-03-22 2021-03-22 Image reconstruction method, system, device and storage medium Active CN113034414B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110304115.XA CN113034414B (en) 2021-03-22 2021-03-22 Image reconstruction method, system, device and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110304115.XA CN113034414B (en) 2021-03-22 2021-03-22 Image reconstruction method, system, device and storage medium

Publications (2)

Publication Number Publication Date
CN113034414A true CN113034414A (en) 2021-06-25
CN113034414B CN113034414B (en) 2022-11-11

Family

ID=76472526

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110304115.XA Active CN113034414B (en) 2021-03-22 2021-03-22 Image reconstruction method, system, device and storage medium

Country Status (1)

Country Link
CN (1) CN113034414B (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113674172A (en) * 2021-08-17 2021-11-19 上海交通大学 Image processing method, system, device and storage medium
CN114978306A (en) * 2022-05-17 2022-08-30 上海交通大学 Method and system for calculating mutual information quantity of optical fiber communication transmission system based on deep learning
CN116405100A (en) * 2023-05-29 2023-07-07 武汉能钠智能装备技术股份有限公司 Distortion signal restoration method based on priori knowledge
CN117319656A (en) * 2023-11-30 2023-12-29 广东工业大学 Quantized signal reconstruction method based on depth expansion

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106972862A (en) * 2017-03-21 2017-07-21 南开大学 Based on the sparse compressed sensing image reconstructing method of group for blocking nuclear norm minimum
CN107154064A (en) * 2017-05-04 2017-09-12 西安电子科技大学 Natural image compressed sensing method for reconstructing based on depth sparse coding
CN109683161A (en) * 2018-12-20 2019-04-26 南京航空航天大学 A method of the inverse synthetic aperture radar imaging based on depth ADMM network
CN109741256A (en) * 2018-12-13 2019-05-10 西安电子科技大学 Image super-resolution rebuilding method based on rarefaction representation and deep learning
CN110570486A (en) * 2019-08-23 2019-12-13 清华大学深圳研究生院 Under-sampling nuclear magnetic resonance image reconstruction method based on deep learning
US20200300957A1 (en) * 2019-03-20 2020-09-24 The Board Of Trustees Of The Leland Stanford Junior University Method for Estimating Systematic Imperfections in Medical Imaging Systems with Deep Learning
CN112258410A (en) * 2020-10-22 2021-01-22 福州大学 Differentiable low-rank learning network image restoration method
CN112365554A (en) * 2020-10-26 2021-02-12 天津大学 Compressed sensing image reconstruction method based on multi-scale residual error neural network

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106972862A (en) * 2017-03-21 2017-07-21 南开大学 Based on the sparse compressed sensing image reconstructing method of group for blocking nuclear norm minimum
CN107154064A (en) * 2017-05-04 2017-09-12 西安电子科技大学 Natural image compressed sensing method for reconstructing based on depth sparse coding
CN109741256A (en) * 2018-12-13 2019-05-10 西安电子科技大学 Image super-resolution rebuilding method based on rarefaction representation and deep learning
CN109683161A (en) * 2018-12-20 2019-04-26 南京航空航天大学 A method of the inverse synthetic aperture radar imaging based on depth ADMM network
US20200300957A1 (en) * 2019-03-20 2020-09-24 The Board Of Trustees Of The Leland Stanford Junior University Method for Estimating Systematic Imperfections in Medical Imaging Systems with Deep Learning
CN110570486A (en) * 2019-08-23 2019-12-13 清华大学深圳研究生院 Under-sampling nuclear magnetic resonance image reconstruction method based on deep learning
CN112258410A (en) * 2020-10-22 2021-01-22 福州大学 Differentiable low-rank learning network image restoration method
CN112365554A (en) * 2020-10-26 2021-02-12 天津大学 Compressed sensing image reconstruction method based on multi-scale residual error neural network

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
GUAN-JU PENG ET AL.: "Adaptive ADMM for Dictionary Learning in Convolutional Sparse Representation", 《IEEE TRANSACTIONS ON IMAGE PROCESSING》 *
XINGYU XIE ET AL.: "Differentiable Linearized ADMM", 《ARXIV》 *
YAN YANG ET AL.: "Deep ADMM-Net for Compressive Sensing MRI", 《30TH CONFERENCE ON NEURAL INFORMATION PROCESSING SYSTEMS (NIPS 2016)》 *

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113674172A (en) * 2021-08-17 2021-11-19 上海交通大学 Image processing method, system, device and storage medium
WO2023019685A1 (en) * 2021-08-17 2023-02-23 上海交通大学 Image processing method, system and apparatus, and storage medium
CN113674172B (en) * 2021-08-17 2023-11-28 上海交通大学 Image processing method, system, device and storage medium
CN114978306A (en) * 2022-05-17 2022-08-30 上海交通大学 Method and system for calculating mutual information quantity of optical fiber communication transmission system based on deep learning
CN116405100A (en) * 2023-05-29 2023-07-07 武汉能钠智能装备技术股份有限公司 Distortion signal restoration method based on priori knowledge
CN116405100B (en) * 2023-05-29 2023-08-22 武汉能钠智能装备技术股份有限公司 Distortion signal restoration method based on priori knowledge
CN117319656A (en) * 2023-11-30 2023-12-29 广东工业大学 Quantized signal reconstruction method based on depth expansion
CN117319656B (en) * 2023-11-30 2024-03-26 广东工业大学 Quantized signal reconstruction method based on depth expansion

Also Published As

Publication number Publication date
CN113034414B (en) 2022-11-11

Similar Documents

Publication Publication Date Title
CN113034414B (en) Image reconstruction method, system, device and storage medium
CN111667884B (en) Convolutional neural network model for predicting protein interactions using protein primary sequences based on attention mechanism
Chen et al. Deep learning with hierarchical convolutional factor analysis
Wilson et al. Thoughts on massively scalable Gaussian processes
Gribonval et al. Dictionary Identification—Sparse Matrix-Factorization via $\ell_1 $-Minimization
Zhang et al. From symmetry to geometry: Tractable nonconvex problems
Le Magoarou et al. Approximate fast graph fourier transforms via multilayer sparse approximations
Lu et al. Sparse spectral factorization: Unicity and reconstruction algorithms
Sheth et al. An application of storage-optimal matdot codes for coded matrix multiplication: Fast k-nearest neighbors estimation
CN109086802B (en) Image classification method based on eight-element convolution neural network
Chen et al. Learning memory augmented cascading network for compressed sensing of images
Yang et al. Parasketch: Parallel tensor factorization via sketching
Sedighin et al. Matrix and tensor completion in multiway delay embedded space using tensor train, with application to signal reconstruction
Xu et al. Learning simple thresholded features with sparse support recovery
CN115017366B (en) Unsupervised video hash retrieval method based on multi-granularity contextualization and multi-structure preservation
CN116958613A (en) Depth multi-view clustering method and device, electronic equipment and readable storage medium
Su et al. Invertible linear transforms based adaptive multi-view subspace clustering
Yang et al. Large scale tensor factorization via parallel sketches
Usevich Decomposing multivariate polynomials with structured low-rank matrix completion
Skretting Sparse signal representation using overlapping frames
Wang Efficient Adaptation of Deep Vision Models
Dang et al. Capsule network with shortcut routing
Yang et al. Hardware-efficient mixed-precision CP tensor decomposition
Kekre et al. Unified fast algorithm for most commonly used transforms using mixed radix and kronecker product
Huang et al. A New Measure of Model Redundancy for Compressed Convolutional Neural Networks

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant