CN110706346B - Space-time joint optimization reconstruction method and system - Google Patents
Space-time joint optimization reconstruction method and system Download PDFInfo
- Publication number
- CN110706346B CN110706346B CN201910875805.3A CN201910875805A CN110706346B CN 110706346 B CN110706346 B CN 110706346B CN 201910875805 A CN201910875805 A CN 201910875805A CN 110706346 B CN110706346 B CN 110706346B
- Authority
- CN
- China
- Prior art keywords
- light field
- data
- field image
- image data
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000005457 optimization Methods 0.000 title claims abstract description 31
- 238000000034 method Methods 0.000 title claims abstract description 21
- 238000003384 imaging method Methods 0.000 claims abstract description 49
- 238000007781 pre-processing Methods 0.000 claims abstract description 15
- 230000002457 bidirectional effect Effects 0.000 claims abstract description 12
- 230000002123 temporal effect Effects 0.000 claims description 24
- 238000013519 translation Methods 0.000 claims description 20
- 230000003287 optical effect Effects 0.000 claims description 18
- 238000005070 sampling Methods 0.000 claims description 11
- 238000012163 sequencing technique Methods 0.000 claims description 9
- 238000013075 data extraction Methods 0.000 claims description 6
- 238000012937 correction Methods 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 3
- 125000004122 cyclic group Chemical group 0.000 claims description 3
- 230000004927 fusion Effects 0.000 claims description 3
- 238000012986 modification Methods 0.000 claims description 3
- 230000004048 modification Effects 0.000 claims description 3
- 230000008707 rearrangement Effects 0.000 claims description 3
- 238000000926 separation method Methods 0.000 claims description 3
- 230000008859 change Effects 0.000 abstract description 3
- 108010033040 Histones Proteins 0.000 description 3
- 238000000386 microscopy Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 241000244206 Nematoda Species 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/10—Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/80—Geometric correction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10056—Microscopic image
- G06T2207/10061—Microscopic image from scanning electron microscope
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
- G06T2219/20—Indexing scheme for editing of 3D models
- G06T2219/2016—Rotation, translation, scaling
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Geometry (AREA)
- Computer Graphics (AREA)
- Software Systems (AREA)
- Architecture (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Investigating, Analyzing Materials By Fluorescence Or Luminescence (AREA)
Abstract
The invention provides a space-time joint optimization reconstruction system based on a scanning light field microscopic imaging system, which comprises: the system comprises a scanning light field acquisition module, a preprocessing module and a space-time combined bidirectional circulating reconstruction module. The invention also provides a space-time joint optimization reconstruction method based on the scanning light field microscopic imaging system. The space-time joint constraint is added, the traditional reconstruction algorithm is optimized, the reconstruction speed is greatly increased, the three-dimensional reconstruction time is effectively reduced, the body imaging time resolution, namely the time resolution sacrificed by scanning, is recovered, and meanwhile, the motion blur and the artifacts caused by the too fast change of the sample are eliminated, so that the performance of the scanning light field microscopic imaging system is improved, and the application of the scanning light field microscopic imaging system in scenes such as high-speed capture of biological tissues is expanded; for the whole time delay three-dimensional video, the time bidirectional circulation algorithm mainly utilizes the continuity and redundancy of four-dimensional information in the time dimension, and the integral signal-to-noise ratio can be improved.
Description
Technical Field
The invention belongs to the technical field of image three-dimensional reconstruction, and particularly relates to a spatio-temporal joint optimization reconstruction method and a spatio-temporal joint optimization reconstruction system based on a scanning light field microscopic imaging system.
Background
Three-dimensional imaging is always the leading edge of a hot spot in the field of microscopic imaging research, and has great significance for scientists to explore life secret and disease mechanisms. Scanning light field microscopy imaging systems are an important three-dimensional imaging technique. The method introduces sub-pixel translation into the traditional light field microscopic imaging system, overcomes the problem of low spatial resolution of the traditional light field microscope, and achieves diffraction limit imaging. Meanwhile, the acquisition speed of the scanning light field microscopic imaging system can reach 100Hz, and the scanning light field microscopic imaging system can be applied to three-dimensional dynamic observation of a plurality of subcellular levels. This is a very promising three-dimensional microscopic imaging technique.
On the other hand, the scanning light field microscopic imaging system has certain defects. The first is that the acquisition speed is high, but the post-processing reconstruction speed is slow, which brings great challenges to the computer performance and cannot be used in real-time imaging; secondly, the actual volume imaging frame rate is far lower than the image acquisition frame rate, the volume imaging frame rate is only about 10Hz, and a three-dimensional volume can be reconstructed because the scanning light field microscopic imaging system needs to acquire a plurality of pieces of scanning light field data; thirdly, biological tissues which move at an ultra-fast speed cannot be imaged, and when the form difference of samples recorded by a plurality of pieces of scanning light field data is too large, the reconstruction result of the scanning light field microscopic imaging system is easy to generate motion blur and artifacts, so that the inaccuracy of the data is caused. Therefore, the scanning light field microscopic imaging system has a great limitation in a high dynamic imaging environment. These deficiencies have limited further development and application of scanning light field microscopy imaging systems.
Disclosure of Invention
In order to solve the technical problems, the invention aims to improve the three-dimensional reconstruction performance of a scanning light field microscopic imaging system by a space-time joint optimization method, recover the time resolution loss caused by scanning as much as possible, remove the motion blur and the artifacts caused by the over-fast change of a sample and greatly improve the reconstruction speed.
The following presents a simplified summary in order to provide a basic understanding of some aspects of the disclosed embodiments. This summary is not an extensive overview and is intended to neither identify key/critical elements nor delineate the scope of such embodiments. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.
The invention adopts the following technical scheme:
in some alternative embodiments, there is provided a spatio-temporal joint optimization reconstruction system, comprising:
the scanning light field acquisition module is used for converting the information of the sample into a light signal, converting the light signal into light field image data and acquiring the light field image data so as to obtain a scanning light field data stack;
the preprocessing module is used for preprocessing the light field image data, correcting the light field image data to accord with a physical model, and constructing an image stack according to a time sequence;
and the space-time joint bidirectional circulation reconstruction module is used for judging the data direction of the light field image, reading data according to the sliding window of the data direction of the light field image, determining time weight according to the sample motion rate, rearranging the pixels of the weighted light field image data, performing spatial up-sampling to obtain four-dimensional phase space data, and reconstructing a three-dimensional space volume by using a Richardson Lucy algorithm.
In some optional embodiments, the spatio-temporal joint bidirectional cyclic reconstruction module comprises:
the data direction judging unit is used for judging the data direction of the light field image, firstly determining the data input direction according to the sequence of time from front to back, and when the data is input into the last frame, reversing the time, wherein the data direction is from the last frame to the first frame;
a sliding window reading unit for reading data by sliding a window, and reading scan data corresponding to one volume in each sliding manner according to the data direction determined by the data direction determining unit;
a temporal weight interleaving unit for determining a temporal weight according to the sample motion rate;
the phase space data extraction unit is used for rearranging the pixels of the weighted light field image data, and performing spatial up-sampling to obtain four-dimensional phase space data;
the space three-dimensional initialization unit is used for initializing a space three-dimensional volume;
and the space three-dimensional reconstruction unit is used for reconstructing a space three-dimensional volume by using the Richardson Lucy algorithm.
In some optional embodiments, the scanning light field acquisition module comprises: the device comprises a sample, a scanning light field unit and an image sensor, wherein information of the sample is converted into an optical signal, then light field image data with sub-pixel translation is generated through the scanning light field unit, and the light field image data is collected through the image sensor to obtain a scanning light field data stack.
In some optional embodiments, the pre-processing module comprises: the data correction unit is used for correcting the light field image data to enable the light field image data to conform to a physical model of light field imaging; and the scanning light field time sequencing unit is used for sequencing the corrected light field image data according to the time sequence.
In some optional embodiments, the data modification unit includes: the rotation subunit is used for rotating the light field image data based on an interpolation algorithm to enable the light field image data to conform to a physical model of light field imaging; the cutting subunit is used for cutting the light field image data and removing redundant boundaries; a translation subunit, configured to perform pixel translation or sub-pixel translation on the light-field image data.
In some optional embodiments, the scan light field time ordering unit comprises: the scanning image single-sheet separation subunit is used for separating the light field image data corresponding to one volume according to different scanning translation amounts; and constructing a time sequence stack subunit, which is used for sequencing the two-dimensional data separated by the single scanning image separating subunit according to the acquisition time to form a time sequence stack.
In some optional embodiments, the time weight interleaving unit comprises: a temporal weight calculation subunit for estimating a temporal weight from the motion rate of the sample; a temporal weight assignment subunit for assigning temporal weights to the light field image data.
In some optional embodiments, the phase space data extraction unit includes: a pixel rearrangement subunit, configured to convert the light field image data into four-dimensional phase space data; and the up-sampling interpolation subunit is used for up-sampling the four-dimensional phase space data so as to be used for aperture fusion in three-dimensional reconstruction.
In some optional embodiments, the spatio-temporal joint optimization reconstruction method comprises:
s1: converting the information of a sample into an optical signal, converting the optical signal into optical field image data, acquiring the optical field image data, preprocessing the optical field image data, separating the optical field image data corresponding to a volume according to different scanning translation amounts, sequencing the separated two-dimensional data according to acquisition time, and constructing an image stack;
s2: judging the direction of the light field image data, determining the data input direction according to the sequence of the time sequence from the front to the back in the step S1, when the data is input to the last frame, reversing the time direction, wherein the data direction is from the last frame to the first frame, and inputting the current data stream format to the step S3;
s3: sliding the window to read data, and reading the data stream and the data direction determined in the step S2;
s4: determining a time weight according to the sample motion rate, and assigning the time weight to the light field image data;
s5: rearranging the pixels of the weighted light field image data, and performing spatial up-sampling interpolation to convert the weighted light field image data into four-dimensional phase spatial data;
s6: reconstructing a spatial three-dimensional volume by using a Richardson Lucy algorithm;
s7: and judging whether the iteration upper limit of the Richardson Lucy algorithm is reached, if so, ending, and otherwise, returning to the step S3.
The invention has the following beneficial effects: the space-time joint constraint is added, the traditional reconstruction algorithm is optimized, the reconstruction speed is greatly increased, the three-dimensional reconstruction time is effectively reduced, the body imaging time resolution, namely the time resolution sacrificed by scanning, is recovered, and meanwhile, the motion blur and the artifacts caused by the too fast change of the sample are eliminated, so that the performance of the scanning light field microscopic imaging system is improved, and the application of the scanning light field microscopic imaging system in scenes such as high-speed capture of biological tissues is expanded; for the whole time delay three-dimensional video, the time bidirectional cyclic algorithm mainly utilizes the continuity and redundancy of four-dimensional information in a time dimension, and the integral signal-to-noise ratio can be improved.
Drawings
FIG. 1 is a schematic block diagram of a spatio-temporal joint optimization reconstruction system of the present invention;
FIG. 2 is a flow chart of the spatio-temporal joint optimization reconstruction method of the present invention;
FIG. 3 is a comparison graph of the three-dimensional reconstruction effect of the space-time joint optimization of nematode histone.
Detailed Description
The following description and the drawings sufficiently illustrate specific embodiments of the invention to enable those skilled in the art to practice them. Other embodiments may incorporate structural, logical, electrical, process, and other changes. The examples merely typify possible variations. Individual components and functions are optional unless explicitly required, and the sequence of operations may vary. Portions and features of some embodiments may be included in or substituted for those of others.
As shown in fig. 1, in some illustrative embodiments, there is provided a spatio-temporal joint optimization reconstruction system based on a scanning light field microscopy imaging system, comprising: the scanning light field acquisition module 100, the preprocessing module 200 and the space-time joint bidirectional cycle reconstruction module 300.
The scan light field acquisition module 100 is configured to convert information of a sample into an optical signal, convert the optical signal into light field image data, and acquire the light field image data to obtain a scan light field data stack.
The scanning light field acquisition module 100 includes: a sample 110, a scanning light field unit 120, and an image sensor 130. After the information of the sample 110 is converted into an optical signal, the scanned light field unit 120 generates light field image data with sub-pixel translation, and the light field image data is collected by the image sensor 130 to obtain a scanned light field data stack.
The scanning light field unit 120 may scan the imaging light path and may also scan the microlens array. The image sensor 130 may be a CMOS, a monochrome sensor, a CCD or a CMOS, or other types of imaging sensors, and is not limited in detail.
The preprocessing module 200 is configured to preprocess the light field image data, correct the light field image data to conform to the physical model, and construct an image stack according to a time sequence.
The pre-processing module 200 includes: a data correction unit 210 and a scanning light field time ordering unit 220. A data correction unit 210, configured to correct the light field image data to conform to a physical model of light field imaging; a scanning light field time ordering unit 220 for ordering the corrected light field image data in time order.
The data modification unit 210 includes: a rotation subunit 211, configured to perform rotation based on an interpolation algorithm on the light field image data, so that the light field image data conforms to a physical model of light field imaging; a cropping subunit 212, cropping the light field image data to remove redundant boundaries; the shift subunit 213 performs pixel shift or sub-pixel shift on the light field image data to reduce errors. The data correction unit 210 corrects the acquired light field image data, and can correct the physical model by interpolation operation of a discrete mathematical matrix.
The scan light field time ordering unit 220 includes: a scanned image single-sheet separation subunit 221, configured to separate light field image data corresponding to one volume according to different scanning translation amounts; and a time sequence stack subunit 222 is constructed, and is used for sequencing the two-dimensional data separated by the single scanned image separating subunit 221 according to the acquisition time to form a time sequence stack.
The space-time joint bidirectional cycle reconstruction module 300 is configured to determine a light field image data direction, read data according to a sliding window of the light field image data direction, determine a time weight according to a sample motion rate, rearrange pixels of the weighted light field image data, perform spatial upsampling to obtain four-dimensional phase space data, and reconstruct a three-dimensional volume of a space by using a richardson luci algorithm.
The space-time joint bidirectional circulation reconstruction module 300 realizes space-time joint bidirectional circulation optimization reconstruction of scanning light field image data by combining low rank and sparsity of four-dimensional information of motion and response of a three-dimensional sample in space, optimizes a traditional three-dimensional reconstruction method, recovers acquisition time resolution and improves reconstruction rate. The method specifically comprises the following steps: a data direction determination unit 310, a sliding window reading unit 320, a time weight interleaving unit 330, a phase space data extraction unit 340, a spatial three-dimensional initialization unit 350, and a spatial three-dimensional reconstruction unit 360.
The data direction determining unit 310 is configured to determine the light field image data direction, determine the data input direction according to the sequence from front to back of the time sequence of the scanning light field time ordering unit 220, when the data is input into the last frame, reverse the time direction, where the data direction is from the last frame to the first frame, and input the current data stream format into the sliding window reading unit 320.
A sliding window reading unit 320 for reading the data by sliding the window, and reading the scan data corresponding to one volume at a time by sliding according to the data direction determined by the data direction determining unit.
And the time weight interleaving unit 330 is configured to determine a time weight according to the sample motion rate, assign a different weight to each two-dimensional light field image, assign a higher weight to data at a time instant, and perform time weight interleaving balance.
And a phase space data extraction unit 340, configured to rearrange pixels of the weighted light field image data, perform spatial upsampling, and acquire four-dimensional phase space data.
And a spatial three-dimensional initialization unit 350, configured to initialize a spatial three-dimensional volume, where a uniform value is used as an initial value in a first reconstruction, and then an average value of the uniform initial values and a reconstruction result of a previous frame are used to replace the uniform initial values.
The spatial three-dimensional reconstruction unit 360 reconstructs a spatial three-dimensional volume using the richardson-lucy algorithm. The space-time joint bidirectional circulation reconstruction module 300 can perform bidirectional circulation and rapid reconstruction. The spatial three-dimensional reconstruction unit 360 may reconstruct a three-dimensional structure, may eliminate the influence of the sample on the signal on the out-of-focus plane by using the imaging stack, and may realize three-dimensional reconstruction of the sample based on the imaging stack. The computational reconstruction process of the spatial three-dimensional reconstruction unit 360 may be implemented on a hardware system such as a general personal computer or a workstation. The computational reconstruction portion may perform a computational reconstruction of the sample using the acquired image information.
The temporal weight interleaving unit 330 calculates temporal weights required for reconstruction, and uses the temporal weights in the scanned light field image data, i.e., estimates temporal weights according to different motion rates of the samples, and assigns the temporal weights to the scanned light field data. The method specifically comprises the following steps: a temporal weight calculation subunit 331 configured to estimate a temporal weight from the motion rate of the sample; a temporal weight assignment subunit 332 for assigning temporal weights to the light-field image data.
The phase space data extracting unit 340 restores the light field image data with the overlap into four-dimensional phase space data, which specifically includes: a pixel rearrangement subunit 341 configured to convert the light field image data into four-dimensional phase space data; an upsampling interpolation subunit 342, configured to upsample the four-dimensional phase spatial data for use in aperture fusion in three-dimensional reconstruction.
Different from the traditional microscopic three-dimensional reconstruction method, the method mainly utilizes the low rank and sparsity of the four-dimensional information of the motion and response of the three-dimensional sample in the space, so that the video reconstruction has more globality. Each scanning is one-time acquisition of global information, so that the time resolution lost due to scanning can be recovered by using a space-time joint optimization algorithm, and the acquisition speed is further improved. If the four-dimensional information, i.e. the delayed 3D video, is reconstructed as a whole, the temporal resolution lost by the periodic scanning process can be recovered using a sliding window.
As shown in fig. 2, in some illustrative embodiments, a spatio-temporal joint optimization reconstruction method based on a scanning light field microscopic imaging system is provided, and the invention is implemented by firstly constructing the scanning light field microscopic imaging system, and performing sub-pixel scanning on an optical path at a collection end by using optical-electromechanical devices such as a scanning galvanometer on the basis of a conventional light field microscope, so as to achieve a microscopic imaging system with the capability of collecting data of a scanning light field.
The space-time joint optimization reconstruction method comprises the following steps:
s1: and acquiring light field image data and preprocessing the light field image data.
The constructed scanning light field microscopic imaging system is used for scanning and collecting light field information of a sample, namely the information of the sample is converted into light signals and then converted into light field image data and the light field image data is collected, after the light field image data is preprocessed, the light field image data corresponding to one volume is separated according to different scanning translation amounts, the separated two-dimensional data is sequenced according to collection time, and an image stack is constructed.
Preprocessing the light field image data includes: and preprocessing such as translation, rotation, cutting and the like to ensure the one-to-one correspondence of the pixels. The rotation is to rotate the light field image data based on an interpolation algorithm so as to enable the light field image data to conform to a physical model of light field imaging; the cutting is to cut the light field image data and remove redundant boundaries; the translation is pixel translation or sub-pixel translation of the light field image data to reduce errors.
S2: and judging the multi-frame data reconstruction direction.
And judging the direction of the light field image data, determining the data input direction according to the sequence from the front to the back of the time sequence of the step S1, reversing the time direction when the last frame is input, wherein the data direction is from the last frame to the first frame, and inputting the current data stream format to the step S3.
S3: the window is slid to read the data.
Sliding the window to read data, and reading the data stream and the data direction determined in the step S2; and ensuring that the kth reading of the data in the [ k, k + N-1] sliding window is carried out. When the data direction is opposite, the sliding frame is reversely read.
S4: time weight staggered balancing is performed.
And determining a time weight according to the sample motion rate, assigning the time weight to the light field image data, giving different weights to each two-dimensional light field image data, and giving higher weight to the data at the time moment. A single light field data at a certain moment is endowed with higher weight, so that the influence of other moments can be relieved, and the motion blur and the artifacts are relieved.
S5: and extracting phase space information.
And rearranging the pixels of the weighted light field image data, and performing spatial up-sampling interpolation to convert the weighted light field image data into four-dimensional phase space data.
S6: and carrying out spatial three-dimensional reconstruction.
Initializing a three-dimensional space volume before reconstruction, wherein a uniform value is used as an initial value during first reconstruction; and then replacing the uniform initial value with the average value of the uniform initial value and the reconstruction result of the previous frame.
Reconstructing a spatial three-dimensional volume by using a Richardson Lucy algorithm;
s7: and judging whether the iteration upper limit of the Richardson Lucy algorithm is reached, if so, ending, otherwise, returning to the step S3, namely, repeating the steps S3 to S6 until the iteration upper limit of the Richardson Lucy algorithm is reached.
Referring to fig. 3, a scanning light field microscopic imaging system is set up to image histone in the nematode, and a space-time joint optimization reconstruction method based on the scanning light field microscopic imaging system is used for carrying out three-dimensional reconstruction on the histone. The results of the three-dimensional reconstruction with and without spatio-temporal joint optimization are compared in fig. 3. The left column represents three-dimensional reconstruction results without spatio-temporal joint optimization, and the right column represents three-dimensional reconstruction results with spatio-temporal joint optimization. It can be clearly seen that the spatio-temporal joint optimization reconstruction method based on the scanning light field microscopic imaging system effectively improves the imaging quality and greatly improves the imaging resolution and contrast.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.
Claims (9)
1. A spatio-temporal joint optimization reconstruction system, comprising:
the scanning light field acquisition module is used for converting the information of the sample into a light signal, converting the light signal into light field image data and acquiring the light field image data so as to obtain a scanning light field data stack;
the preprocessing module is used for preprocessing the light field image data, correcting the light field image data to accord with a physical model, and constructing an image stack according to a time sequence;
and the space-time joint bidirectional circulation reconstruction module is used for judging the data direction of the light field image, reading data according to the sliding window of the data direction of the light field image, determining time weight according to the sample motion rate, rearranging the pixels of the weighted light field image data, performing spatial up-sampling to obtain four-dimensional phase space data, and reconstructing a three-dimensional space volume by using a Richardson Lucy algorithm.
2. The spatio-temporal joint optimization reconstruction system according to claim 1, wherein the spatio-temporal joint bi-directional cyclic reconstruction module comprises:
the data direction judging unit is used for judging the data direction of the light field image, firstly determining the data input direction according to the sequence of time from front to back, and when the data is input into the last frame, reversing the time, wherein the data direction is from the last frame to the first frame;
a sliding window reading unit for reading data by sliding a window, and reading scan data corresponding to one volume in each sliding manner according to the data direction determined by the data direction determining unit;
a temporal weight interleaving unit for determining a temporal weight according to the sample motion rate;
the phase space data extraction unit is used for rearranging the pixels of the weighted light field image data, and performing spatial up-sampling to obtain four-dimensional phase space data;
the space three-dimensional initialization unit is used for initializing a space three-dimensional volume;
and the space three-dimensional reconstruction unit is used for reconstructing a space three-dimensional volume by using the Richardson Lucy algorithm.
3. The spatio-temporal joint optimization reconstruction system of claim 2, wherein the swept light field acquisition module comprises: the device comprises a sample, a scanning light field unit and an image sensor, wherein the information of the sample is converted into an optical signal, then light field image data with sub-pixel translation is generated through the scanning light field unit, and the light field image data is collected through the image sensor to obtain a scanning light field data stack.
4. The spatio-temporal joint optimization reconstruction system according to claim 3, wherein the preprocessing module comprises:
the data correction unit is used for correcting the light field image data to enable the light field image data to conform to a physical model of light field imaging;
and the scanning light field time sequencing unit is used for sequencing the corrected light field image data according to the time sequence.
5. The spatio-temporal joint optimization reconstruction system according to claim 4, wherein the data modification unit comprises:
the rotation subunit is used for rotating the light field image data based on an interpolation algorithm to ensure that the light field image data conforms to a physical model of light field imaging;
the cutting subunit is used for cutting the light field image data and removing redundant boundaries;
a translation subunit, configured to perform pixel translation or sub-pixel translation on the light-field image data.
6. The spatio-temporal joint optimization reconstruction system of claim 5, wherein the scan light field temporal ordering unit comprises:
the scanning image single-sheet separation subunit is used for separating the light field image data corresponding to one volume according to different scanning translation amounts;
and constructing a time sequence stack subunit, which is used for sequencing the two-dimensional data separated by the single scanning image separating subunit according to the acquisition time to form a time sequence stack.
7. The spatio-temporal joint optimization reconstruction system according to claim 6, wherein the temporal weight interleaving unit comprises:
a temporal weight calculation subunit for estimating a temporal weight from the motion rate of the sample;
a temporal weight assignment subunit for assigning temporal weights to the light field image data.
8. The spatio-temporal joint optimization reconstruction system according to claim 7, wherein the phase-space data extraction unit comprises:
a pixel rearrangement subunit, configured to convert the light field image data into four-dimensional phase space data;
and the up-sampling interpolation subunit is used for up-sampling the four-dimensional phase space data so as to be used for aperture fusion in three-dimensional reconstruction.
9. The space-time joint optimization reconstruction method is characterized by comprising the following steps:
s1: converting the information of a sample into an optical signal, converting the optical signal into optical field image data, acquiring the optical field image data, preprocessing the optical field image data, separating the optical field image data corresponding to a volume according to different scanning translation amounts, sequencing the separated two-dimensional data according to acquisition time, and constructing an image stack;
s2: judging the direction of the light field image data, determining the data input direction according to the sequence of the time sequence from the front to the back in the step S1, when the data is input to the last frame, reversing the time direction, wherein the data direction is from the last frame to the first frame, and inputting the current data stream format to the step S3;
s3: sliding the window to read data, and reading the data stream and the data direction determined in the step S2;
s4: determining a time weight according to the sample motion rate, and assigning the time weight to the light field image data;
s5: rearranging the pixels of the weighted light field image data, and performing spatial up-sampling interpolation to convert the weighted light field image data into four-dimensional phase spatial data;
s6: reconstructing the space three-dimensional volume by using a Richardson Lucy algorithm;
s7: and judging whether the iteration upper limit of the Richardson Lucy algorithm is reached, if so, ending, and otherwise, returning to the step S3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910875805.3A CN110706346B (en) | 2019-09-17 | 2019-09-17 | Space-time joint optimization reconstruction method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910875805.3A CN110706346B (en) | 2019-09-17 | 2019-09-17 | Space-time joint optimization reconstruction method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110706346A CN110706346A (en) | 2020-01-17 |
| CN110706346B true CN110706346B (en) | 2022-11-15 |
Family
ID=69195471
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910875805.3A Active CN110706346B (en) | 2019-09-17 | 2019-09-17 | Space-time joint optimization reconstruction method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110706346B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111258044B (en) * | 2020-02-18 | 2021-05-04 | 清华大学 | Mirror scanning light field-based 4 pi-like microscopic imaging method |
| CN113487658B (en) * | 2021-08-31 | 2021-11-16 | 清华大学 | Dynamic scene shooting method and device for scanning light field imaging system |
| CN114494383B (en) * | 2022-04-18 | 2022-09-02 | 清华大学 | Light field depth estimation method based on Richard-Lucy iteration |
| CN117876279B (en) * | 2024-03-11 | 2024-05-28 | 浙江荷湖科技有限公司 | Method and system for removing motion artifact based on scanned light field sequence image |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102509281A (en) * | 2011-11-24 | 2012-06-20 | 浙江大学 | Double-image planar motion blur eliminating method based on transparency |
| CN106530381A (en) * | 2016-10-19 | 2017-03-22 | 浙江大学 | GPU (graphic processing unit) acceleration-based deconvolution algorithm for three-dimensional fluorescence microscopic image |
| WO2018129715A1 (en) * | 2017-01-13 | 2018-07-19 | 浙江大学 | Simultaneous positioning and dense three-dimensional reconstruction method |
| CN108389171A (en) * | 2018-03-08 | 2018-08-10 | 深圳市唯特视科技有限公司 | A kind of light field deblurring and depth estimation method based on Combined estimator fuzzy variable |
| WO2019012658A1 (en) * | 2017-07-13 | 2019-01-17 | オリンパス株式会社 | Image processing device and light-field image pickup device |
-
2019
- 2019-09-17 CN CN201910875805.3A patent/CN110706346B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102509281A (en) * | 2011-11-24 | 2012-06-20 | 浙江大学 | Double-image planar motion blur eliminating method based on transparency |
| CN106530381A (en) * | 2016-10-19 | 2017-03-22 | 浙江大学 | GPU (graphic processing unit) acceleration-based deconvolution algorithm for three-dimensional fluorescence microscopic image |
| WO2018129715A1 (en) * | 2017-01-13 | 2018-07-19 | 浙江大学 | Simultaneous positioning and dense three-dimensional reconstruction method |
| WO2019012658A1 (en) * | 2017-07-13 | 2019-01-17 | オリンパス株式会社 | Image processing device and light-field image pickup device |
| CN108389171A (en) * | 2018-03-08 | 2018-08-10 | 深圳市唯特视科技有限公司 | A kind of light field deblurring and depth estimation method based on Combined estimator fuzzy variable |
Non-Patent Citations (2)
| Title |
|---|
| Scaled Heavy-Ball Acceleration of the Richardson-Lucy Algorithm for 3D Microscopy Image Restoration;Hongbin Wang等;《IEEE Transactions on Image Processing》;20131114;第23卷(第2期);第848-853页 * |
| 一种结合射影运动模糊模型的Richardson-Lucy算法;姜超等;《测绘科学技术学报》;20130615;第30卷(第03期);第288-292页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110706346A (en) | 2020-01-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110706346B (en) | Space-time joint optimization reconstruction method and system | |
| Dong et al. | Multi-scale boosted dehazing network with dense feature fusion | |
| CN109345449B (en) | Image super-resolution and non-uniform blur removing method based on fusion network | |
| Zou et al. | Learning to reconstruct high speed and high dynamic range videos from events | |
| CN110599400B (en) | EPI-based light field image super-resolution method | |
| Chi et al. | Dynamic low-light imaging with quanta image sensors | |
| US20160381369A1 (en) | Image capturing system, image capturing apparatus, decoding apparatus, image capturing method, and display control system | |
| CN113222825B (en) | Infrared image super-resolution reconstruction method based on visible light image training and application | |
| WO2022016350A1 (en) | Light field image processing method, light field image encoder and decoder, and storage medium | |
| Thawakar et al. | Image and video super resolution using recurrent generative adversarial network | |
| CN103268599A (en) | Multi-linear-array charge coupled device (CCD) sub-pixel staggered imaging super-resolution reconstruction method | |
| CN114841856A (en) | Image super-pixel reconstruction method of dense connection network based on depth residual channel space attention | |
| CN108989731A (en) | A method of improving video spatial resolution | |
| Chi et al. | HDR imaging with spatially varying signal-to-noise ratios | |
| Wei et al. | Efficient video compressed sensing reconstruction via exploiting spatial-temporal correlation with measurement constraint | |
| Wagner et al. | Image superresolution for improved automatic target recognition | |
| CN110766612A (en) | Image super-resolution method based on deep learning | |
| Shedligeri et al. | A unified framework for compressive video recovery from coded exposure techniques | |
| CN107123100B (en) | FiDrizzle multi-sampling image reconstruction technology | |
| CN112348745B (en) | Video super-resolution reconstruction method based on residual convolutional network | |
| CN116091337A (en) | Image enhancement method and device based on event signal nerve coding mode | |
| Seyid et al. | Hardware implementation of real-time multiple frame super-resolution | |
| Li et al. | H-vfi: Hierarchical frame interpolation for videos with large motions | |
| Darwish et al. | Event-driven image sensor application: event-driven image segmentation | |
| CN111246213B (en) | Video compressed sensing sampling rate self-adaptive hierarchical block matching reconstruction method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20210618 Address after: Room 301, building 3, 1390 cangxing street, Cangqian street, Yuhang District, Hangzhou City, Zhejiang Province Applicant after: Zhejiang Hehu Technology Co.,Ltd. Address before: 100083 202 / F, block F, Zhizao street, Zhongguancun, 45 Chengfu Road, Haidian District, Beijing Applicant before: Beijing Youke Nuclear Power Technology Development Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |