CN114092329A - Super-resolution fluorescence microscopic imaging method based on sub-pixel neural network - Google Patents
Super-resolution fluorescence microscopic imaging method based on sub-pixel neural network Download PDFInfo
- Publication number
- CN114092329A CN114092329A CN202111376143.9A CN202111376143A CN114092329A CN 114092329 A CN114092329 A CN 114092329A CN 202111376143 A CN202111376143 A CN 202111376143A CN 114092329 A CN114092329 A CN 114092329A
- Authority
- CN
- China
- Prior art keywords
- fluorescence
- pixel
- resolution
- image
- imaging
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 42
- 238000000034 method Methods 0.000 title claims abstract description 36
- 238000000799 fluorescence microscopy Methods 0.000 title claims abstract description 28
- 238000003384 imaging method Methods 0.000 claims abstract description 70
- 239000007850 fluorescent dye Substances 0.000 claims abstract description 39
- 238000002073 fluorescence micrograph Methods 0.000 claims abstract description 33
- 238000012549 training Methods 0.000 claims description 46
- 238000004088 simulation Methods 0.000 claims description 34
- 230000006870 function Effects 0.000 claims description 19
- 239000000523 sample Substances 0.000 claims description 19
- 238000010857 super resolution fluorescence microscopy Methods 0.000 claims description 8
- 238000005457 optimization Methods 0.000 claims description 6
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 3
- 238000002474 experimental method Methods 0.000 abstract description 3
- 238000004364 calculation method Methods 0.000 abstract 1
- 238000010586 diagram Methods 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000008034 disappearance Effects 0.000 description 3
- 238000004880 explosion Methods 0.000 description 3
- 238000000386 microscopy Methods 0.000 description 3
- 238000003062 neural network model Methods 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000004807 localization Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000003213 activating effect Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000012550 audit Methods 0.000 description 1
- 238000004061 bleaching Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Microscoopes, Condenser (AREA)
- Investigating, Analyzing Materials By Fluorescence Or Luminescence (AREA)
Abstract
The invention provides a super-resolution fluorescence microscopic imaging method based on a sub-pixel neural network, which is used for carrying out super-resolution fluorescence positioning microscopic imaging and comprises the following steps: acquiring a fluorescence image sequence of an object to be imaged; positioning each frame of fluorescence image in the fluorescence image sequence based on the ultrahigh resolution imaging model to obtain a positioning result corresponding to each frame of fluorescence image; and (5) taking a superposition result obtained by superposing all the positioning results as an ultrahigh resolution positioning microimaging image of the object to be imaged. According to the ultrahigh-resolution fluorescence positioning microscopic imaging method, in the fluorescent probe positioning process, as long as the low-resolution fluorescence microscopic image obtained by an experiment is input into the ultrahigh-resolution imaging model, an accurate positioning result under the condition of a high-density fluorescent probe can be obtained, any additional operation or manual parameter adjustment is not needed, and the calculation complexity is reduced and the parameter dependence is avoided while the rapid ultrahigh-resolution fluorescence positioning microscopic imaging is realized.
Description
Technical Field
The invention belongs to the technical field of image analysis, and relates to a super-resolution fluorescence microscopic imaging method based on a sub-pixel neural network.
Background
Ultra-high resolution fluorescence Localization Microscopy techniques have been proposed and rapidly developed, such as Stochastic Optical Reconstruction Microscopy (STORM) and Photoactivated Localization Microscopy (PALM). By using the optical switchable probe combined positioning method, the STORM and the PALM can break through the optical diffraction limit, and the spatial resolution of the traditional fluorescence microscopic imaging can be improved by one order of magnitude.
However, ultra-high resolution imaging using STORM and PALM remains extremely challenging. The main limiting factor is poor temporal resolution during imaging in order to maintain the required ultra-high spatial resolution. To overcome this limitation, one possible approach is to increase the density of fluorescent probes that can be activated in each frame, so that more fluorescent probes can be located in each frame of fluorescent image, thereby reducing the number of frames of images required to complete ultra-high resolution imaging, and further improving the imaging time resolution. However, the probability of overlap between high-density fluorescent probes is large. For the traditional single probe positioning method, the real position of each fluorescent probe is difficult to accurately distinguish, and the imaging spatial resolution is influenced.
Disclosure of Invention
In order to solve the problems, the invention provides a super-resolution fluorescence microscopic imaging method based on a sub-pixel neural network, which improves the positioning accuracy, and adopts the following technical scheme:
the invention provides a super-resolution fluorescence microscopic imaging method based on a sub-pixel neural network, which comprises the following steps: step S1, acquiring a fluorescence image sequence of an object to be imaged; step S2, positioning each frame of fluorescence image in the fluorescence image sequence based on the super-high resolution imaging model to obtain a positioning result corresponding to each frame of fluorescence image; step S3, the superposition result obtained by superposing all the positioning results is used as the super-high resolution positioning microscopic imaging image of the object to be imaged, wherein, the super-high resolution imaging model is obtained by taking simulation training data as input and training the sub-pixel convolution neural network by using a loss function, and the simulation training data is obtained by the following steps: step S2-1, based on the fluorescence microscopic imaging system, imaging a single fluorescence probe in the imaging area of the original image for multiple times, and calculating the average half height FHWM of the single fluorescence probe in the multiple imaging result; step S2-2, calculating to obtain a standard deviation delta based on the average half height FHWM:
step S2-3, obtaining a point spread function model I (x, y) based on the standard deviation delta:
wherein I (x, y) is the intensity of the fluorescent probe at (x, y), (x)0,y0) As the true position of the fluorescent probe, I0To correct the amplitude; step S2-4, generating a plurality of fluorescent probes in random distribution in a grid area of a first preset pixel, thereby forming an image containing the plurality of fluorescent probes, and sampling random spatial positions corresponding to the fluorescent probes in the image containing the plurality of fluorescent probes in sequence to obtain sampled spatial positions; s2-5, performing convolution on an impulse function at the real position of the fluorescent probe and a point spread function model I (x, y) by using a forward model so as to simulate a microscopic imaging process and obtain a noiseless fluorescent microscopic simulation image; and step S2-6, adding Poisson noises with different signal-to-noise ratios to the noiseless fluorescence microscopic simulation image to obtain a noisy fluorescence microscopic simulation image, taking the noisy fluorescence microscopic simulation image as simulation training data, and mapping the sampled spatial position to a grid area of a second preset pixel to be used as a training label.
The super-resolution fluorescence microscopic imaging method based on the sub-pixel neural network provided by the invention can also have the characteristic that a fluorescence image sequence is obtained by performing microscopic imaging on an object to be imaged based on a fluorescence microscopic imaging system under the intervention of a light switchable fluorescence probe, each frame of fluorescence image comprises a plurality of fluorescence probes, and each fluorescence probe is randomly distributed in an imaging area.
The invention also provides a super-resolution fluorescence microscopic imaging method based on the sub-pixel neural network, which has the characteristics that the sub-pixel convolution neural network comprises 13 convolution layers, and the 1 st convolution layer is used for extracting the shallow feature of the image; the 2 nd convolutional layer to the 11 th convolutional layer are residual error structures and are used for local residual error learning; the 12 th convolutional layer is connected with the 1 st convolutional layer and used for global residual learning; the 13 th convolutional layer is used to implement the upsampling operation.
The super-resolution fluorescence microscopic imaging method based on the sub-pixel neural network provided by the invention can also have the characteristic that the loss functions are MSE and L1Combination of regularization:
wherein, y is a training label,
is a predicted output image of the sub-pixel convolution neural network, N is the data amount per batch processing, g is a 2D Gaussian kernel,
is a convolution operation.
The super-resolution fluorescence microscopic imaging method based on the sub-pixel neural network provided by the invention can also have the characteristics that the training is as follows: and taking a preset amount of simulation training data as input, comparing a prediction output result and a real output result which are taken as training labels, and performing traversal training on the sub-pixel convolution neural network for a preset number of times by adopting an Adam optimization algorithm, wherein in each traversal training, the value of N is 4, and the Gaussian kernel g has a standard deviation of 1 pixel.
The invention also provides a super-resolution fluorescence microscopic imaging method based on the sub-pixel neural network, which is characterized in that the first predetermined pixel is 32 × 32, and the second predetermined pixel is 128 × 128.
Action and Effect of the invention
According to the super-resolution fluorescence microscopic imaging method based on the sub-pixel neural network, each frame of fluorescence image in a fluorescence image sequence is firstly positioned through a super-resolution imaging model obtained through the trained sub-pixel convolution neural network, and then positioning results corresponding to each frame of fluorescence image are superposed to obtain a super-resolution positioning microscopic imaging image. Because the 2 nd convolutional layer to the 11 th convolutional layer of the sub-pixel convolutional neural network model are residual error structures, namely jump connection is added on the basis of the sub-pixel convolutional neural network, 5 residual error modules are constructed (each two layers in the residual error structures are 1 residual error module), therefore, the training error can be converged to be close to a smaller value, the problems of gradient disappearance, gradient explosion and the like are relieved, and a deeper network can be trained.
The super-resolution fluorescence microscopic imaging method based on the sub-pixel neural network combines the deep learning technology (namely, the sub-pixel convolution neural network model) with the super-resolution fluorescence positioning microscopic imaging, can obtain an accurate positioning result under the condition of a high-density fluorescence probe as long as a low-resolution fluorescence microscopic image obtained by an experiment is input into the sub-pixel convolution neural network model in the fluorescent probe positioning process, does not need any additional operation or manual parameter adjustment, reduces the computational complexity and avoids the parameter dependence while realizing the rapid super-resolution fluorescence positioning microscopic imaging.
Drawings
FIG. 1 is a flow chart of a sub-pixel neural network-based super-resolution fluorescence microscopy imaging method according to an embodiment of the present invention;
FIG. 2 is an exemplary illustration of a fluorescence image according to an embodiment of the invention;
FIG. 3 is a flow chart of generating simulated training data in an embodiment of the present invention;
FIG. 4 is a comparison graph of the positioning result graph of the fluorescent probe and a low-resolution fluorescent microscopic image generated by simulation according to the embodiment of the present invention;
FIG. 5 is a schematic diagram of super-resolution fluorescence microscopy imaging based on a sub-pixel neural network in an embodiment of the present invention.
Detailed Description
In order to make the technical means, the creation features, the achievement purposes and the effects of the invention easy to understand, the following describes a sub-pixel neural network-based super-resolution fluorescence microscopic imaging method of the invention in detail with reference to the embodiments and the accompanying drawings.
< example >
Fig. 1 is a flowchart of a super-resolution fluorescence microscopy imaging method based on a sub-pixel neural network according to an embodiment of the invention.
As shown in fig. 1, a super-resolution fluorescence microscopic imaging method based on a sub-pixel neural network comprises the following steps:
step S1, a sequence of fluorescence images of the object to be imaged is acquired.
FIG. 2 is an exemplary fluorescence image of an embodiment of the invention.
In this embodiment, the fluorescence image sequence is obtained by performing microscopic imaging on the object to be imaged based on a fluorescence microscopic imaging system under the intervention of the optical switchable fluorescence probe, wherein each frame of fluorescence image includes a plurality of fluorescence probes, as shown in fig. 2, and each fluorescence probe is randomly distributed in the imaging region.
The fluorescent image sequence is obtained by the following steps:
and step S1-1, randomly activating a plurality of fluorescent probes in all the fluorescent probes in each microscopic imaging process, and simulating to perform fluorescent microscopic imaging so as to obtain a simulated image.
And step S1-2, randomly changing the position of the fluorescent probe which can be activated in all the fluorescent probes, and repeating the step S1-1 to perform fluorescence microscopy imaging again for a preset number of times so as to obtain final simulation imaging.
In order to simulate the continuous activation-bleaching (bleaching-activation) alternating process of the fluorescent probe in the imaging area, the imaging process is repeated 1000 times, and 1000 frames of original low-resolution fluorescence microscopic images (namely final simulation imaging) are obtained.
And step S1-3, after all final simulation imaging is obtained, adding background noise to the generated original image for simulating the influence of noise in an actual imaging experiment, and taking all final simulation imaging after noise addition as a fluorescence image sequence of the object to be imaged.
And step S2, positioning each frame of fluorescence image in the fluorescence image sequence based on the ultrahigh resolution imaging model to obtain a positioning result corresponding to each frame of fluorescence image.
In this embodiment, the super-high resolution imaging model in step S2 is obtained by training the sub-pixel convolution neural network with the loss function and simulation training data as input.
Considering that a deeper neural network is difficult to train due to gradient disappearance or gradient explosion, in order to overcome this limitation, the embodiment adds a jump connection on the basis of the sub-pixel convolution neural network, and constructs 5 residual modules, so that a training error can converge to a smaller value. Specifically, the method comprises the following steps:
the sub-pixel convolution neural network comprises 13 convolution layers in total, wherein the 1 st convolution layer is used for extracting shallow layer characteristics of the image; the 2 nd convolutional layer to the 11 th convolutional layer are residual error structures (i.e. containing 5 residual error modules, each residual error module containing 2 convolutional layers) and are used for local residual error learning; the 12 th convolutional layer is connected with the 1 st convolutional layer and used for global residual learning; the 13 th convolutional layer is used to implement the upsampling operation, and in this embodiment, the upsampling factor used is 4.
Since the use of the Mean Square Error (MSE) loss function commonly used in image processing usually results in output results that are too smooth to be suitable for the task of accurate positioning of fluorescent probes. Therefore, in this embodiment, the loss functions in the training process are MSE and L1Combination of regularization:
wherein y is the trainingThe number of the labels is such that,
is a predicted output image of a sub-pixel convolutional neural network, N is the number of data processed per batch, g is a 2D Gaussian kernel,
is a convolution operation.
In order to reduce the learning rate when the training error is stable and make the error converge to a smaller value, the embodiment trains the sub-pixel convolution neural network by using a learning rate attenuation strategy, wherein simulation training data is used as input, a training label is used as a predicted output result to be compared with a real output result (10000 pairs of simulation training data and corresponding training labels are used in the embodiment), an Adam optimization algorithm sub-pixel volume and an audit network are used to perform traversal training for 60 cycles (namely, the simulation training data is traversed for 60 times), wherein in each optimization traversal training, the value of the number N of images is 4, the gaussian kernel g has a standard deviation of 1 pixel, and the initial learning rate is 0.001.
Wherein the simulated training data for training is generated based on the estimated point spread function of the fluorescence microscopy imaging system.
FIG. 3 is a flowchart of generating simulated training data according to an embodiment of the present invention.
As shown in fig. 3, the generation of the simulation training data includes the following steps:
step S2-1, based on the fluorescence microscopic imaging system, imaging a single fluorescence probe in the imaging area of the original image for multiple times, and calculating the average half height FHWM of the single fluorescence probe in the multiple imaging result;
step S2-2, calculating to obtain a standard deviation delta based on the average half height FHWM:
step S2-3, obtaining a point spread function model I (x, y) based on the standard deviation delta:
wherein I (x, y) is the intensity of the fluorescent probe at the imaging position (x, y) of the fluorescent probe, (x0,y0) As the true position of the fluorescent probe, I0To correct the amplitude;
step S2-4, generating a plurality of fluorescent probes in random distribution in the grid area of the first preset pixel, thereby forming an image containing the plurality of fluorescent probes, and sequentially sampling random spatial positions corresponding to the fluorescent probes in the image containing the plurality of fluorescent probes to obtain sampled spatial positions;
s2-5, performing convolution on an impulse function at the real position of the fluorescent probe and a point spread function model I (x, y) by using a forward model so as to simulate a microscopic imaging process and obtain a noiseless fluorescent microscopic simulation image;
step S2-6, adding Poisson noises (such as 6, 8, 10, 12dB) with different signal-to-noise ratios to the noiseless fluorescence microscopic simulation image to obtain a noisy fluorescence microscopic simulation image, taking the noisy fluorescence microscopic simulation image as simulation training data, and mapping the sampled spatial position to a grid area of a second preset pixel to be used as a training label.
In this embodiment, the first predetermined pixels are 32 × 32 pixels, and the second predetermined pixels are 128 × 128 pixels.
And step S3, using the superposition result obtained by superposing all the positioning results as the ultra-high resolution positioning microscopic imaging picture of the object to be imaged.
FIG. 4 is a comparison graph of the positioning result of the fluorescent probe and a low-resolution fluorescence microscopic image generated by simulation according to the embodiment of the present invention.
As shown in fig. 4, the first column in the figure is a low-resolution fluorescence microscopic image generated by simulation, and the image shows that the fluorescence position is fuzzy and the resolution is low; the second column in the figure is a schematic diagram of the positioning result of the fluorescent probe obtained based on the method of the embodiment, and the position of the fluorescent probe shown in the figure is clear and has ultrahigh resolution.
FIG. 5 is a schematic diagram of super-resolution fluorescence microscopy imaging based on a sub-pixel neural network in an embodiment of the present invention.
As shown in fig. 5, the left side of the diagram is the low-resolution original fluorescence microscopic image (i.e., the fluorescence image in the fluorescence image sequence in step S1), and the right side of the diagram is the super-resolution positioning microscopic imaging image obtained by the super-resolution fluorescence microscopic imaging method based on the sub-pixel neural network according to the present embodiment, so that it can be seen that the spatial resolution of the imaging can be effectively improved by the super-resolution fluorescence microscopic imaging method based on the sub-pixel neural network according to the present embodiment, and the super-resolution fluorescence positioning microscopic imaging can be realized.
Effects and effects of the embodiments
According to the super-resolution fluorescence microscopic imaging method based on the sub-pixel neural network provided by the embodiment, each frame of fluorescence image in a fluorescence image sequence is positioned by a super-resolution imaging model obtained by the trained sub-pixel convolution neural network, and then positioning results corresponding to each frame of fluorescence image are superposed to obtain a super-resolution positioning microscopic imaging image. In the embodiment, a deep learning technology (namely a sub-pixel convolution neural network model) is combined with the super-resolution fluorescence positioning microscopic imaging, the accurate positioning of the fluorescence probe in the original low-resolution image is realized through the training model, the spatial resolution of the imaging is effectively improved, and the super-resolution fluorescence positioning microscopic imaging is further realized.
In an embodiment, the loss function in the training process of the neural network due to the sub-pixel convolution is MSE and L1Due to the combination of regularization, image detail information can be well reserved, and the defect that when a model is trained by only adopting a Mean Square Error (MSE) loss function, a reconstructed image is too smooth and cannot well reserve the image detail information is overcome.
In the embodiment, the 2 nd convolutional layer to the 11 th convolutional layer are residual error structures, that is, jump connection is added on the basis of the sub-pixel convolutional neural network, and 5 residual error modules are constructed (each two layers in the residual error structure are 1 residual error module), so that the training error can be converged to a smaller value, the problems of gradient disappearance or gradient explosion and the like are solved, and a deeper network can be trained.
In addition, in the embodiment, because the Poisson noises with different signal-to-noise ratios are added in the simulation training data, the ultrahigh-resolution imaging model has better robustness.
The above-described embodiments are merely illustrative of specific embodiments of the present invention, and the present invention is not limited to the description of the above-described embodiments.
Claims (6)
1. A super-resolution fluorescence microscopic imaging method based on a sub-pixel neural network is characterized by comprising the following steps:
step S1, acquiring a fluorescence image sequence of an object to be imaged;
step S2, positioning each frame of fluorescence image in the fluorescence image sequence based on a super-resolution imaging model to obtain a positioning result corresponding to each frame of fluorescence image;
step S3, using the superposition result obtained by superposing all the positioning results as the ultra-high resolution positioning microscopic imaging picture of the object to be imaged,
wherein the ultra-high resolution imaging model is obtained by taking simulation training data as input and training a sub-pixel convolution neural network by using a loss function,
the simulation training data is obtained through the following steps:
step S2-1, based on a fluorescence microscopic imaging system, imaging a single fluorescence probe in an imaging area of an original image for multiple times, and calculating the average half height FHWM of the single fluorescence probe in multiple imaging results;
step S2-2, calculating to obtain a standard deviation delta based on the average half height FHWM:
step S2-3, obtaining a point spread function model I (x, y) based on the standard deviation delta:
wherein I (x, y) is the intensity of the fluorescent probe at (x, y), (x0,y0) As the true position of the fluorescent probe, I0To correct the amplitude;
step S2-4, generating a plurality of fluorescent probes distributed randomly in a grid area of a first predetermined pixel, thereby forming an image containing the plurality of fluorescent probes, and sequentially sampling random spatial positions corresponding to the fluorescent probes in the image containing the plurality of fluorescent probes to obtain sampled spatial positions;
step S2-5, performing convolution on the impulse function at the real position of the fluorescent probe and the point spread function model I (x, y) by using a forward model so as to simulate a microscopic imaging process, and obtaining the noiseless fluorescent microscopic simulation image;
and step S2-6, adding Poisson noises with different signal-to-noise ratios to the noiseless fluorescence microscopic simulation image to obtain a noisy fluorescence microscopic simulation image, taking the noisy fluorescence microscopic simulation image as simulation training data, and mapping the sampled spatial position to a grid area of a second preset pixel to be used as a training label.
2. The super-resolution fluorescence microscopy imaging method based on the sub-pixel neural network is characterized in that:
wherein the sequence of fluorescence images is obtained by microscopic imaging of the object to be imaged based on the fluorescence microscopic imaging system with the intervention of an optically switchable fluorescence probe,
each frame of the fluorescence image comprises a plurality of fluorescence probes, and each fluorescence probe is randomly distributed in an imaging area.
3. The super-resolution fluorescence microscopy imaging method based on the sub-pixel neural network is characterized in that:
the sub-pixel convolution neural network comprises 13 convolution layers, wherein the 1 st convolution layer is used for extracting shallow features of an image;
the 2 nd convolutional layer to the 11 th convolutional layer are residual error structures and are used for local residual error learning;
the 12 th convolutional layer is connected with the 1 st convolutional layer and used for global residual learning;
the 13 th convolutional layer is used to implement the upsampling operation.
4. The super-resolution fluorescence microscopy imaging method based on the sub-pixel neural network is characterized in that:
wherein the loss functions are MSE and L1Combination of regularization:
wherein, y is a training label,
n is the number of data processed per batch, g is a 2D gaussian kernel,
is a convolution operation.
5. The method of claim 4, wherein the method comprises the following steps:
wherein the training is:
taking a preset amount of the simulation training data as input, comparing the training labels as a prediction output result with a real output result, performing optimization traversal training on the sub-pixel convolution neural network for a preset number of times by adopting an Adam optimization algorithm,
in each optimization traversal training, the value of N is 4, and the Gaussian kernel g has the standard deviation of 1 pixel.
6. The super-resolution fluorescence microscopy imaging method based on the sub-pixel neural network is characterized in that:
wherein the first predetermined pixel is 32 × 32, and the second predetermined pixel is 128 × 128.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111376143.9A CN114092329B (en) | 2021-11-19 | 2021-11-19 | Super-resolution fluorescence microscopic imaging method based on sub-pixel neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111376143.9A CN114092329B (en) | 2021-11-19 | 2021-11-19 | Super-resolution fluorescence microscopic imaging method based on sub-pixel neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114092329A true CN114092329A (en) | 2022-02-25 |
| CN114092329B CN114092329B (en) | 2023-01-20 |
Family
ID=80302263
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111376143.9A Active CN114092329B (en) | 2021-11-19 | 2021-11-19 | Super-resolution fluorescence microscopic imaging method based on sub-pixel neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114092329B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115100033A (en) * | 2022-05-20 | 2022-09-23 | 浙江大学 | Fluorescence microscopic image super-resolution reconstruction method and device and computing equipment |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110599399A (en) * | 2019-07-26 | 2019-12-20 | 清华大学 | Fast two-photon imaging method and device based on convolutional neural network |
| EP3598474A1 (en) * | 2018-07-19 | 2020-01-22 | FEI Company | Adaptive specimen image acquisition using an artificial neural network |
| CN111052173A (en) * | 2017-07-31 | 2020-04-21 | 巴斯德研究所 | Method, apparatus and computer program for improving reconstruction of dense super-resolution images from diffraction limited images acquired from single molecule positioning microscopy |
| EP3676750A1 (en) * | 2017-09-01 | 2020-07-08 | Ludwig-Maximilians-Universität München | Detection of biological cells and tracing of cell lineage |
| CN111524064A (en) * | 2020-03-11 | 2020-08-11 | 浙江大学 | Fluorescence microscopic image super-resolution reconstruction method based on deep learning |
| EP3712618A1 (en) * | 2019-03-18 | 2020-09-23 | Euroimmun Medizinische Labordiagnostika AG | Method for detecting a binding of antibodies of a patient sample to double-stranded dna using crithidia luciliae cells and fluorescence microscopy |
| CN112419192A (en) * | 2020-11-24 | 2021-02-26 | 北京航空航天大学 | Convolutional neural network-based ISMS image restoration and super-resolution reconstruction method and device |
| CN112548326A (en) * | 2020-11-17 | 2021-03-26 | 华东师范大学重庆研究院 | Ridge type surface 5G chip femtosecond intelligent processing technology and equipment based on deep learning |
| CN113296259A (en) * | 2021-05-25 | 2021-08-24 | 中国科学院国家天文台南京天文光学技术研究所 | Super-resolution imaging method and device based on aperture modulation subsystem and deep learning |
| WO2021188839A1 (en) * | 2020-03-20 | 2021-09-23 | The Regents Of The University Of California | Single-shot autofocusing of microscopy images using deep learning |
-
2021
- 2021-11-19 CN CN202111376143.9A patent/CN114092329B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111052173A (en) * | 2017-07-31 | 2020-04-21 | 巴斯德研究所 | Method, apparatus and computer program for improving reconstruction of dense super-resolution images from diffraction limited images acquired from single molecule positioning microscopy |
| EP3676750A1 (en) * | 2017-09-01 | 2020-07-08 | Ludwig-Maximilians-Universität München | Detection of biological cells and tracing of cell lineage |
| EP3598474A1 (en) * | 2018-07-19 | 2020-01-22 | FEI Company | Adaptive specimen image acquisition using an artificial neural network |
| EP3712618A1 (en) * | 2019-03-18 | 2020-09-23 | Euroimmun Medizinische Labordiagnostika AG | Method for detecting a binding of antibodies of a patient sample to double-stranded dna using crithidia luciliae cells and fluorescence microscopy |
| CN110599399A (en) * | 2019-07-26 | 2019-12-20 | 清华大学 | Fast two-photon imaging method and device based on convolutional neural network |
| CN111524064A (en) * | 2020-03-11 | 2020-08-11 | 浙江大学 | Fluorescence microscopic image super-resolution reconstruction method based on deep learning |
| WO2021188839A1 (en) * | 2020-03-20 | 2021-09-23 | The Regents Of The University Of California | Single-shot autofocusing of microscopy images using deep learning |
| CN112548326A (en) * | 2020-11-17 | 2021-03-26 | 华东师范大学重庆研究院 | Ridge type surface 5G chip femtosecond intelligent processing technology and equipment based on deep learning |
| CN112419192A (en) * | 2020-11-24 | 2021-02-26 | 北京航空航天大学 | Convolutional neural network-based ISMS image restoration and super-resolution reconstruction method and device |
| CN113296259A (en) * | 2021-05-25 | 2021-08-24 | 中国科学院国家天文台南京天文光学技术研究所 | Super-resolution imaging method and device based on aperture modulation subsystem and deep learning |
Non-Patent Citations (3)
| Title |
|---|
| 李浩宇: "基于深度学习的荧光显微成像技术及应用", 《激光与光电子学进展》 * |
| 肖康: "基于深度学习的低信噪比下的快速超分辨荧光显微成像", 《中国激光》 * |
| 陈奕灿: "基于深度学习的傅里叶叠层显微成像", 《激光与光电子学进展》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115100033A (en) * | 2022-05-20 | 2022-09-23 | 浙江大学 | Fluorescence microscopic image super-resolution reconstruction method and device and computing equipment |
| CN115100033B (en) * | 2022-05-20 | 2023-09-08 | 浙江大学 | Fluorescent microscopic image super-resolution reconstruction method and device and computing equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114092329B (en) | 2023-01-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111754403B (en) | Image super-resolution reconstruction method based on residual learning | |
| CN111062872B (en) | Image super-resolution reconstruction method and system based on edge detection | |
| CN109509152B (en) | Image super-resolution reconstruction method for generating countermeasure network based on feature fusion | |
| CN110310227B (en) | Image super-resolution reconstruction method based on high-low frequency information decomposition | |
| CN108550115B (en) | Image super-resolution reconstruction method | |
| CN106846463B (en) | Microscopic image three-dimensional reconstruction method and system based on deep learning neural network | |
| CN109727195B (en) | Image super-resolution reconstruction method | |
| CN109214989A (en) | Single image super resolution ratio reconstruction method based on Orientation Features prediction priori | |
| CN110675321A (en) | Super-resolution image reconstruction method based on progressive depth residual error network | |
| CN113012172A (en) | AS-UNet-based medical image segmentation method and system | |
| CN113160234A (en) | Unsupervised remote sensing image semantic segmentation method based on super-resolution and domain self-adaptation | |
| CN114022506B (en) | Image restoration method for edge prior fusion multi-head attention mechanism | |
| CN116029902A (en) | Knowledge distillation-based unsupervised real world image super-resolution method | |
| CN112435305A (en) | Ultra-high resolution ultrasonic imaging method based on deep learning | |
| CN114092329B (en) | Super-resolution fluorescence microscopic imaging method based on sub-pixel neural network | |
| CN113344793A (en) | Image super-resolution reconstruction method, device, equipment and storage medium | |
| CN112163998A (en) | Single-image super-resolution analysis method matched with natural degradation conditions | |
| CN115484410A (en) | Event camera video reconstruction method based on deep learning | |
| CN115880158A (en) | Blind image super-resolution reconstruction method and system based on variational self-coding | |
| CN114972332A (en) | Bamboo laminated wood crack detection method based on image super-resolution reconstruction network | |
| CN114549318A (en) | Ultra-high resolution fluorescence microscopic imaging method based on sub-voxel convolution neural network | |
| CN113160057B (en) | RPGAN image super-resolution reconstruction method based on generation countermeasure network | |
| CN114332625A (en) | Remote sensing image colorizing and super-resolution method and system based on neural network | |
| CN111667407B (en) | Image super-resolution method guided by depth information | |
| CN117195960A (en) | Atmospheric turbulence wave front detection method based on shack Hartmann UT conversion model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20240318 Address after: Room 603, No. 10, Lane 999, Dangui Road, China (Shanghai) Pilot Free Trade Zone, Pudong New Area, Shanghai, 2012 (property registration floor is 5 floors) Patentee after: Shanghai Changsheng Medical Technology Co.,Ltd. Country or region after: China Address before: 200433 No. 220, Handan Road, Shanghai, Yangpu District Patentee before: FUDAN University Country or region before: China |