CN113724150A - Structured light microscopic reconstruction method and device without high signal-to-noise ratio true value image - Google Patents
Structured light microscopic reconstruction method and device without high signal-to-noise ratio true value image Download PDFInfo
- Publication number
- CN113724150A CN113724150A CN202110852923.XA CN202110852923A CN113724150A CN 113724150 A CN113724150 A CN 113724150A CN 202110852923 A CN202110852923 A CN 202110852923A CN 113724150 A CN113724150 A CN 113724150A
- Authority
- CN
- China
- Prior art keywords
- neural network
- image
- data
- trained
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 56
- 238000013528 artificial neural network Methods 0.000 claims abstract description 142
- 238000012549 training Methods 0.000 claims abstract description 101
- 239000012472 biological sample Substances 0.000 claims abstract description 72
- 238000012545 processing Methods 0.000 claims abstract description 31
- 239000000523 sample Substances 0.000 claims description 39
- 230000006870 function Effects 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 12
- 238000002474 experimental method Methods 0.000 abstract description 10
- 238000005516 engineering process Methods 0.000 abstract description 4
- 238000010801 machine learning Methods 0.000 abstract description 4
- 238000005286 illumination Methods 0.000 description 11
- 238000000386 microscopy Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 238000000799 fluorescence microscopy Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 238000001228 spectrum Methods 0.000 description 2
- 206010034972 Photosensitivity reaction Diseases 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000002547 anomalous effect Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 230000005284 excitation Effects 0.000 description 1
- 239000007850 fluorescent dye Substances 0.000 description 1
- 102000034287 fluorescent proteins Human genes 0.000 description 1
- 108091006047 fluorescent proteins Proteins 0.000 description 1
- 238000010166 immunofluorescence Methods 0.000 description 1
- 229920002521 macromolecule Polymers 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 210000000056 organ Anatomy 0.000 description 1
- 208000007578 phototoxic dermatitis Diseases 0.000 description 1
- 231100000018 phototoxicity Toxicity 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G06T5/70—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
The utility model provides a need not the micro-reconstruction of structured light and device of high SNR truth value image, relates to structured light technical field and machine learning field, and this scheme includes: acquiring a biological sample image to be processed, inputting the biological sample image to be processed into a pre-trained generator neural network for reconstruction processing, and obtaining a reconstructed image of the biological sample image to be processed; the pre-trained denoising method comprises the steps of adding a pre-trained denoising neural network for reserving structural light striations in a pre-trained generator neural network, wherein training data of the pre-trained denoising neural network are common samples. The pre-trained denoising neural network in the scheme can be separated from a specific biological sample to carry out independent training, can be directly applied to the biological sample, and carries out denoising processing on input biological sample real acquisition data containing noise. Therefore, the method and the device have the advantages that the structured light microscopic reconstruction without high signal-to-noise ratio data is realized, the experiment cost is reduced, and the application range of the structured light microscopic technology is remarkably expanded.
Description
Technical Field
The application relates to the technical field of structured light and the field of machine learning, in particular to a structured light micro-reconstruction method and a device without a high signal-to-noise ratio true value image.
Background
Fluorescence microscopy is an important tool in life science research, and by utilizing operations such as immunofluorescence reaction, fluorescent dyes or fluorescent proteins can be specifically combined with a target structure, so that the finally acquired image only contains the biological structure which is interested by people, and the specific imaging of the marked structure with high signal-to-noise ratio is realized. However, the resolution of the common fluorescence microscopy is limited, and the structured light microscopy can realize 2-fold improvement of the resolution by using about 10 wide-field excitation images containing the moire fringes, and is very suitable for living cell imaging due to the characteristics of high speed and low phototoxicity.
With the development of the field of machine learning in recent years, the academic and industrial fields have tried to use a deep neural network for the structured light reconstruction. Before training the deep neural network, the original data with high signal-to-noise ratio of the sample needs to be obtained as a true value image for network training, however, in an actual biological experiment, the obtaining of the data with high signal-to-noise ratio is higher in both experiment difficulty and experiment cost.
Disclosure of Invention
The present application is directed to solving, at least to some extent, one of the technical problems in the related art.
Therefore, a first objective of the present application is to provide a structured light microscopic reconstruction method without a true value image with a high snr, so as to solve the technical problem that in an actual biological experiment, the acquisition of data with a high snr is high in both experimental difficulty and experimental cost.
A second object of the present application is to provide a structured light reconstruction apparatus that does not require a true-value image with a high snr.
A third object of the present application is to propose a computer device.
A fourth object of the present application is to propose a non-transitory computer-readable storage medium.
In order to achieve the above object, a method for structured light microscopic reconstruction without high true snr image is provided in an embodiment of the present application, including:
acquiring a biological sample image to be processed;
inputting the biological sample image to be processed into a pre-trained generator neural network for reconstruction processing to obtain a reconstructed image of the biological sample image to be processed; the pre-trained generator neural network comprises a pre-trained denoising neural network used for reserving structural light striations, and training data of the pre-trained denoising neural network are common samples.
Optionally, in an embodiment of the present application, the training method of the pre-trained neural network specifically includes:
acquiring common sample training data, wherein the common sample training data comprises noise-containing structured light original data and high signal-to-noise ratio true value data;
inputting the noise-containing structured light original data into a preset denoising neural network, and outputting noiseless image training data;
and training the preset denoising neural network according to the noise-free image training data and the high signal-to-noise ratio true value data to obtain the pre-trained denoising neural network.
Optionally, in an embodiment of the present application, the training method of the pre-trained neural network specifically includes:
acquiring raw data of a biological sample, wherein the raw data of the biological sample comprises low-resolution and noise-containing image data;
inputting raw data of the biological sample to the pre-trained de-noising neural network to output noise-free image data and a first label of the noise-free image data, the noise-free image data having a resolution higher than image data in the raw data;
inputting a preset random variable serving as a seed and the original data of the biological sample into a preset generator neural network to generate an output image and a second label of the output image;
and training the preset generator neural network according to the noiseless image data, the first label, the output image and the second label to obtain the trained generator neural network.
Optionally, in an embodiment of the present application, in the training method of the pre-trained generator neural network, the method includes: the first label is a true label and the second label is a false label.
Optionally, in this embodiment of the present application, training the preset generator neural network according to the noiseless image data and the first label, and the output image and the second label, to obtain a trained generator neural network, including:
inputting the noise-free image data and the first label and the output image and the second label to a pre-trained discriminator neural network to output a probability that an image is true/false;
and calculating a loss function by taking the actual label of the image as a true value according to the probability that the image is true/false, and training the preset generator neural network according to the loss function to obtain the trained generator neural network.
In order to achieve the above object, a second embodiment of the present application provides a structured light microscopy reconstruction apparatus without a true signal-to-noise ratio image, comprising:
the first acquisition module is used for acquiring an image of a biological sample to be processed;
the image reconstruction module is used for inputting the biological sample image to be processed into a pre-trained generator neural network for reconstruction processing to obtain a reconstructed image of the biological sample image to be processed; the pre-trained generator neural network comprises a pre-trained denoising neural network used for reserving structural light striations, and training data of the pre-trained denoising neural network are common samples.
Optionally, in this embodiment of the application, the training apparatus of the pre-trained generator neural network in the image reconstruction module specifically includes:
the device comprises a first acquisition unit, a second acquisition unit and a third acquisition unit, wherein the first acquisition unit is used for acquiring common sample training data, and the common sample training data comprises noise-containing structured light original data and high signal-to-noise ratio true value data;
the first processing unit is used for inputting the noise-containing structured light original data into a preset denoising neural network and outputting noiseless image training data;
and the first training unit is used for training the preset denoising neural network according to the noiseless image training data and the high signal-to-noise ratio true value data to obtain the pre-trained denoising neural network.
Optionally, in this embodiment of the application, the training apparatus of the pre-trained generator neural network in the image reconstruction module specifically includes:
the device comprises a first acquisition unit, a second acquisition unit and a third acquisition unit, wherein the first acquisition unit is used for acquiring common sample training data, and the common sample training data comprises noise-containing structured light original data and high signal-to-noise ratio true value data;
the first processing unit is used for inputting the noise-containing structured light original data into a preset denoising neural network and outputting noiseless image training data;
and the first training unit is used for training the preset denoising neural network according to the noiseless image training data and the high signal-to-noise ratio true value data to obtain the pre-trained denoising neural network.
Optionally, in this embodiment of the application, the training apparatus of the pre-trained generator neural network in the image reconstruction module specifically includes:
a second acquisition unit configured to acquire raw data of a biological sample, the raw data of the biological sample including low-resolution and noisy image data;
a second processing unit for inputting raw data of the biological sample to the pre-trained denoising neural network to output a noise-free image data having a resolution higher than image data in the raw data and a first label of the noise-free image data;
the third processing unit is used for inputting a preset random variable serving as a seed and the original data of the biological sample into a preset generator neural network so as to generate an output image and a second label of the output image;
and the second training unit is used for training the preset generator neural network according to the noiseless image data, the first label, the output image and the second label to obtain the trained generator neural network.
To achieve the above object, a third aspect of the present application provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the computer device implements the method according to the first aspect of the present application.
In order to achieve the above object, a fourth aspect of the present application provides a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the method according to the first aspect of the present application.
To sum up, in the structural light microscopic reconstruction method, the structural light microscopic reconstruction device, the computer device and the non-transitory computer readable storage medium which do not need the true value image with the high signal-to-noise ratio according to the embodiment of the present application, the biological sample image to be processed is input to the pre-trained generator neural network for reconstruction processing, so as to obtain the reconstructed image of the biological sample image to be processed; the pre-trained denoising method comprises the steps of adding a pre-trained denoising neural network for reserving structural light striations in a pre-trained generator neural network, wherein training data of the pre-trained denoising neural network are common samples. Therefore, the original data with high signal to noise ratio of the sample does not need to be obtained to be used as a true value image for network training, the pre-trained denoising neural network used for retaining the structural light striations is obtained through a common sample, then denoising processing of retaining the structural light illumination information is directly carried out on the real data of the biological sample containing noise, and then reconstruction processing is carried out on the biological sample image to be processed through the pre-trained generator neural network so as to obtain a reconstructed image.
Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.
Drawings
The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
fig. 1 is a flowchart of a method for structured light reconstruction without a true-value image with a high signal-to-noise ratio according to an embodiment of the present disclosure;
FIG. 2 is a schematic diagram of a training process of a neural network of a generator according to an embodiment of the present application;
FIG. 3 is a flowchart illustrating an actual application of the embodiment of the present application after training of the neural network of the generator is completed; and
fig. 4 is a schematic structural diagram of a structured light microscopic reconstruction apparatus without a true signal-to-noise ratio image according to an embodiment of the present application.
Detailed Description
Reference will now be made in detail to the embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are exemplary and intended to be used for explaining the present application and should not be construed as limiting the present application.
To facilitate a better understanding of the embodiments of the present application, a brief description will now be given of the conventional and recent academic and industrial community methods for reconstructing the original image of the acquired structured light microscope, the details of which are as follows:
after the original image of the structured light microscope is acquired, a reconstruction algorithm is required. The conventional reconstruction algorithm is mainly divided into two steps:
the first is that: estimation of structured light illumination parameters. Although the optical illumination portion of the structured light microscopy system can be stable, the structured light illumination parameters at the focal plane can be shifted to some extent due to possible tilt of the sample, mismatch of refractive indices of the media, and the like. Therefore, to obtain accurate parameters, an algorithm needs to be used to estimate the actual lighting parameters.
Secondly, the following steps: and separating and splicing all levels of frequency information of the sample in the Fourier domain, and reconstructing a super-resolution frequency spectrum. In the fourier domain, a single original captured picture is a coupling of the information of each level of the sample. In order to extract each independent frequency domain component, different levels of information need to be separated by using an algorithm, and then the information is translated to a position corresponding to a frequency domain, and finally reconstruction of an expanded spectrum is realized. And the expanded information contains higher-frequency information, so that the resolution can be improved.
However, the above conventional structured light microscopy data reconstruction algorithm has a large performance deficiency, and particularly, the reconstructed image under the condition of low noise generates obvious artifacts, thereby seriously affecting the image quality. In such data, we can hardly distinguish real sample information from reconstruction artifacts, and thus cannot find life science phenomena behind them. Under the original data with low signal-to-noise ratio, the quality degradation of the reconstruction by the common algorithm mainly comprises the following two reasons:
the first is that: inaccuracy of parameter estimation. Since the original collected image contains a large amount of noise which is random and not modulated by the structured light illumination, when the intensity of the random information is too large, the information containing modulation cannot be effectively obtained, so that parameter estimation is wrong and even the algorithm cannot be converged.
Secondly, the following steps: frequency domain information separation and splicing are inaccurate. When the signal-to-noise ratio is too low, the information of each order of the sample is coupled with noise very strongly, so that the separation cannot be performed. This noise is further translated to high frequency locations, which introduce high frequency anomalous spikes that lead to reconstruction artifacts.
Because the traditional structured light microscopy algorithm has the defects in low signal-to-noise ratio data, the improvement of the performance of the traditional structured light microscopy algorithm is very practical. With the development of the field of machine learning in recent years, the academic and industrial circles have tried to use a deep neural network to perform structured light reconstruction, and in recent research, most of research methods are supervised learning, that is: the original data with high signal-to-noise ratio of the sample needs to be obtained as a true value image for network training, however, in an actual biological experiment, the data with high signal-to-noise ratio is obtained with high experimental difficulty and experimental cost, so the practical applicability of the method is limited.
Therefore, the embodiment of the application provides a method and a device for reconstructing structured light microscopy without a true value image with a high signal-to-noise ratio, so that the reconstruction quality and the application capability of a structured light microscopy algorithm are improved while the problems are solved.
The following describes a method and an apparatus for structured light reconstruction without high snr true value images according to embodiments of the present application with reference to the drawings.
Fig. 1 is a flowchart of a structured light reconstruction method without a true signal-to-noise ratio image according to an embodiment of the present disclosure.
In order to realize structured light microscopic reconstruction without high signal-to-noise ratio data and reduce experiment cost, the embodiment of the application provides a structured light microscopic reconstruction method without a high signal-to-noise ratio true value image, as shown in fig. 1, the method includes the following steps:
step S10, acquiring a to-be-processed biological sample image, wherein the to-be-processed biological sample image is a biological sample image containing noise.
Step S20, inputting the biological sample image to be processed into a generator neural network trained in advance for reconstruction processing, and obtaining a reconstructed image of the biological sample image to be processed; the pre-trained generator neural network comprises a pre-trained denoising neural network used for reserving the structural light striations, and training data of the pre-trained denoising neural network are various common samples.
According to the method and the device, the original data with high signal to noise ratio of the sample does not need to be obtained and is used as the true value image for network training, the pre-trained denoising neural network used for retaining the structured light striations is obtained through the common sample, then denoising processing of retaining the structured light illumination information is directly carried out on the real acquisition data of the biological sample containing the noise, and then reconstruction processing is carried out on the biological sample image to be processed through the pre-trained generator neural network so as to obtain the reconstructed image.
Fig. 2 is a schematic diagram of a training process of a neural network of a generator in an embodiment of the present application.
Further, in this embodiment of the present application, as shown in fig. 2, the method for training a pre-trained neural network of a generator specifically includes:
and acquiring common sample training data, wherein the common sample training data comprises noise-containing structured light original data and high signal-to-noise ratio true value data.
Inputting the noise-containing structured light original data to a preset denoising neural network, and outputting noiseless image training data;
and training the preset denoising neural network according to the noise-free image training data and the high signal-to-noise ratio true value data to obtain the pre-trained denoising neural network.
Specifically, in the embodiment of the present application, the training data for training the preset denoising neural network is a common sample training denoising neural network. The method comprises the steps of inputting original data of a common sample illuminated by structured light and collected by a camera into a denoising neural network, outputting a noiseless high-resolution image capable of reserving the illumination characteristic of the structured light, and performing supervised training by taking high signal-to-noise ratio data collected by the common sample as a true value. It is particularly emphasized that the generic sample training data in the embodiments of the present application is not specific to biological samples, in other words, the generic sample training data is not the raw data of biological samples, i.e. samples of biological macromolecules, cells, tissues and organs of healthy and diseased organisms are collected, processed, stored and applied in a standardized manner.
Therefore, the preset denoising neural network in the embodiment of the application can be separated from a specific biological sample to carry out independent training, namely, the denoising processing of retaining the structural light striations can be realized on a universal sample, so that the resolution of a common sample cannot be reduced; after the preset denoising neural network finishes training, the method can be directly applied to a biological sample, and input real acquisition data of the biological sample containing noise is input into the denoising neural network trained in advance to perform denoising processing for reserving the structured light illumination information. That is, the normal sample processed by the pre-set denoising neural network is equivalent to the raw data with high signal-to-noise ratio in the actual biological experiment.
Further, in this embodiment of the present application, as shown in fig. 2, the method for training a pre-trained neural network of a generator specifically includes:
raw data of a biological sample is acquired, the raw data of the biological sample including low resolution and noisy image data.
Raw data of a biological sample is input to a pre-trained de-noising neural network to output noise-free image data and a first label of the noise-free image data, the resolution of the noise-free image data being higher than that of the image data in the raw data.
Specifically, since the output of the noise-free image data in the embodiment of the present application is derived from the noise-free image training data, and the noise-free image training data is obtained by performing denoising processing for preserving the structured light illumination information on the common sample training data through a preset denoising neural network, the embodiment of the present application defines the first label of the noise-free image data as a true label.
And taking a preset random variable as a seed, and inputting the seed and the original data of the biological sample into a preset generator neural network to generate an output image and a second label of the output image.
Specifically, since the generation of the output image outputs raw data derived from the seed and the biological sample, in other words, the output image in the embodiment of the present application is data that is not derived from the real image and is obtained by processing the raw data of the seed and the biological sample by the preset generator neural network, the embodiment of the present application defines the second label corresponding to the output image as the label.
And training a preset generator neural network according to the noise-free image data, the first label, the output image and the second label to obtain the trained generator neural network.
According to the method and the device, the original data with high signal to noise ratio of the sample does not need to be obtained and used as a true value image for network training, a pre-trained denoising neural network used for retaining the structured light striations is obtained through a common sample, then denoising processing for retaining the structured light illumination information is directly carried out on the actually-acquired data of the noisy biological sample, and then reconstruction processing is carried out on the biological sample image to be processed through a pre-trained generator neural network to obtain a reconstructed image, namely, the method and the device not only realize structured light microscopic reconstruction without high signal to noise ratio data, reduce experiment cost, but also remarkably improve the application range of the structured light microscopic technology; in addition, the embodiment of the application uses the denoised data to carry out truth data and assists in training the neural network of the generator, and in the process, only the easily obtained high signal-to-noise ratio truth data of the common sample is used, but the difficultly obtained high signal-to-noise ratio truth data of the biological sample is not needed, so that the network can obviously reduce the difficulty in acquiring the structured light microscopic truth data. Therefore, the scheme realizes the structural light microscopic reconstruction algorithm without high signal-to-noise ratio data, and can remarkably improve the application range of the structural light microscopic technology.
Further, in the embodiment of the present application, as shown in fig. 2, the method further includes a pre-trained discriminator neural network, where the training method of the pre-trained discriminator neural network includes: inputting the noise-free image data and the first label as well as the output image and the second label into a pre-trained discriminator neural network to output the probability that the image is true/false;
and calculating a loss function by taking the actual label of the image as a true value according to the probability that the image is true/false, and training a preset generator neural network according to the loss function to obtain the trained generator neural network.
Specifically, in the embodiment of the present application, the generated data of the generator neural network (labeled as "false") and the noise-free image data output by the pre-trained denoising neural network (labeled as "true") are input to the pre-trained discriminator neural network as input data, respectively, to output the probability that the image is true/false; and calculating a loss function by taking the actual label of the image as a true value, and training a preset generator neural network according to the loss function to obtain the trained generator neural network.
Fig. 3 is a flowchart of an actual application of the generator neural network after training is completed in the embodiment of the present application.
After the network in the embodiment of the present application is trained, the reconstruction process applied to a specific biological sample is as shown in fig. 3: the image of the biological sample containing noise is input into an output image of a neural network of a generator, and the output generated image is lifelike enough to an actual image, so that the image can be used as a reconstruction result of the structured light microscope.
Based on the above analysis, it can be seen that the pre-trained generator neural network adopted in the embodiment of the present application includes a pre-trained denoising neural network for preserving the structural light striations, and compared with the prior art, the embodiment of the present application has the following advantages:
1. the denoising neural network in the embodiment of the application can be independently trained from a specific biological sample, namely, denoising of the universal sample with the retained structural light stripe can be realized. After the network is trained, the network can be directly applied to a biological sample, and input noise-containing biological sample real acquisition data is subjected to denoising processing.
2. The denoising neural network in the embodiment of the application can realize denoising of the reserved structured light illumination information, and cannot cause reduction of resolution, so that the embodiment of the application can realize structured light microscopic reconstruction without high signal-to-noise ratio data, and the experiment cost is reduced.
3. The embodiment of the application only uses the high signal-to-noise ratio true value data of the easily obtained common sample, and does not need the high signal-to-noise ratio true value data of the biological sample which is difficult to obtain, so that the network can obviously reduce the difficulty in collecting the structured light microscopic true value data.
In summary, the pre-trained neural network of the generator provided by the embodiment of the present application is a brand-new deep learning network model, realizes a structured light microscopy reconstruction algorithm without high signal-to-noise ratio data, and can significantly improve the application range of the structured light microscopy technology.
In order to implement the above embodiments, the present application further proposes a structured light reconstruction apparatus without a true signal-to-noise ratio image, as shown in fig. 4, the apparatus includes:
a first acquisition module 10, configured to acquire an image of a biological sample to be processed;
the image reconstruction module 20 is configured to input the biological sample image to be processed to a pre-trained generator neural network for reconstruction processing, so as to obtain a reconstructed image of the biological sample image to be processed; the pre-trained generator neural network comprises a pre-trained denoising neural network used for reserving the structural light striations, and training data of the pre-trained denoising neural network are common samples.
Further, in a possible implementation manner of the embodiment of the present application, the training apparatus of the pre-trained generator neural network in the image reconstruction module specifically includes:
the device comprises a first acquisition unit, a second acquisition unit and a third acquisition unit, wherein the first acquisition unit is used for acquiring common sample training data, and the common sample training data comprises noise-containing structured light original data and high signal-to-noise ratio true value data;
the first processing unit is used for inputting the noise-containing structured light original data into a preset denoising neural network and outputting noiseless image training data;
and the first training unit is used for training the preset denoising neural network according to the non-noise image training data and the high signal-to-noise ratio true value data to obtain the pre-trained denoising neural network.
Further, in a possible implementation manner of the embodiment of the present application, the training apparatus of the pre-trained generator neural network in the image reconstruction module specifically includes:
a second acquisition unit configured to acquire raw data of a biological sample, the raw data of the biological sample including low-resolution and noisy image data;
a second processing unit, for inputting the raw data of the biological sample to a pre-trained de-noising neural network to output the noise-free image data and the first label of the noise-free image data, wherein the resolution of the noise-free image data is higher than that of the image data in the raw data;
the third processing unit is used for inputting the seeds and the original data of the biological sample into a preset generator neural network by taking preset random variables as seeds so as to generate an output image and a second label of the output image;
and the second training unit is used for training the preset generator neural network according to the noiseless image data, the first label, the output image and the second label to obtain the trained generator neural network.
In order to implement the foregoing embodiments, the present application further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the method described in the foregoing embodiments is implemented.
In order to implement the above embodiments, the present application also proposes a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method as described in the above embodiments.
In the description herein, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present application, "plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing steps of a custom logic function or process, and alternate implementations are included within the scope of the preferred embodiment of the present application in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware that is related to instructions of a program, and the program may be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
In addition, functional units in the embodiments of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a separate product, may also be stored in a computer readable storage medium.
The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc. Although embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application, and that variations, modifications, substitutions and alterations may be made to the above embodiments by those of ordinary skill in the art within the scope of the present application.
Claims (10)
1. A structured light microscopic reconstruction method without a true value image with a high signal-to-noise ratio is characterized by comprising the following steps:
acquiring a biological sample image to be processed;
inputting the biological sample image to be processed into a pre-trained generator neural network for reconstruction processing to obtain a reconstructed image of the biological sample image to be processed; the pre-trained generator neural network comprises a pre-trained denoising neural network used for reserving structural light striations, and training data of the pre-trained denoising neural network are common samples.
2. The method for structured light reconstruction without true high signal-to-noise ratio (SNR) images according to claim 1, wherein the pre-trained training method for the generator neural network specifically comprises:
acquiring common sample training data, wherein the common sample training data comprises noise-containing structured light original data and high signal-to-noise ratio true value data;
inputting the noise-containing structured light original data into a preset denoising neural network, and outputting noiseless image training data;
and training the preset denoising neural network according to the noise-free image training data and the high signal-to-noise ratio true value data to obtain the pre-trained denoising neural network.
3. The method for structured light reconstruction without true high signal-to-noise ratio (SNR) images according to claim 1, wherein the pre-trained training method for the generator neural network specifically comprises:
acquiring raw data of a biological sample, wherein the raw data of the biological sample comprises low-resolution and noise-containing image data;
inputting raw data of the biological sample to the pre-trained de-noising neural network to output noise-free image data and a first label of the noise-free image data, the noise-free image data having a resolution higher than image data in the raw data;
inputting a preset random variable serving as a seed and the original data of the biological sample into a preset generator neural network to generate an output image and a second label of the output image;
and training the preset generator neural network according to the noiseless image data, the first label, the output image and the second label to obtain the trained generator neural network.
4. The method for structured light reconstruction without true high signal-to-noise ratio (SNR) images according to claim 3, wherein the method for training the pre-trained neural network comprises: the first label is a true label and the second label is a false label.
5. The method for structured light reconstruction without true-to-noise ratio images according to any of claims 2 to 4, wherein training the pre-set generator neural network according to the image data and the first label, and the output image and the second label to obtain a trained generator neural network comprises:
inputting the noise-free image data and the first label and the output image and the second label to a pre-trained discriminator neural network to output a probability that an image is true/false;
and calculating a loss function by taking the actual label of the image as a true value according to the probability that the image is true/false, and training the preset generator neural network according to the loss function to obtain the trained generator neural network.
6. A structured light microscopic reconstruction apparatus without a true high signal-to-noise ratio image, comprising:
the first acquisition module is used for acquiring an image of a biological sample to be processed;
the image reconstruction module is used for inputting the biological sample image to be processed into a pre-trained generator neural network for reconstruction processing to obtain a reconstructed image of the biological sample image to be processed; the pre-trained generator neural network comprises a pre-trained denoising neural network used for reserving structural light striations, and training data of the pre-trained denoising neural network are common samples.
7. The apparatus for structured light microscopic reconstruction without true high signal-to-noise ratio image according to claim 6, wherein the training apparatus of the pre-trained neural network in the image reconstruction module comprises:
the device comprises a first acquisition unit, a second acquisition unit and a third acquisition unit, wherein the first acquisition unit is used for acquiring common sample training data, and the common sample training data comprises noise-containing structured light original data and high signal-to-noise ratio true value data;
the first processing unit is used for inputting the noise-containing structured light original data into a preset denoising neural network and outputting noiseless image training data;
and the first training unit is used for training the preset denoising neural network according to the noiseless image training data and the high signal-to-noise ratio true value data to obtain the pre-trained denoising neural network.
8. The apparatus for structured light microscopic reconstruction without true high signal-to-noise ratio image according to claim 6, wherein the training apparatus of the pre-trained neural network in the image reconstruction module comprises:
a second acquisition unit configured to acquire raw data of a biological sample, the raw data of the biological sample including low-resolution and noisy image data;
a second processing unit for inputting raw data of the biological sample to the pre-trained denoising neural network to output a noise-free image data having a resolution higher than image data in the raw data and a first label of the noise-free image data;
the third processing unit is used for inputting a preset random variable serving as a seed and the original data of the biological sample into a preset generator neural network so as to generate an output image and a second label of the output image;
and the second training unit is used for training the preset generator neural network according to the noiseless image data, the first label, the output image and the second label to obtain the trained generator neural network.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method of any one of claims 1-5 when executing the computer program.
10. A non-transitory computer-readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110852923.XA CN113724150A (en) | 2021-07-27 | 2021-07-27 | Structured light microscopic reconstruction method and device without high signal-to-noise ratio true value image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110852923.XA CN113724150A (en) | 2021-07-27 | 2021-07-27 | Structured light microscopic reconstruction method and device without high signal-to-noise ratio true value image |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113724150A true CN113724150A (en) | 2021-11-30 |
Family
ID=78674089
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110852923.XA Pending CN113724150A (en) | 2021-07-27 | 2021-07-27 | Structured light microscopic reconstruction method and device without high signal-to-noise ratio true value image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113724150A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115293981A (en) * | 2022-08-02 | 2022-11-04 | 中国科学院生物物理研究所 | Denoising and super-resolution reconstruction method and system for structured light illumination fluorescence microscopic image |
| CN115984107A (en) * | 2022-12-21 | 2023-04-18 | 中国科学院生物物理研究所 | Self-supervision multi-mode structure light microscopic reconstruction method and system |
-
2021
- 2021-07-27 CN CN202110852923.XA patent/CN113724150A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115293981A (en) * | 2022-08-02 | 2022-11-04 | 中国科学院生物物理研究所 | Denoising and super-resolution reconstruction method and system for structured light illumination fluorescence microscopic image |
| CN115984107A (en) * | 2022-12-21 | 2023-04-18 | 中国科学院生物物理研究所 | Self-supervision multi-mode structure light microscopic reconstruction method and system |
| CN115984107B (en) * | 2022-12-21 | 2023-08-11 | 中国科学院生物物理研究所 | Self-supervision multi-mode structure light microscopic reconstruction method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Maier et al. | Medical imaging systems: An introductory guide | |
| JP7422825B2 (en) | Focus-weighted machine learning classifier error prediction for microscope slide images | |
| Aguet et al. | Model-based 2.5-D deconvolution for extended depth of field in brightfield microscopy | |
| JP2020529083A (en) | Methods, instruments, and computer programs to improve the reconstruction of high-density ultra-resolution images from diffraction-limited images obtained by single-molecule localization microscopy. | |
| CN113724150A (en) | Structured light microscopic reconstruction method and device without high signal-to-noise ratio true value image | |
| Buranachai et al. | Rapid frequency-domain FLIM spinning disk confocal microscope: lifetime resolution, image improvement and wavelet analysis | |
| Goceri et al. | Quantitative validation of anti‐PTBP1 antibody for diagnostic neuropathology use: Image analysis approach | |
| CN112862805B (en) | Automatic auditory neuroma image segmentation method and system | |
| CN114998471B (en) | Magnetic particle imaging reconstruction method based on RecNet model | |
| JP7431472B2 (en) | Slide-free histological imaging method and system | |
| CA3078256A1 (en) | Methods and systems for 3d structure estimation using non-uniform refinement | |
| Kromp et al. | Deep Learning architectures for generalized immunofluorescence based nuclear image segmentation | |
| Zhao et al. | Deep Learning‐Based Denoising in High‐Speed Portable Reflectance Confocal Microscopy | |
| CN112200726B (en) | Urinary sediment visible component detection method and system based on lensless microscopic imaging | |
| JP2019537736A (en) | System and method for object detection in holographic lens-free imaging with convolution dictionary learning and coding | |
| CN115984107B (en) | Self-supervision multi-mode structure light microscopic reconstruction method and system | |
| CN115841423B (en) | Deep learning-based wide-field illumination fluorescent super-resolution microscopic imaging method | |
| CN116485813A (en) | Zero-sample brain lesion segmentation method, system, equipment and medium based on prompt learning | |
| CN115019130A (en) | Training method of virtual dyeing model and method for generating bright field virtual dyeing image | |
| CN111476125A (en) | Three-dimensional fluorescence microscopic signal denoising method based on generation countermeasure network | |
| Kang et al. | Coordinate-based neural representations for computational adaptive optics in widefield microscopy | |
| CN112967268B (en) | Digital optical tomography method and device based on optical field | |
| CN112258457B (en) | Multi-dimensional feature extraction method of full-volume three-dimensional ultrasonic image | |
| Prigent et al. | SPITFIR (e): A supermaneuverable algorithm for restoring 2D-3D fluorescence images and videos, and background subtraction | |
| US20240135565A1 (en) | Method and device for light field microscopy |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |