CN115220211B - Microscopic imaging system and method based on deep learning and light field imaging - Google Patents
Microscopic imaging system and method based on deep learning and light field imaging Download PDFInfo
- Publication number
- CN115220211B CN115220211B CN202210902191.5A CN202210902191A CN115220211B CN 115220211 B CN115220211 B CN 115220211B CN 202210902191 A CN202210902191 A CN 202210902191A CN 115220211 B CN115220211 B CN 115220211B
- Authority
- CN
- China
- Prior art keywords
- camera sensor
- dimensional image
- resolution
- vcd
- imaging
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003384 imaging method Methods 0.000 title claims abstract description 90
- 238000013135 deep learning Methods 0.000 title claims abstract description 27
- 238000000034 method Methods 0.000 title claims abstract description 15
- 230000003287 optical effect Effects 0.000 claims description 20
- 238000012549 training Methods 0.000 claims description 16
- 230000006870 function Effects 0.000 claims description 10
- 238000000386 microscopy Methods 0.000 claims description 8
- 230000008569 process Effects 0.000 claims description 8
- 238000013528 artificial neural network Methods 0.000 claims description 7
- 238000007334 copolymerization reaction Methods 0.000 claims description 5
- 230000003068 static effect Effects 0.000 claims description 5
- 238000012360 testing method Methods 0.000 claims description 5
- 239000011159 matrix material Substances 0.000 claims description 3
- 238000001914 filtration Methods 0.000 claims 1
- 238000013527 convolutional neural network Methods 0.000 abstract description 8
- 238000005516 engineering process Methods 0.000 description 6
- 238000011160 research Methods 0.000 description 4
- 210000004027 cell Anatomy 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 2
- 238000011282 treatment Methods 0.000 description 2
- 241000244203 Caenorhabditis elegans Species 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000012292 cell migration Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 210000003743 erythrocyte Anatomy 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 244000005700 microbiome Species 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 238000000399 optical microscopy Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
- G02B21/365—Control or image processing arrangements for digital or video microscopes
- G02B21/367—Control or image processing arrangements for digital or video microscopes providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10056—Microscopic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Abstract
The invention discloses a microscopic imaging system and a microscopic imaging method based on deep learning and light field imaging, and belongs to the technical field of light field microscopic imaging. The microscopic imaging system includes: the system comprises a microscopic imaging system, a deep learning network module and an image output module. According to the invention, a two-dimensional image array is obtained through a light field microscope system formed by a micro lens array and a camera sensor, multi-view image information is obtained from the two-dimensional image array after the two-dimensional image array passes through a deep learning network, super-resolution reconstruction is further carried out on the image, and finally a high-resolution three-dimensional reconstructed image with uniform intensity distribution is obtained. Compared with the existing super-resolution reconstruction method based on the convolutional neural network, the image generated by the method has higher spatial resolution, smaller reconstruction artifacts and larger reconstruction throughput, thereby effectively improving the imaging definition; and under the lens array system, the invention also has the advantages of low cost, low system complexity and no need of scanning super-resolution imaging.
Description
Technical Field
The invention relates to a microscopic imaging system and a microscopic imaging method based on deep learning and light field imaging, and belongs to the technical field of light field microscopic imaging.
Background
The light field imaging system mainly collects and captures a four-dimensional light field which is distributed in space through an optical device, and then calculates corresponding images according to different application requirements. The computational light field imaging technology is based on a four-dimensional light field, aims to establish the relation of light in a plurality of dimensions such as a space domain, a visual angle, a spectrum and a time domain, realizes coupling perception, decoupling reconstruction and intelligent processing, and is used for multi-dimensional and multi-scale imaging for a large-range dynamic scene. The light field imaging technology is gradually applied to the fields of life science, industrial detection, national security, unmanned systems, virtual reality/augmented reality and the like, and has important academic research value and industrial application prospect.
Light-Field Microscopy (LFM) is implemented by inserting a microlens array capable of capturing Light Field information on the relay image plane of a conventional optical microscope. The multi-view image and the multi-layer focal plane image can be reconstructed through inversion of 4D light field data, and three-dimensional microscopic imaging can be realized by introducing a deconvolution algorithm and tomographic reconstruction. Since these subsequent treatments can be achieved by one exposure, there are unique advantages to observing moving microorganisms and light sensitive samples. Optical microscopy has long been an important tool in biomedical research by virtue of its non-contact, non-damaging, etc. However, since 1873, it has been considered that the resolution limit of the optical microscope is about 200nm, and it is not possible to clearly observe biological structures having a size of 200nm or less.
The optical field microscope is used for obtaining high-resolution images, which are significant in biological research and medical treatment, wherein the digital microscopic imaging technology is different from the traditional optical microscopic imaging technology, and can obtain biological parameters and morphological information of cells according to reconstructed holograms, so that the optical field microscope is an effective non-contact lossless three-dimensional imaging technology. With the development of image sensors and the improvement of hardware computing capability, the digital holographic microscopic imaging technology has made remarkable progress and breakthrough in the field of living organism cell detection, especially in the field of red blood cell detection. Digital microscopy imaging techniques are now widely used for cell migration analysis and abnormal cell behavior research, as well as in a large number of instruments for imaging medical images.
However, on the one hand, the images obtained by digital microscopy imaging techniques cannot have very high accuracy due to the limitations of the equipment and of the imaging technique; on the other hand, due to limitations in sensor resolution (the ability of the sensor to sense the smallest change measured), light field cameras often trade off spatial resolution (the size or dimension of the smallest unit that can be distinguished in detail) for angular resolution (the ability of an imaging system or system element to differentially distinguish between the smallest spacing of two adjacent objects), resulting in an unclear image. Therefore, limited spatial resolution is a difficulty in the development of light field cameras.
In order to solve the above problems, yoon et al first proposed a convolutional neural network-based light field data super-resolution reconstruction (Yoon Y, jeon H G, yoon D, et al, light Field Image SuperResolution using Convolutional Neural Network [ J ]. IEEE Signal Processing Letters, 2017.), wherein the network can be divided into a spatial resolution reconstruction convolutional neural network and an angular resolution reconstruction convolutional neural network, but the model does not fully utilize the effective information between multi-view images, resulting in failure to obtain a higher resolution reconstruction image, and failure to quickly obtain a reconstruction image free of artifacts and uniform in intensity distribution.
Disclosure of Invention
In order to solve the problems of artifacts, non-uniform resolution, low reconstruction speed and the like in the conventional microscopic imaging, the invention provides a microscopic imaging system and a microscopic imaging method based on deep learning and light field imaging, wherein the technical scheme is as follows:
a first object of the present invention is to provide a microscopic imaging system based on deep learning and light field imaging, comprising, in order: the system comprises a microscope system, a deep learning network module and an image output module;
the microscopy system is used for acquiring a plurality of two-dimensional data of an image, and comprises: a microlens 1, a first dichroic mirror 2, a reflecting mirror 3, a beam splitter 4, a band pass filter 5, a first camera sensor 6, a microlens array 7, a relay lens 8, a second dichroic mirror 9, a second camera sensor 10, a third camera sensor 11;
the microscope 1 collects image data, filters and removes interference light through the first dichroic mirror 2, then makes the collected light enter a horizontal direction from a vertical direction through the reflecting mirror 3, and then makes a part of the collected signal gather to a first camera sensor 6 for wide-field imaging through the beam splitter 4 to perform wide-field imaging; the other part respectively passes through the micro lens array 7, the relay lens 8 and the second dichroic mirror 9 to respectively perform light field imaging on the second camera sensor 10 and the third camera sensor 11;
the deep learning network module is used for reconstructing two-dimensional image data imaged by the first camera sensor 6, the second camera sensor 10 and the third camera sensor 11 into a high-resolution three-dimensional image;
the image output module is used for outputting the reconstructed high-resolution three-dimensional image.
Optionally, the deep learning network module adopts a trained VCD deep network VCD-Net to reconstruct the high-resolution three-dimensional image.
Alternatively, the first camera sensor 6, the second camera sensor 10, and the third camera sensor 11 employ sCMOS cameras.
Optionally, the training process of the VCD deep network VCD-Net includes:
step 1: initializing the VCD-Net, comprising: network parameters and loss functions;
step 2: obtaining a high-resolution three-dimensional image from a copolymerization microscope of a real static sample and synthesized data;
step 3: constructing a fluctuation optical model, inputting the high-resolution three-dimensional image obtained in the step 2 into the fluctuation optical model, and outputting a corresponding two-dimensional image;
step 4: and (3) constructing a training set and a testing set based on the two-dimensional image obtained in the step (3) and the high-resolution three-dimensional image obtained in the step (2), taking the two-dimensional image as input, taking the high-resolution three-dimensional image as output, and training the VCD-Net until convergence to obtain an optimal VCD-Net network model.
Optionally, when light field imaging is performed on the sample, 1: the relay lens 8 of 1 focuses the second camera sensor 10 and the third camera sensor 11 on the back focal plane of the microlens array 7.
Optionally, the fluctuating optical model is:
F＝Hg
wherein, vector F represents the acquired original light field image, vector g represents the reconstructed 3D discrete point cloud of the object, and H is the point spread function matrix representation of the imaging process.
A second object of the present invention is to provide a microscopic imaging method based on deep learning and light field imaging, comprising:
step one: acquiring image data by adopting a microscope lens, inputting the image data into a plurality of camera sensors in a microscope system and respectively performing wide-field imaging and light field imaging by a sensor formed by a microlens array and a CCD (charge coupled device) to obtain a plurality of two-dimensional images;
step two: inputting the two-dimensional image obtained in the first step into a trained deep neural network, and obtaining a reconstructed high-resolution three-dimensional image through the trained deep neural network.
Optionally, the microscope system includes: a microlens 1, a first dichroic mirror 2, a reflecting mirror 3, a beam splitter 4, a band pass filter 5, a first camera sensor 6, a microlens array 7, a relay lens 8, a second dichroic mirror 9, a second camera sensor 10, a third camera sensor 11;
the microscope 1 collects image data, filters and removes interference light through the first dichroic mirror 2, then makes the collected light enter a horizontal direction from a vertical direction through the reflecting mirror 3, and then makes a part of the collected signal gather to a first camera sensor 6 for wide-field imaging through the beam splitter 4 to perform wide-field imaging; the other part passes through the microlens array 7, the relay lens 8 and the second dichroic mirror 9, and light field imaging is performed on the second camera sensor 10 and the third camera sensor 11, respectively.
Optionally, the deep neural network uses a VCD-Net network to reconstruct the two-dimensional image data imaged by the first camera sensor 6, the second camera sensor 10 and the third camera sensor 11 into a high-resolution three-dimensional image.
Optionally, the training process of the VCD-Net network includes:
step 1: constructing and initializing VCD-Net;
step 2: obtaining a high-resolution three-dimensional image from a copolymerization microscope of a real static sample and synthetic data thereof;
step 3: constructing a fluctuation optical model, inputting the high-resolution three-dimensional image obtained in the step 2 into the fluctuation optical model, and outputting a corresponding two-dimensional image;
step 4: and (3) constructing a training set and a testing set based on the two-dimensional image obtained in the step (3) and the high-resolution three-dimensional image obtained in the step (2), taking the two-dimensional image as input, taking the high-resolution three-dimensional image as output, and training the VCD-Net until convergence to obtain an optimal VCD-Net network model.
The invention has the beneficial effects that:
the invention constructs a light field microscopic imaging system based on light field imaging and a deep learning network, acquires a plurality of two-dimensional images through a microscopic system formed by a micro lens array and a camera sensor, acquires multi-view image information from the two-dimensional images through the deep learning network to reconstruct the images in super resolution, and acquires three-dimensional images in high resolution, compared with the existing super resolution reconstruction method based on a convolutional neural network, the image generated by the method has higher spatial resolution (1.0+0.15 um), smaller reconstruction artifacts, and greater reconstruction throughput (200 HZ), thereby effectively improving imaging sharpness; in addition, under the lens array system, the super-resolution light field image generation model based on the depth neural network also has the advantages of low cost, low system complexity and no need of scanning super-resolution imaging, and the reconstruction speed is effectively improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic plan view of a microscopic imaging system according to an embodiment of the present invention.
Wherein, 1-a microscope lens; 2-a first dichroic mirror; a 3-mirror; 4-beam splitters; a 5-band pass filter; 6-a first camera sensor; 7-a microlens array; 8-a relay lens; 9-a second dichroic mirror; 10-a second camera sensor; 11-third camera sensor.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the embodiments of the present invention will be described in further detail with reference to the accompanying drawings.
First, the basic theoretical knowledge related to the present invention is described as follows:
VCD-Net network:
the VCD network is: in a typical Convolutional Neural Network (CNN), some N-th convolutional layer receives a feature map from a previous (N-1) layer and generates a new feature map using a different convolutional kernel. The network ultimately produces a multi-channel output. Where each channel is a nonlinear combination of the original inputs, this concept is similar to the digital refocusing algorithm in light field photography, where each synthetic plane of the reconstructed volume can be treated as a superposition of different views extracted from the light field. Through the cascade layers, our model is expected to gradually convert the original angular information from the light field original image into depth features, finally forming a conventional 3D image stack and reconstructing the scene. In an implementation, the customized VCD-Net is based on a modified U-Net architecture (see https:// closed. Content. Com/development/architecture/1520224). The method comprises a downsampling path and a symmetrical upsampling path, and along the two paths, each layer has three parameters: n, f and s represent the number of output channels, the filter size of the convolution kernel and the step size of the shift kernel, respectively.
Embodiment one:
the embodiment provides a microscopic imaging system based on deep learning and light field imaging, the microscopic imaging system comprises: the system comprises a microscope system, a deep learning network module and an image output module;
the microscopy system is used for acquiring a plurality of two-dimensional data of an image, see fig. 1, comprising: a microlens 1, a first dichroic mirror 2, a reflecting mirror 3, a beam splitter 4, a band pass filter 5, a first camera sensor 6, a microlens array 7, a relay lens 8, a second dichroic mirror 9, a second camera sensor 10, a third camera sensor 11;
the microscope 1 collects image data, filters and removes interference light through the first dichroic mirror 2, then makes the light enter the horizontal direction from the vertical direction through the reflecting mirror 3, and then makes a part of collected signals gather to the first camera sensor 6 for wide-field imaging through the beam splitter 4 to perform wide-field imaging; the other part respectively passes through the micro lens array 7, the relay lens 8 and the second dichroic mirror 9 to respectively perform light field imaging on the second camera sensor 10 and the third camera sensor 11;
the deep learning network module is used for reconstructing two-dimensional image data imaged by the first camera sensor 6, the second camera sensor 10 and the third camera sensor 11 into a high-resolution three-dimensional image;
the image output module is used for outputting the reconstructed high-resolution three-dimensional image.
Embodiment two:
the embodiment relates to a microscopic imaging system based on deep learning and light field imaging, which comprises the following components in sequence: the system comprises a microscope system, a deep learning network module and an image output module;
the microscopy system is used for acquiring a plurality of two-dimensional data of an image, see fig. 1, comprising: a microlens 1, a first dichroic mirror 2, a reflecting mirror 3, a beam splitter 4, a band pass filter 5, a first camera sensor 6, a microlens array 7, a relay lens 8, a second dichroic mirror 9, a second camera sensor 10, a third camera sensor 11; all three camera sensors employ sCMOS cameras.
The microscope 1 collects image data, the interference light is filtered and removed through the first dichroic mirror 2, the collected light enters the horizontal direction from the vertical direction through the reflecting mirror 3, and part of the collected signals is collected to the first camera sensor 6 for wide-field imaging through the beam splitter 4 to carry out wide-field imaging; the other part respectively passes through the micro lens array 7, the relay lens 8 and the second dichroic mirror 9 to respectively perform light field imaging on the second camera sensor 10 and the third camera sensor 11; when light field imaging is performed, 1: the relay lens 8 of 1 focuses the second camera sensor 10 and the third camera sensor 11 on the back focal plane of the microlens array 7.
The deep learning network module adopts a trained VCD deep network VCD-Net to reconstruct a high-resolution three-dimensional image according to two-dimensional image data imaged by the first camera sensor 6, the second camera sensor 10 and the third camera sensor 11;
the training process of VCD-Net includes:
step 1: constructing and initializing VCD-Net, and constructing a VCD-Net deep learning model in a Window10 environment by using Tensorflow1.15.0, tensorlayer 1.8.5 and Python 3;
step 2: obtaining a high-resolution three-dimensional image from a copolymerization microscope of static samples or synthetic data;
step 3: constructing a fluctuation optical model, inputting the high-resolution three-dimensional image obtained in the step 2 into the fluctuation optical model, and outputting a corresponding two-dimensional image;
the wave optics model is: f=hg, where vector F represents the light field, vector g represents the discrete volume being reconstructed, H is a measurement matrix modeling of the forward imaging process, and H is primarily determined by the point spread function of the light field microscope.
The spatially varying point spread function of an optical microscope is modeled using wave optics, the light field point spread function maps the transition from a three-dimensional object to a two-dimensional plane, and is also spatially varying, with a unique point spread function for each point of the region of interest being considered. To generate the point spread function, a scalar diffraction theory calculation is used to image the wavefront of multiple points in the volume through the microlens array.
Step 4: constructing a training set and a test set based on the two-dimensional image acquired in the step 3 and the high-resolution three-dimensional image acquired in the step 2, taking the two-dimensional image as an input, taking the high-resolution three-dimensional image as an output, training the VCD-Net, performing stepwise training by iteratively minimizing the difference between the intermediate output and the reference high-resolution image, obtaining optimized kernel parameters for each layer by setting appropriate loss parameters such as the mean square error of pixel intensities, and effectively converging to one to obtain an optimal VCD-Net network model.
The final high resolution reconstructed image can be obtained by inputting a plurality of two-dimensional images acquired by the microscopy system (images acquired by caenorhabditis elegans in this example) into a trained VCD-Net network model.
The image output module is used for outputting the reconstructed high-resolution three-dimensional image.
Embodiment III:
the embodiment provides a microscopic imaging method based on deep learning and light field imaging, which is implemented by the microscopic imaging system based on the deep learning and the microlens array described in the second embodiment, and comprises the following steps:
step one: acquiring image data by adopting a microscope lens, inputting the image data into a plurality of camera sensors and a microlens array in a microscope system, and respectively performing wide-field imaging and light-field imaging to obtain a plurality of two-dimensional images;
step two: inputting the two-dimensional image obtained in the first step into a trained VCD-Net network model, and obtaining a reconstructed high-resolution three-dimensional image through the trained VCD-Net network model.
Some steps in the embodiments of the present invention may be implemented by using software, and the corresponding software program may be stored in a readable storage medium, such as an optical disc or a hard disk.
The foregoing description of the preferred embodiments of the invention is not intended to limit the invention to the precise form disclosed, and any such modifications, equivalents, and alternatives falling within the spirit and scope of the invention are intended to be included within the scope of the invention.
Claims (6)
1. A microscopic imaging system based on deep learning and light field imaging, the microscopic imaging system comprising, in order: the system comprises a microscope system, a deep learning network module and an image output module;
the microscopy system is used for acquiring a plurality of two-dimensional data of an image, and comprises: a microlens (1), a first dichroic mirror (2), a reflecting mirror (3), a beam splitter (4), a band-pass filter (5), a first camera sensor (6), a microlens array (7), a relay lens (8), a second dichroic mirror (9), a second camera sensor (10), and a third camera sensor (11);
the microscope lens (1) collects image data, interference light is filtered and removed through the first dichroic mirror (2), then the light enters the horizontal direction from the vertical direction through the reflecting mirror (3), and then part of the collected signals is gathered to the first camera sensor (6) for wide-field imaging through the beam splitter (4), so that wide-field imaging is performed; the other part respectively passes through the micro lens array (7), the relay lens (8) and the second dichroic mirror (9), and light field imaging is respectively carried out on the second camera sensor (10) and the third camera sensor (11);
the deep learning network module is used for reconstructing two-dimensional image data imaged by the first camera sensor (6), the second camera sensor (10) and the third camera sensor (11) into a high-resolution three-dimensional image;
the deep learning network module adopts a trained VCD deep network VCD-Net to reconstruct a high-resolution three-dimensional image;
the training process of VCD deep network VCD-Net comprises:
step 1: initializing the VCD-Net, comprising: network parameters and loss functions;
step 2: obtaining a high-resolution three-dimensional image from a copolymerization microscope of a real static sample and synthesized data;
step 3: constructing a fluctuation optical model, inputting the high-resolution three-dimensional image obtained in the step 2 into the fluctuation optical model, and outputting a corresponding two-dimensional image;
step 4: constructing a training set and a testing set based on the two-dimensional image obtained in the step 3 and the high-resolution three-dimensional image obtained in the step 2, taking the two-dimensional image as input, taking the high-resolution three-dimensional image as output, and training the VCD-Net until convergence to obtain an optimal VCD-Net network model;
the image output module is used for outputting the reconstructed high-resolution three-dimensional image.
2. The microscopic imaging system according to claim 1, wherein the first camera sensor (6), the second camera sensor (10) and the third camera sensor (11) employ sCMOS cameras.
3. The microscopic imaging system of claim 1, wherein the sample is light field imaged using 1:1, the relay lens (8) focuses the second camera sensor (10) and the third camera sensor (11) on the back focal plane of the microlens array (7).
4. The microscopic imaging system of claim 1, wherein the fluctuating optical model is:
F＝Hg
wherein, vector F represents the acquired original light field image, vector g represents the reconstructed 3D discrete point cloud of the object, and H is the point spread function matrix representation of the imaging process.
5. A microscopic imaging method based on deep learning and light field imaging, the microscopic imaging method comprising:
step one: acquiring image data by adopting a microscope lens, inputting the image data into a plurality of camera sensors in a microscope system and respectively performing wide-field imaging and light field imaging by a sensor formed by a microlens array and a CCD (charge coupled device) to obtain a plurality of two-dimensional images;
step two: inputting the two-dimensional image obtained in the first step into a trained deep neural network, and obtaining a reconstructed high-resolution three-dimensional image through the trained deep neural network;
the deep neural network adopts a VCD-Net network;
the training process of the VCD-Net network comprises the following steps:
step 1: initializing the VCD-Net, comprising: network parameters and loss functions;
step 2: obtaining a high-resolution three-dimensional image from a copolymerization microscope of a real static sample and synthesized data;
step 3: constructing a fluctuation optical model, inputting the high-resolution three-dimensional image obtained in the step 2 into the fluctuation optical model, and outputting a corresponding two-dimensional image;
step 4: and (3) constructing a training set and a testing set based on the two-dimensional image obtained in the step (3) and the high-resolution three-dimensional image obtained in the step (2), taking the two-dimensional image as input, taking the high-resolution three-dimensional image as output, and training the VCD-Net until convergence to obtain an optimal VCD-Net network model.
6. The microscopic imaging method of claim 5, wherein the microscopic system includes: a microlens (1), a first dichroic mirror (2), a reflecting mirror (3), a beam splitter (4), a band-pass filter (5), a first camera sensor (6), a microlens array (7), a relay lens (8), a second dichroic mirror (9), a second camera sensor (10), and a third camera sensor (11);
the microscope lens (1) collects image data, interference light is removed through filtering of the first dichroic mirror (2), the collected light enters the horizontal direction from the vertical direction through the reflecting mirror (3), and part of the collected signals are collected to the first camera sensor (6) for wide-field imaging through the beam splitter (4) to perform wide-field imaging; the other part respectively passes through the micro lens array (7), the relay lens (8) and the second dichroic mirror (9), and light field imaging is respectively carried out on the second camera sensor (10) and the third camera sensor (11).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210902191.5A CN115220211B (en) | 2022-07-29 | 2022-07-29 | Microscopic imaging system and method based on deep learning and light field imaging |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210902191.5A CN115220211B (en) | 2022-07-29 | 2022-07-29 | Microscopic imaging system and method based on deep learning and light field imaging |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115220211A CN115220211A (en) | 2022-10-21 |
| CN115220211B true CN115220211B (en) | 2024-03-08 |
Family
ID=83614063
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210902191.5A Active CN115220211B (en) | 2022-07-29 | 2022-07-29 | Microscopic imaging system and method based on deep learning and light field imaging |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115220211B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116609942B (en) * | 2023-07-18 | 2023-09-22 | 长春理工大学 | Sub-aperture compressed sensing polarization super-resolution imaging method |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106846463A (en) * | 2017-01-13 | 2017-06-13 | 清华大学 | Micro-image three-dimensional rebuilding method and system based on deep learning neutral net |
| JP2018194634A (en) * | 2017-05-16 | 2018-12-06 | オリンパス株式会社 | Light field microscope |
| CN109596588A (en) * | 2018-12-16 | 2019-04-09 | 华中科技大学 | A kind of high-resolution four-dimension light field micro imaging system based on mating plate illumination |
| CN110441271A (en) * | 2019-07-15 | 2019-11-12 | 清华大学 | Light field high-resolution deconvolution method and system based on convolutional neural networks |
| CN111105415A (en) * | 2019-12-31 | 2020-05-05 | 北京理工大学重庆创新中心 | White blood cell large-field-of-view image detection system and method based on deep learning |
| CN113383225A (en) * | 2018-12-26 | 2021-09-10 | 加利福尼亚大学董事会 | System and method for propagating two-dimensional fluorescence waves onto a surface using deep learning |
| CN114549318A (en) * | 2022-02-23 | 2022-05-27 | 复旦大学 | Ultra-high resolution fluorescence microscopic imaging method based on sub-voxel convolution neural network |
-
2022
- 2022-07-29 CN CN202210902191.5A patent/CN115220211B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106846463A (en) * | 2017-01-13 | 2017-06-13 | 清华大学 | Micro-image three-dimensional rebuilding method and system based on deep learning neutral net |
| JP2018194634A (en) * | 2017-05-16 | 2018-12-06 | オリンパス株式会社 | Light field microscope |
| CN109596588A (en) * | 2018-12-16 | 2019-04-09 | 华中科技大学 | A kind of high-resolution four-dimension light field micro imaging system based on mating plate illumination |
| CN113383225A (en) * | 2018-12-26 | 2021-09-10 | 加利福尼亚大学董事会 | System and method for propagating two-dimensional fluorescence waves onto a surface using deep learning |
| CN110441271A (en) * | 2019-07-15 | 2019-11-12 | 清华大学 | Light field high-resolution deconvolution method and system based on convolutional neural networks |
| CN111105415A (en) * | 2019-12-31 | 2020-05-05 | 北京理工大学重庆创新中心 | White blood cell large-field-of-view image detection system and method based on deep learning |
| CN114549318A (en) * | 2022-02-23 | 2022-05-27 | 复旦大学 | Ultra-high resolution fluorescence microscopic imaging method based on sub-voxel convolution neural network |
Non-Patent Citations (1)
| Title |
|---|
| 基于深度学习的荧光显微成像技术及应用;李浩宇 等;《激光与光电子学进展》;第58卷(第18期);第1811007-1页至第1811007-15页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115220211A (en) | 2022-10-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109580457B (en) | Three-dimensional diffraction chromatography microscopic imaging method based on LED array coding illumination | |
| US9426429B2 (en) | Scanning projective lensless microscope system | |
| Moon et al. | Automated three-dimensional identification and tracking of micro/nanobiological organisms by computational holographic microscopy | |
| CN110441271B (en) | Light field high-resolution deconvolution method and system based on convolutional neural network | |
| CN109884018B (en) | Submicron lens-free microscopic imaging method and system based on neural network | |
| CN104849852B (en) | Light field micro imaging system based on camera array and method | |
| Bishop et al. | Plenoptic depth estimation from multiple aliased views | |
| CN115144371A (en) | Lensless Fourier laminated diffraction tomography microscopic imaging method based on wavelength scanning | |
| CN115220211B (en) | Microscopic imaging system and method based on deep learning and light field imaging | |
| Li et al. | Quantitative phase imaging (QPI) through random diffusers using a diffractive optical network | |
| Huang et al. | Light-field reconstruction and depth estimation from focal stack images using convolutional neural networks | |
| Zhang et al. | Super-resolution generative adversarial network (SRGAN) enabled on-chip contact microscopy | |
| Nguyen et al. | Convolutional neural network for Fourier ptychography video reconstruction: learning temporal dynamics from spatial ensembles | |
| Macias-Garza et al. | Digital reconstruction of three-dimensional serially sectioned optical images | |
| CN110989155A (en) | Lens-free microscopic imaging device based on optical filter array and reconstruction method | |
| Sokol et al. | The Use of Digital Interferometry Devices to Analyze the State of Red Blood Cell Membranes | |
| Verinaz-Jadan et al. | Shift-invariant-subspace discretization and volume reconstruction for light field microscopy | |
| Kauvar et al. | Aperture interference and the volumetric resolution of light field fluorescence microscopy | |
| CN114118199A (en) | Image classification method and system for fault diagnosis of intelligent pump cavity endoscope | |
| JP7378164B2 (en) | Optical system and optical image processing method applying image restoration | |
| Kwon et al. | Image quality enhancement of 4D light field microscopy via reference impge propagation-based one-shot learning | |
| Guzzi et al. | A Deep Prior Method for Fourier Ptychography Microscopy | |
| Henrot et al. | Fast deconvolution of large fluorescence hyperspectral images | |
| Proppe et al. | 3d-2d neural nets for phase retrieval in noisy interferometric imaging | |
| CN113592716B (en) | Light field image space domain super-resolution method, system, terminal and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |