CN109884018B - Submicron lens-free microscopic imaging method and system based on neural network - Google Patents
Submicron lens-free microscopic imaging method and system based on neural network Download PDFInfo
- Publication number
- CN109884018B CN109884018B CN201910220004.3A CN201910220004A CN109884018B CN 109884018 B CN109884018 B CN 109884018B CN 201910220004 A CN201910220004 A CN 201910220004A CN 109884018 B CN109884018 B CN 109884018B
- Authority
- CN
- China
- Prior art keywords
- resolution
- image
- images
- training
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The invention discloses a submicron lens-free microscopic imaging method and system based on a neural network, which comprises the following steps: under the condition of coherent illumination, obtaining a hologram corresponding to interference fringes generated by an object to be imaged through a non-transparent microscope; performing phase recovery based on the hologram of the object to be imaged, and acquiring two-dimensional amplitude images of different cross sections of the object to be imaged at a first resolution and two-dimensional phase images of the first resolution; and inputting the two-dimensional amplitude image with the first resolution and the two-dimensional phase image with the first resolution into a trained generation countermeasure network to obtain the two-dimensional amplitude image with the second resolution and the two-dimensional phase image with the second resolution, wherein the second resolution is greater than the first resolution. The invention solves the problem of insufficient resolution of the lens-free microscope by utilizing the neural network, can strengthen the resolution of any cell image, allows the lens-free imaging system to break through the limitation of the pixel size and realizes the submicron resolution.
Description
Technical Field
The invention relates to the technical field of biomedical microscopic imaging, in particular to a submicron lens-free microscopic imaging method and system based on a neural network.
Background
In the instant biomedical detection process, morphological information of micro samples such as cells and microorganisms and fluorescence image information after biochemical reactions such as nucleic acids and antigen antibodies are often analyzed, and the acquisition of the information usually requires microscopic imaging equipment. The traditional optical microscope mainly comprises a light source, an optical lens and a light detector. The main function of the optical lens is to optically magnify the sample and focus it on the photodetector for imaging. However, the optical lens usually needs to be used in cooperation with components such as an optical lens barrel, an aperture, a focusing system and the like to obtain a clear image, so that the volume and complexity of the microscope are greatly increased, and the optical lens becomes a large resistance which must be overcome when the microscope is used in the field of instant detection.
Compared with the traditional microscope, the lensless microscope is taken as a novel microscope system, and is favored in the biomedical field by the characteristics of large visual field and simplicity. The lens-less imaging technique is a technique in which a sample is brought into close contact with a photodetector such as a charge-coupled device (CCD) or a Complementary Metal Oxide Semiconductor (CMOS) chip, and the sample is directly imaged without using an optical element. According to different imaging principles, the lens-free imaging technology is divided into two types, namely shadow imaging and digital holographic imaging, and the two types of lens-free imaging technologies have advantages and disadvantages respectively. The structure of the shadow lens-free imaging system is the simplest, incoherent light is adopted for illumination, the optical principle of light along linear propagation is utilized, the target information is obtained by acquiring the projection of a sample through a light detector, and the ratio of the obtained image to the original sample is close to 1; the digital holographic lens-free imaging system adopts coherent light illumination, interference fringes are generated through the interference process of the light, and the interference fringes are received by a light detector so as to acquire spatial three-dimensional information of a sample.
However, the current image sensor can only receive light intensity information but cannot receive light phase information, and in biomedical applications, transparent or semitransparent samples such as cells are often required to be observed, the transmittance distribution of the samples is relatively uniform, and the intensity information change is small after light penetrates through the samples, so that high-quality sample images are difficult to obtain only through light intensity detection. Contact lensless microscopes are therefore not suitable for this type of detection. However, the change of the refractive index of the sample affects the optical path of the light passing through the sample, so that the phase information of the light is obviously changed, and the properties of the sample can be directly and accurately reflected by detecting the phase information of the light. The phase information of the light is key to restoring high resolution details of the image. The digital holographic lens-free imaging system can meet the requirement of the biomedical real-time detection application, interference fringes are generated by coherent superposition of reference light waves and light waves scattered by an object, the interference fringes are recorded into a hologram by a CMOS or CCD, an original light field can be restored by a phase iterative recovery algorithm, and the physical form characteristics of a sample are obtained by the restored amplitude and phase information.
The resolution of both current lens-less imaging systems is limited by the pixel size of the photodetector CMOS or CCD. The smaller the picture element size theoretically, the more subtle the information that can be received. The pixel size of the CMOS or CCD which is common on the market at present is in the order of mum, and the resolution of a common optical microscope is generally close to the diffraction limit, namely hundreds of nanometers. In addition, due to the design of the photodetector, the sensor surface of a CCD or CMOS that can acquire a color pattern has a layer of bayer filters, each of which has a size of 3 × 3, and the removal of this bayer filter can improve the resolution, but, on the contrary, the photosensor also loses its ability to acquire color images.
Disclosure of Invention
Aiming at the defects of the prior art, the invention aims to solve the technical problem that the resolution of the image acquired by the existing lens-free imaging technology is low.
In order to achieve the above object, in one aspect, the present invention provides a neural network-based submicron lens-free microscopic imaging method, including the following steps:
under the condition of coherent illumination, obtaining a hologram corresponding to interference fringes generated by an object to be imaged;
performing phase recovery based on the hologram of the object to be imaged, and acquiring two-dimensional amplitude images of different cross sections of the object to be imaged at a first resolution and two-dimensional phase images of the first resolution;
inputting the two-dimensional amplitude image of the first resolution and the two-dimensional phase image of the first resolution into a trained antagonistic network (GAN) to obtain a two-dimensional amplitude image of a second resolution and a two-dimensional phase image of a second resolution, wherein the second resolution is greater than the first resolution.
Specifically, the trained generation countermeasure network GAN is obtained by mapping and training a low-resolution image and a corresponding high-resolution image, which are acquired in advance, and is used for learning a mapping relationship between the low-resolution image and the corresponding high-resolution image, and restoring the received low-resolution image based on the mapping relationship to obtain the corresponding high-resolution image.
Specifically, the first resolution is a low resolution, and the second resolution is a high resolution, and the high resolution is typically a submicron level.
Optionally, the lens-free microscopic imaging method further comprises the following steps:
performing multi-angle coherent illumination on the object to be imaged to obtain two-dimensional amplitude images with different cross sections of the object to be imaged at different angles and two-dimensional phase images with different first resolutions;
and carrying out biaxial compensation on the two-dimensional amplitude images with the first resolution and the two-dimensional phase images with the first resolution of different cross sections of the object to be imaged at different angles to obtain the spatial frequency information lost in the multi-angle coherent illumination, and recovering the three-dimensional image of the object to be imaged.
Optionally, the generation of the countermeasure network is trained by:
pre-acquiring a high-resolution image of an object for training;
carrying out fuzzy processing on the acquired high-resolution image to obtain a corresponding low-resolution image for training; the high-resolution images of the object used for training and the corresponding low-resolution images form a training set;
determining a generating countermeasure network architecture comprising a generator architecture for generating image details in the second resolution image and a discriminator architecture for judging the authenticity of the image details generated by the generator architecture;
training a generated countermeasure network architecture by adopting the training set, learning the mapping relation between the low-resolution images and the corresponding high-resolution images to obtain a trained generated countermeasure network, dividing the training set into training data and test data, training and learning the generated countermeasure network architecture by utilizing the training data, and evaluating by utilizing the test data until the generated countermeasure network architecture has the capacity of establishing mapping from the low-resolution images to the high-resolution images.
Optionally, the blurring process is performed on the acquired high-resolution image, specifically by using the following formula:
Im＝D(K*I)+N
wherein, ImIs an intensity distribution matrix of a high resolution image of the object, I denotes the intensity distribution matrix of the corresponding low resolution image; k is the point spread function of the optical system, expressed as a Gaussian convolution kernel; is the convolution between I and K; d, acting on the convolution result, representing discretization of the camera sensor; n represents additive white gaussian noise.
In another aspect, the present invention provides a neural network-based submicrometer lensless microscopy imaging system, comprising:
an illumination unit for providing coherent illumination light;
the detection imaging unit is used for obtaining a hologram corresponding to interference fringes generated by an object to be imaged based on the coherent illumination light;
the phase recovery unit is used for carrying out phase recovery on the basis of the hologram of the object to be imaged, and acquiring two-dimensional amplitude images with first resolution and two-dimensional phase images with first resolution of different cross sections of the object to be imaged;
and the high-resolution recovery unit is used for inputting the two-dimensional amplitude image with the first resolution and the two-dimensional phase image with the first resolution into the trained generation countermeasure network to obtain the two-dimensional amplitude image with the second resolution and the two-dimensional phase image with the second resolution, wherein the second resolution is greater than the first resolution.
Optionally, the lensless microscopic imaging system further comprises a moving unit and a three-dimensional image restoration unit;
the moving unit is used for moving the illumination unit to carry out multi-angle coherent illumination on the object to be imaged to obtain two-dimensional amplitude images with different cross sections of the object to be imaged at different angles and two-dimensional phase images with different first resolutions;
the three-dimensional image recovery unit is used for carrying out double-axis compensation on the two-dimensional amplitude images with the first resolution and the two-dimensional phase images with the first resolution of different cross sections of the object to be imaged at different angles to obtain lost spatial frequency information in multi-angle coherent illumination, and recovering the three-dimensional image of the object to be imaged.
Optionally, the lensless microscopic imaging system further comprises: generating a confrontation network training unit;
the generation countermeasure network training unit is used for acquiring a high-resolution image of an object for training in advance; carrying out fuzzy processing on the acquired high-resolution image to obtain a corresponding low-resolution image for training; the high-resolution images of the object used for training and the corresponding low-resolution images form a training set; determining a generating countermeasure network architecture comprising a generator architecture for generating image details in the second resolution image and a discriminator architecture for discriminating authenticity of image details generated by the generator architecture; training a generated countermeasure network architecture by adopting the training set, learning the mapping relation between the low-resolution images and the corresponding high-resolution images to obtain a trained generated countermeasure network, dividing the training set into training data and test data, training and learning the generated countermeasure network architecture by utilizing the training data, and evaluating by utilizing the test data until the generated countermeasure network architecture has the capacity of establishing mapping from the low-resolution images to the high-resolution images.
Optionally, the generation countermeasure network training unit performs a blurring process on the acquired high-resolution image, and is specifically implemented by the following formula:
Im＝D(K*I)+N
wherein, ImIs an intensity distribution matrix of a high resolution image of the object, I denotes the intensity distribution matrix of the corresponding low resolution image; k is the point spread function of the optical system, expressed as a Gaussian convolution kernel; is the convolution between I and K; d, acting on the convolution result, representing discretization of the camera sensor; n represents additive white gaussian noise.
Generally, compared with the prior art, the above technical solution conceived by the present invention has the following beneficial effects:
the invention provides a submicron lens-free microscopic imaging method and system based on a neural network, which do not need any lens or high-power laser unit, have simple and portable integral structure, can simultaneously realize high resolution and large visual field and can reconstruct a phase image of a cell.
The invention uses the neural network to solve the problem of insufficient resolution of the lens-free microscope, can strengthen the resolution of any cell image, and allows the lens-free imaging system to break through the limitation of the pixel size and realize the submicron resolution.
The multi-angle illumination method provided by the invention can realize three-dimensional reconstruction of imaging cells, and the reconstructed axial resolution is in the micrometer scale, so that cells which are overlapped with each other can be distinguished, and a cell sample with higher density can be imaged.
The submicron lens-free microscopic imaging method and system based on the neural network, provided by the invention, can be further applied to the field of biomedicine while solving the problems.
Drawings
FIG. 1 is a flow chart of a method for providing a neural network-based submicrometer lensless microscopic imaging in accordance with the present invention;
FIG. 2 is a schematic diagram of an architecture for generating an antagonistic neural network according to the present invention;
FIG. 3 illustrates the imaging effect of the lens-free imaging system provided by the present invention;
FIG. 4 is a flow chart of a neural network training method provided by the present invention;
FIG. 5a is a diagram of a raw phase image recovered from a hologram without neural network processing according to an embodiment of the present invention;
fig. 5b is a diagram of the final effect obtained by using the method of increasing the resolution according to the embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In addition, the technical features involved in the embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
Aiming at the defects or shortcomings, the invention provides a solution and a corresponding system building method: aiming at the defect that the pixel size is not small enough, the invention bypasses the limitation of the pixel size by means of software, and utilizes a neural network, in particular, a generation antagonistic network GAN can be developed to recover a high-resolution lensless image from a single low-resolution measurement, thereby realizing the resolution of submicron level on the whole sensor plane. Experimentally, our lensless system can capture lensless images of cultured cells quickly, and then input a trained generative countermeasure network GAN for super-resolution recovery with real-time efficiency, and this imaging method based on deep learning can recover images with a large field of view at an extremely fast speed (within 1 second), with an area of the field of view of approximately 95 mm square, increasing the resolution of about 1.7 μm, without changing the settings of the existing microscope.
The application proposes a method for removing the bayer filter on the sensor surface, whereby a grey-scale map of the original light field can be obtained with higher resolution. The present application can also recover from multi-wavelength illumination due to the ability to acquire color images lost from the removal of the bayer filter.
In addition, the application provides a lens-free microscope system capable of constructing a three-dimensional image of an imaged object, spatial frequency information lost in multi-angle illumination is obtained by multi-angle illumination and double-axis compensation, and the system can restore the three-dimensional image of the original object through back projection calculation.
Fig. 1 is a flow chart of a submicron lens-free microscopic imaging method based on a neural network, which comprises the following steps:
s100, under the condition of coherent illumination, obtaining a hologram corresponding to interference fringes generated by an object to be imaged through a non-transparent microscope;
s200, performing phase recovery based on the hologram of the object to be imaged, and acquiring two-dimensional amplitude images of different cross sections of the object to be imaged at a first resolution and two-dimensional phase images of the first resolution;
s300, inputting the two-dimensional amplitude image with the first resolution and the two-dimensional phase image with the first resolution into a trained GAN to obtain a two-dimensional amplitude image with the second resolution and a two-dimensional phase image with the second resolution, wherein the second resolution is higher than the first resolution, the trained generation countermeasure network GAN is obtained by mapping and training a low-resolution image and a corresponding high-resolution image which are acquired in advance, and is used for learning a mapping relation between the low-resolution image and the corresponding high-resolution image, and restoring the received low-resolution image based on the mapping relation to obtain the corresponding high-resolution image.
Specific details of the individual steps can be found in the detailed description of the examples below.
The invention aims to develop a lens-free microscope, the resolution of which can reach submicron level and even approach the diffraction limit.
The invention provides a low-cost lens-free coherent microscope system, which comprises the following components from top to bottom: illumination unit, filtering unit, detection imaging unit:
and the lighting unit can directly use a common or narrow-band LED light source. The illumination unit also comprises a filtering unit, the illumination light of the filtering unit is converted into coherent illumination light, the spatial domain filtering and the frequency domain filtering are included, a 50um-100um magnitude aperture is used as a spatial filter, and a filter with the bandwidth of +/-10 or less is placed below the aperture for band-pass filtering. Note that the aperture and filter should be close together, and the distance from the illumination unit needs to be adjusted properly to avoid additional interference patterns being collected on the detection imaging unit due to additional diffracted light generated by the distance between the two.
The detection imaging unit can be a common CMOS or CCD on the market, and the pixel size is 2.2um or below.
And the phase recovery unit is used for carrying out phase recovery on the basis of the hologram of the object to be imaged, and acquiring the two-dimensional amplitude image with the first resolution and the two-dimensional phase image with the first resolution of different cross sections of the object to be imaged.
The high-resolution recovery unit is used for inputting the two-dimensional amplitude image with the first resolution and the two-dimensional phase image with the first resolution into the trained antagonistic network GAN to obtain a two-dimensional amplitude image with the second resolution and a two-dimensional phase image with the second resolution, wherein the second resolution is greater than the first resolution; the trained generation countermeasure network GAN is obtained by mapping and training a low-resolution image and a corresponding high-resolution image which are acquired in advance, and is used for learning a mapping relation between the low-resolution image and the corresponding high-resolution image, and restoring the received low-resolution image based on the mapping relation to obtain the corresponding high-resolution image.
And the moving unit is used for moving the illumination unit so as to carry out multi-angle coherent illumination on the object to be imaged to obtain two-dimensional amplitude images with first resolution and two-dimensional phase images with first resolution of different cross sections of the object to be imaged at different angles.
And the three-dimensional image recovery unit is used for carrying out double-axis compensation on the two-dimensional amplitude images with the first resolution and the two-dimensional phase images with the first resolution of different cross sections of the object to be imaged at different angles to obtain lost spatial frequency information in multi-angle coherent illumination and recover the three-dimensional image of the object to be imaged.
A GAN training unit which previously acquires a high-resolution image of an object for training; carrying out fuzzy processing on the acquired high-resolution image to obtain a corresponding low-resolution image for training; the high-resolution images of the object used for training and the corresponding low-resolution images form a training set; determining a GAN architecture comprising a generator structure for generating image details in the second resolution image and a discriminator structure for discriminating authenticity of the image details generated by the generator structure; and training the GAN framework by adopting the training set, learning the mapping relation between the low-resolution images and the corresponding high-resolution images to obtain a trained GAN network, wherein the training set is divided into training data and test data, the GAN framework performs training learning by utilizing the training data and performs evaluation by utilizing the test data until the GAN framework has the capacity of establishing mapping from the low-resolution images to the high-resolution images.
The culture dish is directly placed on a CMOS or CCD, an illumination light source is turned on, interference fringes generated by the cells under coherent illumination are collected, and the interference images are called coaxial holograms. Then, the iterative algorithm is used for eliminating the double-image artifact, and the phase recovery processing is carried out, so that the light intensity and phase information passing through the cell can be reconstructed, and the amplitude and phase images are obtained.
The invention provides an optimization method for improving the resolution of an acquired image through a deep learning network, which comprises the following steps:
the cells were sampled beforehand using a high-resolution optical microscope, and a large number of high-resolution cell images were obtained. The acquired high-resolution images are down-sampled, the high-resolution images are blurred into low-resolution images, and then the two groups of corresponding image sets are used as training sets for generating a countermeasure network (GAN network) for training, so that the capability of mapping the low-resolution images to the high-resolution image reconstruction is established. Depending on such a neural network, we can recover a high resolution image from an arbitrary low resolution image.
The method also requires the following steps:
1. acquiring a proper cell original image in advance, and carrying out blurring processing on the acquired high-resolution image:
an image down-sampling model is used for realizing the transformation from a High Resolution (HR) image to a Low Resolution (LR) image, and the image mapping of LR-HR is used for training a GAN network, so that the GAN network has the capability of restoring an original High Resolution image from the Low Resolution image. It should be noted that, in order to ensure that the trained model can accurately restore the original image acquired by the lens-free system, the image down-sampling model designed by the inventor must be capable of generating a low-resolution image very close to the lens-free image.
The down-sampling model of a conventional optical microscope system is:
Im＝D(K*I)+N
wherein ImIs the continuous true intensity distribution of the sample to be imaged; k is the point spread function of the optical system, expressed as a Gaussian convolution kernel; is the convolution between I and K; d, acting on the convolution result, representing discretization of the camera sensor; n represents additive white gaussian noise. Among these parameters, white gaussian noise is mainly caused by statistical thermal noise of CCD/CMOS sensors; i ismIs our image obtained by an optical microscope, which is actually a down-sampling of I. There are two parameters to optimize for this equation: the size of the gaussian kernel and the variance of the noise distribution in the convolution step.
The image collected by the high-power optical objective lens (X10) is approximate to the actual continuous image of the sample, then Gaussian filtering downsampling is carried out on the image by using different Sigma values, and the downsampled image is compared with the lens-free original image to determine the optimal Sigma value, namely the size of the optimal Gaussian mask. And then, Gaussian white noise with different variance values is added to the down-sampled image, and the optimal noise variance is determined by comparing the original image without the lens. After these two key parameters are determined, the down-sampling model can generate a low-resolution image that is close enough to the raw image without the lens.
2. Establishing a framework for generating the countermeasure network GAN by utilizing a computer:
referring to fig. 2: architecture of GAN networks.
Part a is the structure of the generator, conv and resnet are abbreviations for the convolutional layers and the remaining network blocks. The parameters of the convolutional layer are given in the format "k-s-n", where k is the kernel size, s is the stride, and n is the number of profiles (i.e., the output channels of the layer). The depth of each convolutional layer roughly represents the number of its feature maps, and the lateral dimension represents the size of its input. It has 16 residual blocks.
B is the structure of the discriminator. Each convolutional layer in the discriminator is a combination of convolutional layer, batch normalization operation, and ReLU activation function. After the architecture of the neural network is implemented in any programming language, the training set obtained in step 1 is divided into two parts, one part is used as training data, and the other part is used as test data, and the neural network is trained and evaluated until the neural network has the capability of establishing the mapping from low resolution to high resolution images.
3. Inputting the image with the need of improving the resolution into the neural network can obtain a clearer image. Note that the source of this image: the primitive cells must belong to a certain class of reference cells used in the training process.
The invention provides a multi-angle illumination imaging method capable of constructing a three-dimensional image of an object, which needs to modify an illumination unit of a lens-free microscope system provided by the application. Specifically, the angle of the illumination source needs to be changed while ensuring that the source-sample distance is constant, and generally, the illumination source needs to be moved within a range of ± 50 °, the projections of cells at different heights will be shifted to different degrees on the CMOS plane, and the series of images will contain the depth information of the cells. After we remove the dual images from these original holograms using an iterative algorithm and perform a phase recovery process, they can reconstruct amplitude and phase images of different cross-sections of the sample volume. This solution requires and, as a result of this numerical reconstruction process, we can distinguish overlapping lensless holograms of cells, thereby increasing the density of cells we can use.
The invention aims to provide a microscopic imaging system which has low cost and simple and portable device, can simultaneously realize high resolution and large field of view and can realize real-time display of three-dimensional information, and provides a plurality of methods which can be simultaneously used for improving the resolution of the system on the basis of the system, so that the performance of the system is close to that of an optical microscope, but the optical aberration caused by any lens is avoided and the advantages of a microscope without the lens are reserved.
The invention provides a schematic diagram of a lens-free microscopic imaging method, which comprises the following steps:
and shielding ambient light and powering on the LED light source.
Placing the filtering unit in front of the LED unit, and adjusting the LED light source to be close to the small hole of the spatial filter as much as possible;
placing the sample above the image processing unit;
and connecting the image processing unit to a computer to acquire an image.
The original image is obtained through the steps, the double images are eliminated through an iterative algorithm, the phase recovery processing is carried out, the originally shot cell image can be obtained through reduction from the stripe, and the three schemes provided by the application can be used for strengthening the resolution.
The following is a specific embodiment provided by the present invention:
example 1
The embodiment of the invention provides a neural network processing method for improving resolution, which is used for enhancing a final image obtained by lensless microscopic imaging. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The embodiment of the invention uses an imaging sensor (model: Aptina MT9P031), the space size of which is 2.2 μm, the effective area of which is 5.7mm multiplied by 4.28mm, and the imaging sensor is used for a light sensor part without lens imaging, and the near-field light signals of cells cultured on the surface of the sensor are digitized. Firstly, turning on an LED light source, carrying out filtering illumination, placing a sample on a sensor plane, starting a CMOS (complementary metal oxide semiconductor) drive, and transmitting an obtained digital image to a computer through a USB (universal serial bus) data line. And eliminating double image artifacts generated by interference by using an iterative algorithm on a computer, and performing phase recovery processing according to a Rayleigh-Sovifili diffraction formula to obtain a cell pattern recovered from diffraction fringes. We use the recovered cell pattern as input to the neural network to obtain a high resolution output pattern. Fig. 3 shows the imaging effect of the lens-less imaging system provided by the present invention, as shown in fig. 3, the image is an original hologram which is not processed by the coherence recovery algorithm, and the phase diagram and the intensity diagram of the original object can be obtained after the original hologram is recovered by the coherence recovery algorithm.
To do this, we first need to create a trained GAN network, and the training process is as follows in fig. 4:
several High Resolution (HR) images of the cells were first obtained using a conventional high magnification microscope (fig. 4a, step 1). By reproducing an image degradation model of the transfer function of the lensless imaging process, a simulated Low Resolution (LR) image is generated whose image details are similar to real experimental images acquired by a lensless microscope (fig. 4a, step 2). Targeting the high resolution image and the low resolution image LR simulation as input, the generative countermeasure network (GAN) is made to iteratively learn the mapping from the low resolution image to its corresponding high resolution target until the quality of the output image tested is sufficiently close to the image resolution that the optical microscope can output (fig. 4a, step 3). This trained GAN can then be used to make super-resolution inferences about the original lensless microscope image. Experimentally, our portable on-chip device can quickly capture lens-free images of cultured cells (fig. 4b, step 1) and then input to a trained GAN network for super-resolution recovery with real-time efficiency (fig. 4b, step 2). The GAN network can rapidly output super-resolution lens-free images in less than one second. Thus, this GAN-enabled contact image preserves a large field of view from unit magnification measurements, while recovering the high resolution details originally destroyed by sensor pixel discretization.
Specifically, fig. 5a is a low-resolution image without being processed by a neural network, fig. 5b is a high-resolution image after being processed by the neural network, three areas of Hela cells in the culture dish are selected and observed in stages, and the three areas are sampled and restored in real time after 3h, 6h, 9h, 12h, 15h, 18h and 21h after the culture is started, so that the observed images are obtained. It can be seen that the resolution of the three small patches obtained by the recovery is significantly improved, in particular already in the sub-micron order, as can be seen from fig. 5b compared to the overall pattern provided by fig. 5 a.
It will be understood by those skilled in the art that the foregoing is only a preferred embodiment of the present invention, and is not intended to limit the invention, and that any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (6)
1. A submicron lens-free microscopic imaging method based on a neural network is characterized by comprising the following steps:
under the condition of coherent illumination, obtaining a hologram corresponding to interference fringes generated by an object to be imaged;
performing phase recovery based on the hologram of the object to be imaged, and acquiring two-dimensional amplitude images of different cross sections of the object to be imaged at a first resolution and two-dimensional phase images of the first resolution;
inputting the two-dimensional amplitude image with the first resolution and the two-dimensional phase image with the first resolution into a trained generation countermeasure network to obtain a two-dimensional amplitude image with a second resolution and a two-dimensional phase image with the second resolution, wherein the second resolution is greater than the first resolution;
performing multi-angle coherent illumination on the object to be imaged to obtain two-dimensional amplitude images with different cross sections of the object to be imaged at different angles and two-dimensional phase images with different first resolutions;
and carrying out biaxial compensation on the two-dimensional amplitude images with the first resolution and the two-dimensional phase images with the first resolution of different cross sections of the object to be imaged at different angles to obtain the spatial frequency information lost in the multi-angle coherent illumination, and recovering the three-dimensional image of the object to be imaged.
2. The lensless microscopy imaging method of claim 1, wherein the generation of the countermeasure network is trained by:
pre-acquiring a high-resolution image of an object for training;
carrying out fuzzy processing on the acquired high-resolution image to obtain a corresponding low-resolution image for training; the high-resolution images of the object used for training and the corresponding low-resolution images form a training set;
determining a generating countermeasure network architecture comprising a generator architecture for generating image details in the second resolution image and a discriminator architecture for judging the authenticity of the image details generated by the generator architecture;
training a generated countermeasure network architecture by adopting the training set, learning the mapping relation between the low-resolution images and the corresponding high-resolution images to obtain a trained generated countermeasure network, dividing the training set into training data and test data, training and learning the generated countermeasure network architecture by utilizing the training data, and evaluating by utilizing the test data until the generated countermeasure network architecture has the capacity of establishing mapping from the low-resolution images to the high-resolution images.
3. The lensless microscopy imaging method according to claim 2, wherein the blurring of the acquired high resolution image is performed by the following formula:
Im＝D(K*I)+N
wherein, ImIs an intensity distribution matrix of a high resolution image of the object, I denotes the intensity distribution matrix of the corresponding low resolution image; k is the point spread function of the optical system, expressed as a Gaussian convolution kernel; is the convolution between I and K; d, acting on the convolution result, representing discretization of the camera sensor; n represents additive white gaussian noise.
4. A neural network-based sub-micron lensless microscopy imaging system, comprising:
an illumination unit for providing coherent illumination light;
the detection imaging unit is used for obtaining a hologram corresponding to interference fringes generated by an object to be imaged based on the coherent illumination light;
the phase recovery unit is used for carrying out phase recovery on the basis of the hologram of the object to be imaged, and acquiring two-dimensional amplitude images with first resolution and two-dimensional phase images with first resolution of different cross sections of the object to be imaged;
the high-resolution recovery unit is used for inputting the two-dimensional amplitude image with the first resolution and the two-dimensional phase image with the first resolution into the trained generation countermeasure network to obtain a two-dimensional amplitude image with the second resolution and a two-dimensional phase image with the second resolution, wherein the second resolution is greater than the first resolution;
the moving unit is used for moving the illuminating unit so as to carry out multi-angle coherent illumination on the object to be imaged to obtain two-dimensional amplitude images with first resolution and two-dimensional phase images with first resolution of different cross sections of the object to be imaged at different angles;
and the three-dimensional image recovery unit is used for carrying out double-axis compensation on the two-dimensional amplitude images with the first resolution and the two-dimensional phase images with the first resolution of different cross sections of the object to be imaged at different angles to obtain lost spatial frequency information in multi-angle coherent illumination and recover the three-dimensional image of the object to be imaged.
5. The lensless microscopy imaging system of claim 4, further comprising: generating a confrontation network training unit;
the generation countermeasure network training unit is used for acquiring a high-resolution image of an object for training in advance; carrying out fuzzy processing on the acquired high-resolution image to obtain a corresponding low-resolution image for training; the high-resolution images of the object used for training and the corresponding low-resolution images form a training set; determining a generating countermeasure network architecture comprising a generator architecture for generating image details in the second resolution image and a discriminator architecture for discriminating authenticity of image details generated by the generator architecture; training a generated countermeasure network architecture by adopting the training set, learning the mapping relation between the low-resolution images and the corresponding high-resolution images to obtain a trained generated countermeasure network, dividing the training set into training data and test data, training and learning the generated countermeasure network architecture by utilizing the training data, and evaluating by utilizing the test data until the generated countermeasure network architecture has the capacity of establishing mapping from the low-resolution images to the high-resolution images.
6. The lensless microscopic imaging system of claim 5, wherein the generation countermeasure network training unit performs blurring on the acquired high-resolution image, and is implemented by the following formula:
Im＝D(K*I)+N
wherein, ImIs an intensity distribution matrix of a high resolution image of the object, I denotes the intensity distribution matrix of the corresponding low resolution image; k is the point spread function of the optical system, expressed as a Gaussian convolution kernel; is the convolution between I and K; d, acting on the convolution result, representing discretization of the camera sensor; n represents additive white gaussian noise.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910220004.3A CN109884018B (en) | 2019-03-22 | 2019-03-22 | Submicron lens-free microscopic imaging method and system based on neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910220004.3A CN109884018B (en) | 2019-03-22 | 2019-03-22 | Submicron lens-free microscopic imaging method and system based on neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109884018A CN109884018A (en) | 2019-06-14 |
| CN109884018B true CN109884018B (en) | 2020-09-18 |
Family
ID=66933599
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910220004.3A Active CN109884018B (en) | 2019-03-22 | 2019-03-22 | Submicron lens-free microscopic imaging method and system based on neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109884018B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110308547B (en) * | 2019-08-12 | 2021-09-07 | 青岛联合创智科技有限公司 | Dense sample lens-free microscopic imaging device and method based on deep learning |
| CN111221118B (en) * | 2020-02-26 | 2022-06-28 | 南京理工大学 | Microscopic imaging method based on phase coding single lens |
| JP2023519423A (en) * | 2020-04-08 | 2023-05-10 | ビージーアイ シェンチェン | LENSLESS MICROSCOPY IMAGING SYSTEM AND METHOD AND BIOCHEMICAL DETECTION SYSTEM AND METHOD |
| CN111652815B (en) * | 2020-05-26 | 2023-05-05 | 浙江大学 | Mask plate camera image restoration method based on deep learning |
| CN112130306B (en) * | 2020-09-17 | 2022-02-18 | 浙江大学山东工业技术研究院 | CMOS holographic microscopic imaging device and method applied to cell segmentation |
| CN112461360B (en) * | 2020-10-26 | 2021-10-12 | 北京理工大学 | High-resolution single photon imaging method and system combined with physical noise model |
| CN112200726B (en) * | 2020-10-29 | 2023-04-07 | 陈根生 | Urinary sediment visible component detection method and system based on lensless microscopic imaging |
| CN112798504B (en) * | 2020-12-07 | 2022-06-07 | 西安电子科技大学 | Large-field-of-view high-throughput flow cytometry analysis system and analysis method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107101943A (en) * | 2017-05-18 | 2017-08-29 | 大连海事大学 | A kind of optofluidic is without lens holographic imaging activity of microalgae detection means and method |
| CN107154023A (en) * | 2017-05-17 | 2017-09-12 | 电子科技大学 | Face super-resolution reconstruction method based on generation confrontation network and sub-pix convolution |
| WO2018023039A1 (en) * | 2016-07-29 | 2018-02-01 | William Marsh Rice University | Lensless imaging device for microscopy and fingerprint biometric |
| CN108051930A (en) * | 2017-12-29 | 2018-05-18 | 南京理工大学 | Big visual field super-resolution dynamic phasing is without lens microscopic imaging device and reconstructing method |
| CN108508588A (en) * | 2018-04-23 | 2018-09-07 | 南京大学 | A kind of multiple constraint information without lens holographic microphotography phase recovery method and its device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10706258B2 (en) * | 2017-02-22 | 2020-07-07 | University Of Connecticut | Systems and methods for cell identification using lens-less imaging |
-
2019
- 2019-03-22 CN CN201910220004.3A patent/CN109884018B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018023039A1 (en) * | 2016-07-29 | 2018-02-01 | William Marsh Rice University | Lensless imaging device for microscopy and fingerprint biometric |
| CN107154023A (en) * | 2017-05-17 | 2017-09-12 | 电子科技大学 | Face super-resolution reconstruction method based on generation confrontation network and sub-pix convolution |
| CN107101943A (en) * | 2017-05-18 | 2017-08-29 | 大连海事大学 | A kind of optofluidic is without lens holographic imaging activity of microalgae detection means and method |
| CN108051930A (en) * | 2017-12-29 | 2018-05-18 | 南京理工大学 | Big visual field super-resolution dynamic phasing is without lens microscopic imaging device and reconstructing method |
| CN108508588A (en) * | 2018-04-23 | 2018-09-07 | 南京大学 | A kind of multiple constraint information without lens holographic microphotography phase recovery method and its device |
Non-Patent Citations (4)
| Title |
|---|
| Generative adversarial network(GAN)enabled on-chip contact microscopy;Xiongchao Chen et al.;《bioRxiv a license》;20181126;第1页右栏-第3页 * |
| High -throughput, high-resolution deep learning microscopy based on registrationfree generative adversarial network;Hao Zhang et al.;《Biomedical optics express》;20190204;第10卷(第3期);第106-108页，2 Results * |
| Synthetic aperture-based on-chip microscopy;Wei Luo et al.;《Light：Science& Applications》;20150327;第4卷;第2-3页 * |
| 无透镜显微成像技术在即时检测中的应用进展;李聪慧等;《中国激光》;20180228;第45卷(第2期);0207018-1至10 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109884018A (en) | 2019-06-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109884018B (en) | Submicron lens-free microscopic imaging method and system based on neural network | |
| US10769765B2 (en) | Imaging systems and methods of using the same | |
| McLeod et al. | Unconventional methods of imaging: computational microscopy and compact implementations | |
| Guo et al. | Fourier ptychography for brightfield, phase, darkfield, reflective, multi-slice, and fluorescence imaging | |
| US9426429B2 (en) | Scanning projective lensless microscope system | |
| US9892812B2 (en) | Fourier ptychographic x-ray imaging systems, devices, and methods | |
| JP2011523057A (en) | System and method for generating an optically sectioned image using both structured and uniform illumination | |
| JP2022507259A (en) | Systems and methods for converting holographic microscopic images into microscopic images of various modality | |
| Guo et al. | Lensfree auto-focusing imaging using nuclear norm of gradient | |
| CN110989155B (en) | Lens-free microscopic imaging device based on optical filter array and reconstruction method | |
| CN115220211B (en) | Microscopic imaging system and method based on deep learning and light field imaging | |
| Maalouf | Contribution to fluorescence microscopy, 3D thick samples deconvolution and depth-variant PSF | |
| CN112288633A (en) | Novel sub-pixel resolution diffraction imaging method | |
| Wu et al. | Three-dimensional identification of microorganisms using a digital holographic microscope | |
| Cossairt et al. | Compressive reconstruction for 3d incoherent holographic microscopy | |
| Proppe et al. | 3d-2d neural nets for phase retrieval in noisy interferometric imaging | |
| JP7378164B2 (en) | Optical system and optical image processing method applying image restoration | |
| TWI831078B (en) | Optical system and optical image processing method by applying image restoration | |
| Hu et al. | Extended depth of field reconstruction with complex field estimation | |
| Henrot et al. | Fast deconvolution of large fluorescence hyperspectral images | |
| Shevkunov et al. | CNN-assisted quantitative phase microscopy for biological cell imaging. | |
| Hoffman | Structured Illumination for In-Vivo Sectioning and Imaging | |
| Wu | Deep Learning-Enabled Computational Imaging in Optical Microscopy and Air Quality Monitoring | |
| Choi et al. | Deep learning approach to coherent noise reduction in optical diffraction tomography | |
| Sun et al. | Improvement of imaging quality of Fourier ptychographic microscopy |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |