CN116183489A

CN116183489A - Bimodal real-time microscopic imaging device and method

Info

Publication number: CN116183489A
Application number: CN202211608068.9A
Authority: CN
Inventors: 王中阳; 肖康; 孙静
Original assignee: Shanghai Advanced Research Institute of CAS
Current assignee: Shanghai Advanced Research Institute of CAS
Priority date: 2022-12-14
Filing date: 2022-12-14
Publication date: 2023-05-30

Abstract

The invention provides a bimodal real-time microscopic imaging device, comprising: an illumination assembly configured to illuminate a sample to be measured; an imaging assembly configured to receive an optical signal from a sample to be measured and emit a scattered optical signal and a fluorescent signal in different optical paths; a full-field camera configured to receive scattered light signals emitted from the imaging assembly; a fluorescence imaging two-dimensional array detector and a GISC camera configured to switchably receive fluorescence signals emanating from the imaging assembly with one another; and the image operation processor is electrically connected with the full light field camera, the fluorescence imaging two-dimensional array detector and the GISC camera. The bimodal real-time microscopic imaging device and the bimodal real-time microscopic imaging method solve the problems that the spatial resolution is low, the bimodal simultaneous real-time imaging cannot be realized, the multicolor real-time imaging is difficult and the like in the existing bimodal phase and fluorescence imaging.

Description

Bimodal real-time microscopic imaging device and method

Technical Field

The invention relates to the field of microscopic imaging, in particular to a bimodal (phase/fluorescence) real-time microscopic imaging device and a bimodal (phase/fluorescence) real-time microscopic imaging method.

Background

The optical microscopic imaging technology has the outstanding advantages of no damage, non-contact, high specificity, high sensitivity, high living body friendliness, capability of providing dynamic imaging of functional information and the like, and becomes the first choice for living organism and living cell imaging. Since the fluorescent protein discoverers issued by the norben chemical prize in 2008, japan scientists, under village repair, and the dominant contributors, the american scientists, marty Chalfie and Qian Yongjian, fluorescent labeling technology has been widely developed and used in biomedical science. In particular, the combination of fluorescence microscopy, fluorescence confocal microscopy and recent development of super-resolution fluorescence microscopy technologies such as random optical reconstruction super-resolution imaging (STORM), structured light illumination imaging (SIM) and the like has been extended to aspects of biomedical research in terms of dynamic in-vivo labeling microscopic observation of biomarkers and target molecules. However, in combination with the wide-field super-resolution imaging technology, there is a disadvantage in real-time performance of multi-color marker imaging.

STORM technology based on single-molecule positioning method has the advantages of high spatial resolution and large field of view, but the high-precision single-molecule positioning requirement limits the molecular density of marked fluorescent dye, and in order to reconstruct a super-resolution image, image sampling of thousands of frames to tens of thousands of frames is usually required for reconstruction, and the time resolution is too low to achieve imaging instantaneity.

The time resolution of the SIM technology is limited only by the sampling frame rate of the detector, and can realize large field of view and low illumination intensity, so that the SIM technology has certain leading advantages in three indexes of time resolution, field of view and illumination intensity, but the spatial resolution is limited to 100nm under the constraint of an imaging mechanism. The SIM technology can realize the real-time performance of monochromatic imaging, but needs structured light illumination and multi-frame reconstruction imaging.

In order to avoid the problem of multicolor crosstalk, the current multicolor imaging is mainly used by matching with a rotary filter plate through multi-wavelength laser time-sharing sequential excitation, but the current rotary filter plate has the fastest switching speed of 35-50ms, which seriously affects the real-time performance of multicolor imaging of living biological cells.

At present, the inventor has developed a wide-field fluorescence super-resolution microscopic imaging technology based on sparse constraint Ghost Imaging (GISC) in the early stage, and combines the quantum imaging technology with a compressed sensing technology of a modern information theory, so that the method has higher image information acquisition efficiency compared with the SIM and STORM technologies, and the spatial resolution, the time resolution and the triangular constraint space of a field of view of a super-resolution fluorescence microscope can be continuously improved; and the technology has the capability of realizing super-resolution imaging by a single frame, and the resolution reaches 80nm. Also in combination with the STORM technique, the frame rate required for STORM imaging can be reduced by more than a factor of hundred. Particularly, the GISC super-resolution imaging technology has the characteristic of modulation coding imaging, can integrate the multicolor coding technology to realize single-frame multicolor super-resolution imaging, and solves the problem of low multicolor imaging speed. The GISC imaging technology obtains national patent authorization (China patent number: ZL 201510394995.9); meanwhile, a nucleic acid detection and gene sequencing scheme based on the GISC imaging technology is also provided, and the national patent is granted (Chinese patent number: ZL 202010982860.5) and International patent application PCT/CN2021/114636 is filed.

In addition, the real-time observation of the optical microscopy technology in living cell imaging and biochip aspects requires not only functional observation after fluorescent labeling, but also label-free dynamic evolution observation of the whole cell. At present, transparent living cells are observed in a label-free manner, various optical phase microscopic imaging technologies are mainly developed, and interference-based phase imaging technologies such as a phase contrast microscope, a differential interference microscope, a holographic microscope and the like are mainly used. However, the acquisition of the real morphology information of the cells requires quantification of the detected phases, and usually requires detection of illumination imaging with a plurality of preset phases, and quantitative phase information is obtained through repeated reconstruction, so that real-time observation is difficult to realize through phase imaging based on an interference technology. Another type of technique is phase recovery algorithm based on coherent diffraction, mainly including Coherent Diffraction Imaging (CDI) and fourier stack imaging (FPM), and generally, in order to increase algorithm convergence of phase recovery, multiple angle illuminations need to be detected and stack imaging is required, and then the phase is recovered by aperture synthesis, which is also disadvantageous in terms of real-time performance of phase imaging.

The inventor independently develops a full light field camera imaging technology in the early stage, realizes the phase recovery of an object image through strong constraint of a real image surface and a Fourier surface, thereby obtaining the phase information of a light field, has the capability of realizing phase recovery imaging by single frame sampling, enables the optical phase microscopic imaging to be realized in real time, and has filed (application number: 202111548124. X) a core patent.

In the aspect of bimodal microscopic imaging, in 2020, the university of Beijing institute of molecular medicine research Chen Liangyi and Shi Kebin cooperate to combine the label-free optical ODT (optical diffraction tomography) technology with the SIM fluorescent imaging technology, so as to realize a new bimodal super-resolution microscope (SR-FACT), and apply for a related patent with the publication number of CN 111610621B. The fluorescence super-resolution is better than 100 nanometers, the resolution of label-free three-dimensional imaging is 200 nanometers in the XY direction and about 500 nanometers in the Z direction, the time resolution of bimodal living cell super-resolution imaging can reach 0.8Hz, and real-time imaging cannot be realized.

Meanwhile, the combination of two-photon fluorescence imaging and FPM technology by the university of Beijing team Shi Kebin also filed a patent with publication number CN113702288B related to bimodal microscopy imaging.

To sum up, the prior art approaches to bimodal imaging are based on ODT technology and SIM or two-photon fluorescence imaging technology. However, since the dual-mode super-resolution microscope (SR-FACT) technology requires multi-angle illumination for realizing the phase imaging technology, multi-frame acquisition, and complex modulation of an illumination light source for the SIM technology, and resolution can only be increased by two times of diffraction limit, the dual-mode imaging in the prior art has the defects of long imaging time, complex instrument structure, low resolution, incapability of synchronous real-time imaging, and the like.

The full light field camera can realize phase imaging by adopting only two-dimensional array detectors to collect real image plane and Fourier plane intensity information at the same time without multi-angle illumination and multi-frame collection. The GISC multicolor coding super-resolution microscopy has the advantages that super-resolution imaging can be realized on the basis of no need of modulating an illumination light source, the super-resolution capability of 80nm of a single frame is realized, and multicolor imaging can be finished by single exposure.

Therefore, it is necessary to propose a bimodal microscopic imaging technique integrating the full-light-field camera technique and the GISC polychromatic coding super-resolution microscopic technique, so as to solve the problems of slow bimodal imaging time, complex structure, low resolution and the like in the prior art, and realize real-time imaging.

Disclosure of Invention

The invention aims to provide a bimodal real-time microscopic imaging device and a bimodal real-time microscopic imaging method, so as to solve the problems that the spatial resolution in the existing bimodal phase and fluorescence imaging is low, the bimodal real-time imaging can not be carried out simultaneously, the multicolor real-time imaging is difficult, and the like.

In order to achieve the above object, the present invention provides a bimodal real-time microscopic imaging device comprising: an illumination assembly configured to illuminate a sample to be measured; an imaging assembly configured to receive an optical signal from a sample to be measured and emit a scattered optical signal and a fluorescent signal in different optical paths; a full-field camera configured to receive scattered light signals emitted from the imaging assembly; a fluorescence imaging two-dimensional array detector and a GISC camera configured to switchably receive fluorescence signals emanating from the imaging assembly with one another; and the image operation processor is electrically connected with the full light field camera, the fluorescence imaging two-dimensional array detector and the GISC camera.

The illumination assembly includes a plurality of excitation light sources, an optical fiber combiner coupled to the plurality of excitation light sources, and an optical fiber collimator coupled to the optical fiber combiner by a single optical fiber.

The illumination assembly is configured to emit monochromatic light through one of a plurality of excitation light sources, or to emit polychromatic light through the plurality of excitation light sources in time sequence or simultaneously; and the laser output of the excitation light source is combined into a single optical fiber through an optical fiber beam combiner, and the sample to be measured is collimated and irradiated through the optical fiber collimator.

The sample to be measured is arranged on a displacement table, and the displacement table is a two-dimensional displacement table.

The whole light field camera is arranged to split scattered light signals by using a whole light field camera beam splitter, one path of the scattered light signals split into two paths is directly detected by a first whole light field camera two-dimensional array detector on an imaging surface, the other path of the scattered light signals is detected by a second whole light field camera two-dimensional array detector on a rear focal surface after being transformed by a whole light field camera imaging lens, and then the phase of the scattered light signals of an object is obtained by reconstruction recovery through a Fourier iterative algorithm.

The GISC camera comprises a random phase modulator arranged at a first distance Z1 from the fluorescent signal imaging surface and a GISC camera two-dimensional array detector arranged at a second distance Z2 behind the random phase modulator so as to record fluorescent speckle signals, wherein the first distance Z1 and the second distance Z2 meet the speckle detection optimization condition; the GISC camera is set to carry out association operation reconstruction on the obtained fluorescence speckle signals and a preset calibration matrix to obtain multicolor super-resolution fluorescence signals.

The imaging component is arranged to emit the scattered light signal to a first light path, and the full light field camera is positioned on the first light path; the imaging component is arranged to switchably emit the fluorescent signal to a second light path and a third light path, the fluorescent imaging two-dimensional array detector is located on the second light path, and the GISC camera is located on the third light path.

The imaging component comprises an objective lens, a dichroic mirror and a switching turning mirror which are sequentially arranged along the trend of the light path; the dichroic mirror is configured to reflect and emit a scattered light signal in the light signal to the first light path and transmit and emit a fluorescent light signal in the light signal to the second light path; the switching turning mirror is arranged to reflect the fluorescent signal propagating along the second optical path to the third optical path when turning to the first position, and to enable the fluorescent signal propagating along the second optical path to continue to propagate along the second optical path when turning to the second position; the switching turning mirror is positioned on the second optical path.

The imaging component is arranged to emit the scattered light signal to a first light path, and the full light field camera is positioned on the first light path; the imaging component is configured to emit the fluorescent signal to a second optical path, and the fluorescent imaging two-dimensional array detector and the GISC camera are switchably located on the second optical path.

The image operation processor is provided with a data processing module; the image operation processor is configured to:

s1: outputting signals acquired by the full-light field camera and the GISC camera or the full-light field camera and the fluorescence imaging two-dimensional array detector synchronously;

s2: caching the acquired data and transmitting the data to a data processing module of an image operation processor;

s3: calculating and reconstructing the acquired signals by using a prefabricated full light field phase recovery algorithm and a GISC super-resolution algorithm in the data processing module to obtain a reconstruction result;

s4: and outputting a reconstruction result.

In another aspect, the present invention provides a method for phase and fluorescence microscopy bimodal imaging under monochromatic illumination, comprising:

a0: building the bimodal real-time microscopic imaging device according to the description above;

a1: the illumination component emits monochromatic light, and the fluorescent imaging two-dimensional array detector is switched to receive fluorescent signals emitted from the imaging component;

a2: controlling the full light field camera and the fluorescence imaging two-dimensional array detector to synchronously acquire signals by utilizing an image operation processor, caching acquired data, and calculating and reconstructing the acquired data by utilizing a data processing module to obtain a reconstruction result; and then outputting a reconstruction result.

In another aspect, the present invention provides a method for phase and GISC fluorescence super-resolution bimodal imaging under monochromatic illumination, comprising:

b0: building the bimodal real-time microscopic imaging device according to the description above;

b1: the illumination component emits monochromatic light, and the GISC camera is switched to receive fluorescent signals emitted from the imaging component;

b2: controlling the full light field camera and the GISC camera to synchronously acquire signals by utilizing an image operation processor, caching acquired data, and calculating and reconstructing the acquired data by utilizing a data processing module to obtain a reconstruction result; and then outputting a reconstruction result.

On the other hand, the invention provides a full-color phase and GISC multi-color fluorescence super-resolution bimodal imaging method under multi-color time sequence illumination, which comprises the following steps:

c0: building the bimodal real-time microscopic imaging device according to the description above;

c1: the illumination assembly is utilized to emit monochromatic light each time according to the time sequence so as to emit light with different wavelengths, and the GISC camera is switched to receive fluorescent signals emitted from the imaging assembly;

c2: controlling the full light field camera and the GISC camera to synchronously perform time sequence acquisition signals by utilizing an image operation processor, caching acquired data, and performing calculation and reconstruction on the time sequence acquired data by utilizing a data processing module to obtain a time sequence image as a reconstruction result; then outputting a reconstruction result; and synthesizing a full-color phase image and a multicolor fluorescence image according to the time sequence image.

On the other hand, the invention provides a full-color phase and GISC multi-color fluorescence super-resolution bimodal imaging method under multi-color simultaneous illumination, which comprises the following steps:

d0: building the bimodal real-time microscopic imaging device according to the description above; the first full-field camera two-dimensional array detector and the second full-field camera two-dimensional array detector in the full-field camera of the bimodal real-time microscopic imaging device are all RGB color array detectors, so that the full-field camera forms a color full-field camera;

d1: the illumination component is used for emitting polychromatic light simultaneously, and the GISC camera is switched to receive fluorescent signals emitted from the imaging component;

d2: controlling the color full-light field camera and the GISC camera to synchronously acquire signals by utilizing an image operation processor, and caching acquired data; decomposing data output by the color full-light field camera according to RGB three-color channels by utilizing an image operation processor to obtain decomposed three-color channel signals, and respectively calculating and reconstructing the decomposed three-color channel signals by utilizing a data processing module to obtain a reconstruction result, and re-superposing the reconstruction result to generate a full-color phase image; calculating and reconstructing data output by the GISC camera by using an image operation processor to obtain a multicolor fluorescence image; thereby obtaining a full-color phase image and a multicolor fluorescence image.

The dual-mode real-time microscopic imaging device and the method integrate the full-light-field camera imaging technology and the GISC multi-color coding super-resolution microscopic technology, and adopt a unified optical fiber beam-combining collimation wide-field illumination mode. The phase imaging mode adopts double-sided constraint Fourier iterative phase recovery, phase information can be recovered only by single exposure, the fluorescent super-resolution imaging mode adopts a GISC technology, and the imaging mode can realize spatial resolution of 80nm under wide-field illumination. Therefore, the invention integrates the full light field camera imaging technology and the GISC super-resolution microscopy technology, replaces the existing bimodal imaging mode which needs to modulate the excitation light and illuminate at multiple angles and is difficult to image the biological multicolor microscopy in real time, has simple design, compact structure and lower cost, does not need to modulate the excitation light and illuminate at multiple angles, has the high-resolution imaging capability of a single frame and has better instantaneity.

Existing phase imaging methods often require more than 2 times the nyquist sampling rate to recover the phase. According to the method and the device for realizing bimodal real-time microscopic imaging, the full light field camera module is used for carrying out light splitting treatment on scattered signals by means of the beam splitter in the full light field camera, so that the intensity signals of the real image surface and the Fourier surface are collected simultaneously, enough information solving phases can be obtained through double-sided constraint of the real image surface and the Fourier surface, phase recovery is carried out, phase object reconstruction imaging can be realized under the condition of approaching one time of Nyquist sampling, and the full light field camera has a lower sampling rate compared with the traditional mode. The real image surface and fourier surface signal acquisition realized through lens transformation not only forms double-sided intensity constraint, but also introduces prior constraint (analytic constraint of the real image surface caused by lens fourier surface transformation property, band-limited constraint of the fourier surface caused by object lens cutting-off of the fourier surface, limited constraint caused by object size imaging on the real image surface, and the like) into an imaging system.

The optical filters are often required to be switched for realizing multicolor by the prior dual-mode fluorescence imaging, the optical fiber beam combiner is adopted to couple light with different wavelengths into a single optical fiber, so that the simultaneous illumination of multicolor light can be realized, and the adopted GISC multicolor super-resolution imaging camera module can realize multicolor imaging by compressing and encoding information of a light field in a high-dimensional space to a two-dimensional array detector.

In summary, the device and the method for bimodal real-time microscopic imaging integrate the phase recovery capability of the full-light field camera imaging technology and the super-resolution imaging capability of the GISC super-resolution microscopic technology, realize the efficient fluorescence excitation mode of single exposure multicolor imaging by matching multicolor illumination, have the advantages of effectively reducing the sampling requirement of a detector, simplifying the light path, improving the imaging information acquisition efficiency and the like, and provide powerful imaging means for multicolor marking and unmarked bimodal real-time biological (fluorescence and phase) imaging.

Drawings

Fig. 1 is a schematic structural view of a bimodal real-time microscopic imaging device according to one embodiment of the present invention.

Fig. 2 is a block diagram of the GISC camera.

Fig. 3 is a schematic diagram of a phase and fluorescence microscopy bimodal imaging method under monochromatic illumination.

Fig. 4 is a schematic diagram of a phase and GISC fluorescence super-resolution imaging method under monochromatic illumination.

FIG. 5 is a schematic diagram of a full color phase and GISC fluorescence super-resolution dual-mode imaging method under time-sequential polychromatic illumination.

FIG. 6 is a schematic diagram of a full color phase and GISC fluorescence super-resolution dual-mode imaging method under simultaneous illumination of multiple colors.

Detailed Description

The following specific examples are presented to illustrate the present invention, and those skilled in the art will readily appreciate the additional advantages and capabilities of the present invention as disclosed herein. The invention may be practiced or carried out in other embodiments that depart from the specific details, and the details of the present description may be modified or varied from the spirit and scope of the present invention. It should be noted that the following embodiments and features in the embodiments may be combined with each other without conflict.

It should be noted that the illustrations provided in the following embodiments merely illustrate the basic concept of the present invention by way of illustration, and only the components related to the present invention are shown in the drawings and are not drawn according to the number, shape and size of the components in actual implementation, and the form, number and proportion of the components in actual implementation may be arbitrarily changed, and the layout of the components may be more complicated.

First embodiment bimodal real-time microscopic imaging device

Fig. 1 is a schematic structural diagram of a bimodal real-time microscopic imaging device according to an embodiment of the present invention. As shown in fig. 1, the bimodal real-time microscopic imaging device comprises an illumination assembly configured to illuminate a sample 4 to be measured (the illumination assembly comprises a plurality of laser light sources 1, one optical fiber combiner 2 connected to a plurality of excitation light sources, and an optical fiber collimator 3 connected to the optical fiber combiner 2 through a single optical fiber 2'); an imaging assembly (which includes an objective lens 5, a dichroic mirror 6, a first reflecting mirror 81, and a switching turning mirror 9) configured to receive an optical signal from a sample 4 to be measured and emit a scattered optical signal and a fluorescent signal in different optical paths; a plenoptic field camera 7 arranged to receive scattered light signals emerging from the imaging assembly, in particular the dichroic mirror 6 of the imaging assembly, and preferably to receive the scattered light signals via the imaging lens 111 of the imaging assembly; a fluorescence imaging two-dimensional array detector 10 and a GISC camera 12 arranged to switchably receive fluorescence signals emitted from the imaging assembly (in particular, the dichroic mirror 6 of the imaging assembly) with each other; and an image operation processor 13 electrically connected with the full light field camera 7, the fluorescence imaging two-dimensional array detector 10 and the GISC camera 12.

The illumination assembly is configured to emit monochromatic light through one of a plurality of excitation light sources, or to emit polychromatic light through the plurality of excitation light sources in time sequence or simultaneously; and the laser output of the excitation light source is combined into a single optical fiber through an optical fiber beam combiner, and the sample to be measured is collimated and irradiated through the optical fiber collimator. When the illumination component emits polychromatic light, after the excitation light irradiates the sample 4 to be measured, the excitation light forms scattered light on the sample 4 to be measured, meanwhile, the polychromatic fluorescence can be excited simultaneously by an efficient multi-wavelength excitation mode, and the bimodal signal of polychromatic fluorescence signals and scattered signals is collected simultaneously by combining the spectral action of the dichroic mirror.

The illumination assembly comprises a plurality of excitation light sources 1, one optical fiber combiner 2 connected to the plurality of excitation light sources 1, and an optical fiber collimator 3 connected to the optical fiber combiner 2 via a single optical fiber 2'. The excitation light source 1 includes, but is not limited to, continuous or pulse output coherent lasers with center wavelengths of 488nm, 532nm, 639nm, etc., and the phase imaging and the fluorescence imaging share the same light source. The excitation light source satisfies a certain spatial coherence and a certain time coherence, and the lack of coherence will cause the light source used for phase imaging to be insufficient to form a better diffraction pattern on the fourier surface, so that a laser or an LED with good coherence must be used as the excitation light source. The excitation light source 1 is configured to emit excitation light, which may be used to illuminate the sample 4 to be measured, to generate a scattered light signal, and to excite fluorescent markers (e.g., fluorescent dyes, fluorescent markers, autofluorescent proteins, autofluorescent biomolecules, autofluorescent tissues, etc.) of the sample 4 to be measured to emit a fluorescent signal. Scattered light from the illumination source is used for label-free phase imaging.

The optical fiber beam combiner 2 is arranged to output laser of the excitation light source 1 into a single optical fiber, and the optical fiber collimator 3 is arranged at the tail end of the single optical fiber and is arranged to irradiate a sample 4 to be measured; the single optical fiber can be a multimode optical fiber, a single mode optical fiber or a polarization maintaining optical fiber; the output end of the single-mode fiber or the polarization maintaining fiber adopts a proper fiber collimator 3 according to the bright field requirement so as to obtain illumination meeting the illumination field requirement.

The optical fiber collimator 3 is configured to collimate the outgoing light of the single optical fiber 2' and irradiate the sample 4 to be measured in the form of collimated light and wide-field illumination.

The sample 4 to be measured is mounted on a displacement table 14. The displacement table 14 is a two-dimensional displacement table, on which an installation position with the sample 4 to be measured is arranged, and the light source assembly is aligned with the sample 4 to be measured. The displacement stage 14 may employ a micrometer-scale or nanometer-scale displacement stage or a combination of both to meet different experimental scenarios depending on microscopic imaging field of view requirements and resolution requirements.

The imaging component is located at one side of the sample 4 to be measured far away from the illumination component, and is configured to receive the optical signal from the sample 4 to be measured through the objective lens 5 and emit the scattered optical signal and the fluorescent signal in the optical signal to different optical paths respectively. In the present embodiment, the dichroic mirror 6 in the imaging assembly is configured to emit the scattered light signal to a first optical path, and the full-field camera 7 is located on the first optical path; the switching turning mirror 9 in the imaging assembly is configured to switchably emit the fluorescent signal to the second optical path and the third optical path, the fluorescent imaging two-dimensional array detector 10 is located on the second optical path, and the GISC camera 12 is located on the third optical path.

The imaging component includes, but is not limited to, an objective lens 5 and imaging lenses (such as a first imaging lens 111, a second imaging lens 112, and a third imaging lens 113) matched with the objective lens 5, and auxiliary optical elements such as a dichroic mirror 6, a filter, a switching mirror 9, and the like. In this embodiment, the imaging assembly includes the objective lens 5, the dichroic mirror 6, the first mirror 81, and the switching mirror 9, which are sequentially arranged along the direction of the optical path, and in other embodiments, the first mirror 81 may be omitted. The dichroic mirror 6 is arranged such that scattered light signals of the light signals are reflected and emitted to a first light path and fluorescent signals of the light signals are transmitted and emitted to a second light path, and the plenoptic field camera 7 is located on the first light path. The switching turning mirror 9 is provided with: the fluorescent signal propagating along the second optical path is reflected onto the third optical path when rotated to the first position, and the fluorescent signal propagating along the second optical path is caused to continue to propagate along the second optical path when rotated to the second position. Accordingly, the first mirror 81, the switching mirror 9 and the fluorescence imaging two-dimensional array detector 10 are all located on the second optical path, and the GISC camera 12 is located on the third optical path. Therefore, when the switching rotating mirror 9 rotates to the second position, the fluorescence signal enters the GISC camera 12, and when the switching rotating mirror rotates to the first position, the fluorescence signal directly enters the fluorescence imaging two-dimensional array detector 10 to directly perform wide-field fluorescence microscopic imaging.

A first imaging lens 111 is arranged between the full-light-field camera 7 and the dichroic mirror 6 (i.e. in front of the full-light-field camera 7), a second imaging lens 112 is arranged between the dichroic mirror 6 and the fluorescence imaging two-dimensional array detector 10 (i.e. in front of the fluorescence imaging two-dimensional array detector 10), and a second reflecting mirror 82 and a third imaging lens 113 are arranged between the dichroic mirror 6 and the GISC camera 12 (i.e. in front of the GISC camera 12). Further, a bandpass filter may be provided between the dichroic mirror 6 and the first, second, and

third imaging lenses

111, 112, 113.

In other embodiments, as in the present embodiment, the imaging assembly is configured to emit the scattered light signal to a first optical path, and the full-field camera 7 is located on the first optical path; the difference from the present embodiment is that the imaging component may also be configured to emit the fluorescence signal to a second optical path, where the fluorescence imaging two-dimensional array detector 10 and the GISC camera 12 are switchably located on the second optical path, i.e. one of the fluorescence imaging two-dimensional array detector 10 and the GISC camera 12 is removed from the second optical path, and the other is switched on the second optical path. Correspondingly, the imaging assembly comprises an objective lens 5 and a dichroic mirror 6 arranged in sequence along the course of the light path.

Thus, the full-field camera 7 receives the scattered light signal from the first imaging assembly 111, and the fluorescence imaging two-dimensional array detector 10 and the GISC camera 12 switchably receive the fluorescence signal from the second imaging assembly 112 or the third imaging assembly 113.

The fluorescence imaging two-dimensional array detector 10 can adopt a high sampling rate CCD camera, an EMCCD camera, a CMOS camera, an sCMOS camera or a full-color camera.

The full-field camera 7 may refer to the patent document with application number 202111548124.X related to the full-field camera. The full light field camera 7 comprises a full light field camera beam splitter, a first full light field camera two-dimensional array detector and a full light field camera beam splitter, a full light field camera imaging lens and a second full light field camera two-dimensional array detector which are sequentially arranged in the other beam splitting direction of the full light field camera beam splitter.

Thus, the full-field camera 7 is configured to split the scattered light signal by using the full-field camera beam splitter, and one of the scattered light signals split into two paths is directly detected by the first full-field camera two-dimensional array detector on the imaging plane to acquire a real image plane intensity signal; the other path is detected by a second full light field camera two-dimensional array detector at the rear focal plane after being transformed by the full light field camera imaging lens so as to acquire Fourier plane intensity signals; reconstruction recovery is then performed by a fourier iterative algorithm to obtain the phase of the scattered light signal of the object. Therefore, the Fourier transform principle of the lens is utilized, the first full-field camera two-dimensional array detector and the second full-field camera two-dimensional array detector are respectively placed on the real image surface and the Fourier surface, meanwhile, the intensity information of the real image surface and the Fourier surface is collected, and the full-field phase recovery algorithm is matched to achieve phase recovery. Due to the strong constraint of the real image plane and the Fourier plane, single-frame phase recovery imaging is realized, so that real-time phase imaging is realized.

Full light field phase recovery algorithms for full light field cameras include, but are not limited to, fourier iterative phase recovery algorithms such as Gerchberg-Saxton (GS) algorithm, hybrid input-output (HIO) algorithm, and Young's (Y-G) algorithm. The input data of the full light field phase recovery algorithm are the intensity signal of the scattering signal on the real image surface and the intensity signal of the Fourier surface (namely double-sided constraint and double-sided acquisition intensity constraint), and the output data is the phase of the object to be detected.

The double-sided a priori constraints incorporated by the full-light-field phase recovery algorithm include, but are not limited to: fourier plane band-limit constraints limited by objective aperture, band-limit constraints on the size of the object itself (i.e. the imaged size of the sample size at the real image plane), analytical constraints (Ke Xili man equation, shannon interpolation, kramers-kronig relation). These two-sided prior constraints are all constraints inherent to the physical system in the present invention (real image plane imaging size, fourier plane band-limit, sparseness constraint, resolution characteristics). The prior constraint provides constraints inherent to the physical system (imaging size, fourier plane band-limit, sparse constraint, resolution) for the phase recovery algorithm. In addition to measuring the intensities of the real image and fourier surfaces, the physical design of the full-field camera may provide band-limited constraints on the fourier surfaces and analytical constraints on the real image surfaces. The band-limited constraint can reduce the number of equations, ensure uniqueness under low sampling rate, and real image plane analytic constraint can accelerate the convergence process, so that the phase reconstruction quality is further optimized.

As shown in fig. 2, the GISC camera 12 includes a random phase modulator 121 disposed at a first distance Z1 from the imaging plane of the fluorescent signal and a GISC camera two-dimensional array detector 122 disposed at a second distance Z2 behind the random phase modulator, and the light entering the GISC camera 12 reaches the random phase modulator 121 after passing through the first distance Z1 and reaches the GISC camera two-dimensional array detector 122 after passing through the second distance Z2, wherein the Z1, Z2 distance adjustment should meet the optimization requirement of the speckle contrast after modulation. In the present embodiment, the GISC camera 12 is a GISC multi-color super-resolution camera. Thus, the GISC camera 12 performs random compression encoding on fluorescent signals with different wavelengths by using the random phase modulator 121 to obtain an encoded signal, where the encoded signal is a fluorescent speckle signal formed after the random phase modulator 121 performs random phase modulation on an incident light field. The GISC camera two-dimensional array detector 122 collects the coded signals, namely records the fluorescence speckle signals, and the GISC camera is set to perform association operation reconstruction on the obtained fluorescence speckle signals and a preset calibration matrix to obtain multicolor super-resolution fluorescence signals.

The GISC super-resolution algorithm comprises the steps of performing correlation operation on the coded signals and the calibrated system matrix to obtain a super-resolution imaging result. The system matrix is formed by combining random coded speckle intensity distribution normalization and vectorization corresponding to different wavelengths at different positions calibrated in advance by the GISC system.

The association operation includes but is not limited to: sparse constraint, compressed sensing, least squares fitting, maximum likelihood estimation, and deep learning. The correlation operation needs a speckle field Y acquired in the imaging process, a matrix A is calibrated in advance, a polychromatic super-resolution fluorescent image to be reconstructed is X, and the three satisfy a matrix relation Y=AX. The correlation algorithm reversely solves X through Y through the sparse constraint algorithm, the compressed sensing algorithm and the like.

The calibration method of the system matrix is various, including:

(1) calibrating a displacement table: and placing a single-color fluorescent ball on the sample stage, moving the displacement stage according to imaging precision, and acquiring a fluorescent speckle signal by a GISC camera every time the displacement stage is moved. And (5) traversing all positions in the view field, replacing the wavelength of the monochromatic fluorescent pellets, and repeating the steps until the requirement of imaging polychromacy is met. All collected fluorescence speckle signals are normalized, vectorized and matrixed to form a calibration matrix A.

(2) Calculating imaging scaling: the known phase distribution is obtained through the design and development of the multicolor random phase modulator, a GISC microscopic full-link simulation model is established according to the fluctuation optics and geometry optics theory, the fluorescence light field distribution after random phase modulation is obtained, and then a high-precision calibration matrix is constructed through a large number of high-precision simulation calculations. The method can effectively reduce calibration errors caused by instable systems and light sources in the actual calibration process.

(3) Phase measurement scaling: the phase distribution of the phase modulator is measured with high precision by utilizing a quantitative phase imaging technology, the phase error introduced by a processing error is reduced, and the precise phase distribution of the random phase modulator is obtained, so that the GISC microscopic full-link simulation model is corrected, and the GISC microscopic full-link simulation model is more in line with a real system, so that the precision of a calibration matrix is improved.

(4) Deep learning scaling: the multicolor fluorescence signal is used as X, the GISC system collects speckle field signals as Y, and a training set is established by collecting mass fluorescence imaging data. And establishing a function mapping relation between X and Y according to the GISC system parameters, namely Y=f (X|theta). And solving theta through a back propagation algorithm to determine the functional relation between the multicolor fluorescence signal and the speckle field to finish calibration.

The GISC super-resolution algorithm and the full-field phase-recovery algorithm may run in the image arithmetic processor 13. The image operation processor 13 may include various miniaturized high-performance computing devices capable of parallel operation, such as a computer, an FPGA chip, or a single chip microcomputer. The image operation processor 13 has a data processing module, which may be a built-in fourier iteration module or an in-chip multi-core MCU of the FPGA, and is configured to execute a pre-stored full light field phase recovery algorithm and a GISC super-resolution algorithm to reconstruct an image to be reconstructed. It should be noted that, the wide-field fluorescence imaging result acquired by the fluorescence imaging two-dimensional array detector 10 is obtained without an algorithm for recovering the image, so that the image operation processor 13 is not required for reconstructing the image.

The flow of specific execution of the image operation processor 13 is as follows:

step S1: outputting signals acquired synchronously by the full-field camera 7 and the GISC camera 12 or by the full-field camera 7 and the fluorescence imaging two-dimensional array detector 10;

step S2: the data acquired by the buffer memory of the image operation processor 13 is transmitted to a data processing module of the image operation processor 13;

step S3: and calculating and reconstructing the acquired signals by using one of a prefabricated full light field phase recovery algorithm and a GISC super-resolution algorithm in the data processing module to obtain a reconstruction result. It should be noted that, the wide-field fluorescence imaging result acquired by the fluorescence imaging two-dimensional array detector 10 does not need to be recovered by an algorithm, and the intensity distribution detected by the detector is the required result, so that the wide-field fluorescence imaging method is not required to execute the step S3.

Step S4: and outputting a reconstruction result to realize bimodal real-time imaging. In this embodiment, the reconstruction result is output to the display device.

Thus, the polychromatic or monochromatic coherent excitation light source 1 is coupled into a single fiber with a fiber combiner 2, the output end of which incorporates a fiber collimator 3 to design an illumination field to illuminate the sample. The excitation light irradiates the sample 4 to be measured to generate scattered light, and simultaneously, the excitation light excites a fluorescent signal generated by multicolor fluorescent marks in the sample. Subsequently, the scattered light and the fluorescent signal are collected simultaneously by the objective lens 5, the scattered light and the fluorescent signal are separated by the dichroic mirror 6, the scattered signal is reflected by the dichroic mirror 6, and the fluorescent signal is transmitted through the dichroic mirror. The scattered signal is imaged by the first imaging lens 111 to the full light field camera 7 for phase imaging. The fluorescent signal is imaged to the GISC camera 12 through the third imaging lens 113 for single-frame multi-color super-resolution imaging. In addition, the fluorescent signal can be imaged to the fluorescent imaging two-dimensional array detector 10 by switching the rotating mirror 9 and the second imaging lens 112 for conventional wide-field fluorescent microscopic imaging.

Second embodiment phase and fluorescence microscopy bimodal imaging method under monochromatic illumination

As shown in fig. 3, the phase and fluorescence microscopy bimodal imaging method under monochromatic illumination comprises the steps of:

step A0: constructing the bimodal real-time microscopic imaging device;

as described above, the bimodal real-time microscopic imaging device includes an illumination assembly (including an excitation light source 1, an optical fiber 2', an optical fiber collimator 3) configured to illuminate a sample 4 to be measured, an imaging assembly (including an objective lens 5, a dichroic mirror 6, a switching mirror 9, a first imaging lens 111, a second imaging lens 112, a third imaging lens 113, a first reflecting mirror 81) configured to receive an optical signal from the sample 4 to be measured, a full light field camera 7 configured to receive a scattered optical signal from the imaging assembly, a fluorescence imaging two-dimensional array detector 10 and a full light field camera 7 configured to switchably receive fluorescent signals from the imaging assembly with each other, a fluorescence imaging two-dimensional array detector 10, and an image operation processor 13 electrically connected to the full light field camera 7. The illumination assembly comprises an excitation light source 1 and a single optical fiber 2' connected to the excitation light source 1 and a fiber collimator 3, but since the illumination assembly is used hereinafter to emit monochromatic light, only a single excitation light source 1 needs to be turned on due to a single wavelength of excitation light, fig. 3 shows only one excitation light source 1 compared to fig. 1.

In this embodiment, the two-dimensional array detector 10 for fluorescence imaging is an sCMOS camera, and is used for detecting weak fluorescence signals, so that a sCMOS with high quantum efficiency and sensitivity is required.

In this embodiment, the imaging assembly includes an objective lens 5, a dichroic mirror 6, and a first reflecting mirror 81 sequentially arranged along the direction of the optical path, and the objective lens 5 is 100×,0.8NA.

In the present embodiment, the imaging assembly used includes a first imaging lens 111 in front of the full-field camera 7 and a second imaging lens 112 in front of the fluorescent imaging two-dimensional array detector 10, and the focal length of the first imaging lens 111 and the second imaging lens 112 is 180mm.

In the present embodiment, the full-field camera 7 includes a full-field camera imaging lens, the focal length of which may be 100mm. The 2048×2048 CMOS array detector that can be employed for both the first and second full-field camera two-dimensional array detectors used in the full-field camera 7 has a pixel size of 6.45 μm×6.45 μm.

Step A1: and the illumination component emits monochromatic light, and the fluorescent imaging two-dimensional array detector of the bimodal real-time microscopic imaging device is switched to receive fluorescent signals emitted from the imaging component so as to perform traditional wide-field fluorescent microscopic imaging.

In this embodiment, the illumination assembly includes a single excitation light source 1 that is turned on, an optical fiber 2' connected to the excitation light source 1, and an optical fiber collimator 3, where when the illumination assembly emits monochromatic light, the excitation light source 1 with a wavelength corresponding to the excitation light source 1 is selected to be turned on according to the excitation wavelength of the dye molecule of the object to be measured.

Step A2: the image operation processor 13 is used for controlling the full light field camera 7 and the fluorescence imaging two-dimensional array detector 10 to synchronously acquire signals, caching the acquired data, and the data processing module is used for calculating and reconstructing the acquired signals to obtain a reconstruction result; and then outputting a reconstruction result to realize bimodal real-time imaging.

In this embodiment, the image operation processor 13 is an image operation processor developed based on FPGA, and is connected with the full-field camera and the sCMOS camera through the USB3.0 interface to receive the camera acquisition data, and the calculated result outputs the reconstruction result to the display device through the DP1.2 interface.

Therefore, the excitation light of the single-beam excitation light source 1 is coupled to the optical fiber 2', after being collimated by the optical fiber collimator 3, the scattered light and fluorescence generated by the sample irradiated onto the sample 4 to be measured in a near-collimation and wide-field illumination mode are collected by the objective lens 5 and are divided into two paths through the dichroic mirror 6, one path of the scattered light and the fluorescence is transmitted to the full-light field camera 7 to collect real-image plane and Fourier plane information, the other path of the scattered light and the fluorescence is transmitted to the sCMOS camera 10 to carry out fluorescence microscopic imaging, and the collected signals are input into the graphic arithmetic processor 13 in a USB3.0 transmission mode to be operated, so that real-time single-color illumination phase and fluorescence microscopic dual-mode imaging is realized.

Third embodiment phase and GISC fluorescence super-resolution bimodal imaging method under monochromatic illumination

As shown in fig. 4, the phase and fluorescence super-resolution microscopic bimodal imaging method under monochromatic illumination comprises the following steps:

step B0: constructing the bimodal real-time microscopic imaging device;

as described above, the bimodal real-time microscopic imaging apparatus includes an illumination assembly (including an excitation light source 1, an optical fiber 2', an optical fiber collimator 3) configured to illuminate a sample 4 to be measured, an imaging assembly (including an objective lens 5, a dichroic mirror 6, a first imaging lens 111, a second imaging lens 112, a third imaging lens 113, a first mirror 81, a second mirror 82, a switching mirror 9) configured to receive an optical signal from the sample 4 to be measured, a full light field camera 7 configured to receive a scattered optical signal from the imaging assembly, a fluorescence imaging two-dimensional array detector 10 and a GISC camera 12 configured to switchably receive a fluorescent signal from the imaging assembly with each other, and an image operation processor 13 electrically connected to each of the full light field camera 7, the fluorescence imaging two-dimensional array detector 10, and the GISC camera 12. The illumination assembly comprises a single on excitation light source 1 and a single optical fiber 2' and a cluster 3, only one excitation light source 1 being shown in fig. 4, since the illumination assembly is used hereinafter to emit light of a single wavelength.

In the present embodiment, the imaging assembly used includes a first imaging lens 111 in front of the full-field camera 7 and a third imaging lens 113 in front of the GISC camera 12, the focal length of the first imaging lens 111 and the third imaging lens 113 being 180mm.

In this embodiment, the full-field camera 7 comprises a fourier imaging lens, the focal length of which may be 100mm. The 2048×2048 CMOS array detector that can be employed for both the first and second full-field camera two-dimensional array detectors used in the full-field camera 7 has a pixel size of 6.45 μm×6.45 μm.

Step B1: the illumination assembly emits monochromatic light, and the GISC camera 12 of the bimodal real-time microscopic imaging device is switched to receive fluorescent signals emitted from the imaging assembly; accordingly, the fluorescence imaging two-dimensional array detector 10 does not receive a fluorescence signal at this time.

In this embodiment, the illumination assembly includes a single excitation light source 1, an optical fiber 2', and an optical fiber collimator 3, and when the illumination assembly emits monochromatic light, the laser light sources 1 with corresponding wavelengths among the plurality of excitation light sources 1 are selected to be turned on according to the excitation wavelength of the dye molecules of the object to be measured.

Step B2: the image operation processor 13 is used for controlling the full light field camera 7 and the GISC camera 12 to synchronously acquire signals, caching the acquired data, and the data processing module is used for calculating and reconstructing the acquired data to obtain a reconstruction result; and then outputting a reconstruction result to realize bimodal real-time imaging.

In this embodiment, the image operation processor 13 is an image operation processor developed based on FPGA, and is connected with the full-field camera, the GISC super-resolution camera, and the sCMOS camera through the USB3.0 interface to receive the camera acquisition data, and the calculated result outputs the reconstruction result to the display device through the DP1.2 interface.

Because the wavelength of the scattered signal (the reconstructed phase information of the full-field camera) is consistent with the excitation wavelength, the wavelength of the phase information can be known according to different laser time sequence excitation sequences under the condition of time sequence detection, and the full-field camera is not required to have the color resolution capability.

Therefore, the single-wavelength excitation light source 1 is collimated by the optical fiber 2' through the optical fiber collimator 3, scattered light and fluorescence generated by irradiating the sample to be measured 4 in a near-collimation and wide-field illumination mode are collected by the objective lens 5 and are divided into two paths through the dichroic mirror 6, one path of the scattered light and the fluorescence is transmitted to the full-light-field camera 7 to collect real-image surface and Fourier surface information, the other path of the scattered light and the fluorescence enters the GISC camera 12, the collected signals are input into the control and image operation processor 13 taking the multi-core MCU in the FPGA chip as a core in a USB3.0 transmission mode to operate, and real-time phase and GISC fluorescence super-resolution dual-mode imaging is realized.

Full-color phase and GISC multi-color fluorescent super-resolution bimodal imaging method under multi-color time sequence illumination of fourth embodiment

As shown in fig. 5, the full-color phase and GISC multi-color fluorescence super-resolution bimodal imaging method under multi-color sequential illumination includes the following steps:

step C0: constructing the bimodal real-time microscopic imaging device;

as described above, the bimodal real-time microscopic imaging apparatus includes an illumination assembly configured to illuminate a sample 4 to be measured, an imaging assembly (including an excitation light source 1, a fiber combiner 2, a fiber 2', a fiber collimator 3, a timing controller 15) configured to receive an optical signal from the sample 4 to be measured, a full-field camera 7 configured to receive a scattered optical signal emitted from the imaging assembly (including an objective lens 5, a dichroic mirror 6, a switching mirror 9, a first imaging lens 111, a second imaging lens 112, a third imaging lens 113, a first mirror 81, a second mirror 82), a fluorescence imaging two-dimensional array detector 10 and a GISC camera 12 configured to switchably receive a fluorescence signal emitted from the imaging assembly with each other, and an image operation processor 13 electrically connected to each of the full-field camera 7, the fluorescence imaging two-dimensional array detector 10, and the GISC camera 12.

In this embodiment, the two-dimensional array detector for fluorescence imaging 10 is an sCMOS camera, and the two-dimensional array detector for fluorescence imaging 10 is not used in this mode.

In this embodiment, the imaging assembly includes an objective lens 5, a dichroic mirror 6, and a first reflecting mirror 81 sequentially arranged along the direction of the optical path, where the objective lens 5 is 100×,0.8NA.

Step C1: the GISC camera 12 is switched to receive the fluorescent signal emitted from the imaging assembly 113 by sequentially emitting monochromatic light each time to emit light of different wavelengths by using the illumination assembly under the control of the timing controller; accordingly, the fluorescence imaging two-dimensional array detector 10 does not receive a fluorescence signal at this time.

The illumination assembly comprises a plurality of excitation light sources 1 and an optical fiber combiner 2 connected with the excitation light sources 1, wherein the optical fiber combiner 2 couples a plurality of optical fibers to a single optical fiber, and the tail end of the single optical fiber is connected with an optical fiber collimator 3. Since the multi-color light is emitted in time sequence by the lighting assembly, the lighting assembly further comprises a time sequence controller, the time sequence controller 15 being connected to the plurality of excitation light sources 1, arranged to switch on one excitation light source 1 at a time in accordance with the sequence. The excitation light source 1 which is turned on comprises laser light sources with center wavelengths of 488nm, 532nm and 639 nm. The sequential excitation of the laser adopts a STM32 type time schedule controller, and the switching of the excitation light source 1 is controlled by pulse signal output.

Step C2: the image operation processor 13 is used for controlling the full light field camera 7 and the GISC camera 12 to synchronously perform time sequence acquisition signals, caching the acquired data, and the data processing module is used for calculating and reconstructing the acquired data to obtain a time sequence image as a reconstruction result; and then outputting a reconstruction result, and synthesizing full-color phase and multi-color fluorescent images according to the time sequence images.

In this embodiment, the full-field camera 7 can simultaneously image the transmittance and the phase of the object 4 to be measured, detect the transmittance and the fourier plane intensities of three colors of blue (488 nm), green (532 nm) and red (639 nm) respectively according to the excitation sequence, and then reconstruct the transmittance and the fourier plane intensities respectively, where the reconstruction result is a time-series image, and the time-series image includes a real image plane intensity image acquired in time series and a phase image reconstructed by a corresponding algorithm. And re-superimposing the reconstructed light field information (namely, the reconstruction result) to obtain a full-color phase map.

Therefore, the time sequence controller 15 turns on the excitation light source 1 with single wavelength each time according to the sequence, the laser excited each time is coupled to a single optical fiber through the optical fiber beam combiner 2, the excitation light in the single optical fiber is collimated by the optical fiber collimator 3 and irradiates onto the sample 4 to be detected in a wide field mode, scattered light and fluorescence generated by the sample are collected by the objective lens 5 and are divided into two paths through the dichroic mirror 6, one path of the scattered light and fluorescence is used for collecting real image surface and Fourier surface information to the full-light field camera 7, the other path of the scattered light and fluorescence enters the GISC camera 12 to collect, the collected signals are input into the control and image operation processor 13 taking the multi-core MCU in the FPGA chip to operate in a USB3.0 transmission mode, and full-color phase and GISC multi-color fluorescence super-resolution dual-mode imaging, namely full-color phase super-resolution imaging and multi-color super-resolution fluorescence imaging is realized.

Fifth embodiment full color phase and GISC multicolor fluorescence super-resolution bimodal imaging method under multicolor simultaneous illumination

As shown in fig. 6, the full-color phase and GISC multi-color fluorescence super-resolution bimodal imaging method under multi-color simultaneous illumination includes the following steps:

step D0: constructing the bimodal real-time microscopic imaging device;

as described above, the bimodal real-time microscopic imaging apparatus includes an illumination assembly (including an excitation light source 1, a fiber combiner 2, a fiber collimator 3) configured to illuminate a sample 4 to be measured, an imaging assembly (including an objective lens 5, a dichroic mirror 6, a switching mirror 9, a first imaging lens 111, a second imaging lens 112, a third imaging lens 113, a first mirror 81, a second mirror 82) configured to receive an optical signal from the sample 4 to be measured, a full-field camera 7 configured to receive a scattered optical signal emitted from the imaging assembly, a fluorescence imaging two-dimensional array detector 10 and a GISC camera 12 configured to switchably receive a fluorescent signal emitted from the imaging assembly with each other, and an image operation processor 13 electrically connected to each of the full-field camera 7, the fluorescence imaging two-dimensional array detector 10, and the GISC camera 12.

In the present embodiment, the fluorescence imaging two-dimensional array detector 10 is an sCMOS camera, and the fluorescence imaging two-dimensional array detector 10 is not used.

In the present embodiment, the full-color full-field camera 7 is a full-color full-field camera, which includes a full-field camera imaging lens, the focal length of which may be 100mm. The first full-field camera two-dimensional array detector and the second full-field camera two-dimensional array detector used in the full-field camera 7 are both RGB color array detectors, so that the full-field camera constitutes a color full-field camera; and the first full-light field camera two-dimensional array detector and the second full-light field camera two-dimensional array detector can both adopt 2688 multiplied by 2200 CMOS array detectors, and the pixel size is 4.54 mu m multiplied by 4.54 mu m.

Step D1: the illumination assembly is used for emitting polychromatic light simultaneously, and the GISC camera 12 is switched to receive fluorescent signals emitted by the third imaging lens 113 of the imaging assembly; accordingly, the fluorescence imaging two-dimensional array detector 10 does not receive a fluorescence signal at this time.

The illumination assembly comprises a plurality of excitation light sources 1 and an optical fiber combiner 2 connected with the excitation light sources 1, wherein the optical fiber combiner 2 couples a plurality of optical fibers to a single optical fiber 2', and the tail end of the single optical fiber 2' is connected with an optical fiber collimator 3. Since the polychromatic light is emitted simultaneously by means of the illumination assembly, the illumination assembly is arranged to simultaneously switch on a plurality of excitation light sources 1. The excitation light source 1 which is turned on comprises laser light sources with center wavelengths of 488nm, 532nm and 639 nm.

Step D2: using the image operation processor 13 to control the full light field camera 7 (i.e. the color full light field camera) and the GISC camera 12 to synchronously acquire signals, and buffering the acquired data; decomposing the data output by the color full-light field camera according to RGB three-color channels by utilizing an image operation processor 13 to obtain decomposed three-color channel signals, and respectively calculating and reconstructing the decomposed three-color channel signals by utilizing a data processing module to obtain a reconstruction result, and re-superposing the reconstruction result to generate a full-color phase image; using an image operation processor 13 to calculate and reconstruct the data output by the GISC camera to obtain a multicolor fluorescence image; thus, a full-color phase image and a multicolor fluorescence image are obtained, and dual-mode real-time imaging is realized.

In this embodiment, the full-field camera 7 can simultaneously image the transmittance and the phase of the object 3 to be measured, after the transmittance of three colors of blue (488 nm), green (532 nm) and red (639 nm) can be detected simultaneously, the intensities collected by the real image plane and the fourier plane are decomposed according to the channel of RGB three colors in the subsequent calculation process, the phase information is respectively reconstructed, and then the polychromatic light field obtained after the decomposition and reconstruction is superimposed to obtain the full-color phase map.

In this embodiment, the image operation processor 13 is an image operation processor developed based on FPGA, and is connected with the full-light field camera 7, the GISC camera 12, and the sCMOS camera (i.e. the fluorescence imaging two-dimensional array detector 10) through a USB3.0 interface to receive the camera acquisition data, and the calculated result outputs the reconstruction result to the display device through a DP1.2 interface.

Therefore, a plurality of excitation light sources 1 are started simultaneously, the laser excited simultaneously is coupled to a single optical fiber 2 'through an optical fiber beam combiner 2, excitation light in the single optical fiber 2' is collimated by an optical fiber collimator 3 and then irradiates on a sample 4 to be detected in a wide field mode, scattered light and fluorescence generated by the sample 4 are collected by an objective lens 5 and are divided into two paths through a dichroic mirror 6, one path of the scattered light and fluorescence enters a GISC camera 7 to collect real image surface and Fourier surface information, the other path of the scattered light and fluorescence enters a GISC camera 12 to collect, and the collected signals are input into a control and image operation processor 13 taking a multi-core MCU in an FPGA chip as a core to operate in a USB3.0 transmission mode, so that full-color phase and GISC multi-color fluorescence super-resolution bimodal imaging is realized.

In summary, bimodal imaging includes, but is not limited to, four common approaches: phase and fluorescence microscopy bimodal imaging under monochromatic illumination, phase and GISC fluorescence super-resolution bimodal imaging under monochromatic illumination, full-color phase and GISC polychromatic fluorescence super-resolution bimodal imaging under time sequence polychromatic illumination, full-color phase and GISC polychromatic fluorescence super-resolution bimodal imaging under polychromatic illumination. Bimodal imaging means that the bimodal real-time microscopic imaging device of the present invention has two imaging modes that are simultaneously in operation.

The phase and fluorescence microscopic bimodal imaging mode under the monochromatic illumination can be used for exciting a fluorescence signal and a scattering signal of a sample through a single laser, then the fluorescence signal and the scattering signal are collected through an objective lens and are separated by utilizing a dichroic mirror, and the fluorescence signal and the scattering signal enter a full-light field camera and a two-dimensional array detector respectively for bimodal imaging.

The full-color phase and GISC fluorescence super-resolution bimodal imaging mode under the time sequence polychromatic illumination can control the excitation time sequence of laser through a time sequence controller, and fluorescence signals and scattering signals of a sample are excited sequentially through the time sequence laser and then are collected through an objective lens, separated by a dichroic mirror and respectively enter a full-light field camera and a GISC polychromatic super-resolution camera to carry out bimodal imaging. The full-color phase imaging is that the full-light field camera can image the transmittance and the phase of a plurality of wavelengths of an object to be detected, so that full-color phase imaging is realized.

Meanwhile, the full-color phase and GISC fluorescence super-resolution bimodal imaging mode under multicolor illumination can be realized by coupling a plurality of laser beams with different wavelengths into one laser beam through an optical fiber beam gathering device, exciting fluorescent signals and scattered signals of a sample by the laser beams, collecting the fluorescent signals and scattered signals through an objective lens, and separating the fluorescent signals and scattered signals by utilizing a dichroic mirror to enter a full-color full-light field camera and a GISC multicolor super-resolution camera respectively for bimodal imaging. Full color full light field cameras refer to replacing the two-dimensional array detector in the full light field camera with a full color two-dimensional array detector.

During the handoff, the GISC camera 12 does not need to move position. Since the full-field camera imaging lens employs a tele lens, the wavelength is not sensitive to the full-field camera 7. Therefore, the lens position is not required to be moved, and only the objective lens 5 is required to search the focal plane.

Compared with the Fourier laminated imaging technology or the coherent diffraction imaging technology, the phase microscopic imaging realized in the mode does not need to enable the spatial Fourier spectrum information to have an overlapped part or obtain enough information recovery phase through oversampling, so that redundant information is reduced, required data can be reduced, and the experimental sampling time is reduced to meet the real-time imaging requirement.

The fluorescence polychromatic super-resolution imaging realized by the method does not need to switch a complex electromechanical control system such as a filter, and the method has single-frame polychromatic imaging capability by utilizing the random encoding and compression of light field high-dimensional information.

Although the full-light-field camera imaging technology and the GISC multicolor coding super-resolution microscopy technology are both the prior art, the integration of the full-light-field camera imaging technology and the GISC multicolor coding super-resolution microscopy technology has the difficulty that:

1 a high degree of integration of the illumination mode. Compared with the traditional bimodal imaging, the device integrates and designs the different illumination modes of the bimodal imaging into an independent unit. The invention integrates the illumination modes of phase imaging and fluorescence imaging to a high degree, and uses the optical fiber beam combiner and the optical fiber collimator as the integration key components, so that the multicolor illumination of the multicolor fluorescence imaging can be integrated into a single optical fiber to vertically irradiate a sample.

2 single frame imaging capability. The phase imaging part of the traditional dual-mode imaging needs multi-angle illumination, multi-frame acquisition meets the constraint condition of solving the phase information, and the double-sided acquisition constraint of the full-light field camera can meet the requirement of solving the phase information under the low sampling rate. In the traditional bimodal super-resolution imaging part, multi-angle moire modulation of the SIM technology is needed, and therefore multi-frame acquisition is needed. The invention adopts the GISC super-resolution algorithm of intensity association and compressed sensing, and can perform super-resolution imaging on fluorescent signals acquired by a single frame.

3 multicolor imaging capability. The traditional bimodal imaging has no report of realizing multicolor imaging at present, and the invention is expected to realize breakthrough. The super-resolution of polychromatic depends on the compression encoding capability of the random phase modulator for the multi-wavelength signal. Polychromatic phase imaging techniques rely on polychromatic illumination and polychromatic acquisition of real image plane intensities of full-field cameras.

The foregoing description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention, and various modifications can be made to the above-described embodiment of the present invention. All simple, equivalent changes and modifications made in accordance with the claims and the specification of the present application fall within the scope of the patent claims. The present invention is not described in detail in the conventional art.

Claims

1. A bimodal real-time microscopic imaging device comprising:

an illumination assembly configured to illuminate a sample to be measured;

an imaging assembly configured to receive an optical signal from a sample to be measured and emit a scattered optical signal and a fluorescent signal in different optical paths;

a full-field camera configured to receive scattered light signals emitted from the imaging assembly;

a fluorescence imaging two-dimensional array detector and a GISC camera configured to switchably receive fluorescence signals emanating from the imaging assembly with one another; and

And the image operation processor is electrically connected with the full light field camera, the fluorescence imaging two-dimensional array detector and the GISC camera.

2. The dual modality live microscopy imaging system of claim 1, wherein the illumination assembly includes a plurality of excitation light sources, a fiber optic combiner coupled to the plurality of excitation light sources, and a fiber optic collimator coupled to the fiber optic combiner by a single fiber.

3. The bimodal real-time microscopic imaging device according to claim 2, wherein the illumination assembly is configured to emit monochromatic light by turning on one of the plurality of excitation light sources, or to emit polychromatic light by the plurality of excitation light sources in time sequence or simultaneously; and the laser output of the excitation light source is combined into a single optical fiber through an optical fiber beam combiner, and the sample to be measured is collimated and irradiated through the optical fiber collimator.

4. The dual-modality real-time microscopic imaging device of claim 1, wherein the sample to be measured is mounted on a displacement table, the displacement table being a two-dimensional displacement table.

5. The bimodal real-time microscopic imaging device according to claim 1, wherein the full-field camera is configured to split the scattered light signal by using a full-field camera beam splitter, wherein one of the scattered light signals split into two paths is directly detected by the first full-field camera two-dimensional array detector on the imaging surface, and the other path is detected by the second full-field camera two-dimensional array detector on the back focal surface after being transformed by the full-field camera imaging lens, and then the phase of the scattered light signal of the object is obtained by reconstruction recovery by a fourier iterative algorithm.

6. The bimodal real-time microscopic imaging device according to claim 1, wherein the GISC camera includes a random phase modulator positioned a first distance Z1 from the imaging plane of the fluorescent signal, and a GISC camera two-dimensional array detector positioned a second distance Z2 behind the random phase modulator to record the fluorescent speckle signal, the first distance Z1 and the second distance Z2 satisfying speckle detection optimization conditions; the GISC camera is set to carry out association operation reconstruction on the obtained fluorescence speckle signals and a preset calibration matrix to obtain multicolor super-resolution fluorescence signals.

7. The bimodal real time microscopic imaging device according to claim 1, wherein the imaging assembly is configured to emit the scattered light signal to a first light path and the full light field camera is located on the first light path; the imaging component is arranged to switchably emit the fluorescent signal to a second light path and a third light path, the fluorescent imaging two-dimensional array detector is located on the second light path, and the GISC camera is located on the third light path.

8. The bimodal real-time microscopic imaging device according to claim 5, wherein the imaging assembly comprises an objective lens, a dichroic mirror and a switching turning mirror which are sequentially arranged along the trend of the light path; the dichroic mirror is configured to reflect and emit a scattered light signal in the light signal to the first light path and transmit and emit a fluorescent light signal in the light signal to the second light path; the switching turning mirror is arranged to reflect the fluorescent signal propagating along the second optical path to the third optical path when turning to the first position, and to enable the fluorescent signal propagating along the second optical path to continue to propagate along the second optical path when turning to the second position; the switching turning mirror is positioned on the second optical path.

9. The bimodal real time microscopic imaging device according to claim 1, wherein the imaging assembly is configured to emit the scattered light signal to a first light path and the full light field camera is located on the first light path; the imaging component is configured to emit the fluorescent signal to a second optical path, and the fluorescent imaging two-dimensional array detector and the GISC camera are switchably located on the second optical path.

10. The bimodal real-time microscopic imaging device according to claim 1, wherein the image operation processor has a data processing module;

the image operation processor is configured to:

step S1: outputting signals acquired by the full-light field camera and the GISC camera or by the full-light field camera and the fluorescence imaging two-dimensional array detector synchronously;

step S2: caching the acquired data and transmitting the data to a data processing module of an image operation processor;

step S3: calculating and reconstructing the acquired signals by using a prefabricated full light field phase recovery algorithm and a GISC super-resolution algorithm in the data processing module to obtain a reconstruction result;

step S4: and outputting a reconstruction result.

11. A method for phase and fluorescence microscopy bimodal imaging under monochromatic illumination, comprising:

Step A0: building the bimodal real-time microscopic imaging device according to one of claims 1-10;

step A1: the illumination component emits monochromatic light, and the fluorescent imaging two-dimensional array detector is switched to receive fluorescent signals emitted from the imaging component;

step A2: controlling the full light field camera and the fluorescence imaging two-dimensional array detector to synchronously acquire signals by utilizing an image operation processor, caching acquired data, and calculating and reconstructing the acquired data by utilizing a data processing module to obtain a reconstruction result; and then outputting a reconstruction result.

12. The phase and GISC fluorescence super-resolution bimodal imaging method under monochromatic illumination is characterized by comprising the following steps:

step B0: building the bimodal real-time microscopic imaging device according to one of claims 1-10;

step B1: the illumination component emits monochromatic light, and the GISC camera is switched to receive fluorescent signals emitted from the imaging component;

step B2: controlling the full light field camera and the GISC camera to synchronously acquire signals by utilizing an image operation processor, caching acquired data, and calculating and reconstructing the acquired data by utilizing a data processing module to obtain a reconstruction result; and then outputting a reconstruction result.

13. The full-color phase and GISC multicolor fluorescence super-resolution bimodal imaging method under multicolor time sequence illumination is characterized by comprising the following steps:

step C0: building the bimodal real-time microscopic imaging device according to one of claims 1-10;

step C1: the illumination assembly is utilized to emit monochromatic light each time according to the time sequence so as to emit light with different wavelengths, and the GISC camera is switched to receive fluorescent signals emitted from the imaging assembly;

step C2: controlling the full light field camera and the GISC camera to synchronously perform time sequence acquisition signals by utilizing an image operation processor, caching acquired data, and performing calculation and reconstruction on the time sequence acquired data by utilizing a data processing module to obtain a time sequence image as a reconstruction result; then outputting a reconstruction result; and synthesizing a full-color phase image and a multicolor fluorescence image according to the time sequence image.

14. The full-color phase and GISC multi-color fluorescent super-resolution bimodal imaging method under multi-color simultaneous illumination is characterized by comprising the following steps:

step D0: building the bimodal real-time microscopic imaging device according to one of claims 1-10; the first full-field camera two-dimensional array detector and the second full-field camera two-dimensional array detector in the full-field camera of the bimodal real-time microscopic imaging device are all RGB color array detectors, so that the full-field camera forms a color full-field camera;

Step D1: the illumination component is used for emitting polychromatic light simultaneously, and the GISC camera is switched to receive fluorescent signals emitted from the imaging component;

step D2: controlling the color full-light field camera and the GISC camera to synchronously acquire signals by utilizing an image operation processor, and caching acquired data; decomposing data output by the color full-light field camera according to RGB three-color channels by utilizing an image operation processor to obtain decomposed three-color channel signals, and respectively calculating and reconstructing the decomposed three-color channel signals by utilizing a data processing module to obtain a reconstruction result, and re-superposing the reconstruction result to generate a full-color phase image; calculating and reconstructing data output by the GISC camera by using an image operation processor to obtain a multicolor fluorescence image; thereby obtaining a full-color phase image and a multicolor fluorescence image.