WO2023280292A1

WO2023280292A1 - Fast-scanning and three-dimensional imaging method and device for large-volume scattered sample

Info

Publication number: WO2023280292A1
Application number: PCT/CN2022/104531
Authority: WO
Inventors: 戴琼海; 陈一帆; 吴嘉敏
Original assignee: 清华大学
Priority date: 2021-07-08
Filing date: 2022-07-08
Publication date: 2023-01-12
Also published as: CN113554744A

Abstract

The present application provides a fast-scanning and three-dimensional imaging method and device for a large-volume scattered sample, the method comprising: imaging the sample to obtain a light field image of the sample in different directions; rearranging the light field image to obtain sub-aperture images under different angles; obtaining a point spread function of the light field imaging system by means of computing simulation; and using an aberration estimation and synchronous reconstruction algorithm to perform three-dimensional reconstruction according to the point spread function and on the basis of the sub-aperture images. The method used in the present invention enables rapid in-situ three-dimensional imaging of the large-volume sample. A three-dimensional refractive index distribution of the sample is obtained by means of multi-angle aberration estimation, and a three-dimensional reconstruction result free from aberration is finally obtained by means of reconstruction.

Description

A fast-scanning three-dimensional imaging method and device for a large-volume scattering sample

Cross References to Related Applications

This application is based on the Chinese patent application with the application number 202110773712.7 and the filing date is July 08, 2021, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby incorporated by reference into this application.

technical field

The invention relates to the technical field of optical imaging systems, in particular to a fast-scanning three-dimensional imaging method and device for large-volume scattering samples.

Background technique

Light field imaging is a fast three-dimensional imaging method, which records four-dimensional light field information by adding a microlens array in the optical path. Since the light field records the spatial and angular information at the same time, the three-dimensional information of the scene can be obtained by processing a single or multiple light field images.

Computed tomography is a three-dimensional imaging method, which uses detectors to image the imaging object from multiple angles to obtain spatial and angular information, and then performs back-projection reconstruction through algorithms to obtain high-quality three-dimensional information of the detected object.

In the process of multi-angle sampling, multiple lenses are often required, which makes the system complex, or the object needs to be rotated, making it impossible to perform real-time imaging in situ. In macroscopic imaging scenarios, optical elements such as lenses will introduce aberrations; in microscopic imaging, the scattering and refraction of large samples will cause non-negligible aberrations, and the aberration space at different positions in the sample is inconsistent. These aberrations will lead to inaccurate reconstruction results. In practical applications, aberration information needs to be extracted and corrected during the reconstruction process. (In addition, because the point spread function of the actual collection is different from that of the simulation, it is generally necessary to calibrate the point spread function of the system, which makes the reconstruction process more complicated.)

Contents of the invention

The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

In order to solve the deficiencies in the prior art, the purpose of the present invention is to combine computed tomography and light field imaging methods to perform in-situ rapid multi-angle scanning imaging for large-volume fluorescent samples, and perform phase difference correction and accurate three-dimensional reconstruction.

For this reason, the first purpose of the present invention is to propose a fast scanning three-dimensional imaging method for large-volume scattering samples, in order to combine computed tomography and light field imaging methods to perform in-situ rapid multi-angle scanning imaging for large-volume fluorescent samples, and Perform phase correction and accurate 3D reconstruction.

The second object of the present invention is to propose a fast-scanning three-dimensional imaging device for large-volume scattering samples.

In order to achieve the above purpose, the embodiment of the first aspect of the present invention proposes a fast-scanning three-dimensional imaging method for large-volume scattering samples, including:

S1, imaging the sample, and obtaining light field images of the sample in different directions;

S2, rearranging the light field images to obtain sub-aperture images at different angles;

S3, obtaining the point spread function of the light field imaging system through calculation and simulation;

S4, registering the image pair, and calculating a coordinate transformation relationship of the image pair;

S5. According to the point spread function and based on the sub-aperture image, perform three-dimensional reconstruction using a phase difference estimation synchronous reconstruction algorithm.

In addition, the fast-scanning three-dimensional imaging method for large-volume scattering samples according to the above-mentioned embodiments of the present invention may also have the following additional technical features:

Further, in an embodiment of the present invention, the imaging of the sample and obtaining light field images of the sample in different directions include:

The sample is imaged from different angles, imaged at multiple angles at certain angular intervals each time, and imaged after each rotation is completed.

Further, in one embodiment of the present invention, the point spread function of the light field imaging system obtained through calculation and simulation includes:

The forward propagation process of the optical path is simulated, the complex light field is calculated, and the light intensity distribution is obtained through the forward propagation process of the optical path through phase modulation, so as to obtain the sub-aperture point spread function corresponding to each depth.

Further, in an embodiment of the present invention, the phase matrix and the three-dimensional reconstruction result are alternately optimized based on the alternating direction multiplier method.

Further, in an embodiment of the present invention, the phase matrix and the intensity matrix are initialized with all zeros.

Further, in an embodiment of the present invention, the intensity matrix and the phase matrix are optimized, wherein,

Select a rotation imaging angle, and obtain a phase difference correction matrix through order accumulation, the phase difference correction matrix performs forward projection on the intensity matrix, and combines the phase difference correction matrix to perform correction to obtain a forward projection result, and calculates the forward projection The calculation result is obtained from the error matrix between the result and the current sub-aperture true value, and based on the calculation result, a gradient descent algorithm is respectively performed on the intensity matrix and the phase matrix for optimization.

Further, in an embodiment of the present invention, said optimizing the intensity matrix also includes:

Back projection is performed in combination with the phase difference correction matrix, and the error matrix is projected onto the volume, so as to optimize the intensity matrix.

Further, in an embodiment of the present invention, the intensity matrix and the phase matrix are optimized using sub-apertures of all rotation angles sequentially or in random order.

Further, in an embodiment of the present invention, the optimization of the intensity matrix and the phase matrix is repeated until the two converge, and the intensity matrix is the result of the three-dimensional reconstruction after phase correction.

The fast-scanning three-dimensional imaging method for a large-volume scattering sample in the embodiment of the present invention obtains light field images of the sample in different directions by imaging the sample, rearranges the light field images to obtain sub-aperture images at different angles, and obtains through calculation and simulation The point spread function of the light field imaging system, according to the point spread function, and based on the sub-aperture image, the 3D reconstruction is performed using the phase difference estimation simultaneous reconstruction algorithm. By adopting the above method, the present invention can quickly perform in-situ rapid three-dimensional imaging on large-volume samples, obtain the three-dimensional refractive index distribution of the sample through multi-angle aberration estimation, and finally obtain the three-dimensional reconstruction result with phase difference removed.

In order to achieve the above purpose, the embodiment of the second aspect of the present invention proposes a fast-scanning three-dimensional imaging device for large-volume scattering samples, including:

an imaging module, configured to image the sample, and obtain light field images of the sample in different directions;

A rearrangement module, configured to rearrange the light field images to obtain sub-aperture images at different angles;

A computing module, used to obtain the point spread function of the light field imaging system through computational simulation;

The reconstruction module is configured to perform three-dimensional reconstruction using a phase difference estimation synchronous reconstruction algorithm based on the sub-aperture image according to the point spread function.

The fast-scanning three-dimensional imaging device for a large-volume scattering sample in the embodiment of the present invention obtains light field images of the sample in different directions by imaging the sample, rearranges the light field images to obtain sub-aperture images at different angles, and obtains through calculation and simulation The point spread function of the light field imaging system, according to the point spread function, and based on the sub-aperture image, the 3D reconstruction is performed using the phase difference estimation simultaneous reconstruction algorithm. By adopting the above method, the present invention can quickly perform in-situ rapid three-dimensional imaging on large-volume samples, obtain the three-dimensional refractive index distribution of the sample through multi-angle aberration estimation, and finally obtain the three-dimensional reconstruction result with phase difference removed.

Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, wherein:

Fig. 1 is a flowchart of a fast scanning three-dimensional imaging method for a large-volume scattering sample according to an embodiment of the present invention;

FIG. 2 is a schematic structural diagram of an example of a rotating light field imaging system according to an embodiment of the present invention;

3 is a schematic diagram of an optical path of a light field imaging system according to an embodiment of the present invention;

Fig. 4 is a schematic diagram of a single light field rearrangement according to an embodiment of the present invention;

5 is a schematic diagram of a light field diagram and rearranged sub-aperture images according to an embodiment of the present invention;

FIG. 6 is a flowchart of a phase difference estimation synchronization reconstruction algorithm according to an embodiment of the present invention;

Fig. 7 is a schematic structural diagram of a fast-scanning three-dimensional imaging device for large-volume scattering samples according to an embodiment of the present invention.

detailed description

Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention.

The method and device for fast-scanning three-dimensional imaging of a large-volume scattering sample according to an embodiment of the present invention will be described below with reference to the accompanying drawings.

The rapid scanning three-dimensional imaging method of large-volume scattering samples according to the embodiment of the present application provides a fast scanning three-dimensional imaging method for large-volume scattering samples. Through a self-designed rotating mirror group, a single objective lens can be used to quickly collect fixed samples from multiple angles. And for each angle collected by the objective lens, the light field is used to simultaneously record the angle and position information of the fluorescent signal of the sample. Afterwards, the sub-aperture image is obtained by rearranging the light field data at multiple angles, and through the convex optimization algorithm ADMM (augmented Lagrangian penalty function method), based on the intensity and position information of multiple angles, for the sample at each angle The aberration is estimated, and then the three-dimensional distribution of the refractive index of the sample is obtained. In the reconstruction process, the three-dimensional intensity distribution of the fluorescent sample is corrected by using the refractive index distribution matrix, so as to obtain high-precision three-dimensional reconstruction results.

Fig. 1 is a flow chart of a fast scanning three-dimensional imaging method for a large-volume scattering sample provided by an embodiment of the present invention.

As shown in Figure 1, the rapid scanning 3D imaging method of large-volume scattering samples includes:

Step S1, imaging the sample to obtain light field images of the sample in different directions.

Specifically, the sample is imaged by a light field rotation imaging system to obtain light field images in different directions. Fig. 2 is a schematic structural diagram of an example of a rotating light field imaging system according to an embodiment of the present invention.

1) First, embed the sample in a transparent container with a smooth surface, and fix it in front of the rotating mirror group. As shown in Figure 2, during the imaging process, the rotating table drives the lens group to move, so that the objective lens can image the sample from different angles, and rotate at a certain angle interval each time for multi-angle imaging.

2) The acquisition module adopts a light field imaging system, which performs imaging after each rotation is completed. The example point light source in Figure 3 is a point source, L1 and L2 are lenses, a microlens array ML is placed on the back focal plane of L2, and an image sensor is placed on the sensor plane of the back focal plane of the microlens for light field acquisition. In the figure, f1 is the focal length of L1, f2 is the focal length of L2, and fml is the focal length of ML. The collected information is 4D information, including spatial information (2D) and angle information (2D). For example, in a microscopic scene, the light source is a fluorescent sample, and the excitation light path is not restricted. L1 and L2 are the objective lens and the tube lens, respectively.

Step S2, rearranging the light field images to obtain sub-aperture images at different angles.

It can be understood that the sub-aperture image is obtained by rearranging the light field diagram. Figure 4 is a schematic diagram of rearranging a single light field. The specific size and rearrangement order of the diagram depends on the actual situation. In Figure 3, for the sake of simplicity, each microlens corresponds to 3*3 sensor pixels, Figure 4(a) is an example of the light field diagram, and Figure 4(b), (c) are respectively taken after corresponding positions in the light field diagram. By splicing, the images of each sub-aperture can be obtained. Two of the sub-aperture images are shown in the figure, and a total of 3*3 sub-aperture images can be obtained in this example. For this light field map, it should be interpolated to the size of the original light field map 9*9. For the light field diagram obtained by scanning the light field, it also needs to be rearranged, the basic process remains unchanged, and the order of rearrangement needs to be determined according to the actual situation. Figure 5 is an example of the light field diagram and the rearranged sub-aperture image. As shown in Figure 5, the sub-aperture image obtained from the light field map contains information of different angles of the object. The example in Figure 5 has a total of 13*13 sub-apertures, corresponding to 13*13 sub-aperture images. Define the sub-aperture image corresponding to the sub-aperture (u, v) as I(u, v). Figure 4(a) is the original light field diagram, and Figure 4(b) is an example of sub-aperture images corresponding to four sub-apertures, and the corresponding apertures are (2,9),(3,9),(7,7), (1,1).

In step S3, the point spread function of the light field imaging system is obtained through calculation and simulation.

It can be understood that, before reconstruction, it is first necessary to obtain the point spread function (PSF) of the light field imaging system. Specifically, first, the forward propagation process of the optical path is simulated by a computer, and the simulated sub-aperture point spread function psf is calculated. For example, the schematic diagram of the light field system in Figure 3, starting from the point source, using a computer to simulate the forward propagation process of the optical path, calculates the complex light field on the front surface of the microlens array ML, and through the phase modulation of the microlens array, after experiencing a propagation process, the light intensity distribution of the point light source on the sensor plane on the sensor surface can be obtained. Thus, the sub-aperture point spread function corresponding to each depth can be calculated. In Fig. 3, L1 and L2 are lenses, f1 is the focal length of L1, f2 is the focal length of L2, and fml is the focal length of the microlens.

Step S4, according to the point spread function, and based on the sub-aperture image, use the phase difference estimation synchronous reconstruction algorithm to perform three-dimensional reconstruction.

It can be understood that after preprocessing the light field image and obtaining the point spread function (PSF) of the light field imaging system, the phase difference estimation and simultaneous reconstruction algorithm is used to simultaneously estimate the phase matrix from multiple angles and eliminate the phase difference for multi-angle three-dimensional reconstruction. Under the assumption of continuity, the problem is a convex problem, so the present invention alternately optimizes the phase matrix and the three-dimensional reconstruction result based on the alternate direction multiplier method (ADMM). FIG. 6 is a flowchart of a phase difference estimation synchronous reconstruction algorithm according to an embodiment of the present invention, as shown in the figure:

c. For the range of 3D reconstruction, perform all-zero initialization for the phase matrix and the intensity matrix.

d. To optimize the intensity matrix, first select a rotational imaging angle, and first accumulate the phase matrix according to the order of this direction to obtain the phase difference correction matrix in this direction. Select a sub-aperture image at this angle, and the matrix forward-projects the intensity matrix through the point spread function (PSF) of the light field imaging system, and corrects it in combination with the direction difference correction matrix to obtain the forward projection result. Calculate the error matrix between the forward projection result and the true value of the current sub-aperture, and perform optimization based on the gradient descent algorithm for the intensity matrix. (And combine the direction phase difference correction matrix to perform back projection, project the error matrix onto the volume, and optimize the intensity matrix.) Then optimize the intensity matrix using sub-apertures of all rotation angles sequentially or randomly.

e. To optimize the phase matrix, firstly select a rotation imaging angle, firstly accumulate the phase matrix according to the order of this direction, and obtain the phase difference correction matrix in this direction. Select a sub-aperture image at this angle, and the matrix forward-projects the intensity matrix through the point spread function (PSF) of the light field imaging system, and corrects it in combination with the direction difference correction matrix to obtain the forward projection result. Calculate the error matrix between the forward projection result and the true value of the current sub-aperture, and perform optimization based on the gradient descent algorithm for the phase matrix. The phase matrix is then optimized using sub-apertures for all rotation angles sequentially or out-of-order.

f. Repeat steps d and e until the two converge, then the intensity matrix is the 3D reconstruction result after phase correction.

According to the fast-scanning three-dimensional imaging method of a large-volume scattering sample provided by an embodiment of the present invention, by imaging the sample, the light field images of the sample in different directions are obtained, and the light field images are rearranged to obtain sub-aperture images at different angles. The point spread function of the light field imaging system is obtained by simulation. According to the point spread function and based on the sub-aperture image, the phase difference estimation synchronous reconstruction algorithm is used for 3D reconstruction. By adopting the above method, the present invention can quickly perform in-situ rapid three-dimensional imaging on large-volume samples, obtain the three-dimensional refractive index distribution of the sample through multi-angle aberration estimation, and finally obtain the three-dimensional reconstruction result with phase difference removed.

Fig. 7 is a schematic structural diagram of a fast-scanning three-dimensional imaging device for a large-volume scattering sample according to an embodiment of the present invention.

As shown in FIG. 7, the large-volume scattering sample rapid scanning three-dimensional imaging device 10 includes:

The imaging module 100 , the rearrangement module 200 , the calculation module 300 and the reconstruction module 400 .

The imaging module 100 is configured to image the sample and obtain light field images of the sample in different directions;

A rearrangement module 200, configured to rearrange the light field images to obtain sub-aperture images at different angles;

Calculation module 300, used to obtain the point spread function of the light field imaging system through calculation and simulation;

The reconstruction module 400 is configured to perform 3D reconstruction using a phase difference estimation synchronous reconstruction algorithm based on the point spread function and based on the sub-aperture image.

According to the large-volume scattering sample rapid scanning three-dimensional imaging device proposed in the embodiment of the present invention, by imaging the sample, the light field images of the sample in different directions are obtained, and the light field images are rearranged to obtain sub-aperture images at different angles. The point spread function of the light field imaging system is obtained by simulation. According to the point spread function and based on the sub-aperture image, the phase difference estimation synchronous reconstruction algorithm is used for 3D reconstruction. By adopting the above method, the present invention can quickly perform in-situ rapid three-dimensional imaging on large-volume samples, obtain the three-dimensional refractive index distribution of the sample through multi-angle aberration estimation, and finally obtain the three-dimensional reconstruction result with phase difference removed.

In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the described specific features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples and features of different embodiments or examples described in this specification without conflicting with each other.

Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention, those skilled in the art can make the above-mentioned The embodiments are subject to changes, modifications, substitutions and variations.

Claims

A fast-scanning three-dimensional imaging method for a large-volume scattering sample, characterized in that it includes:

Imaging the sample to obtain light field images of the sample in different directions;

Rearranging the light field images to obtain sub-aperture images at different angles;

The point spread function of the light field imaging system is obtained through calculation and simulation;

According to the point spread function and based on the sub-aperture image, a phase difference estimation synchronous reconstruction algorithm is used to perform three-dimensional reconstruction.
The fast-scanning three-dimensional imaging method for a large-volume scattering sample according to claim 1, wherein said imaging the sample to obtain light field images of the sample in different directions comprises:

The sample is imaged from different angles, imaged at multiple angles at a certain angular interval each time, and imaged after each rotation is completed.
The fast-scanning three-dimensional imaging method for a large-volume scattering sample according to claim 1, wherein the point spread function of the light field imaging system obtained through calculation and simulation includes:

The forward propagation process of the optical path is simulated, the complex light field is calculated, and the light intensity distribution is obtained through the forward propagation process of the optical path through phase modulation, so as to obtain the sub-aperture point spread function corresponding to each depth.
The fast-scanning three-dimensional imaging method for a large-volume scattering sample according to claim 1, wherein the phase matrix and the three-dimensional reconstruction results are alternately optimized based on the alternating direction multiplier method.
The fast-scanning three-dimensional imaging method for a large-volume scattering sample according to claim 4, wherein the phase matrix and the intensity matrix are initialized with all zeros.
The fast-scanning three-dimensional imaging method for a large-volume scattering sample according to claim 5, wherein the intensity matrix and the phase matrix are optimized, wherein,

Select a rotation imaging angle, and obtain a phase difference correction matrix through order accumulation, the phase difference correction matrix performs forward projection on the intensity matrix, and combines the phase difference correction matrix to perform correction to obtain a forward projection result, and calculates the forward projection The calculation result is obtained from the error matrix between the result and the current sub-aperture true value, and based on the calculation result, a gradient descent algorithm is respectively performed on the intensity matrix and the phase matrix for optimization.
The fast-scanning three-dimensional imaging method for a large-volume scattering sample according to claim 6, wherein said optimizing the intensity matrix further comprises:

Back projection is performed in combination with the phase difference correction matrix, and the error matrix is projected onto the volume, so as to optimize the intensity matrix.
The fast-scanning three-dimensional imaging method for a large-volume scattering sample according to claim 7, wherein the intensity matrix and the phase matrix are optimized using sub-apertures of all rotation angles sequentially or randomly.
The method for rapid scanning three-dimensional imaging of a large-volume scattering sample according to claim 8, wherein the intensity matrix and the phase matrix are repeatedly optimized until the two converge, and the intensity matrix is a phase-corrected The 3D reconstruction results.
A fast-scanning three-dimensional imaging device for a large-volume scattering sample, comprising:

an imaging module, configured to image the sample, and obtain light field images of the sample in different directions;

A rearrangement module, configured to rearrange the light field images to obtain sub-aperture images at different angles;

A computing module, used to obtain the point spread function of the light field imaging system through computational simulation;

The reconstruction module is configured to perform three-dimensional reconstruction using a phase difference estimation synchronous reconstruction algorithm based on the sub-aperture image according to the point spread function.