WO2023005671A1

WO2023005671A1 - Correction method and apparatus for large-field-of-view high-resolution light field microscopic system

Info

Publication number: WO2023005671A1
Application number: PCT/CN2022/105493
Authority: WO
Inventors: 戴琼海; 吴嘉敏
Original assignee: 清华大学
Priority date: 2021-07-28
Filing date: 2022-07-13
Publication date: 2023-02-02
Also published as: CN113327211B; CN113327211A

Abstract

The present application relates to the technical fields of computer graphics and computer vision. Provided are a correction method and apparatus for a large-field-of-view high-resolution light field microscopic system. The method comprises: acquiring light field images at a plurality of axial positions of a calibration target; acquiring a white light field image, which is not added to the calibration target; acquiring a noise image; processing the white light field image and the noise image, so as to obtain a light field decoding parameter; decoding the light field images by using the light field decoding parameter, so as to obtain four-dimensional light field data; and correcting the distortion of an imaging system by using the four-dimensional light field data, which corresponds to the light field images at the plurality of axial positions. By means of the present application comprising the solution, a large-field-of-view light field is corrected by using light field images of a calibration target at different axial positions, thereby alleviating the imaging distortion effect of the large-field-of-view light field, and improving a large-field-of-view high-resolution light field microscopic three-dimensional reconstruction result.

Description

Calibration method and device for large-field-of-view high-resolution light-field microscopy system

Cross References to Related Applications

This application is based on a Chinese patent application with application number 202110854589.1 and a filing date of July 28, 2022, and claims the priority of this Chinese patent application. The entire content of this Chinese patent application is hereby incorporated by reference into this application.

technical field

The present application relates to the technical fields of computer graphics and computer vision, and in particular to a correction method and device for a large-field-of-view high-resolution light-field microsystem.

Background technique

Light field microscopy is an important tool for three-dimensional observation of cells. There is a compromise between spatial resolution and field of view in traditional light field microscopy systems, and the field of view of light field microscopy systems with micron-level resolution is generally 1 mm or hundreds of microns. However, many biological activities, such as brain nerve activity and tumor cell migration, have a relatively wide range of activities, generally at the millimeter level or larger. In order to realize the three-dimensional observation of cell activity process with micron-scale resolution within a large field of view, a high-resolution light field microscope system with a large field of view was constructed. However, there is obvious field curvature in the imaging of the large field of view system, which is not conducive to the subsequent calculation and reconstruction of the three-dimensional spatial structure of the sample.

Contents of the invention

This application aims to solve one of the technical problems in the related art at least to a certain extent.

For this reason, the first purpose of this application is to propose a large-field-of-view high-resolution light-field microscopy system correction method, which solves the technical problem that the field curvature existing in the large-field light-field microscopy system causes image distortion at various angles of view. By collecting the light field images of the calibration plate at different axial positions, estimating the optical path difference of each viewing angle, using the estimated optical path difference data to correct the images of each viewing angle, and then reconstructing the spatial structure of the sample with the three-dimensional deconvolution method, the large viewing angle is realized. Field light field correction alleviates the influence of imaging distortion in the large field of view light field, and improves the results of large field of view high-resolution light field microscopic 3D reconstruction.

The second purpose of the present application is to propose a correction device for a large-field-of-view high-resolution light-field microscope system.

The third object of the present application is to propose a non-transitory computer-readable storage medium.

In order to achieve the above purpose, the embodiment of the first aspect of the present application proposes a correction method for a large-field-of-view high-resolution light field microscope system, including: collecting light field images at multiple axial positions of the calibration plate; white light field image; collect noise image; process white light field image and noise image to obtain light field decoding parameters; use light field decoding parameters to decode light field image to obtain 4D light field data; use multiple axial positions The 4D light field data corresponding to the light field image at is corrected for imaging system distortion.

Optionally, in one embodiment of the present application, the light field images at multiple axial positions of the calibration plate are collected, specifically:

Place the calibration plate near the front focal plane of the objective lens, keep the calibration plate perpendicular to the optical axis of the system, and translate the large-field light-field microscope system or the calibration plate along the axial direction to realize the positioning of the calibration plate by the large-field light-field microscope system Light field imaging at different axial positions.

Optionally, in one embodiment of the present application, the white light field image without adding the calibration plate is collected, specifically:

Keep the same light intensity as the light field image of the calibration plate, use the same exposure time as the light field of the calibration plate, or use a variety of different exposure times to collect white light field images respectively.

Optionally, in one embodiment of the present application, the noise image is collected, specifically:

Turn off or block the illumination of the large-field light-field microscope system, and collect noise images of the large-field light-field microscope system under different exposure times.

Optionally, in one embodiment of the present application, the white light field image and the noise image are processed to obtain light field decoding parameters, specifically:

By analyzing the white light field image and noise image, eliminating bad camera pixels, reducing the influence of ambient stray light and camera dark current, determining the coordinates of the center point of each microlens, the angle between the microlens and the camera, and then determining the light field decoding parameters.

Optionally, in one embodiment of the present application, light field decoding parameters are used to decode the light field image of the calibration plate to obtain 4-dimensional light field data, and the calibration plate is split to be imaged under different viewing angles.

Optionally, in one embodiment of the present application, the imaging system distortion is corrected using 4-dimensional light field data corresponding to light field images at multiple axial positions, specifically:

Using the characteristics of the 4-dimensional light field data corresponding to the light field images at multiple axial positions, determine the focal plane parameters of the objective lens of the large-field light-field microscope system, so as to determine the position of each point in the large-field light-field microscope system Variation law of optical path difference at different viewing angles;

The images of each viewing angle are corrected according to the variation rule of the obtained optical path difference, and then the three-dimensional deconvolution method is used to reconstruct the spatial structure of the sample.

In order to achieve the above purpose, the embodiment of the second aspect of the present application proposes a correction device for a large-field-of-view high-resolution light field microscope system, including: an acquisition module, a processing module, a decoding module, and a correction module, wherein,

The acquisition module is used to collect light field images at multiple axial positions of the calibration plate, white light field images and noise images not added to the calibration plate;

A processing module, configured to process white light field images and noise images to obtain light field decoding parameters;

The decoding module is used to decode the light field image using light field decoding parameters to obtain 4-dimensional light field data;

The correction module is used to correct the distortion of the imaging system by using the 4-dimensional light field data corresponding to the light field images at multiple axial positions.

In order to achieve the above purpose, the embodiment of the third aspect of the present application proposes a non-transitory computer-readable storage medium. When the instructions in the storage medium are executed by the processor, a large-field-of-view high-resolution light-field display can be executed. Calibration methods for microsystems.

The large-field-of-view high-resolution light-field microsystem calibration method, the large-field-of-view high-resolution light-field microsystem calibration device, and the non-temporary computer-readable storage medium of the embodiments of the present application solve the problem of the large-field-of-view light-field microsystem The existing field curvature leads to the technical problem of image distortion at each viewing angle. By collecting the light field images of the calibration plate at different axial positions, the optical path difference of each viewing angle is estimated, and the estimated optical path difference data is used to correct the images of each viewing angle, and then the three-dimensional The deconvolution method reconstructs the spatial structure of the sample, realizes the correction of the large field of view light field, alleviates the influence of imaging distortion in the large field of view light field, and improves the microscopic three-dimensional reconstruction results of the large field of view and high-resolution light field.

Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.

Description of drawings

The above and/or additional aspects and advantages of the present application will become apparent and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, wherein:

Fig. 1 is a flow chart of a correction method for a large-field-of-view high-resolution light-field microscope system provided in Embodiment 1 of the present application;

Fig. 2 is another flow chart of the calibration method of the large-field-of-view high-resolution light-field microscope system according to the embodiment of the present application;

FIG. 3 is a schematic structural diagram of a correction device for a large-field-of-view high-resolution light-field microscope system provided in Embodiment 2 of the present application.

Detailed ways

Embodiments of the present application are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary, and are intended to explain the present application, and should not be construed as limiting the present application.

The calibration method and device of the large-field-of-view high-resolution light-field microscopy system according to the embodiments of the present application will be described below with reference to the accompanying drawings.

FIG. 1 is a flow chart of a correction method for a large-field-of-view high-resolution light-field microscope system provided in Embodiment 1 of the present application.

As shown in Figure 1, the calibration method of the large-field-of-view high-resolution light-field microscope system includes the following steps:

Step 101, collecting light field images at multiple axial positions of the calibration plate;

Step 102, collecting the white light field image without adding the calibration plate;

Step 103, collecting noise images;

Step 104, processing the white light field image and the noise image to obtain light field decoding parameters;

Step 105, using light field decoding parameters to decode the light field image to obtain 4D light field data;

Step 106 , using the 4-dimensional light field data corresponding to the light field images at multiple axial positions to correct the distortion of the imaging system.

The correction method of the large-field-of-view high-resolution light field microscope system in the embodiment of the present application collects the light field images at multiple axial positions of the calibration plate; collects the white light field image without adding the calibration plate; collects the noise image; Process the image and the noise image to obtain the light field decoding parameters; use the light field decoding parameters to decode the light field image to obtain 4D light field data; use the 4D light field data corresponding to the light field images at multiple axial positions Corrects for imaging system distortion. Therefore, it can solve the technical problem that the field curvature of the large-field light field microscope system causes the image distortion of each viewing angle. By collecting the light field images of the calibration plate at different axial positions, the optical path difference of each viewing angle can be estimated, and the estimation can be obtained The optical path difference data corrects the images of each viewing angle, and then uses the three-dimensional deconvolution method to reconstruct the sample space structure, which realizes the correction of the large field of view light field, alleviates the influence of imaging distortion in the large field of view light field, and improves the height of the large field of view. Resolved light field microscopy 3D reconstruction results.

Further, in the embodiment of the present application, the light field images at multiple axial positions of the calibration plate are collected, specifically:

The purpose of collecting multiple light field images of the calibration plate at the axial position is to improve the estimation accuracy of the parameters of the focal surface in the field of view.

Further, in the embodiment of the present application, the white light field image without adding the calibration plate is collected, specifically:

Further, in the embodiment of the present application, the noise image is collected, specifically:

Further, in the embodiment of the present application, the white light field image and the noise image are processed to obtain the light field decoding parameters, specifically:

Further, in the embodiment of the present application, the light field image of the calibration plate is decoded using the light field decoding parameters to obtain 4-dimensional light field data, and the calibration plate is split to be imaged under different viewing angles.

Further, in the embodiment of the present application, the distortion of the imaging system is corrected by using the 4-dimensional light field data corresponding to the light field images at multiple axial positions, specifically:

By imaging the light field with the calibration plate at different axial positions, the distortion calibration of the entire field of view is realized, and the optical path difference of the images of each viewing angle of the light field is determined.

FIG. 2 is another flow chart of the calibration method of the large-field-of-view high-resolution light-field microscopy system according to the embodiment of the present application.

As shown in Figure 2, the calibration method of the large-field-of-view high-resolution light field microscopy system includes: step S101, collecting white light field images; step S102, collecting noise images, keeping the light intensity constant, and collecting white light fields under different exposure times Image and noise image; step S103, keep the light intensity constant, adjust the axial position of the calibration plate, and collect the light field image sequence of the calibration plate; step S104, use the results of step S101 and step S102 to calculate and determine the microlens center coordinates, microlens and Parameters such as the angle between the cameras and the distance between the centers of the microlenses; step S105, use the results of steps S103 and S104 to decode and generate the 4-dimensional light field data of the calibration plate at different axial positions; step S106, use the image features of each viewing angle to estimate the focus Surface parameters, so as to determine the optical path difference of each point of the image.

As shown in Figure 3, the calibration device of the large-field-of-view high-resolution light-field microscope system includes: an acquisition module, a processing module, a decoding module, and a calibration module, wherein,

The collection module 10 is used to collect light field images at multiple axial positions of the calibration plate, white light field images and noise images not added to the calibration plate;

A processing module 20, configured to process the white light field image and the noise image to obtain light field decoding parameters;

A decoding module 30, configured to decode the light field image using light field decoding parameters to obtain 4-dimensional light field data;

The correction module 40 is configured to use 4-dimensional light field data corresponding to the light field images at multiple axial positions to correct distortion of the imaging system.

The calibration device of the large-field-of-view high-resolution light-field microscope system in the embodiment of the present application includes: an acquisition module, a processing module, a decoding module, and a calibration module, wherein the acquisition module is used to acquire light at multiple axial positions of the calibration plate. field image, the white light field image and the noise image without adding the calibration plate; the processing module is used to process the white light field image and the noise image to obtain the light field decoding parameters; the decoding module is used to use the light field decoding parameters to decode the light field image Decoding is performed to obtain 4-dimensional light field data; a correction module is used to correct the distortion of the imaging system by using the 4-dimensional light field data corresponding to the light field images at multiple axial positions. Therefore, it can solve the technical problem that the field curvature of the large-field light field microscope system causes the image distortion of each viewing angle. By collecting the light field images of the calibration plate at different axial positions, the optical path difference of each viewing angle can be estimated, and the estimation can be obtained The optical path difference data corrects the images of each viewing angle, and then uses the three-dimensional deconvolution method to reconstruct the sample space structure, which realizes the correction of the large field of view light field, alleviates the influence of imaging distortion in the large field of view light field, and improves the height of the large field of view. Resolved light field microscopy 3D reconstruction results.

In order to realize the above-mentioned embodiments, the present application also proposes a non-transitory computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the large-field-of-view high-resolution light-field microscope System Calibration Method.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the described specific features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples and features of different embodiments or examples described in this specification without conflicting with each other.

In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present application, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

Any process or method descriptions in flowcharts or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing custom logical functions or steps of a process , and the scope of preferred embodiments of the present application includes additional implementations in which functions may be performed out of the order shown or discussed, including in substantially simultaneous fashion or in reverse order depending on the functions involved, which shall It should be understood by those skilled in the art to which the embodiments of the present application belong.

The logic and/or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium, For use with instruction execution systems, devices, or devices (such as computer-based systems, systems including processors, or other systems that can fetch instructions from instruction execution systems, devices, or devices and execute instructions), or in conjunction with these instruction execution systems, devices or equipment for use. For the purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use in or in conjunction with an instruction execution system, device or device. More specific examples (non-exhaustive list) of computer-readable media include the following: electrical connection with one or more wires (electronic device), portable computer disk case (magnetic device), random access memory (RAM), Read Only Memory (ROM), Erasable and Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program can be printed, since the program can be read, for example, by optically scanning the paper or other medium, followed by editing, interpretation or other suitable processing if necessary. The program is processed electronically and stored in computer memory.

It should be understood that each part of the present application may be realized by hardware, software, firmware or a combination thereof. In the embodiments described above, various steps or methods may be implemented by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented by any one or a combination of the following techniques known in the art: a discrete Logic circuits, ASICs with suitable combinational logic gates, Programmable Gate Arrays (PGA), Field Programmable Gate Arrays (FPGA), etc.

Those of ordinary skill in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. During execution, one or a combination of the steps of the method embodiments is included.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, each unit may exist separately physically, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. If the integrated modules are realized in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium.

The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, and the like. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limitations on the present application, and those skilled in the art can make the above-mentioned The embodiments are subject to changes, modifications, substitutions and variations.

Claims

A correction method for a large field of view and high resolution light field microscope system, characterized in that it comprises the following steps:

Collect light field images at multiple axial positions of the calibration plate;

collecting white light field images not added to the calibration plate;

Acquire noisy images;

Processing the white light field image and the noise image to obtain light field decoding parameters;

Decoding the light field image using the light field decoding parameters to obtain 4-dimensional light field data;

The distortion of the imaging system is corrected by using the 4-dimensional light field data corresponding to the light field images at the multiple axial positions.
The correction method for a large-field-of-view high-resolution light-field microscope system according to claim 1, wherein the collection of light-field images at multiple axial positions of the calibration plate is specifically:

Place the calibration plate near the front focal plane of the objective lens, keep the calibration plate perpendicular to the optical axis of the system, and translate the large-field light-field microscope system or the calibration plate along the axial direction to realize the positioning of the calibration plate by the large-field light-field microscope system Light field imaging at different axial positions.
The method for correcting a large-field-of-view high-resolution light field microscope system according to claim 1, wherein the collection of white light field images without adding the calibration plate is specifically:

Keep the same light intensity as the light field image of the calibration plate, use the same exposure time as the light field of the calibration plate, or use a variety of different exposure times to collect white light field images respectively.
The method for correcting a large-field-of-view high-resolution light-field microscope system according to claim 1, wherein said collecting noise images is specifically:

Turn off or block the illumination of the large-field light-field microscope system, and collect noise images of the large-field light-field microscope system under different exposure times.
The method for correcting a large-field-of-view high-resolution light field microscope system according to claim 1, wherein said processing said white light field image and said noise image obtains light field decoding parameters, specifically:

By analyzing the white light field image and the noise image, eliminating camera dead pixels, reducing the influence of ambient stray light and camera dark current, determining the coordinates of the center point of each microlens, the angle between the microlens and the camera, and then determining the light field decoding parameter.
The correction method for a large-field-of-view high-resolution light field microscope system according to claim 1, wherein the light field image of the calibration plate is decoded using the light field decoding parameters to obtain 4-dimensional light field data, and the calibration plate is split Imaging at different viewing angles.
The method for correcting a large-field-of-view high-resolution light field microscope system according to claim 1, wherein the correction of distortion of the imaging system is carried out by using the 4-dimensional light field data corresponding to the light field images at the multiple axial positions ,Specifically:

The characteristics of the 4-dimensional light field data corresponding to the light field images at the multiple axial positions are used to determine the focal plane parameters of the objective lens of the large field of view light field microscope system, thereby determining the large field of view light field display. Variation law of optical path difference at each point in the microsystem at each viewing angle;

The images of each viewing angle are corrected according to the variation rule of the obtained optical path difference, and then the three-dimensional deconvolution method is used to reconstruct the spatial structure of the sample.
A correction device for a large-field-of-view high-resolution light-field microscope system, characterized in that it includes an acquisition module, a processing module, a decoding module, and a correction module, wherein,

The collection module is used to collect light field images at multiple axial positions of the calibration plate, white light field images and noise images not added to the calibration plate;

The processing module is configured to process the white light field image and the noise image to obtain light field decoding parameters;

The decoding module is configured to use the light field decoding parameters to decode the light field image to obtain 4-dimensional light field data;

The correction module is configured to use the 4-dimensional light field data corresponding to the light field images at the multiple axial positions to correct the distortion of the imaging system.
A non-transitory computer-readable storage medium, on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the large-field-of-view high-resolution light as described in any one of claims 1-7 is realized. Calibration methods for field microscopy systems.