CN114690391A - Light sheet fluorescence microscope, image processing system and image processing method - Google Patents
Light sheet fluorescence microscope, image processing system and image processing method Download PDFInfo
- Publication number
- CN114690391A CN114690391A CN202011601521.4A CN202011601521A CN114690391A CN 114690391 A CN114690391 A CN 114690391A CN 202011601521 A CN202011601521 A CN 202011601521A CN 114690391 A CN114690391 A CN 114690391A
- Authority
- CN
- China
- Prior art keywords
- image
- dimensional
- dimensional images
- sample
- image processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 77
- 238000003672 processing method Methods 0.000 title claims abstract description 23
- 238000003384 imaging method Methods 0.000 claims abstract description 36
- 230000003287 optical effect Effects 0.000 claims abstract description 28
- 238000000034 method Methods 0.000 claims description 24
- 230000008569 process Effects 0.000 claims description 11
- 238000009877 rendering Methods 0.000 claims description 11
- 238000005070 sampling Methods 0.000 claims description 7
- 238000012937 correction Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 239000011159 matrix material Substances 0.000 description 6
- 238000004590 computer program Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000013507 mapping Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000005284 excitation Effects 0.000 description 1
- 238000000799 fluorescence microscopy Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000035515 penetration Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
- G02B21/365—Control or image processing arrangements for digital or video microscopes
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/16—Microscopes adapted for ultraviolet illumination ; Fluorescence microscopes
Abstract
The application discloses a light sheet fluorescence microscope, an image processing system, an image processing method and a removable recording medium. The light sheet fluorescence microscope includes: a sample stage; a light emitting unit configured to emit a light sheet toward a sample on the sample stage, the light sheet being at a predetermined angle with respect to a surface of the sample stage; and an imaging unit configured to image the sample to acquire a plurality of two-dimensional images of the sample on a plane parallel to the light sheet for reconstruction processing of a three-dimensional image based on the plurality of two-dimensional images, wherein an optical axis of the imaging unit is perpendicular to the light sheet, and the imaging unit is further configured such that a first physical direction in the physical space is mapped to a first image direction in the image space corresponding to a data row direction of a memory for storing the plurality of two-dimensional images, wherein the first physical direction is a normal direction of a surface of the sample stage, and wherein the predetermined angle is greater than 0 degree and less than 90 degrees.
Description
Technical Field
The present disclosure relates to the field of image processing, and in particular to an optical sheet fluorescence microscope, an image processing system, an image processing method, and a removable recording medium.
Background
Light Sheet Fluorescence Microscopy collimates the laser beam to form a "Light Sheet" that illuminates only the plane to be viewed without background Light, so that it can be imaged at high speed (e.g., 512 × 5121000 frames/s, 2048 × 204850 frames/s) directly using an imaging device such as a CCD camera, for example, and three-dimensional reconstruction such as Maximum Intensity Projection (MIP) or Minimum Intensity Projection (min-IP) is performed.
Disclosure of Invention
The following presents a simplified summary of the disclosure in order to provide a basic understanding of some aspects of the disclosure. However, it should be understood that this summary is not an exhaustive overview of the disclosure. It is not intended to identify key or critical elements of the disclosure or to delineate the scope of the disclosure. Its sole purpose is to present some concepts of the disclosure in a simplified form as a prelude to the more detailed description that is presented later.
An object of the present disclosure is to provide an improved optical sheet fluorescence microscope, an image processing system including the optical sheet fluorescence microscope, and an image processing method based on the optical sheet fluorescence microscope.
According to an aspect of the present disclosure, there is provided a light sheet fluorescence microscope including: a sample stage; a light emitting unit configured to emit a sheet of light toward a sample on the sample stage, the sheet of light being at a predetermined angle to a surface of the sample stage; and an imaging unit configured to image the sample to acquire a plurality of two-dimensional images of the sample on a plane parallel to the optical sheet for reconstruction processing of a three-dimensional image based on the plurality of two-dimensional images, wherein an optical axis of the imaging unit is perpendicular to the optical sheet, and the imaging unit is further configured such that a first physical direction in a physical space is mapped to a first image direction in an image space corresponding to a data row direction of a memory for storing the plurality of two-dimensional images, wherein the first physical direction is a normal direction of a surface of the sample stage, and wherein the predetermined angle is greater than 0 degree and less than 90 degrees.
According to another aspect of the present disclosure, there is provided an image processing system including the above-described light sheet fluorescence microscope, the image processing system further including: an image processing unit configured to process a plurality of two-dimensional images of the sample acquired via the imaging unit to reconstruct a three-dimensional image of the sample.
According to another aspect of the present disclosure, an image processing method based on the above light sheet fluorescence microscope is provided, including: acquiring, with the imaging unit, a plurality of two-dimensional images of the sample on a plane parallel to the light sheet for reconstruction processing of a three-dimensional image based on the plurality of two-dimensional images while moving the light sheet or the sample stage in a second physical direction in a physical space, wherein the second physical direction is parallel to a surface of the sample stage and orthogonal to a third physical direction in the physical space, and wherein the third physical direction is parallel to the surface of the sample stage and parallel to the light sheet.
According to other aspects of the present disclosure, there are also provided computer program code and a computer program product for implementing the above-described method according to the present disclosure, and a computer readable storage medium having recorded thereon the computer program code for implementing the above-described method according to the present disclosure.
Additional aspects of the disclosed embodiments are set forth in the description section that follows, wherein the detailed description is presented to fully disclose the preferred embodiments of the disclosed embodiments without imposing limitations thereon.
Drawings
The disclosure may be better understood by reference to the following detailed description taken in conjunction with the accompanying drawings, in which like or similar reference numerals are used throughout the figures to designate like or similar components. The accompanying drawings, which are incorporated in and form a part of the specification, further illustrate preferred embodiments of the present disclosure and explain the principles and advantages of the present disclosure, are incorporated in and form a part of the specification. Wherein:
fig. 1 is a block diagram showing a functional configuration example of an optical sheet fluorescence microscope 100 according to an embodiment of the present disclosure;
fig. 2 is a schematic diagram illustrating a hardware configuration example of the optical sheet fluorescence microscope 100 according to an embodiment of the present disclosure;
fig. 3A shows an example of a mapping relationship between a physical space and an image space of a related art light-sheet fluorescence microscope in an example case where the imaging unit 106 includes a CCD camera;
fig. 3B illustrates an example of a mapping relationship between a physical space and an image space of the light-sheet fluorescence microscope 100 of the embodiment of the present disclosure in an example case where the imaging unit 106 includes a CCD camera;
fig. 4 is a block diagram showing a functional configuration example of an image processing system 400 according to an embodiment of the present disclosure;
fig. 5A and 5B show an example of an original two-dimensional image and an example of an expanded two-dimensional image, respectively;
FIG. 6 illustrates an example of three-dimensional reconstruction performance of an image processing system 400 according to an embodiment of the present disclosure with an exemplary sub-sampling operation employed;
fig. 7 is a flowchart illustrating an example of a flow of an image processing method according to an embodiment of the present disclosure; and
fig. 8 is a block diagram showing an example structure of a personal computer employable in the embodiments of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure will be described hereinafter with reference to the accompanying drawings. In the interest of clarity and conciseness, not all features of an actual implementation are described in the specification. It will of course be appreciated that in the development of any such actual embodiment, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which will vary from one implementation to another. Moreover, it will be appreciated that such a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking for those of ordinary skill in the art having the benefit of this disclosure.
Note that, herein, "first physical direction", "second physical direction", and "third physical direction" respectively denote a first direction, a second direction, and a third direction in a physical space. In addition, "first image direction", "second image direction", and "third image direction" respectively denote a first direction, a second direction, and a third direction in the image space.
Here, it should be further noted that, in order to avoid obscuring the present disclosure with unnecessary details, only the device structures and/or processing steps closely related to the scheme according to the present disclosure are shown in the drawings, and other details not so relevant to the present disclosure are omitted.
Embodiments according to the present disclosure are described in detail below with reference to the accompanying drawings.
First, an implementation example of the light sheet fluorescence microscope 100 according to the embodiment of the present disclosure will be described with reference to fig. 1 and 2. Fig. 1 is a block diagram illustrating a functional configuration example of an optical sheet fluorescence microscope 100 according to an embodiment of the present disclosure. Fig. 2 is a schematic diagram illustrating a hardware configuration example of the optical sheet fluorescence microscope 100 according to an embodiment of the present disclosure.
As shown in fig. 1 and 2, the light sheet fluorescence microscope 100 may include a sample stage 102, a light emission unit 104, and an imaging unit 106.
Light emitting unit 104 may be configured to emit a sheet of light at a predetermined angle θ with respect to the surface of sample stage 102 toward a sample on sample stage 102. Wherein the predetermined angle θ may be greater than 0 degree and less than 90 degrees. For example, the predetermined angle θ may be greater than 0 degrees and less than 60 degrees. However, the predetermined angle θ is not limited to the above example, but a person skilled in the art may set the predetermined angle θ according to actual needs.
For example, as shown in fig. 2, the light emitting unit 104 may include a light source and an excitation objective lens.
The imaging unit 106 may be configured to image the specimen to acquire a plurality of two-dimensional images of the specimen on a plane parallel to the light sheet (hereinafter, the acquired two-dimensional images are also referred to as "slice images") for reconstruction processing of a three-dimensional image based on the acquired plurality of two-dimensional images. The optical axis of the imaging unit 106 is perpendicular to the light sheet, and the imaging unit 106 may be further configured such that a first physical direction in the physical space (e.g., a y "axis direction shown in fig. 2) is mapped to a first image direction in the image space (e.g., a logical y axis direction shown in fig. 3A and 3B) corresponding to a data row direction of a memory for storing the above-described plurality of two-dimensional images, wherein the first physical direction is a normal direction of the surface of the sample stage. For example, the first physical direction may be mapped to a first image direction corresponding to a data line direction of a memory for storing an image by rotating the imaging unit 106 by a certain angle (e.g., 90 degrees) with respect to a related art imaging unit in which the first physical direction is mapped to a second image direction (e.g., an x-axis direction shown in fig. 3A and 3B).
For example, as shown in fig. 2, the imaging unit 106 may include an imaging objective lens, a filter, a tube lens, and an imaging device.
For example, the imaging unit 106 may include a Charge Coupled Device (CCD) camera. Furthermore, those skilled in the art can make the imaging unit 106 include a suitable imaging device other than a CCD camera according to actual needs, which will not be described herein.
Note that although a specific configuration of the light emitting unit 104 and the imaging unit 106 is shown in fig. 2, the specific configuration shown in fig. 2 is merely an example, and a person skilled in the art may adopt an appropriate specific configuration of the light emitting unit 104 and the imaging unit 106 according to actual needs.
Fig. 3A and 3B illustrate an example of a mapping relationship between a physical space and an image space of a related art optical sheet fluorescence microscope and an example of a mapping relationship between a physical space and an image space of the optical sheet fluorescence microscope 100 of an embodiment of the present disclosure, respectively, in an example case where the imaging unit 106 includes a CCD camera.
As shown in fig. 3A, for a prior art light sheet fluorescence microscope, a first physical direction (e.g., the y "axis direction) is mapped to a second image direction (e.g., the x axis direction) in the image space. On the other hand, as shown in fig. 3B, for the light sheet fluorescence microscope 100 of the embodiment of the present disclosure, a first physical direction (e.g., y "axis direction) is mapped to a first image direction (e.g., y axis direction) in the image space. Since the first image direction corresponds to a data line direction of a memory (e.g., a memory), that is, the first image direction coincides with a memory address direction (e.g., a memory line direction stored in a C + + memory), it is possible to efficiently copy image data of an acquired two-dimensional image to a desired position in the memory, reduce data processing time, and improve data processing efficiency.
Note that, in fig. 3A and 3B, for convenience of explanation, a case where the optical sheet is perpendicular to a first physical plane (i.e., a plane parallel to the surface of the sample stage, for example, an x "z" plane) in a physical space defined by a second physical direction (for example, a z "axis direction) and a third physical direction (for example, an x" axis direction) is shown, however, in actual use, due to the light path characteristics of the optical sheet fluorescence microscope, as shown in fig. 1, the optical sheet is not perpendicular to the first physical plane, but the optical sheet is at a predetermined angle θ greater than 0 degrees and less than 90 degrees with the first physical plane.
According to an embodiment of the present disclosure, there may also be provided an image processing system including the above-described optical sheet fluorescence microscope 100. Fig. 4 is a block diagram showing a functional configuration example of an image processing system 400 according to an embodiment of the present disclosure.
As shown in fig. 4, the image processing system 400 may include the light sheet fluorescence microscope 100 and an image processing unit 402. The image processing unit 402 may be configured to process a plurality of two-dimensional images of the sample acquired via the imaging unit 106 to reconstruct a three-dimensional image of the sample.
For example, according to an embodiment of the present disclosure, the image processing unit 402 may expand the plurality of two-dimensional images based on the predetermined angle θ, a distance between adjacent two-dimensional images of the plurality of two-dimensional images, and the number of images of the plurality of two-dimensional images, thereby acquiring a plurality of expanded two-dimensional images, and reconstruct a three-dimensional image of the sample based on the plurality of expanded two-dimensional images using a MIP or MinIP based three-dimensional reconstruction method. Wherein the plurality of expanded two-dimensional images are aligned with each other in a first image direction (e.g., y-axis direction). The image processing unit 402 may construct a projection coordinate system based on a predetermined angle θ at the time of three-dimensional graphics rendering, thereby correcting an angle between the expanded two-dimensional image and a second image plane (for example, xy plane in fig. 5B) defined by the first image direction and the second image direction in the image space. For example, the image processing unit 402 may construct the projection coordinate system based on the predetermined angle θ such that the height direction of the reconstructed three-dimensional image (i.e., the direction corresponding to the first image direction) is parallel to the vertical direction in the projection space.
For example, image processing unit 402 may perform three-dimensional graphics rendering using an OpenGL-based GPU rendering method.
For example, the image processing unit 402 may also output image data of the expanded two-dimensional image into a file (such as a TIF file) so that a user may analyze the image data offline and/or perform offline three-dimensional reconstruction based on saved image data. For example, the image processing unit 402 may perform real-time three-dimensional graphics rendering while saving a plurality of extended two-dimensional images.
The image expansion processing performed by the image processing unit 402 will be further described with reference to fig. 5A and 5B in conjunction with a specific example. Note that in the following description of the image expansion processing, for convenience of description, it is assumed that the specimen stage or the optical sheet is moved in the z "axis direction shown in fig. 2, and the image to be acquired is a slice image of the specimen corresponding to the x" y "plane shown in fig. 2 (i.e., an image parallel to the xy plane in the image space).
As described above, the light sheet is not perpendicular to the first physical plane due to the light path characteristics of a light sheet fluorescence microscope. The depth of penetration of the laser through the sample is fixed due to energy constraints, and during movement of the sample stage or slide along the z "axis as shown in fig. 1, as shown in fig. 5A, two problems arise: 1) as the number of layers increases, the bottom of the acquired image in the y-axis (an example of the first image direction) increases from low to high; 2) the inclination angle theta' between the acquired slice image of the sample and the actual xy plane is 90-theta. Both of the above problems cause some distortion of the slice body (i.e., the three-dimensional image) reconstructed based on the original slice image acquired by the light sheet fluorescence microscope.
As shown in fig. 5A, since the slice image is perpendicular to the yz plane and the xz 'plane and the slice image is at a predetermined angle θ with the xz plane, the y-axis is at an angle θ with respect to the z' -axis corresponding to the bottom of the plurality of slice images. Therefore, the height H' of the sheet body in the y-axis, which is constituted by the acquired plural sheet images, can be represented by the following formula (1).
H' ═ H + N step cos (θ) formula (1)
In equation (1) above, H is the height of a single slice image on the y-axis, step is the step size of movement of the sample stage or slide (i.e., the distance between adjacent slice images), N is the number of slice images and N is a natural number greater than 0.
As can be seen from the above formula (1), the height H' of the sheet body is greater than the height of the individual sheet image. Thus, for example, a plurality of expanded images aligned with each other in the y-axis may be acquired by expanding a plurality of slice images in the y-axis.
As an example, each image may be expanded by zero-padding an area other than the area occupied by the image in a predetermined image area for the image. The predetermined image area is determined based on a predetermined angle, a distance between adjacent two-dimensional images of the plurality of two-dimensional images, and the number of images of the plurality of two-dimensional images.
For example, the height of the predetermined image area in the y-axis may be set as the height H' of the sheet body, and the width of the predetermined image area in the x-axis may be set as the width of the sheet body (i.e., the width of the sheet image).
The coordinate y of the bottom of the N (N is more than 0 and less than or equal to N) slice image on the y axis in the whole slice bodynRepresented by the following formula (2):
ynh' -nstep cos (θ) formula (2)
For example, for the nth slice image, the coordinate y of the slice image on the y-axis determined via equation (2) above may be basednThe slice image is copied to a predetermined image area, and the image is expanded by zero-padding an area other than the area occupied by the slice image in the predetermined image area, as shown in fig. 5B. In fig. 5B, the expanded portion of each slice image is shown by a dotted line frame, and the slice body is shown by a solid line frame.
As an example, the expansion of the slice image may be done simultaneously with the image data transfer of the slice image. For example, a region corresponding to the slice body in a memory (e.g., a memory) for storing the plurality of two-dimensional images may be filled with zero values in advance, and then the image data of the N slice images may be copied to corresponding regions in the region corresponding to the slice body based on the coordinates of the N slice images on the y axis determined via equation (2) above to obtain image data of the slice body composed of the N expanded slice images. Since the memory copy operation (memcpy) is very efficient, the extension of the slice image can be efficiently achieved.
As described above, there is some distortion of the slice body (i.e., the three-dimensional image) reconstructed based on the original slice image acquired by the light sheet fluorescence microscope. To eliminate or reduce warping, for example, warping correction may be performed prior to three-dimensional reconstruction. In the prior art, the distortion correction is performed by a rotational matrix transform. For example, in the case where the sheet body is X × Y × Z and the number of channels is C, the calculated amount of the distortion correction by the rotation matrix conversion is a linear multiple of X × Y × Z × C. For example, a typical configuration of an optical sheet fluorescence microscope is X Y2048, Z256, and C4. For this configuration of the light sheet fluorescence microscope, the number of elements of the rotational matrix is 2048 × 256 × 4 — 8G.
In practical use, it is desirable to reconstruct a three-dimensional image in real time to facilitate a user to view the three-dimensional structure of a sample in real time during photographing. In order to reconstruct a three-dimensional image in real time, it is desirable that the time for three-dimensional reconstruction processing of one slice body should not exceed the acquisition time of the next slice body. For example, for the configuration of the above-described light sheet fluorescence microscope, if the exposure time is 20ms, the acquisition time of one sheet body is Z × C × 20ms ≈ 20 s. Due to the large computational load of the rotation matrix transformation, in order to complete the distortion correction and the three-dimensional image reconstruction within the acquisition time of one slice, the prior art method performs the processing of the distortion correction and the three-dimensional image reconstruction on the GPU or GPU cluster (see, for example, https:// www.intelligent-imaging. com/wp-content/updates/2019/01/DDN-Micro volume-White-Paper-V1. pdf). However, this method requires loading the entire slice block data to the GPU, and thus requires more GPU memory. For example, for the configuration of the above-described light sheet fluorescence microscope, even if only the source sheet body and the destination sheet body are stored, a GPU memory of 16GB is required regardless of other intermediate results. In addition, the exchange of 16GB of data from memory to video memory and vice versa requires a data bus with greater throughput. The need for large capacity GPU memory and large throughput data buses results in higher cost for the prior art approach.
As described above, the image processing system 400 according to the embodiment of the present disclosure may obtain the distortion-corrected slice body by expanding the plurality of slice images in the first image direction to obtain a plurality of expanded slice images aligned with each other in the first direction and introducing the tilt angle at the time of three-dimensional graphic rendering. Such processing of the image processing system 400 according to the embodiment of the present disclosure makes it unnecessary to perform tilt correction on an original image by rotational matrix transformation, so that data processing speed and efficiency can be improved, saving processing time and memory of a processor such as a GPU. Further, such processing of the image processing system 400 according to the embodiment of the present disclosure makes it possible to realize real-time three-dimensional reconstruction on, for example, an ordinary notebook computer or a workstation, and thus can reduce costs.
Considering that the user mainly wants to be able to see the rough effect quickly first in real-time three-dimensional reconstruction, a somewhat lower resolution may be tolerated, so for example the image processing unit 402 may sub-sample a plurality of expanded images and reconstruct a three-dimensional image of the sample based on the expanded images obtained via the sub-sampling, in order to reduce the amount of data, so that the processing time and/or the requirements on the processing performance of the image processing unit 402 may be further reduced.
For example, the plurality of expanded images may be 1/2, 1/4, 1/8, etc. sub-sampled according to a user's hardware configuration (e.g., a video memory size of the GPU).
For example, according to an embodiment of the present disclosure, the image processing unit 402 may perform a reconstruction process of a three-dimensional image of a sample in a C + + environment.
Fig. 6 illustrates an example of three-dimensional reconstruction performance of an image processing system 400 in accordance with an embodiment of the present disclosure, with an exemplary sub-sampling operation being employed. For the example of fig. 6, 1/2 subsampling is performed for X and Y, and not for Z. As shown in fig. 6, for the sheet body of X × Y × Z × C × 1024 × 241 × 4, the three-dimensional reconstruction times in the configurations of the notebook computer (configured as Intel i5-7200@2.5G, GPU-Nvidia GeForce 940MX,2GB) and the Workstation (configured as (HP Z6 workation, Intel Xeon 51223.62666 MHz 4C cpu. GPU-Nvidia P4000,8GB) are 9s and 8s, respectively, both of which are less than the photographing time (20s), thus making it possible to facilitate the user to view the three-dimensional structure of the sample in real time during photographing.
Having described the image processing system according to an embodiment of the present disclosure above, the present disclosure also provides the following embodiments of an image processing method based on a light sheet fluorescence microscope, corresponding to the above-described embodiments of the image processing system.
Fig. 7 is a flowchart illustrating an example of a flow of an image processing method 700 according to an embodiment of the present disclosure. As shown in fig. 7, an image processing method 700 according to an embodiment of the present disclosure may begin at a start step S702 and end at an end step S708. The image processing method 700 may include a two-dimensional image acquisition step S704.
In the two-dimensional image acquisition step S704, a plurality of two-dimensional images of the specimen on a plane parallel to the optical sheet are acquired with the imaging unit 106 for the reconstruction process of the three-dimensional image based on the plurality of two-dimensional images while moving the optical sheet or the specimen stage in a second physical direction (for example, the z "direction shown in fig. 2) in the physical space. The second physical direction is parallel to the surface of the sample stage and orthogonal to a third physical direction in physical space (e.g., the x "direction shown in fig. 2). The third physical direction is parallel to the surface of the sample stage and parallel to the light sheet.
For example, according to an embodiment of the present disclosure, the image processing method 700 may further include a three-dimensional image reconstruction step S706. In the three-dimensional image reconstruction step, a plurality of images of the sample acquired via the imaging unit 106 may be processed to reconstruct a three-dimensional image of the sample. For example, the three-dimensional image reconstruction step S706 may be implemented by the image processing unit 402 in the above image processing system 400, and thus specific details may be referred to the above detailed description of the image processing unit 402.
For example, according to an embodiment of the present disclosure, in the three-dimensional image reconstruction step S706, the plurality of two-dimensional images may be expanded based on the predetermined angle θ, the distance between adjacent two-dimensional images of the plurality of two-dimensional images, and the number of images of the plurality of two-dimensional images, thereby obtaining a plurality of expanded two-dimensional images, and a three-dimensional image of the sample may be reconstructed based on the plurality of expanded two-dimensional images using a MIP or MinIP based three-dimensional reconstruction method. Wherein the plurality of expanded two-dimensional images are aligned with each other in a first image direction (e.g., y-axis direction). Further, the projection coordinate system may be constructed based on a predetermined angle at the time of three-dimensional graphics rendering, thereby correcting an angle between the expanded two-dimensional image and a second image plane (for example, xy plane in fig. 5B) defined by the first image direction and the second image direction in the image space. For example, the projection coordinate system may be constructed based on the predetermined angle θ such that the height direction of the reconstructed three-dimensional image (i.e., the direction corresponding to the first image direction) is parallel to the vertical direction in the projection space. For example, in the three-dimensional image reconstruction step S706, a GPU rendering method based on OpenGL may be adopted for three-dimensional image rendering.
As an example, the expansion of the slice image may be done simultaneously with the image data transfer of the slice image. For example, a region corresponding to the slice body in a memory (e.g., a memory) for storing the plurality of two-dimensional images may be filled with zero values in advance, and then the image data of the N slice images may be copied to corresponding regions in the region corresponding to the slice body based on the coordinates of the N slice images on the y axis determined via equation (2) above to obtain image data of the slice body composed of the N expanded slice images. Since the memory copy operation (memcpy) is very efficient, the extension of the slice image can be efficiently achieved.
As described above, the image processing method 700 according to the embodiment of the present disclosure may obtain the distortion-corrected slice body by expanding the plurality of slice images in the first image direction to obtain a plurality of expanded slice images aligned with each other in the first direction and introducing the tilt angle at the time of three-dimensional graphic rendering. Such processing of the image processing method 700 according to the embodiment of the present disclosure makes it unnecessary to perform tilt correction on an original image through rotational matrix transformation, so that data processing speed and efficiency can be improved, saving processing time and memory of a processor such as a GPU. Further, such processing of the image processing method 700 according to the embodiment of the present disclosure makes it possible to implement real-time three-dimensional reconstruction on, for example, an ordinary notebook computer or a workstation, and thus can reduce costs.
In the three-dimensional image reconstruction step S706, the plurality of expanded images may be sub-sampled, and a three-dimensional image of the sample may be reconstructed based on the expanded images obtained via the sub-sampling, so as to reduce the amount of data, so that the processing time and/or the demand for processing performance may be further reduced.
For example, the plurality of expanded images may be 1/2, 1/4, 1/8 sub-sampled according to a hardware configuration of the user (e.g., a video memory size of the GPU).
For example, according to an embodiment of the present disclosure, the three-dimensional image reconstruction step S706 may be performed in a C + + environment.
It should be noted that although the functional configurations and operations of the light sheet fluorescence microscope, the image processing system, and the image processing method according to the embodiments of the present disclosure are described above, this is merely an example and not a limitation, and a person skilled in the art may modify the above embodiments according to the principles of the present disclosure, for example, functional blocks and operations in the respective embodiments may be added, deleted, or combined, and such modifications fall within the scope of the present disclosure.
In addition, it should be further noted that the method embodiments herein correspond to the system embodiments described above, and therefore, the contents that are not described in detail in the method embodiments may refer to the descriptions of the corresponding parts in the system embodiments, and the description is not repeated here.
In addition, the present disclosure also provides a storage medium and a program product. It should be understood that the machine-executable instructions in the storage medium and the program product according to the embodiments of the present disclosure may also be configured to perform the above-described image processing method, and thus, the contents not described in detail herein may refer to the description of the corresponding parts previously, and the description will not be repeated herein.
Accordingly, storage media for carrying the above-described program products comprising machine-executable instructions are also included in the present disclosure. Including, but not limited to, floppy disks, optical disks, magneto-optical disks, memory cards, memory sticks, and the like.
Further, it should be noted that the above series of processes and means may also be implemented by software and/or firmware. In the case of implementation by software and/or firmware, a program constituting the software is installed from a storage medium or a network to a computer having a dedicated hardware structure, such as a general-purpose personal computer 800 shown in fig. 8, which is capable of executing various functions and the like when various programs are installed.
In fig. 8, a Central Processing Unit (CPU)801 executes various processes in accordance with a program stored in a Read Only Memory (ROM)802 or a program loaded from a storage section 808 to a Random Access Memory (RAM) 803. In the RAM 803, data necessary when the CPU 801 executes various processes and the like is also stored as necessary.
The CPU 801, the ROM802, and the RAM 803 are connected to each other via a bus 804. An input/output interface 805 is also connected to the bus 804.
The following components are connected to the input/output interface 805: an input portion 806 including a keyboard, a mouse, and the like; an output section 807 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker and the like; a storage section 808 including a hard disk and the like; and a communication section 809 including a network interface card such as a LAN card, a modem, and the like. The communication section 809 performs communication processing via a network such as the internet.
A drive 810 is also connected to the input/output interface 805 as needed. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 810 as necessary, so that a computer program read out therefrom is installed in the storage portion 808 as necessary.
In the case where the above-described series of processes is realized by software, a program constituting the software is installed from a network such as the internet or a storage medium such as the removable medium 811.
It will be understood by those skilled in the art that such a storage medium is not limited to the removable medium 811 shown in fig. 8 in which the program is stored, distributed separately from the apparatus to provide the program to the user. Examples of the removable medium 811 include a magnetic disk (including a floppy disk (registered trademark)), an optical disk (including a compact disk read only memory (CD-ROM) and a Digital Versatile Disk (DVD)), a magneto-optical disk (including a Mini Disk (MD) (registered trademark)), and a semiconductor memory. Alternatively, the storage medium may be the ROM802, a hard disk included in the storage section 808, or the like, in which programs are stored and which are distributed to users together with the apparatus including them.
The preferred embodiments of the present disclosure are described above with reference to the drawings, but the present disclosure is of course not limited to the above examples. Various changes and modifications within the scope of the appended claims may be made by those skilled in the art, and it should be understood that these changes and modifications naturally will fall within the technical scope of the present disclosure.
For example, a plurality of functions included in one unit may be implemented by separate devices in the above embodiments. Alternatively, a plurality of functions implemented by a plurality of units in the above embodiments may be implemented by separate devices, respectively. In addition, one of the above functions may be implemented by a plurality of units. Needless to say, such a configuration is included in the technical scope of the present disclosure.
In this specification, the steps described in the flowcharts include not only the processing performed in time series in the described order but also the processing performed in parallel or individually without necessarily being performed in time series. Further, even in the steps processed in time series, needless to say, the order can be changed as appropriate.
Claims (16)
1. A light sheet fluorescence microscope, comprising:
a sample stage;
a light emitting unit configured to emit a sheet of light toward a sample on the sample stage, the sheet of light being at a predetermined angle to a surface of the sample stage; and
an imaging unit configured to image the sample to acquire a plurality of two-dimensional images of the sample on a plane parallel to the light sheet for reconstruction processing of a three-dimensional image based on the plurality of two-dimensional images,
wherein an optical axis of the imaging unit is perpendicular to the light sheet, and the imaging unit is further configured such that a first physical direction in a physical space is mapped to a first image direction in an image space corresponding to a data line direction of a memory for storing the plurality of two-dimensional images,
wherein the first physical direction is a normal direction of a surface of the sample stage, an
Wherein the predetermined angle is greater than 0 degrees and less than 90 degrees.
2. The light sheet fluorescence microscope of claim 1, wherein the predetermined angle is greater than 0 degrees and less than 60 degrees.
3. The light sheet fluorescence microscope of claim 1 or 2, wherein the imaging unit comprises a CCD camera.
4. An image processing system comprising the light sheet fluorescence microscope of any of claims 1 to 3, the image processing system further comprising:
an image processing unit configured to process a plurality of two-dimensional images of the sample acquired via the imaging unit to reconstruct a three-dimensional image of the sample.
5. The image processing system of claim 4, wherein the image processing unit reconstructs a three-dimensional image of the sample by: expanding the plurality of two-dimensional images in the first image direction based on the predetermined angle, a distance between adjacent two-dimensional images of the plurality of two-dimensional images, and a number of images of the plurality of two-dimensional images, thereby obtaining a plurality of expanded two-dimensional images, wherein the plurality of expanded two-dimensional images are aligned with each other in the first image direction; and
reconstructing a three-dimensional image of the sample based on the plurality of extended two-dimensional images using a three-dimensional reconstruction method based on a maximum intensity projection or a minimum intensity projection,
wherein a projection coordinate system is constructed based on the predetermined angle at the time of three-dimensional graphic rendering.
6. The image processing system of claim 5, wherein the image processing unit is further configured to sub-sample the plurality of expanded two-dimensional images and reconstruct a three-dimensional image of the sample based on the expanded two-dimensional images obtained via sub-sampling.
7. The image processing system according to claim 5 or 6, wherein, for each of the plurality of two-dimensional images, the two-dimensional image is expanded by zero-value-filling a region other than a region occupied by the two-dimensional image in a predetermined image region,
wherein the predetermined image area is determined based on the predetermined angle, a distance between adjacent two-dimensional images of the plurality of two-dimensional images, and the number of images of the plurality of two-dimensional images.
8. The image processing system according to claim 7, wherein the expansion of the plurality of two-dimensional images is completed while transferring image data of the plurality of two-dimensional images to the memory.
9. The image processing system according to claim 8, wherein the image processing unit performs the reconstruction processing of the three-dimensional image of the sample in a C + + environment.
10. An image processing method based on the light sheet fluorescence microscope according to any one of claims 1 to 3, comprising:
acquiring a plurality of two-dimensional images of the sample on a plane parallel to the sheet of light with the imaging unit while moving the sheet of light or the sample stage in a second physical direction in a physical space for a reconstruction process of a three-dimensional image based on the plurality of two-dimensional images,
wherein the second physical direction is parallel to the surface of the sample stage and orthogonal to a third physical direction in physical space, an
Wherein the third physical direction is parallel to the surface of the sample stage and parallel to the lightsheet.
11. The image processing method of claim 10, wherein the three-dimensional image of the sample is reconstructed by: expanding the plurality of two-dimensional images in the first image direction based on the predetermined angle, a distance between adjacent two-dimensional images of the plurality of two-dimensional images, and a number of images of the plurality of two-dimensional images, thereby obtaining a plurality of expanded two-dimensional images, wherein the plurality of expanded two-dimensional images are aligned with each other in the first image direction; and reconstructing a three-dimensional image of the sample based on the plurality of extended two-dimensional images using a three-dimensional reconstruction method based on a maximum intensity projection or a minimum intensity projection,
wherein a projection coordinate system is constructed based on the predetermined angle at the time of three-dimensional graphic rendering.
12. The image processing method of claim 11, wherein the processing further comprises sub-sampling the plurality of expanded two-dimensional images and reconstructing a three-dimensional image of the sample based on the expanded images obtained via sub-sampling.
13. The image processing method according to claim 11 or 12, wherein, for each of the plurality of two-dimensional images: the two-dimensional image is expanded by zero-padding an area other than the area occupied by the two-dimensional image in a predetermined image area,
wherein the predetermined image area is determined based on the predetermined angle, a distance between adjacent two-dimensional images of the plurality of two-dimensional images, and the number of images of the plurality of two-dimensional images.
14. The image processing method according to claim 13, wherein the expansion of the plurality of two-dimensional images is completed while transferring the image data of the plurality of two-dimensional images to the memory.
15. The image processing method according to claim 14, wherein the reconstruction process of the three-dimensional image of the sample is performed in a C + + environment.
16. A removable recording medium storing a program that, when executed by a computer, causes the computer to execute the method according to any one of claims 10 to 15.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011601521.4A CN114690391A (en) | 2020-12-29 | 2020-12-29 | Light sheet fluorescence microscope, image processing system and image processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011601521.4A CN114690391A (en) | 2020-12-29 | 2020-12-29 | Light sheet fluorescence microscope, image processing system and image processing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114690391A true CN114690391A (en) | 2022-07-01 |
Family
ID=82132196
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011601521.4A Pending CN114690391A (en) | 2020-12-29 | 2020-12-29 | Light sheet fluorescence microscope, image processing system and image processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114690391A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1376895A (en) * | 2001-03-25 | 2002-10-30 | 欧姆龙株式会社 | Optical metering installation |
| CN103743714A (en) * | 2014-01-14 | 2014-04-23 | 苏州大猫单分子仪器研发有限公司 | Inclined wide-field optical section scanning imaging microscope system and imaging method thereof |
| CN110168609A (en) * | 2017-01-09 | 2019-08-23 | 卡尔蔡司显微镜有限责任公司 | For generating the method and digital microscope of the threedimensional model of sample in digital microscope |
| CN110335220A (en) * | 2019-05-16 | 2019-10-15 | 中国林业科学研究院资源信息研究所 | A kind of image interfusion method based on parallel algorithms |
| CN111220615A (en) * | 2019-10-29 | 2020-06-02 | 怀光智能科技（武汉）有限公司 | Inclined three-dimensional scanning microscopic imaging system and method |
-
2020
- 2020-12-29 CN CN202011601521.4A patent/CN114690391A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1376895A (en) * | 2001-03-25 | 2002-10-30 | 欧姆龙株式会社 | Optical metering installation |
| CN103743714A (en) * | 2014-01-14 | 2014-04-23 | 苏州大猫单分子仪器研发有限公司 | Inclined wide-field optical section scanning imaging microscope system and imaging method thereof |
| CN110168609A (en) * | 2017-01-09 | 2019-08-23 | 卡尔蔡司显微镜有限责任公司 | For generating the method and digital microscope of the threedimensional model of sample in digital microscope |
| CN110335220A (en) * | 2019-05-16 | 2019-10-15 | 中国林业科学研究院资源信息研究所 | A kind of image interfusion method based on parallel algorithms |
| CN111220615A (en) * | 2019-10-29 | 2020-06-02 | 怀光智能科技（武汉）有限公司 | Inclined three-dimensional scanning microscopic imaging system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20190163716A1 (en) | Method and apparatus for performing operation of convolutional layers in convolutional neural network | |
| Xu et al. | High-performance iterative electron tomography reconstruction with long-object compensation using graphics processing units (GPUs) | |
| DE102005008824B4 (en) | A method of registering two images using a graphics processor and the computer readable program storage device therefor | |
| CN109643458B (en) | System and method for automated sinogram completion, combining, and completion by combining | |
| US11244481B2 (en) | Multi-scale image reconstruction of three-dimensional objects | |
| JP6259531B2 (en) | Image filtering method and CT system | |
| JP7083015B2 (en) | Super-resolution processing equipment, methods and programs | |
| JP2013125519A (en) | Image processing apparatus, data structure, and image generating apparatus | |
| JP7292893B2 (en) | Medical image processing device and medical image diagnostic device | |
| JP4769441B2 (en) | Image processing apparatus and image processing method | |
| US9728001B2 (en) | Processing and rendering of large image files | |
| JP2003334188A (en) | Three-dimensional back projection method and x-ray ct apparatus | |
| CN114690391A (en) | Light sheet fluorescence microscope, image processing system and image processing method | |
| JP4032357B2 (en) | Image information processing apparatus and method, and program | |
| Brix et al. | Visualization of large volumetric multi-channel microscopy data streams on standard PCs | |
| Bromberger et al. | FPGA-accelerated Richardson-Lucy deconvolution for 3D image data | |
| CN115005847A (en) | Static CT system based on addressable nanowire cold cathode X-ray planar source | |
| Lee et al. | Fast hybrid CPU-and GPU-based CT reconstruction algorithm using air skipping technique | |
| Sun et al. | Spatial resolution enhancement based on detector displacement for computed tomography | |
| US20070064000A1 (en) | System and method for interleaved slice volume rendering | |
| JP6443574B1 (en) | Ray casting program, search control data, search control data generation method, and ray casting apparatus | |
| Fernández et al. | Electron tomography of complex biological specimens on the Grid | |
| Wang et al. | Image gradient decomposition for parallel and memory-efficient ptychographic reconstruction | |
| US20040240719A1 (en) | Method for producing images in spiral computed tomography, and a spiral CT unit | |
| JP2005326332A (en) | Image data processor, image data processing method and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |