JP2023130196A - Laser scan type microscope system - Google Patents

Abstract

To suppress a distortion of an image in in-vivo imaging.SOLUTION: A laser scan type microscope 1 comprises: a scanner 104 that scans a bio-sample in a main scan direction and in a sub scan direction; a controller 200 that controls the main scan direction; and a processing device 20. The processing device 20 is configured to estimate an oscillation direction of a sample S on the basis of a preliminary image, and next, the processing device 20 is configured to cause the controller 200 to get the main scan direction close to the oscillation direction. Also, the processing device 20 is configured to estimate waveform information on the oscillation on the basis of an object image created by scanning performed in a state where the main scan direction gets close to the oscillation direction. Further, the processing device 20 is configured to perform a pixel shift to the object image in accordance with an amount of displacement occurring in each area of the sample S to be calculated based on the waveform information.SELECTED DRAWING: Figure 1

Description

TECHNICAL FIELD The disclosure herein relates to laser scanning microscopy systems.

Laser scanning microscope systems, typified by confocal microscopes, are known as observation devices used for in vivo imaging to observe living biological samples. Since the laser scanning microscope system has high resolution in the optical axis direction, it is possible to observe the inside of a biological sample, as described in Patent Document 1, for example.

Patent No. 4694139

By the way, in a laser scanning microscope system that scans a sample two-dimensionally, the information of each pixel forming an image reflects the state of the sample at different times. This creates a time difference of up to one frame between pixels, so when a biological sample is scanned, the image may be distorted due to vibrations caused by life-sustaining activities such as pulsation and breathing.

According to the technique described in Patent Document 1, it is possible to prevent focus shift in the optical axis direction due to pulsation. However, image distortion cannot be suppressed. For this reason, there is a need for new technology that suppresses image distortion caused by pulsation, breathing, and the like.

In view of the above circumstances, an object of one aspect of the present invention is to provide a technique for suppressing image distortion in in vivo imaging.

A laser scanning microscope system according to one aspect of the present invention includes a scanner that scans a biological sample in a main scanning direction and a sub-scanning direction, and a control device. The control device is configured to estimate a vibration direction of the biological sample based on a preliminary image that is an image of the biological sample generated in a first scan of the biological sample by the scanner, and to adjust the main scanning direction to the vibration direction of the biological sample. Based on a target image that is an image of the biological sample generated by a second scan of the biological sample by the scanner performed with the main scanning direction close to the vibration direction and the main scanning direction close to the vibration direction, estimating waveform information of vibrations of the biological sample; and calculating vibrations occurring in each region from the start of the second scan to the time of scanning each of the plurality of regions of the biological sample, which is calculated based on the waveform information. and performing a pixel shift on the target image according to the amount of displacement.

According to the above aspect, image distortion can be suppressed in in vivo imaging.

1 is a diagram illustrating the configuration of a laser scanning microscope system 1 according to an embodiment of the present invention. 1 is a diagram illustrating the configuration of a confocal microscope 100. FIG. FIG. 3 is a diagram for explaining a scanning method. FIG. 3 is a diagram for explaining an example of distortion occurring in an image area. FIG. 5 is a diagram for explaining the deformation of cells due to the distortion that occurs in FIG. 4. FIG. FIG. 7 is a diagram for explaining another example of distortion occurring in an image area. FIG. 7 is a diagram for explaining the deformation of cells due to the distortion that occurs in FIG. 6; FIG. 7 is a diagram for explaining yet another example of distortion occurring in an image area. FIG. 9 is a diagram for explaining the deformation of cells due to the distortion that occurs in FIG. 8; FIG. 3 is a diagram for explaining distortion that occurs in a confocal image. 2 is a flowchart of processing performed by the laser scanning microscope system 1. FIG. FIG. 3 is a diagram for explaining the relationship between a vibration waveform and a displacement amount. FIG. 3 is a diagram for explaining input and output of a learned model M1 according to the first embodiment. 3 is a flowchart of learning processing according to the first embodiment. FIG. 7 is a diagram for explaining input and output of a learned model M2 according to the second embodiment. FIG. 7 is a diagram for explaining input and output of a learned model M3 according to the second embodiment. It is a flow chart of vibration direction learning processing concerning a 2nd embodiment. It is a flowchart of waveform information learning processing concerning a 2nd embodiment. It is a flow chart of vibration direction estimation processing concerning a 3rd embodiment. FIG. 3 is a diagram illustrating the configuration of a laser scanning microscope system according to a fourth embodiment. FIG. 7 is a diagram for explaining input and output of a trained model M4 according to a fourth embodiment. It is a flowchart of learning processing concerning a 4th embodiment. FIG. 2 is a diagram illustrating the hardware configuration of a computer for realizing the processing device according to the embodiment described above. 4 is a diagram illustrating the configuration of a two-photon excitation microscope 400. FIG. 5 is a diagram illustrating the configuration of a confocal microscope 500. FIG. 6 is a diagram illustrating the configuration of a confocal microscope 600. FIG.

FIG. 1 is a diagram illustrating the configuration of a laser scanning microscope system 1 according to an embodiment of the present invention. The laser scanning microscope system 1 is a laser confocal microscope that includes a microscope device 10 and a processing device 20, and generates a confocal image of a sample S. The sample S is, for example, a biological sample such as a mouse, although it is not particularly limited. The microscope device 10 includes a confocal microscope 100 and a controller 200 that controls the confocal microscope 100.

FIG. 2 is a diagram illustrating the configuration of the confocal microscope 100. FIG. 3 is a diagram for explaining the scanning method. Hereinafter, a method for generating confocal images will be described with reference to FIGS. 2 and 3.

As shown in FIG. 2, the confocal microscope 100 includes a laser 101, a beam expander 102, a dichroic mirror 103, a scanner 104, a pupil projection optical system 105, and an objective lens 106 on the illumination optical path. ing.

The laser 101 oscillates laser light in the visible, ultraviolet, and infrared regions, for example. The output of the laser light emitted from the laser 101 is controlled by, for example, a controller 200. The beam expander 102 is an optical system that adjusts the luminous flux of the laser beam (collimated light) from the laser 101 according to the pupil diameter of the objective lens 106. The dichroic mirror 103 is a light separating means that separates excitation light (laser light) and detection light (fluorescence) from the sample S, and separates the laser light and fluorescence based on the difference in wavelength.

The scanner 104 two-dimensionally scans the sample S on the stage 107 with laser light. The scanner 104 includes, for example, two galvanometer mirrors that scan the sample S in mutually orthogonal directions, and performs raster scanning as shown in FIG. 3. That is, the scanner 104 scans the sample S in the main scanning direction with one galvano mirror, and shifts the scanning line SL in the sub-scanning direction perpendicular to the main scanning direction with the other galvano mirror, thereby scanning the sample S in the main scanning direction. The sample S is scanned in the sub-scanning direction. Note that the main scanning direction and the sub-scanning direction are controlled by the controller 200 changing the drive timing and drive speed of two galvanometer mirrors that scan in mutually orthogonal directions. Note that the galvanometer mirror is an example of a scanner mirror included in the scanner 104, and a MEMS mirror or the like can also be used.

The pupil projection optical system 105 is an optical system that projects the image of the scanner 104 onto the pupil position of the objective lens 106. Note that the objective lens 106 may include a correction ring for correcting spherical aberration that changes depending on the observation depth.

The confocal microscope 100 further includes a mirror 108, a confocal lens 109, a confocal diaphragm 110, a condensing lens 111, and a photodetector 112 on the detection optical path (transmission optical path of the dichroic mirror 103). We are prepared. The signal output from the photodetector 112 is output to the A/D converter 113.

The confocal lens 109 is a lens that focuses fluorescence onto the confocal aperture 110. The confocal diaphragm 110 is a diaphragm arranged at a position optically conjugate with the focal plane of the objective lens 106. A pinhole is formed in the confocal diaphragm 110 to transmit fluorescence generated from the focal position of the objective lens 106. The condensing lens 111 is a lens that guides the fluorescence that has passed through the confocal diaphragm 110 to the photodetector 112.

The photodetector 112 is, for example, a photomultiplier tube (PMT), and outputs an analog signal corresponding to the amount of incident fluorescent light. The A/D converter 113 converts the analog signal from the photodetector 112 into a digital signal (luminance signal) and outputs it to the processing device 20.

In the laser scanning microscope system 1 configured as described above, the confocal microscope 100 scans the sample S with laser light using the scanner 104, and collects fluorescence from each position of the sample S using the confocal optical system ( The light is detected by a photodetector 112 via an objective lens 106, a pupil projection optical system 105, and a confocal lens 109). Further, the confocal microscope 100 outputs a digital signal indicating the light intensity at each spot Sp (see FIG. 3) on the biological sample by sampling the detected fluorescence signal with the A/D converter 113. Thereafter, the processing device 20 generates a confocal image based on the digital signal from the confocal microscope 100 and the scanning position information of the scanner 104. Furthermore, the processing device 20 converts the confocal image into an image of the sample S in which distortion is suppressed by performing pixel shift, which will be described later.

FIG. 4 is a diagram for explaining an example of distortion occurring in an image area. FIG. 5 is a diagram for explaining the deformation of cells due to the distortion that occurs in FIG. 4. FIG. 6 is a diagram for explaining another example of distortion occurring in an image area. FIG. 7 is a diagram for explaining the deformation of cells due to the distortion that occurs in FIG. 6. FIG. 8 is a diagram for explaining yet another example of distortion occurring in an image area. FIG. 9 is a diagram for explaining the deformation of cells due to the distortion that occurs in FIG. 8. FIG. 10 is a diagram for explaining distortion that occurs in a confocal image. Image distortion that occurs in confocal images will be described below with reference to FIGS. 4 to 10.

The confocal image is generated by sequentially scanning each position of the sample S with a laser beam, as shown in FIG. Therefore, the information of each pixel of the image is information of the sample S at different times. Furthermore, the sample S constantly vibrates due to life support activities such as pulsation and breathing. For this reason, a different displacement according to a time difference due to vibration is added to the information of the sample S at a different time with respect to the information of the sample S at the reference time. As a result, the image is distorted due to the difference in displacement amount between pixels.

The above-described distortions occurring in confocal images can be roughly classified into two patterns depending on the relationship between the vibration direction and the main scanning direction.
(1st pattern) When the main scanning direction is perpendicular to the vibration direction (2nd pattern) When the main scanning direction is parallel to the vibration direction

In the first pattern in which the main scanning direction is perpendicular to the vibration direction, distortion occurs in the image due to expansion and contraction of elements within the scanning range SR. Specifically, for example, if scanning is performed from the upper left to the lower right of the page, the scanning is performed during a period when the sample S is displaced downward due to vibration, as shown in FIG. The regions (region R1, region R3) are visualized in the image IM11 as regions (region R11, region R13) that are elongated vertically than originally. Furthermore, the region (region R2) that is scanned during the period in which the sample S is displaced upward due to vibration is visualized in the image IM11 as a region (region R12) that has shrunk in the vertical direction than originally. As a result, as shown in FIG. 5, when multiple cells (cell C1, cell C2, cell C3) are included in the scanning range SR, cell C1, cell C2, and cell C3 are each It expands and contracts along the vibration direction and transforms into cells C11, C12, and C13.

In the second pattern in which the main scanning direction is parallel to the vibration direction, the elements within the scanning range SR move in the vibration direction, causing distortion in the image. Specifically, for example, if scanning is performed from the upper left to the lower right of the page, the line scanned during the period when the sample S is displaced to the right due to vibration will be shifted to the right. The image is visualized at the position shown. Further, a line scanned during a period in which the sample S is displaced to the left due to vibration is visualized at a position shifted to the left. Therefore, as shown in FIG. 6, the elongated areas (area R4, area R5, area R6) parallel to the sub-scanning direction are replaced by areas (area R14) that are curved in order left and right along the sub-scanning direction in the image IM12. , region R15, and region R16). As a result, as shown in FIG. 7, when the scanning range SR includes a plurality of cells (cell C1, cell C2, cell C3), the cell C1, cell C2, and cell C3 are each included in the image IM12. , they move left and right, and are visualized as cells C21, C22, and C23 in positions different from their original positions.

The vibration direction usually includes both a main scanning direction component and a sub scanning direction component. Therefore, the general distortion that occurs in a confocal image is a combination of distortions that occur in these two patterns. In other words, as shown in FIG. 8, each area (area R1, area R2, area R3) within the scanning range SR is transformed into a complicated shape (area R21, area R22, area R23) by expansion/contraction and shifting within the image IM13. It is transformed and visualized. As a result, as shown in FIG. 9, when the scanning range SR includes multiple cells (cell C1, cell C2, cell C3), the image of the cells (cell C31, cell C32, cell C33) However, complex deformations that combine expansion and contraction and shifts occur.

Further, the actual shape of the sample S is not a simplified shape as shown in FIGS. 5, 7, and 9. In particular, when the sample S is a biological sample, it most likely does not have a fixed shape like samples for industrial use. This makes estimating the original shape even more difficult. Therefore, when a confocal image IM100 of a mouse brain as shown in FIG. 10 is obtained, it is not easy to judge whether this confocal image correctly visualizes the mouse brain. Even if it were possible to recognize that a confocal image is distorted, it is not easy to correct the distortion correctly.

Note that regions R101 to R103 shown in FIG. 10 indicate the actual shapes of each image region (image region IM101 to image region IM103) of the confocal image IM100. Image areas IM101 and IM102 show the state of a specific region in the brain, and image area IM103 shows how nerve fibers in the brain become thicker or thinner. However, in reality, as shown in regions R101 and R102, each of the specific parts has a size different from the size on the image. Further, as shown in region R103, the nerve fibers have almost uniform thickness.

FIG. 11 is a flowchart of processing performed by the laser scanning microscope system 1. FIG. 12 is a diagram for explaining the relationship between vibration waveform and displacement amount. Hereinafter, a method for suppressing the above-described distortion occurring in a confocal image will be described with reference to FIGS. 11 and 12.

The laser scanning microscope system 1 first performs a first scan (step S1). Here, the microscope device 10 scans the sample S with the scanner 104 according to instructions from the processing device 20, and the processing device 20 generates an image of the sample S based on the signal from the microscope device 10. Note that the image generated in step S1 will hereinafter be referred to as a preliminary image.

Next, the laser scanning microscope system 1 estimates the vibration direction (step S2). Here, the processing device 20 estimates the vibration direction of the sample S based on the preliminary image generated in step S1. Image distortion caused by vibration is a combination of distortions caused by two patterns, as described above, and has regularity. By using this regularity, the processing device 20 estimates the vibration direction from the preliminary image.

The specific method of estimating the vibration direction from the preliminary image is not particularly limited. For example, the vibration direction may be estimated using a learned model generated by machine learning. Alternatively, the vibration direction may be estimated by comparing a plurality of preliminary images. Alternatively, the vibration direction may be estimated using any image processing.

After estimating the vibration direction, the laser scanning microscope system 1 adjusts the main scanning direction (step S3). Here, the processing device 20 controls the controller 200 so that the main scanning direction approaches the vibration direction estimated in step S1. That is, the processing device 20 controls the controller 200 to cause the controller 200 to bring the main scanning direction closer to the vibration direction estimated in step S1.

The method of adjusting the main scanning direction is not particularly limited; The driving speed may be changed.

After that, the laser scanning microscope system 1 performs a second scan (step S4). Here, the microscope device 10 scans the sample S with the scanner 104 in accordance with instructions from the processing device 20 with the main scanning direction brought close to the vibration direction, and the processing device 20 scans the sample S based on the signal from the microscope device 10. Generate an image of S. Note that the image generated in step S4 will hereinafter be referred to as a target image.

By bringing the vibration direction closer to the main scanning direction in step S3, the component included in the vibration orthogonal to the main scanning direction becomes smaller. Thereby, distortion (expansion/contraction) of the above-described first pattern included in the target image generated in step S4 can be suppressed. Note that suppressing distortion of the first pattern is more effective than suppressing distortion of the second pattern. This is because, in order to correct the distortion of the first pattern, it is necessary to cancel the expansion and contraction that occurs in the image, but such correction causes a loss of information and deteriorates the resolution. That is, by suppressing the distortion of the first pattern, it is possible to reduce the loss of information that occurs during image correction, and it is possible to perform image correction with less deterioration in resolution.

Note that, in step S3, it is more desirable that the processing device 20 causes the controller 200 to cause the main scanning direction to match the vibration direction. By matching the main scanning direction with the vibration direction, distortion of the first pattern included in the target image can be suppressed to an extremely low level.

Next, the laser scanning microscope system 1 estimates vibration waveform information (step S5). Here, the processing device 20 estimates waveform information of the vibration of the sample S based on the target image generated in step S4. The waveform information may be output as a parameter set of a predetermined function representing the vibration waveform. For example, assuming that the vibration waveform is a sine wave, it may be expressed by a function defined by a parameter set consisting of amplitude A, period T, phase B, and angle θ (vibration direction). In this case, the processing device 20 may output the amplitude A, period T, phase B, and angle θ as vibration waveform information. Alternatively, the vibration waveform whose vibration direction (angle θ) is known may be assumed to be a sine wave, and the vibration waveform may be output as a parameter set of a predetermined function representing the vibration waveform. In this case, the processing device 20 may output the amplitude A, period T, and phase B as vibration waveform information.

Note that the specific method of estimating vibration waveform information from the target image is not particularly limited. For example, vibration waveform information may be estimated using a learned model generated by machine learning. Alternatively, vibration waveform information may be estimated by comparing a plurality of target images.

Finally, the laser scanning microscope system 1 performs pixel shift on the target image (step S6). Here, the processing device 20 calculates the amount of displacement of each of the plurality of regions of the sample S from the waveform information estimated in step S5, and performs pixel shift on the target image according to the calculated amount of displacement. In the second scan in step S4, an image (target image) in which distortion of the first pattern is suppressed is generated in advance. Therefore, by correcting the distortion of the second pattern by pixel shifting according to the amount of displacement, it is possible to obtain a corrected image in which distortion caused by vibration is suppressed.

Note that the amount of displacement in each region refers to the amount of displacement that occurs in each region from the start of the second scan to the time point when each of the plurality of regions of the sample S is scanned. As shown in FIG. 12, by determining the vibration waveform W(p), it is possible to calculate the displacement amount D of the area corresponding to each pixel p.

Further, each region of the sample S is a region on the sample S corresponding to one line. The amount of displacement of a region may be the amount of displacement of a pixel representing the region (for example, a pixel scanned first), or may be the average amount of displacement of pixels included in the region.

Pixel shifting is performed in area units. For this reason, it is desirable that the difference in displacement amount (distortion magnitude) between pixels within a region is sufficiently small.

Step S6 will be explained in more detail. In order to correct the distortion of the second pattern, it is sufficient to reduce the difference in the amount of displacement occurring between the regions. Therefore, in step S6, the processing device 20 may determine the shift amount of each region (each line) so as to compensate for the difference in the calculated displacement amount of each region.

Specifically, the processing device 20 may, for example, determine the shift amount so that the displacement amounts of the plurality of regions are equalized to the maximum amplitude of vibration (amplitude Amax shown in FIG. 12). Further, the processing device 20 may determine the shift amount so that the displacement amount is equal to zero. In other words, the shift amount may be an amount that cancels the displacement that occurs in each region, that is, the amount of displacement itself. Moreover, since the difference in the displacement amount only needs to be reduced by compensating for the difference in the displacement amount, the displacement amounts of the plurality of regions after the pixel shift do not necessarily have to match.

After determining the shift amount, the processing device 20 shifts each pixel of the target image in the main scanning direction by the determined shift amount for the line to which the pixel belongs. In other words, pixels are shifted by a predetermined amount for each line. This makes it possible to reduce the difference in displacement amount between lines, thereby correcting the distortion of the second pattern.

As described above, by executing the process shown in FIG. 11, the laser scanning microscope system 1 can obtain an image in which image distortion caused by vibrations caused by life-sustaining activities of a biological sample is corrected. Therefore, according to the laser scanning microscope system 1, image distortion can be suppressed in in vivo imaging. Further, in the laser scanning microscope system 1, the target image is acquired by setting the main scanning direction so that the expansion/contraction deformation included in the preliminary image is converted into a pixel shift. Therefore, correction can be performed simply by performing pixel shift processing without performing scaling processing on the target image. This makes it possible to reduce the calculation load and also avoid deterioration in resolution. Therefore, it is possible to correct the target image (confocal image) and obtain a high resolution image with suppressed distortion in a short time.

Hereinafter, specific examples of the laser scanning microscope system 1 described above will be described in each embodiment.
<First embodiment>
FIG. 13 is a diagram for explaining input and output of the trained model M1 according to this embodiment. FIG. 14 is a flowchart of learning processing according to this embodiment. This embodiment will be described below with reference to FIGS. 13 and 14. Note that the configuration of the laser scanning microscope system according to this embodiment (hereinafter simply referred to as the laser scanning microscope system) is the same as the laser scanning microscope system 1 shown in FIGS. 1 and 2. Further, the flow of processing executed by the laser scanning microscope system according to this embodiment is as shown in FIG.

In the laser scanning microscope system according to this embodiment, the vibration direction estimation process (step S2) and the vibration waveform information estimation process (step S5) shown in FIG. This is performed using a trained model M1 that has learned the relationship between vibration waveform information (amplitude A, angle θ, period T, and phase B) of the sample in question.

Specifically, in step S2, the processing device 20 inputs the preliminary image and the scanning time of the scanner 104 at the time of acquiring the preliminary image to the learned model M1, and calculates the pulsation information (waveform) output from the learned model M1. The vibration direction is estimated by acquiring the vibration direction (angle θ) included in the information).

Further, in step S5, the processing device 20 inputs the target image and the scanning time of the scanner 104 at the time of acquiring the target image to the learned model M1, and inputs the amplitude A included in the pulsation information output from the learned model M1. , period T, and phase B, the vibration waveform is estimated. The reason for inputting the scanning time is that even if the deformation in the image is the same, the heart rate varies depending on the scanning time.

Note that the learned model M1 shown in FIG. 13 may be generated by performing the learning process shown in FIG. 14 in advance. Hereinafter, an example will be described in which the learned model M1 is trained using the same laser scanning microscope system used for observation, but learning may be performed using another laser scanning microscope system.

The laser scanning microscope system first scans a sample of the same type as the sample S (for example, a mouse) with the sample S stopped vibrating (step S11). Here, the state in which the vibrations are stopped is, for example, the state in which the mouse is dead.

Next, the laser scanning microscope system converts the image obtained in step S11 (image without vibration) into an image with vibration using the pulsation information (step S12). The pulsation information is, for example, information representing the vibration waveform of the pulsation of the sample. Like the vibration waveform information described above, the pulsation information is a parameter set of a predetermined function that represents a vibration waveform due to pulsation. Here, the processing device 20 converts the non-vibration image into an image with vibration by adding to the non-vibration image the displacement that occurs in each pixel up to the time of scanning, which is calculated using the pulsation information. In addition, in step S12, a plurality of images with vibration are generated by changing the parameter values of the pulsation information, and images with vibration necessary and sufficient for learning are generated.

Although the method for calculating the amount of displacement to be added to each pixel is not particularly limited, it can be calculated as follows, for example. The pixel position in the main scanning direction is x, the pixel position in the sub-scanning direction is y, the number of pixels in the main scanning direction is W, the scanning time per pixel tp, the value obtained by dividing the retrace time by the scanning time per pixel is R Then, the time t(x,y) at which each pixel is scanned can be approximated by the following equation.
t(x,y)=(y×(W+R)+x)×tp

When the pulsation is approximated by a sine wave, the pixel value li (x, y) of the image without vibration, the component of the pulsation amplitude in the main scanning direction is Ax, and the component of the pulsation amplitude in the sub-scanning direction is Ay. , the pixel value lo(x,y) of the image with vibration can be expressed by the following formula.
lo (x, y) = li (x', y')

However, x', y', Ax, and Ay are as follows.
x'=x+Ax×sin((2π/T)×(t(x,y)+B))
y′=y+Ay×sin((2π/T)×(t(x,y)+B))
Ax=Acosθ
Ay=Asinθ

Although we have shown an example in which the amount of displacement is calculated and added for each pixel, if the scanning time per line is sufficiently short compared to the vibration period, for example, the displacement amount can be calculated and added for each line using the first pixel of the line. The amount of displacement may be calculated and the same amount of displacement may be assigned to each line. This makes it possible to shorten the time it takes to generate a large amount of learning data.

Finally, the laser scanning microscope system learns the combination of the image with vibration, pulsation information, and scanning time (step S13). Here, the processing device 20 trains the trained model M1 to output the pulsation information used in step S12 in response to the input of the combination of the image with vibration generated in step S12 and the scanning time of step S11. let

According to the laser scanning microscope system according to this embodiment, image distortion caused by pulsation can be corrected. Therefore, according to the laser scanning microscope system, image distortion can be suppressed in in vivo imaging.

<Second embodiment>
FIG. 15 is a diagram for explaining input and output of the trained model M2 according to this embodiment. FIG. 16 is a diagram for explaining input and output of the trained model M3 according to this embodiment. FIG. 17 is a flowchart of vibration direction learning processing according to this embodiment. FIG. 18 is a flowchart of waveform information learning processing according to this embodiment. This embodiment will be described below with reference to FIGS. 15 to 18. Note that the configuration of the laser scanning microscope system according to this embodiment (hereinafter simply referred to as the laser scanning microscope system) is the same as the laser scanning microscope system 1 shown in FIGS. 1 and 2. Further, the flow of processing executed by the laser scanning microscope system according to this embodiment is as shown in FIG.

In the laser scanning microscope system according to the present embodiment, the vibration direction estimation process (step S2) shown in FIG. The vibration waveform information estimation process (step S5), which is performed using model M2 and shown in FIG. This is done using the completed model M3. Note that the trained model M3 differs from the trained model M1 in that learning is performed using an image obtained by converting an image (an image without vibration) into an image with vibration using pulsation information regarding the main scanning direction. There is.

Specifically, in step S2, the processing device 20 estimates the vibration direction by inputting the preliminary image into the learned model M2 and acquiring the vibration direction (angle θ) output from the learned model M2. do.

Further, in step S5, the processing device 20 inputs the target image and the scanning time of the scanner 104 at the time of acquiring the target image to the trained model M3, and inputs the pulse information regarding the main scanning direction output from the trained model M3. The vibration waveform is estimated by acquiring the included amplitude A, period T, and phase B.

The learned model M2 shown in FIG. 15 may be generated by performing the learning process shown in FIG. 17 in advance. Note that the processing in step S21 and step S22 shown in FIG. 17 is similar to the processing in step S11 and step S12 shown in FIG.

Finally, the laser scanning microscope system learns the combination of the image with vibration and the vibration direction (step S23). Here, the processing device 20 sets the trained model M2 to output the vibration direction (angle θ) included in the pulsation information used in step S22 in response to the input of the image with vibration generated in step S22. Let them learn.

The trained model M3 shown in FIG. 16 may be generated by performing the learning process shown in FIG. 18 in advance. Note that the process in step S31 shown in FIG. 18 is similar to the process in step S11 shown in FIG.

Thereafter, the image obtained in step S31 (image without vibration) is converted into an image with vibration using pulsation information regarding the main scanning direction (step S32). The pulsation information regarding the main scanning direction is, for example, information representing a vibration waveform regarding the main scanning direction component of the pulsation of the sample. The pulsation information regarding the main scanning direction may not include the angle θ, or the angle θ may be given as a fixed value (for example, θ=0).

Finally, the laser scanning microscope system learns the combination of the image with vibration and the pulsation information regarding the main scanning direction (step S33). Here, the processing device 20 inputs the pulsation information (amplitude A, period T, phase The trained model M3 is trained to output B).

The laser scanning microscope system according to this embodiment can also correct image distortion caused by pulsation. Therefore, according to the laser scanning microscope system, image distortion can be suppressed in in vivo imaging.

<Third embodiment>
FIG. 19 is a flowchart of vibration direction estimation processing according to this embodiment. This embodiment will be described below with reference to FIG. 19. Note that the configuration of the laser scanning microscope system according to this embodiment (hereinafter simply referred to as the laser scanning microscope system) is the same as the laser scanning microscope system 1 shown in FIGS. 1 and 2. Further, the flow of processing executed by the laser scanning microscope system according to this embodiment is as shown in FIG.

In the laser scanning microscope system according to this embodiment, the vibration direction estimation process shown in FIG. 19 is performed in step S2 shown in FIG. Specifically, the laser scanning microscope system first performs scanning multiple times by changing the main scanning direction (step S41). That is, a plurality of first scans are performed by the scanner 104 in a plurality of states in which the main scanning directions are different from each other, and a plurality of preliminary images are generated. For example, the main scanning direction may be changed 18 times by 10 degrees, and scanning may be performed multiple times each time the main scanning direction is changed.

After that, the laser scanning microscope system compares the plurality of preliminary images generated in step S41 (step S42) and estimates the vibration direction (step S43). Here, in step S42, the processing device 20 extracts the same feature point from a plurality of preliminary images acquired in the same main scanning direction, and extracts the maximum variation amount of the coordinates of the feature point in the main scanning direction for each main scanning direction. may be obtained. In step S43, the processing device 20 may identify the main scanning direction in which the maximum amount of variation is the largest, and estimate that main scanning direction as the vibration direction.

The laser scanning microscope system according to this embodiment can also correct image distortion caused by pulsation. Therefore, according to the laser scanning microscope system, image distortion can be suppressed in in vivo imaging.

<Fourth embodiment>
FIG. 20 is a diagram illustrating the configuration of a laser scanning microscope system according to this embodiment. FIG. 21 is a diagram for explaining input and output of the learned model M4 according to this embodiment. FIG. 22 is a flowchart of learning processing according to this embodiment. The laser scanning microscope system shown in FIG. 20 differs from the laser scanning microscope system 1 in that it includes a sensor 300 attached to the sample S. Other points are similar to the laser scanning microscope system 1. The sensor 300 is a sensor that measures the waveform of a biological sample, and is, for example, an electrocardiogram sensor. The sensor waveform (electrocardiogram waveform) measured by the sensor 300 is output to the controller 200. Further, the flow of processing executed by the laser scanning microscope system according to this embodiment is as shown in FIG.

In the laser scanning microscope system according to this embodiment, the vibration waveform information estimation process (step S5) shown in FIG. This is performed using a trained model M4 that has learned the relationship with the sensor waveform obtained by measurement.

Specifically, in step S5, the processing device 20 inputs the sensor waveform obtained by measuring the vibration of the sample S with the sensor 300 during target image acquisition (during the second scan) into the learned model M4. Then, the vibration waveform is estimated by acquiring the amplitude A, period T, and phase B included in the pulsation information regarding the main scanning direction output from the trained model M4.

The learned model M4 shown in FIG. 21 may be generated by performing the learning process shown in FIG. 22 in advance. The laser scanning microscope system first scans a sample of the same type as the sample S (for example, a mouse) while performing an electrocardiogram measurement with the vibration direction and the main scanning direction aligned (step S51). Thereby, a combination of an image in which the vibration direction and the main scanning direction match and a sensor waveform during image acquisition can be obtained.

Next, the laser scanning microscope system calculates pulsation information regarding the main scanning direction from the image obtained in step S51 using the trained model (step S52). Here, for example, the trained model M3 described above may be used. Note that steps S51 and S52 are repeatedly performed until sufficient sensor waveform and pulsation information are obtained for learning.

Finally, the laser scanning microscope system learns the combination of the sensor waveform acquired in step S51 and the pulsation information calculated in step S52 using the image corresponding to the sensor waveform (step S53). Here, the processing device 20 causes the trained model M4 to learn to output the vibration direction (angle θ) included in the pulsation information calculated in step S52 in response to the input of the sensor waveform acquired in step S52. .

The laser scanning microscope system according to this embodiment can also correct image distortion caused by pulsation. Therefore, according to the laser scanning microscope system, image distortion can be suppressed in in vivo imaging.

FIG. 23 is a diagram illustrating the hardware configuration of a computer for realizing the processing device according to the embodiment described above. The hardware configuration shown in FIG. 23 includes, for example, a processor 21, a memory 22, a storage device 23, a reading device 24, a communication interface 26, and an input/output interface 27. Note that the processor 21, memory 22, storage device 23, reading device 24, communication interface 26, and input/output interface 27 are connected to each other via a bus 28, for example.

The processor 21 may be, for example, a single processor, a multiprocessor, or a multicore processor. The processor 21 reads and executes a program stored in the storage device 23 to execute the control process illustrated in FIG. 11 and the like.

The memory 22 is, for example, a semiconductor memory and may include a RAM area and a ROM area. The storage device 23 is, for example, a hard disk, a semiconductor memory such as a flash memory, or an external storage device.

The reading device 24 accesses the removable storage medium 25 according to instructions from the processor 21, for example. The removable storage medium 25 is realized by, for example, a semiconductor device, a medium through which information is input/output by magnetic action, a medium through which information is input/output by optical action, or the like. Note that the semiconductor device is, for example, a USB (Universal Serial Bus) memory. Further, a medium in which information is input/output by magnetic action is, for example, a magnetic disk. Examples of media on which information is input and output through optical action include CDs (Compact Discs)-ROMs, DVDs (Digital Versatile Disks), and Blu-ray Discs (Blu-ray is a registered trademark).

The communication interface 26 communicates with other devices according to instructions from the processor 21, for example. The input/output interface 27 is, for example, an interface between an input device and an output device. The input device is, for example, a device such as a keyboard, mouse, or touch panel that receives instructions from a user. The output device is, for example, a display device such as a display, and an audio device such as a speaker.

The program executed by the processor 21 is provided to the computer in the following format, for example.
(1) Installed in the storage device 23 in advance.
(2) Provided by removable storage medium 25.
(3) Provided by a server such as a program server.

Note that the hardware configuration of the computer for realizing the processing device described with reference to FIG. 23 is an example, and the embodiment is not limited to this. For example, some of the configurations described above may be deleted, or new configurations may be added. In another embodiment, for example, some or all of the functions of the processing device described above may be implemented using an FPGA (Field Programmable Gate Array), an SoC (System-on-a-Chip), or an ASIC (Application Specific Integrated Circuit). it), and It may be implemented as hardware such as a PLD (Programmable Logic Device).

The embodiments described above are specific examples to facilitate understanding of the invention, and the present invention is not limited to these embodiments. Variations on the embodiments described above and alternatives to the embodiments described above may be included. In other words, the components of each embodiment can be modified without departing from the spirit and scope thereof. Further, new embodiments can be implemented by appropriately combining a plurality of components disclosed in one or more embodiments. Further, some components may be deleted from the components shown in each embodiment, or some components may be added to the components shown in the embodiments. Furthermore, the processing procedures shown in each embodiment may be performed in a different order as long as there is no contradiction. That is, the laser scanning microscope system of the present invention can be modified and changed in various ways without departing from the scope of the claims.

FIG. 24 is a diagram illustrating the configuration of a two-photon excitation microscope 400. In the embodiment described above, an example was shown in which the laser scanning microscope system includes the confocal microscope 100, but the two-photon excitation microscope 400 shown in FIG. 24 may be included instead of the confocal microscope 100.

As shown in FIG. 24, the two-photon excitation microscope 400 includes a laser 401, a scanner 402, a pupil projection optical system 403, a mirror 404, a dichroic mirror 405, and an objective lens 406 on the illumination optical path. There is.

The laser 401 is, for example, an ultrashort pulse laser, and oscillates laser light in the near-infrared region. The scanner 402 is, for example, two galvanometer mirrors that scan in the main scanning direction and the sub-scanning direction. Pupil projection optical system 403 projects the image of scanner 402 onto the pupil position of objective lens 406 . The dichroic mirror 405 is a light separating means that separates excitation light (laser light) and detection light (fluorescence) from the sample S, and separates the laser light and fluorescence depending on the wavelength.

The two-photon excitation microscope 400 further includes a pupil projection optical system 407 and a photodetector 408 on the detection optical path (reflection optical path of the dichroic mirror 405). The signal output from photodetector 408 is output to A/D converter 409.

The pupil projection optical system 407 is an optical system that projects the pupil image of the objective lens 406 onto the photodetector 408. The photodetector 408 is, for example, a photomultiplier tube (PMT), and outputs an analog signal according to the amount of incident fluorescent light. The A/D converter 409 converts the analog signal from the photodetector 408 into a digital signal (luminance signal) and outputs it to the processing device 20.

Even in a multiphoton excitation microscope such as the two-photon excitation microscope shown in FIG. 24, distortion occurs in images for the same reason as in a confocal microscope. Therefore, the image distortion may be corrected by performing the process shown in FIG.

In the embodiment described above, an example was shown in which the controller 200 changes the drive timing and drive speed of two galvano mirrors that scan in mutually orthogonal directions, thereby bringing the main scanning direction closer to the vibration direction. It may also be adjusted by other methods.

For example, the laser scanning microscope system may include a confocal microscope 500 shown in FIG. 25 that includes an image rotator 501 between the scanner 104 and the dichroic mirror 103 instead of the confocal microscope 100. In this case, the controller 201 that controls the image rotator 501 may change the orientation of the image rotator 501 so that the main scanning direction approaches the vibration direction. Even when the main scanning direction is adjusted using such a method, image distortion can be corrected by performing the processing shown in FIG. 11. Note that by using the image rotator 501, a resonant scanner with restrictions on changing the rotation direction may be used as the scanner 104.

Further, the laser scanning microscope system may include, instead of the confocal microscope 100, a confocal microscope 600 shown in FIG. 26 that includes a rotation stage 601 on which the sample S is placed. In this case, the controller 202 that controls the rotation stage 601 may change the orientation of the rotation stage 601 so that the main scanning direction approaches the vibration direction. Even when the main scanning direction is adjusted using such a method, image distortion can be corrected by performing the processing shown in FIG. 11. Note that by using the rotation stage 601, a resonant scanner with restrictions on changing the rotation direction may be used as the scanner 104.

Furthermore, in the above-described embodiments, the fluorescence observation method for detecting fluorescence has been described as an example, but the observation method performed with the laser scanning microscope system is not limited to the fluorescence observation method. The laser scanning microscope system described above may be applied to other observation methods.

Further, in the embodiment described above, an example was shown in which a biological sample is observed, but the sample is not limited to a biological sample. Even with industrial samples, if regular vibrations occur for some reason during scanning, the distortion caused by the vibrations can be corrected by applying the above-described laser scanning microscope system.

Furthermore, in the above-described embodiments, vibrations caused by pulsations have been described as an example, but image distortion may be corrected by estimating vibrations caused by breathing instead of or in addition to pulsations. Like pulsations, vibrations caused by breathing can be regarded as periodic vibrations, so vibrations that are a combination of pulsations and breathing may be treated as waveform information. Furthermore, in the embodiment described above, an example was shown in which the waveform information is estimated assuming that the vibration is a sine wave vibration, but the vibration waveform is not limited to a sine wave shape. The vibration waveform may be approximated by a function closer to actual vibration.

Further, in the embodiment described above, an example was shown in which the shift amount is determined for each region, which is the unit of calculation of the displacement amount, but the calculation units of the displacement amount and the shift amount may be different. For example, the amount of displacement may be calculated for each region consisting of a plurality of pixels, and the amount of shift may be determined for each pixel from the amount of displacement calculated for each region.

As used herein, the expression "based on A" does not mean "based only on A," but "based at least on A," and furthermore, "based at least in part on A." It also means "te". That is, "based on A" may be based on B in addition to A, or may be based on a part of A.

1 Laser scanning microscope system 10 Microscope device 20 Processing device 21 Processor 100, 500, 600 Confocal microscope 101, 401 Laser 104, 402 Scanner 200, 201, 202 Controller 300 Sensor 400 Two-photon excitation microscope 501 Image rotator 601 Rotation stage
M1, M2, M3 trained models

Claims (10)

a scanner that scans a biological sample in the main scanning direction and the sub-scanning direction;
a controller that controls the main scanning direction;
comprising a processing device;
The processing device includes:
estimating a vibration direction of the biological sample based on a preliminary image that is an image of the biological sample generated in a first scan of the biological sample by the scanner;
causing the controller to bring the main scanning direction closer to the vibration direction;
A plurality of the biological samples from the start of a second scan of the biological sample by the scanner, which is performed with the main scanning direction close to the vibration direction, which is calculated based on waveform information of the vibration of the biological sample. A plurality of lines of a target image, which is an image of the biological sample generated in the second scan, so as to compensate for the difference in the amount of displacement that has occurred in each region up to the time of each scan of the region. determining the amount of shift of each of the plurality of lines corresponding to the area;
Shifting the plurality of lines of the target image in the main scanning direction by only the shift amount of the plurality of lines;
A laser scanning microscope system configured to perform. The laser scanning microscope system according to claim 1,
The laser scanning microscope system is characterized in that the processing device is further configured to estimate vibration waveform information of the biological sample based on the target image. The laser scanning microscope system according to claim 1 or 2,
The scanner includes two scanner mirrors that scan in mutually orthogonal directions,
The laser scanning microscope system is characterized in that the controller controls the two scanner mirrors so that the main scanning direction approaches the vibration direction. In the laser scanning microscope system according to claim 1 or 2, further:
Includes image rotator
The laser scanning microscope system is characterized in that the controller changes the orientation of the image rotator so that the main scanning direction approaches the vibration direction. In the laser scanning microscope system according to claim 1 or 2, further:
including a rotation stage on which the biological sample is placed;
The laser scanning microscope system is characterized in that the controller changes the orientation of the rotation stage so that the main scanning direction approaches the vibration direction. The laser scanning microscope system according to any one of claims 1 to 5,
A laser scanning microscope system characterized in that estimating the vibration direction includes inputting the preliminary image to a trained model that has learned a relationship between an image and a vibration direction of a sample shown in the image. . The laser scanning microscope system according to any one of claims 1 to 5,
Preferably, estimating the vibration direction includes comparing a plurality of preliminary images generated in a plurality of first scans by the scanner performed in a plurality of states in which the main scanning directions are different from each other. Laser scanning microscope system. The laser scanning microscope system according to claim 2,
Estimating the vibration waveform information involves using a trained model that has learned the relationship between the image, the scanning time, and the vibration waveform of the sample shown in the image to estimate the target image and the time required to acquire the target image. A laser scanning microscope system comprising inputting a scanning time of a scanner. The laser scanning microscope system according to claim 1, further comprising:
comprising a sensor that measures vibrations of the biological sample,
The processing device is further configured to estimate the waveform information based on the vibration waveform of the biological sample measured by the sensor,
Estimating the waveform information means that the sensor uses a trained model that has learned the relationship between the vibration waveform of the sample shown in the image and the sensor waveform obtained when the sensor measures the vibration of the sample. A laser scanning microscope system comprising: inputting a sensor waveform obtained by measuring vibrations of the biological sample during a second scan. The laser scanning microscope system according to claim 2,
Estimating the vibration direction involves using a trained model that has learned the relationship between the image, the scanning time, and the waveform of the sample shown in the image to estimate the preliminary image and the scanning time of the scanner required to acquire the preliminary image. including entering
A laser scanning microscope system characterized in that estimating the vibration waveform information includes inputting the target image and the scanning time of the scanner required to acquire the target image into the learned model. .

Images