JP2022032268A - Optical microscope system and method for generating visual field extended image - Google Patents

Abstract

To extend the actual visual field of an optical microscope.SOLUTION: An optical microscope system 10 comprises: at least one galvanomirror 16, 18 which observation light 42 from one of a plurality of observation positions enters; an objective lens 20 which the observation light 42 reflected by the at least one galvanomirror 16, 18 enters; a focus-variable lens 22 which the observation light 42 having passed through the objective lens 20 enters; an imaging device 24 for imaging the observation light 42 having passed through the focus-variable lens 22 and generating an imaged image; mirror drive mechanisms 26, 28 for biaxially driving the at least one galvanomirror 16, 18 and sequentially switching the plurality of observation positions; a lens drive mechanism 30 for switching the focal distance of the focus-variable lens 22 in accordance with changes of the geometric distance of the optical path of the observation light 42 from each observation position to the objective lens 20 due to switching of the plurality of observation positions; and a control device 32 for generating a visual field extended image on the basis of the plurality of imaged images in which the plurality of observation positions are imaged.SELECTED DRAWING: Figure 1

Description

Patent Law Article 30, Paragraph 2 Application Applicable Robotics and Mechatronics Lecture 2020 in Kanazawa, Online Proceedings, May 27, 2 (Posted) Robotics and Mechatronics Lecture 2020 in Kanazawa, Online Poster Presentation, Ordinance May 28, 2nd year (announcement date)

The present disclosure relates to an optical microscope.

Optical microscopes are widely used for magnifying and observing samples. In recent years, a configuration in which an optical image magnified by an objective lens and an image pickup lens is imaged by using an image pickup element and image data of a sample is acquired is generally used. The size of the actual field of view of an optical microscope is determined by the magnification of the objective lens, the magnification of the image pickup lens, and the size of the image pickup element. There is a need. As another method for expanding the actual field of view, a steering mirror is provided between the objective lens and the image pickup lens, and the tilt of the optical axis of the part of the optical path from the sample to the image pickup element that passes through the objective lens is used as the optical axis of the objective lens. On the other hand, a variable configuration has been proposed.

Japanese Patent Publication No. 2008-52882

In order to expand the actual field of view by the above proposal, it is necessary to increase the size of the objective lens. Further, an adaptive optics system for compensating the distortion of the image plane due to the change in the inclination of the optical path of the portion passing through the objective lens is required, which leads to a significant increase in cost.

The present disclosure has been made in view of these problems, and one of its exemplary purposes is to provide a technique for expanding the actual field of view of an optical microscope.

The optical microscope system of one aspect of the present disclosure includes a stage that supports a sample, a lighting device that illuminates the sample on the stage, and a plurality of optical microscope systems that are set so that the two-dimensional coordinates in the x-direction and the y-direction on the stage are different. At least one galvano mirror in which observation light from any one of the observation positions of the above is incident, an objective lens in which observation light reflected by at least one galvano mirror is incident, and observation light that has passed through the objective lens are incident. A variable focus lens, an image pickup device that captures the observation light that has passed through the variable focus lens to generate an image, and a mirror drive mechanism that drives at least one galvano mirror in two axes to switch between multiple observation positions in order. A lens drive mechanism that switches the focal distance of the variable focus lens according to changes in the geometrical distance of the optical path of the observation light from each observation position to the objective lens by switching between multiple observation positions, and a plurality of images of multiple observation positions. It is provided with a control device for generating a field expansion image based on the captured image of the above.

Another aspect of the present disclosure is a method of generating a field-extended image using an optical microscope. The optical microscope is one of a stage that supports the sample, a lighting device that illuminates the sample on the stage, and a plurality of observation positions that are set so that the two-dimensional coordinates in the x and y directions on the stage are different. At least one galvano mirror in which observation light from one is incident, an objective lens in which observation light reflected by at least one galvano mirror is incident, a focus variable lens in which observation light that has passed through the objective lens is incident, and a variable focus lens. It includes an image pickup device that captures the observation light that has passed through the lens and generates an image captured image. This generation method consists of a step of driving at least one galvanometer mirror on two axes to switch between multiple observation positions in order, and a geometrical geometric path of the optical path of the observation light from each observation position to the objective lens by switching between multiple observation positions. A step of switching the focal length of the variable focus lens according to a change in distance, a step of sequentially capturing a plurality of observation positions to generate a plurality of captured images, and a step of generating a field expansion image based on a plurality of captured images. And prepare.

It should be noted that any combination of the above components and those in which the components and expressions of the present disclosure are mutually replaced between methods, systems and the like are also effective as aspects of the present disclosure.

According to the present disclosure, the actual field of view of the optical microscope can be expanded.

It is a figure which shows typically the structure of the optical microscope system which concerns on embodiment. It is a perspective view which shows typically the arrangement of the 1st galvano mirror and the 2nd galvano mirror. It is a figure which shows typically the arrangement of a plurality of observation positions. It is a figure which shows typically the switching order of a plurality of imaging ranges. It is a figure which shows typically the whole extended image and the partially expanded image. It is a figure which shows typically the screen example displayed on the display device. It is a figure which shows typically the influence of the inclination of the optical path of the observation light according to the observation position. It is a flowchart which shows the method of generating the field of view expansion image which concerns on embodiment.

First, the outline of the present disclosure will be described. The present disclosure relates to a technique for expanding the actual field of view of an optical microscope. The actual field of view of the optical microscope is the range in which the sample to be observed can be seen. The size of the actual field of view of the sample is determined by the magnification M1 of the objective lens, the magnification M2 of the image pickup lens, and the size S of the image pickup element, and is S / (M1 × M2). Therefore, when the magnification is fixed, the size S of the image sensor must be increased in order to increase the actual field of view. Since the size of the image sensor that is generally distributed is fixed, it is necessary to use a custom-sized image sensor in order to increase the actual field of view. Further, in order to increase the size of the image pickup device, it is necessary to increase the size of the image pickup lens, which leads to a significant increase in cost.

In the present disclosure, a plurality of captured images obtained by capturing a plurality of observation positions are generated, and a plurality of captured images are combined in real time to generate a field-extended image. In the present disclosure, a galvano mirror is arranged between the sample and the objective lens, and a plurality of observation positions can be switched at high speed by driving the galvano mirror. In particular, by driving the galvano mirror in two axes, it is possible to image a plurality of observation positions having different two-dimensional coordinates in the x-direction and the y-direction. Further, by using a high-speed camera as an image pickup device, it is possible to acquire a plurality of captured images necessary for generating a field-expanded image in a short time. Further, a variable focus lens is used to suppress the deviation of the focal position of the objective lens due to the difference in the observation position. By switching the focal length of the variable focus lens according to the observation position, a plurality of captured images are in focus. In one embodiment of the present disclosure, by using a camera at 500 frames per second and combining 25 captured images, the size of the actual field of view in the x-direction and y-direction can be expanded by about 5 times, and per second. The field-extended image can be updated in 17 cycles.

FIG. 1 is a diagram schematically showing a configuration of an optical microscope system 10 according to an embodiment. The optical microscope system 10 has an inverted microscope configuration. The optical microscope system 10 includes a stage 12, a lighting device 14, a first galvano mirror 16, a second galvano mirror 18, an objective lens 20, a variable focus lens 22, an image pickup device 24, and a first mirror drive mechanism. 26, a second mirror drive mechanism 28, a lens drive mechanism 30, and a control device 32 are provided.

In FIG. 1, a coordinate system with respect to the stage 12 is set. The direction orthogonal to the support surface 12a of the stage 12 is the z direction, and the directions parallel to the support surface 12a of the stage 12 are the x direction and the y direction. In the present specification, the x direction may be referred to as a horizontal direction, the y direction may be referred to as a vertical direction, and the z direction may be referred to as a vertical direction.

The stage 12 has a support surface 12a for horizontally supporting the sample 40 and an opening 12b for passing the observation light 42 from the sample 40. The sample 40 to be observed is not particularly limited, but cells such as humans and animals can be observed. The sample 40 is housed in a sample dish 38 made of a transparent material such as glass, and the sample dish 38 is arranged on the stage 12.

Above the stage 12, a holding pipette 46 and an injection pipette 48 for operating the sample 40 are provided. For example, the holding pipette 46 is used to fix the cells, and the injection pipette 48 is used to perform cell manipulation such as gene transfer into the cells. Such cell manipulation is generally performed continuously on a plurality of cells. By providing an expanded field of view, it is possible to visually recognize a plurality of cells at the same time while magnifying the cells to be manipulated to an appropriate magnification, and it is possible to enhance the convenience of cell manipulation.

The lighting device 14 is provided above the stage 12 and irradiates the sample 40 on the stage 12. The lighting device 14 projects illumination light 44 such as white light toward the sample 40. The lighting device 14 may be capable of projecting the illumination light 44 of visible light having a specific wavelength selected for fluorescence observation or the like. The lighting device 14 projects, for example, the illumination light 44 having a uniform illuminance distribution on the stage 12. The illuminating device 14 is configured to provide transmitted illumination. An illumination device that provides reflected illumination may be provided, and for example, illumination light may be projected toward the sample 40 via a beam splitter arranged between the variable focus lens 22 and the image pickup apparatus 24. ..

The first galvano mirror 16 and the second galvano mirror 18 are provided directly below the opening 12b of the stage 12. The first galvano mirror 16 is configured to rotate about an axis in the z direction. The second galvano mirror 18 is configured to rotate about an axis in the x direction. The second galvano mirror 18 is arranged so as to reflect the observation light 42 from the sample 40 toward the first galvano mirror 16. The first galvano mirror 16 is arranged so as to reflect the observation light 42 reflected by the second galvano mirror 18 toward the objective lens 20. The rotation range of the first galvano mirror 16 and the second galvano mirror 18 is not particularly limited, but is, for example, ± 10 degrees.

FIG. 2 is a perspective view schematically showing the arrangement of the first galvano mirror 16 and the second galvano mirror 18. In FIG. 2, the vertical direction is reversed from that in FIG. 1 for the sake of clarity. The second galvano mirror 18 is arranged at a position separated by a second distance d2 in the −z direction from the origin O of the stage 12. The first galvano mirror 16 is arranged at a position separated from the second galvano mirror 18 by a first distance d1 in the −y direction.

The first galvano mirror 16 changes the position of the observation position 50 on the stage 12 in the x direction. The second galvanometer mirror 18 changes the position of the observation position 50 on the stage 12 in the y direction. The observation position 50 corresponds to the emission position of the observation light 42 reflected by the second galvano mirror 18 and the first galvano mirror 16 and incident on the objective lens 20. The two-dimensional coordinates (x, y) of the observation position 50 are set by using the first rotation angle α of the first galvano mirror 16 and the second rotation angle β of the second galvano mirror 18 with respect to the origin O. It can be described as x = (d1 + d2) tanα and y = d2tanβ.

When the observation position 50 changes, the geometric distance of the optical path of the observation light 42 from the observation position 50 to the objective lens 20 changes. Specifically, the geometric distance of the first optical path L1 from the first reflection point 16a of the observation light 42 in the first galvano mirror 16 to the second reflection point 18a of the observation light 42 in the second galvano mirror 18 (that is,). The length) and the geometric distance (that is, the length) of the second optical path L2 from the second reflection point 18a to the observation position 50 change. The length of the first optical path L1 is d1 / cosα, and the length of the second optical path L2 is d2 · √ (1 + tan ² α + tan ² β).

In the present embodiment, the observation position 50 is configured to scan the observation position 50 in two dimensions by using the first galvano mirror 16 driven by one axis and the second galvano mirror 18 driven by one axis in combination. In another embodiment, one galvanometer mirror driven by two axes may be used to scan the observation position 50 in two dimensions. Therefore, the optical microscope system 10 may include at least one galvano mirror, and by driving the at least one galvano mirror in two axes, a two-dimensional scan of the observation position 50 may be realized. At least one galvano mirror may be a MEMS (Micro Electro Mechanical Systems) mirror.

Returning to FIG. 1, the objective lens 20 is arranged at a position where the observation light 42 from the first galvano mirror 16 is incident. The objective lens 20 is arranged at a position separated from the first galvano mirror 16 in the + x direction. It is desirable that the objective lens 20 has a relatively long working distance (WD). Specifications such as the magnifying power and working distance of the objective lens 20 are not particularly limited, but for example, an ultra-long working type objective lens having a working distance of 20 mm to 40 mm at a magnifying power of 10 to 50 times can be used. ..

The variable focus lens 22 is arranged at a position where the observation light 42 that has passed through the objective lens 20 is incident. The variable focus lens 22 is arranged between the objective lens 20 and the image pickup apparatus 24, and is arranged adjacent to or close to the objective lens 20, for example. The focal length variable lens 22 is configured so that the focal length is variable within a predetermined range. The variable focus lens 22 may be a convex lens having only a positive refractive power, a concave lens having only a negative refractive power, or may be configured to be able to switch between positive and negative refractive powers. ..

The variable focus lens 22 is composed of, for example, a liquid lens, and is configured to have a variable focal length by deforming a flexible transparent film that seals the liquid lens. The shape of the transparent film is controlled by changing the pressure applied to the transparent film. For example, the focal length of the variable focal length lens 22 can be electrically controlled by using an electromagnetic actuator or a piezoelectric element. The variable focus lens 22 is configured to make the effective working distance of the combination of the objective lens 20 and the variable focus lens 22 variable within a range of about 2 mm, for example.

The image pickup apparatus 24 captures the observation light 42 that has passed through the variable focus lens 22 and generates an captured image. The image pickup device 24 includes an image pickup lens 24a and an image pickup element 24b. The image pickup lens 24a forms an image of the observation light 42 on the image pickup element 24b. The image pickup device 24b is an image sensor such as a CMOS sensor, and can generate an image pickup image at a high frame rate. The frame rate of the image pickup apparatus 24 is not particularly limited, but is preferably 100 frames or more per second, and more preferably 500 frames or more per second or 1,000 frames or more per second.

The image pickup apparatus 24 takes an image pickup range 52 centered on the observation position 50 shown in FIG. 2 as an image pickup target. The shape of the image pickup range 52 corresponds to the shape of the image pickup element 24b, and is, for example, rectangular. The shapes of the image pickup devices 24b that are generally distributed differ in the vertical size and the horizontal size, and the horizontal size is larger than the vertical size. The ratio of the vertical size to the horizontal size is, for example, 3: 4 or 9:16. In this case, the width wx in the x direction of the imaging range 52 is larger than the width wy in the y direction of the imaging range 52, and the ratio of the width wx in the x direction to the width wy in the y direction is 3: 4 or 9:16. be.

The objective lens 20, the varifocal lens 22, and the image pickup device 24 are arranged along the observation axis A extending in the x direction, and are fixed to, for example, a lens barrel extending in the x direction. An additional reflecting mirror (not shown) may be provided between the variable focus lens 22 and the image pickup device 24, or the observation axis A may be folded back in the middle.

The first mirror drive mechanism 26 drives the first galvano mirror 16 and controls the first rotation angle α of the first galvano mirror 16. The first mirror drive mechanism 26 is configured to scan the observation position 50 on the stage 12 in the x direction. The second mirror drive mechanism 28 drives the second galvano mirror 18 and controls the second rotation angle β of the second galvano mirror 18. The second mirror drive mechanism 28 is configured to scan the observation position 50 on the stage 12 in the y direction.

The first mirror drive mechanism 26 and the second mirror drive mechanism 28 change the rotation angles α and β of the first galvano mirror 16 or the second galvano mirror 18 in synchronization with the image pickup cycle of the image pickup device 24. The first mirror drive mechanism 26 and the second mirror drive mechanism 28 make the first galvano mirror 16 or the second galvano mirror 18 stationary in the frame imaged by the image pickup device 24. The first mirror drive mechanism 26 and the second mirror drive mechanism 28 rotate the first galvano mirror 16 or the second galvano mirror 18 at the timing between frames when the image pickup device 24 does not take an image to switch the observation position 50.

The lens drive mechanism 30 drives the variable focus lens 22 and switches the focal length of the variable focus lens 22. The lens drive mechanism 30 operates in synchronization with the first mirror drive mechanism 26 and the second mirror drive mechanism 28, and changes the focal length of the variable focus lens 22 according to the observation position 50 on the stage 12. When the observation position 50 on the stage 12 is changed, the geometric distance of the optical path of the observation light 42 reflected by the first galvano mirror 16 and the second galvano mirror 18 changes. In the lens drive mechanism 30, the focal position of the objective lens 20 and the observation position 50 are deviated from each other due to the change in the geometrical distance of the optical path of the observation light 42 according to the change in the observation position 50, and the captured image is out of focus. The variable focus lens 22 is operated so as to prevent this from happening.

The lens driving mechanism 30 adjusts the effective working distance of the objective lens 20 so that the observation position 50 is included in the depth of field of the objective lens 20 even if the observation position 50 changes. The lens driving mechanism 30 drives the variable focus lens 22 so that the geometrical distance of the optical path of the observation light 42 from the objective lens 20 to the observation position 50 matches the effective working distance of the objective lens 20. Is preferable. That is, the lens driving mechanism 30 focuses so that the amount of change in the geometrical distance of the optical path of the observation light 42 according to the change in the observation position 50 and the amount of change in the effective working distance of the objective lens 20 match. It is preferable to drive the variable lens 22.

The control device 32 controls the overall operation of the optical microscope system 10. The control device 32 can be realized by an element such as a CPU and a memory of a computer or a mechanical device in terms of hardware, and can be realized by a computer program or the like in terms of software. The control device 32 is composed of, for example, a general-purpose personal computer. The control device 32 is connected to a display device 34 such as a liquid crystal display and an input device 36 such as a keyboard and a mouse.

The control device 32 controls the operation of the first mirror drive mechanism 26 and the second mirror drive mechanism 28 so that the plurality of observation positions 50 are switched in a predetermined order. The control device 32 controls the operation of the lens drive mechanism 30 and controls the effective working distance of the objective lens 20 according to the switching of the observation position 50. The control device 32 controls the operation of the lens driving mechanism 30 so that an image captured in focus is generated at each of the plurality of observation positions 50.

FIG. 3 is a diagram schematically showing the arrangement of the plurality of observation positions 50, and illustrates a plurality of imaging ranges 52 centered on each of the plurality of observation positions 50. The plurality of imaging ranges 52 are set so as to overlap each other, and the plurality of observation positions 50 are determined so that no gap is formed between the plurality of imaging ranges 52. In the example shown in FIG. 3, 25 observation positions 50 are set. The shape of the image pickup range 52 is a rectangular shape corresponding to the shape of the image pickup element 24b.

The plurality of observation positions 50 include a reference observation position 50a, a plurality of first perimeter observation positions 50b, and a plurality of second perimeter observation positions 50c. The reference observation position 50a corresponds to the origin O in FIG. The plurality of first surrounding observation positions 50b are set around the reference observation position 50a. In the illustrated example, eight first peripheral observation positions 50b are set. The plurality of second perimeter observation positions 50c are set around the first perimeter observation position 50b. In the illustrated example, 16 second perimeter observation positions 50c are set. Further, additional peripheral observation positions may be set around the plurality of second peripheral observation positions 50c.

The plurality of imaging ranges 52 include a reference imaging range 52a, a plurality of first peripheral imaging ranges 52b, and a plurality of second peripheral imaging ranges 52c. The reference imaging range 52a is a rectangular range centered on the reference observation position 50a, and is shown by a thick line frame in FIG. The first peripheral imaging range 52b is a rectangular range centered on the first peripheral observation position 50b, and is shown by a thin line frame in FIG. The first peripheral imaging range 52b is set so as to be adjacent to or overlap with the reference imaging range 52a. The second peripheral imaging range 52c is a rectangular range centered on the second peripheral observation position 50c, and is shown by a broken line frame in FIG. The second peripheral imaging range 52c is set so as to be adjacent to or overlap the first peripheral imaging range 52b.

The plurality of first peripheral observation positions 50b are set so that the geometric distances of the optical paths of the observation light 42 are common to each other. That is, at each of the plurality of first peripheral observation positions 50b, the total length of the first optical path L1 and the second optical path L2 shown in FIG. 2 is the same. Similarly, the plurality of second ambient observation positions 50c are set so that the geometric distances of the optical paths of the observation light 42 are common to each other. A plurality of observation positions 50 having the same optical path length exist, for example, on an arc centered on the origin O or on an elliptical arc. Therefore, the plurality of first peripheral observation positions 50b are set on an arc or an elliptical arc centered on the reference observation position 50a. Similarly, the plurality of second peripheral observation positions 50c are set on an arc or an elliptical arc centered on the reference observation position 50a.

The shape of an arc or an elliptical arc having the same optical path length is determined by the magnitude relationship between the first distance d1 and the second distance d2 in FIG. When the first distance d1 and the second distance d2 are equal (d1 = d2), a plurality of observation positions 50 having the same optical path length exist on an arc. On the other hand, when the first distance d1 and the second distance d2 are not equal (d1 ≠ d2), a plurality of observation positions 50 having the same optical path length exist on the elliptical arc. When the first distance d1 is larger than the second distance d2 (d1> d2), the elliptical arc has a major axis in the x direction and a minor axis in the y direction. When the first distance d1 is smaller than the second distance d2 (d1 <d2), an elliptical arc has a minor axis in the x direction and a major axis in the y direction. In the examples of FIGS. 2 and 3, d1> d2, and the lengths of the optical paths are common at a plurality of observation positions 50 on an elliptical arc having a major axis in the x direction and a minor axis in the y direction.

In the example of FIG. 3, the width of the imaging range 52 in the x direction is made larger than the width of the imaging range 52 in the y direction. That is, the width of the ellipse in the major axis direction (that is, the x direction) is increased and the width of the ellipse in the minor axis direction (that is, the y direction) is decreased in correspondence with the shape of the elliptical arc having the same optical path length. There is. By associating the shape of the imaging range 52 with the shape of an elliptical arc having the same optical path length, a plurality of imaging ranges 52 can be efficiently arranged.

The plurality of first perimeter observation positions 50b may not be strictly arranged on an arc or an elliptical arc having the same optical path length, and may be arranged slightly deviated from the arc or the elliptical arc. For example, a plurality of first perimeter observation positions 50b may be set so that the plurality of first perimeter imaging ranges 52b overlap with an arc or an elliptical arc. The same applies to the plurality of second perimeter observation positions 50c. For example, a plurality of second perimeter observation positions 50c may be set so that the plurality of second perimeter imaging ranges 52c overlap with an arc or an elliptical arc.

FIG. 4 is a diagram schematically showing the switching order of the plurality of imaging ranges 52. The plurality of imaging ranges 52 are sequentially switched in a spiral as shown by the arrow B. First, the reference observation position 50a is switched to and the reference imaging range 52a is imaged. Next, the first peripheral observation position 50b is switched to, and the plurality of first peripheral observation positions 50b are sequentially switched along an arc or an elliptical arc to image a plurality of first peripheral imaging ranges 52b. When all of the plurality of first perimeter imaging ranges 52b are imaged, the second perimeter observation position 50c is then switched to, and the plurality of second perimeter observation positions 50c are sequentially switched along an arc or an elliptical arc. A plurality of second peripheral imaging ranges 52c are imaged. As a result, the entire imaging range 52 is imaged, and one cycle of imaging is completed.

The cycle of imaging a plurality of imaging ranges 52 is repeatedly executed. When all of the plurality of second peripheral imaging ranges 52c are imaged, the reference observation position 50a is then switched to image the reference imaging range 52a, and the next imaging cycle is executed. By switching the observation position 50 spirally along an arc or an elliptical arc, it is possible to continuously image a plurality of imaging ranges 52b or 52c having the same optical path length. As a result, the number of times the focal length of the variable focal length lens 22 is switched can be reduced, and faster switching can be realized. The switching order in FIG. 4 is only an example, and a plurality of imaging ranges 52 may be switched in the reverse order of FIG.

The control device 32 acquires the captured image generated by the image pickup device 24 and performs image processing. The control device 32 combines a plurality of captured images captured in each of the imaging ranges 52 of the plurality of observation positions 50 to generate a field-extended image, and displays the image on the display device 34. The control device 32 may generate an overall extended image in which a plurality of captured images corresponding to all the observation positions 50 are combined. The control device 32 may generate a partially expanded image obtained by cutting out a part of the entire expanded image. The cutout range of the partially expanded image may be arbitrarily specified based on, for example, an input operation from the input device 36. The partially expanded image may be generated based on only one captured image, or may be generated based on a plurality of captured images. The control device 32 may simultaneously generate the entire expanded image and the partially expanded image and display them on the display device 34.

FIG. 5 is a diagram schematically showing a fully expanded image 56 and a partially expanded image 58. The overall expanded image 56 is generated by combining all of the plurality of captured images 54 that image each of the plurality of imaging ranges 52 in FIG. The plurality of captured images 54 are arranged at positions corresponding to each observation position 50 and superimposed on each other. The control device 32 generates the overall expanded image 56 by synthesizing a plurality of captured images 54 that overlap each other. For the portion where the two or more captured images 54 overlap, two or more captured images 54 may be combined by using a technique such as alpha blending, or only one of the captured images 54 may be preferentially used. When only one of the captured images 54 is used, an image having a short distance from the reference observation position 50a may be prioritized, or an image having a new imaging timing may be prioritized. The partially expanded image 58 is generated by arbitrarily cutting out a part of the entire expanded image 56. In FIG. 5, the cut-out range of the partially expanded image 58 is shown by a alternate long and short dash line.

FIG. 6 is a diagram schematically showing an example of a screen displayed on the display device 34. In the example of FIG. 6, the entire expanded image 56 and the partially expanded image 58 are displayed on the display device 34 at the same time. A frame 60 showing a cutout range of the partially expanded image 58 is superimposed and displayed on the entire expanded image 56. For example, by changing the position and size of the frame 60 using an input device 36 such as a mouse, the position and size of the cutout range displayed as the partially expanded image 58 can be changed. Since the control device 32 generates the entire expanded image 56, the partially expanded image 58 having different positions and sizes of the cutout range can be generated in real time. That is, it is possible to change the position, range, and magnifying power of the actual field of view by software processing without moving the stage 12 to change the observation position or exchanging the lens to change the magnifying power.

When the control device 32 generates the partially expanded image 58, the control device 32 may change the switching pattern of the plurality of imaging ranges 52. For example, the frequency of switching to the imaging range 52 included in the partially expanded image 58 may be relatively high, and the frequency of switching to the imaging range 52 not included in the partially expanded image 58 may be relatively low. For example, in one cycle for capturing the entire imaging range 52, the number of switching to the imaging range 52 included in the partially expanded image 58 is set to two or more, and the switching to the imaging range 52 not included in the partially expanded image 58 is performed. The number of times may be only once. By increasing the frequency of switching to the imaging range 52 that is not included in the partially expanded image 58, the frequency of updating the partially expanded image 58 can be increased, and the convenience of work while visually recognizing the partially expanded image 58 can be improved. can.

The control device 32 does not use the captured image 54 captured in each imaging range 52 as it is, but performs predetermined image processing on the captured image 54 to generate the entire expanded image 56 or the partially expanded image 58. You may use it. The control device 32 may correct the distortion of the image due to the captured image 54 being imaged obliquely with respect to the support surface 12a of the stage 12. The control device 32 may correct the effect that the focus positions of the central portion and the outer peripheral portion of the captured image 54 are displaced in the z direction due to the captured image 54 being imaged obliquely with respect to the support surface 12a of the stage 12. The control device 32 may, for example, cut out only the central portion of the captured image 54 that is in focus with the sample 40 and use it for generating the entire expanded image 56 or the partially expanded image 58.

FIG. 7 is a diagram schematically showing the influence of the inclination of the optical path of the observation lights 42a, 42b, 42c according to the observation positions 50a, 50b, 50c. FIG. 7 is an enlargement of the vicinity of the sample 40 of FIG. 1, and shows the inclination of the optical path of the observation lights 42a to 42c in the x direction. Further, the width wx in the x direction and the width wz in the z direction of the imaging ranges 52a, 52b, 52c at each observation position 50a to 50c are shown. The width wz in the z direction of the imaging range 52a to 52c is a range in the depth direction in which the focus is achieved, and corresponds to the depth of field of the objective lens 20.

At the reference observation position 50a, the observation light 42a is parallel to the z direction and is not tilted with respect to the support surface 12a of the stage 12. Therefore, the captured image 54 that captures the reference imaging range 52a does not need to be corrected. On the other hand, at the first perimeter observation position 50b and the second perimeter observation position 50c, the observation light 42a is tilted at an angle α1 or α2 in the x direction with respect to the z direction. Therefore, it is preferable that the captured image 54 that captures the first peripheral imaging range 52b and the second peripheral imaging range 52c is corrected according to the inclination.

As the first correction process, the distortion of the image due to the captured image 54 being imaged obliquely with respect to the sample 40 may be corrected. For example, at the first peripheral observation position 50b, the coordinates are converted from the plane orthogonal to the optical path of the observation light 42b to the plane orthogonal to the z direction, and the captured image 54 in which the first peripheral imaging range 52b is imaged is supported by the stage 12. By projecting onto the surface 12a, distortion of the image can be corrected. The optical path of the observation light 42b at the first peripheral observation position 50b may be inclined at an angle β1 (not shown) in the y direction with respect to the z direction. In that case, it is preferable to correct both the inclination of the angle α1 in the x direction and the angle β1 in the y direction. Similarly, at the second peripheral observation position 50c, the first correction process may be performed based on the angle α2 in the x direction and the angle β2 (not shown) in the y direction of the inclination of the optical path of the observation light 42c.

As the second correction process, only the central portion of the captured image 54 that is in focus with the sample 40 may be cut out. For example, at the first peripheral observation position 50b, only the first range Wb where the thickness t of the sample 40 and the first peripheral imaging range 52b overlap may be cut out. The width of the first range Wb in the x direction uses the thickness t of the sample 40, the depth of field wz of the objective lens 20, and the angle α1 of the inclination of the optical path of the observation light 42b at the first peripheral observation position 50b in the x direction. Can be calculated geometrically. The optical path of the observation light 42b at the first peripheral observation position 50b may be inclined at an angle β1 in the y direction with respect to the z direction. In that case, the width of the first range Wb in the y direction is the thickness t of the sample 40, the depth of field D of the objective lens 20, and the angle of inclination of the optical path of the observation light 42b at the first peripheral observation position 50b in the y direction. It can be calculated geometrically using β1. Similarly, at the second peripheral observation position 50c, the thickness t of the sample 40 and the second peripheral imaging range 52c overlap each other based on the angle α2 in the x direction and the angle β2 in the y direction of the inclination of the optical path of the observation light 42c. Only the range Wc may be cut out.

When the above-mentioned second correction process is performed, it is preferable that the arrangement of the first peripheral observation position 50b and the second peripheral observation position 50c is set according to the cutout ranges Wb and Wc of the captured image 54. Specifically, it is preferable that the arrangement of the first peripheral observation position 50b and the second peripheral observation position 50c is set so that the cutout ranges Wb and Wc of the captured image 54 are adjacent to or overlap each other. As a result, it is possible to prevent the occurrence of a blank region that cannot be captured when a field-of-view expansion image is generated based on an image obtained by cutting out a part of a plurality of captured images 54.

Next, the flow of operation of the optical microscope system 10 will be described. FIG. 8 is a flowchart showing a method of generating a field-of-view expansion image according to the embodiment. First, at least one galvano mirror is driven to switch the observation position 50 (S10). If the length of the optical path of the observation light 42 changes before and after switching the observation position 50 (Y in S12), the focal length of the variable focus lens 22 is switched (S14). If the length of the optical path has not changed in S12 (N in S12), the processing in S14 is skipped. Next, the observation position 50 after switching is imaged by the image pickup apparatus 24 to generate an image captured image 54 (S16). If correction of the captured image 54 is necessary (Y in S18), the captured image 54 is corrected according to the observation position 50 (S20). If the correction of the captured image 54 is unnecessary (N in S18), the process of S20 is skipped. Subsequently, a visual field expansion image is generated using the captured image 54 before or after the correction (S24), and the generated visual field expansion image is displayed on the display device 34 (S24).

The processes of S10 to S24 in FIG. 8 are repeatedly executed. For example, a plurality of observation positions 50 are sequentially switched based on a predetermined order shown in FIG. 4, a plurality of captured images 54 in which a plurality of observation positions 50 are imaged are generated, and a field of view is generated based on the plurality of captured images 54. An extended image is generated. After the generation of the field-of-view expansion image corresponding to the entire of the plurality of observation positions 50, the display of a part of the field-of-view expansion image is updated based on the newly captured captured image 54.

According to an example of the present embodiment, the image pickup apparatus 24 generates an image pickup image 54 every 2 milliseconds with an image pickup cycle of 500 frames per second. The plurality of observation positions 50 are switched every 2 milliseconds in synchronization with the imaging cycle, and the first galvano mirror 16 and the second galvano mirror 18 are driven every 2 milliseconds. On the other hand, the variable focus lens 22 does not have to be driven every 2 milliseconds. For example, the focal length of the variable focal length lens 22 may be fixed for 16 milliseconds required to image all of the eight first peripheral observation positions 50b. Similarly, the focal length of the variable focus lens 22 may be fixed for 32 milliseconds required to image all of the 16 second peripheral observation positions 50c. The focal length of the variable focal length lens 22 is switched when the reference observation position 50a is switched to the first peripheral observation position 50b, when the first peripheral observation position 50b is switched to the second peripheral observation position 50c, and when the second peripheral observation position is switched. It may be executed only when switching from the reference observation position 50a from 50c.

The response speed of the variable focus lens 22 is preferably as fast as the image pickup cycle of the image pickup device 24, but in the case of the generally available variable focus lens 22, the image pickup cycle of the image pickup device 24 (for example, 2 mm) is preferable. Seconds) may not be realized. An example of the response speed of the variable focus lens 22 is about 4 milliseconds. In this case, the variable focus lens 22 cannot be driven in synchronization with the switching of the observation position 50 every 2 milliseconds, and it is necessary to wait for the switching of the observation position 50 until the switching of the focal length of the variable focus lens 22 is completed. There is. Therefore, in the process of S14 in FIG. 8, the observation position 50 may be fixed without being switched in 4 milliseconds until the switching of the focal length of the variable focal length lens 22 is completed. The captured image 54 captured by the image pickup apparatus 24 during the switching of the focal length of the variable focus lens 22 may not be used for generating the field-extended image.

According to the present embodiment, the number of times the focal length of the variable focal length lens 22 is switched (for example, 3 times) can be significantly reduced as compared with the number of times the observation position 50 is switched (for example, 25 times) in one imaging cycle. As a result, the waiting time required for switching the focal length of the variable focal length lens 22 can be minimized, the observation position 50 can be switched at a higher speed, and the interval at which the entire field-extended image is updated can be shortened. .. By updating the field-enhanced image at high frequency, it is possible to provide an appropriate field-of-view-expanded image for applications that require real-time display such as cell manipulation.

The present disclosure has been described above based on the embodiments. It is understood by those skilled in the art that the present disclosure is not limited to the above embodiment, various design changes are possible, various modifications are possible, and such modifications are also within the scope of the present disclosure. It is about to be.

10 ... Optical microscope system, 12 ... Stage, 14 ... Illumination device, 16 ... First galvano mirror, 18 ... Second galvano mirror, 20 ... Objective lens, 22 ... Variable focus lens, 24 ... Imaging device, 26 ... First mirror Drive mechanism, 28 ... Second mirror drive mechanism, 30 ... Lens drive mechanism, 32 ... Control device, 34 ... Display device, 36 ... Input device, 40 ... Sample, 42 ... Observation light, 44 ... Illumination light, 50 ... Observation position , 50a ... Reference observation position, 50b ... First peripheral observation position, 50c ... Second peripheral observation position, 52 ... Imaging range, 52a ... Reference imaging range, 52b ... First peripheral imaging range, 52c ... Second peripheral imaging range, 54 ... captured image, 56 ... whole expanded image, 58 ... partially expanded image.

Claims (10)

The stage that supports the sample and
A lighting device that illuminates the sample on the stage,
At least one galvanometer mirror to which observation light from any one of a plurality of observation positions set so that the two-dimensional coordinates in the x-direction and the y-direction on the stage are incident is incident.
An objective lens to which the observation light reflected by the at least one galvano mirror is incident, and
A varifocal lens to which the observation light that has passed through the objective lens is incident,
An image pickup device that captures the observation light that has passed through the focus variable lens and generates an image.
A mirror drive mechanism that drives the at least one galvano mirror in two axes and switches the plurality of observation positions in order.
A lens drive mechanism that changes the focal length of the variable focus lens according to a change in the geometrical distance of the optical path of the observation light from each observation position to the objective lens by switching the plurality of observation positions.
An optical microscope system comprising: a control device for generating a field-extended image based on a plurality of captured images obtained by capturing the plurality of observation positions. The optical microscope system according to claim 1, wherein the mirror drive mechanism sequentially switches the plurality of observation positions in synchronization with the image pickup cycle of the image pickup apparatus. The plurality of observation positions include a reference observation position and a plurality of ambient observation positions set around the reference observation position.
The optical microscope system according to claim 1 or 2, wherein the mirror drive mechanism switches to the reference observation position before or after switching the plurality of ambient observation positions in order. The optical microscope according to claim 3, wherein the plurality of ambient observation positions are set so that the geometrical distances of the optical paths of the observation light from each observation position to the objective lens are set to be common to each other. system. The optical microscope system according to claim 3 or 4, wherein the plurality of ambient observation positions are set on an arc or an elliptical arc centered on the reference observation position. The image pickup apparatus is configured to take a rectangular range centered on each observation position and having a width in the x direction larger than the width in the y direction as an image pickup target.
The optical microscope system according to claim 5, wherein the plurality of ambient observation positions are set on an elliptical arc whose x-direction is a major axis and the y-direction is a horizontal axis. The at least one galvano mirror includes a first galvano mirror that rotates about the axis in the z direction of the stage and a second galvano mirror that rotates about the axis in the x direction, and the first galvano mirror is the first galvano mirror. , The observation light reflected by the second galvano mirror is arranged so as to be reflected toward the objective lens.
The optics according to claim 6, wherein the first distance from the first galvano mirror to the second galvano mirror is longer than the second distance from the second galvano mirror to the sample on the stage. Microscope system. The control device generates the field-extended image by combining a plurality of images obtained by cutting out at least a part of each of the plurality of captured images.
The control device according to any one of claims 1 to 7, wherein the control device changes the cutting range of the captured image corresponding to each observation position according to the inclination of the optical path of the observation light at each observation position. The optical microscope system described. The control device generates a fully expanded image based on all of the plurality of captured images, generates a partially expanded image based on at least one of the plurality of captured images, and the totally expanded image and the partially expanded image. Can be displayed on the display device at the same time.
One of claims 1 to 8, wherein the mirror drive mechanism makes the frequency of switching to the observation position included in the partially expanded image higher than that of switching to the observation position not included in the partially expanded image. The optical microscope system described in. It is a method of generating a field-expanded image using an optical microscope, and the optical microscope is a method.
The stage that supports the sample and
A lighting device that illuminates the sample on the stage,
At least one galvanometer mirror to which observation light from any one of a plurality of observation positions set so that the two-dimensional coordinates in the x-direction and the y-direction on the stage are incident is incident.
An objective lens to which the observation light reflected by the at least one galvano mirror is incident, and
A varifocal lens to which the observation light that has passed through the objective lens is incident,
An image pickup device that captures the observation light that has passed through the focus variable lens and generates an image capture image is provided.
The generation method is
A step of driving the at least one galvano mirror in two axes to switch the plurality of observation positions in order, and
A step of switching the focal length of the variable focus lens according to a change in the geometrical distance of the optical path of the observation light from each observation position to the objective lens by switching the plurality of observation positions.
The step of capturing the plurality of observation positions in order to generate a plurality of captured images, and
A method for generating a field-expanded image, which comprises a step of generating a field-expanded image based on the plurality of captured images.

Images