WO2023127001A1 - Microscope system, projection unit, and selection assistance method - Google Patents

Abstract

A microscope system 1 comprises: a microscope 100 that forms an optical image of a sperm sample which contains an object of selection; an imaging device 143 for acquiring a digital image of the sperm sample; a processing device 200 that generates an object image which is an image of the object of selection, such generation being on the basis of sperm detection and sperm form evaluation with respect to the digital image; and a projection device 153 that displays the object image on an image surface where the optical image is formed, the object image being displayed larger than the object of selection in the optical image.

Description

Microscope system, projection unit, and sorting support method

The disclosure of this specification relates to a microscope system, a projection unit, and a selection support method.

With the trend toward later marriage and childbearing, the number of patients undergoing fertility treatment is increasing year by year, and the demand for assisted reproductive technology (ART) is also increasing.

ART is a general term for technologies such as in vitro fertilization (IVF) and microinsemination, in which eggs and sperm removed from humans are fertilized outside the body, and the collected sperm is injected into the uterus and fertilized with the egg inside the body. artificial insemination.

ART-related technology is described in Patent Document 1, for example. Patent Document 1 describes a microscope suitable for intracytoplasmic sperm injection (ICSI) used in microinsemination, which is a type of ART. Note that ICSI is a method of directly injecting sperm into an egg by piercing an injection pipette containing sperm into the egg fixed with a holding pipette.

WO2012/150689

By the way, in order to increase the success rate of ICSI, it is important to select sperm and inject sperm suitable for fertilization into the egg. However, whether the quality of the sperm obtained by the sorting work depends largely on the experience of the embryo cultivator who performs the work, and a difference in fertilization rate tends to occur among the embryo culturists.

Although the sperm sorting work has been described above as an example, the above problems can occur not only in the sperm sorting work but also in any target sorting work.

In light of the circumstances described above, an object of one aspect of the present invention is to provide a technology that supports the selection of objects in a sample.

A microscope system according to an aspect of the present invention includes a microscope that forms an optical image of a sperm sample containing a sorting target, an imaging device that acquires a digital image of the sperm sample, and sperm detection and sperm morphology in the digital image. A processing device for generating a target image, which is the image to be selected based on the evaluation, and displaying the target image on an image plane on which the optical image is formed in a size larger than the target for selection in the optical image. and an optical device.

A projection unit according to an aspect of the present invention is a projection unit attached to a microscope, and includes an imaging unit that acquires a digital image of a sperm sample, and sperm detection based on the digital image and evaluation of the sperm morphology. a processing unit that generates a target image that is an image of a sorting target contained in a sample; a projection unit for displaying in a larger size.

A selection support method according to an aspect of the present invention forms an optical image of a sperm sample containing a selection target, acquires a digital image of the sperm sample, and performs sperm detection and sperm morphology evaluation on the digital image. A target image that is an image to be sorted is generated, and the target image is displayed on an image plane on which the optical image is formed in a size larger than that of the sorting target in the optical image.

According to the above aspect, it is possible to support the quality judgment of the sample in the sorting work.

1 is a diagram illustrating the configuration of a microscope system 1; FIG. 1 is a diagram illustrating the configuration of a microscope 100; FIG. 4 is a diagram illustrating the configuration of an operation unit of the input device 50; FIG. 2 is a diagram illustrating a hardware configuration of a processing device 200; FIG. It is a flowchart which shows an example of the procedure of ICSI by an embryologist. 3 is a diagram illustrating the configuration of a drop formed as a sample 300 in a petri dish 310; FIG. It is a flowchart which shows an example of the sperm selection procedure by an embryonic culture person. 9 is a flowchart showing an example of sorting support processing; 6 is a flowchart showing an example of target image generation processing; 4 is a flowchart showing an example of image display processing; 4 is a diagram showing an example of an optical image generated by the microscope 100; FIG. FIG. 2 is a diagram showing an example of a digital image acquired by an imaging device 143; FIG. FIG. 10 is a diagram showing an example of object detection results for a digital image; FIG. 4 is a diagram for explaining an example of a method of generating a target image; 4 is a diagram showing an example of an image seen through an eyepiece lens 101. FIG. FIG. 10 is a diagram for explaining another example of a method of generating a target image; FIG. 10 is a diagram showing another example of an image seen through the eyepiece 101; 9 is a flowchart showing another example of sorting support processing; 9 is a flowchart showing another example of image display processing; FIG. 4 is a diagram for explaining an example of a method of generating an auxiliary image; FIG. FIG. 10 is a diagram showing still another example of an image seen through the eyepiece 101; 2 is a diagram illustrating the configuration of a microscope system 2; FIG.

[First Embodiment]
FIG. 1 is a diagram illustrating the configuration of a microscope system 1. As shown in FIG. FIG. 2 is a diagram illustrating the configuration of the microscope 100. As shown in FIG. FIG. 3 is a diagram illustrating the configuration of the operation unit of the input device 50. As shown in FIG. FIG. 4 is a diagram illustrating the configuration of the processing device 200. As shown in FIG.

The microscope system 1 is a system for observing a sample by looking through an eyepiece 101. Specifically, the microscope system 1 is an inverted microscope system equipped with a transmitted illumination system 120 used for microscopic insemination, particularly sperm sorting. The microscope system 1 is used, for example, by an embryologist. A sample to be observed is a sperm suspension containing sperm contained in a petri dish or the like in the case of sperm sorting.

The microscope system 1 includes at least a microscope 100, an imaging device 143, a projection device 153, and a processing device 200. Microscope 100 forms an optical image of a sample containing sperm to be sorted (that is, a sperm sample). Imaging device 143 acquires a digital image of the specimen. The processing device 200 generates an image to be sorted (hereinafter referred to as a target image) based on object detection on the digital image. Here, the selection target is a target whose quality is judged by the user, and means a target whose selection or non-selection is determined as a result of the quality judgment. The projection device 153 is an example of an optical device that displays a target image in a size larger than the selection target in the optical image on the image plane where the optical image is formed. In this specification, "displaying an image (image)" refers to forming an image (image) in a visible manner. position).

The microscope system 1 uses the projection device 153 to display the target image in a size larger than the selection target in the optical image on the image plane where the optical image of the sample is formed by the microscope 100 . In other words, when the sorting target (sperm) in the optical image is displayed within, for example, a 1 mm×1 mm area on the image plane, the target image is displayed on the image plane in a size larger than 1 mm×1 mm. As a result, the user who is looking through the eyepiece 101 and observing the sample can observe the sperm to be sorted included in the optical image in detail with a larger size target image without taking his eyes off the eyepiece 101. can do. Therefore, it is possible to accurately specify and collect good sperm in a short period of time. Therefore, according to the microscope system 1, it is possible to support the sperm sorting work of users such as embryologists.

A specific example of the configuration of the microscope system 1 will be described in detail below with reference to FIGS. 1 to 4. FIG. As shown in FIG. 1, the microscope system 1 includes a microscope controller 10, a display device 30, and a plurality of input devices in addition to the microscope 100, the imaging device 143, the projection device 153, and the processing device 200 described above. (input device 40 , input device 50 , input device 60 , input device 70 ) and identification device 80 . The microscope system 1 is also connected to a database server 20 storing various data. In this example, the imaging device 143 and the projection device 153 are arranged inside the microscope body 110 of the microscope 100 .

The microscope 100 is an inverted microscope equipped with an eyepiece 101. The microscope 100 includes a microscope main body 110, a plurality of objective lenses 102 attached to the microscope main body 110, a stage 111, a transmitted illumination system 120, and an eyepiece tube 170, as shown in FIG. Further, the microscope 100 includes modulation elements for visualizing unstained samples such as sperm and ovum in each of the illumination optical path and the observation optical path, as will be described later. A user, such as an embryologist, can use the microscope 100 to examine a sample using four microscopy techniques: bright field (BF) observation, polarized light (PO) observation, differential interference contrast (DIC) observation, and modulation contrast (MC) observation. can be observed. Modulation contrast observation is also called relief contrast (RC) observation.

A plurality of objective lenses 102 are attached to a revolver 112 . As shown in FIG. 2, the multiple objective lenses 102 include an objective lens 102a for BF observation, an objective lens 102b for PO observation and DIC observation, and an objective lens 102c for MC observation. In addition, the modulator 104 is included in the objective lens 102c. The modulator 104 includes three regions with different transmittances (for example, a region with a transmittance of about 100%, a region with a transmittance of about 5%, and a region with a transmittance of about 0%).

Although FIG. 2 illustrates three objective lenses according to the microscopy method, the plurality of objective lenses 102 may include objective lenses with different magnifications for each microscopy method. In the following, 4x objective lens for BF observation, 10x, 20x, and 40x objective lenses for MC observation, 20x objective lens for PO observation, and 60x objective lens for DIC observation are included. will be described as an example.

The revolver 112 is a switching device that switches objective lenses arranged on the optical path among the plurality of objective lenses 102 . The revolver 112 switches the objective lens arranged on the optical path according to the microscope method and observation magnification. The objective lens arranged on the optical path by the revolver 112 guides the transmitted light that has passed through the sample to the eyepiece lens 101 .

A sample placed in a container is placed on the stage 111 . The container is, for example, a petri dish, and the sample contains reproductive cells such as sperm and ovum. The stage 111 moves in the optical axis direction of the objective lens 102 arranged on the optical path and in the direction orthogonal to the optical axis of the objective lens 102 . Note that the stage 111 may be a manual stage or an electric stage.

The transmitted illumination system 120 illuminates the sample placed on the stage 111 from above the stage 111 . Transillumination system 120 includes a light source 121 and a universal condenser 122, as shown in FIGS. The light source 121 may be, for example, an LED (Light Emitting Diode) light source or a lamp light source such as a halogen lamp light source.

The universal condenser 122 includes a polarizer 123 (first polarizing plate), a plurality of optical elements housed in a turret 124, and a condenser lens 128, as shown in FIG. The polarizer 123 is used for MC observation, PO observation and DIC observation. The turret 124 accommodates a plurality of optical elements that are switched for use according to the microscope method. A DIC prism 125 is used for DIC observation. Aperture plate 126 is used for BF observation and PO observation. The optical element 127 is a combination of a slit plate 127a, which is a light shielding plate in which a slit is formed, and a polarizing plate 127b (second polarizing plate) arranged so as to partially cover the slit. used.

The eyepiece tube 170 includes the eyepiece lens 101 . The imaging lens 103 is arranged between the eyepiece lens 101 and the objective lens 102 . The imaging lens 103 forms an optical image of the sample on an image plane IP between the eyepiece lens 101 and the imaging lens 103 based on the transmitted light. A target image, which will be described later, is also formed on the image plane IP based on the light from the projection device 153 . As a result, the optical image and the target image are displayed on the image plane IP. A user of the microscope system 1 uses the eyepiece 101 to observe the optical image formed on the image plane IP and the virtual image of the target image.

The microscope body 110 includes a laser assisted hatching unit 130, an imaging unit 140, and a projection unit 150, as shown in FIGS. The microscope body 110 also includes an intermediate magnification unit 160, as shown in FIG. Furthermore, the microscope main body 110 includes a DIC prism 105 and an analyzer 106 that can be inserted into and removed from the optical path.

The laser assisted hatching unit 130 is a laser unit arranged between the objective lens 102 and the imaging lens 103, as shown in FIG. The laser assisted hatching unit 130 irradiates the sample with laser light by introducing the laser light from between the objective lens 102 and the imaging lens 103 . More specifically, the laser-assisted hatching unit 130 irradiates, for example, a laser beam onto the zona pellucida surrounding an embryo grown from a fertilized egg. Laser assisted hatching unit 130 includes splitter 131 , scanner 133 , lens 134 and laser 135 . Splitter 131 is, for example, a dichroic mirror. The scanner 133 is, for example, a galvanometer scanner, and adjusts the irradiation position of the laser light in a direction perpendicular to the optical axis of the objective lens 102 . A lens 134 converts the laser beam into a parallel beam. Thereby, the laser light is focused on the sample by the objective lens 102 .

The imaging unit 140, as shown in FIG. 2, includes a splitter 141 and an imaging device 143 that acquires a digital image of the sample based on transmitted light. The imaging unit 140 is arranged between the imaging lens 103 and the eyepiece 101 . Splitter 141 is, for example, a half mirror. The imaging lens 103 forms an optical image of the sample on the light receiving surface of the imaging device included in the imaging device 143 . The imaging device 143 is, for example, a digital camera that acquires a digital image, and the imaging device included in the imaging device 143 is, for example, a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, or the like. is. The imaging device detects light from a sample and converts the detected light into an electrical signal by photoelectric conversion. The imaging unit 140 outputs digital images acquired by the imaging device 143 to the processing device 200 .

The projection unit 150 is arranged between the imaging lens 103 and the eyepiece lens 101 . The projection unit 150 includes a splitter 151, a lens 152, and a projection device 153, as shown in FIG. Splitter 151 is, for example, a half mirror. The projection device 153 projects the target image generated by the processing device 200 . More specifically, the lens 152 converges the light from the projection device 153 onto the image plane IP of the imaging lens 103, that is, the image plane IP where an optical image is formed, so that the projection device 153 is projected onto the image plane IP. Project the target image.

The intermediate variable power unit 160 is arranged between the objective lens 102 and the imaging lens 103 . As shown in FIG. 2, the intermediate magnification unit 160 includes a plurality of lenses (lens 161, lens 162, and lens 163), and by switching the lenses arranged on the optical path among these, an image is formed on the image plane. Change the magnification of the optical image displayed. By using the intermediate magnification unit 160, the magnification of the optical image can be changed without switching the objective lens 102 located near the sample.

The DIC prism 105 and the analyzer 106 are arranged between the objective lens 102 and the imaging lens 103. A DIC prism 105 is used for DIC observation. Analyzer 106 is used for PO viewing and DIC viewing.

In the microscope 100, when performing MC observation, a polarizer 123 and an optical element 127 are arranged on the illumination light path as modulation elements (hereinafter referred to as first modulation elements) for modulating the illumination light irradiated to the sample. A modulator 104 is arranged on the observation optical path as a modulating element (hereinafter referred to as a second modulating element) that modulates the transmitted light. Further, when performing PO observation, the polarizer 123 is arranged on the illumination optical path as a first modulation element, and the analyzer 106 is arranged on the observation optical path as a second modulation element. When performing DIC observation, the polarizer 123 and the DIC prism 125 are arranged on the illumination optical path as the first modulation elements, and the analyzer 106 and the DIC prism 105 are arranged on the observation optical path as the second modulation elements. . Thereby, it is possible to visualize an unstained sample and, for example, to sort sperm.

The microscope controller 10 is a device that controls the microscope 100. The microscope controller 10 is connected to the processing device 200 , the input device 50 and the microscope 100 and controls the microscope 100 according to commands from the processing device 200 or the input device 50 .

The display device 30 is, for example, a liquid crystal display, a plasma display, an organic EL display, a CRT display, an LED matrix panel, or the like.

The input device 40 includes a handle 41 and a handle 42. By operating the handle 41 and the handle 42, the operation of a micromanipulator (not shown) that moves the pipettes 43 and 44 is controlled. Pipettes 43 and 44 are used to manipulate samples in microinsemination procedures, including sperm sorting. Pipette 43 is, for example, a holding pipette, and pipette 44 is, for example, an injection pipette.

The input device 50 is a hand switch device for changing settings related to the microscopy method and observation magnification of the microscope 100 . As shown in FIG. 3, the input device 50 has, for example, six buttons (buttons 51 to 56), and the user can quickly switch the settings of the microscope 100 simply by pressing these buttons. can be done.

When the user presses the button 51, the setting of the microscope 100 is switched to BF observation with an observation magnification of 4x (hereinafter referred to as BF4x observation). When the user presses the button 52, the setting of the microscope 100 is switched to the setting of MC observation with an observation magnification of 10 times (hereinafter referred to as MC10x observation). When the user presses the button 53, the setting of the microscope 100 is switched to MC observation with an observation magnification of 20 times (hereinafter referred to as MC20x observation). When the user presses the button 54, the setting of the microscope 100 is switched to MC observation with an observation magnification of 40 times (hereinafter referred to as MC40x observation). When the user presses the button 55, the setting of the microscope 100 is switched to PO observation with an observation magnification of 20 times (hereinafter referred to as PO 20× observation). When the user presses the button 56, the setting of the microscope 100 is switched to DIC observation with an observation magnification of 60 times (hereinafter referred to as DIC60x observation).

The input device 60 is a keyboard. Input device 70 is a mouse. The input device 60 and the input device 70 are each connected to the processing device 200 . Note that the microscope system 1 may include other input devices (not shown) such as a touch panel, a voice input device, and a foot pedal.

The identification device 80 is a device that acquires identification information attached to the sample. Attached to the sample includes, for example, the case where the identification information is attached to a container containing the sample. The identification information is information that identifies the sample, and more specifically, for example, information that identifies the patient who provided the sample. The identification device 80 may be, for example, a barcode reader, an RFID (registered trademark) reader, a QR code (registered trademark) reader, or the like.

The processing device 200 generates a target image based on object detection on the digital image acquired by the imaging device 143 . The generated target image is output to the projection device 153 of the microscope 100 directly or via the microscope controller 10 . The processing device 200 is connected to the microscope 100, the microscope controller 10, the display device 30, the input device 60, the input device 70, and the identification device 80, as shown in FIG. The processing device 200 is also connected to the database server 20 .

The processing device 200 includes an image analysis unit 210, an image generation unit 220, and a storage unit 230 as functional components related to target image generation.

The image analysis unit 210 detects an object on the digital image, and determines sperm to be selected (hereinafter simply referred to as a selection target) from the sperm detected by the object detection (hereinafter referred to as selection candidates). perform image analysis, including target determination; The image analysis performed by the image analysis unit 210 may include candidate evaluation for evaluating selection candidates detected by object detection, in addition to object detection and target determination, and may include tracking processing for tracking selection candidates. may The selection candidate is a candidate to be selected, and the selection target is determined from among the selection candidates.

　The method of object detection is not particularly limited. The image analysis unit 210 may, for example, perform object detection using a learned model stored in the storage unit 230, and detect objects classified as sperm as sorting candidates. Algorithms of the trained model are not particularly limited, but deep learning models such as SSD, YOLO, and FasterR-CNN may be used, for example.

The method of target determination is to evaluate the morphology of candidates for selection detected by object detection and determine the target to be selected. Specifically, the image analysis unit 210 recognizes the morphology of the candidate for selection based on the digital image, and determines whether the candidate for selection is classified into a normal morphology, an abnormal morphology, or an indistinguishable morphology based on the recognized morphology. Selection targets are determined by calculating the probability. Here, "indistinguishable" means that the sperm is out of focus or faces to the side and the morphology cannot be determined accurately. The image analysis unit 210 may detect surplus cytoplasm and other abnormal sites included in the selection candidates based on the digital image, and may count them. Based on the digital image, the image analysis unit 210 may calculate parameters related to the shape and size of the candidates for selection, such as the circularity of the sperm head, the thickness of the neck, and the asymmetry of the neck position. .

The image generation unit 220 generates a target image, which is an image to be sorted determined by the image analysis unit 210, based on the digital image. That is, the image generator 220 generates a target image to be sorted based on sperm detection and sperm morphology evaluation for the digital image.

When the projection device 153 projects the target image onto the image plane, the image generation unit 220 displays the target image projected onto the image plane in a size larger than the selection target in the optical image similarly projected onto the image plane. , the size of the target image to be generated may be adjusted by enlarging or reducing the image clipped from the digital image by digital zooming.

The area to be sorted that is cut out from the digital image is, for example, an area that contains the entire sperm detected by object detection, but it is not necessary to cut out the entire sperm. A setting of the microscope system 1 may be able to change which range of the sorting target is to be cut out. Therefore, the image generation unit 220 may cut out the entire sperm region (from the head to the tail) or cut out the region from the head to the neck of the sperm according to the settings of the microscope system 1. Alternatively, only the head region may be cut out, only the neck region may be cut out, only the tail region may be cut out, or a specified region of the sperm may be cut out. The settings of the microscope system 1 may be changed according to, for example, which part of the object to be sorted is focused by the user for quality judgment.

The target image generated by the image generation unit 220 is output to the projection device 153. As a result, the projection device 153 projects the target image onto the image plane, and the target image is projected onto the image plane in a size larger than that of the selection target in the optical image. Thereby, the target image is preferably displayed on the image plane at a higher overall magnification than the optical image.

The storage unit 230 stores learned models used in image analysis performed by the image analysis unit 210 . Specifically, the storage unit 230 stores a trained model for object detection performed to detect sperm that are candidates for selection and a trained model for candidate evaluation that evaluates sperm that are candidates for selection. there is The image analysis unit 210 performs object detection (sperm detection) and candidate evaluation using the learned model stored in the storage unit 230 .

Note that the processing device 200 may be a general-purpose computer or a dedicated computer. The processing device 200 is not particularly limited to this configuration, but may have, for example, a physical configuration as shown in FIG. Specifically, the processing device 200 may include a processor 201, a storage device 202, an input interface (I/F) 203, an output interface (I/F) 204, and a communication device 205. They may be connected to each other by bus 206 .

The processor 201 may include hardware, and the hardware may include, for example, at least one of circuitry for processing digital signals and circuitry for processing analog signals. Processor 201 may include, for example, one or more circuit devices (eg, ICs) or one or more circuit elements (eg, resistors, capacitors) on a circuit board. The processor 201 may be a CPU (central processing unit). Also, the processor 201 may use various types of processors including GPUs (Graphics Processing Units) and DSPs (Digital Signal Processors). The processor 201 may be a hardware circuit having an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). Processor 201 may include amplifier circuits, filter circuits, etc. for processing analog signals. The processor 201 functions as the image analysis unit 210 and the image generation unit 220 described above by executing programs stored in the storage device 202 .

The storage device 202 may include memory and/or other storage devices. The memory may be, for example, random access memory (RAM). The memory may be a semiconductor memory such as SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory). Storage device 202 may be, for example, a register, a magnetic storage device such as a hard disk drive, an optical storage device such as an optical disk drive, an internal or external hard disk drive, a solid state storage device, a CD-ROM, a DVD, or other optical or magnetic storage device. It may be a disk storage device or other storage device. The storage device 202 stores programs executed by the processor 201, learned models, and other data, and functions as the storage unit 230 described above. Note that the storage device 202 is an example of a non-temporary computer-readable storage medium.

The input I/F 203 is connected to an input device operated by a user of the microscope system 1 (for example, an embryologist), receives an operation signal according to the operation of the input device, and outputs it to the processor 201 .

The output I/F 204 is connected to the display device 30. The output I/F 204 may be further connected to an audio output device such as a speaker that outputs audio, a light emitting device such as a lamp that outputs light, and a vibration device such as a vibrator that outputs vibration, which are not shown.

The communication device 205 is a device that exchanges data with the microscope 100 and other devices. The communication device 205 may be a communication device that exchanges data by wire, or may be a communication device that exchanges data wirelessly. The programs and learned models stored in the storage device 202 may be obtained by the communication device 205 from another device via the Internet.

Figure 5 is a flow chart showing an example of the ICSI procedure by an embryologist. FIG. 6 is a diagram illustrating the configuration of a drop formed as the sample 300 in the petri dish 310. As shown in FIG. FIG. 7 is a flow chart showing an example of a sperm sorting procedure by an embryologist. FIG. 8 is a flowchart showing an example of sorting support processing. FIG. 9 is a flowchart illustrating an example of target image generation processing. FIG. 10 is a flowchart illustrating an example of image display processing. FIG. 11 is a diagram showing an example of an optical image generated by the microscope 100. As shown in FIG. FIG. 12 is a diagram showing an example of a digital image acquired by the imaging device 143. As shown in FIG. FIG. 13 is a diagram showing an example of object detection results for a digital image. FIG. 14 is a diagram for explaining an example of a method of generating a target image. FIG. 15 is a diagram showing an example of an image seen through the eyepiece lens 101. As shown in FIG. Specific utilization of the sperm sorting support method performed by the microscope system 1 in ICSI will be described below with reference to FIGS. 5 to 15 .

First, the user prepares a sample (step S1). Here, the user prepares a sample 300 containing a plurality of drops in a petri dish 310 and places it on the stage 111, as shown in FIG. 6, for example.

A drop 301 is a drop for cleaning and is used for cleaning the pipette. The drop 302 is a sperm suspension drop, for example, a drop of a sperm suspension in a PVP solution. A drop 303 is an oocyte manipulation drop, and is, for example, an oocyte in an m-HTF solution. The m-HTF solution is a Hepps-containing HTF solution supplemented with 10% serum. These drops are coated with mineral oil.

Next, the user sets up the microscope system 1 (step S2). Here, for example, the user presses the button 51 of the input device 50 to switch the setting of the microscope system 1 to BF4×observation. After that, the positions of the pipettes 43 and 44 are adjusted by operating the input device 40 and the pipettes 43 and 44 are brought into focus. Furthermore, the stage 111 is moved to wash the pipettes 43 and 44 with the drop 301 (washing drop).

When the setup is completed, the user confirms the state of the egg (egg cell) in the drop 303 (egg manipulation drop) (step S3). Here, for example, the user presses the button 53 of the input device 50 to switch the setting of the microscope system 1 to MC20x observation. The morphology of the ovum is observed by MC20x observation, and the ovum is selected. Further, for example, the button 55 of the input device 50 may be pressed to switch the setting of the microscope system 1 to PO20x observation. By observing the mitotic spindle of the ovum by PO20x observation, the degree of maturity of the ovum may be determined, and the ovum may be further selected.

When the egg sorting is finished, the user sorts the sperm according to the procedure shown in FIG. 7 (step S4). First, the user presses, for example, the button 53 of the input device 50 to switch the setting of the microscope system 1 to MC20x observation. Then, the stage 111 is moved to move the observation position to the drop 302 (sperm floating drop), and the sperm is focused by MC20× observation (step S11).

Next, the user selects sperm suitable for fertilization by MC20x observation (step S12). Conventionally, in this process, embryologists judged sperm quality based on sperm morphology and motility observed in optical images, and sorted sperm based on that judgment. However, with MC20x, a sufficient field of view is secured for the size of the sperm, which has the advantage of making it easy to understand the motility of the sperm. Too much. For this reason, it is necessary to observe the spermatozoa selected in step S12 at a higher magnification, which causes the problem that the sorting operation takes time and requires reworking.

In light of such problems, in step S12, the microscope system 1 classifies the sperm morphology into good, poor, and indistinguishable on the image plane, and the sperm determined to be good is larger than the target sperm in the optical image. Display by size. By projecting the image in a size larger than the sorting object in the optical image, it is possible to assist the user in determining the morphology of the sperm, which has been difficult in this step in the past. As a result, it is possible to suppress rework due to different judgments in the post-process, and for example, it is possible to omit switching to MC40x observation in step S14 and shorten the time required for sperm sorting. Furthermore, by projecting the optical image and the target image onto the same image plane, it is possible to accurately identify the morphology of the sperm of interest (sperm to be sorted) using the target image, while mainly confirming the motility of the sperm using the optical image. can be grasped. Therefore, since sperm can be sorted by confirming both motility and morphology, it is possible to suppress rework due to different determinations in the subsequent steps.

Specifically, in step S12, the microscope system 1 performs the selection support process shown in FIG. 8, so that the target image is projected onto the image plane together with the optical image. In the selection support process, first, the microscope system 1 projects an optical image of the sample onto the image plane (step S21). Here, the microscope 100 forms, for example, an optical image O1 shown in FIG. 11 on the image plane. A region 143R shown in FIG. 11 indicates a region imaged by the imaging device 143 in step S22.

The microscope system 1 acquires a digital image at the same time as step S21 (step S22). Here, the imaging device 143 acquires, for example, a digital image D1 of the sample shown in FIG. 12 based on the light from the sample, and outputs the acquired digital image D1 to the processing device 200 .

After that, the microscope system 1 generates a target image based on the digital image (step S23). Here, the processing device 200 performs the target image generation processing shown in FIG. In the target image generation process, the processing device 200 first performs object detection on the digital image D1 (step S31). Here, for example, the image analysis unit 210 performs object detection by inputting the digital image D1 as an input image into the trained model, and detects spermatozoa as sorting candidates as shown in FIG. Note that FIG. 13 shows a box B attached to the position of an object (selection candidate) classified by the object detection.

When a selection candidate is detected by object detection, the processing device 200 evaluates the selection candidate (step S32). Here, the image analysis unit 210 evaluates each of the selection candidates. Specifically, the image analysis unit 210 evaluates the forms of selection candidates.

After that, the processing device 200 determines sorting targets based on the evaluation performed in step S32 (step S33). Here, based on the evaluation, the image analysis unit 210 determines the selection candidate with the highest evaluation among the selection candidates as the selection target. Specifically, for example, as shown in FIG. 13, the central sperm with the highest probability of determining good sperm is determined as the selection target.

When the selection target is determined, the processing device 200 cuts out the selection target area from the digital image (step S34). Here, for example, as shown in FIG. 14, the image generation unit 220 cuts out a selection target area from the digital image D1 to generate the target image T1. The size of the target image T1 may be adjusted in step S34 so that it is displayed larger than the selection target in the optical image when projected onto the image plane.

When the target image is generated in step S23, the microscope system 1 displays the target image on the image plane in a size larger than the selection target in the optical image (step S24). Here, the processing device 200 performs the image display processing shown in FIG. In the image display process, the processing device 200 first determines the overall magnification of the target image T1 displayed on the image plane (step S41), and further determines the size of the target image T1 (step S42). In step S41, when the total magnification of the optical image formed on the image plane is, for example, 200 times, the image analysis unit 210 sets the total magnification of the target image T1 projected on the image plane to a magnification exceeding 200 times (for example, 1000 times). times). Further, in step S42, when the selection target in the optical image formed on the image plane is projected onto a region of 1 mm×1 mm, the image analysis unit 210 determines the size of the target image T1 to be 1 mm×1 mm or more. do.

Further, the processing device 200 determines the position of the target image T1 displayed on the image plane (step S43). Here, the processing device 200 determines the position of the target image while avoiding the region of the sperm to be sorted included in the optical image, for example, so that the region of the sperm to be sorted included in the optical image does not overlap the target image. do. Note that the position of the target image may be arranged at a previously specified position such as the upper side, the lower side, the right side, or the left side with respect to the center of the field of view.

Finally, the processing device 200 displays the target image T1 with the overall magnification determined in step S41 and the size determined in step S42 at the position determined in step S43 (step S44). Here, the processing device 200 assigns each pixel of the target image T1 to a pixel of the projection device 153 based on the overall magnification determined in step S41, the size determined in step S42, and the position determined in step S43. As a result, the projection device 153 projects the target image T1 in a size larger than the selection target in the optical image at a position that does not overlap the selection target in the optical image on the image plane. This allows the user to simultaneously confirm the optical image O1 and the target image T1 displayed on the image plane, as shown in FIG. 15, for example. Therefore, sperm suitable for fertilization can be selected with high accuracy by confirming both sperm morphology and motility. In addition, in step S12, the sorting support process shown in FIG. 8 is repeatedly performed, so that the sorting target spermatozoa moving within the field of view can be tracked, so that the sorting target spermatozoa can be carefully observed in the target image. can be done.

When the sperm are sorted in step S12, the user immobilizes the sperm by scratching the tail of the sperm by RC20x observation (step S13). Here, the user immobilizes the sperm by rubbing the tail of the sperm against the bottom surface of the petri dish 310 with a pipette.

After that, the user may observe the morphology of the immobilized sperm in more detail and further sort the sperm (step S14). Here, for example, the user presses the button 54 of the input device 50 to switch the setting of the microscope system 1 to MC40× observation. The user may then further sort the sperm with MC40x viewing. Here, too, the microscope system 1 performs the sorting support process shown in FIG. 8 in the same manner as in step S12, and projects the target image onto the image plane in a size larger than the sorting target in the optical image. May assist in sperm sorting operations.

After the sperm selection is completed, the user then takes the selected sperm into the pipette 44, which is an injection pipette, and moves the observation position to the drop 303 (egg manipulation drop) (step S15). Complete the sperm sorting sequence shown in .

When sperm selection is completed, the user confirms the position of the spindle in preparation for sperm injection (step S5). Here, the user observes the egg selected in step S3 in the drop 303 and confirms the position of the mitotic spindle of the egg. Specifically, for example, the user presses the button 55 of the input device 50 to switch the setting of the microscope system 1 to PO20× observation. After that, the user changes the orientation of the spindle by operating the pipette 43, which is a holding pipette, so that the spindle of the egg visualized by PO20x observation is positioned in the direction of 12 o'clock or 6 o'clock. This is to prevent the spindle from being damaged by the pipette thrust into the ovum from the 3 o'clock or 9 o'clock direction in step S6, which will be described later.

Finally, the user injects the sperm into the egg (step S6) and ends the ICSI. Here, for example, the user presses the button 53 of the input device 50 to switch the setting of the microscope system 1 to MC20x observation. After that, in MC20× observation, the user fixes the ovum whose orientation has been adjusted in step S5 with a holding pipette 43 and pierces it with an injection pipette 44 . After that, good sperm is injected into the egg from the pipette 44 .

After completing the series of ICSI procedures shown in Figure 5, the user returns the sperm-injected egg to the incubator and cultures it. Also, the user may operate the processing device 200 using the input device 60 and the input device 70 to store the information obtained by ICSI in the database server 20 . For example, sperm and egg patient information (maternal clinical data, sperm containing sperm test results, etc.), sperm and egg images, sperm injected image, sorted sperm image, ICSI working time, etc. data (for example, type, concentration, PH, etc.) of the culture medium may be associated with each other and stored in the database server 20 .

As described above, in the microscope system 1, in ICSI, the target image, which is the image of the sperm to be sorted, is projected onto the image plane in a size larger than that of the sorting target in the optical image. The sperm size is about 60 μm, and an objective lens of at least 20× is used to distinguish good sperm. Since the field number of an inverted microscope is generally about 22, the actual field of view is about Φ1 mm. It is very difficult to select free-moving spermatozoa with a size of about 60 μm within the real field of view Φ1 mm. In general, sperm that are presumed to be good sperm have high motility, and the ICSI work needs to be done in a short time. A good/bad judgment must be made. Even in a work environment with such strict restrictions, the microscope system 1 can determine good sperm candidates from a morphological point of view while mainly confirming the motility of sperm in optical images. Then, it is possible to confirm the morphology of the sperm at the same time by projecting it onto the image plane at a higher total magnification than the optical image. As a result, good sperm can be selected appropriately and in a short period of time based on both motility and morphology of the sperm, so that the fertilization success rate can be improved. Therefore, according to the microscope system 1, it is possible to effectively assist the sorting operation of the spermatozoa in the sample.

[Second embodiment]
FIG. 16 is a diagram for explaining another example of the target image generation method. FIG. 17 is a diagram showing another example of an image seen through the eyepiece 101. In FIG. This embodiment will be described below with reference to FIGS. 16 and 17. FIG. Note that the configuration of the microscope system according to the present embodiment (hereinafter also simply referred to as the microscope system) is the same as that of the microscope system 1 .

This embodiment differs from the first embodiment in that a plurality of target images are displayed on the image plane. Other points are the same as in the first embodiment. Specifically, in step S33 of the target image generation process shown in FIG. 9, the processing device 200 determines a plurality of selection targets based on the evaluation performed in step S32. The number of sorting targets may be predetermined, such as two. Further, the number of selection targets may be determined based on the evaluation in step S32, and the processing device 200 may, for example, determine all selection candidates evaluated to exceed a certain criterion as selection targets. good.

Accordingly, in step S34, for example, as shown in FIG. 16, the processing device 200 cuts out a plurality of selection target regions from the digital image to generate a plurality of target images (target image T1, target image T2). .

Also with the microscope system according to the present embodiment, good sperm are projected onto the image plane in a size larger than that of the sorting object in the optical image, so that it is possible to assist the user in determining the morphology of the sperm. Since both the motility and morphology of the sperm can be simultaneously confirmed from the target image, good sperm can be selected appropriately and in a short period of time based on both the motility and morphology of the sperm. This is the same as the microscope system 1 according to the embodiment of . Further, in the microscope system according to the present embodiment, as shown in FIG. 17, a plurality of spermatozoa included in the optical image O1 is determined as a sorting target, and a plurality of target images (target image T1, target image T2) are optically selected. displayed with the image. Therefore, since the possibility of early discovery of good sperm that exceeds the user's criteria for judging quality increases, sperm can be sorted more efficiently than the microscope system 1 .

[Third embodiment]
FIG. 18 is a flow chart showing another example of the sorting support process. FIG. 19 is a flowchart showing another example of image display processing. FIG. 20 is a diagram for explaining an example of a method of generating an auxiliary image. FIG. 21 is a diagram showing still another example of an image seen through the eyepiece 101. In FIG. The present embodiment will be described below with reference to FIGS. 18 to 21. FIG. Note that the configuration of the microscope system according to the present embodiment (hereinafter also simply referred to as the microscope system) is the same as that of the microscope system 1 .

This embodiment differs from the first embodiment in that the sorting support process shown in FIG. 18 is performed instead of the sorting support process shown in FIG. Other points are the same as in the first embodiment.

In the selection support process shown in FIG. 18, the microscope system projects an optical image of the sample onto the image plane (step S51), acquires a digital image (step S52), and generates a target image (step S53). The processing from step S51 to step S53 is the same as the processing from step S21 to step S23 in FIG.

After that, the microscope system generates an auxiliary image for the target image (step S54). Here, the processing device 200 generates the auxiliary image. The auxiliary image is an image that provides the user with information indicating whether the morphology of the sperm to be sorted is close to good, poor, or indistinguishable, and information on the size of the sperm head to be sorted.

In step S54, for example, as shown in FIG. 20, the processing device 200 first generates an evaluation result ER by evaluating the sorting target based on the target image T1, and then generates an auxiliary image ER based on the evaluation result ER. A1 may be generated. Note that in FIG. 20, the auxiliary image A1 includes evaluation information E1 regarding the morphology of the object to be sorted, and a marker M1 indicating the size of the head of the object to be sorted. The marker M1 is also an example of a marker indicating a target position or area.

In addition, the auxiliary image A1 may include a marker indicating the position or region of excess cytoplasm in the sperm instead of or in addition to the marker M1 indicating the head size in the sperm. That is, the auxiliary image A1 may include a marker indicating the presence or absence of surplus cytoplasm. The auxiliary image A1 may also include markers indicating the degree of circularity of the head, the thickness of the neck, and the asymmetry between the positions of the head and neck. In other words, it may include a marker indicating the target position or area to be sorted. In addition, as shown in FIG. 20, the evaluation information E1 preferably includes numerical information indicating the size of the head. Also, the auxiliary image A1 may include a marker indicating the position or region of the space or other defective part of the sperm.

Note that FIG. 20 shows an example in which the auxiliary image A1 is generated based on the target image T1, but the auxiliary image A1 is information obtained in the process of generating the target image T1 in step S53 (for example, candidate evaluation information). result).

When the target image and the auxiliary image are generated, the microscope system displays the target image and the auxiliary image on the image plane (step S55). 19, the target image T1 and the auxiliary image A1 are displayed on the image plane together with the optical image O1, as shown in FIG.

In the image display process, the processing device 200 first determines the overall magnification of the target image T1 arranged on the image plane (step S61), then determines the size of the target image T1 (step S62), and further The position of the target image T1 displayed on the image plane is determined (step S63). The processing from step S61 to step S63 is the same as the processing from step S41 to step S43 in FIG.

Further, the processing device 200 determines the position of the auxiliary image A1 arranged on the image plane (step S64). In step S64, the processing device 200 may determine the position of the auxiliary image A1 in relation to the target image T1. The processing device 200 may determine the positional relationship between the auxiliary image A1 and the target image T1, for example, such that the marker M1 in the auxiliary image A1 overlaps the region of interest in the target image T1. That is, the processing device 200 generates a projection image including the auxiliary image A1 and the target image T1 in the determined positional relationship.

Finally, the processing device 200 places the target image T1 having the overall magnification determined in step S61 and the size determined in step S62 at the position determined in step S63, and the auxiliary image A1 at the position determined in step S64. display (step S65). That is, the processing device 200 projects the projection image generated in step S64 onto the image plane. Here, the processing device 200 assigns each pixel of the projection image to a pixel of the projection device 153 . That is, each pixel of the target image T1 is assigned to a pixel of the projection device 153 based on the overall magnification determined in step S61, the size determined in step S62, and the position determined in step S63. Pixels are assigned to pixels of projection device 153 based on the positions determined in step S64. As a result, the projection device 153 projects the target image T1 at a position that does not overlap the selection target in the optical image O1 on the image plane in a size larger than the selection target in the optical image, and also projects the auxiliary image A1 on the image plane. Project to the appropriate position of Thereby, the user can confirm the optical image O1, the target image T1, and the auxiliary image A1 at the same time, as shown in FIG.

With the microscope system according to the present embodiment as well, the target image is projected onto the image plane in a size larger than that of the sorting target in the optical image. Since both the motility and morphology of the sperm can be simultaneously confirmed from the target image, good sperm can be selected appropriately and in a short period of time based on both the motility and morphology of the sperm. This is the same as the microscope system 1 according to the embodiment of . In addition, as shown in FIG. 21, the microscope system according to the present embodiment projects an auxiliary image that provides information related to the target image onto the image plane, thereby further assisting the user in determining the quality of the sperm. be able to. In particular, by superimposing the marker indicating the target position or area of the selection target on the target image having a size larger than the selection target in the optical image, the problem of being too small and difficult to see when superimposed on the optical image is avoided. , to inform the user of the part to which attention should be paid. Therefore, according to the microscope system according to the present embodiment, it is possible to select sperm suitable for fertilization with high accuracy, so that it is possible to suppress the difference in fertilization rate between embryologists.

The above-described embodiments are specific examples to facilitate understanding of the invention, and the invention is not limited to these embodiments. Modifications of the above-described embodiments and alternatives to the above-described embodiments may be included. That is, each embodiment is capable of modifying its components without departing from its spirit and scope. Further, new embodiments can be implemented by appropriately combining multiple components disclosed in one or more embodiments. Also, some components may be omitted from the components shown in each embodiment, or some components may be added to the components shown in the embodiments. Furthermore, the order of the processing procedures shown in each embodiment may be changed as long as there is no contradiction. That is, the microscope system, projection unit, and selection support method of the present invention can be modified and modified in various ways without departing from the scope of the claims.

Although the microscope system 1 has been exemplified in the above-described embodiment, the configuration of the microscope system is not limited to this example. For example, the microscope system 2 shown in FIG. 22 may be used. The microscope system 2 differs from the microscope system 1 in that a microscope 400 is provided instead of the microscope 100 . A microscope 400 has a projection unit 500 between a microscope main body 410 and a lens barrel 420 .

Projection unit 500 is a projection unit for a microscope, and includes a projection section (splitter 151, lens 152, and projection device 153) corresponding to projection unit 150 shown in FIG. 1, and an imaging section corresponding to imaging unit 140 shown in FIG. (splitter 141 and imaging device 143 ) and an image processor 510 . The image processing unit 510 functions as the image analysis unit 210, the image generation unit 220, and the storage unit 230 shown in FIG.

With the projection unit 500 and the microscope system 2, the same effects as those of the microscope system 1 can be obtained. In addition, by using the projection unit 500, an existing microscope system can be expanded to obtain the above effects, so the existing microscope system can be effectively utilized.

In the above-described embodiment, the case of sorting sperm is described as an example, but the sorting target is not limited to sperm. In addition, it is desirable that the sorting target be selected based on both motility and morphology.

In the above-described embodiment, an example in which the target image is projected onto the image plane by the projection device 153 is shown. A device may be used. Further, in the above-described embodiment, the target image is displayed on the image plane in a size larger than the selection target in the optical image. It may be displayed in a size equal to or larger than that of the sorting object inside.

Also, in the above-described embodiment, an example of generating one target image for each of the sorting targets was shown, but two or more target images may be generated for each of the sorting targets. For example, a target image obtained by cropping the entire sperm area and a target image obtained by cropping only the sperm head area may be generated and arranged on the image plane at different overall magnifications. As a result, it is possible to observe the entire sperm at an appropriate magnification while observing a specific portion of the sperm at a higher magnification.

Note that the target image may be updated periodically based on, for example, the frame rate of the moving image captured by the imaging device 143, and the update frequency may be changeable according to settings. That is, the target image may be projected onto the image plane as a moving image, or may be projected as a still image. By projecting the target image as a still image, the user can carefully observe the morphology of the sperm. Further, by projecting the target image as the sperm image, it is possible to avoid a situation in which the auxiliary image is projected at a position shifted from the predetermined position with respect to the target image due to delay in creation of the auxiliary image. On the other hand, by projecting the target image as a moving image, the user can observe the morphology of the sperm in detail without being affected by the timing of photographing. This in turn means that the processing device tracks the sorting object. Further, even when projecting as a moving image, by adjusting the updating frequency, it is possible to avoid the inconvenience caused by the delay in creating the auxiliary image.

As used herein, the phrase "based on A" does not mean "based only on A", but means "based at least on A", and furthermore, "based at least in part on A". It also means "te". That is, "based on A" may be based on B in addition to A, or may be based on a portion of A.

1, 2 microscope system 10 microscope controller 20 database server 30 display device 40, 50, 60, 70 input device 41, 42 handle 43, 44 pipette 80 identification device 100 microscope 101 eyepiece 102, 102a-102c objective lens 103 imaging lens 104 Modulators 105, 125 DIC Prism 106 Analyzer 110 Microscope Body 111 Stage 112 Revolver 120 Transmitted Illumination System 121 Light Source 122 Universal Condenser 123 Polarizer 124 Turret 126 Aperture Plate 127 Optical Element 127a Slit Plate 127b Polarizer 128 Condenser Lens 130 Laser Assisted Hatching Unit 131, 141, 151 splitter 133 scanner 135 laser 140 imaging unit 143 imaging device 143R imaging region 150 projection unit 153 projection device 160 intermediate magnification unit 170 eyepiece tube 200 processing device 201 processor 202 storage device 203 input interface 204 output interface 205 communication Controller 206 Bus 210 Image analysis unit 220 Image generation unit 230 Storage unit 300 Sample 310 Petri dish 400 Microscope 410 Microscope body 420 Lens barrel 500 Projection unit 510 Image processing unit A1 Auxiliary image B Box D1 Digital image ER Evaluation result E1 Evaluation information O1 Optics Image M1 Markers T1, T2 Target image

Claims (9)

1. a microscope that forms an optical image of a sperm sample containing a sorting target;
   an imaging device for acquiring a digital image of the sperm sample;
   a processing device for generating a target image, which is the image to be sorted, based on sperm detection and sperm morphology evaluation in the digital image;
   and an optical device that displays the target image on an image plane on which the optical image is formed in a size larger than that of the selection target in the optical image.
2. The microscope system of claim 1, wherein
   A microscope system, wherein the processing device tracks sorting candidates detected in the sperm detection.
3. In the microscope system according to claim 1 or claim 2,
   The processing device determines the selection target based on the classification result of the sperm morphology being classified as good, poor, or indistinguishable from among the selection candidates detected in the sperm detection. .
4. In the microscope system according to any one of claims 1 to 3,
   The microscope system, wherein the processing device performs the sperm detection using a trained model.
5. A microscope system according to claim 3, wherein
   The processing device generates an auxiliary image for the target image,
   The optical device displays the auxiliary image together with the target image on the image plane,
   The microscope system, wherein the auxiliary image includes evaluation information regarding evaluation results classified into good, bad, and indistinguishable.
6. A microscope system according to claim 3, wherein
   The processing device generates an auxiliary image for the target image,
   The optical device displays the auxiliary image together with the target image on the image plane,
   A microscope system, wherein the auxiliary image includes a marker indicating a position or area of interest in the target image.
7. A microscope system according to claim 6, wherein
   The auxiliary image includes markers indicating the size of the sperm head in the target image, the presence or absence of excess cytoplasm, the roundness of the head, the thickness of the neck, and the asymmetry between the head and neck positions. A microscope system characterized by:
8. A projection unit attached to a microscope,
   an imaging unit that acquires a digital image of the sperm sample;
   a processing unit that generates a target image that is an image to be sorted contained in the sperm sample based on sperm detection and sperm morphology evaluation for the digital image;
   a projection unit that displays the target image in a size larger than that of the sorting target in the optical image on an image plane formed by the microscope on which the optical image of the sperm sample is formed. .
9. forming an optical image of a sperm sample containing a sorting target;
   obtaining a digital image of the sperm sample;
   generating a target image, which is the image to be sorted, based on sperm detection and sperm morphology evaluation for the digital image;
   A selection support method, wherein the target image is displayed on an image plane on which the optical image is formed in a size larger than that of the selection target in the optical image.

Images