WO2021200003A1 - 顕微鏡システム、投影ユニット、及び、精子選別支援方法 - Google Patents

顕微鏡システム、投影ユニット、及び、精子選別支援方法 Download PDF

Info

Publication number
WO2021200003A1
WO2021200003A1 PCT/JP2021/009685 JP2021009685W WO2021200003A1 WO 2021200003 A1 WO2021200003 A1 WO 2021200003A1 JP 2021009685 W JP2021009685 W JP 2021009685W WO 2021200003 A1 WO2021200003 A1 WO 2021200003A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
sperm
microscope system
microscope
information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/009685
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
円香 木村
拓人 山根
敏征 服部
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Olympus Corp
Original Assignee
Olympus Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Olympus Corp filed Critical Olympus Corp
Priority to EP21779707.5A priority Critical patent/EP4130843A4/en
Priority to CN202180025391.0A priority patent/CN115485602A/zh
Priority to JP2022511751A priority patent/JPWO2021200003A1/ja
Publication of WO2021200003A1 publication Critical patent/WO2021200003A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G02B21/365—Control or image processing arrangements for digital or video microscopes
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N15/00—Investigating characteristics of particles; Investigating permeability, pore-volume or surface-area of porous materials
    • G01N15/10—Investigating individual particles
    • G01N15/14—Optical investigation techniques, e.g. flow cytometry
    • G01N15/1429—Signal processing
    • G01N15/1433—Signal processing using image recognition
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
    • G06V20/695—Preprocessing, e.g. image segmentation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
    • G06V20/698—Matching; Classification
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61D—VETERINARY INSTRUMENTS, IMPLEMENTS, TOOLS, OR METHODS
    • A61D19/00—Instruments or methods for reproduction or fertilisation
    • A61D19/02—Instruments or methods for reproduction or fertilisation for artificial insemination
    • C—CHEMISTRY; METALLURGY
    • C12—BIOCHEMISTRY; BEER; SPIRITS; WINE; VINEGAR; MICROBIOLOGY; ENZYMOLOGY; MUTATION OR GENETIC ENGINEERING
    • C12M—APPARATUS FOR ENZYMOLOGY OR MICROBIOLOGY; APPARATUS FOR CULTURING MICROORGANISMS FOR PRODUCING BIOMASS, FOR GROWING CELLS OR FOR OBTAINING FERMENTATION OR METABOLIC PRODUCTS, i.e. BIOREACTORS OR FERMENTERS
    • C12M21/00—Bioreactors or fermenters specially adapted for specific uses
    • C12M21/06—Bioreactors or fermenters specially adapted for specific uses for in vitro fertilization
    • C—CHEMISTRY; METALLURGY
    • C12—BIOCHEMISTRY; BEER; SPIRITS; WINE; VINEGAR; MICROBIOLOGY; ENZYMOLOGY; MUTATION OR GENETIC ENGINEERING
    • C12M—APPARATUS FOR ENZYMOLOGY OR MICROBIOLOGY; APPARATUS FOR CULTURING MICROORGANISMS FOR PRODUCING BIOMASS, FOR GROWING CELLS OR FOR OBTAINING FERMENTATION OR METABOLIC PRODUCTS, i.e. BIOREACTORS OR FERMENTERS
    • C12M47/00—Means for after-treatment of the produced biomass or of the fermentation or metabolic products, e.g. storage of biomass
    • C12M47/04—Cell isolation or sorting
    • C—CHEMISTRY; METALLURGY
    • C12—BIOCHEMISTRY; BEER; SPIRITS; WINE; VINEGAR; MICROBIOLOGY; ENZYMOLOGY; MUTATION OR GENETIC ENGINEERING
    • C12N—MICROORGANISMS OR ENZYMES; COMPOSITIONS THEREOF; PROPAGATING, PRESERVING, OR MAINTAINING MICROORGANISMS; MUTATION OR GENETIC ENGINEERING; CULTURE MEDIA
    • C12N5/00—Undifferentiated human, animal or plant cells, e.g. cell lines; Tissues; Cultivation or maintenance thereof; Culture media therefor
    • C12N5/06—Animal cells or tissues; Human cells or tissues
    • C12N5/0602—Vertebrate cells
    • C12N5/0608—Germ cells
    • C12N5/0612—Germ cells sorting of gametes, e.g. according to sex or motility
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N15/00—Investigating characteristics of particles; Investigating permeability, pore-volume or surface-area of porous materials
    • G01N15/10—Investigating individual particles
    • G01N2015/1006—Investigating individual particles for cytology
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N15/00—Investigating characteristics of particles; Investigating permeability, pore-volume or surface-area of porous materials
    • G01N15/10—Investigating individual particles
    • G01N2015/1028—Sorting particles
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10056—Microscopic image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • the disclosure of this specification relates to a microscope system, a projection unit, and a sperm sorting support method.
  • ART is a general term for techniques such as in vitro fertilization (IVF: In vitro fertilization) and microfertilization, in which eggs and sperms taken out from humans are fertilized in vitro. Distinguished from in vitro fertilization.
  • Patent Document 1 describes a microscope suitable for an intracytoplasmic sperm injection method (ICSI) used in microinsemination, which is a type of ART.
  • ICSI is a method of directly injecting sperm into an egg by piercing an injection pipette containing sperm into an egg fixed with a holding pipette.
  • an object of one aspect of the present invention is to provide a technique for easily grasping high-quality sperms in sperm sorting work.
  • the microscope system is based on a microscope, a photographing device that acquires a photographed image of a sample containing sperm placed on the stage of the microscope, and the photographed image acquired by the photographing device.
  • An image processing device that generates an auxiliary image including auxiliary information that contributes to sorting and the auxiliary image generated by the image processing device are superimposed on an optical image of the sample formed on an image plane on the optical path of the microscope. It is provided with a superimposing device.
  • the projection unit according to one aspect of the present invention is a projection unit used by being mounted on a microscope, and is auxiliary information that contributes to sperm selection based on a photographed image of a sample containing sperm placed on the stage of the microscope.
  • An image processing device that generates an auxiliary image including To be equipped with.
  • the sperm sorting support method acquires a photographed image of a sample containing sperm placed on a stage of a microscope, and based on the acquired image, an auxiliary image including auxiliary information contributing to sperm separation. Is generated, and the generated auxiliary image is superimposed on the optical image of the sample formed on the image plane on the optical path of the microscope.
  • good quality sperm can be easily grasped in the sperm sorting operation.
  • FIG. 1 is a diagram illustrating the configuration of the microscope system 1.
  • FIG. 2 is a diagram illustrating the configuration of the microscope 100.
  • FIG. 3 is a diagram illustrating the configuration of the operation unit of the input device 50.
  • FIG. 4 is a diagram illustrating the configuration of the image processing device 200.
  • the microscope system 1 is a system for observing a sample by looking through the eyepiece 101.
  • the microscope system 1 is an inverted microscope system provided with a transmission illumination system 120 used for microinsemination, particularly sperm sorting, and is used by, for example, an embryo culture technician.
  • the sample to be observed by the microscope system 1 is a sperm suspension containing sperm contained in a petri dish or the like during sperm sorting work.
  • the microscope system 1 includes at least a microscope 100, a photographing device 143, a projection device 153 which is an example of a superimposing device, and an image processing device 200.
  • the microscope system 1 superimposes an auxiliary image on the optical image by projecting the auxiliary image on the image plane on which the optical image of the sample is formed by the optical system of the microscope 100 by using the projection device 153.
  • the auxiliary image contains information that contributes to sperm separation (hereinafter referred to as auxiliary information).
  • the auxiliary image is created by the image processing device 200 based on the captured image of the sample captured by the imaging device 143.
  • the embryo culture person of the microscope system 1 who looks into the eyepiece 101 and observes the sample can obtain information that contributes to sperm selection without taking his eyes off the eyepiece 101, so that good sperm can be shortened. It is possible to accurately identify and collect by time. Therefore, according to the microscope system 1, it is possible to support the sperm sorting work of the embryo cultivator.
  • the microscope system 1 includes a microscope 100 including an imaging device 143 and a projection device 153, and an image processing device 200. Further, the microscope system 1 includes a microscope controller 10, a display device 30, a plurality of input devices (input device 40, input device 50, input device 60, input device 70), and an identification device 80. Further, the microscope system 1 is connected to a database server 20 in which various data are stored.
  • the microscope 100 is an inverted microscope provided with an eyepiece 101, and as shown in FIG. 1, the microscope main body 110, a plurality of objective lenses 102 attached to the microscope main body 110, a stage 111, a transmission illumination system 120, and a transmission illumination system 120.
  • the eyepiece tube 170 is provided.
  • the microscope 100 is provided with a modulation element for visualizing an unstained sample such as a sperm or an egg in each of the illumination optical path and the observation optical path.
  • Users such as embryo culture specialists use the microscope 100 to perform four microscopy methods: bright field (BF) observation, polarization (PO) observation, differential interference contrast (DIC) observation, and modulation contrast (MC) observation. Can be observed.
  • the modulated contrast observation is also referred to as relief contrast (RC) observation.
  • the plurality of objective lenses 102 are attached to the revolver 112. As shown in FIG. 2, the plurality of objective lenses 102 include an objective lens 102a for BF observation, an objective lens 102b for PO observation and DIC observation, and an objective lens 102c for MC observation. Further, the objective lens 102c includes a modulator 104.
  • the modulator 104 includes three regions having different transmittances (for example, a region having a transmittance of about 100%, a region having a transmittance of about 5%, and a region having a transmittance of about 0%).
  • the plurality of objective lenses 102 may include a plurality of objective lenses having different magnifications for each microscopy method.
  • a 4x objective lens for BF observation a 10x, 20x, 40x objective lens for MC observation, a 20x objective lens for PO observation, and a 60x objective lens for DIC observation are included. Will be described as an example.
  • the revolver 112 is a switching device for switching objective lenses arranged on an optical path among a plurality of objective lenses 102.
  • the revolver 112 switches the objective lens arranged on the optical path according to the microscopy and the observation magnification.
  • the objective lens arranged on the optical path by the revolver 112 guides the transmitted light transmitted through the sample to the eyepiece 101.
  • the sample contained in the container is placed on the stage 111.
  • the container is, for example, a petri dish, and the sample contains germ cells such as sperm and eggs.
  • the stage 111 moves in the direction of the optical axis of the objective lens 102 arranged on the optical path and in the direction orthogonal to the optical axis of the objective lens 102.
  • the stage 111 may be a manual stage or an electric stage.
  • the transmission illumination system 120 illuminates the sample placed on the stage 111 from above the stage 111.
  • the transmission illumination system 120 includes a light source 121 and a universal capacitor 122.
  • the light source 121 may be, for example, an LED (Light Emitting Diode) light source or a lamp light source such as a halogen lamp light source.
  • the universal capacitor 122 includes a polarizer 123 (first polarizing plate), a plurality of optical elements housed in the turret 124, and a capacitor lens 128.
  • Polarizer 123 is used in MC observation, PO observation and DIC observation.
  • the turret 124 contains a plurality of optical elements that are switched and used according to the microscopy method.
  • the DIC prism 125 is used for DIC observation.
  • the opening plate 126 is used for BF observation and PO observation.
  • the optical element 127 is a combination of a slit plate 127a, which is a light-shielding plate on which a slit is formed, and a polarizing plate 127b (second polarizing plate) arranged so as to cover a part of the slit, and is observed by MC observation. used.
  • the eyepiece tube 170 includes an eyepiece lens 101.
  • the imaging lens 103 is arranged between the eyepiece lens 101 and the objective lens 102.
  • the imaging lens 103 forms an optical image of the sample on the image plane IP between the eyepiece lens 101 and the imaging lens 103 based on the transmitted light.
  • an auxiliary image described later is also formed on the image plane IP based on the light from the projection device 153.
  • the auxiliary image is superimposed on the optical image in the image plane IP.
  • the user of the microscope system 1 uses the eyepiece lens 101 to observe a virtual image of an image in which an auxiliary image is superimposed on an optical image formed on the image plane IP.
  • the microscope main body 110 includes a laser assisted hatching unit 130, an imaging unit 140, and a projection unit 150. Further, as shown in FIG. 2, the microscope main body 110 includes an intermediate scaling unit 160. Further, the microscope main body 110 includes a DIC prism 105 and an analyzer 106 so as to be removable with respect to an optical path.
  • the laser assisted hatching unit 130 is a laser unit arranged between the objective lens 102 and the imaging lens 103.
  • the laser assisted hatching unit 130 irradiates the sample with the laser beam by introducing the laser beam between the objective lens 102 and the imaging lens 103. More specifically, the laser assisted hatching unit 130 irradiates, for example, a laser beam on a zona pellucida surrounding an embryo grown from a fertilized egg.
  • the laser assisted hatching unit 130 includes a splitter 131, a scanner 133, a lens 134, and a laser 135.
  • the splitter 131 is, for example, a dichroic mirror.
  • the scanner 133 is, for example, a galvano scanner, and adjusts the irradiation position of the laser beam in a direction orthogonal to the optical axis of the objective lens 102.
  • the lens 134 converts the laser beam into a parallel luminous flux. As a result, the laser beam is focused on the sample by the objective lens 102.
  • the photographing unit 140 includes a splitter 141 and a photographing device 143 that acquires a photographed image of a sample based on transmitted light.
  • the photographing unit 140 is arranged between the imaging lens 103 and the eyepiece lens 101.
  • the splitter 141 is, for example, a half mirror.
  • the imaging lens 103 forms an optical image of the sample on the light receiving surface of the image pickup device included in the photographing apparatus 143.
  • the photographing device 143 is, for example, a digital camera, and the image sensor included in the photographing device 143 is, for example, a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal-Oxide-Semiconducor) image sensor, or the like.
  • the image sensor detects light from the sample and converts the detected light into an electric signal by photoelectric conversion.
  • the photographing unit 140 outputs the photographed image acquired by the photographing device 143 to the image processing device 200.
  • the microscope system 1 is used for sperm sorting, but the details of sperm, for example, the tail part are about ⁇ 0.5 ⁇ m.
  • the total magnification is 10x.
  • the pitch of the pixel projected image on the object surface is 0.345 ⁇ m, and the tail portion of the sperm can also be discriminated.
  • the region consisting of effective pixels has a size that fills the entire field of view.
  • the projection unit 150 is arranged between the imaging lens 103 and the eyepiece lens 101. As shown in FIG. 2, the projection unit 150 includes a splitter 151, a lens 152, and a projection device 153.
  • the splitter 151 is, for example, a half mirror.
  • the projection device 153 projects an auxiliary image generated by the image processing device 200. More specifically, the projection device 153 causes the lens 152 to collect the light from the projection device 153 at the same position as the image plane of the imaging lens 103, that is, the image plane IP on which the optical image is formed. , The auxiliary image is projected on the optical image formed on the image plane IP on the optical path.
  • the size of sperm from the head to the tail is about 60 ⁇ m, and the short side of the size of the head is about 3 ⁇ m.
  • the sperm image is 1.2 mm x 0.06 mm. It becomes the size of.
  • the minimum rectangle is about 1.5 mm ⁇ 0.1 mm.
  • the projection magnification of the lens 152 is 1x, it is composed of light emitting elements (in the case of a single color) with a pitch of 0.05 mm or less.
  • the projected projection device 153 may be used. This makes it possible to display an auxiliary image in which the user can recognize a gap of 0.1 mm.
  • the projection device 153 not only fills the field of view of the eyepiece 101 having a field of view of ⁇ 22, but also projects an auxiliary image in a range having a field of view of ⁇ 23 or more, which is one size larger than that.
  • a projection device 153 having an effective light emitting region corresponding to a range of a field of view of ⁇ 23 or more is used.
  • sperms in the peripheral portion of the visual field that have entered the visual field from outside the visual field of the eyepiece 101 are also included in the auxiliary image. Therefore, it is possible to recognize good sperms from all sperms in the visual field including the peripheral portion of the visual field of the eyepiece 101.
  • the effective pixel region of the photographing device 143 also needs to correspond to the range of the number of visual fields ⁇ 23 or more in the eyepiece.
  • the intermediate scaling unit 160 is arranged between the objective lens 102 and the imaging lens 103.
  • the intermediate magnification unit 160 includes a plurality of lenses (lens 161 and lens 162, lens 163), and is formed on the image plane by switching the lens arranged on the optical path between them. Change the magnification of the optical image to be made. By using the intermediate magnification unit 160, the magnification of the optical image can be changed without switching the objective lens 102 located near the sample.
  • the DIC prism 105 and the analyzer 106 are arranged between the objective lens 102 and the imaging lens 103.
  • the DIC prism 105 is used for DIC observation.
  • Analyzer 106 is used in PO and DIC observations.
  • a polarizer 123 and an optical element 127 are arranged on the illumination optical path as a modulation element (hereinafter referred to as a first modulation element) that modulates the illumination light applied to the sample.
  • a modulator 104 is arranged on the observation optical path as a modulation element (hereinafter, referred to as a second modulation element) that modulates the transmitted light.
  • the polarizer 123 is arranged on the illumination optical path as the first modulation element
  • the analyzer 106 is arranged on the observation optical path as the second modulation element.
  • the polarizer 123 and the DIC prism 125 are arranged on the illumination optical path as the first modulation element, and the analyzer 106 and the DIC prism 105 are arranged on the observation optical path as the second modulation element. .. Thereby, it is possible to visualize the unstained sample, and for example, sperm can be sorted.
  • the microscope controller 10 is a device that controls the microscope 100.
  • the microscope controller 10 is connected to the image processing device 200, the input device 50, and the microscope 100, and controls the microscope 100 in response to a command from the image processing device 200 or the input device 50.
  • the display device 30 is, for example, a display device such as a liquid crystal display, a plasma display, an organic EL display, a CRT display, or an LED matrix panel.
  • the input device 40 includes a handle 41 and a handle 42. By operating the handle 41 and the handle 42, the operation of the pipette 43 and the micromanipulator (not shown) for moving the pipette 44 is controlled. Pipette 43 and pipette 44 are used to manipulate the sample in microinsemination operations, including sperm sorting.
  • the pipette 43 is, for example, a holding pipette
  • the pipette 44 is, for example, an injection pipette.
  • the input device 50 is a hand switch device for changing the settings related to the microscope method and the observation magnification of the microscope 100. As shown in FIG. 3, the input device 50 has, for example, six buttons (buttons 51 to 56), and the user can quickly switch the settings of the microscope 100 simply by pressing these buttons. Can be done.
  • the setting of the microscope 100 is switched to the setting of BF observation (hereinafter referred to as BF4 ⁇ observation) with an observation magnification of 4 times.
  • the setting of the microscope 100 is switched to the setting of MC observation (hereinafter referred to as MC10 ⁇ observation) at an observation magnification of 10 times.
  • the setting of the microscope 100 is switched to the setting of MC observation (hereinafter referred to as MC20 ⁇ observation) at an observation magnification of 20 times.
  • the setting of the microscope 100 is switched to the setting of MC observation (hereinafter referred to as MC40 ⁇ observation) having an observation magnification of 40 times.
  • the setting of the microscope 100 is switched to the setting of PO observation (hereinafter referred to as PO20 ⁇ observation) at an observation magnification of 20 times.
  • the setting of the microscope 100 is switched to the setting of DIC observation (hereinafter referred to as DIC 60 ⁇ observation) having an observation magnification of 60 times.
  • the input device 60 is a keyboard.
  • the input device 70 is a mouse.
  • the input device 60 and the input device 70 are each connected to the image processing device 200.
  • the microscope system 1 may include other input devices (not shown) such as a touch panel, a voice input device, and a foot pedal.
  • the identification device 80 is a device that acquires identification information added to the sample.
  • the term "added to the sample” includes, for example, the case where the identification information is attached to the container for accommodating the sample.
  • the identification information is information that identifies the sample, and more specifically, for example, information that identifies the patient who provided the sample.
  • the identification device 80 may be, for example, a barcode reader, an RFID (registered trademark) reader, a QR code (registered trademark) reader, or the like.
  • the image processing device 200 generates an auxiliary image based on the captured image acquired by the photographing device 143.
  • the generated auxiliary image is output to the projection device 153 of the microscope 100, directly or via the microscope controller 10.
  • the image processing device 200 is connected to the microscope 100, the microscope controller 10, the display device 30, the input device 60, the input device 70, and the identification device 80.
  • the image processing device 200 is also connected to the database server 20.
  • the image processing device 200 includes an image analysis unit 210, an image generation unit 220, and a storage unit 230 as functional components related to the generation of an auxiliary image including auxiliary information that contributes to sperm selection. Twice
  • the image analysis unit 210 performs image analysis including object detection for sperm for a captured image and recommendation degree estimation for estimating the recommendation degree of sperm detected by the object detection. Specifically, the image analysis unit 210 performs object detection and recommendation degree estimation using, for example, a learned model stored in the storage unit 230. Object detection and recommendation degree estimation may be performed using two or more trained models, or may be performed using a single trained algorithm. The algorithm of the trained model is not particularly limited, but for example, a deep learning model such as SSD, YOLO, or FasterR-CNN may be used.
  • the image generation unit 220 generates an auxiliary image including auxiliary information that contributes to sperm selection based on the result of the image analysis of the image analysis unit 210.
  • the auxiliary information includes information on the sperm recommendation level (hereinafter referred to as recommendation level information) generated based on the result of the recommendation level estimation. That is, the image generation unit 220 generates an auxiliary image including at least recommendation degree information.
  • the recommendation level information may be numerical information such as DFI described later, or symbol information indicating a grade such as DFI rank and MM rank described later. Further, the recommendation degree information may be a combination of one or more numerical information and one or more symbol information.
  • the recommendation degree information may include at least one or more numerical information and at least one or more symbolic information.
  • the auxiliary information may further include information regarding the sperm position (hereinafter referred to as position information) generated based on the result of object detection. That is, the image generation unit 220 may generate an auxiliary image including position information in addition to the recommendation degree information.
  • the recommendation degree estimation performed by the image analysis unit 210 may include an integrity estimation for estimating the DNA integrity of the sperm detected by the object detection.
  • the integrity information may include integrity information regarding sperm DNA integrity generated based on the results of integrity estimation. It is desirable that the integrity information includes at least one of DFI and a rank corresponding to DFI, which will be described later.
  • the DFI or DFI rank preferably includes at least two of the latest information, average information, and maximum information.
  • the recommendation level estimation performed by the image analysis unit 210 may include quality estimation for estimating the quality of at least one of sperm morphology and motility detected by object detection. In that case, the recommendation level information may include quality information regarding at least one of the morphological quality of sperm and the quality of sperm motility generated based on the result of quality estimation.
  • Non-Patent Document 1 Wang, Y. et al., Prediction of DNA Integrity from Morphological Parameters Using a Single-Sperm DNA Fragmentation Index 20 SCDNA, VSC. There is a correlation between completeness and sperm morphology.
  • the completeness estimation performed by the image analysis unit 210 is performed using a trained model in which the DNA completeness of sperm with respect to the sperm morphology extracted from the image is learned by utilizing this fact.
  • the recommendation level information included in the auxiliary image may have an aspect according to the recommendation level estimated by the recommendation level estimation. More specifically, for example, it may have a color corresponding to the recommendation level estimated by the recommendation level estimation. Further, the position information included in the auxiliary image may also have an aspect corresponding to the recommendation level estimated by the recommendation level estimation, as in the recommendation level information, and may be, for example, the recommendation level estimated by the recommendation level estimation. It may have a corresponding color.
  • the storage unit 230 stores the trained model used in the image analysis performed by the image generation unit 220.
  • the storage unit 230 stores a learned model for detecting an object targeting sperm, and the image generation unit 220 performs object detection using the learned model stored in the storage unit 230. Further, for example, when the image generation unit 220 generates the completeness information, the storage unit 230 stores a learned model in which the DNA completeness of the sperm with respect to the sperm morphology is learned.
  • the image generation unit 220 performs integrity estimation using the trained model stored in the storage unit 230.
  • the storage unit 230 stores a learned model that learns the quality of sperm with respect to at least one of sperm morphology and sperm motility. ing.
  • the image generation unit 220 performs quality estimation using the trained model stored in the storage unit 230.
  • the object detection, completeness estimation, and quality estimation may be performed using a single trained model stored in the storage unit 230, and two or more trained models stored in the storage unit 230 may be used. It may be done using a model.
  • the image processing device 200 may be a general-purpose computer or a dedicated computer.
  • the image processing device 200 is not particularly limited to this configuration, but may have, for example, a physical configuration as shown in FIG.
  • the image processing device 200 may include a processor 201, a storage device 202, an input interface (I / F) 203, an output interface (I / F) 204, and a communication device 205. , They may be connected to each other by bus 206.
  • the processor 201 may include hardware, which may include, for example, at least one of a circuit for processing digital signals and a circuit for processing analog signals.
  • Processor 201 may include, for example, one or more circuit devices (eg, ICs) or one or more circuit elements (eg, resistors, capacitors) on a circuit board.
  • the processor 201 may be a CPU (central processing unit). Further, various types of processors including GPU (Graphics processing unit) and DSP (Digital Signal Processor) may be used for the processor 201.
  • the processor 201 may be a hardware circuit having an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).
  • the processor 201 can include an amplifier circuit, a filter circuit, and the like for processing an analog signal.
  • the processor 201 functions as the image analysis unit 210 and the image generation unit 220 described above by executing the program stored in the storage device 202.
  • the storage device 202 may include a memory and / or other storage device.
  • the memory may be, for example, a random access memory (RAM).
  • the memory may be a semiconductor memory such as SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory).
  • the storage device 202 may include, for example, a register, a magnetic storage device such as a hard disk device, an optical storage device such as an optical disk device, an internal or external hard disk drive, a solid state storage device, a CD-ROM, a DVD, or other optical or magnetic. It may be a disk storage device or another storage device.
  • the storage device 202 stores programs, trained models, and other data executed by the processor 201, and functions as the storage unit 230 described above.
  • the storage device 202 is an example of a non-temporary computer-readable storage medium.
  • the input I / F 203 is connected to an input device operated by a user of the microscope system 1 (for example, an embryo culture technician), receives an operation signal corresponding to the operation of the input device, and outputs the operation signal to the processor 201.
  • a user of the microscope system 1 for example, an embryo culture technician
  • the output I / F 204 is connected to the display device 30.
  • the output I / F 204 may be further connected to an audio output device such as a speaker that outputs audio, a light emitting device such as a lamp or light that outputs light, or a vibrating device such as a vibrator that outputs vibration, which is not shown. ..
  • the communication device 205 is a device that exchanges data with the microscope 100 and other devices.
  • the communication device 205 may be a communication device that exchanges data by wire, or may be a communication device that exchanges data wirelessly.
  • the program or learned model stored in the storage device 202 may be acquired by the communication device 205 from another device via the Internet.
  • FIG. 5 is a flowchart showing an example of the procedure for creating a learning data set.
  • FIG. 6 is a flowchart showing an example of the learning process.
  • the procedure for constructing the trained model stored in the storage unit 230 will be described with reference to FIGS. 5 and 6 by taking as an example the case of constructing the trained model for estimating the DNA integrity.
  • the preparation procedure shown in FIG. 5 may be performed by the microscope system 1 or may be performed by another device or system. Further, the learning process shown in FIG. 6 may be performed by a computer different from the image processing device 200, and the image processing device 200 may acquire the completed trained model via a network or a recording medium. .. In the following, a case where the microscope system 1 performs the creation procedure shown in FIG. 5 and the image processing apparatus 200 performs the learning process shown in FIG. 6 will be described as an example.
  • the microscope system 1 first captures a moving image of sperm (step S1).
  • the operator observes the sperm suspension using the microscope system 1, for example, at a magnification of 40, and picks up the sperms contained in the sperm suspension one by one.
  • the worker is, for example, an embryo culture person.
  • the microscope system 1 captures a moving image of each sperm picked up by, for example, the imaging unit 140.
  • the magnification at this time is, for example, 40 times, 60 times, and the like.
  • the operator rotates the sperm using the tip of a pipette or the like. As a result, the microscope system 1 photographs the entire circumference of the sperm head and neck.
  • the microscope system 1 stains the sperm whose moving image has been taken and then takes a picture again (step S2).
  • the operator fixes each of the spermatozoa imaged in step S1 at a position in the assigned container, and then stains the sperm in the container with a dye that labels DNA damage.
  • the operator observes the sperm in the stained container by a fluorescence observation method, and further, photographs the sperm with the microscope system 1 to acquire a stained image of the sperm.
  • the microscope system 1 calculates the DNA fragmentation index (DFI: DNA fragmentation index) of each sperm from the stained image acquired in step S2 (step S3).
  • DFI is an index of DNA integrity, and the smaller the value, the better.
  • the specific calculation method of DFI is not particularly limited, but for example, Non-Patent Document 2 (McCallum, C. et al., Deep learning-based selection of human sperm with high DNA integrity, Communications 2, Communications Biol). As described in 2019)), it may be calculated using an AO (Acridine Orange) test.
  • the microscope system 1 labels the unstained image of sperm extracted from the moving image acquired in step S1 with the DFI calculated in step S3 (step S4).
  • the microscope system 1 assigns DFI to each frame image (unstained image) of the moving image, so that the unstained image of the sperm and the DFI of the sperm are stored in association with each other.
  • the association between the moving image taken in step S1 and the DFI calculated in step S3 can be performed by managing the ID assigned to the moving image and the position in the container.
  • the learning process is performed by using a large number of unstained images labeled according to the procedure of FIG. 5 as a learning data set.
  • the image processing apparatus 200 first acquires the created training data set (step S11).
  • the image processing apparatus 200 trains the sperm into a model that estimates the integrity of the DNA (step S12).
  • the image processing apparatus 200 trains, verifies, and tests the model using the training data set.
  • the learning data set acquired in step S11 may be divided into training data / verification data and test data in step S12. In that case, it is desirable that the training and validation data be used for both training and validation using cross-validation.
  • the training data is data used for training the model
  • the verification data is data used for verifying the model.
  • the test data is data used for testing the model.
  • the image processing device 200 repeats the above processing until the model clears the test, and when the processing is cleared, that is, when learning is completed (step S13YES), the processing of FIG. 6 is completed.
  • the obtained trained model is stored in the storage unit 230 of the image processing device 200.
  • this trained model has a label indicating DNA completeness calculated from a non-stained image obtained by photographing the sperm before staining and a stained image obtained by photographing the sperm after staining attached to the unstained image.
  • FIG. 7 is a flowchart showing another example of the procedure for creating the learning data set.
  • FIG. 8 is a diagram showing an example of a screen for creating a learning data set.
  • the learning of a model for estimating DNA completeness for sperm has been described as an example, but quality estimation for estimating the morphological quality of sperm and the quality of sperm motility have been described.
  • the training data set may be created by the creation procedure shown in FIG. 7 instead of the creation procedure shown in FIG.
  • the procedure for constructing the trained model stored in the storage unit 230 when constructing a trained model that estimates the quality of at least one of sperm morphology and motility. Will be described as an example.
  • the microscope system 1 first captures a moving image or a still image of sperm (step S21).
  • the operator observes the sperm suspension using the microscope system 1 at a magnification of, for example, 20 times, and photographs the sperm in the sperm suspension as a moving image or a still image with the photographing unit 140.
  • the microscope system 1 cuts out an image of the sperm portion from the captured image (moving image or still image) and displays it side by side (step S22).
  • the microscope system 1 reads out the moving image or still image taken in step S21, cuts out an image of the sperm portion from the moving image or still image, and displays them side by side on the display device 30 as shown in FIG.
  • the operator evaluates the morphological quality of sperm and the quality of sperm motility from the images of sperm displayed side by side. Therefore, it is desirable that the worker is a skilled embryo cultivator with a high fertilization success rate.
  • the image obtained by cutting out the sperm portion will be referred to as a reference image.
  • the microscope system 1 labels each reference image with a quality evaluation by the embryo cultivator (step S23).
  • the microscope system 1 assigns a quality rank to each sperm reference image (unstained image) to associate and store the unstained sperm image and the sperm quality.
  • the image (T1, T10, T14, %) Clicked with the button B1 on the window W1 selected is saved in association with the label of rank A indicating the quality. .. Further, the image (T2, T3, T6, T8, T9, T11, T15 %) Clicked with the button B2 selected is saved in association with the label of rank B indicating the quality. Further, the image (T4, T5, T13, T16 %) Clicked with the button B3 selected is saved in association with the label of rank C indicating the quality. Further, the image (T7, T12 ...) Clicked with the button B4 selected is saved in association with the label of rank D indicating the quality. Note that rank A is assigned to the image evaluated with the highest quality, and rank D is assigned to the image evaluated with the lowest quality. That is, ranks A, B, C, and D indicate that the quality deteriorates in this order.
  • the learning process shown in FIG. 6 may be performed by using a large number of unstained images labeled in the procedure of FIG. 7 as a learning data set.
  • the obtained trained model is stored in the storage unit 230 of the image processing device 200.
  • this trained model contains a data set including a reference image obtained by photographing the sperm and a label attached to the reference image indicating the quality of the sperm evaluated by the embryo culture specialist based on the reference image. It is a trained model whose parameters are adjusted by supervised learning used as training data.
  • FIG. 9 is a flowchart showing an example of sperm sorting support processing.
  • FIG. 10 is a flowchart showing an example of the auxiliary image generation process.
  • FIG. 11 is a diagram for explaining image processing on a captured image.
  • FIG. 12 is a diagram for explaining an image projected on the image plane.
  • the microscope system 1 first projects an optical image onto the image plane (step S31).
  • the optical system including the objective lens 102 and the imaging lens 103 is placed on the optical path of the microscope 100, for example, based on the light from the sperm suspension which is a sample placed on the stage of the microscope 100.
  • An optical image O1 of a sperm suspension is formed as shown in.
  • the microscope system 1 acquires a captured image at the same time as step S31 (step S32).
  • the photographing device 143 acquires a photographed image of the sperm suspension based on the light from the sperm suspension, and outputs the photographed image to the image processing device 200.
  • the microscope system 1 performs the auxiliary image generation process shown in FIG. 10 (step S33).
  • the image processing device 200 first detects sperm from the captured image (step S41).
  • the image analysis unit 210 detects an object targeting sperm by inputting the captured image M as an input image into the trained model, for example, as shown in FIG.
  • the image analysis unit 210 may detect sperms by classifying them according to the quality of sperm morphology or motility.
  • FIG. 11 shows how sperms detected by object detection are classified from rank A to rank C. In this case, it may be considered that the object detection in step S41 also serves as the recommendation degree estimation in step S42, which will be described later.
  • the image processing device 200 estimates the degree of recommendation from the image of the sperm detected in step S41 (step S42).
  • the image analysis unit 210 estimates the sperm recommendation level by inputting the detected sperm images (images M1 to M6) into the trained model as input images. do.
  • FIG. 11 shows an example in which DFI was calculated as a recommendation level.
  • the image processing device 200 After that, the image processing device 200 generates an auxiliary image including sperm recommendation degree information (step S43).
  • the image generation unit 220 generates recommendation degree information regarding the sperm recommendation degree based on the result of the recommendation degree estimation in step S42, and for example, an auxiliary image including the recommendation degree information as shown in FIG. Generate A1.
  • the microscope system 1 When the auxiliary image generation process is completed, the microscope system 1 superimposes the auxiliary image on the optical image (step S34), and ends the process shown in FIG.
  • the projection device 153 projects the auxiliary image A1 generated in step S33 on the image plane IP on which the optical image O1 is formed, whereby the auxiliary image A1 is projected on the optical image O1 as shown in FIG. Are superimposed.
  • the embryo culture person who is the user of the microscope system 1 looks into the eyepiece 101 and sees the optical image O1 containing sperm and the auxiliary image A1 containing auxiliary information contributing to sperm selection as shown in FIG. You can sort sperm while looking at the image on which. Therefore, according to the microscope system 1, it is possible to support the sperm sorting work by the embryo cultivator.
  • the auxiliary information includes sperm recommendation level information, it becomes easy to judge good sperm, so that sperm sorting work can be performed quickly in a short time.
  • the recommendation level information includes the completeness information regarding the DNA completeness of the sperm, the sperm can be selected in consideration of the presence or absence of DNA damage which is difficult to judge from the optical image O1 alone. Therefore, according to the microscope system 1, the embryo culture technician can select good sperms with higher reliability than the conventional judgment based on morphology and motility.
  • FIG. 13 is a flowchart showing an example of the ICSI procedure.
  • FIG. 14 is a diagram illustrating the configuration of a drop formed as a sample 300 in a petri dish 310.
  • FIG. 15 is a flowchart showing an example of a sperm sorting procedure.
  • FIG. 16 is a flowchart showing another example of the auxiliary image generation process.
  • FIG. 17 is a diagram for explaining a configuration example of the auxiliary image setting window.
  • 18 to 24 are diagrams illustrating the relationship between the auxiliary image setting and the image seen from the eyepiece 101, respectively.
  • FIGS. 13 to 24 specific utilization of the sperm sorting support method performed by the microscope system 1 in ICSI will be described with reference to FIGS. 13 to 24.
  • the user prepares a sample (step S51).
  • a sample 300 containing a plurality of drops in the petri dish 310 and arranges the sample 300 on the stage 111.
  • Drop 301 is a cleaning drop and is used for cleaning pipettes.
  • the drop 302 is a sperm floating drop, for example, a sperm suspension added dropwise to a PVP solution.
  • the drop 303 is an egg manipulation drop, for example, an egg is put in an m-HTF solution.
  • the m-HTF solution is a Hepps-containing HTF solution to which 10% serum is added. These drops are covered with mineral oil.
  • the user sets up the microscope system 1 (step S52).
  • the user presses the button 51 of the input device 50 to switch the setting of the microscope system 1 to BF4 ⁇ observation.
  • the input device 40 is operated to adjust the positions of the pipette 43 and the pipette 44 to focus on the pipette 43 and the pipette 44.
  • the stage 111 is moved to wash the pipette 43 and the pipette 44 with the drop 301 (washing drop).
  • the user confirms the growth state of the egg (egg cell) in the drop 303 (drop for egg operation) (step S53).
  • the user presses the button 53 of the input device 50 to switch the setting of the microscope system 1 to MC20 ⁇ observation.
  • Eggs are selected by observing the morphology of the eggs with MC20 ⁇ observation.
  • the button 55 of the input device 50 may be pressed to switch the setting of the microscope system 1 to PO20 ⁇ observation.
  • the maturity of the egg may be determined, and the egg may be further selected.
  • step S54 the user selects the sperm according to the procedure shown in FIG. 15 (step S54).
  • the stage 111 is moved to move the observation position to the drop 302 (sperm floating drop), and the sperm is focused by MC20 ⁇ observation (step S61).
  • step S62 the user selects good sperms suitable for fertilization by MC20 ⁇ observation.
  • an embryo culture person judges the quality of sperm based on the morphology and motility of sperm observed in an optical image, and selects good sperm based on the judgment.
  • the criteria for determining good sperm are not always clear. For this reason, sperms are often selected based on the experience and intuition of the embryo cultivator, and as a result, the judgment differs depending on the embryo cultivator. This was the cause of the difference in fertilization success rate among embryo culture specialists.
  • the microscope system 1 supports the sperm sorting work of the embryo cultivator by superimposing an auxiliary image including auxiliary information contributing to sperm sorting on the optical image.
  • step S62 the microscope system 1 performs the auxiliary image generation process shown in FIG. 16, and as a result, the generated auxiliary image is projected onto the image plane.
  • the image processing device 200 detects sperm from the captured image of the drop 302 (sperm floating drop) captured by the imaging device 143, and generates position information regarding the position of the detected sperm (step S71).
  • object detection for sperm is performed using the trained model.
  • the image processing device 200 estimates the degree of recommendation from the sperm image detected in the captured image (step S72).
  • recommendation degree estimation such as completeness estimation and quality estimation is performed using the trained model.
  • the DFI is estimated for each sperm by the completeness estimation, and the sperm morphology and motility rank (hereinafter referred to as MM rank) are determined for each sperm by the quality estimation.
  • MM rank sperm morphology and motility rank
  • the image processing device 200 generates recommendation degree information according to the setting by the user (step S73).
  • the image processing device 200 is set in step S72 according to the settings previously made by the user on the window W2 shown in FIG. 17 displayed on the display device 30 by using the input device 60 and the input device 70.
  • Recommendation level information is generated from the obtained estimation results.
  • the window W2 is an auxiliary image setting window for setting recommendation level information included in the auxiliary image, and includes an area R1 for setting a display format and an area R2 for setting display items.
  • a list box L1 for selecting a DFI value As shown in FIG. 17, in the area R1, as a display format of the recommendation level, a list box L1 for selecting a DFI value, a list box L2 for selecting a DFI rank, and an MM rank are selected.
  • the list box L3 is included.
  • the DFI value is the DFI itself, and the DFI rank is obtained by dividing the DFI into a predetermined range and assigning a rank to each range.
  • the DFI value can be included as the recommendation level information in the auxiliary image.
  • the DFI rank can be included as the recommendation degree information in the auxiliary image.
  • the MM rank can be included as the recommendation degree information in the auxiliary image. It is also possible to select a plurality of list boxes L3 from the list box L1, and in that case, information in a plurality of display formats can be included as recommendation degree information.
  • a list box L6 for selecting an average and a list box L7 for selecting a circumference average are included.
  • Raw is DFI estimated from the latest captured image.
  • the maximum value is the maximum value of DFI estimated within the period set by the slider SL2.
  • the threshold value set by the slider SL1 is the upper limit of the maximum value included in the recommendation degree information.
  • the time average is the average value of DFI estimated within the period set by the slider SL4.
  • the threshold value set by the slider SL3 is the upper limit of the average value included in the recommendation degree information.
  • the image processing device 200 When the recommendation level information is generated, the image processing device 200 generates an auxiliary image including the recommendation level information and the position information in a mode corresponding to the recommendation level (step S74).
  • the image processing apparatus 200 may generate, for example, an auxiliary image including recommendation degree information and position information in colors corresponding to the recommendation degree. More specifically, if the position information is generated as a bounding box surrounding the sperm, the thickness and line type of the lines constituting the bounding box included in the auxiliary image may be different depending on the degree of recommendation. ..
  • the microscope system 1 performs the auxiliary image generation process shown in FIG. 16 and projects the generated auxiliary image onto the image plane, so that in step S62, the user can use the optical image O1 as shown in FIG. Sperm can be selected while looking at the superimposed auxiliary image A11.
  • FIG. 18 shows an example of an image projected in the field of view of the eyepiece 101 when the DFI value is selected as the display format and the maximum value is selected as the display item.
  • the displayed DFI is the maximum value for the last 2 seconds and is limited to 0.05 or less by the threshold setting.
  • the sperm selection in step S62 selects good sperms from a large number of sperms contained in the sperm suspension, and it is expected that many sperms are also contained in the visual field.
  • the threshold setting by the slider SL1 is effective.
  • the threshold value By setting the threshold value, only the recommendation level information of sperms having DFI below the threshold value (here, the maximum value of DFI) can be projected, so that the amount of information superimposed on the optical image O1 is appropriately limited. be able to.
  • innumerable sperm recommendation level information is projected in the field of view, and the visibility of the optical image (sperm) is reduced, and there are too many pieces of information to make it difficult to understand the necessary information. It can be avoided.
  • by limiting the amount of information using a threshold value it becomes possible for the user to concentrate on only the sperms with high evaluation and observe them.
  • step S62 since the sperm freely move around in the sperm suspension, the direction and shape of the sperm in the captured image change from moment to moment. Therefore, it is expected that the estimated DFI will change from moment to moment even for the same sperm.
  • the period setting by the slider SL2 is effective. By setting the period, it is possible to extract or reconstruct necessary information from a large number of estimation results obtained within the set period and project it. This makes it possible to provide the user with the recommendation level of sperm without being influenced by the temporary estimation result at a specific time.
  • the position information is formed as a bounding box surrounding the sperm head. As shown in FIG. 18, location information may be formed to identify a major portion of sperm rather than the entire sperm. Such display of position information is also particularly suitable in step S62, where a large number of sperms are expected to be detected.
  • FIG. 18 shows an example in which one recommendation degree information is superimposed on one sperm
  • a plurality of recommendation degree information may be superimposed.
  • FIG. 19 shows an example of an image projected in the field of view of the eyepiece 101 when the DFI value is selected as the display format and raw and the maximum value are selected as the display items. ..
  • the displayed DFI is the latest value (raw) and the maximum value for the last 2 seconds, and is limited to 0.05 or less by the threshold setting.
  • the auxiliary image A13 including a plurality of information for each sperm may be superimposed on the optical image O1.
  • the user can hit the sperm to be observed in the DFI rank and observe the sperm while checking the DFI value, so that the sperm can be efficiently selected.
  • FIG. 20 shows an example of an image projected in the field of view of the eyepiece 101 when the DFI value and the DFI rank are selected as the display format and the time average is selected as the display item. There is.
  • the DFI displayed together with the DFI rank is the time average value for the last 1 second, and is limited to 0.05 or less by the threshold setting.
  • FIGS. 18 to 20 an example in which information on DNA completeness is superimposed as recommendation degree information is shown, but as shown in FIG. 21, the MM rank based on the sperm morphology and motility that has been conventionally performed is shown.
  • the auxiliary image A14 including the above as the recommendation degree information may be superimposed on the optical image O1.
  • the auxiliary image A15 including both the MM rank and the information on the DNA integrity (here, the DFI value) as the recommendation degree information may be superimposed on the optical image O1.
  • the DFI value and the MM rank are selected as the display format and the time average is selected as the display item.
  • the DFI displayed together with the MM rank is the time average value for the last 1 second, and is limited to 0.05 or less by the threshold setting.
  • step S62 by superimposing the auxiliary image on the optical image O1, the user can easily recognize the sperm of particular interest by the auxiliary image, so that good sperm can be selected. Therefore, the number of sperms confirmed by the user can be significantly reduced. Therefore, the sperm sorting work becomes easy and the sperm sorting can be performed in a short time, so that the burden of the sorting work is greatly reduced.
  • step S62 When good sperms are selected in step S62, the user injures the tail of the good sperms by RC20 ⁇ observation and immobilizes the good sperms (step S63). Here, the user immobilizes the good sperm by rubbing the tail of the good sperm against the bottom surface of the petri dish 310 with a pipette.
  • the user observes the morphology of the immobilized good sperm in more detail and further selects the good sperm (step S64).
  • the user for example, presses the button 54 of the input device 50 to switch the setting of the microscope system 1 to MC40 ⁇ observation. After that, the user selects better sperm by MC40 ⁇ observation.
  • the microscope system 1 performs the auxiliary image generation process shown in FIG. 16 as in step S64, and as a result, the generated auxiliary image is projected onto the image plane to perform the sperm separation work of the embryo incubator. Support.
  • the microscope system 1 performs the auxiliary image generation process shown in FIG. 16 and projects the generated auxiliary image onto the image plane, so that in step S64, the user can use the optical image O2 as shown in FIG. Sperm can be selected while looking at the superimposed auxiliary image A21.
  • FIG. 23 shows an example of an image projected in the field of view of the eyepiece 101 when the DFI value and the DFI rank are selected as the display format and raw and the circumferential average are selected as the display items. Has been done.
  • the sperm selection in step S64 is performed in a state where a specific sperm is immobilized, and it is expected that a single sperm is included in the visual field.
  • the user can observe the sperm from any direction by adjusting the direction of the sperm with the tip of the pipette, and by rolling the sperm with the tip of the pipette, the head of the sperm and the sperm can be observed. The entire circumference of the neck can be photographed. Therefore, as shown in FIG. 23, the calculation setting of the peripheral average by selecting the list box L7 is effective. By calculating the perimeter average, it is possible to provide the user with highly reliable DNA integrity regardless of the imaging direction.
  • step S64 the image processing apparatus 200 calculates the sperm rotation amount, and when the sperm rotation amount is insufficient (for example, the rotation amount is less than 360 °), as shown in FIG. 24, the sperm rotation amount is insufficient.
  • the auxiliary image A22 including the information to be notified may be generated, and the projection device 153 may superimpose the auxiliary image A22 on the optical image O2. That is, the image analysis performed by the image analysis unit 210 may include a process of estimating the recommended operation in addition to the object detection and the recommendation degree estimation, and the image analysis unit 210 is selecting sperm based on the captured image.
  • the notification of the recommended operation may be notified by voice, light, vibration, or the like instead of or in addition to the notification by using the auxiliary image so as to be visible. That is, the notification device may be a projection device 153, or may be a speaker, a lamp, a vibrator, or the like (not shown).
  • step S65 When the selection of good sperm by MC40 ⁇ observation is completed, the user further observes the head of the good sperm in detail and further selects the good sperm according to the size of the blank present in the head (step S65). ).
  • the user selects good spermatozoa with small blanks as better spermatozoa by DIC60 ⁇ observation.
  • step S65 may be performed under MC40 ⁇ observation. In this case, the user selects good sperms by recognizing the bright spots generated on the head as blanks.
  • the user takes the selected good sperm into the pipette 44, which is an injection pipette, moves the observation position to the drop 303 (drop for egg manipulation) (step S66), and performs a series of sperm selection shown in FIG. End the procedure of.
  • the user confirms the position of the mitotic spindle in preparation for good sperm injection (step S55).
  • the user observes the egg selected in step S53 existing in the drop 303 and confirms the position of the spindle of the egg.
  • the user presses, for example, the button 55 of the input device 50 to switch the setting of the microscope system 1 to PO20 ⁇ observation.
  • the user changes the orientation of the spindle by operating the pipette 43, which is a holding pipette, so that the spindle of the egg visualized by PO20 ⁇ observation is located in the direction of 12 o'clock or 6 o'clock. This is to avoid damaging the mitotic spindle by the pipette thrust into the egg from the 3 o'clock or 9 o'clock direction in step S56, which will be described later.
  • the user injects sperm into the egg (step S56) and ends ICSI.
  • the user presses the button 53 of the input device 50 to switch the setting of the microscope system 1 to MC20 ⁇ observation.
  • the user fixes the egg whose orientation was adjusted in step S55 with the pipette 43, which is a holding pipette, and pierces the pipette 44, which is an injection pipette, by observing MC20 ⁇ . Then, good sperm is injected into the egg from the pipette 44.
  • the user When the series of ICSI procedures shown in FIG. 13 is completed, the user returns the sperm-injected egg to the incubator and incubates it. Further, the user may operate the processing device 200 by using the input device 60 and the input device 70, and save the information obtained by ICSI in the database server 20. For example, image data of sperm-injected eggs, image data of selected good sperms, ICSI working hours, sperm and egg patient information (maternal clinical data, test results of sperm-containing semen, etc.), Data of sperm and egg culture solutions (eg, type, concentration, PH, etc.) may be associated and stored in the database server 20.
  • an auxiliary image contributing to sperm separation is projected on the image plane in ICSI.
  • the size of sperm is about 60 ⁇ m, and an objective lens of at least 20 times is used to distinguish good sperm.
  • the number of fields of view of an inverted microscope is about 22, so the actual field of view is about ⁇ 1 mm.
  • the work of selecting sperms that move around freely in this region of the actual field of view ⁇ 1 mm is a very difficult work.
  • sperms presumed to be good sperms have high motility, and ICSI work needs to be performed in a short time. Therefore, in sperm sorting work, the morphology of sperms that move relatively quickly is quickly observed.
  • FIG. 25 is a diagram showing the configuration of the microscope 400.
  • the projection device 153 arranged on the optical path branched from the optical path between the objective lens 102 and the eyepiece lens 101 is illustrated, but the device for superimposing the auxiliary image is an objective as shown in FIG. It may be arranged on the optical path between the lens 102 and the eyepiece 101.
  • the microscope 400 differs from the microscope 100 in that instead of the projection unit 150, a display device 401 is included on an image plane on which an optical image is formed.
  • the display device 401 is a transmissive image display device, and is an example of a superimposition device that superimposes an auxiliary image on an optical image by displaying the auxiliary image directly on the image plane. Even when the microscope 400 is used, it is possible to support the sperm sorting of the embryo incubator as in the case of using the microscope 100.
  • FIG. 26 is a diagram showing the configuration of the microscope system 2.
  • the imaging device 143 and the projection device 153 are provided in the microscope 100, and the image processing device 200 independent of the microscope 100 exchanges data with the imaging device 143 and the projection device 153 to obtain an optical image of the auxiliary image.
  • the photographing apparatus, the projection apparatus, and the image processing apparatus may be integrally configured as shown in FIG.
  • the projection unit 600 mounted between the microscope main body 510 of the microscope 500 and the lens barrel 520 includes an imaging device 143, a projection device 153, and an image processing device (image analysis unit 210, image generation unit 220, A storage unit 230) may be provided.
  • an image processing device image analysis unit 210, image generation unit 220, A storage unit 230
  • the DFI value, DFI rank, and MM rank are illustrated as the recommendation level information included in the auxiliary image, but the recommendation level information is not limited to these.
  • the recommendation level information may be a sperm quality determination result.
  • object detection and recommendation degree estimation may be performed using a trained model that has learned the quality of sperm.
  • FIG. 27 is a diagram showing still another example of the relationship between the auxiliary image setting and the image seen from the eyepiece lens 101. As shown in FIG. 27, by selecting the list box L8 provided in the area R1 for selecting the recommended display format, the auxiliary image A16 including the quality determination result may be superimposed on the optical image O1. ..
  • the auxiliary image A16 may include, for example, a bounding box A16a that encloses the sperm that is most frequently judged as "good” by the quality judgment performed for each frame.
  • the sperm having the largest value calculated by multiplying the number of times determined to be "good” by the value of VSL (linear velocity) may be surrounded by the bounding box A16a.
  • VSL linear velocity
  • the sperm movement locus A16b surrounded by the bounding box A16a may be included in the auxiliary image A16.
  • FIG. 28 is a diagram showing still another example of the relationship between the auxiliary image setting and the image seen from the eyepiece lens 101. 29 and 30 are diagrams for explaining the merits of the bar display. As shown in FIG. 28, by selecting the list box L8 provided in the area R1 for selecting the recommended display format, the auxiliary image A17 including the bar indicating the pass / fail judgment result is superimposed on the optical image O3. You may. That is, the recommendation level may be presented to the user by displaying a bar.
  • the three bars showing the pass / fail judgment results shown in FIG. 28 show the probabilities of each classification of "good”, “bad”, and “undecidable” output from the trained model by length. be.
  • FIG. 31 is a diagram for explaining a desirable change example regarding the display format of the auxiliary image.
  • the simple display using the bounding box and the bar display have merits, and by using them properly according to the situation, it is possible to efficiently and surely select good sperm.
  • the quality of each sperm is determined on the optical image O1 as shown in FIG.
  • Auxiliary image A17 including a bounding box in which the result is expressed in color may be superimposed.
  • the immobilized sperm quality determination result is displayed on the optical image O2.
  • Auxiliary image A22 represented by using a plurality of bars may be superimposed.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Chemical & Material Sciences (AREA)
  • Theoretical Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Biomedical Technology (AREA)
  • Analytical Chemistry (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Biochemistry (AREA)
  • Immunology (AREA)
  • Pathology (AREA)
  • Medical Informatics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Dispersion Chemistry (AREA)
  • Signal Processing (AREA)
  • Optics & Photonics (AREA)
  • Investigating Or Analysing Biological Materials (AREA)
  • Apparatus Associated With Microorganisms And Enzymes (AREA)
  • Image Analysis (AREA)
PCT/JP2021/009685 2020-03-31 2021-03-10 顕微鏡システム、投影ユニット、及び、精子選別支援方法 Ceased WO2021200003A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP21779707.5A EP4130843A4 (en) 2020-03-31 2021-03-10 MICROSCOPE SYSTEM, PROJECTION UNIT AND SPERM SORTING SUPPORT METHOD
CN202180025391.0A CN115485602A (zh) 2020-03-31 2021-03-10 显微镜系统、投影单元以及精子筛选辅助方法
JP2022511751A JPWO2021200003A1 (https=) 2020-03-31 2021-03-10

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020063284 2020-03-31
JP2020-063284 2020-03-31

Publications (1)

Publication Number Publication Date
WO2021200003A1 true WO2021200003A1 (ja) 2021-10-07

Family

ID=77928722

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/009685 Ceased WO2021200003A1 (ja) 2020-03-31 2021-03-10 顕微鏡システム、投影ユニット、及び、精子選別支援方法

Country Status (4)

Country Link
EP (1) EP4130843A4 (https=)
JP (1) JPWO2021200003A1 (https=)
CN (1) CN115485602A (https=)
WO (1) WO2021200003A1 (https=)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220293219A1 (en) * 2021-03-09 2022-09-15 Thread Robotics Inc. System and method for dfi-based gamete selection
CN115074237A (zh) * 2022-07-29 2022-09-20 贵州省畜牧兽医研究所 一种山羊育种优选精子的显微操作取出采集装置
US11694344B2 (en) 2021-11-05 2023-07-04 Thread Robotics Inc. System and method for automated cell positioning
JP2023096267A (ja) * 2021-12-27 2023-07-07 株式会社エビデント 顕微鏡システム、投影ユニット、及び、選別支援方法
WO2023248958A1 (ja) * 2022-06-24 2023-12-28 株式会社エビデント 顕微鏡システム、投影ユニット、選別支援方法、及び、記録媒体
TWI866209B (zh) * 2022-06-08 2024-12-11 好孕行生醫股份有限公司 精子分選方法
TWI876619B (zh) * 2023-10-25 2025-03-11 國立陽明交通大學 精蟲分選裝置
JP7692232B1 (ja) 2024-09-09 2025-06-13 株式会社アークス 精子選別補助装置、精子選別補助システム、及び精子選別補助プログラム
US12406368B2 (en) 2022-10-17 2025-09-02 Vitruvianmd Pte Ltd Visual analysis of sperm DNA fragmentation
JP7751345B1 (ja) * 2025-04-24 2025-10-08 株式会社アークス 精子選別補助装置、投影装置、及び顕微鏡システム

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0829694A (ja) * 1994-07-20 1996-02-02 Nikon Corp 画像処理装置付き顕微鏡
JP2012198249A (ja) * 2012-07-27 2012-10-18 Chiba Univ 精子機能の検査方法
WO2012150689A1 (ja) 2011-05-02 2012-11-08 オリンパス株式会社 顕微鏡、及び、顕微鏡を用いた顕微授精方法
JP2017092730A (ja) * 2015-11-11 2017-05-25 ソニー株式会社 情報処理装置、情報処理方法、プログラム及び情報処理システム
WO2020008920A1 (ja) * 2018-07-02 2020-01-09 ソニー株式会社 医療用観察システム、医療用観察装置、及び医療用観察装置の駆動方法
WO2020059522A1 (ja) * 2018-09-21 2020-03-26 公立大学法人横浜市立大学 生殖医療支援システム
WO2020138279A1 (ja) * 2018-12-28 2020-07-02 オリンパス株式会社 顕微鏡システム

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPWO2020066043A1 (ja) * 2018-09-28 2021-08-30 オリンパス株式会社 顕微鏡システム、投影ユニット、及び、画像投影方法
WO2020066042A1 (ja) * 2018-09-28 2020-04-02 オリンパス株式会社 顕微鏡システム、投影ユニット、及び、画像投影方法
JP2023130870A (ja) * 2022-03-08 2023-09-21 株式会社エビデント 顕微鏡システム、重畳ユニット、及び、動作方法

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0829694A (ja) * 1994-07-20 1996-02-02 Nikon Corp 画像処理装置付き顕微鏡
WO2012150689A1 (ja) 2011-05-02 2012-11-08 オリンパス株式会社 顕微鏡、及び、顕微鏡を用いた顕微授精方法
JP2012198249A (ja) * 2012-07-27 2012-10-18 Chiba Univ 精子機能の検査方法
JP2017092730A (ja) * 2015-11-11 2017-05-25 ソニー株式会社 情報処理装置、情報処理方法、プログラム及び情報処理システム
WO2020008920A1 (ja) * 2018-07-02 2020-01-09 ソニー株式会社 医療用観察システム、医療用観察装置、及び医療用観察装置の駆動方法
WO2020059522A1 (ja) * 2018-09-21 2020-03-26 公立大学法人横浜市立大学 生殖医療支援システム
WO2020138279A1 (ja) * 2018-12-28 2020-07-02 オリンパス株式会社 顕微鏡システム

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
MCCALLUM, C. ET AL.: "Deep learning-based selection of human sperm with high DNA integrity", COMMUNICATIONS BIOLOGY, vol. 2, 2019, pages 250, XP055843297, DOI: 10.1038/s42003-019-0491-6
See also references of EP4130843A4
WANG, Y. ET AL.: "Prediction of DNA Integrity from Morphological Parameters Using a Single-Sperm DNA Fragmentation Index Assay", ADVANCED SCIENCE, 2019

Cited By (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220293219A1 (en) * 2021-03-09 2022-09-15 Thread Robotics Inc. System and method for dfi-based gamete selection
US11481900B2 (en) 2021-03-09 2022-10-25 Thread Robotics Inc. System and method for automated gamete selection
US11734822B2 (en) 2021-03-09 2023-08-22 Thread Robotics Inc. System and method for automated gamete selection
US11694344B2 (en) 2021-11-05 2023-07-04 Thread Robotics Inc. System and method for automated cell positioning
JP7693533B2 (ja) 2021-12-27 2025-06-17 株式会社エビデント 顕微鏡システム、投影ユニット、及び、選別支援方法
JP2023096267A (ja) * 2021-12-27 2023-07-07 株式会社エビデント 顕微鏡システム、投影ユニット、及び、選別支援方法
EP4209820A1 (en) * 2021-12-27 2023-07-12 Evident Corporation Microscope system and sorting support method
TWI866209B (zh) * 2022-06-08 2024-12-11 好孕行生醫股份有限公司 精子分選方法
WO2023248958A1 (ja) * 2022-06-24 2023-12-28 株式会社エビデント 顕微鏡システム、投影ユニット、選別支援方法、及び、記録媒体
CN115074237A (zh) * 2022-07-29 2022-09-20 贵州省畜牧兽医研究所 一种山羊育种优选精子的显微操作取出采集装置
US12406368B2 (en) 2022-10-17 2025-09-02 Vitruvianmd Pte Ltd Visual analysis of sperm DNA fragmentation
US12608804B2 (en) 2022-10-17 2026-04-21 Vitruvianmd Pte Ltd. Visual analysis of sperm DNA fragmentation
TWI876619B (zh) * 2023-10-25 2025-03-11 國立陽明交通大學 精蟲分選裝置
JP7692232B1 (ja) 2024-09-09 2025-06-13 株式会社アークス 精子選別補助装置、精子選別補助システム、及び精子選別補助プログラム
WO2026054092A1 (ja) * 2024-09-09 2026-03-12 株式会社アークス 精子選別補助装置、精子選別補助システム、及び精子選別補助プログラム
JP2026050293A (ja) * 2024-09-09 2026-03-19 株式会社アークス 精子選別補助装置、精子選別補助システム、及び精子選別補助プログラム
JP7751345B1 (ja) * 2025-04-24 2025-10-08 株式会社アークス 精子選別補助装置、投影装置、及び顕微鏡システム

Also Published As

Publication number Publication date
EP4130843A4 (en) 2024-05-01
CN115485602A (zh) 2022-12-16
EP4130843A9 (en) 2024-06-05
JPWO2021200003A1 (https=) 2021-10-07
EP4130843A1 (en) 2023-02-08

Similar Documents

Publication Publication Date Title
WO2021200003A1 (ja) 顕微鏡システム、投影ユニット、及び、精子選別支援方法
JP7214753B2 (ja) 顕微鏡システム
US11636627B2 (en) System for histological examination of tissue specimens
US9522396B2 (en) Apparatus and method for automatic detection of pathogens
US11662565B2 (en) Microscope system, projection unit, and image projection method
US9145572B2 (en) Observation system, recording medium, and control method of observation system
US12394170B2 (en) Cell image analysis method and cell image analyzer
CA3100751A1 (en) A method for automated non-invasive measurement of sperm motility and morphology and automated selection of a sperm with high dna integrity
JP7253273B2 (ja) 生殖医療支援システム
JP2018000048A (ja) 多能性幹細胞の無染色評価支援方法、プログラム、演算装置
JP7692232B1 (ja) 精子選別補助装置、精子選別補助システム、及び精子選別補助プログラム
WO2021200002A1 (ja) 顕微鏡システム、投影ユニット、及び、検卵支援方法
WO2023248958A1 (ja) 顕微鏡システム、投影ユニット、選別支援方法、及び、記録媒体
JP7751345B1 (ja) 精子選別補助装置、投影装置、及び顕微鏡システム
CN115713767A (zh) 基于监测显微镜测量的机器学习算法自动化训练
JP6790734B2 (ja) 装置、方法、およびプログラム
JP7693533B2 (ja) 顕微鏡システム、投影ユニット、及び、選別支援方法
WO2023127001A1 (ja) 顕微鏡システム、投影ユニット、及び、選別支援方法
JP2025158211A (ja) 顕微鏡システム
JP2024059329A (ja) 顕微鏡システム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21779707

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022511751

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2021779707

Country of ref document: EP

Effective date: 20221031