WO2023248958A1

WO2023248958A1 - Microscope system, projection unit, sorting assisting method, and recording medium

Info

Publication number: WO2023248958A1
Application number: PCT/JP2023/022472
Authority: WO
Inventors: 敏征服部; 拓人山根
Original assignee: 株式会社エビデント
Priority date: 2022-06-24
Filing date: 2023-06-16
Publication date: 2023-12-28

Abstract

A microscope system (1) comprises: a microscope (100) that forms an image of a sample containing sperm; an imaging device (143) that acquires the image of the sample; a processing device (200) that generates, on the basis of the acquired image, an auxiliary image containing information related to grading of the sperm; and a projection device (153) that superimposes the auxiliary image onto the image plane where the microscope (100) forms the image. The processing device (200) extracts at least one portion of the sperm from the image by utilizing a segmentation model generated by means of deep learning. The processing device (200) additionally grades the sperm on the basis of a measured value of a feature quantity measured from the at least one portion that was extracted and preregistered grading criteria indicating the relationship between the feature quantity and the grade.

Description

Microscope system, projection unit, sorting support method, and recording medium

The disclosure of this specification relates to a microscope system, a projection unit, a sorting support method, and a recording medium.

Now that people are getting married later and having children later in life, the number of patients undergoing infertility treatment is increasing year by year, and the demand for assisted reproductive technology (ART) is also increasing.

ART is a general term for techniques such as in vitro fertilization (IVF) and microinsemination, in which eggs and sperm extracted from humans are fertilized outside the body. It is distinguished from artificial insemination.

Technology related to ART is described in Patent Document 1, for example. Patent Document 1 describes a microscope suitable for intracytoplasmic sperm injection (ICSI) used in microinsemination, which is a type of ART. Note that ICSI is a method in which sperm is directly injected into an egg by inserting an injection pipette containing sperm into an egg fixed with a holding pipette.

International Publication No. 2012/150689

By the way, in order to increase the success rate of ICSI, it is important to select sperm and inject the sperm suitable for fertilization into the egg. However, whether or not the sperm obtained through the sorting process is of good quality largely depends on the experience of the embryo cultivator who is working on the process, and differences in fertilization rates tend to occur between embryo cultivators. For this reason, consideration is being given to using artificial intelligence (AI) to support the sperm selection work performed by embryologists.

However, if AI is entrusted with determining the quality of sperm, the basis for the determination becomes a black box. For this reason, there is a problem in that embryo culturists do not have a sense of security regarding the judgment results, making it difficult for embryo cultivators to make final decisions. Another issue is that the standards for good and bad differ depending on the facility (for example, a university hospital or clinic) that the embryo cultivator belongs to, so it is necessary to build an AI that learns the standards of each facility. There is also. It is difficult in terms of time to prepare for the introduction of such a system that imposes a heavy burden on each facility before the start of operation while conducting daily operations.

In view of the above circumstances, an object of one aspect of the present invention is to provide a new technique that supports the sperm selection work performed by embryo culturists.

A microscope system according to one aspect of the present invention includes a microscope that forms an image of a sample containing sperm, an imaging device that acquires an image of the sample, and information regarding grading of the sperm based on the image acquired by the imaging device. and a superimposition device that superimposes the auxiliary image on an image plane on which the microscope forms the image. The image processing device extracts one or more parts of the sperm from the image using a segmentation model generated by deep learning, and measures feature quantities measured from the one or more extracted parts. The sperm is graded based on the value and a pre-registered grading standard indicating a relationship between the feature amount and the grade.

A projection unit according to one aspect of the present invention is a projection unit attached to a microscope, and includes an imaging section that acquires an image of a sample containing sperm, and a projection unit that is configured to perform grading of the sperm based on the image acquired by the imaging section. The apparatus includes an image processing section that generates an auxiliary image containing information, and a superimposition section that superimposes the auxiliary image on an image plane on which the microscope forms the image. The image processing unit extracts one or more parts of the sperm from the image using a segmentation model generated by deep learning, and measures feature quantities measured from the one or more extracted parts. The sperm is graded based on the value and a pre-registered grading standard indicating a relationship between the feature amount and the grade.

A sorting support method according to one aspect of the present invention includes forming an image of a sample containing sperm, acquiring an image of the sample, and an auxiliary image containing information regarding grading of the sperm based on the acquired image. and superimposing the auxiliary image on an image plane on which the image is formed. Generating the auxiliary image includes extracting one or more portions of the sperm from the image using a segmentation model generated by deep learning, and measuring from the extracted one or more portions. grading the sperm based on the measured value of the feature amount and a pre-registered grading standard indicating a relationship between the feature amount and the grade.

In one aspect of the present invention, the recording medium is a non-temporary recording medium that stores a program, and the program causes a computer to image a sample containing sperm by using a segmentation model generated by deep learning. extracting one or more parts of the spermatozoa, a measured value of a feature quantity measured from the one or more extracted parts, and a pre-registered grading standard indicating the relationship between the feature quantity and the grade; A process of grading the spermatozoa is performed based on.

According to the above aspect, it is possible to support determination of the quality of a sample in sperm selection work performed by an embryonic cultivator.

1 is a diagram illustrating the configuration of a microscope system 1. FIG. 1 is a diagram illustrating the configuration of a microscope 100. FIG. 5 is a diagram illustrating the configuration of an operation section of an input device 50. FIG. 2 is a diagram illustrating a hardware configuration of a processing device 200. FIG. It is a flow chart which shows an example of the procedure of ICSI by an embryo culturist. 3 is a diagram illustrating the configuration of a drop formed as a sample 300 in a petri dish 310. FIG. It is a flowchart which shows an example of the sperm selection procedure by an embryo culturist. It is a flowchart which shows an example of selection support processing. 3 is a flowchart illustrating an example of auxiliary image generation processing. 3 is a diagram showing an example of an optical image generated by the microscope 100. FIG. 5 is a diagram showing an example of a captured image acquired by an imaging device 143. FIG. FIG. 3 is a diagram showing an example of object detection results for a captured image. FIG. 3 is a diagram showing an example of a portion extracted by segmentation of a captured image. It is a figure showing an example of grading standard T1. FIG. 3 is a diagram showing an example of a GUI for display settings. 5 is a diagram showing an example of an image seen through the eyepiece lens 101. FIG. 7 is a diagram showing another example of an image seen through the eyepiece lens 101. FIG. 7 is a diagram showing still another example of an image seen through the eyepiece lens 101. FIG. 7 is a diagram showing still another example of an image seen through the eyepiece lens 101. FIG. 7 is a diagram showing still another example of an image seen through the eyepiece lens 101. FIG. 7 is a diagram showing still another example of an image seen through the eyepiece lens 101. FIG. 1 is a diagram illustrating the configuration of a microscope system 2. FIG.

FIG. 1 is a diagram illustrating the configuration of a microscope system 1. FIG. 2 is a diagram illustrating the configuration of the microscope 100. FIG. 3 is a diagram illustrating the configuration of the operation section of the input device 50. FIG. 4 is a diagram illustrating the configuration of the processing device 200.

The microscope system 1 is a system for observing a sample by looking through an eyepiece 101. Specifically, the microscope system 1 is an inverted microscope system equipped with a transmitted illumination system 120 and used for microinsemination, particularly sperm selection. The microscope system 1 is used, for example, by an embryologist. During sperm sorting work, the sample to be observed is a sperm suspension containing sperm stored in a petri dish or the like.

The microscope system 1 includes at least a microscope 100, an imaging device 143, a projection device 153, and a processing device 200. The microscope 100 forms an image (optical image) of a sample containing sperm to be sorted. The imaging device 143 acquires an image (captured image) of the sample. The processing device 200 is an example of an image processing device that generates an auxiliary image based on an image acquired by the imaging device 143. The projection device 153 is an example of a superimposition device that superimposes an auxiliary image on the image plane on which the microscope 100 forms an optical image, and displays the auxiliary image on the image plane.

Note that the auxiliary image is an image that includes information regarding sperm grading, and is generated by a combination of processing using AI and rule-based processing. Further, the selection target is an object whose acceptability is determined by the user, and is an object whose selection or non-selection is determined as a result of the acceptability determination. In addition, in this specification, "displaying an image" refers to forming an image (image) so that it is visible. In other words, "displaying an image" means forming an image (image) on a visible surface position).

Hereinafter, a specific example of the configuration of the microscope system 1 will be described in detail with reference to FIGS. 1 to 4. As shown in FIG. 1, the microscope system 1 includes, in addition to the above-mentioned microscope 100, imaging device 143, projection device 153, and processing device 200, a microscope controller 10, a display device 30, and a plurality of input devices. (input device 40, input device 50, input device 60, input device 70), and an identification device 80. Further, the microscope system 1 is connected to a database server 20 in which various data are stored. Note that in this example, the imaging device 143 and the projection device 153 are arranged within the microscope body 110 of the microscope 100.

The microscope 100 is an inverted microscope equipped with an eyepiece 101. As shown in FIG. 1, the microscope 100 includes a microscope body 110, a plurality of objective lenses 102, a stage 111, a transmitted illumination system 120, and an eyepiece tube 170 attached to the microscope body 110. Further, as will be described later, the microscope 100 includes modulation elements in each of the illumination optical path and the observation optical path for visualizing unstained samples such as sperm and eggs. Users such as embryo cultivators use the microscope 100 to examine samples using four microscopy methods: bright field (BF) observation, polarized light (PO) observation, differential interference interference (DIC) observation, and modulated contrast (MC) observation. can be observed. Note that modulated contrast observation is also referred to as relief contrast (RC) observation.

A plurality of objective lenses 102 are attached to a revolver 112. As shown in FIG. 2, the plurality of objective lenses 102 include an objective lens 102a for BF observation, an objective lens 102b for PO observation and DIC observation, and an objective lens 102c for MC observation. Further, the objective lens 102c includes a modulator 104. The modulator 104 includes three regions with different transmittances (for example, a region with a transmittance of about 100%, a region with a transmittance of about 5%, and a region with a transmittance of about 0%).

Although FIG. 2 illustrates three objective lenses depending on the microscopy method, the plurality of objective lenses 102 may include a plurality of objective lenses with different magnifications for each microscopy method. In the following cases, a 4x objective lens for BF observation, a 10x, 20x, and 40x objective lens for MC observation, a 20x objective lens for PO observation, and a 60x objective lens for DIC observation are included. This will be explained using an example.

The revolver 112 is a switching device that switches the objective lens placed on the optical path among the plurality of objective lenses 102. The revolver 112 switches the objective lens placed on the optical path depending on the microscopy method and observation magnification. The objective lens placed on the optical path by the revolver 112 guides the transmitted light that has passed through the sample to the eyepiece 101 .

A sample placed in a container is placed on the stage 111. The container is, for example, a petri dish, and the sample contains reproductive cells such as sperm and eggs. The stage 111 moves in the optical axis direction of the objective lens 102 arranged on the optical path and in the direction orthogonal to the optical axis of the objective lens 102. Note that the stage 111 may be a manual stage or an electric stage.

The transmitted illumination system 120 illuminates the sample placed on the stage 111 from above the stage 111. The transmitted illumination system 120 includes a light source 121 and a universal condenser 122, as shown in FIGS. 1 and 2. The light source 121 may be, for example, an LED (Light Emitting Diode) light source or a lamp light source such as a halogen lamp light source.

As shown in FIG. 2, the universal condenser 122 includes a polarizer 123 (first polarizing plate), a plurality of optical elements housed in a turret 124, and a condenser lens 128. The polarizer 123 is used in MC observation, PO observation, and DIC observation. The turret 124 houses a plurality of optical elements that are switched and used depending on the microscopy method. DIC prism 125 is used for DIC observation. The aperture plate 126 is used for BF observation and PO observation. The optical element 127 is a combination of a slit plate 127a, which is a light-shielding plate in which a slit is formed, and a polarizing plate 127b (second polarizing plate) arranged so as to cover a part of the slit. used.

The eyepiece tube 170 includes an eyepiece lens 101. The imaging lens 103 is arranged between the eyepiece lens 101 and the objective lens 102. The imaging lens 103 forms an optical image of the sample on an image plane IP between the eyepiece lens 101 and the imaging lens 103 based on transmitted light. Further, an auxiliary image, which will be described later, is also formed on the image plane IP based on light from the projection device 153. As a result, the optical image and the auxiliary image are displayed on the image plane IP. A user of the microscope system 1 uses the eyepiece lens 101 to observe the virtual image of the optical image and the auxiliary image formed on the image plane IP.

The microscope main body 110 includes a laser assisted hatching unit 130, an imaging unit 140, and a projection unit 150, as shown in FIGS. 1 and 2. Further, the microscope main body 110 includes an intermediate variable magnification unit 160, as shown in FIG. Furthermore, the microscope main body 110 includes a DIC prism 105 and an analyzer 106 that can be inserted into and removed from the optical path.

The laser assisted hatching unit 130 is a laser unit placed between the objective lens 102 and the imaging lens 103, as shown in FIG. The laser assisted hatching unit 130 irradiates the sample with laser light by introducing the laser light from between the objective lens 102 and the imaging lens 103. More specifically, the laser assisted hatching unit 130 irradiates the zona pellucida surrounding an embryo grown from a fertilized egg with laser light, for example. Laser assisted hatching unit 130 includes a splitter 131, a scanner 133, a lens 134, and a laser 135. The splitter 131 is, for example, a dichroic mirror. The scanner 133 is, for example, a galvano scanner, and adjusts the irradiation position of the laser beam in a direction perpendicular to the optical axis of the objective lens 102. Lens 134 converts the laser beam into a parallel beam of light. Thereby, the laser beam is focused onto the sample by the objective lens 102.

As shown in FIG. 2, the imaging unit 140 includes a splitter 141 and an imaging device 143 that acquires a captured image of the sample based on transmitted light. Imaging unit 140 is arranged between imaging lens 103 and eyepiece 101. The splitter 141 is, for example, a half mirror. The imaging lens 103 forms an optical image of the sample on the light-receiving surface of an image sensor included in the imaging device 143. The imaging device 143 is, for example, a digital camera that acquires a captured image, and the imaging device included in the imaging device 143 is, for example, a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, etc. It is. The image sensor detects light from the sample and converts the detected light into an electrical signal through photoelectric conversion. The imaging unit 140 outputs the captured image acquired by the imaging device 143 to the processing device 200.

The projection unit 150 is arranged between the imaging lens 103 and the eyepiece 101. Projection unit 150 includes a splitter 151, a lens 152, and a projection device 153, as shown in FIG. The splitter 151 is, for example, a half mirror. The projection device 153 projects the auxiliary image generated by the processing device 200. More specifically, the lens 152 focuses the light from the projection device 153 on the image plane of the imaging lens 103, that is, the image plane IP where an optical image is formed, so that the projection device 153 focuses on the image plane IP. Project an auxiliary image.

The intermediate variable magnification unit 160 is arranged between the objective lens 102 and the imaging lens 103. As shown in FIG. 2, the intermediate variable magnification unit 160 includes a plurality of lenses (lens 161, lens 162, and lens 163), and by switching the lenses placed on the optical path among these lenses, images are formed on the image plane. Change the magnification of the optical image. By using the intermediate variable magnification unit 160, the magnification of the optical image can be changed without switching the objective lens 102 located near the sample.

The DIC prism 105 and analyzer 106 are arranged between the objective lens 102 and the imaging lens 103. DIC prism 105 is used for DIC observation. Analyzer 106 is used for PO observation and DIC observation.

In the microscope 100, when performing MC observation, a polarizer 123 and an optical element 127 are arranged on the illumination light path as a modulation element (hereinafter referred to as a first modulation element) that modulates the illumination light irradiated onto the sample. A modulator 104 is arranged on the observation optical path as a modulation element (hereinafter referred to as a second modulation element) that modulates transmitted light. Furthermore, when performing PO observation, a polarizer 123 is placed on the illumination optical path as a first modulation element, and an analyzer 106 is placed on the observation optical path as a second modulation element. When performing DIC observation, a polarizer 123 and a DIC prism 125 are placed on the illumination optical path as a first modulation element, and an analyzer 106 and a DIC prism 105 are placed on the observation optical path as a second modulation element. . Thereby, it is possible to visualize an unstained sample, and for example, sperm selection can be performed.

The microscope controller 10 is a device that controls the microscope 100. The microscope controller 10 is connected to the processing device 200, the input device 50, and the microscope 100, and controls the microscope 100 according to commands from the processing device 200 or the input device 50.

The display device 30 is, for example, a liquid crystal display, a plasma display, an organic EL display, a CRT display, an LED matrix panel, or the like.

The input device 40 includes a handle 41 and a handle 42. By operating the handle 41 and the handle 42, the operation of a micromanipulator (not shown) that moves the pipette 43 and the pipette 44 is controlled. Pipette 43 and pipette 44 are used to manipulate samples in microinsemination work including sperm sorting. Pipette 43 is, for example, a holding pipette, and pipette 44 is, for example, an injection pipette.

The input device 50 is a hand switch device for changing settings regarding the microscopy method and observation magnification of the microscope 100. As shown in FIG. 3, the input device 50 has, for example, six buttons (buttons 51 to 56), and the user can quickly change the settings of the microscope 100 by simply pressing these buttons. Can be done.

When the user presses the button 51, the settings of the microscope 100 are switched to BF observation with a 4x observation magnification (hereinafter referred to as BF4×observation). When the user presses the button 52, the settings of the microscope 100 are switched to settings for MC observation with an observation magnification of 10x (hereinafter referred to as MC10x observation). When the user presses the button 53, the settings of the microscope 100 are switched to settings for MC observation at a magnification of 20 times (hereinafter referred to as MC20x observation). When the user presses the button 54, the settings of the microscope 100 are switched to settings for MC observation at a magnification of 40 times (hereinafter referred to as MC40x observation). When the user presses the button 55, the settings of the microscope 100 are switched to settings for PO observation at a magnification of 20x (hereinafter referred to as PO20x observation). When the user presses the button 56, the settings of the microscope 100 are switched to settings for DIC observation at an observation magnification of 60x (hereinafter referred to as DIC60x observation).

The input device 60 is a keyboard. The input device 70 is a mouse. The input device 60 and the input device 70 are each connected to the processing device 200. Note that the microscope system 1 may include other input devices (not shown) such as a touch panel, a voice input device, and a foot pedal.

The identification device 80 is a device that acquires identification information attached to a sample. Note that "attached to a sample" includes, for example, a case where identification information is attached to a container containing the sample. The identification information is information that identifies the sample, and more specifically, for example, information that identifies the patient who provided the sample. The identification device 80 may be, for example, a barcode reader, an RFID (registered trademark) reader, a QR code (registered trademark) reader, or the like.

The processing device 200 generates an auxiliary image based on the captured image acquired by the imaging device 143. The generated auxiliary image is output to the projection device 153 of the microscope 100, either directly or via the microscope controller 10. Note that the processing device 200 is connected to the microscope 100, the microscope controller 10, the display device 30, the input device 60, the input device 70, and the identification device 80, as shown in FIG. The processing device 200 is also connected to a database server 20.

The processing device 200 includes an image analysis section 210, an image generation section 220, and a storage section 230 as functional components related to the generation of auxiliary images.

The image analysis unit 210 performs segmentation and feature measurement on the captured image, and grades the sperm based on these results. Further, the image analysis unit 210 may perform object detection on the captured image in addition to segmentation and feature quantity measurement, and may grade the sperm based on these results.

In object detection, the image analysis unit 210 detects sperm from the captured image. The object detection algorithm is not particularly limited as long as it can detect the position of an object classified as a sperm. For object detection, an object detection model generated by deep learning, such as SSD, YOLO, FasterR-CNN, etc., may be used.

In the segmentation, the image analysis unit 210 extracts the sperm part from the captured image. The segmentation algorithm is not particularly limited as long as one or more parts of each sperm can be extracted instead of extracting the entire sperm in one block. A segmentation model generated by deep learning is used for segmentation.

In feature quantity measurement, the image analysis unit 210 measures feature quantities from one or more parts of the sperm extracted by segmentation. It is sufficient that the feature quantity can be measured, and the algorithm for measuring the feature quantity is not particularly limited. To measure the feature amount, for example, the measured value of the feature amount may be calculated by performing pre-programmed arithmetic processing using a rule-based model.

In the grading, the image analysis unit 210 grades the sperm based on the measured values of the measured feature amounts and pre-registered grading criteria. The grading standard is information indicating the relationship between the feature amount and the grade of the sperm, and may be information that uniquely determines the grade of the sperm from the measured feature amount. The grading criteria are registered in the microscope system 1 by being stored in the storage unit 230 in advance.

The image generation unit 220 generates an auxiliary image including information regarding grading based on the information obtained by the above-described analysis process performed by the image analysis unit 210. The auxiliary image generated by the image generation unit 220 is output to the projection device 153. Thereby, the projection device 153 projects the auxiliary image onto the image plane, and the auxiliary image is displayed superimposed on the optical image.

The storage unit 230 stores an AI model (AIM: for example, the above-mentioned object detection model, segmentation model, etc.), a rule-based model (RBM), and a grading standard (G standard) used in image analysis performed by the image analysis unit 210. Remember.

Note that the processing device 200 may be a general-purpose computer or a dedicated computer. Although the processing device 200 is not particularly limited to this configuration, it may have a physical configuration as shown in FIG. 4, for example. Specifically, the processing device 200 may include a processor 201, a storage device 202, an input interface (I/F) 203, an output interface (I/F) 204, and a communication device 205. They may be connected to each other by a bus 206.

The processor 201 may include hardware, and the hardware may include, for example, at least one of a circuit for processing digital signals and a circuit for processing analog signals. Processor 201 may include one or more circuit devices (eg, ICs) or one or more circuit elements (eg, resistors, capacitors), eg, on a circuit board. The processor 201 may be a CPU (central processing unit). Further, various types of processors including a GPU (Graphics processing unit) and a DSP (Digital Signal Processor) may be used as the processor 201. The processor 201 may be a hardware circuit including an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). Processor 201 can include an amplifier circuit, a filter circuit, etc. for processing analog signals. The processor 201 functions as the image analysis section 210 and image generation section 220 described above by executing a program stored in the storage device 202.

The storage device 202 may include memory and/or other storage devices. The memory may be, for example, random access memory (RAM). The memory may be a semiconductor memory such as SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory). The storage device 202 may be, for example, a register, a magnetic storage device such as a hard disk drive, an optical storage device such as an optical disk drive, an internal or external hard disk drive, a solid state storage device, a CD-ROM, a DVD, or other optical or magnetic storage device. It may also include disk storage or other storage devices. The storage device 202 stores the program executed by the processor 201, the various models described above, and the grading criteria in a rewritable memory, and functions as the storage unit 230 described above. Note that the storage device 202 is an example of a non-transitory computer-readable storage medium.

The input I/F 203 is connected to an input device operated by a user of the microscope system 1 (for example, an embryo culturist), receives an operation signal corresponding to an operation on the input device, and outputs it to the processor 201.

The output I/F 204 is connected to the display device 30. The output I/F 204 may further be connected to an audio output device such as a speaker that outputs audio, a light emitting device such as a lamp that outputs light, a vibration device such as a vibrator that outputs vibration, etc. (not shown).

The communication device 205 is a device that exchanges data with the microscope 100 and other devices. The communication device 205 may be a communication device that exchanges data by wire, or may be a communication device that exchanges data wirelessly. The programs, various models, and grading standards stored in the storage device 202 may be acquired by the communication device 205 from another device via the Internet.

With the microscope system 1 configured as described above, it is possible to support the sperm selection work performed by embryonic cultivators while solving the problems that would arise if the determination of the quality of sperm is left to AI. Specifically, it is as follows.

Because the grade of the sperm is determined based on the measured value of the feature quantity measured from the sperm part and the grading criteria registered in advance, the basis for determining the grade is clear. Therefore, by registering appropriate grading standards in advance, the embryo culturist can trust the sperm grading performed by the microscope system 1 and use it for final decision making.

Furthermore, by registering and using the grading standards in the microscope system 1, it is possible to respond to standards for good and bad that differ from facility to facility by simply rewriting the grading standards. Therefore, the microscope system 1 can be used in various facilities with different standards, and the microscope system 1 can be introduced into each facility in a relatively short period of time.

Furthermore, unlike grading standards, segmentation models can be used in common regardless of facility. In other words, no learning work is required to construct a model that meets the standards of each facility. Therefore, without imposing an excessive burden on the facility when introducing the system, it is possible to perform high-accuracy and high-speed region extraction using artificial intelligence (segmentation model) to extract specific regions (parts) of sperm. This makes it possible to provide a system with a high level of balance between installation cost and performance.

In this way, Microscope System 1 solves two problems at the same time: high introduction costs due to the need for learning according to the facility's own standards, and a black box regarding the basis of judgment, and enables embryo culturists to It is possible to appropriately support the sperm sorting work carried out by

Note that it is preferable that the image analysis unit 210 described above measures at least the feature quantities of the head, midpiece, tail, and vacuole of the sperm extracted by using a segmentation model. This is because, in sperm selection, the quality of sperm is often evaluated using features that characterize these parts (head, midpiece/tail, and vacuole).

More specifically, the measured features include the length of the head, the width of the head, the width of the middle piece and the tail, the length of the middle piece and the tail, and the length of the middle piece and tail relative to the head. It is desirable that at least one of the slope of vacuoles and the number of vacuoles be included. This is because these feature amounts are particularly often used in sperm selection. Thereby, in grading, it is possible to appropriately respond to various grading standards that differ from facility to facility using these feature amounts.

It is further preferable that feature quantities be measured using a rule-based model. As a result, even if a new feature quantity is used as a grading standard, it can be handled simply by modifying the measurement program, and the impact on the field can be minimized.

Furthermore, it is desirable that the image analysis unit 210 extract at least the head, midpiece/tail, and vacuole of the sperm from the captured image using a segmentation model. This is because by distinguishing and extracting these portions using the segmentation model, the feature amounts of these portions can be easily measured using the rule-based model in feature amount measurement. Therefore, it is possible to enjoy the above-mentioned advantages obtained by using the rule-based model.

Furthermore, the image analysis unit 210 may use a segmentation model to extract one or more portions of sperm from the region in the captured image narrowed down by the object detection model. By excluding regions other than sperm from the segmentation target region in advance using an object detection model, it is possible to prevent regions other than sperm from being mistakenly identified as sperm parts and extracted.

It is preferable that the image generation unit 220 described above generates an auxiliary image that includes at least one of grade information indicating the grade of sperm and measured values of feature quantities outside the numerical range that should be satisfied by sperm of the highest grade. That is, the information regarding grading included in the auxiliary image preferably includes at least one of the grade information and the measured value of the feature amount outside the numerical range to be satisfied, and it is desirable that at least one of the two be displayed on the image plane.

By displaying grade information on the image plane, it is possible to directly understand the grade of sperm. Furthermore, by displaying the measured values of the feature amounts outside the numerical range, it is possible to understand whether there are feature amounts that do not match the conditions and to what extent they do not match the conditions. Therefore, a user who is aware of the grading standards can indirectly understand the grade. Furthermore, by displaying the measured values of feature quantities outside the numerical range, it is possible to understand the basis for judgment as well as the grade. In this way, by including the above-mentioned information in the auxiliary image, the user can understand the grade of sperm and the basis for its determination from the auxiliary image.

Note that it is more desirable for the image generation unit 220 to generate an auxiliary image that includes both grade information indicating the grade of sperm and measured values of feature amounts outside the numerical range that should be satisfied. However, these pieces of information may be switched and displayed as appropriate. Therefore, the image generation unit 220 only needs to generate an auxiliary image that includes at least one of the two.

Furthermore, it is preferable that the image generation unit 220 changes the configuration of information regarding grading according to the display settings. That is, the content and number of information included in the auxiliary image may be changed. As a result, in the microscope system 1, it is possible to change the information displayed on the image plane as an auxiliary image depending on the user or the sperm donor, thereby avoiding deterioration in the visibility of the optical image due to too much information. be able to.

For example, in a certain display setting, the image generation unit 220 may generate an auxiliary image that includes grade information, and may generate an auxiliary image that includes grade information (for example, only the best grade G1, only grade G1 and grade G2, etc.). ) may also be used to generate an auxiliary image that includes only grade information. For example, by displaying only the grade information indicating grade G1, the embryo cultivator can easily identify the sperm that best matches the grading criteria. On the other hand, depending on the sperm donor, there may be almost no grade G1 sperm. In such a case, the display settings may be changed to display grade information corresponding to multiple grades.

In another display setting, the image generation unit 220 may generate an auxiliary image that includes a measured value of a feature amount (hereinafter also referred to as an abnormal value) outside the numerical value range that should be satisfied. For example, by displaying abnormal values along with grade information, the embryo cultivator can understand the basis for the judgment, and can perform sperm selection work with peace of mind. Furthermore, if too much information is displayed within the field of view, it becomes difficult to observe the sperm itself. For this reason, for example, when a veteran embryo cultivator is working, the display of grade information may be turned off and only abnormal values may be displayed.

It is desirable that the storage unit 230 described above stores the grading criteria in a rewritable memory. Thereby, it is possible to update the grading standards according to the facility, and the grading standards registered in the microscope system 1 can be easily adjusted for each facility. In particular, a dedicated GUI for updating the grading criteria may be provided, and the processing device 200 may update the registered grading criteria in accordance with grading settings made on the GUI. Note that the grading settings may be performed by directly rewriting the settings file.

Furthermore, the storage unit 230 may store a plurality of grading standards in a rewritable memory. Thereby, it is possible to select a grading standard that matches the standards of the institution for each facility, and it is possible to easily change the grading standard used for grading for each facility. In particular, a dedicated GUI for selecting a grading standard may be provided, and the processing device 200 may determine a grading standard to be used for grading from a plurality of grading standards in accordance with grading settings made on the GUI. good. Note that the grading settings may be performed by directly rewriting the settings file.

Furthermore, it is preferable that the grading criteria stored in the storage unit 230 include information regarding one or more combinations of feature amounts and numerical ranges that should be satisfied by sperm of the highest grade in the measurement value of the feature amount. In particular, it is desirable to include information regarding a plurality of combinations, that is, it is desirable that the grade be determined using a plurality of different feature quantities. Thereby, it is possible to determine whether the sperm is good or bad with higher accuracy than when the grade is determined using a single feature amount.

FIG. 5 is a flowchart showing an example of an ICSI procedure performed by an embryonic cultivator. FIG. 6 is a diagram illustrating the configuration of a drop formed as a sample 300 in a petri dish 310. FIG. 7 is a flowchart showing an example of a sperm selection procedure by an embryologist. FIG. 8 is a flowchart illustrating an example of the sorting support process. FIG. 9 is a flowchart illustrating an example of auxiliary image generation processing. FIG. 10 is a diagram showing an example of an optical image generated by the microscope 100. FIG. 11 is a diagram showing an example of a captured image acquired by the imaging device 143. FIG. 12 is a diagram showing an example of object detection results for a captured image. FIG. 13 is a diagram showing an example of a portion extracted by segmentation of a captured image. FIG. 14 is a diagram showing an example of the grading standard T1. FIG. 15 is a diagram showing an example of a GUI for display settings. 16 to 21 are diagrams showing examples of images seen through the eyepiece lens 101. Hereinafter, with reference to FIGS. 5 to 21, specific utilization of the sperm selection support method performed by the microscope system 1 in ICSI will be described.

First, the user prepares a sample (step S1). Here, for example, as shown in FIG. 6, the user creates a sample 300 containing a plurality of drops in a petri dish 310 and places it on the stage 111.

The drop 301 is a cleaning drop and is used to clean the pipette. The drop 302 is a sperm suspension drop, for example, a sperm suspension dropped into a PVP solution. The drop 303 is a drop for manipulating eggs, for example, an egg is placed in an m-HTF solution. Note that the m-HTF solution is a Hepps-containing HTF solution to which 10% serum was added. These drops are coated with mineral oil.

Next, the user sets up the microscope system 1 (step S2). Here, the user, for example, presses the button 51 of the input device 50 to switch the setting of the microscope system 1 to BF4× observation. Thereafter, the input device 40 is operated to adjust the positions of the

pipettes

43 and 44, and the

pipettes

43 and 44 are brought into focus. Furthermore, the stage 111 is moved to wash the

pipettes

43 and 44 with the drop 301 (washing drop).

When the setup is completed, the user checks the state of the egg (egg cell) in the drop 303 (egg manipulation drop) (step S3). Here, the user, for example, presses the button 53 of the input device 50 to switch the setting of the microscope system 1 to MC20x observation. Observe the morphology of the eggs using MC20x observation and select the eggs. Further, for example, the setting of the microscope system 1 may be switched to PO20x observation by pressing the button 55 of the input device 50. By observing the spindle of the egg by PO20x observation, the degree of maturity of the egg may be determined, and the eggs may be further sorted.

When the egg selection is completed, the user performs sperm selection according to the procedure shown in FIG. 7 (step S4). First, the user presses the button 54 on the input device 50 to switch the settings of the microscope system 1 to MC40x observation, for example. Then, the stage 111 is moved to move the observation position to the drop 302 (sperm floating drop), and the sperm is focused on by MC40× observation (step S11).

Next, the user selects sperm suitable for fertilization by MC40x observation (step S12). Traditionally, in this process, an embryologist would judge the quality of the sperm based on the morphology and motility of the sperm observed using optical images, and then select the sperm based on that judgment. However, determining the quality of sperm depends largely on the experience of embryologists, and the problem has been the variation in skills among embryologists and the resulting disparity in fertilization rates. In addition, standards for good and bad embryonics often differ depending on the facility to which the embryologist belongs, and when an embryologist moves from one facility to another, he or she is required to perform work in accordance with the standards of the new facility.

In consideration of such issues, the microscope system 1 grades the sperm included in the captured image in step S12, and displays an auxiliary image containing information regarding the grading on the image plane. This allows embryologists to judge the quality of sperm by referring to auxiliary images without relying solely on subjective judgment, thereby reducing disparities in fertilization rates among embryologists. . In addition, when generating the auxiliary image, AI is used to measure features from the extracted sperm part, and the grading standards are rewritten by grading the sperm based on the measured features and pre-registered grading standards. You can easily adapt to the standards of each facility by simply Furthermore, since grading is performed according to pre-registered grading standards, the basis for the judgment is clear and the embryo cultivator can easily accept the judgment result. Therefore, it is expected that the embryo culturist will actively utilize the auxiliary images for sperm selection, so that the quality and quantity of the sorting work performed by the embryo culturist can be improved at the same time.

Specifically, in step S12, the microscope system 1 performs the sorting support process shown in FIG. 8, so that the auxiliary image is displayed on the image plane together with the optical image. In the sorting support process, first, the microscope system 1 forms an optical image of the sample on the image plane (step S21). Here, the microscope 100 forms, for example, an optical image O1 shown in FIG. 10 on the image plane. Note that the region 143R shown in FIG. 10 indicates the region photographed by the imaging device 143 in step S22.

Simultaneously with step S21, the microscope system 1 acquires a captured image (step S22). Here, the imaging device 143 acquires a captured image D1 of the sample shown in FIG. 11, for example, based on light from the sample, and outputs the acquired captured image D1 to the processing device 200.

After that, the microscope system 1 generates an auxiliary image based on the captured image (step S23). Here, the processing device 200 performs the auxiliary image generation process shown in FIG. In the auxiliary image generation process, the processing device 200 first performs object detection on the captured image D1 (step S31). Here, the image analysis unit 210 detects the position of the object classified as sperm by inputting the captured image D1 to an object detection model obtained by deep learning. Note that FIG. 12 shows a state in which a box B is attached to the position of an object whose sperm has been classified by object detection.

When the position of the sperm is detected by object detection, the processing device 200 performs segmentation on the captured image D1 (step S32). Here, the image analysis unit 210 extracts the head, middle segment/tail, and vacuole of the sperm by inputting the captured image D1 into a segmentation model obtained by deep learning. Note that FIG. 13 shows how the head H of the sperm, the middle segment/tail MT of the sperm, and the vacuole V are distinguished by segmentation.

After that, the processing device 200 measures the feature amounts of the head, midpiece/tail, and vacuole extracted in step S32 (step S33). Here, the image analysis unit 210 measures the feature amount through predetermined calculation processing using a rule-based model. The feature amount to be measured may be determined in advance regardless of the grading standard, or may be determined according to the grading standard used for grading in step S34. In addition, in FIG. 13, the length and width of the head are measured as the feature quantities of the head, and the length and width of the middle piece and tail are measured as the feature quantities of the middle piece and tail. The number of vacuoles is exemplified as a characteristic amount of vacuoles.

Once the feature amounts are measured, the processing device 200 reads out the grading criteria (step S34). Here, the image analysis unit 210 reads the grading criteria from the storage unit 230. The grading standard is stored in the storage unit 230 in a table format, such as the grading standard T1 shown in FIG. Contains information regarding one or more combinations. In addition, in the grading standard T1 shown in FIG. 14, the numerical range that should be satisfied by sperm graded to grade G1, which is the highest grade, is 4.5<HA<5.5 for head length HA; Examples include 3<HW<4 for the width of the part, V=0 for the number of vacuoles V, and NW<1.5 for the width NW of the middle piece/tail part. All units are μm.

As shown in Figure 14, the conditions under which sperm are graded to a specific grade (for example, grade G3) may be defined using the conditions under which sperm are graded to other grades (grade G1, grade G2, grade G4). good. Further, as shown in FIG. 14, a plurality of different conditions for grading to a specific grade (grade G4) may be defined as long as they do not contradict each other.

Note that when a plurality of grading standards are stored in the storage unit 230, the image analysis unit 210 selects a grading standard to be used for grading from the plurality of grading standards stored in the storage unit 230 according to the grading settings. may be determined. That is, in step S34, the image analysis unit 210 may determine a grading standard according to the grading settings, and read the determined grading standard from the storage unit 230. Grading settings may be performed, for example, by the embryologist directly selecting grading criteria, or by the embryologist logging into the microscope system 1 and setting user information (for example, the facility to which the user belongs). information). Hereinafter, the case where the grading standard T1 shown in FIG. 14 is read out will be described as an example.

Once the grading criteria are read, the processing device 200 grades the sperm (step S35). Here, the image analysis unit 210 grades the sperm based on the measured value of the feature amount measured in step S33 and the grading standard read out in step S34.

Once the sperm has been graded, the processing device 200 generates an auxiliary image according to the display settings (step S36). Here, the image generation unit 220 first determines information regarding the grading of sperm to be included in the auxiliary image according to the display settings. That is, the configuration of information regarding grading is changed according to the display settings.

The display settings can be arbitrarily set by the embryo culturist on the GUI as shown in FIG. 15, for example. In region R1, it is possible to set grade information to be included in the auxiliary image. The image generation unit 220 determines whether grade information is included in the information regarding grading and for which grade the grade information is included, depending on the settings of the region R1. In region R2, it is possible to set measurement value information to be included in the auxiliary image. The image generation unit 220 determines whether to include measurement value information in the information regarding grading and which feature quantity measurement value to include, depending on the settings of the region R2. Furthermore, when "only abnormal values" is selected, the image generation unit 220 may decide to extract only abnormal values from the measured values and include them in the information regarding grading.

Once the configuration of information regarding grading is determined, the image generation unit 220 generates an auxiliary image including information regarding the grading of the determined configuration and outputs it to the projection device 153. Here, the image generation unit 220 generates the auxiliary image so that information regarding sperm grading is displayed near the sperm region in the optical image and at a position that does not overlap with the sperm region. Note that the position of the information regarding grading may be determined based on the position of the sperm detected by object detection or the position of the part of the sperm extracted by segmentation.

Once the auxiliary image is generated, the microscope system 1 superimposes the auxiliary image on the image plane (step S24). Here, the projection device 153 superimposes the auxiliary image on the image plane on which the optical image is formed. Thereby, the user can simultaneously check the optical image O1 displayed on the image plane and the auxiliary image, for example, as shown in FIGS. 16 to 21.

By displaying the auxiliary image, it is possible to easily understand the quality of the morphology of the sperm from the auxiliary image. Therefore, the embryo culturist can select sperm from among the noteworthy sperm narrowed down by the auxiliary images, focusing mainly on the motility of the sperm ascertained from the optical image O1.

Note that the example shown in FIG. 16 is an example of an image displayed on the image plane when only the checkbox C11 in the area R1 is selected in the display setting GUI shown in FIG. 15. It shows how the auxiliary image A1 overlaps the optical image O1. The auxiliary image A1 includes only grade information indicating grade G1 sperm. With such display settings, the embryologist's attention can be focused on the highest grade sperm.

The example shown in FIG. 17 is an example of an image displayed on the image plane when check box C11 and check box C12 in area R1 are selected in the display setting GUI shown in FIG. 15. It shows how the auxiliary image A2 overlaps the optical image O1. The auxiliary image A2 includes grade information indicating grade G1 sperm and grade information indicating grade G2 sperm. With such display settings, a plurality of spermatozoa can be displayed as selection candidates even when there are few spermatozoa of the highest grade.

The example shown in FIG. 18 is an example of an image displayed on the image plane when check box C21, check box C22, check box C23, and check box C25 in area R2 are selected in the display setting GUI shown in FIG. It is. It shows how the auxiliary image A3 overlaps the optical image O1. Auxiliary image A3 shows the measured values of head length (HA), head width (HW), number of vacuoles (V), and midpiece/tail width (NW) for each sperm. It is included. In addition, so that you can understand at a glance whether a measured value is an abnormal value (that is, a measured value outside the numerical range that should be met for sperm of the highest grade), abnormal measured values are distinguished from other measured values. Displayed in different ways (for example, different colors, different font sizes, different fonts, addition of marks indicating abnormal values, etc.). With such display settings, the embryo cultivator can select sperm while recognizing the characteristic amounts of each sperm.

The example shown in FIG. 19 is an example of an image displayed on the image plane when only check box C26 in area R2 is selected in the display setting GUI shown in FIG. 15. It shows how the auxiliary image A4 overlaps the optical image O1. The auxiliary image A4 includes only abnormal values among the measured values of the feature quantities of the sperm. With such display settings, only the abnormal values of each sperm are displayed, so that the highest grade sperm without abnormal values can be easily distinguished. Furthermore, the degree of abnormality can be recognized from abnormal values for sperm of grades other than the highest.

In the example shown in FIG. 20, in the display setting GUI shown in FIG. 15, check box C11 and check box C12 in area R1 are selected, check box C21, check box C22, check box C23, and check box This is an example of an image displayed on the image plane when C25 is selected. It shows how the auxiliary image A5 overlaps the optical image O1. Auxiliary image A5 includes grade information indicating grade G1 sperm, grade information indicating grade G2 sperm, and for each sperm, head length (HA), head width (HW), and vacuole. number (V), and the measured values of the width (NW) of the middle piece and tail. With such display settings, it is possible to understand both the sperm grade and the measurement value.

The example shown in FIG. 21 is displayed on the image plane when check box C11 and check box C12 in area R1 are selected and check box C26 in area R2 is selected in the display setting GUI shown in FIG. 15. This is an example of an image. It shows how the auxiliary image A6 overlaps the optical image O1. The auxiliary image A6 includes grade information indicating sperm of grade G1, grade information indicating sperm of grade G2, and an abnormal value among the measured values of the feature quantity of the sperm. With such display settings, information about the recommendation level (grade) of sperm and the degree of abnormality can be provided with a minimum amount of display.

Note that the display settings described above are just examples, and other display settings may be possible. For example, it may be possible to set the display of grades G3 and G4 in addition to grades G1 and G2. Furthermore, display of measured values of feature quantities other than the feature quantities shown in FIG. 15 may also be settable.

Once the sperm are sorted in step S12, the user damages the tail of the sperm using RC40x observation to immobilize the sperm (step S13). Here, the user immobilizes the sperm by rubbing the tail of the sperm against the bottom of the Petri dish 310 with a pipette.

After that, the user may observe the morphology of the immobilized sperm in more detail and further select the sperm (step S14). Here, the user may use, for example, the intermediate magnification unit 160 to change the magnification to higher than 40 times, and may further select the sperm by observing at a higher magnification than in step S12. . Here, the microscope system 1 may support the embryonic cultivator's sperm selection work by performing the selection support process shown in FIG. 8 and displaying an auxiliary image on the image plane, similarly to step S12.

When the sperm sorting is completed, the user then takes the sorted sperm into the pipette 44, which is an injection pipette, and moves the observation position to the drop 303 (egg manipulation drop) (step S15), as shown in FIG. Complete the sequence of steps for sperm selection shown in .

When sperm sorting is completed, the user confirms the position of the spindle in preparation for sperm injection (step S5). Here, the user observes the egg selected in step S3 that is present in the drop 303 and confirms the position of the spindle of the egg. Specifically, the user presses the button 55 of the input device 50, for example, to switch the setting of the microscope system 1 to PO20x observation. Thereafter, the user changes the orientation of the oocyte spindle visualized by PO20x observation by operating the pipette 43, which is a holding pipette, so that it is positioned at the 12 o'clock or 6 o'clock direction. This is to prevent the spindle from being damaged by the pipette that is thrust into the egg from the 3 o'clock or 9 o'clock direction in step S6, which will be described later.

Finally, the user injects the sperm into the egg (step S6) and ends the ICSI. Here, the user, for example, presses the button 53 of the input device 50 to switch the setting of the microscope system 1 to MC20x observation. Thereafter, under MC20x observation, the user fixes the oocyte whose orientation was adjusted in step S5 with a pipette 43, which is a holding pipette, and pierces it with a pipette 44, which is an injection pipette. Thereafter, good spermatozoa are injected into the egg from the pipette 44.

After completing the series of ICSI procedures shown in FIG. 5, the user returns the sperm-injected eggs to the incubator and culture them. Further, the user may operate the processing device 200 using the input device 60 and the input device 70 to save information obtained by ICSI in the database server 20. For example, images of eggs injected with sperm, images of sorted sperm, ICSI work hours, patient information on sperm and eggs (mother's clinical data, test results of semen containing sperm, etc.), sperm and eggs, etc. Culture solution data (for example, type, concentration, PH, etc.) may be associated and stored in the database server 20.

As described above, the microscope system 1 grades the sperm by segmenting the sperm and measuring the morphological features, and displays the obtained information on the image plane as an auxiliary image. This allows the embryo cultivator to easily and uniformly judge the quality of the sperm in terms of morphology. Therefore, in conjunction with the sperm motility determined from the optical image, embryo culturists can select good sperm from a sample, and this can reduce disparities in fertilization rates among embryo culturists. .

Additionally, it is possible to easily accommodate grading standards that vary from facility to facility by simply rewriting the information stored in the memory. Therefore, it is possible to optimize the system and provide a high level of service to each facility. Furthermore, the feature values required for grading, which are performed using different standards for each facility, are measured using the same standards regardless of the facility. More specifically, by using a common AI independent of facilities for region extraction for measuring feature quantities, high accuracy and high-speed processing are both achieved. This makes it possible to provide a high level of service while reducing the cost of introducing the system to each facility.

Therefore, according to the microscope system 1, it is possible to effectively support the embryo cultivator's work of sorting sperm within a sample.

The embodiments described above are specific examples to facilitate understanding of the invention, and the present invention is not limited to these embodiments. Variations on the embodiments described above and alternatives to the embodiments described above may be included. In other words, the components of each embodiment can be modified without departing from the spirit and scope thereof. Further, new embodiments can be implemented by appropriately combining a plurality of components disclosed in one or more embodiments. Further, some components may be deleted from the components shown in each embodiment, or some components may be added to the components shown in the embodiments. Furthermore, the processing procedures shown in each embodiment may be performed in a different order as long as there is no contradiction. That is, the microscope system, projection unit, and sorting support method of the present invention can be modified and changed in various ways without departing from the scope of the claims.

In the embodiment described above, the microscope system 1 was illustrated, but the configuration of the microscope system is not limited to this example. For example, a microscope system 2 shown in FIG. 22 may be used. The microscope system 2 differs from the microscope system 1 in that it includes a microscope 400 instead of the microscope 100. The microscope 400 includes a projection unit 500 between a microscope main body 410 and a lens barrel 420.

The projection unit 500 is a projection unit for a microscope, and includes a superimposing section (splitter 151, lens 152, and projection device 153) corresponding to the projection unit 150 shown in FIG. 1, and an imaging section corresponding to the imaging unit 140 shown in FIG. (splitter 141 and imaging device 143), and an image processing section 510. The image processing section 510 functions as the image analysis section 210, image generation section 220, and storage section 230 shown in FIG.

The projection unit 500 and the microscope system 2 can also provide the same effects as the microscope system 1. Further, by using the projection unit 500, the above-described effects can be obtained by expanding an existing microscope system, so that the existing microscope system can be effectively utilized.

In the embodiment described above, an example was shown in which the projection device 153 projects the auxiliary image onto the image plane, but it is sufficient if the auxiliary image can be displayed on the image plane, and instead of the projection device 153, a transmissive liquid crystal placed on the image plane may be used. A device may also be used.

In the embodiment described above, an example was shown in which the image generation unit 220 changes the configuration of information related to grading included in the auxiliary image according to the display settings. The structure of the information may be changed. For example, if the length and width of the head, the number of vacuoles, and the width of the midpiece and tail are used in the grading criteria, an auxiliary image may be used to display the measured values of these features. may be generated.

As used herein, the expression "based on A" does not mean "based only on A," but "based at least on A," and furthermore, "based at least in part on A." It also means "te". That is, "based on A" may be based on B in addition to A, or may be based on a part of A.

1, 2: Microscope system 10: Microscope controller 20: Database server 30:

Display device

40, 50, 60, 70: Input device 80: Identification device 100, 400: Microscope 101: Eyepiece 140: Imaging unit 143: Imaging device 150 : Projection unit 153 : Projection device 160 : Intermediate magnification unit 170 : Eyepiece tube 200 : Processing device 201 : Processor 202 : Storage device 210 : Image analysis section 220 : Image generation section 230 : Storage section 300 : Sample 500 : Projection unit 510: Image processing units A1 to A6: Auxiliary image D1: Captured image H: Head IP: Image plane MT: Tail O1: Optical image V: Vacuole

Claims

a microscope that forms an image of a sample containing sperm;
an imaging device that acquires an image of the sample;
an image processing device that generates an auxiliary image including information regarding grading of the sperm based on the image acquired by the imaging device;
a superimposing device that superimposes the auxiliary image on an image plane on which the microscope forms the image;
The image processing device includes:
extracting one or more parts of the sperm from the image using a segmentation model generated by deep learning;
The method is characterized in that the sperm is graded based on the measured value of the feature amount measured from the one or more extracted portions and a pre-registered grading standard indicating the relationship between the feature amount and the grade. Microscope system.
The microscope system according to claim 1,
The grading criteria include information regarding one or more combinations of the feature quantity and a numerical range that the measured value of the feature quantity should satisfy in sperm of the highest grade,
The microscope system is characterized in that the information regarding the grading included in the auxiliary image includes a measured value of the feature quantity outside a numerical range to be satisfied.
The microscope system according to claim 2,
The microscope system is characterized in that the information regarding the grading included in the auxiliary image includes grade information indicating the grade of the sperm.
The microscope system according to claim 1,
The microscope system is characterized in that the image processing device changes the configuration of the information regarding the grading according to display settings.
The microscope system according to claim 1,
The microscope system is characterized in that the image processing device changes the configuration of information regarding the grading according to the grading standard.
The microscope system according to claim 3,
A microscope system characterized in that the grade information has different aspects for each grade.
The microscope system according to claim 3,
A microscope system characterized in that the grade information has a different color for each grade.
The microscope system according to claim 1,
The microscope system is characterized in that the auxiliary image includes information regarding the grading at a position that does not overlap with sperm in the image when displayed on the image plane.
The microscope system according to claim 1,
The microscope system is characterized in that the image processing device extracts at least a head, a middle segment, a tail, and a vacuole of the sperm from the image using the segmentation model.
The microscope system according to claim 9,
measuring feature quantities of at least the head, the middle piece and tail, and the vacuole extracted using the segmentation model using a rule-based model;
The feature quantities measured by the rule-based model include the length of the head, the width of the head, the width of the middle piece and the tail, the length of the middle piece and the tail, and the length of the head with respect to the head. A microscope system comprising at least one of the number of vacuoles, the inclination of the middle piece and the tail.
The microscope system according to claim 1,
The image processing device includes:
detecting the position of the object classified as the sperm from the image using an object detection model generated by deep learning;
A microscope system characterized in that the segmentation model is used to extract one or more parts of the sperm from a region in the image narrowed down by the object detection model.
The microscope system according to claim 1,
The image processing device includes:
comprising a rewritable memory for storing the grading criteria;
A microscope system characterized in that the grading standard is updated according to grading settings.
The microscope system according to claim 1,
The image processing device includes:
Equipped with rewritable memory that stores multiple grading standards,
A microscope system characterized in that the grading standard used for grading the sperm is determined from the plurality of grading standards according to grading settings.
The microscope system according to claim 1,
The microscope is an inverted microscope equipped with an eyepiece.
A microscope system characterized by:
A projection unit attached to a microscope,
an imaging section that obtains images of samples containing sperm;
an image processing unit that generates an auxiliary image including information regarding grading of the sperm based on the image acquired by the imaging unit;
a superimposing unit that superimposes the auxiliary image on an image plane on which the microscope forms the image;
The image processing unit includes:
extracting one or more parts of the sperm from the image using a segmentation model generated by deep learning;
The method is characterized in that the sperm is graded based on the measured value of the feature amount measured from the one or more extracted portions and a pre-registered grading standard indicating the relationship between the feature amount and the grade. projection unit.
forming an image of a sample containing sperm;
obtaining an image of the sample;
generating an auxiliary image containing information regarding grading of the sperm based on the acquired image;
superimposing the auxiliary image on an image plane on which the image is formed;
Generating the auxiliary image comprises:
extracting one or more parts of the sperm from the image using a segmentation model generated by deep learning;
grading the sperm based on a measured value of a feature amount measured from the one or more extracted portions and a pre-registered grading standard indicating a relationship between the feature amount and the grade. A selection support method characterized by:
A non-temporary recording medium that records a program,
The program is installed on a computer,
extracting one or more parts of the sperm from an image of the sample containing the sperm using a segmentation model generated by deep learning;
Executing a process of grading the sperm based on a measured value of a feature amount measured from the one or more extracted portions and a pre-registered grading standard indicating a relationship between the feature amount and the grade. A recording medium characterized by.