JP7091635B2 - Object detector, image analysis device, object detection method, image analysis method, program, and training data - Google Patents

Description

The present disclosure relates to a technique for analyzing an image of a sample.

In recent years, with the spread of time-lapse devices, it has become possible to observe changes over time in a sample. For example, Patent Document 1 describes a method of image processing cells in a culture vessel with a time-lapse device.

Japanese Unexamined Patent Publication No. 2009-152827

However, in the method of acquiring a photographed image while illuminating the object as in Patent Document 1, the appearance of the object in the photographed image differs depending on the position and direction of the illumination, so that the accuracy of image analysis is correct. There is a problem that is reduced.

It is a main object of the present disclosure to provide an image analysis apparatus capable of preventing a decrease in detection accuracy due to the influence of lighting.

The present application includes a plurality of means for solving the above-mentioned problems. For example, an object detector that detects a detection object contained in the sample from a photographed image of the sample is a sample at the time of learning. Learning is performed by the input data which is the captured image of the above and the teacher data prepared for each divided portion in which the region including the detection object included in the captured image is divided into a plurality of portions so as to include the detection object. At the time of detection, the captured image of the actual sample is received as input data, and the detection result of the detection target contained in the captured image is output.

The present application includes a plurality of means for solving the above problems, but to give another example, an image analysis device for detecting a detection target contained in the sample from the photographed image of the sample is a photographed image of the sample. The object detector is provided with an object detector for detecting an object to be detected contained in the sample, and the object detector includes input data which is a photographed image of the sample and the detection object included in the photographed image. It has been learned by a plurality of training data including the teacher data prepared for each divided portion in which the region is divided into a plurality of portions so as to include the detection target .
The present application includes a plurality of means for solving the above problems, but to give another example, an image analysis device for detecting a detection target contained in the sample from the photographed image of the sample is a photographed image of the sample. The object detector is provided with an object detector for detecting an object to be detected contained in the sample, and the object detector includes input data which is a photographed image of the sample and the detection object included in the photographed image. It has been trained by a plurality of training data including teacher data prepared for each divided portion in which the region is divided into a plurality of portions, and the divided portion includes a distribution of pixel values due to the structure of the detection object. ..

The present application includes a plurality of means for solving the above problems, but to give another example, the present application is provided with a computer and executed by an object detector that detects an object to be detected contained in the sample from a photographed image of the sample. At the time of learning, the program to be executed is divided into a plurality of parts so as to include the input data which is a photographed image of the sample and the area including the detection object included in the photographed image. The computer functions as a means for learning based on the teacher data prepared in the above, a means for receiving a photographed image of an actual sample as input data at the time of detection, and a means for outputting the detection result of the detection target contained in the photographed image. Let me.

The present application includes a plurality of means for solving the above problems, but to give another example, it is executed by an image analysis device equipped with a computer and detecting an object to be detected contained in the sample from a photographed image of the sample. Program causes the computer to function as an object detector that detects an object to be detected contained in the sample from the captured image of the sample, and the object detector uses input data that is a captured image of the sample. , The region including the detection target included in the captured image is divided into a plurality of portions so as to include the detection target, and the training data including the teacher data prepared for each divided portion has been learned. ..
The present application includes a plurality of means for solving the above problems, but to give another example, it is executed by an image analysis device equipped with a computer and detecting an object to be detected contained in the sample from a photographed image of the sample. Program causes the computer to function as an object detector that detects an object to be detected contained in the sample from the captured image of the sample, and the object detector uses input data that is a captured image of the sample. , The region including the detection target included in the captured image has been learned by a plurality of learning data including the teacher data prepared for each divided portion divided into a plurality of portions, and the divided portion is the detection. Includes the distribution of pixel values due to the structure of the object.

The present application includes a plurality of means for solving the above-mentioned problems, but to give another example, a computer is provided, and the probability for each number of detection objects contained in the sample is calculated from the photographed image of the sample. The trained model for operating the computer comprises a first neural network and a second neural network coupled so that the output from the first neural network is input. A plurality of neural networks including 1 input data which is a photographed image of the sample and teacher data prepared for each divided portion in which a region including the detection target included in the captured image is divided into a plurality of portions. It has been learned from the training data of the above, and it is configured to output a divided image in which a region including the detection target object included in the captured image is divided into a plurality of parts based on the captured image of the sample. The second neural network learns from the input data including the detection information in which the detection target is detected by the object detector and the teacher data showing the probability for each number of the detection objects included in the detection information. It has already been completed, and it is configured to output the probability for each number of the detection objects included in the detection information based on the detection information in which the detection object is detected by the object detector.

The present application includes a plurality of means for solving the above problems, but to give another example, an object executed by an object detector that detects an object to be detected contained in the sample from a photographed image of the sample. At the time of learning, the detection method is for each divided portion in which the input data which is the captured image of the sample and the region including the detection target included in the captured image are divided into a plurality of portions so as to include the detection target. It includes a step of learning with the prepared teacher data and a step of receiving a photographed image of an actual sample as input data at the time of detection and outputting a detection result of a detection object included in the photographed image.

The present application includes a plurality of means for solving the above problems, but to give another example, an image analysis method executed by an image analysis device that detects an object to be detected contained in the sample from a photographed image of the sample. The object detector comprises an object detection step of detecting an object to be detected contained in the sample from the photographed image of the sample, and the object detector includes input data which is a photographed image of the sample. , The region including the detection target included in the captured image is divided into a plurality of portions so as to include the detection target, and the training data including the teacher data prepared for each divided portion has been learned. ..
The present application includes a plurality of means for solving the above problems, but to give another example, an image analysis method executed by an image analysis device that detects an object to be detected contained in the sample from a photographed image of the sample. The object detector comprises an object detection step of detecting an object to be detected contained in the sample from the photographed image of the sample, and the object detector includes input data which is a photographed image of the sample. , The region including the detection target included in the captured image has been trained by a plurality of learning data including the teacher data prepared for each divided portion divided into a plurality of portions, and the divided portion is the detection. Includes the distribution of pixel values due to the structure of the object.

According to the present disclosure, it is possible to prevent a decrease in detection accuracy due to the influence of lighting in an image analysis device.

The configuration of the image analysis system of the fertilized egg according to the embodiment is shown. An example of a photographed image of a culture vessel and a fertilized egg is shown. It is a flowchart of the pronucleus number estimation process. It is a figure explaining the detection method of a fertilized egg. An example of learning data used for the detection process of the outer periphery of the pronucleus is shown. It is a figure explaining the 1st detection method which detects the peripheral part of the pronucleus. It is a figure explaining the 2nd detection method which detects the peripheral part of the pronucleus. It is a figure explaining the method of machine learning by dividing the anterior hypothalium outer peripheral part into four. It is a figure explaining the method of calculating the probability of the number of pronuclei. An example of the data used for the process of calculating the probability of the number of pronuclei is shown. It is a figure which shows the multi-stage structure of a neural network. An example of determining the number of pronuclei at each time is shown. It is a graph which shows the determination result of the number of pronuclei at all time. It is a table which shows the determination accuracy of the number of pronuclei according to the method of the pronucleus outer peripheral part detection processing. An example of displaying the number of pronuclei by discrete values is shown. This is an example of displaying the number of pronuclei by continuous values. This is a display example of an image showing the detection result of the outer peripheral portion of the pronucleus by the second detection method.

Hereinafter, preferred embodiments of the present disclosure will be described.
[System configuration]
FIG. 1 shows the configuration of an image analysis system for a fertilized egg according to an embodiment. As shown in the figure, the image analysis system 100 is roughly classified into a photographing device 50, a storage device 55, and an image analysis device 60.

The photographing apparatus 50 has an incubator 9 and photographs and stores an image of a fertilized egg in a culture vessel 10 placed in a chamber (chamber) (not shown) provided in the incubator 9. A plurality of fertilized eggs are placed in each culture container 10, and the photographing apparatus 50 associates them with identification information for identifying the fertilized eggs, for example, identification information of a well containing the fertilized eggs. The captured image is transmitted to the storage device 55. Specifically, the captured image 50 records the XY coordinates corresponding to the positions of the wells in the culture vessel 10 in advance, and each well may be identified from the XY coordinate information, or at the start of imaging, etc. In addition, the XY coordinates associated with each well position may be finely adjusted and the coordinates may be individually registered as the well position.

The photographing device 50 includes a light source 51, a camera 52, a control unit 53, and a communication unit 54. The light source 51 illuminates the fertilized egg in the culture vessel 10 from above for photographing. The light source 51 may be a point light source, a surface light source, a plurality of point light sources, or a surface light source. The illumination by the light source 51 is controlled by the control unit 53.

The camera 52 is movably arranged below the culture vessel 10 and photographs the fertilized egg in a state of being illuminated by the light source 51. The camera 52 is moved by an actuator (not shown) or the like. The movement of the camera 52 by the actuator and the shooting by the camera 52 are controlled by the control unit 53. The camera 52 photographs a plurality of fertilized eggs in the same culture vessel 10 at different times at a predetermined time from the start of observation. When photographing a plurality of fertilized eggs in the same culture vessel 10, the culture vessel 10 may move instead of the camera 52, and the culture vessel 10 does not move either the camera 52 or the culture vessel 10. The structure may be such that the entire fertilized egg inside is photographed.

The control unit 53 controls lighting by the light source 51, movement of the camera 52, and shooting by the camera 52. Further, when the control unit 53 acquires the captured image from the camera 52, the control unit 53 controls the communication unit 54 to transmit the captured image to the storage device 55.

The storage device 55 has a control unit, a storage unit, a communication unit, and the like (not shown), receives a photographed image transmitted from each photographing device 50, and stores the received photographed image. In this case, for example, the storage device 55 provides a folder for each incubator 9 provided in the photographing device 50, and stores the photographed image in association with the corresponding folder. The storage device 55 may further provide a subfolder for each chamber with respect to the above-mentioned folder, and classify and store captured images for each chamber. Further, when the storage device 55 receives the acquisition request of the captured image from the image analysis device 60, the storage device 55 transmits the captured image specified by the acquisition request to the image analysis device 60.

The image analysis device 60 is configured by a PC (Personal Computer) or the like, and detects the pronucleus contained in the fertilized egg based on the photographed image of the fertilized egg. The fertilized egg is an example of the "specimen" of the present disclosure, and the pronucleus is an example of the "detection target" of the present disclosure. The image analysis device 60 mainly includes a communication unit 61, an input unit 62, a storage unit 63, a display unit 64, and a control unit 65.

The communication unit 61 receives a photographed image of the fertilized egg from the storage device 55 under the control of the control unit 65. The input unit 62 is a keyboard, a mouse, a wheel device capable of rotation operation, a touch panel, a voice input device, and the like, and is an input signal for designating a fertilized egg used in the pronuclear number estimation process described later, a photographed image thereof, and the like. Is generated and supplied to the control unit 65. The storage unit 63 stores the program executed by the control unit 65 and the information necessary for executing the program. For example, the storage unit 63 stores a program required for the pronuclear number estimation process. The display unit 64 displays the detection result of the number of pronuclei based on the control of the control unit 65. The control unit 65 controls the entire image analysis device 60. The control unit 65 is an example of the "object detector" and the "number detector" of the present disclosure, and the display unit 64 is an example of the "result output unit" of the present disclosure.

[Shooted image]
FIG. 2 shows an example of a photographed image of the culture vessel 10 and the fertilized egg. The culture container 10 is a circular container, and a storage portion 11 is formed in the central portion. A plurality of wells 12 (5 × 5 = 25 in the example of FIG. 2) are formed in the accommodating portion 11. The well 12 is a recess for accommodating a fertilized egg, and one fertilized egg is accommodated in one well 12. As shown in FIG. 2A, the accommodating portion 11 is filled with the culture solution 13.

FIG. 2B shows a captured image 20 of a portion of one well 12. When the fertilized egg is housed in the well 12, the photographed image 20 includes the well 12 and the fertilized egg 30 in the well 12. In the example of FIG. 2B, the fertilized egg 30 contains two pronuclei 32. The photographing apparatus 50 generates and stores a plurality of photographed images of the same fertilized egg (that is, the same well 12) at predetermined time intervals by so-called time-lapse observation. The photographing device 50 shall capture an image for each well 12.

[Pre-nuclear number estimation process]
Next, the pronuclear number estimation process will be described. The pronucleus number estimation process is a process of detecting the pronuclei contained in the fertilized egg based on the photographed image of the fertilized egg and estimating the number of pronuclei, and is executed by the image analysis device 60. FIG. 3 is a flowchart of the pronucleus number estimation process. This process is realized by the control unit 65 of the image analysis device 60 executing a program prepared in advance.

First, the control unit 65 acquires a photographed image of the fertilized egg to be processed at each time from the storage device 55 (step S11). Specifically, the control unit 65 acquires captured images obtained from the storage device 55 at predetermined time intervals during a predetermined time (for example, about 24 hours) after fertilization for the fertilized egg to be processed.

Next, the control unit 65 performs a fertilized egg detection process (step S12). The fertilized egg detection process is a process of detecting a fertilized egg from each image obtained in step S11. In the present embodiment, the control unit 65 detects the fertilized egg by Hough transform. The Hough transform is a method for detecting a specific structure (straight line, perfect circle, ellipse, etc.) in an image, and also detects an object that is partially hidden or slightly deformed. Can be done. Since the fertilized egg is close to a perfect circle, the control unit 65 detects the perfect circle by Hough transform here.

FIG. 4A shows an example of a photographed image 20 of a fertilized egg. The control unit 65 analyzes the brightness (luminance) of the pixels inside the well 12, and detects a pixel whose brightness suddenly changes from the adjacent pixel as a circumferential candidate pixel. Then, as shown in FIG. 4B, the control unit 65 detects the perfect circle that most overlaps with the circumferential candidate pixels as the fertilized egg 30. The control unit 65 detects a perfect circle belonging to a general fertilized egg size range as a fertilized egg. Since the Hough transform performs detection based on the degree of overlap with the circumferential candidate pixels, it is possible to correctly detect the fertilized egg 30 even if a part of the fertilized egg is hidden or partially deformed. can.

Next, the control unit 65 performs a pronucleus peripheral portion detection process as a process of detecting a region including a detection target (step S13). The pronucleus outer peripheral portion detection process is a process of detecting a region including the outer peripheral portion that determines the outer peripheral contour shape of the pronucleus as a detection target existing in the fertilized egg 30 detected in step S12 (“pronucleus outer peripheral portion”). It is called "detection".) Here, machine learning (deep learning) using a neural network is used to detect the outer periphery of the pronucleus, but the present invention is not limited to the neural network, and other machine learning (for example, SVM: support vector machine) is used. , RAMDOM FOREST: Random Forest) algorithm can also be used. Although the pronucleus is not a perfect circle or an ellipse, the outer peripheral shape often changes smoothly. Therefore, although it is not actually circular, in the front nucleus outer peripheral portion detection process, a donut-shaped circular shape such as a circle or an ellipse may be applied to detect the front nucleus outer peripheral portion. Hereinafter, two examples of the detection method of the outer peripheral portion of the pronucleus will be described.

(First detection method)
FIG. 5A shows an example of learning data used for machine learning of the outer periphery of the pronucleus by the first detection method. The training data includes training input data and teacher data which is a target output from the neural network. As shown in the figure, as the input data for learning, an image 25 of the fertilized egg 30 (hereinafter, also referred to as “fertilized egg image”) 25 is used. The fertilized egg image 25 is an image obtained by cutting out a portion of the fertilized egg 30 from the photographed image 20 including the entire well 12 shown in FIG.

FIG. 6 schematically shows a method of detecting the outer peripheral portion of the pronucleus by the first detection method. The detection of the outer periphery of the pronucleus uses a neural network 71 learned by supervised machine learning. First, at the time of learning, the learning data shown in FIG. 5A, that is, the learning input data and the teacher data are given to the neural network 71 and trained. At this time, specifically, as shown in FIG. 6, a region of a predetermined size (for example, a region of 30 × 30 pixels; hereinafter referred to as an “image patch”) 21 is extracted from the fertilized egg image 25, and the image is taken. The training input data is input to the neural network 71 in units of the patch 21. The image patch 21 is cut out while shifting the position one pixel at a time in the fertilized egg image 25. On the other hand, the teacher data is prepared as a pixel value (white / black) at the center of each image patch 21. The teacher data shown in FIG. 5A is obtained by integrating the pixel values indicating the target output for each image patch 21 for the fertilized egg image 25. In the integrated teacher data, the region including the detection target includes the region including the outer peripheral portion that determines the outer peripheral contour shape of the detection target, and particularly the region including the outer peripheral portion, the luminance distribution due to the structure of the outer peripheral portion is obtained. It is preferable to be able to identify the including region (for example, the region where light and darkness is generated due to the outer peripheral portion and the outer peripheral portion around the outer peripheral portion, and the shadow portion). In this way, the control unit 65 gives the learning input data and the teacher data prepared in advance to the neural network 71 in units of the image patch 21 to perform learning.

When the training of the neural network 71 is completed using the plurality of training data, the trained neural network 71 is then used to perform an actual calculation, that is, to detect the outer periphery of the pronucleus from the fertilized egg image 25. .. Specifically, the control unit 65 inputs the actual fertilized egg image 25 into the trained neural network 71, and outputs the actual detection result. As a result, as shown in FIG. 6, the pronucleus outer peripheral portion 32a contained in the fertilized egg is detected from the actual fertilized egg image 25.

(Second detection method)
FIG. 5B shows an example of learning data used for machine learning of the outer periphery of the pronucleus by the second detection method. The training data includes training input data and teacher data which is a target output from the neural network. Here, as the input data for learning, the fertilized egg image 25 is used as in the first detection method. On the other hand, as the teacher data, as shown in the figure, teacher data is prepared for each portion (referred to as "divided portion") in which the outer peripheral portion of the anterior hypothalium is divided into four parts. Specifically, the teacher data whose target output is the upper left portion of the outer periphery of the anterior nucleus is defined as "A channel teacher data", and the teacher data whose target output is the upper right portion is defined as "B channel teacher data". , The teacher data whose target output is the lower left portion is referred to as "C channel teacher data", and the teacher data whose target output is the lower right portion is referred to as "D channel teacher data". Further, the teacher data whose target output is a portion other than the outer peripheral portion of the anterior hypothalium is referred to as “E-channel teacher data”. Although the teacher data of the E channel is not essential, there is an effect that the learning efficiency can be improved by using the teacher data. Here, as in the first detection method, the teacher data of each channel includes a region including an outer peripheral portion that determines the outer peripheral contour shape of the detection target as a region including the detection target, and particularly a region including the outer peripheral portion. As a result, it is preferable to be able to identify a region including a luminance distribution due to the structure of the outer peripheral portion (for example, a region where light and darkness is generated due to the outer peripheral portion and the outer peripheral portion around the outer peripheral portion, and a shadow portion). The shadow portion actually appears as light and darkness generated by the intensity modulation of the illumination light by the light source 51 by the detection object. Therefore, the shadow portion of the detection target is shown as a distribution of pixel values, for example, luminance values in the fertilized egg image 25. Then, the neural network 71 is trained using the five teacher data of the A to E channels for one learning input data.

FIG. 7 schematically shows a method of detecting the outer peripheral portion of the pronucleus by the second detection method. The learned neural network 71 is used to detect the outer periphery of the pronucleus. At the learning stage, the learning data shown in FIG. 5B, that is, the learning input data and the teacher data are given to the neural network 71 and trained. At this time, the neural network 71 is trained using the five teacher data of the A to E channels for one input data. Also in the second detection method, as shown in FIG. 7, input data is given to the neural network 71 in units of image patches 21 of a predetermined size for learning.

When the learning of the neural network 71 is completed using the plurality of training data, next, the learning of the neural network 71 is used to detect the outer periphery of the pronucleus from the actual fertilized egg image 25. Specifically, the control unit 65 inputs the fertilized egg image 25, which is the target of the actual analysis, into the trained neural network 71, and outputs the actual detection result. As a result, as shown in FIG. 7, detection results corresponding to each of the A to E channels can be obtained from the actual fertilized egg image 25.

In the second detection method, the reason why the teacher data is prepared for each divided portion of the anterior hypothalamic peripheral portion and machine learning is performed is as follows. First, by dividing the outer peripheral portion of the pronucleus, the influence of illumination by the light source 51 can be reduced. FIG. 8A schematically shows how the pronucleus 32 is illuminated by the light source 51. In the photographing apparatus 50, since the light source 51 is fixedly arranged, a bright region and a dark region are formed around and inside the pronucleus 32 in the fertilized egg 30 depending on the positional relationship between the well 12 in which the fertilized egg 30 is housed and the light source 51. There is an area. That is, as shown in FIG. 8A, of the periphery and the inside of the pronucleus 32, the side exposed to the light from the light source 51 is a bright region, and the opposite side is a dark region. Also, the area between the bright area and the dark area is an area where the brightness changes moderately. As described above, even in one pronucleus 32, there are regions having different brightness depending on the relationship with the position of the light source 51, so it is preferable to learn them separately as regions having different properties in machine learning. Therefore, in the second detection method, teacher data is prepared for each divided portion in which the region including the periphery and the inside of the pronucleus 32 is divided into four, and machine learning is performed. This makes it possible to prevent a decrease in detection accuracy due to a difference in the illumination state due to the light source 51.

In this case, theoretically, it is preferable to divide the outer peripheral portion of the pronucleus based on the direction of the light source 51. For example, the outer peripheral portion of the anterior hypothalium is divided into a portion close to the light source 51, a portion far from the light source 51 on the opposite side thereof, and two intermediate portions between them. As another method, the outer peripheral portion of the pronucleus is evenly divided without considering the position of the illumination light from the light source 51 so that learning can be performed in consideration of the influence of the brightness distribution of the illumination light. May be good. In this case, it is effective to increase the number of divisions, and it is particularly desirable that the number of divisions is 4 or more. For example, the outer peripheral portion of the anterior hypothalium is divided into eight parts, and machine learning is performed using the teacher data for each divided part. In this method, the direction of the actual light source 51 is not considered, but by increasing the number of divisions, the influence of the illumination light from the light source 51 can be dispersed and the influence on the detection of each division portion can be reduced.

Another reason for performing machine learning by dividing the outer peripheral portion of the pronucleus is that it is possible to secure the detection accuracy when the two outer peripheral portions of the pronucleus overlap and look like an ellipse. Since the two pronuclei appearing in the fertilized egg 30 are adjacent to each other, some of them may overlap and appear as one ellipse as a whole. In such a case, it is difficult to distinguish whether there are two circles corresponding to the pronucleus or one large ellipse. In this respect, by dividing the outer peripheral portion of the pronucleus and performing detection for each divided portion, it is possible to correctly detect two pronuclei that partially overlap and look like an ellipse.

In the above example, the outer peripheral portion of the anterior hypothalium is divided into 4 or 8, but the number of divisions is not limited to these, and other divisions such as 2 divisions, 3 divisions, and 6 divisions may be used. However, as the number of divisions increases, the processing load in machine learning and actual detection of the outer periphery of the pronucleus increases, so it is desirable to determine the number of divisions in relation to the processing load. Further, in the above example, the outer peripheral portion of the anterior hypothalium is evenly divided (for example, in the case of four divisions, it is divided into 90 degree portions), but the size of each divided portion may be different. For example, as shown in FIG. 8B, in consideration of the distribution of brightness due to the illumination light, the bright divided portion (B channel) and the dark divided portion (C channel) are enlarged, and the remaining two divided portions (the remaining two divided portions (channel B) are enlarged. A channel and D channel) may be made smaller accordingly.

Returning to FIG. 3, when the outer peripheral portion of the pronucleus is detected, the control unit 65 then performs the pronuclear number probability calculation process (step S14). The probability calculation process for the number of pronuclei is a process for calculating the probability of the number of pronuclei. Here, the control unit 65 calculates the probability of the number of pronuclei by machine learning. FIG. 9 schematically shows a method of calculating the probability of the number of pronuclei. FIG. 10 shows an example of input / output data in the pronuclear number probability detection process. The learning data includes the fertilized egg image 25 and the detection result of the outer periphery of the pronucleus of any of the channels A to D as input data for learning. The fertilized egg image 25 is not essential. The teacher data has a probability of 0, 1 and 2 pronuclei (denoted as "0PN", "1PN" and "2PN", respectively). At the time of learning, the neural network 72 is trained using a large number of the above learning data.

At the time of calculation, the actual fertilized egg image 25 and the detection result of the actual pronucleus outer periphery of any of the corresponding channels A to D are input to the trained neural network 72 as input data. In this case as well, the fertilized egg image 25 is not essential. The neural network 72 calculates and outputs the probability of the number of pronuclei based on the actual fertilized egg image 25 and the detection result of the actual pronucleus outer periphery. Thus, for one fertilized egg image 25, the probability that the number of pronuclei contained in the fertilized egg image 25 is 0 (0PN), the probability that it is 1 (1PN), and the probability that it is 2 (2PN). Are obtained respectively.

As described above, the present embodiment is characterized in that a multi-stage configuration of two neural networks is used in the pronuclear number estimation process. That is, as shown in FIG. 11, the trained first neural network 71 generates an actual pronucleus outer periphery detection result from an actual fertilized egg image and outputs it to the trained second neural network 72. .. Then, the trained second neural network 72 calculates and outputs the actual probabilistic number of pronuclei from the actual detection result of the outer periphery of the pronucleus.

Next, the control unit 65 determines whether or not the processing of steps S12 to S14 has been completed for all the fertilized egg images 25 prepared in advance (step S15). When the processing is not completed for all the fertilized egg images 25 (step S15: No), the processing returns to step S12, and the control unit 65 executes the processing of steps S12 to S14 for the next fertilized egg image 25.

In this way, when the processing of steps S12 to S14 is completed for all the fertilized egg images 25 (step S15: Yes), the control unit 65 integrates the probabilities of the number of pronuclei calculated for each fertilized egg image 25 and each time. The number of pronuclei in (step S16) is determined. Specifically, first, the control unit 65 graphs the probabilities of the number of pronuclei (0PN, 1PN, 2PN) at all times as illustrated in FIG. Next, the control unit 65 determines the number of pronuclei having the maximum probability under a predetermined prior knowledge (constraint) based on the graph of the number of pronuclears. In this embodiment, the transition of the most probable pronucleus number is calculated from the probability of the pronucleus number calculated at all times and the prior knowledge (constraint condition) by the hidden Markov model.

Specifically, in this embodiment, at least one of the following is used as prior knowledge (constraint) based on the nature of the pronucleus in the fertilized egg.
(1) The number of pronuclei that can occur is 0PN, 1PN, and 2PN, and the number of pronuclei at the start of observation is 0PN.
(2) The number of pronuclei may increase with the passage of time, but it does not decrease.
(3) Statistically, there are times when pronuclei are likely to occur and times when pronuclei are likely to disappear. Typically, the pronucleus develops about 5 hours after the start of observation and disappears about 20 hours.
(4) Statistically, the frequency of occurrence of 0PN, 1PN, and 2PN is known. Typically, the frequency of occurrence of 0PN, 1PN, and 2PN is about 10%, 10%, and 80%, respectively.

FIG. 12A shows an example of determining the number of pronuclei using the above-mentioned prior knowledge (constraint condition). It is assumed that the transition of the probability of each pronucleus number (0PN, 1PN, 2PN) is obtained as shown in each graph of FIG. 12 (A). In this example, the probability of 1PN temporarily increases around 15 to 18 hours from the start of observation, but considering the above prior knowledge (1) to (4), the judgment result may be 2PN. It is reasonable.

FIG. 12B shows another determination example of the number of pronuclei. It is assumed that the transition of the probability of each pronucleus number (0PN, 1PN, 2PN) is obtained as shown in each graph of FIG. 12B. In this case, the probabilities frequently switch between 1PN and 2PN in the period of 12 to 22 hours from the start of observation, but considering the above prior knowledge (1) to (4), the judgment result should be 2PN. Is appropriate.

In this way, the control unit 65 determines the number of pronuclei at each time using the hidden Markov model. Then, when the number of pronuclei at each time is obtained, the control unit 65 then outputs the maximum number of pronuclei at all times (step S17). That is, the control unit 65 outputs the maximum number of pronuclei over the entire time as the final number of pronuclei of the fertilized egg. FIG. 13A is an example of a graph showing the determination result of the number of pronuclei at each time for a fertilized egg. In the example of FIG. 13A, since the maximum number of pronuclei over the previous time is 2PN, the determination result of the number of pronuclei of the fertilized egg is 2PN. FIG. 13B is a graph showing the determination results of the number of pronuclei at each time for other fertilized eggs. In the example of FIG. 13B, since the maximum number of pronuclei over the entire time is 1PN, the determination result of the number of pronuclei of the fertilized egg is 1PN. In this way, the pronucleus number is estimated for one fertilized egg, and the pronuclear number estimation process is completed.

FIG. 14 is a table showing the determination accuracy of the number of pronuclei according to the method of detecting the outer peripheral portion of the pronucleus according to the present embodiment. In the table of FIG. 14, the “original image” is the result of detecting the number of pronuclei by image analysis of the fertilized egg image 25 without detecting the outer periphery of the pronucleus by machine learning. In this case, the correct answer rate for determining the number of pronuclei is about 40 to 60%, and the correct answer rate for determining 2PN is only 52%.

Next, "original image + pronucleus outer peripheral portion" means that the above-mentioned first detection method (see FIG. 6) is executed in the pronucleus outer peripheral portion detection process, that is, the pronucleus outer peripheral portion is divided. This is the result when the outer peripheral part of the anterior nucleus is detected by machine learning as the entire outer peripheral part of the anterior nucleus. In this case, the correct answer rate for determining the number of pronuclei is 60% or more, and the correct answer rate for determining 2PN is 74%. Therefore, the determination accuracy is higher than that in the case where the outer peripheral portion of the pronucleus is not detected.

Next, "original image + pronucleus outer periphery (4 divisions)" means that the above-mentioned second detection method (see FIG. 7) is executed in the pronucleus outer periphery detection process, that is, the forearm outer periphery. This is the result when the part is divided into four parts and the outer peripheral part of the anterior hypothalium is detected by machine learning. In this case, the correct answer rate for determining the number of pronuclei is 80% or more, and the correct answer rate for determining 2PN is 92%. Therefore, it can be seen that the determination accuracy can be further improved by dividing the outer peripheral portion of the anterior nucleus and detecting the outer peripheral portion of the anterior nucleus.

[Display of the number of pronuclei]
Next, a display example of the number of pronuclei obtained by the pronuclear number estimation process will be described. The control unit 65 displays the number of pronuclei obtained by the pronuclear number estimation process on the display unit 64 of the image analysis device 60. FIG. 15 shows a display example when the number of pronuclei is displayed as a discrete value. FIG. 15A is an example showing the transition of the number of pronuclei over time by a line graph. FIG. 15B is an example showing the transition of the number of pronuclei over time by color coding. FIG. 15C is an example showing the transition of the number of pronuclei over time by coloring with a gradation. In this example, the period during which the number of pronuclei changes is represented by a gradation. FIG. 15 (D) is an example showing the transition of the number of pronuclei over time by the area of color. In this example, the period during which the number of pronuclei changes is represented by the change in area.

FIG. 16 is an example in which the number of pronuclei is displayed by continuous values. The horizontal axis shows the elapsed time from the start of observation, and the vertical axis shows the number of pronuclei. In this example, by presenting information on a time zone in which the number of pronuclei is difficult to be determined as a continuous value, it is possible to provide more detailed information that can be used for visual judgment by an observer such as an embryo culture technician.

FIG. 17 is an example of an image showing the detection result of the outer peripheral portion of the pronucleus by the above-mentioned second detection method. Specifically, the actual detection results of the A to D channels obtained by using the trained neural network 71 shown in FIG. 7 are displayed as one image. Here, the detection results of each channel are displayed in different display modes so that the observer can distinguish them. In FIG. 17A, each divided portion in which the outer peripheral portion of the anterior hypothalium is divided into four is shown by different hatching. FIG. 17B is an enlarged view of one pronucleus portion in FIG. 17A, which is schematically shown by a circle. As shown in FIG. 17B, the actual detection results of channels A to D are displayed by different hatching. Here, for convenience of illustration, each divided portion is distinguished by hatching, but in reality, each divided portion may be distinguished and displayed by a different color or the like. By displaying each divided portion separately by color coding or the like in this way, it becomes easy to distinguish the pronuclei when the two pronuclei overlap. Further, since one pronucleus is displayed in a pattern of four colors, the number of pronuclei can be easily detected by the number of the patterns. Further, if each pronucleus is displayed in the same four-color pattern, it can be considered that the detection process of the outer periphery of the pronucleus is performed correctly. Therefore, by looking at this image, the pronucleus number estimation process is performed. There is also the advantage that the reliability of the can be indirectly confirmed.

[Modification example]
In the above pronucleus number estimation process, the pronucleus number probability is calculated by the pronucleus number probability calculation process in step S14, and the pronucleus number at each time is determined based on the pronuclear number probability in step S16. However, instead, the number of pronuclei may be determined by a method that does not use probability, and the result may be output.

10 Culture container 12 wells 20 Photographed image 25 Fertilized egg image 30 Fertilized egg 50 Photographing device 51 Light source 52 Camera 55 Storage device 60 Image analysis device 64 Display unit 65 Control unit

Claims (20)

An object detector that detects an object to be detected contained in the sample from a photographed image of the sample.
At the time of learning, a teacher prepared for each divided portion in which the input data which is the captured image of the sample and the region including the detection target included in the captured image are divided into a plurality of portions so as to include the detection target. Learning with data,
An object detector that receives a captured image of an actual sample as input data at the time of detection and outputs the detection result of the detected object contained in the captured image. An image analysis device that detects an object to be detected contained in the sample from the captured image of the sample.
A target detector for detecting a detection target contained in the sample from the photographed image of the sample is provided.
The object detector divides the input data, which is a photographed image of the sample, and the region including the detection object contained in the photographed image into a plurality of portions so as to include the detection object, for each divided portion. An image analysis device that has been trained by a plurality of training data including the prepared teacher data. An image analysis device that detects an object to be detected contained in the sample from the captured image of the sample.
A target detector for detecting a detection target contained in the sample from the photographed image of the sample is provided.
The object detector includes input data which is a photographed image of the sample and teacher data prepared for each divided portion in which a region including the detection object included in the captured image is divided into a plurality of portions. It has been trained by multiple training data and has been trained.
The divided portion is an image analysis device including a distribution of pixel values due to the structure of the detection object. The image analysis apparatus according to claim 3, wherein the divided portion includes a distribution of pixel values due to the structure of the outer peripheral portion of the detection object. The image analysis apparatus according to any one of claims 2 to 4, wherein the divided portion is a portion obtained by dividing the outer peripheral portion of the detection object in a direction perpendicular to the circumferential direction. The image analysis according to any one of claims 2 to 5, wherein the object detector divides a region including the detection object and displays it in an identifiable manner in the same manner as the divided portion included in the learning data. Device. The image analysis apparatus according to any one of claims 2 to 6, further comprising a result output unit for outputting number information regarding the number of detection objects detected by the object detector. A number detector for calculating the probability for each number of the detection objects from the detection information in which the detection target is detected by the object detector is provided.
The result output unit outputs the number information based on the probability of each number detected by the number detector.
The number detector is a learning including input data including detection information in which a detection target is detected by the object detector, and teacher data indicating a probability for each number of the detection objects included in the detection information. The image analysis apparatus according to claim 7, which has been learned from the data. A number detector for calculating the probability for each number of the detection objects from the detection information in which the detection target is detected by the object detector is provided.
The result output unit outputs the number information based on the probability of each number detected by the number detector.
The number detector is
At the time of learning, learning is performed by input data including detection information in which the detection object is detected by the object detector and teacher data indicating the probability of each number of the detection objects included in the detection information.
The image analysis according to claim 7, wherein at the time of detection, the detection information in which the detection target is detected by the object detector is received as input data, and the probability for each number of the detection targets included in the detection information is output. Device. The result output unit analyzes the probability of each number output by the number detector under a predetermined constraint condition over a predetermined period, determines the number of the detection target objects, and outputs the number information. Item 8. The image analysis apparatus according to Item 8. The image analysis apparatus according to claim 10, wherein the result output unit determines that the maximum number of the number obtained in the predetermined period is the number of detection objects contained in the sample. The image analysis apparatus according to any one of claims 7 to 11, wherein the result output unit displays a graph showing a continuous value of the number of detection objects over time as the number information. The image analysis according to any one of claims 7 to 12, wherein the result output unit displays an image showing an object detected by the object detector in a different display mode for each divided portion. Device. A program equipped with a computer and executed by an object detector that detects an object to be detected contained in the sample from a photographed image of the sample.
At the time of learning, a teacher prepared for each divided portion in which the input data which is the captured image of the sample and the region including the detection target included in the captured image are divided into a plurality of portions so as to include the detection target. A means of learning with data,
A means of receiving a captured image of an actual sample as input data at the time of detection and outputting the detection result of the detection target contained in the captured image.
A program that makes the computer function as. A program equipped with a computer and executed by an image analysis device that detects an object to be detected contained in the sample from the captured image of the sample.
The computer is made to function as an object detector for detecting an object to be detected contained in the sample from a photographed image of the sample.
The object detector divides the input data, which is a photographed image of the sample, and the region including the detection object contained in the photographed image into a plurality of portions so as to include the detection object, for each divided portion. A program that has been trained by multiple training data including the prepared teacher data. A program equipped with a computer and executed by an image analysis device that detects an object to be detected contained in the sample from the captured image of the sample.
The computer is made to function as an object detector for detecting an object to be detected contained in the sample from a photographed image of the sample.
The object detector includes input data which is a photographed image of the sample and teacher data prepared for each divided portion in which a region including the detection object included in the captured image is divided into a plurality of portions. It has been trained by multiple training data and has been trained.
The divided portion is a program including a distribution of pixel values due to the structure of the detected object. A trained model for operating the computer so as to have a computer and calculate the probability for each number of detection objects contained in the sample from the captured image of the sample, the first neural network and the above. It consists of a second neural network combined so that the output from the first neural network is input.
The first neural network obtains input data which is a photographed image of the sample and teacher data prepared for each divided portion in which a region including the detection object included in the captured image is divided into a plurality of portions. It has been trained by a plurality of training data including the sample, and is configured to output a divided image obtained by dividing a region including the detection object included in the captured image into a plurality of portions based on the captured image of the sample. ,
The second neural network is based on input data including detection information in which the detection target is detected by the object detector and teacher data indicating the probability of each number of the detection targets included in the detection information. It is configured to output the probability for each number of the detection objects included in the detection information based on the detection information that has been learned and the detection object is detected by the object detector. Trained model. It is an object detection method executed by an object detector that detects an object to be detected contained in the sample from a photographed image of the sample.
At the time of learning, a teacher prepared for each divided portion in which the input data which is the captured image of the sample and the region including the detection target included in the captured image are divided into a plurality of portions so as to include the detection target. The process of learning with data and
At the time of detection, the process of receiving the captured image of the actual sample as input data and outputting the detection result of the detection target contained in the captured image, and
An object detection method comprising. It is an image analysis method executed by an image analysis device that detects an object to be detected contained in the sample from a photographed image of the sample.
The object detector comprises an object detection step of detecting an object to be detected contained in the sample from a photographed image of the sample.
The object detector divides the input data, which is a photographed image of the sample, and the region including the detection object contained in the photographed image into a plurality of portions so as to include the detection object, for each divided portion. An image analysis method that has been trained by a plurality of training data including the prepared teacher data. It is an image analysis method executed by an image analysis device that detects an object to be detected contained in the sample from a photographed image of the sample.
The object detector comprises an object detection step of detecting an object to be detected contained in the sample from a photographed image of the sample.
The object detector includes input data which is a photographed image of the sample and teacher data prepared for each divided portion in which a region including the detection object included in the captured image is divided into a plurality of portions. It has been trained by multiple training data and has been trained.
The divided portion is an image analysis method including a distribution of pixel values due to the structure of the detection object.

Images