JP2021126074A - Dyed image generation system, dyed image generation method and program - Google Patents

Abstract

To provide a technique capable of analyzing a fertilized egg which is dyed without extinction of the fertilized egg.SOLUTION: A dyed image generation system comprises: a visible light radiation image acquiring part for analysis which acquires a visible light radiation image for analysis being an image obtained by imaging, by radiating visible light to a fertilized egg to be analyzed; and a dyed image generating part for, inputting the visible light generation image for analysis to a machine learning model which is machine learned on the basis of a visible light radiation image for learning which is an image obtained by imaging by radiating visible light to a fertilized egg for learning in a solvent including dyeing solution, and an ultraviolet radiation image for learning being an image obtained by imaging by radiating ultraviolet to the fertilized egg for learning, the machine learning model generating an image of the dyed fertilized egg from the image of the fertilized egg under illumination by the visible light, and generating a dyed image being an image in which the fertilized egg to be analyzed is dyed.SELECTED DRAWING: Figure 1

Description

The present invention relates to a stained image generation system, a stained image generation method, and a stained image generation program.

In fertilized eggs obtained by artificial insemination, the fertilized egg is expected to have a high fertility rate. In order to analyze whether or not a fertilized egg is normal and whether or not there is a chromosomal abnormality, a technique for performing analysis based on various characteristics of a fertilized egg has been conventionally known. For example, Patent Document 1 discloses a technique for performing analysis based on a change over time in a fertilized egg. Patent Document 2 discloses a technique for performing analysis based on the amount of cell cortex of a fertilized egg. Patent Document 3 discloses a technique for generating a classifier based on normal image data and abnormal image data of a fertilized egg.

Japanese Unexamined Patent Publication No. 2018-171039 Japanese Unexamined Patent Publication No. 2019-17289 JP-A-2019-97425

The above-mentioned prior art does not directly visually recognize and analyze the state of chromosomes. It is preferable to directly visually recognize the state of the chromosome for analysis of whether or not the fertilized egg is normal and whether or not there is a chromosomal abnormality.

In order to directly observe the chromosomes, fertilized eggs are stained with a staining solution. According to the staining solution, the chromosomes fluoresce and are stained under ultraviolet illumination, so that the structure of the chromosomes and the like can be visually recognized and analyzed. However, when the fertilized egg is stained with a staining solution, the embryo is killed, so that the fertilized egg after the analysis cannot be used.
The present invention has been made in view of the above problems, and an object of the present invention is to make it possible to analyze a fertilized egg in a stained state without killing the fertilized egg.

In order to achieve the above-mentioned object, the stained image generation system includes an analysis visible light irradiation image acquisition unit that acquires an analysis visible light irradiation image which is an image taken by irradiating the fertilized egg to be analyzed with visible light. The learning visible light irradiation image which is an image taken by irradiating the learning fertilized egg in a solvent containing a staining solution with visible light and the learning ultraviolet irradiation image which is an image taken by irradiating the learning fertilized egg with ultraviolet light. A machine learning model that is machine-learned based on an image and that generates a stained image of a fertilized egg from an image of a fertilized egg under visible light illumination is irradiated with visible light for analysis. It includes a stained image generation unit that inputs an image and generates a stained image that is an image in which the fertilized egg to be analyzed is stained.

That is, in the stained image generation system, a machine learning model that generates an image in which the fertilized egg is stained is generated in advance from an image taken by irradiating the fertilized egg with visible light. Therefore, if the fertilized egg to be analyzed is photographed under visible light illumination and input to the machine learning model, it is possible to generate a stained image which is an image in which the fertilized egg to be analyzed is stained. By analyzing the stained image, it is possible to estimate whether or not the fertilized egg to be analyzed is normal, whether or not there is a chromosomal abnormality, and the like. Since the staining solution is not used for the fertilized egg to be analyzed, the fertilized egg is not killed by the analysis. Therefore, it is possible to improve the birth rate in artificial insemination in infertility treatment and artificial insemination in the livestock industry by selecting fertilized eggs that are highly likely to be normal from the fertilized eggs to be analyzed.

It is a schematic block diagram of a dyed image generation system. It is a block diagram of a stained image generation system. It is a figure explaining the generative model and the discriminative model. It is a flowchart of a machine learning process. It is a flowchart of a dyed image generation process.

Here, embodiments of the present invention will be described in the following order.
(1) Configuration of stained image generation system:
(2) Machine learning process:
(3) Stained image generation processing:
(4) Other embodiments:

(1) Configuration of stained image generation system:
FIG. 1 is a diagram showing a schematic configuration of a fertilized egg analysis system 1 including a stained image generation system 10, and FIG. 2 is a block diagram of the fertilized egg analysis system 1. The fertilized egg analysis system 1 includes a stained image generation system 10, a microscope 40, and a display 50. Although omitted in FIGS. 1 and 2, various user interfaces (keyboard, mouse, etc.) may be connected to the stained image generation system 10. In the present embodiment, the object photographed by the fertilized egg analysis system 1 is a fertilized egg existing in the solvent in the petri dish S. In the present embodiment, the chromosome of the fertilized egg or the like is the analysis target, and the microscope 40 is used to enable the analysis of the chromosome.

The microscope 40 according to the present embodiment includes an objective lens Lo and an eyepiece lens Le, and the user can magnify and observe the fertilized egg through the eyepiece lens Le while rotating the rotating portion R. The magnification is, for example, about 100 times. The microscope 40 according to this embodiment includes a camera 41, a visible light source 42, and a UV light source 43. The camera 41 is a camera provided with an area sensor that detects the intensity of visible light (for example, 400 nm to 750 nm). In this embodiment, the camera 41 does not output ultraviolet rays.

The visible light source 42 is a light source that irradiates the observation target with white light. The UV light source 43 is a light source that irradiates an observation target with ultraviolet rays (for example, light having a wavelength smaller than 400 nm). In the microscope 40, the light source can be switched manually or automatically. Therefore, when the camera 41 takes a picture with the visible light source 42 turned on and the UV light source 43 turned off, the fertilized egg can be taken under visible light illumination. Further, when the camera 41 takes a picture with the UV light source 43 turned on and the visible light source 42 turned off, the fertilized egg can be taken under ultraviolet illumination.

In the present embodiment, the stained image generation system 10 is a stationary computer, and includes a control unit 20 and a recording medium 30. Further, the microscope 40 and the display 50 are connected to the stained image generation system 10 via an interface (not shown). The control unit 20 of the stained image generation system 10 can output various signals to the microscope 40 and acquire various information such as image data from the microscope 40. Further, the stained image generation system 10 can control the display 50 to display an arbitrary image.

The control unit 20 includes a CPU, RAM, ROM, and GPU (not shown), and can execute various programs recorded on the recording medium 30 or the like. In the present embodiment, the control unit 20 can execute the stained image generation program 21 and the machine learning program 22. The stained image generation program 21 has a function of generating a stained image which is an image in which the fertilized egg is stained based on the visible light irradiation image for analysis taken by the camera 41 under visible light illumination. This is a program to be executed by the control unit 20. When the stained image generation program 21 is executed, the control unit 20 functions as a visible light irradiation image acquisition unit 21a for analysis and a stained image generation unit 21b.

In the present embodiment, the generation of the stained image is performed based on the result of machine learning performed in advance. The machine learning program 22 is based on a learning visible light irradiation image in which a learning fertilized egg is photographed by a camera 41 under visible light illumination and a learning ultraviolet irradiation image in which a learning fertilized egg is photographed under ultraviolet illumination. This is a program that causes the control unit 20 to execute a function of machine learning a model for generating a stained image. When the machine learning program 22 is executed, the control unit 20 functions as the machine learning unit 22a.

(2) Machine learning process:
In the present embodiment, the machine learning process is a process of optimizing the training model that forms the neural network. In this embodiment, machine learning is performed on a model that generates a stained image from an image of a fertilized egg taken under visible light illumination. Therefore, in the present embodiment, the model means both in the case where the gradation value for each pixel of the input image data is used as the input value and the gradation value for each pixel of the image data of the generated image is used as the output value. It is information showing an expression for deriving the correspondence of data.

The model may be in various modes, but in the present embodiment, it is assumed that a model called pix2pix is machine-learned. Note that pix2pix is disclosed in, for example, Image-to-Image Translation with Conditional Adversarial Networks (arXiv: 1611.00704v3 [cs.CV] 26 Nov 2018).

FIG. 3 is a diagram schematically showing a machine learning model in this embodiment. In the present embodiment, the generative model G and the discriminative model D are prepared as training models. The generative model G is a model that generates another image from one image. The discriminative model D is a model that compares the first image and the second image and discriminates whether or not the second image is a fake image generated from the first image.

In the present embodiment, the image Ivt taken by irradiating the fertilized egg with visible light is input to the generative model G, and machine learning is performed so as to generate the image Iuf taken by irradiating the fertilized egg with ultraviolet rays. Further, in the present embodiment, the image Ivt taken by irradiating the fertilized egg with visible light is used as the first image, and the image Iff generated by the generative model G or the image taken by irradiating the fertilized egg with ultraviolet rays. Iut is input to the identification model D as a second image. Then, false when the second image is the generated image If, and true when the second image is the image Iut taken under ultraviolet illumination by putting the fertilized egg of the first image in the staining solution. The discriminative model D is machine-learned to output. The structure of the model and the setting of the loss function when performing machine learning may be performed by various methods. For example, the above-mentioned Image-to-Image Translation with Conditional Adversarial Networks (arXiv: 1611.07004v3 [cs.CV]] Various known technologies such as 26 Nov 2018) are available.

In performing the machine learning as described above, the teacher data 30a is recorded in advance on the recording medium 30. The control unit 20 optimizes the training model of the neural network based on the teacher data 30a, and records the optimized machine-learned model on the recording medium 30 as the machine-learning model 30b. For the teacher data 30a, a plurality of sets of an image taken by irradiating the fertilized egg with visible light and an image taken by irradiating the fertilized egg with ultraviolet rays in a state where the dyeing solution is contained in the solvent are prepared. It is generated by.

In order to efficiently proceed with machine learning of the generative model G and to achieve sufficient accuracy, images of fertilized eggs in the same state under visible light illumination and ultraviolet illumination are combined into a set of teacher data 30a. Is preferable. That is, the fertilized egg easily moves and rotates in the solvent, so that it easily takes a different posture. If the postures are the same, the components of the fertilized egg (cell nuclei, chromosomes, etc.) are at the same position in the image, so the difference between the image taken under visible light illumination and the image taken under ultraviolet illumination is Mainly with or without staining. In this case, since the generative model G simply converts the unstained fertilized egg into the stained fertilized egg, a relatively simple conversion may be performed.

However, if the change in position and rotation of the fertilized egg are allowed, it is necessary to reproduce the change in position, posture, and the like in addition to the change due to staining of the components of the fertilized egg by the generative model G. Such learning is extremely difficult, and it is virtually impossible to machine-learn a generative model capable of performing high-precision reproduction that can withstand chromosome analysis and the like.

Therefore, in the present embodiment, the image taken under visible light illumination and the image taken under ultraviolet illumination are photographed so that the fertilized egg is in the same state (or substantially the same state). Specifically, the user prepares a solvent in which the fertilized egg for learning is present in the petri dish S and sets it in the microscope 40. Next, the user puts the staining solution in the solvent. As the dyeing solution, for example, a Hoechst stain such as Hoechst (registered trademark) 33342 can be used, and the amount is, for example, about 1 mg / ml with respect to the solvent.

In the present embodiment, the fertilized learning egg fluoresces when irradiated with ultraviolet rays in a solvent containing a staining solution, but fluorescence is not observed (or almost not observed) even when irradiated with visible light. Therefore, if the fertilized eggs for learning are photographed under visible light illumination and ultraviolet illumination after adding the staining solution to Chare S, the fertilized eggs in the same state (or almost the same state) can be seen with visible light and ultraviolet rays. It is possible to take an image when illuminated by each of the above.

The control unit 20 according to the present embodiment can perform such shooting by the function of the machine learning program 22. Specifically, the staining solution is put into the petri dish S, the microscope 40 is operated to place the fertilized egg for learning in the field of view of the camera 41, and then the user gives an imaging instruction. When a shooting instruction is given, the control unit 20 controls the visible light source 42 to output visible light, and controls the UV light source 43 to prevent the output of ultraviolet rays. Further, the control unit 20 controls the camera 41 to take a picture of the fertilized egg for learning. The captured visible light irradiation image for learning is recorded in the RAM.

Further, the control unit 20 controls the visible light source 42 so that visible light is not output, and controls the UV light source 43 to output ultraviolet rays. Further, the control unit 20 controls the camera 41 to take a picture of the fertilized egg for learning. The captured ultraviolet irradiation image for learning is recorded in the RAM. Then, the control unit 20 makes a set of data in which the captured visible light irradiation image for learning and the ultraviolet irradiation image for learning are associated with each other, and records the captured data as teacher data 30a on the recording medium 30. By repeating the above processing, the control unit 20 creates teacher data 30a having a plurality of sets of data.

The number of vertical and horizontal pixels in the learning visible light irradiation image and the learning ultraviolet irradiation image constituting the teacher data 30a is a default number determined according to the generative model G. Therefore, the control unit 20 may perform enlargement / reduction processing and trimming processing in order to match the predetermined number. Further, when creating the teacher data 30a, the efficiency of the data creation process may be improved by various methods. For example, the image may be taken in a state where a plurality of fertilized learning eggs are present in the field of view of the camera 41, and images of each fertilized learning egg may be cut out.

Further, in order to carry out machine learning, it is preferable that a sufficient amount of data of the teacher data 30a is secured. Therefore, in order to increase the amount of data, various padding processes, for example, rotation, movement, inversion, enlargement, and reduction of the image may be performed to obtain different sets of teacher data 30a.

In the teacher data 30a, the fertilized egg for learning may be photographed in the same state, and in this sense, the ultraviolet irradiation image for learning may be photographed first, and the visible light irradiation image for learning may be photographed later. However, as in the present embodiment, it is more preferable that the learning fertilized egg is irradiated with ultraviolet rays and the learning ultraviolet irradiation image is taken after the learning visible light irradiation image is taken. That is, it may take a little time from the time when the staining solution is put into the solvent until the fertilized egg for learning can output fluorescence. In addition, when the color of the fertilized egg for learning changes slightly depending on the staining solution under visible light, it is considered that the longer the time from the start of staining, the greater the effect.

Therefore, if the learning visible light irradiation image is first taken and then the learning ultraviolet irradiation image is taken, the learning visible light irradiation image is taken with the influence of the dyeing solution suppressed, and the dyeing solution is used. It is possible to perform an ultraviolet irradiation image for learning in a state where fluorescence is output. According to the experiment of the applicant, the deviation between the image of the visible light irradiation for learning and the image of the other fertilized egg that is allowed when the image of one fertilized egg is used as a reference is It is known to be about 4%. Therefore, it is preferable to prepare the teacher data 30a so that the deviation is within 4%. In addition, when the learning visible light irradiation image and the learning ultraviolet irradiation image are taken and the deviation between them is larger than 4%, image processing such as rotation processing is performed to reduce the deviation between the two to less than 4%. The teacher data 30a may be used as the configuration.

The teacher data 30a as described above is prepared in advance before machine learning is performed. When the teacher data 30a is prepared, machine learning can be performed using the teacher data 30a. That is, when a learning visible light irradiation image is input to the generation model G prepared in advance as a training model, a stained image is generated. On the other hand, when the generated stained image and the visible light irradiation image for learning of the group at the time of generating the stained image are input to the discrimination model D, false or true is output. When the learning visible light irradiation image and the learning ultraviolet irradiation image are input to the discriminative model D, false or true is output.

Therefore, if the learning proceeds so that the discriminative model D outputs false when the generated stained image is input and the discriminative model D outputs true when the learning ultraviolet irradiation image is input, the generation is generated. The stained image generated by the model G becomes close to the real image (the image obtained by photographing the stained fertilized egg).

FIG. 4 is a flowchart showing the machine learning process. The machine learning process is executed in advance before the generation of the stained image of the fertilized egg to be analyzed is performed. Further, before the machine learning process is performed, the teacher data 30a is prepared in advance. When the machine learning process is started, the control unit 20 acquires a training model by the function of the machine learning unit 22a (step S100). In the present embodiment, the control unit 20 uses an image of a fertilized egg taken under visible light illumination as an input value, puts the fertilized egg in a staining solution, and emits fluorescence under ultraviolet illumination. Acquires the training model (information such as the filter indicating the model, the activation function, the loss function, etc.) of the generation model G whose output value is.

Further, the control unit 20 was stained by using an image of the fertilized egg taken under visible light illumination and an image of the stained fertilized egg (generated stained image or photographed stained image) as input values. The training model (information such as the filter indicating the model, the activation function, the loss function, etc.) of the identification model D whose output value is whether the image of the fertilized egg is true or false is acquired.

Next, the control unit 20 acquires the teacher data 30a by the function of the machine learning unit 22a (step S105). In the present embodiment, the teacher data 30a is information in which the learning visible light irradiation image and the learning ultraviolet irradiation image are associated with each other.

Next, the control unit 20 acquires test data by the function of the machine learning unit 22a (step S110). In the present embodiment, a part of the teacher data 30a is extracted and used as test data for confirming whether or not learning has been generalized. The test data is not used for machine learning.

Next, the control unit 20 determines the initial value by the function of the machine learning unit 22a (step S115). That is, the control unit 20 gives initial values to the variable parameters (filter weight, bias, etc.) to be learned in the training model acquired in step S100. The initial value may be determined by various methods. Of course, the initial values may be adjusted so that the parameters are optimized in the learning process, or the learned parameters may be acquired from various databases and used.

Next, the control unit 20 performs learning by the function of the machine learning unit 22a (step S120). That is, the control unit 20 inputs the learning visible light irradiation image of the teacher data 30a acquired in step S105 to the generation model G of the training model acquired in step S100, and outputs a stained image. Next, the control unit 20 inputs a learning visible light irradiation image and a comparison image to the identification model D, and outputs either true or false. The comparative image is either a learning ultraviolet irradiation image (that is, a true image) or a stained image generated by the generative model G (that is, a fake image). Which one to use is statistically determined in advance.

When the final output value by the training model, that is, the stained image generated by the generative model G and the true / false output identified by the identification model D is obtained, the control unit 20 determines the output value and the correct answer. The error is specified based on the loss function that evaluates the error between the output value and the correct answer value based on the value (the information indicating the authenticity of the learning ultraviolet irradiation image shown by the teacher data 30a and the comparison image). When the loss function E is obtained, the control unit 20 updates the parameters by a predetermined optimization algorithm, for example, a stochastic gradient descent method or the like. That is, the control unit 20 repeats the process of updating the parameters based on the differentiation of the parameters of the loss function E a predetermined number of times.

The loss function E can be in various known modes, but an element for making the stained image clearer may be added. That is, the details of the image generated by a model such as GAN (Generative Adversarial Network) tend to be blurred. On the other hand, when analyzing a stained image of a fertilized egg, details of chromosomes and cells are often analyzed. Therefore, if a stained image is generated without reproducing the details of the cells, it is difficult to use it for analysis. Therefore, in the generative model G, the loss function may be defined so that the loss decreases as the generated stained image becomes as sharp as the learning ultraviolet irradiation image.

Specifically, smoothing processing and contour enhancement processing (LoG (Laplacian of Gaussian)) are applied and applied to the stained image generated based on the learning visible light irradiation image and the learning ultraviolet irradiation image. A loss function indicating the difference between the later images may be included in the loss function of the generation model G. When the parameters are adjusted so that this loss function becomes small, the generative model G can be machine-learned so that the sharpness of the stained image becomes high. As a result, the stained image becomes sharp and it is possible to prevent the details from becoming unclear.

When the parameters are updated a predetermined number of times as described above, the control unit 20 determines whether or not the generalization of the training model is completed (step S125). That is, the control unit 20 inputs the learning visible light irradiation image of the test data acquired in step S110 into the training model generation model G to generate a stained image. Further, the control unit 20 inputs the generated stained image and the visible light irradiation image for learning of the test data into the identification model D of the training model, and determines the authenticity. Then, the control unit 20 acquires the number of true outputs of the test data and divides it by the total number of samples of the test data to acquire the estimation accuracy. In the present embodiment, the control unit 20 determines that the generalization is completed when the estimation accuracy is equal to or higher than the threshold value.

In addition to the evaluation of generalization performance, the validity of hyperparameters may be verified. That is, in a configuration in which a hyperparameter that is a variable quantity other than the variable parameter to be learned, for example, the number of nodes is tuned, the control unit 20 verifies the validity of the hyperparameter based on the verification data. Is also good. The verification data may be acquired by extracting the verification data from the teacher data 30a in advance by the same processing as in step S110 and securing the verification data as data not used for training.

If it is not determined in step S125 that the generalization of the training model is complete, the control unit 20 repeats step S120. That is, the process of updating the variable parameters to be learned is further performed. On the other hand, when it is determined in step S125 that the generalization of the training model is completed, the control unit 20 records the machine-learned training model as the machine learning model 30b on the recording medium 30 (step S130). In the present embodiment, in order to enable additional learning of the machine learning model 30b, the generation model G and the identification model D are recorded on the recording medium 30 as the machine learning model 30b, but a stained image is generated. If it is assumed that the above is performed and the identification is not performed, the generation model G may be recorded as the machine learning model 30b, and the identification model D may not be recorded.

(3) Stained image generation processing:
Next, an image of the fertilized egg to be analyzed (for example, the fertilized egg whose birth rate is estimated after artificial insemination or the fertilized egg whose chromosomal abnormality is analyzed after artificial insemination) under visible light illumination. The stained image generation process performed based on the above will be described with reference to the flowchart shown in FIG. The stained image generation process is executed after the machine learning model 30b is recorded on the recording medium 30. When executing the stained image generation process, the user prepares a solvent containing the fertilized egg to be analyzed in the petri dish S and sets it in the microscope 40. Here, the stain solution cannot be put into the solvent in the petri dish S. That is, in the present embodiment, in order to maintain the life of the fertilized egg, a visible light irradiation image for analysis is taken in a state where the staining solution is not contained in the solvent.

With such a petri dish S set in the microscope 40, the user operates the microscope 40 to turn on the visible light source 42 and move the fertilized egg to be analyzed within the field of view of the camera 41. In this state, when the user instructs the start of the dyed image generation process by an operation unit such as a mouse or a keyboard (not shown), the control unit 20 starts the execution of the dyed image generation program 21.

When the execution of the stained image generation program 21 is started, the control unit 20 irradiates the fertilized egg to be analyzed with visible light by the function of the visible light irradiation image acquisition unit 21a for analysis (step S200). That is, the control unit 20 outputs a control signal to the visible light source 42 and turns on the visible light source 42 with a predetermined intensity. At this time, the UV light source 43 is turned off.

Next, the control unit 20 captures the analysis visible light irradiation image by the camera 41 by the function of the analysis visible light irradiation image acquisition unit 21a (step S205). That is, the control unit 20 controls the camera 41 and acquires image data of the fertilized egg to be analyzed that has been irradiated with visible light. Further, the control unit 20 records the captured image data on the recording medium 30 (visible light irradiation image data 30c for analysis shown in FIG. 2).

Next, the control unit 20 inputs the visible light irradiation image for analysis into the machine learning model by the function of the stained image generation unit 21b (step S210). That is, the control unit 20 acquires the machine learning model 30b recorded on the recording medium 30, inputs the visible light irradiation image for analysis into the generation model G, and acquires the stained image.

Next, the control unit 20 displays the dyed image by the function of the dyed image generation unit 21b (step S215). That is, the control unit 20 controls the display 50 to display the stained image acquired in step S210. The display mode of the dyed image may be various modes, and in the present embodiment, the image taken in step S205 is displayed as a still image. Of course, a plurality of stained images may be displayed at the same time, or the visible light irradiation image for analysis taken in step S205 and the stained image may be displayed side by side. Further, in step S205, the moving image data is generated by continuously capturing the visible light irradiation image for analysis, the moving image data of the dyed image generated by the generation model G of the machine learning model 30b is generated, and the moving image data of the dyed image is generated. May be configured to display.

According to the present embodiment as described above, the visible light output when the fertilized egg in the solvent containing the staining solution is irradiated with ultraviolet rays based on the image of the fertilized egg to be analyzed under visible light illumination. It is possible to generate a stained image equivalent to the image formed by the fluorescence of. In order to generate the stained image, the staining solution is not put in the solvent of the fertilized egg to be analyzed. In addition, the fertilized egg is not irradiated with ultraviolet rays. Therefore, the fertilized egg is not killed to produce a stained image, nor is it caused damage by UV light.

Then, by analyzing the stained image, it is possible to estimate whether or not the fertilized egg to be analyzed is normal, whether or not there is a chromosomal abnormality, and the like. Therefore, it is possible to improve the birth rate in artificial insemination in infertility treatment and artificial insemination in the livestock industry by selecting fertilized eggs that are highly likely to be normal from the fertilized eggs to be analyzed.

(4) Other embodiments:
The above embodiment is an example for carrying out the present invention, as long as a machine learning model is generated based on images taken of a fertilized egg in a solvent containing a staining solution under visible light illumination and ultraviolet illumination. In addition, various other embodiments can be adopted. For example, the stained image generation system may be realized by a plurality of devices, or may be a system in which machine learning is performed by a server and a dyed image is generated by a client. In this configuration, for example, teacher data may be collected from a plurality of clients, and machine learning may be performed on the server based on the teacher data.

Further, at least a part of the visible light irradiation image acquisition unit 21a for analysis, the stained image generation unit 21b, and the machine learning unit 22a may be divided into a plurality of devices and exist. For example, the machine learning unit 22a may have a configuration in which the process of creating the teacher data 30a and the process of performing machine learning based on the teacher data 30a are performed by different devices. Of course, some configurations of the above-described embodiments may be omitted, and the order of processing may be changed or omitted. For example, when the system that executes machine learning and the stained image generation system 10 are configured by different devices, the microscope 40 used in the stained image generation system 10 does not have to include the UV light source 43.

The analysis visible light irradiation image acquisition unit only needs to be able to acquire an analysis visible light irradiation image which is an image taken by irradiating the fertilized egg to be analyzed with visible light. That is, the analysis visible light irradiation image acquisition unit only needs to be able to acquire the analysis visible light irradiation image which is an image of the fertilized egg input to the machine learning model. The visible light irradiation image for analysis may be taken under visible light illumination, but it may be taken under the same lighting conditions as when generating teacher data for learning a machine learning model. preferable. Further, since the visible light irradiation image for analysis is input to the machine learning model, it needs to be in the format (number of pixels, number of channels) input to the machine learning model. Of course, arbitrary image processing may be performed to match the format, and various image processing such as trimming, enlargement / reduction, edge enhancement, color conversion, contrast adjustment, and rotation may be performed.

Visible light may be irradiated in various states. That is, the visible light may be applied to the fertilized egg so that the visible light irradiation image for analysis can be taken, and the positional relationship between the fertilized egg and the visible light source can be various. Of course, the positional relationship may be variable, and the intensity of visible light may be variable. The number of channels for expressing the visible light irradiation image for analysis is arbitrary, but it is preferable that the visible light irradiation image for analysis is represented by typically 3 channels in order to cover the wavelength range of visible light. .. The fertilized egg is preferably held and photographed in a state in which life is maintained. For example, a configuration in which a fertilized egg existing in a solvent such as water in a petri dish is photographed is assumed.

The stained image generation unit may input a visible light irradiation image for analysis into the machine learning model and generate a stained image which is an image in which the fertilized egg to be analyzed is stained. That is, a machine learning model that generates a stained image from an image of a fertilized egg under visible light illumination is generated in advance. Then, the stained image generation unit generates a stained image by inputting an analysis visible light irradiation image obtained by photographing a fertilized egg to be analyzed into the machine learning model. The dyeing liquid may be any liquid that outputs fluorescence from the fertilized egg under ultraviolet illumination when placed in a solvent, and for example, various dyeing agents other than the Hoechst stain can be adopted. The solvent is preferably a liquid that can hold the fertilized egg in a state in which the life of the fertilized egg is maintained, and is, for example, water.

The machine learning model may be any model that converts the image of the fertilized egg under visible light illumination into the image of the stained fertilized egg, and various configurations other than the above configurations can be adopted. That is, the machine learning model may be an image-image conversion model obtained by various aspects of DCGAN (Deep Convolutional Generative Adversarial Networks), aspects derived from them, conditional GAN, or the like.

The mode of machine learning is not limited, for example, the number of layers and nodes constituting the neural network, the type of activation function, the type of loss function, the type of gradient descent method, the type of gradient descent optimization algorithm, etc. Presence / absence of mini-batch learning, number of batches, learning rate, initial value, type / presence / absence of overlearning suppression method, presence / absence of convolution layer, filter size in convolution calculation, filter type, padding / stride type, pooling layer type Machine learning may be performed by appropriately selecting various elements such as the presence / absence, the presence / absence of a fully connected layer, and the presence / absence of a recursive structure. Of course, learning may be performed by other machine learning, for example, deep learning, support vector machine, clustering, reinforcement learning, or the like. Further, machine learning may be performed in which the structure of the model (for example, the number of layers, the number of nodes per layer, etc.) is automatically optimized.

The image of the fertilized egg such as the visible light irradiation image for analysis and the visible light irradiation image for learning may be in a state where the analysis target such as a chromosome can be visually recognized. Therefore, it is preferable to take an image using a magnifying device such as a microscope, and the magnification and the like can be adjusted depending on the analysis application and the like.

Furthermore, the method of generating a machine learning model based on images of fertilized eggs in a solvent containing a staining solution under visible light illumination and ultraviolet illumination as in the present invention is also applicable as a program or method. It is possible. Further, the system, program, and method as described above can be assumed to be realized as a single device or by a plurality of devices, and include various aspects. For example, it is possible to provide a microscope or the like provided with the above means. In addition, some of them are software and some of them are hardware, which can be changed as appropriate. Further, the invention is also established as a recording medium for a program that controls a system. Of course, the recording medium of the software may be a magnetic recording medium or a semiconductor memory, and any recording medium to be developed in the future can be considered in exactly the same way.

1 ... fertilized egg analysis system, 10 ... stained image generation system, 20 ... control unit, 21 ... stained image generation program, 21a ... visible light irradiation image acquisition unit for analysis, 21b ... stained image generation unit, 22 ... machine learning program, 22a ... Machine learning unit, 30 ... Recording medium, 30a ... Teacher data, 30b ... Machine learning model, 30c ... Visible light irradiation image data for analysis, 40 ... Microscope, 41 ... Camera, 42 ... Visible light source, 43 ... UV light source, 50 ... Display

Claims (6)

A visible light irradiation image acquisition unit for analysis that acquires a visible light irradiation image for analysis, which is an image taken by irradiating a fertilized egg to be analyzed with visible light.
The learning visible light irradiation image which is an image taken by irradiating the learning fertilized egg in a solvent containing a staining solution with visible light and the learning ultraviolet irradiation image which is an image taken by irradiating the learning fertilized egg with ultraviolet light. A machine learning model that is machine-learned based on an image and that generates a stained image of a fertilized egg from an image of a fertilized egg under visible light illumination is irradiated with visible light for analysis. A stained image generation unit that inputs an image and generates a stained image that is an image in which the fertilized egg to be analyzed is stained.
Stained image generation system including. The irradiation of the fertilized egg for learning with the ultraviolet rays and the acquisition of the ultraviolet irradiation image for learning are performed after the acquisition of the visible light irradiation image for learning.
The stained image generation system according to claim 1. The visible light irradiation image for analysis is
It is an image of the fertilized egg to be analyzed that exists in a solvent that does not contain the stain solution.
The stained image generation system according to claim 1 or 2. The machine learning model is
Smoothing processing and contour enhancement processing are applied to the stained image generated based on the learning visible light irradiation image and the learning ultraviolet irradiation image so as to reduce the difference between the images after application. A machine-learned model,
The stained image generation system according to any one of claims 1 to 3. A visible light irradiation image acquisition process for analysis to acquire a visible light irradiation image for analysis, which is an image taken by irradiating a fertilized egg to be analyzed with visible light,
The learning visible light irradiation image which is an image taken by irradiating the learning fertilized egg in a solvent containing a staining solution with visible light and the learning ultraviolet irradiation image which is an image taken by irradiating the learning fertilized egg with ultraviolet light. A machine learning model that is machine-learned based on an image and that generates a stained image of a fertilized egg from an image of a fertilized egg under visible light illumination is irradiated with visible light for analysis. A stained image generation step of inputting an image and generating a stained image which is an image in which the fertilized egg to be analyzed is stained.
Stained image generation method including. Computer,
Visible light irradiation image acquisition unit for analysis, which acquires a visible light irradiation image for analysis, which is an image taken by irradiating a fertilized egg to be analyzed with visible light.
The learning visible light irradiation image which is an image taken by irradiating the learning fertilized egg in a solvent containing a staining solution with visible light and the learning ultraviolet irradiation image which is an image taken by irradiating the learning fertilized egg with ultraviolet light. A machine learning model that is machine-learned based on an image and that generates a stained image of a fertilized egg from an image of a fertilized egg under visible light illumination is irradiated with visible light for analysis. A stained image generation unit that inputs an image and generates a stained image that is an image in which the fertilized egg to be analyzed is stained.
A stained image generation program that functions as.

Images