JP2021018582A - Learning device, determination device, microscope, trained model, and program - Google Patents

Abstract

To provide a learning device, determination device, microscope, trained model, and program that can prevent deterioration in accuracy of segmentation.SOLUTION: In an image processing system, a learning execution unit 13 of a learning unit 1 includes: a similar image learning unit 130 that executes first learning based on a similar image, which is an image of a similar object, an object having a similar appearance to a target object, which is a target to be determined; and a target image learning unit 132 that executes second learning based on teacher data which is a set of a target image, which is an image obtained by capturing an image of the target, and a correct image, which is an image showing the target object in the target image, and a result of the first learning by the similar image learning unit.SELECTED DRAWING: Figure 2

Description

The present invention relates to learning devices, determination devices, microscopes, trained models, and programs.

In medicine, images of cells and tissues may be used for diagnosis. Image recognition by a computer is used to determine the tissues and cells imaged in those images.
A convolutional neural network (CNN: Convolutional Neural Network) is known as a deep learning method having excellent performance in image recognition (Non-Patent Document 1). Although CNN has excellent performance in image recognition, it requires a huge amount of teacher data for learning.

In a segmentation method using deep learning such as CNN, it is known that the generalization performance of the learner becomes low when there are not enough teacher data or when it is difficult to prepare a sufficient number of teacher data. The low generalization performance of the learner means that the accuracy of segmentation is reduced.

Oraf Ronneberger, Philipp Fisher, Thomas Brox, "U-Net: Convolutional Networks for Biomedical Image Segmentation", [online], May 18, 2015, Internet org / abs / 1505.04597>

One aspect of the present invention includes a similar image learning unit that executes the first learning based on a similar image that is an image of a similar object that is an object similar in appearance to the target object to be determined, and the target object. The teacher data, which is a set of the target image which is the image captured by, and the correct answer image which is the image in which the target object is shown in the target image, and the result of the first learning by the similar image learning unit. It is a learning device including a target image learning unit that executes a second learning based on the above.

One aspect of the present invention is a determination device including a determination unit that determines the target object in the target image based on the learning result by the learning device.

One aspect of the present invention is a microscope provided with the above learning device.

One aspect of the present invention is a microscope provided with the above-mentioned determination device.

One aspect of the present invention is a trained model trained by the learning device described above.

One aspect of the present invention comprises a similar image learning step in which a computer performs a first learning based on a similar image which is an image of a similar object which is an object similar in appearance to the target object to be determined. Based on the teacher data which is a set of the target image which is the image of the target object and the correct image which is the image showing the target object in the target image, and the learning result of the similar image learning step. This is a program for executing the target image learning step for executing the second learning.

One aspect of the present invention is a program for causing a computer to execute a determination step of determining the target object in the target image based on the learning result of the above program.

It is a figure which shows an example of the image processing system which concerns on 1st Embodiment. It is a figure which shows an example of the structure of the image processing apparatus which concerns on 1st Embodiment. It is a figure which shows an example of the data flow in the deep learning which concerns on 1st Embodiment. It is a figure which shows an example of the machine learning processing of the image processing apparatus which concerns on 1st Embodiment. It is a figure which shows an example of the learning process which concerns on 1st Embodiment. It is a figure which shows an example of the calculation method of the degree of similarity which concerns on 1st Embodiment. It is a figure which shows an example of the relationship between each image which concerns on 1st Embodiment, and learning processing. It is a figure which shows an example of the determination process which concerns on 1st Embodiment. It is a figure which shows an example of the segmentation result which concerns on 1st Embodiment. It is a figure which shows an example of the relationship between the similarity degree of the similar image which concerns on 1st Embodiment, and the determination accuracy. It is a figure which shows an example of the relationship between the spatial density of a similar object in the similar image which concerns on 1st Embodiment, and the determination accuracy. It is a figure which shows an example of the image processing system which concerns on 2nd Embodiment. It is a figure which shows an example of the relationship between the image which concerns on 2nd Embodiment, and learning processing. It is a figure which shows an example of the determination image which concerns on 2nd Embodiment. It is a figure which shows an example of the determination accuracy which concerns on 2nd Embodiment. It is a figure which shows an example of the image processing system which concerns on 3rd Embodiment. It is a figure which shows an example of the image processing system which concerns on 4th Embodiment. It is a figure which shows an example of correspondence with a target object which concerns on a modification of embodiment, and the degree of similarity. It is a figure which shows an example of the calculation process of the distance between image sets of the blood vessel-like image which concerns on the modification of embodiment. It is a figure which shows an example of the blood vessel image and the blood vessel-like image which concerns on the modification of embodiment.

(First Embodiment)
Hereinafter, the first embodiment will be described in detail with reference to the drawings. FIG. 1 is a diagram showing an example of an image processing system ST according to the present embodiment. As an example, the image processing system ST includes an electron microscope M2, an optical microscope M1, an electron microscope image database DB2, a DAPI image database DB1, and an image processing device T1.

In the image processing system ST, the brain tissue cell nucleus A2, which is the cell nucleus of the brain tissue, is imaged by the electron microscope M2. The brain tissue cell nucleus A2 is, for example, the granule cell nucleus of the mouse cerebellum. The image of the brain tissue cell nucleus A2 captured by the electron microscope M2 is supplied to the image processing device T1 from the electron microscope image database DB2 as the target image PA1. A part of the target image PA1 supplied to the image processing device T1 is supplied to the learning unit 1 of the image processing device T1 and used for learning processing, and the remaining part is supplied to the determination unit 2 of the image processing device T1 for determination. Used for processing.

The image processing device T1 is, for example, a computer. The image processing device T1 determines the image of the brain tissue cell nucleus A2 in the target image PA1, and classifies the region included in the target image PA1 into an image region of the brain tissue cell nucleus A2 and a region other than that. That is, the image processing device T1 performs segmentation on the target image PA1. The brain tissue cell nucleus A2 is an example of a target object to be determined.

The image processing device T1 performs segmentation based on deep learning. The image processing device T1 includes a learning unit 1 and a determination unit 2. The learning unit 1 executes learning based on deep learning. The determination unit 2 determines the region of the image of the target object in the target image PA1 based on the result learned by the learning unit 1.
The learning unit 1 executes learning based on the teacher data LD, which is a set of the target image PA1 and the nuclear correct answer image PT2. Here, the nuclear correct answer image PT2 is an image in which the region of the image of the brain tissue cell nucleus A2 is shown in advance in the target image PA1.

The nuclear correct image PT2 is a target image PA1 in which the region showing the brain tissue cell nucleus A2 is painted by, for example, a human with medical knowledge, the inside of this region is painted or the outline of this region is drawn. Created based on image PA1. Since it takes time and effort to create the nuclear correct image PT2, it may not be possible to prepare a sufficient number of nuclear correct images PT2. That is, the image processing device T1 may not be able to prepare a sufficient number of teacher data LDs for learning.

In the image processing system ST, even if the image processing device T1 cannot prepare a sufficient number of teacher data LDs for learning, the learning unit 1 uses the similar image PR1 for learning to perform generalization of learning. Suppress the decrease in. Here, the similar image PR1 is an image of a similar object which is an object similar in appearance to the target object. An object having a similar appearance to the target object is, for example, an object having a similar shape such as a contour to the target object. The image of a similar object is, for example, an image obtained by capturing a similar object with a microscope provided with an image sensor.

The image processing device T1 learns the feature amount of the similar object by using the similar image PR1. In the image processing system ST, iPS cell nucleus A1, which is a fluorescent image of the cell nucleus of human induced pluripotent stem cells (iPS cells) in the early stage of culture, is adopted as an example of a similar object having a shape similar to that of brain tissue cell nucleus A2. The shape of the brain tissue cell nucleus A2 is granular, and the shape of the iPS cell nucleus A1 is similar to the granular shape.

In this embodiment, an example in which the similar object is an object having a shape similar to that of the target object will be described, but the present invention is not limited to this. The similar object may be an object having a similar pattern such as a structure or a pattern to the target object.
Further, in the present embodiment, an example in which the image of the similar object is an image captured by a microscope including an image sensor will be described, but the present invention is not limited to this. The image of the similar object may be an image in which the similar object is drawn on a paper surface or the like as long as the appearance is similar to the target object.

The optical microscope M1 observes iPS cell nuclei A1 stained with DAPI (4', 6-diamidino-2-phenylindole) by fluorescence observation. The iPS cell nucleus A1 is imaged by the optical microscope M1 and accumulated in the DAPI image database DB1 as a similar image source image PM. The learning unit 1 selects an image region of the iPS cell nucleus A1 in the similar image source image PM by image recognition, and generates a nuclear correct image PT1 showing the selected region. Here, the nuclear correct answer image PT1 is an image in which the region of the image of the iPS cell nucleus A1 is shown in the similar image original image PM in which the iPS cell nucleus A1 is captured.

The image processing device T1 learns the feature amount of the iPS cell nucleus A1 using the nuclear correct image PT1. Here, the feature amount of iPS cell nucleus A1 is a granular shape. The shape of the iPS cell nucleus A1 and the shape of the brain tissue cell nucleus A2 are similar in that they are granular.
The image processing device T1 learns an extractor for extracting the brain tissue cell nucleus A2, which is a target object, using the learned features as foresight information. Here, the foresight information is information in which features similar to the appearance of the target object are learned before the learning of the extractor for extracting the target object is executed.

In the image processing system ST, even when the number of teacher data LDs used by the image processing device T1 for learning deep learning is not sufficient, foresight information can be used for learning, so that the generalization performance of learning is deteriorated. Can be suppressed. That is, in the image processing system ST, the accuracy of segmentation can be improved even when the number of teacher data LDs is not sufficient.

Next, with reference to FIG. 2, the details of the configuration of the image processing device T1 will be described. FIG. 2 is a diagram showing an example of the configuration of the image processing device T1 according to the present embodiment. The learning unit 1 is an example of a learning device, and the determination unit 2 is an example of a determination device.

First, the configuration of the learning unit 1 will be described. The learning unit 1 includes a similar image source image acquisition unit 10, a similar object extraction unit 11, a similarity calculation unit 12, a learning execution unit 13, a teacher data acquisition unit 14, and a learning result output unit 15.
The similar image source image acquisition unit 10, the similar object extraction unit 11, the similarity calculation unit 12, the learning execution unit 13, the teacher data acquisition unit 14, and the learning result output unit 15 are each an image processing unit T1. CPU (Central Processing Unit) is a module realized by reading a program from ROM (Read Only Memory) and executing processing.

Here, the similar image source image acquisition unit 10 and the similar object extraction unit 11 will be described with reference to FIG. 3 in addition to FIG. FIG. 3 is a diagram showing an example of a data flow in deep learning according to the present embodiment.

The similar image source image acquisition unit 10 acquires the similar image source image PM supplied from the similarity source image database 3. The similar original image database 3 of FIG. 3 is the DAPI image database DB1 of FIG. 1 as an example in this embodiment.
The similar image source image acquisition unit 10 may acquire the similar image source image PM from a database outside the image processing system ST instead of the similar image source image database 3. The external database of the image processing system ST is, for example, a public database such as an image server that publishes images on the web.

The similar object extraction unit 11 extracts a similar object from the similar image source image PM acquired by the similar image source image acquisition unit 10. The similar object extraction unit 11 generates a similar image PR from the similar image original image PM based on the extracted similar object.

Returning to FIG. 2, the description of the configuration of the learning unit 1 will be continued.
The similarity calculation unit 12 sets the image of the similar object in the similar image PR and the image of the similar object in the target image PA based on the similar image PR generated by the similar object extraction unit 11 and the target image PA acquired by the target image acquisition unit 140. The degree of similarity with the image of the target object is calculated. The similarity calculation unit 12 calculates the similarity as, for example, the L1 distance between the Hu moment of the image of the target object and the image of the target object. In the present embodiment, the smaller the similarity value, the more similar the image of the similar object in the similar image PR and the image of the target object in the target image PA.

The learning execution unit 13 executes learning based on deep learning. In the following, the convolutional neural network (CNN: Convolutional Neural Network) used by the learning execution unit 13 for learning is referred to as a neural network NN. The learning execution unit 13 includes a similar image learning unit 130, a shape loss calculation unit 131, a target image learning unit 132, a VAT (Virtual Adversarial Training) loss calculation unit 133, and a cross entropy loss calculation unit 134.
Here, the learning execution unit 13 will be described with reference to FIG. 3 in addition to FIG.

The similar image learning unit 130 executes the first learning based on the similar image PR generated by the similar object extraction unit 11. Here, the first learning is, for example, learning by a convolutional autoencoder (CAE) that uses a similar image PR as an input. The similar image learning unit 130 extracts the characteristics of the similar object in the similar image PR by using the similar image PR as an input and using CAE. In the following, the CAE trained by the similar image learning unit 130 using the similar image PR as an input is referred to as CAEf. CAEf is a neural network different from the neural network NN.

The shape loss calculation unit 131 calculates the shape loss L3 based on the predicted image PP calculated by the target image learning unit 132 and the CAEf learned by the similar image learning unit 130. The predicted image PP is an output of the neural network NN when the target image PA is input.

The target image learning unit 132 executes the second learning based on the teacher data LD, which is a set of the target image PA and the correct image PT, and the result of the first learning by the similar image learning unit 130. Here, the target image PA is acquired by the target image acquisition unit 140, and the correct image PT is acquired by the correct image acquisition unit 141. In the second learning, the weight of the neural network NN that minimizes the loss function L is calculated using the teacher data LD, which is a set of the target image PA and the correct image PT, and the calculated weight is used as the neural network. This is a process for updating the weight of the network NN.

This loss function L is an amount obtained by multiplying the cross entropy loss L1, the VAT loss L2, and the shape loss L3 by predetermined weights. As described above, the shape loss L3 is calculated based on the CAEf generated by the similar image learning unit 130 in the first learning, and the shape loss L3 is added to the loss function L. Therefore, the loss function L is based on the result of the first learning by the similar image learning unit 130.

The VAT loss calculation unit 133 calculates the VAT loss L2 based on the target image PA acquired by the target image acquisition unit 140.
The cross entropy loss calculation unit 134 calculates the cross entropy loss L1 based on the predicted image PP which is the output of the neural network NN when the target image PA is input and the correct image PT acquired by the correct image acquisition unit 141. To do.

Returning to FIG. 2, the description of the configuration of the learning unit 1 will be continued.
The teacher data acquisition unit 14 acquires the teacher data LD from the teacher data database 4. The teacher data acquisition unit 14 includes a target image acquisition unit 140 and a correct answer image acquisition unit 141.
The target image acquisition unit 140 acquires the target image PA from the teacher data database 4. The teacher data database 4 is the electron microscope image database DB2 of FIG. 1 as an example in this embodiment. The target image PA is a part of the target image PA1 stored in the electron microscope image database DB2.

The correct image acquisition unit 141 acquires the correct image PT from the teacher data database 4. The correct image PT is created in advance by a human based on the target image PA and stored in the electron microscope image database DB2. The correct image PT is, for example, the nuclear correct image PT2 of FIG.

The learning result output unit 15 outputs the learning result of the learning execution unit 13 to the determination unit 2. The learning result of the learning execution unit 13 is a neural network NN whose weight is updated by the learning process.

Next, the configuration of the determination unit 2 will be described. The determination unit 2 includes a determination image acquisition unit 20, a determination execution unit 21, and a determination result output unit 22.
The determination image acquisition unit 20 acquires the determination image PE from the determination image database 5. The determination image database 5 is the electron microscope image database DB2 of FIG. 1 as an example in the present embodiment. The determination image PE is, for example, a part of the target image PA1 stored in the electron microscope image database DB2.

The determination execution unit 21 is based on the determination image PE acquired by the determination image acquisition unit 20 and the neural network NN whose weight is updated by the learning process by the learning execution unit 13, and is the area of the image of the target object in the target image PA. To judge. That is, the determination execution unit 21 performs segmentation on the target image PA.
The determination result output unit 22 displays the segmentation result, which is the result of the determination by the determination execution unit 21, on a display device (not shown).

The processing of the image processing system ST will be described with reference to FIGS. 4 to 8.
FIG. 4 is a diagram showing an example of machine learning processing of the image processing device T1 according to the present embodiment. In the example shown in FIG. 4, the target object is the cell nucleus of the second site. Further, the cell nucleus of the first site is used as a similar object to the cell nucleus of the second site, which is the target object. As an example, the cell nucleus of the first site is the iPS cell nucleus A1, and the cell nucleus of the second site is the brain tissue cell nucleus A2. Further, in FIG. 4, the first microscope is an optical microscope M1 and the second microscope is an electron microscope M2.

Step S10: The first observation condition is set in the first microscope. The first observation condition is an observation condition for observing the cell nucleus of the first site with the first microscope. As an example, the observation conditions include a microscope observation method such as fluorescence observation and bright field observation, and pixel resolution of the captured image. The observation condition in the first observation condition of the present embodiment is, for example, fluorescence observation.

Step S20: The cell nucleus of the first site is imaged with the first microscope. In this example, the iPS cell nucleus A1 stained with DAPI is observed by fluorescence observation with an optical microscope M1. The first microscope generates a similar image source image PM based on the result of imaging the cell nucleus of the first site.

Step S30: The nuclear correct image PT1 is generated. In the present embodiment, the nuclear correct image PT1 is generated from the similar image source image PM by the learning unit 1 of the image processing device T1. The learning unit 1 selects an image region of the iPS cell nucleus A1 in the similar image source image PM by image recognition, and generates a nuclear correct image PT1 showing the selected region.

Step S40: The image processing device T1 changes the pixel resolution of the nuclear correct image PT1 so as to match the pixel resolution indicated by the second observation condition. The processing of changing the pixel resolution is, for example, upsampling or downsampling.

Step S50: The image processing device T1 learns the characteristics of a similar object using the nuclear correct image PT1. Here, for example, CAE is used in learning the characteristics of similar objects.

Step S60: The second observation condition is set in the second microscope. The second observation condition is an observation condition for observing the cell nucleus of the second site with a second microscope.

Step S70: The cell nucleus of the second site is imaged with the second microscope. In this example, the brain tissue cell nucleus A2 is observed with an electron microscope M2. The electron microscope M2 generates a target image PA based on the result of imaging the cell nucleus of the second site.

Step S80: The plurality of target image PAs are divided into an image for learning and an image for determination. For example, among the plurality of target image PAs, the number corresponding to a predetermined ratio is used for the learning image, and the remaining number is used for the determination image.

Step S90: A nuclear correct image PT2 is created for the target image PA for learning. Here, the nuclear correct answer image PT2 is created by painting the inside of the region showing the brain tissue cell nucleus A2 in the target image PA1 by a person having medical knowledge or drawing the outline of this region. Will be done. The work of creating the nuclear correct answer image PT2 is performed by, for example, using image processing software or the like to trace the region showing the brain tissue cell nucleus A2 of the nuclear correct answer image PT2 displayed on the touch panel or the display with a touch pen or a mouse. It is said.
The pair of the target image PA for learning and the nuclear correct image PT2 is used as the teacher data LD of the learning process performed by the image processing device T1.

Step S100: The image processing device T1 learns the feature amount of the cell nucleus of the second site by using the pair of the target image PA for learning and the nuclear correct answer image PT2 as the teacher data LD by using deep learning. In this deep learning, as a loss function, a function in which shape loss and VAT loss are added to cross entropy loss is used.

Step S110: The image processing device T1 determines the image of the cell nucleus of the second site with respect to the image for determination based on the learning result.

In this embodiment, as described in step S30, an example in which the nuclear correct image PT1 is generated from the similar image source image PM has been described, but the present invention is not limited to this. A similar image source image PM may be used as the nuclear correct image PT1. For example, the image in which the iPS cell nucleus A1 stained by DAPI is observed by fluorescence observation as the nuclear correct image PT1 may be used as it is without extracting the region of the image of the iPS cell nucleus A1.

Next, the learning process of the learning unit 1 will be described with reference to FIG. FIG. 5 is a diagram showing an example of the learning process according to the present embodiment.

Step S200: The similar image source image acquisition unit 10 acquires the similar image source image PM supplied from the similarity source image database 3. The similar image source image acquisition unit 10 supplies the acquired similar image source image PM to the similar object extraction unit 11.

Step S210: The teacher data acquisition unit 14 acquires the teacher data LD from the teacher data database 4. Here, the target image acquisition unit 140 acquires the target image PA, while the correct image acquisition unit 141 acquires the correct image PT. As described above, the correct image PT is created from the target image PA, and the correct image PT and the target image PA are associated with each other.
The target image acquisition unit 140 supplies the acquired target image PA to the target image learning unit 132 and the VAT loss calculation unit 133. The correct image acquisition unit 141 supplies the acquired correct image PT to the cross entropy loss calculation unit 134 and the similarity calculation unit 12.

Step S220: The similar object extraction unit 11 extracts a similar object from the similar image source image PM acquired by the similar image source image acquisition unit 10. The similar object extraction unit 11 extracts a similar object from the similar image source image PM by using, for example, a binarization process or an image recognition algorithm. The similar object extraction unit 11 uses, for example, template matching as an image recognition algorithm.
The similar object extraction unit 11 generates a similar image PR from the similar image original image PM based on the extracted similar object. The similar object extraction unit 11 supplies the generated similar image PR to the similarity calculation unit 12 and the similar image learning unit 130.

Step S230: The similarity calculation unit 12 sets the image of the similar object in the similar image PR and the target based on the similar image PR generated by the similar object extraction unit 11 and the target image PA acquired by the target image acquisition unit 140. The degree of similarity with the image of the target object in the image PA is calculated.

Here, with reference to FIG. 6, the similarity calculated by the similarity calculation unit 12 will be described. FIG. 6 is a diagram showing an example of a method for calculating the degree of similarity according to the present embodiment. FIG. 6 describes a case where the degree of similarity is calculated for the similar image PR1 which is an example of the similar image PR. The similar image PR1 includes a plurality of target object images which are images of the target object. Each of the plurality of target object images is represented as a target object image SPi (i = 1, 2, ..., N: N is the number of images of the target object included in the similar image PR1).

The similarity calculation unit 12 selects, for example, the target object image SP1 from the plurality of target object images included in the similar image PR1. The similarity calculation unit 12 calculates the image moment for the selected target object image SP1. The image moment is defined by, for example, equation (1).

Here, the coordinates x and the coordinates y indicate the x-coordinate and the y-coordinate of a certain point included in the target object image SP1 when a two Cartesian coordinate system (xy coordinate system) is set for the target object image SP1. The pixel value I (x, y) indicates the pixel value of the target object image SP1. In the present embodiment, since the similar image PR1 is a binarized image, the pixel value I (x, y) is 0 or 1. The numbers i and j are natural numbers indicating the powers of the coordinates x and the coordinates y on the right side of the equation (1) and the order of the image moments on the left side of the equation (1).

The similarity calculation unit 12 calculates the image moment of the equation (1) represented by the two Cartesian coordinate system (xy coordinate system). The similarity calculation unit 12 calculates the image moment for the target object image SP1 with respect to a predetermined order. The predetermined order is the order required to calculate the Hu moment.

The similarity calculation unit 12 calculates the Hu moment from the calculated image moment. Hu moments are seven types of invariants with respect to image translation, scaling, rotation, etc., and are represented by polynomials of image moments. Here, image scaling is represented by the spatial density of similar objects in the image. The spatial density of a similar object in an image is calculated as, for example, the ratio of the area of the image region of the similar object to the image to the area of the entire image.
As described above, the similarity calculation unit 12 quantifies the image moments of a predetermined order.

The similarity calculation unit 12 calculates the similarity between the target object image SP1 and the target object as the L1 distance between the Hu moment of the target object image SP1 and the Hu moment of the target object.
The similarity calculation unit 12 calculates the similarity with the target object for the target object image SPI (i = 1, 2, ..., N: N is the number of images of the target object included in the similar image PR1). .. As an example, the similarity calculation unit 12 calculates the average value of the calculated similarity for the image of the target object included in the similar image PR1 as the similarity of the similar image PR1. The method of calculating the similarity is not limited to the mode using the Hu moment described above. For example, the outer shape of the region may be represented by the rΘ coordinate system with the center of gravity as O to create a histogram, and the similarity may be calculated based on the difference (L1 distance, etc.) of the histogram.

Returning to FIG. 5, the description of the learning process is continued. In the following, the learning process will be described with reference to FIG. 7 in addition to FIG. FIG. 7 is a diagram showing an example of the relationship between each image and the learning process according to the present embodiment.

Step S240: The similar image learning unit 130 executes learning by CAE using the similar image PR generated by the similar object extraction unit 11. In FIG. 7, features are extracted from the similar image PR by the encoder f that inputs the similar image PR, and learning is performed so that the output of the decoder g that inputs the output of the encoder f matches the similar image PR. .. The similar image learning unit 130 generates CAEf as a result of training the CAE using the similar image PR as an input.

As described above, the learning by CAE using the similar image PR as an input is the first learning. Therefore, the similar image learning unit 130 executes the first learning using the similar image PR generated by the similar object extraction unit 11. That is, the similar image learning unit 130 executes the first learning as the similar image PR by using the image showing the similar object extracted by the similar object extraction unit 11.

The similarity calculation unit 12 may exclude the similar image PR whose similarity is equal to or less than a predetermined value from the similar images used in the learning by CAE. As described above, the smaller the value of the similarity, the more similar the image of the similar object in the similar image PR and the image of the target object in the target image PA.
When a similar image PR having a similarity of less than or equal to a predetermined value is excluded from the similar image used in learning by CAE, the similar object has a degree of similarity to or less than a predetermined value. In this case, if the value indicating the degree of similarity in appearance with respect to the target object is, for example, the reciprocal of the degree of similarity, the value indicating the degree of similarity in appearance with respect to the target object is equal to or greater than a predetermined value.

Step S250: The VAT loss calculation unit 133 calculates the VAT loss L2 based on the target image PA acquired by the target image acquisition unit 140. The VAT loss calculation unit 133 supplies the calculated VAT loss L2 to the target image learning unit 132.

The VAT loss L2 is described as item _{Ladv in} FIG. This term _Ladv is expressed by the equation (2).

In equation (2), r _vadv is represented by equation (3).

In the formulas (2) and (3), the value x indicates the pixel value of the target image PA, and the value y indicates the pixel value of the correct image PT.
This term _Ladv corresponds to so-called local distribution smoothing, and improves the robustness of segmentation by adding a perturbation to the target image PA.

Step S260: The target image learning unit 132 calculates the predicted image PP based on the target image PA acquired by the target image acquisition unit 140 and the neural network NN. Here, the target image learning unit 132 calculates the output of the neural network NN by using the pixel value of the target image PA acquired by the target image acquisition unit 140 as an input of the neural network NN. The target image learning unit 132 calculates the predicted image PP by using the calculated output as the pixel value of the predicted image PP.

In FIG. 7, the output of the neural network NN when the target image PA is used as the input of the neural network NN is shown as a value φ _θ . The parameter θ indicates a set of total weights between each node included in the neural network NN. The value φ _θ corresponds to the pixel value of the predicted image PP and is the segmentation prediction result.

Step S270: The cross entropy loss calculation unit 134 receives the cross entropy loss based on the predicted image PP which is the output of the neural network NN when the target image PA is input and the correct image PT acquired by the correct image acquisition unit 141. Calculate L1. The cross-entropy loss calculation unit 134 supplies the calculated cross-entropy loss L1 to the target image learning unit 132.

The cross entropy loss L1 is described in FIG. 7 as the term L _{cross entropy} . The term L _{cross entropy} is a _{cross entropy} between the value φ _θ (x), which is the result of segmentation prediction, and the value y, which is the pixel value of the correct image PT. The term L _{cross entropy} is a loss function for two-class classification. In the present embodiment, the two classes are two classes, a class corresponding to the area of the image of the target object and a class corresponding to the area other than the image of the target object.

Step S280: The shape loss calculation unit 131 is based on the predicted image PP calculated by the target image learning unit 132, the correct image PT acquired by the correct image acquisition unit 141, and the CAEf trained by the similar image learning unit 130. Then, the shape loss L3 is calculated. The shape loss calculation unit 131 supplies the calculated shape loss L3 to the target image learning unit 132.

The shape loss L3 is described as item L _{shape in} FIG. Distance this section L _shape has a value of the output of CAEf which receives a segmentation prediction result value phi _theta (x), the value of the output of CAEf to enter a value y is the pixel value of the correct image PT Is. The term L _happe is represented by equation (4).

The term L _shape corresponds to a shape regularization term using a similar image PR. The term L _shape compares the distance between the value φ _θ (x), which is the result of segmentation prediction, and the value y, which is the pixel value of the correct image PT, with a filter that extracts the characteristics of similar objects in the similar image PR. By doing so, the search range of the parameter θ is limited. The filter that extracts the characteristics of the similar object of the similar image PR is the CAEf trained by the similar image learning unit 130.

Step S290: The target image learning unit 132 includes a cross entropy loss L1 calculated by the cross entropy loss calculation unit 134, a VAT loss L2 calculated by the VAT loss calculation unit 133, and a shape loss L3 calculated by the shape loss calculation unit 131. The loss function L is calculated based on.

In FIG. 7, the loss function L is the sum of the cross entropy loss L1, the VAT loss L2 multiplied by the parameter λ _a , and the shape loss L3 multiplied by the parameter λ _s . The parameter λ _a is the weight of the VAT loss L2 with respect to the cross entropy loss L1, and the parameter λ _s is the weight of the shape loss L3 with respect to the cross entropy loss L1.

Step S2100: The target image learning unit 132 updates the weight (parameter θ) of the neural network NN so as to minimize the calculated loss function L. Here, the target image learning unit 132 updates the weight of the neural network NN based on, for example, the gradient descent method.

That is, the target image learning unit 132 is similar to the teacher data LD, which is a set of the target image PA, which is an image in which the target object is captured, and the correct image PT, which is the image in which the target object is shown in the target image PA. The second learning is executed based on the result of the first learning by the image learning unit 130. Here, the target image learning unit 132 uses the loss function L to which the shape loss L3 is added for the second learning. The shape loss L3 is based on CAEf, which is the result of the first learning by the similar image learning unit 130. That is, the target image learning unit 132 executes the second learning based on the loss function L based on the result of the first learning by the similar image learning unit 130 and the teacher data LD.

In FIG. 7, the value φ _θ (x), which is the output of the neural network NN when the target image PA is input, approaches the pixel value of the correct image PT by updating the parameter θ using the similar image PR. It is shown.

Step S2110: The target image learning unit 132 determines whether or not the learning end condition is satisfied. The learning end condition is, for example, that the value of the loss function L converges. When it is determined that the learning end condition is satisfied (step S2110; YES), the target image learning unit 132 executes the process of step S2120. On the other hand, when the target image learning unit 132 determines that the learning end condition is not satisfied (step S2110; NO), the learning execution unit 13 executes the process of step S260 again.

Step S2120: The target image learning unit 132 outputs the neural network NN whose weight has been updated by the learning process to the learning result output unit 15. The neural network NN whose weight is updated by the learning process is a model learned by the learning unit 1. The neural network NN is an example of a trained model trained by a learning device.
With the above, the learning execution unit 13 ends the learning process.

In the learning process shown in FIG. 5, the process of acquiring the similar image PR in step S200 and the process of acquiring the teacher data LD in step S210 may be executed in a different order. Further, the process of calculating the cross entropy loss L1 in step S270 and the process of calculating the shape loss L3 in step S280 may be executed in a different order.

Next, the determination process of the determination unit 2 will be described with reference to FIG. FIG. 8 is a diagram showing an example of the determination process according to the present embodiment.
Step S300: The determination image acquisition unit 20 acquires the determination image PE from the determination image database 5. The determination image acquisition unit 20 supplies the acquired determination image PE to the determination execution unit 21.

Step S310: The determination execution unit 21 performs segmentation on the target image PA. Here, the determination execution unit 21 is an image of the target object in the target image PA based on the determination image PE acquired by the determination image acquisition unit 20 and the neural network NN whose weight is updated by the learning process by the learning execution unit 13. Judge the area of. The determination execution unit 21 supplies the segmentation result to the determination result output unit 22.

Step S320: The determination result output unit 22 causes the display device to display the segmentation result by the determination execution unit 21. The determination result output unit 22 may store the segmentation result by the determination execution unit 21 in a database or the like external to the image processing device T1.
With the above, the determination unit 2 ends the determination process.

Next, the segmentation results will be described with reference to FIGS. 9 to 11.
FIG. 9 is a diagram showing an example of the segmentation result according to the present embodiment. In the example shown in FIG. 9, the segmentation result images PS1 to PS4 are shown as the segmentation results obtained by performing the segmentation on the determination image PE1.

The determination image PE1 is, for example, a Z-stack image in which the granule cell nuclei of the mouse cerebellum are imaged. The determination image PE1 includes three cross-sectional images in which the cross-sections of the granule cell nuclei are captured in three different directions. According to the determination image PE1 including the three cross-sectional images, the segmentation result images PS1 to PS4 also show the segmentation results for each of the three cross-sectional images.
The segmentation result by the image processing apparatus T1 is the segmentation result image PS4. The segmentation result images PS1 to PS3 are shown for comparison with the segmentation result by the image processing apparatus T1.

The segmentation result image PS1 is a segmentation result by a neural network trained by using the cross entropy loss as a loss function. The segmentation result image PS2 is a segmentation result by a neural network in which VAT loss is added to the cross entropy loss and the learning is performed by using it as a loss function. The segmentation result image PS3 is a segmentation result by a neural network in which shape loss is added to cross entropy loss and learning is performed by using it as a loss function.

The determination accuracy of the region of the image of the target object in the target image PA is higher in the order of the segmentation result image PS4, the segmentation result image PS3, the segmentation result image PS2, and the segmentation result image PS1. That is, the segmentation result by the image processing device T1 is the highest as compared with the other segmentation results.

Since the segmentation result image PS2 has improved judgment accuracy as compared with the segmentation result image PS1 and the segmentation result image PS4 has improved judgment accuracy as compared with the segmentation result image PS3, VAT loss is added to the loss function. It can be seen that the determination accuracy is improved as compared with the case where the added person is not added.
When VAT loss is added to the loss function, more memory is required than when VAT loss is not added in the learning process.

On the other hand, the segmentation result image PS3 has improved determination accuracy as compared with the segmentation result image PS1, and the segmentation result image PS4 has improved determination accuracy as compared with the segmentation result image PS2. It can be seen that the determination accuracy is improved when is added as compared with the case where is not added.
Even when the VAT loss is not added to the loss function, the determination accuracy is improved by adding the shape loss. In order to improve the determination accuracy of segmentation, it is preferable to add shape loss and VAT loss to the loss function, but if the amount of memory available in the learning process is not sufficient, depending on the amount of memory, The VAT loss does not have to be added to the loss function.

FIG. 10 is a diagram showing an example of the relationship between the similarity and the determination accuracy of the similar image PR according to the present embodiment. Graph G1 shows the value of the Dice coefficient with respect to the degree of similarity between the target image PA and the similar image PR. The degree of similarity is the L1 distance between the Hu moments of the target image PA and the similar image PR as described above, and is the shape of the image of the target object included in the target image PA and the image of the similar object included in the similar image PR. The smaller the value, the more similar the shape of. In FIG. 10, the Dice coefficient indicates the degree of similarity between the segmentation result image and the correct answer image, and the higher the segmentation result image, the higher the segmentation accuracy.

In the graph G1, the point P1a corresponds to the similar image PR2a, the point P1b corresponds to the similar image PR2b, and the point P1c corresponds to the similar image PR2c. The similar image PR2a, the similar image PR2b, and the similar image PR2c have a higher degree of similarity to the target image PA in this order. In the similar image PR2b, only one similar object is included, and the shape of the similar object is not similar to the shape of the granule cell nucleus which is the target object as compared with the shape of the similar object contained in the similar image PR2a. In the similar image PR2c, the shape of the similar object is polygonal, and the shape of the similar object is not similar to the shape of the granule cell nucleus as compared with the shape of the similar object contained in the similar image PR2a and the similar image PR2b.
From graph G1, it can be seen that the accuracy of segmentation decreases as the similarity increases.

Next, with reference to FIG. 11, the relationship between the spatial density of similar objects and the determination accuracy will be described.
FIG. 11 is a diagram showing an example of the relationship between the spatial density of similar objects and the determination accuracy in the similar image PR according to the present embodiment. In FIG. 11, the similar image PR2b is obtained by extracting an object region of the similar image PR2a and displaying one of the regions. Further, in the similar image PR2c, the Douglas-Pueker algorithm is applied to a plurality of object regions extracted from the similar image PR2a, and the object region is represented as a polygon having vertices on the contour of the object region. The spatial density of similar objects was changed as described above.

Graph G2 shows the value of the Dice coefficient with respect to the spatial density of the similar object in the similar image PR. The Dice coefficient is the same as in FIG.
In the graph G2, the point P2a corresponds to the similar image PR2a, the point P2b corresponds to the similar image PR2b, and the point P2c corresponds to the similar image PR2c. Similar image PR2a, similar image PR2b, and similar image PR2c have higher spatial densities of similar objects in this order.
From graph G2, it can be seen that the spatial density of similar objects does not affect the accuracy of segmentation.

In the present embodiment, the similar object extraction unit 11 extracts a similar object from the similar image source image PM acquired by the similar image source image acquisition unit 10, and generates a similar image PR based on the extracted similar object. An example of the case has been described, but the present invention is not limited to this. The similar image original image PM may be used as it is as the similar image PR. When the similar image original image PM is used as it is as the similar image PR, for example, the similar object extraction unit 11 uses the similar image original image PM acquired by the similar image original image acquisition unit 10 as it is as the similar image PR, and the similarity calculation unit 12 and It is supplied to the similar image learning unit 130. Alternatively, when the similar image original image PM is used as it is as the similar image PR, the similar object extraction unit 11 may be omitted from the configuration of the learning unit 1.

Further, in the present embodiment, an example in which the similarity between the image of the similar object in the similar image PR and the image of the target object in the target image PA is calculated by the similarity calculation unit 12 has been described. Not exclusively. The similarity calculation unit 12 may be omitted from the configuration of the learning unit 1.

As described above, the learning device according to the present embodiment (in this example, the learning unit 1) includes a similar image learning unit 130 and a target image learning unit 132.
The similar image learning unit 130 is an image of a similar object (iPS cell nucleus A1 in this example) which is an object similar in appearance to the target object (brain tissue cell nucleus A2 in this example) to be determined. The first learning (in this example, the learning by the convolutional auto encoder using the similar image PR as the input) is executed based on the PR.
In the target image learning unit 132, the target image PA, which is an image obtained by capturing the target object (brain tissue cell nucleus A2 in this example), and the target object (brain tissue cell nucleus A2 in this example) are shown in the target image PA. Based on the teacher data LD, which is a set of the correct image PT, which is the image, and the result of the first learning by the similar image learning unit 130 (in this example, the learning by the convolutional autoencoder using the similar image PR as an input). Second learning (in this example, the weight of the neural network NN that minimizes the loss function L was calculated and calculated using the teacher data LD, which is a set of the target image PA and the correct image PT. The process of updating the weight of the neural network NN by the weight) is executed.

With this configuration, in the learning device according to the present embodiment (learning unit 1 in this example), even if the number of teacher data LDs is not sufficient and the determination accuracy is low, the first learning (in this example). , A loss function using the teacher data LD, which is a set of the target image PA and the correct image PT in this example, based on the result of the second learning (in this example, the learning by the convolutional autoencoder using the similar image PR as the input). Since the process of calculating the weight of the neural network NN that minimizes L and updating the weight of the neural network NN with the calculated weight) can be executed, even if there are not enough teacher data LDs, The accuracy of the determination can be improved as compared with the case where it is not based on the result of the first learning (in this example, the learning by the convolution autoencoder using the similar image PR as the input).

Here, the similar image PR may be acquired from a database external to the image processing system ST such as a public database, or as described above, the similar image PR acquired from an external database of the image processing system ST such as a public database. It may be generated from the image source image PM. When the similar image PR is acquired from a database outside the image processing system ST such as a public database, a pre-prepared image group including similar objects can be used, and the trouble of preparing the similar image PR can be saved. Can be done.
In the learning device according to the present embodiment (in this example, the learning unit 1), even if the number of teacher data LDs is not sufficient, the first learning is performed using an image group prepared in advance in an external database. By executing this, it is possible to suppress a decrease in determination accuracy due to an insufficient number of teacher data LDs.

Further, the learning device according to the present embodiment (in this example, the learning unit 1) further includes a similar object extraction unit 11. The similar object extraction unit 11 extracts a similar object (iPS cell nucleus A1 in this example) from the similar image PR. Further, the similar image learning unit 130 uses the image showing the similar object (iPS cell nucleus A1 in this example) extracted by the similar object extraction unit 11 as the similar image PR for the first learning (in this example). , Learning by a convolutional auto-encoder that uses a similar image PR as an input) is executed.

With this configuration, in the learning device according to the present embodiment (in this example, the learning unit 1), the similar object extraction unit 11 can perform the similar object (in this example, the learning unit 1) from the similar image PR without the user painting a similar object from the similar image PR. In this example, since the iPS cell nucleus A1) can be extracted, the trouble of extracting a similar object can be saved.

Further, in the learning device according to the present embodiment (learning unit 1 in this example), the first learning (this example) using an image showing a similar object (iPS cell nucleus A1 in this example) as a similar image PR. In the case where the image showing the similar object (iPS cell nucleus A1 in this example) extracted as the similar image PR is not used (for example, because the learning by the convolution auto encoder using the similar image PR as the input) can be executed. Compared with the case where the similar image original image PM is used), the efficiency of the first learning (in this example, the learning by the convolution auto encoder using the similar image PR as an input) can be improved. Increasing the efficiency of the first learning means, for example, reducing the number of similar image PRs required to obtain the same degree of accuracy of the second learning (accuracy of segmentation in this example).

Further, in the learning device according to the present embodiment (learning unit 1 in this example), the similar object (iPS cell nucleus A1 in this example) has a similar appearance to the target object (brain tissue cell nucleus A2 in this example). The value indicating the degree (in this example, the reciprocal of the similarity) is equal to or greater than a predetermined value.

With this configuration, in the learning device (learning unit 1 in this example) according to the present embodiment, a value indicating the degree of similarity in appearance to the target object (brain tissue cell nucleus A2 in this example) (degree of similarity in this example). First learning using a similar image PR of a similar object (in this example, iPS cell nucleus A1) whose (inverse number) is greater than or equal to a predetermined value (in this example, learning by a convolutional autoencoder using the similar image PR as an input) (In this example, the weight of the neural network NN that minimizes the loss function L is calculated using the teacher data LD, which is a set of the target image PA and the correct image PT. Then, the accuracy of the processing of updating the weight of the neural network NN by the calculated weight is a value indicating the degree of similarity in appearance with respect to the target object (in this example, the brain tissue cell nucleus A2) (in this example, the inverse of the degree of similarity). ) Is less than a predetermined value, which can be increased as compared with the case of using a similar image of a similar object.

Further, in the learning device according to the present embodiment (in this example, the learning unit 1), the target image learning unit 132 is the first learning by the similar image learning unit 130 (in this example, the convolution using the similar image PR as an input). Using the second learning (in this example, the teacher data LD, which is a pair of the target image PA and the correct image PT) based on the loss function L based on the result of the learning by the autoencoder and the teacher data LD, A process of calculating the weight of the neural network NN so as to minimize the loss function L and updating the weight of the neural network NN with the calculated weight) is executed.

With this configuration, in the learning device according to the present embodiment (in this example, the learning unit 1), the loss function L used for the second learning is combined with the first learning (in this example, the convolution using the similar image PR as an input). Since the result of (learning by autoencoder) can be reflected, the accuracy of judgment is higher than that in the case where the loss function L is not based on the result of the first learning (in this example, learning by the convolutional autoencoder using the similar image PR as an input). Can be enhanced.

Further, the determination device according to the present embodiment (in this example, the determination unit 2) includes the determination unit 2. The determination unit 2 determines the target object (in this example, the neural network NN whose weight is updated by the learning process of the learning unit 1) in the target image PA based on the learning result by the learning device (in this example, the learning unit 1). In this example, the brain tissue cell nucleus A2) is determined.

With this configuration, in the determination device (determination unit 2 in this example) according to the present embodiment, the learning result by the learning device (learning unit 1 in this example) (in this example, the weight is weighted by the learning process of the learning unit 1). Since the determination can be made based on the updated neural network NN), the learning result by the learning device (in this example, the learning unit 1) (in this example, the weight is updated by the learning process of the learning unit 1). Judgment accuracy can be improved as compared with the case where it is not based on.

(Second Embodiment)
Hereinafter, the second embodiment of the present invention will be described in detail with reference to the drawings.
In the first embodiment, the case where the image processing system processes an image of the brain tissue cell nucleus captured by an electron microscope as a target image and an image of the iPS cell nucleus captured by an optical microscope as a similar image will be described. did. In the present embodiment, the case where the image processing system processes the image of the iPS cell nucleus captured by the optical microscope as the target image and the image of the brain tissue cell nucleus captured by the electron microscope as a similar image will be described.
The image processing system according to this embodiment is referred to as an image processing system STa, and the image processing apparatus is referred to as an image processing apparatus T1a.

FIG. 12 is a diagram showing an example of the image processing system STa according to the present embodiment. As an example, the image processing system STa includes an electron microscope M2, an optical microscope M1a, an electron microscope image database DB2, a DAPI image database DB1, a bright field image database DB3a, and an image processing device T1a.
Comparing the image processing system STa (FIG. 12) according to the present embodiment with the image processing system ST (FIG. 1) according to the first embodiment, the optical microscope M1a, the bright field image database DB3a, and the image processing apparatus T1a different. Here, the functions of the other components (electron microscope M2, electron microscope image database DB2, and DAPI image database DB1) are the same as those of the first embodiment. The description of the same function as that of the first embodiment is omitted, and in the second embodiment, the parts different from those of the first embodiment will be mainly described.

In the image processing system STa, the optical microscope M1a observes the iPS cell nucleus A1 by bright-field observation, and observes the iPS cell nucleus A1 stained by DAPI by fluorescence observation. The iPS cell nucleus A1 observed by bright-field observation is imaged by an optical microscope M1a and accumulated in the bright-field image database DB3a as a target image PAa. A part of the target image PAa stored in the bright field image database DB3a is supplied to the learning unit 1a of the image processing device T1a and used for learning processing, and the remaining part is supplied to the determination unit 2 of the image processing device T1a for determination. Used for processing.
On the other hand, the iPS cell nucleus A1 observed by fluorescence observation is imaged by the optical microscope M1a and accumulated in the DAPI image database DB1 as the correct image source image PNa.

The learning unit 1a selects an image region of the iPS cell nucleus A1 in the correct image source image PNa by image recognition, and generates a nuclear correct image PT1a showing the selected region. The learning unit 1a executes learning based on the teacher data LDa, which is a set of the target image PAa and the kernel correct image PT1a.

In the image processing system STa, the brain tissue cell nucleus A2 is imaged by the electron microscope M2. The image of the brain tissue cell nucleus A2 imaged by the electron microscope M2 is accumulated in the electron microscope image database DB2 as a similar image source image PMa.
The nuclear correct image PT2a is created based on the image of the brain tissue cell nucleus A2 accumulated in the electron microscope image database DB2. Here, in the nuclear correct answer image PT2a, the inside of this region is painted or the outline of this region is drawn by a person having medical knowledge about the region showing the brain tissue cell nucleus A2 in the image of the brain tissue cell nucleus A2. Created by The created nuclear correct image PT2a is stored in the electron microscope image database DB2 and supplied to the image processing apparatus T1a as a similar image PR1a.

The learning unit 1a learns the feature amount of the iPS cell nucleus A1 using the similar image PR1a. In the image processing system STa, the brain tissue cell nucleus A2 is adopted as an example of a similar object having a shape similar to that of the iPS cell nucleus A1.

In the learning process of the learning unit 1a and the learning process of the learning unit 1 described with reference to FIGS. 4 and 5, the cell nucleus of the first site, the cell nucleus of the second site, the first microscope, the second microscope, and the first observation condition, And the second observation condition is different. In the learning process of the learning unit 1a, the cell nucleus of the first site is the brain tissue cell nucleus A2, and the cell nucleus of the second site is the iPS cell nucleus A1. Further, in the learning process of the learning unit 1a, the first microscope is an electron microscope M2, and the second microscope is an optical microscope M1a.

Further, the learning process of the learning unit 1a and the learning process of the learning unit 1 are different in that the method of generating the core correct image of step S30 and step S90 in FIG. 4 is replaced with the following steps S30a and S90a. ..
Step S30a: The nuclear correct image PT2a is created. Here, in the nuclear correct answer image PT2a, the inside of the region showing the brain tissue cell nucleus A2 in the similar image original image PMa is painted or the outline of this region is drawn by a person having medical knowledge. Created by.

Step S90a: A nuclear correct image PT1a is generated for the target image PAa for learning. In the present embodiment, the nuclear correct image PT1a is generated from the target image PAa by the learning unit 1a of the image processing device T1a. The learning unit 1a selects an image region of the iPS cell nucleus A1 in the target image PAa by image recognition, and generates a nuclear correct answer image PT1a showing the selected region. The pair of the target image PAa for learning and the kernel correct image PT1a is used as the teacher data LDa of the learning process performed by the image processing device T1a.

Next, with reference to FIG. 13, a specific example of how the image of the iPS cell nucleus A1 and the image of the brain tissue cell nucleus A2 are used in the learning process will be described. FIG. 13 is a diagram showing an example of the relationship between the image and the learning process according to the present embodiment.

In FIG. 13, the output of the neural network NN when the target image PAa, which is an image of the iPS cell nucleus A1, is used as the input of the neural network NN is shown as a value φ _θ . It is shown that the value φ _θ (x), which is the output of the neural network NN when the target image PAa is input, approaches the pixel value of the correct image PTa by updating the parameter θ using the similar image PRa. .. The correct image PTa is a nuclear correct image PT1a showing the region of the image of the iPS cell nucleus A1.

In FIG. 13, the nuclear correct image PT2a showing the region of the image of the brain tissue cell nucleus A2 is used as the similar image PRa, and the characteristics of the shape of the brain tissue cell nucleus A2 are learned by CAEf.
In FIG. 13, the loss function L is calculated based on the target image PAa, the correct image PTa, and the CAEf learned from the shape characteristics of the brain tissue cell nucleus A2 imaged in the similar image PRa.

Next, the segmentation result of the image processing apparatus T1a will be described with reference to FIGS. 14 and 15.
FIG. 14 is a diagram showing an example of the determination image PE1a according to the present embodiment. The determination image PE1a is, for example, a Z-stack image in which the iPS cell nucleus A1 is imaged by an optical microscope M1a based on bright-field observation. The determination image PE1a includes three cross-sectional images in which the cross-sections of the iPS cell nucleus A1 are imaged in three different directions. In the determination image PE1a, the depths other than the depth corresponding to the middle depth of the iPS cell nucleus A1 imaged in the Z stack image are defocused.

FIG. 15 is a diagram showing an example of determination accuracy according to the present embodiment. Graph G31, graph G32, graph G33, and graph G34 show the value of the Dice coefficient for each depth of the Z stack image as the accuracy of segmentation.
Graph G34 shows the segmentation accuracy by the image processing apparatus T1a. The segmentation accuracy shown in the graph G34 is the segmentation accuracy by the neural network in which the VAT loss and the shape loss are added to the cross entropy loss and learned by using it as a loss function.

Graphs G31, G32, and graph G33 are shown for comparison with the segmentation accuracy of the image processing apparatus T1a. The segmentation accuracy shown in the graph G31 is the segmentation accuracy by the neural network trained by using the cross entropy loss as a loss function. The segmentation accuracy shown in the graph G32 is the segmentation accuracy by the neural network in which the shape loss is added to the cross entropy loss and the learning is performed by using it as a loss function. The segmentation accuracy shown in the graph G33 is the segmentation accuracy by the neural network in which the VAT loss is added to the cross entropy loss and the learning is performed by using it as a loss function.

From the comparison between the graph G32 and the graph G31 and the comparison between the graph G34 and the graph G33, when the learning is performed by adding the shape loss to the loss function, the learning is performed without adding the shape loss to the loss function. It can be seen that the segmentation accuracy is improved as compared with the case where it is performed. That is, the segmentation accuracy is improved by using the shape of the brain tissue cell nucleus A2 as the foresight information.

Further, from the comparison between the graph G33 and the graph G31 and the comparison between the graph G34 and the graph G32, when the VAT loss is added to the loss function and learning is performed, the learning is performed without adding the VAT loss to the loss function. It can be seen that the segmentation accuracy is improved as compared with the case where.

From the graph G31, the graph G32, the graph G33, and the graph G34, it can be seen that the segmentation accuracy by the image processing apparatus T1a is higher than the other segmentation accuracy. That is, the segmentation accuracy by the neural network in which the shape loss and the VAT loss are added to the cross entropy loss and learned by using it as a loss function is the highest as compared with other segmentation accuracy.

(Third Embodiment)
Hereinafter, a third embodiment of the present invention will be described in detail with reference to the drawings.
In the first embodiment and the second embodiment, in the image processing system, one of an image in which the brain tissue cell nucleus is captured by an electron microscope and an image in which the iPS cell nucleus is captured by an optical microscope is a target image. The case where the other image is regarded as a similar image has been described. In this embodiment, a case where one of the images captured by different imaging methods of the brain tissue cell nucleus is used as a target image and the other is used as a similar image will be described. In the third embodiment, the image of the brain tissue cell nucleus captured by the optical microscope is used as a similar image, and the image of the brain tissue cell nucleus captured by the electron microscope is used as the target image.
The image processing system according to this embodiment is referred to as an image processing system STb, and the image processing apparatus is referred to as an image processing apparatus T1b.

FIG. 16 is a diagram showing an example of the image processing system STb according to the present embodiment. As an example, the image processing system STb includes an electron microscope M2, an optical microscope M1a, an electron microscope image database DB2, a DAPI image database DB1b, a bright field image database DB3b, and an image processing device T1b.
Comparing the image processing system STb (FIG. 16) according to the present embodiment with the image processing system STa (FIG. 12) according to the second embodiment, the DAPI image database DB1b, the brightfield image database DB3b, and the image processing apparatus T1b Is different. Here, the functions of the other components (optical microscope M1a, electron microscope M2, and electron microscope image database DB2) are the same as those of the second embodiment. The description of the same function as that of the second embodiment will be omitted, and in the third embodiment, the parts different from those of the second embodiment will be mainly described.

In the image processing system STb, the optical microscope M1a observes the brain tissue cell nucleus A2 by bright-field observation, and observes the brain tissue cell nucleus A2 stained by DAPI by fluorescence observation. The brain tissue cell nucleus A2 observed by bright-field observation is imaged by an optical microscope M1a, and the captured image is accumulated in the bright-field image database DB3b. On the other hand, the brain tissue cell nucleus A2 observed by fluorescence observation is imaged by an optical microscope M1a and accumulated in the DAPI image database DB1b as a similar image source image PMb.

The learning unit 1b selects an image region of the brain tissue cell nucleus A2 in the similar image source image PMb by image recognition, and generates a nuclear correct image PT1b showing the selected region. The created nuclear correct image PT1b is stored in the DAPI image database DB1b and supplied to the image processing device T1b as a similar image PR1b.

In the image processing system STb, the brain tissue cell nucleus A2 is imaged by the electron microscope M2. The image of the brain tissue cell nucleus A2 imaged by the electron microscope M2 is accumulated in the electron microscope image database DB2 as the target image PA1b. A part of the target image PA1b stored in the electron microscope image database DB2 is supplied to the learning unit 1b of the image processing device T1b and used for the learning process, and the remaining part is supplied to the determination unit 2 of the image processing device T1b for determination. Used for processing.

The nuclear correct image PT2b is created based on the image of the brain tissue cell nucleus A2 accumulated in the electron microscope image database DB2. Here, in the nuclear correct answer image PT2b, the inside of this region is painted or the outline of this region is drawn by a person having medical knowledge about the region showing the brain tissue cell nucleus A2 in the image of the brain tissue cell nucleus A2. Created by The created nuclear correct image PT2b is stored in the electron microscope image database DB2 and supplied to the image processing apparatus T1b as the nuclear correct image PT1b.

The learning unit 1b learns the feature amount of the brain tissue cell nucleus A2 using the similar image PR1b. In the image processing system STb, as an example of a similar object having a shape similar to that of the brain tissue cell nucleus A2 imaged by the electron microscope M2, the brain tissue cell nucleus A2 stained by DAPI and imaged by fluorescence observation on the optical microscope M1a is adopted. ..

The cell nucleus of the first site is different between the learning process of the learning unit 1b and the learning process of the learning unit 1 described with reference to FIGS. 4 and 5. In the learning process of the learning unit 1b, the cell nucleus of the first site is the brain tissue cell nucleus A2.

Further, the learning process of the learning unit 1b and the learning process of the learning unit 1 are different in that the method of generating the core correct image in step S30 of FIG. 4 is replaced with the following step S30b.
Step S30b: The nuclear correct image PT1b is created. Here, in the nuclear correct image PT1b, the inside of the region showing the brain tissue cell nucleus A2 in the similar image original image PMb is painted or the outline of this region is drawn by a person having medical knowledge. Created by.

In the present embodiment, the similar image PRb is imaged and generated by the optical microscope M1a which is the first microscope under the first observation condition, and the target image PAb is imaged by the electron microscope M2 which is the second microscope under the second observation condition. Is generated. That is, in the present embodiment, the imaging method is different between the similar image PRb and the target image PAb.

The electron microscope has a higher resolution than the optical microscope. On the other hand, in an electron microscope, observation is not as easy as observation with an optical microscope because a high voltage is required for electron beam irradiation and the inside of the microscope needs to be kept in a vacuum. Observation with an optical microscope is more efficient than observation with an electron microscope in that observation is simple.
The case where the brain tissue cell nucleus A2 stained by DAPI is imaged by fluorescence observation with the optical microscope M1a to generate an image is more than the case where the image is generated by being imaged by the electron microscope M2 of the brain tissue cell nucleus A2. The similar image PRb can be efficiently generated.

Further, the case where the nuclear correct image PT1b is generated from the similar image original image PMb by image recognition is the case where the nuclear correct image PT2b is created by humans from the image of the brain tissue cell nucleus A2 captured by the electron microscope M2. In comparison, similar image PRb can be generated efficiently.

The imaging method may be the same for the similar image PRb and the target image PAb. For example, the similar image PRb and the target image PAb may both be imaged by the first microscope under the first observation condition. Alternatively, the similar image PRb and the target image PAb may both be imaged by the second microscope under the second observation condition.

As described above, in the learning apparatus according to the present embodiment (in this example, the learning unit 1b), the similar image PRb and the target image PAb have an imaging method (in this example, the optical microscope M1a which is the first microscope). The imaging under the first observation condition or the imaging under the second observation condition by the electron microscope M2 which is the second microscope) is different from each other.

With this configuration, the learning device according to the present embodiment (in this example, the learning unit 1b) can use an imaging method that is more efficient than the imaging method of the target image PAb as the imaging method of the similar image PRb. Similar image PR can be obtained more efficiently than when the imaging method is common between the similar image PRb and the target image PAb.

When, for example, an image captured by an imaging method different from the imaging method in which the target image is captured is used as the similar image PRb, for example, the segmentation of the image captured by the first imaging method is performed by the second imaging method. Since the captured image may be used and the image captured by the first imaging method may be used for the segmentation of the image captured by the second imaging method, the efficiency of segmentation can be improved.

(Fourth Embodiment)
Hereinafter, a fourth embodiment of the present invention will be described in detail with reference to the drawings.
In the third embodiment, another example will be described in which the image processing system uses one of the images captured by different imaging methods in the brain tissue cell nucleus as the target image and the other as the similar image. To do. In the fourth embodiment, the image of the brain tissue cell nucleus captured by the optical microscope is used as the target image, and the image of the brain tissue cell nucleus captured by the electron microscope is used as the similar image.
The image processing system according to this embodiment is referred to as an image processing system STc, and the image processing apparatus is referred to as an image processing apparatus T1c.

FIG. 17 is a diagram showing an example of the image processing system STc according to the present embodiment. As an example, the image processing system STc includes an electron microscope M2, an optical microscope M1a, an electron microscope image database DB2c, a DAPI image database DB1c, a brightfield image database DB3c, and an image processing device T1c.
Comparing the image processing system STc (FIG. 17) according to the present embodiment and the image processing system STb (FIG. 16) according to the third embodiment, the DAPI image database DB1c, the brightfield image database DB3c, and the electron microscope image database DB2c , And the image processing device T1c are different. Here, the functions of the other components (optical microscope M1a, electron microscope M2) are the same as those of the third embodiment. The description of the same function as that of the third embodiment will be omitted, and in the fourth embodiment, the parts different from those of the third embodiment will be mainly described.

In the image processing system STc, the optical microscope M1a observes the brain tissue cell nucleus A2 by bright-field observation, and observes the brain tissue cell nucleus A2 stained by DAPI by fluorescence observation. The brain tissue cell nucleus A2 observed by bright-field observation is imaged by an optical microscope M1a and accumulated in the bright-field image database DB3b as a target image PAc. On the other hand, the brain tissue cell nucleus A2 observed by fluorescence observation is imaged by an optical microscope M1a and accumulated in the DAPI image database DB1b as a nuclear correct image PT1c.

In the image processing system STb, the brain tissue cell nucleus A2 is imaged by the electron microscope M2. The image of the brain tissue cell nucleus A2 imaged by the electron microscope M2 is accumulated in the electron microscope image database DB2c as a similar image source image PMc.

The nuclear correct image PT2c is created based on the image of the brain tissue cell nucleus A2 accumulated in the electron microscope image database DB2c. Here, in the nuclear correct answer image PT2c, the inside of this region is painted or the outline of this region is drawn by a person having medical knowledge about the region showing the brain tissue cell nucleus A2 in the image of the brain tissue cell nucleus A2. Created by The created nuclear correct image PT2c is stored in the electron microscope image database DB2c and supplied to the image processing apparatus T1c as a similar image PRc.

The learning unit 1c learns the feature amount of the brain tissue cell nucleus A2 using the similar image PR1c. In the image processing system STc, the brain tissue cell nucleus A2 imaged by the electron microscope M2 is adopted as an example of a similar object having a shape similar to that of the brain tissue cell nucleus A2 imaged by the optical microscope M1a.

The cell nucleus of the second site is different between the learning process of the learning unit 1c and the learning process of the learning unit 1a of the second embodiment. In the learning process of the learning unit 1c, the cell nucleus of the second site is the brain tissue cell nucleus A2. Other than that, the learning process of the learning unit 1c and the learning process of the learning unit 1a of the second embodiment are the same.

In the present embodiment, the similar image PRc is imaged and generated by the electron microscope M2 which is the second microscope under the second observation condition, and the target image PAc is imaged by the optical microscope M1a which is the first microscope under the first observation condition. Is generated. That is, in the present embodiment, the imaging method is different between the similar image PRc and the target image PAc.

In the third embodiment and the fourth embodiment described above, an example in which fluorescence observation is performed by an optical microscope has been described, but the present invention is not limited to this. In the third embodiment and the fourth embodiment, the fluorescence observation may be performed by a confocal microscope, a super-resolution microscope, or the like.

(Modification example)
In each of the above-described embodiments, an example in which the target object is a cell nucleus having a granular shape and the similarity is calculated as the L1 distance between Hu moments has been described, but the present invention is not limited to this. With reference to FIG. 18, other examples of the target object and the similarity will be described.

FIG. 18 is a diagram showing an example of correspondence between the target object and the degree of similarity according to the modified example of the embodiment. When the target object is a cell nucleus, mitochondria, neurovexil, etc., the shape of the target object is granular. When the shape of the target object is granular, the similarity is calculated as the L1 distance between the image of the target object and the Hu moment of each of the images of the target object, as in each of the above-described embodiments.
On the other hand, when the target object is a neurite, a blood vessel, a cell membrane (or cytoplasm), a filament, or the like, the shape of the target object is elongated. When the shape of the target object is an elongated shape, the similarity is calculated as, for example, the distance between image sets of blood vessel-like images.

Here, with reference to FIGS. 19 and 20, the distance between image sets of blood vessel-like images will be described. FIG. 19 is a diagram showing an example of a process of calculating the distance between image sets of a blood vessel-like image according to a modified example of the embodiment. FIG. 20 is a diagram showing an example of a blood vessel image and a blood vessel-like image according to a modified example of the embodiment.
The calculation process shown in FIG. 19 is executed by, for example, the similarity calculation unit 12 of the image processing devices T1, T1a, T1b, and T1c. In the example of the calculation process shown in FIG. 19, the distance between the image sets is calculated for the image set A and the image set B. The image set A is, for example, a plurality of target images, and the image set B is, for example, a plurality of similar images. In the image set A which is a plurality of target images, the target object is, for example, a blood vessel, and in the image set B which is a plurality of similar images, the similar object is, for example, a neurite.

Step S400: The similarity calculation unit 12 acquires the image set A. The blood vessel image PV1 of FIG. 20 is an example of an image included in the image set A, and the blood vessel is imaged.
Step S410: The similarity calculation unit 12 applies the blood vessel extraction filter to the acquired image set A. The blood vessel extraction filter is a filter for extracting the characteristics of the shape of blood vessels. The shape characteristics of blood vessels include the characteristic of being elongated.

Step S420: The similarity calculation unit 12 generates a blood vessel-like image VA as a result of applying the blood vessel extraction filter to the image set A. The blood vessel-like image PV2 of FIG. 20 is an example of a blood vessel-like image VA generated by applying a blood vessel extraction filter to the blood vessel image PV1.
Step S430: The similarity calculation unit 12 executes a histogram extraction process on the blood vessel-like image VA.

Step S440: The similarity calculation unit 12 acquires the image set B.
Step S450: The similarity calculation unit 12 applies the blood vessel extraction filter to the acquired image set B.
Step S460: The similarity calculation unit 12 generates a blood vessel-like image VB as a result of applying the blood vessel extraction filter to the image set B.
Step S470: The similarity calculation unit 12 executes a histogram extraction process on the blood vessel-like image VB.

Step S480: The similarity calculation unit 12 calculates the distance between the histograms of the blood vessel-like image VA and the blood vessel-like image VB as the similarity. Here, the distance between histograms is, for example, the L1 distance.

In each of the above-described embodiments, an example in which the learning units 1, 1a, 1b, 1c, and the determination unit 2 are provided in the image processing devices T1, T1a, T1b, and T1c, respectively, has been described, but the present invention is limited to this. Absent. The learning units 1, 1a, 1b, 1c, and the determination unit 2 may be provided as separate devices in each of the image processing systems ST, STa, STb, and STc as learning devices and determination devices, respectively.

In each of the above-described embodiments, in each of the image processing systems ST, STa, STb, and STc, the image processing devices T1, T1a, T1b, and T1c are separate from the microscopes such as the optical microscopes M1, M1a, and the electron microscope M2. An example of the case where the system is provided as is described. That is, in each embodiment, an example in which the learning device and the determination device are provided as a separate body from the microscope has been described, but the present invention is not limited to this. The learning device and the determination device may be provided integrally with the microscope. That is, in the image processing system ST, the microscope may include the learning device described above. Further, in the image processing system ST, the microscope may be provided with the above-mentioned determination device. Further, in the image processing system ST, the microscope may include the above-mentioned learning device and the above-mentioned determination device.

In addition, a part of the image processing apparatus T1, T1a, T1b, T1c in the above-described embodiment, for example, learning units 1, 1a, 1b, 1c, and determination unit 2 may be realized by a computer. In that case, the program for realizing this control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by the computer system and executed. The "computer system" referred to here is a computer system built into the image processing devices T1, T1a, T1b, and T1c, and includes hardware such as an OS and peripheral devices. Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, or a storage device such as a hard disk built in a computer system. Furthermore, a "computer-readable recording medium" is a medium that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In that case, a program may be held for a certain period of time, such as a volatile memory inside a computer system serving as a server or a client. Further, the above-mentioned program may be a program for realizing a part of the above-mentioned functions, and may further realize the above-mentioned functions in combination with a program already recorded in the computer system.
Further, a part or all of the image processing devices T1, T1a, T1b, and T1c in the above-described embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the image processing devices T1, T1a, T1b, and T1c may be individually made into a processor, or a part or all of them may be integrated into a processor. Further, the method of making an integrated circuit is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. Further, when an integrated circuit technology that replaces an LSI appears due to advances in semiconductor technology, an integrated circuit based on this technology may be used.

Although one embodiment of the present invention has been described in detail with reference to the drawings, the specific configuration is not limited to the above, and various design changes and the like are made without departing from the gist of the present invention. It is possible to do.

T1, T1a, T1b, T1c ... Image processing device, 1, 1a, 1b, 1c ... Learning unit, 2 ... Judgment unit, 130 ... Similar image learning unit, 132 ... Target image learning unit, PR ... Similar image, PA ... Target Image, PT ... Correct image, LD ... Teacher data

Claims (12)

A similar image learning unit that executes the first learning based on a similar image that is an image of a similar object that is an object similar in appearance to the target object to be determined,
Teacher data that is a set of a target image that is an image of the target object and a correct image that is an image showing the target object in the target image, and the first learning by the similar image learning unit. The target image learning unit that executes the second learning based on the result of
A learning device equipped with. A similar object extraction unit for extracting the similar object from the similar image is further provided.
The learning device according to claim 1, wherein the similar image learning unit executes the first learning by using an image showing the similar object extracted by the similar object extraction unit as the similar image. The learning device according to claim 1 or 2, wherein the similar object has a value indicating a degree of similarity in appearance with respect to the target object of a predetermined value or more. The learning device according to any one of claims 1 to 3, wherein the imaging method is different between the similar image and the target image. The target image learning unit is any of claims 1 to 4 that executes the second learning based on the loss function based on the result of the first learning by the similar image learning unit and the teacher data. The learning device according to item 1. A determination device including a determination unit that determines the target object in the target image based on the learning result of the learning device according to any one of claims 1 to 5. A microscope provided with the learning apparatus according to any one of claims 1 to 5. A microscope provided with the determination device according to claim 6. The learning device according to any one of claims 1 to 5.
A microscope including a determination device including a determination unit for determining the target object in the target image based on the learning result of the learning device. A trained model trained by the learning device according to any one of claims 1 to 5. On the computer
A similar image learning step of executing the first learning based on a similar image which is an image of a similar object which is an object similar in appearance to the target object to be determined,
Based on the teacher data which is a set of the target image which is the image obtained by the target object and the correct image which is the image showing the target object in the target image, and the learning result of the similar image learning step. A program for executing the target image learning step for executing the second learning. On the computer
A program for executing a determination step of determining the target object in the target image based on the learning result of the program according to claim 11.