WO2018221625A1 - System and method for diagnostic support using pathological image of skin tissue - Google Patents

Abstract

A diagnostic support system for performing dermatosis diagnostic support comprises an output data generation unit that generates output data related to dermatosis on the basis of the outputs from: (A) a learned model of machine learning in which the inputs are an image representing the distribution of dermal cells in the skin tissue included in a pathological image, an image representing the distribution of epidermal cells in skin tissue, and an image representing the distribution of atypical cells in the skin tissue; and (B) a learned model for a first image representing the distribution of dermal cells in the skin tissue to be processed, a second image representing the distribution of epidermal cells in the skin tissue to be processed, and a third image representing the distribution of atypical cells in the skin tissue to be processed.

Description

System and method for diagnosis support using pathological image of skin tissue

The present invention relates to a technique for supporting diagnosis using a pathological image of a skin tissue.

One paper describes what uses a convolutional neural network in two stages to detect Mitosis cells from a pathological image of a breast cancer pathological tissue. Here, it is sufficient to detect a specific type of cell, and it is not a problem to make a diagnosis in consideration of the distribution of the specific type of cell.

Other papers also use a convolutional neural network to detect the tumor area locally from prostate cancer pathological images and detect breast cancer metastasis in the sentinel lymph nodes locally. For this purpose, a convolutional neural network is described. In this paper, in the analysis of prostate cancer pathological images, the histogram of likelihood of tumors or tumors from the entire pathological image was obtained, and the presence or absence of breast cancer metastasis was obtained locally. It is determined by scoring the maximum likelihood of whether there is metastasis, and it is not problematic to make a diagnosis in consideration of the distribution of specific types of cells.

Furthermore, some documents consider that changes in cell nuclei and surrounding tissues are important in identifying the nature of tumors, and subimages centered on cell nuclei, pores, cytoplasm, and stroma from pathological images. Is extracted and a sub-image is input as a learning pattern and an input pattern, so that the presence / absence of a tumor and the benign / malignant tumor of the tumor are determined with high accuracy based on the sub-image. This document considers that tissue collected in pathological examination is stained (stained with hematoxylene, eosin, etc.), so that cell nuclei and surrounding tissues are stained in unique colors. By extracting sub-images centering on cell nuclei, vacancies, cytoplasm, stroma, etc. from the pathological image, simultaneously extracting color information of cell nuclei and storing both as feature candidates, it is possible to obtain tumors with higher accuracy. The presence / absence and determination of benign / malignant tumor are also disclosed. However, elements characteristic of the pathological image in the skin tissue are not considered.

JP 2006-153742 A

An object of the present invention is, according to one aspect, to provide a novel technique for providing diagnosis support for a skin disease.

The first diagnosis support system of the present invention includes (A) an image representing the distribution of dermal cells in the skin tissue, an image representing the distribution of epidermal cells in the skin tissue, and the distribution of atypical cells in the skin tissue. A learned model of machine learning using as input an image representing the image, (B) a first image representing a distribution of dermal cells in the skin tissue to be processed, and a first image representing a distribution of epidermal cells in the skin tissue to be processed And an output data generation unit that generates output data related to the skin disease based on the output from the learned model for the second image and the third image representing the distribution of atypical cells in the skin tissue to be processed.

The second diagnosis support system of the present invention includes (A) an image representing the distribution of dermal cells in the skin tissue included in the pathological image, an image representing the distribution of epidermal cells in the skin tissue, and the distribution of immune cells in the skin tissue. A learned model of machine learning that receives an image representing the distribution of the cells forming the lumen in the skin tissue and an output representing the differentiation of a plurality of non-neoplastic skin diseases, and (B) processing A first image representing the distribution of dermal cells in the subject skin tissue, a second image representing the distribution of epidermal cells in the skin tissue to be treated, and a third image representing the distribution of immune cells in the skin tissue to be treated. Based on the output of the learned model of machine learning for the image and the fourth image representing the distribution of the cells that make up the lumen in the skin tissue to be processed, the discrimination result of a plurality of types of non-neoplastic skin diseases And an output data generating unit for generating output data including.

FIG. 1A is a diagram showing an example of a pathological image of a skin tissue containing malignant melanoma. FIG. 1B is a diagram illustrating an example of a pathological image of a skin tissue including pigmented nevus. FIG. 2 is a diagram illustrating an example of a convolutional neural network. FIG. 3A is a diagram showing an example of an image representing the distribution of dermal cells in skin tissue known to be malignant melanoma. FIG. 3B is a diagram showing an example of an image representing the distribution of epidermal cells in skin tissue known to be malignant melanoma. FIG. 3C is a diagram showing an example of an image representing the distribution of atypical cells in skin tissue known to be malignant melanoma. FIG. 4A is a diagram illustrating an example of an image representing a distribution of dermal cells in a skin tissue known to be pigmented nevus. FIG. 4B is a diagram showing an example of an image representing the distribution of epidermal cells in skin tissue known to be pigmented nevus. FIG. 4C is a diagram illustrating an example of an image representing the distribution of atypical cells in skin tissue known to be pigmented nevus. FIG. 5 is a diagram illustrating a functional block configuration of the information processing apparatus according to the first embodiment. FIG. 6 is a diagram illustrating a processing flow executed by the information processing apparatus according to the first embodiment. FIG. 7 is a diagram for explaining divided images. FIG. 8A is a diagram for explaining an initial image of the distribution of dermal cells. FIG. 8B is a diagram for explaining an initial image of the distribution of epidermal cells. FIG. 8C is a diagram for explaining an initial image of atypical cell distribution. FIG. 9 is a diagram showing an example of an image representing the distribution of immune cells in skin tissue known to be malignant melanoma. FIG. 10 is a diagram showing an example of an image representing the distribution of immune cells in skin tissue known as pigmented nevus. FIG. 11 is a diagram showing an example of an image representing a distribution of cells that form a lumen in skin tissue known as malignant melanoma. FIG. 12 is a diagram illustrating an example of an image representing a distribution of cells that form a lumen in skin tissue known as pigmented nevus. FIG. 13 is a diagram illustrating a functional block configuration of the information processing apparatus according to the second embodiment. FIG. 14 is a diagram illustrating a processing flow executed by the information processing apparatus according to the second embodiment. FIG. 15A is a diagram illustrating a relationship between a disease name to be identified, a cell name that is derived from an atypical cell related to a tumor, and a cell type used for processing. FIG. 15B is a diagram illustrating a relationship between a disease name to be identified, a cell name from which an atypical cell related to a tumor is derived, and a cell type used for processing. FIG. 15C is a diagram illustrating a relationship between a disease name to be identified, a cell name derived from an atypical cell related to a tumor, and a cell type used for processing. FIG. 16 is a diagram showing the relationship between the names of non-neoplastic diseases and the types of cells used for treatment.

[Embodiment 1]
The purpose of this embodiment is to distinguish malignant melanoma from pigmented nevus. Malignant melanoma is a cancer of pigment cells present in the epidermis. On the other hand, pigmented nevus is a benign tumor of pigment cells, and it is often difficult to distinguish them.

FIG. 1A shows an example of a pathological image of a skin tissue containing malignant melanoma (however, a gray scale image). FIG. 1B shows an example of a pathological image of a skin tissue including pigmented nevus (however, a gray scale image). Thus, it is difficult to distinguish between the two. Such a pathological image is a digital image obtained by scanning a preparation of a pathological tissue.

In the present embodiment, as a first-stage process, the distribution of epidermal cells, the distribution of dermal cells, and the distribution of atypical cells in the skin tissue are specified in such a pathological image. The atypical cells are either pigmented cells that are malignant melanoma or pigmented nevi that are benign tumors. Hereinafter, an atypical cell shall mean a cell that has become tumorous or is not normal.

Next, as a second stage process, an image representing the distribution of epidermal cells (an image representing the distribution is also referred to as a heat map), an image representing the distribution of dermal cells, and an image representing the distribution of atypical cells are used as malignant cells. Distinguish between melanoma and benign tumor, pigmented nevus, and others. However, you may make it distinguish malignant melanoma from others.

In the present embodiment, it is intended to estimate the malignancy of a pathological tissue based on the distribution of atypical cells relative to the distribution of epidermal cells and the distribution of dermal cells. That is, it is intended to predict the malignancy of a pathological tissue based on the distribution of the atypical cells in which part of the epidermis or dermis. In other words, it evaluates how an atypical cell group enters a dermis cell group or epidermis cell group that expresses the structure of the skin and exists from the normal time. Such prediction and evaluation are realized by performing learning on a convolutional neural network (CNN: Convolutional Neural Network).

The convolutional neural network has a structure as shown in FIG. It has a structure similar to the well-known convolutional neural network called AlexNet, and there are multiple combinations of convolutional layers and subsampling layers (pooling layers) between the input layer and the output layer ( In the figure, 2) a set is provided, and two additional coupling layers are added. However, there are various variations in the structure of the convolutional neural network, and these may be used. For example, the number of combinations of convolution layers and subsampling layers may be increased.

As described above, an image representing the distribution of epidermal cells, an image representing the distribution of dermal cells, and an image representing the distribution of atypical cells are input to the input layer. On the other hand, in the output layer, for example, the likelihood of malignant melanoma, the likelihood of pigmented nevus, and the likelihood in other cases are output.

As is well known, the convolutional layer has parameters for the filter and activation function, the subsampling layer also has parameters for subsampling, and the fully connected layer also has parameters for the activation function, so An appropriate value or the like is set for them by learning. A known method is used as an algorithm for learning.

FIG. 3A shows an image representing the distribution of dermal cells in skin tissue known to be malignant melanoma. FIG. 3B shows an image representing the distribution of epidermal cells in skin tissue known to be malignant melanoma. FIG. 3C shows an image representing the distribution of atypical cells in skin tissue known to be malignant melanoma. Learning is performed for the purpose that when such an image is input to the input layer, an output having a likelihood of being malignant melanoma of 1 is output from the output layer. In addition, in these figures, it has shown that the cell to which its attention exists exists in the white part.

On the other hand, FIG. 4A shows an image representing the distribution of dermal cells in the skin tissue known to be pigmented nevus. FIG. 4B shows an image representing the distribution of epidermal cells in skin tissue known to be pigmented nevus. FIG. 4C shows an image representing the distribution of atypical cells in skin tissue known to be pigmented nevus. When such an image is input to the input layer, learning is performed for the purpose of outputting from the output layer a likelihood of pigmentous nevus being 1. In these figures, the white cells indicate that the cells of interest exist.

Note that a convolutional neural network may also be used in the first stage processing. In this case, a partial image of a pathological image including skin tissue (a partial image having a size similar to that of one cell or a partial image including one extracted cell) is input, and dermal cells and epidermis are input. A trained convolutional neural network that outputs the likelihood of cells, atypical cells, and others (glass portion, etc.) is used. However, for the first stage processing, the distribution of epidermal cells, the distribution of dermal cells, and the distribution of atypical cells may be extracted from the pathological image using other techniques such as region extraction.

Hereinafter, the configuration of the information processing apparatus that performs such processing and details of the processing contents will be described.

FIG. 5 shows a functional configuration example of the information processing apparatus according to the present embodiment.

The information processing apparatus according to the present embodiment includes an input unit 101, a pathological image storage unit 103, a first preprocessing unit 105, a first image storage unit 107, a first CNN 109, and a first post processing unit 111. , Second image storage unit 113, second pre-processing unit 115, third image storage unit 117, second CNN 119, second post-processing unit 121, result storage unit 123, and first learning processing unit 131 And a second learning processing unit 133.

The input unit 101 receives an input of a pathological image to be processed from a user and stores it in the pathological image storage unit 103.

As described above, the pathological image is a digital image obtained by scanning a preparation of a pathological tissue. Here, for example, it is an image scanned by enlarging the objective 20 times, for example, an RGB image of 1M × 1M pixels in length and width.

The first preprocessing unit 105 divides the pathological image stored in the pathological image storage unit 103 into a size of about one cell for calculation in the first CNN 109, and divides the pathological image (hereinafter referred to as a divided image). ) Is stored in the first image storage unit 107. For example, a square divided image of 32 × 32 pixels is generated.

The first CNN 109 is a convolutional neural network learned to determine whether each divided image stored in the first image storage unit 107 corresponds to a dermal cell, epidermal cell, atypical cell, or other cell. is there. Accordingly, the first CNN 109 outputs, for example, respective likelihoods for the divided images related to the processing.

Based on the output from the first CNN 109, the first post-processing unit 111 generates an initial image that represents the distribution of dermal cells, an initial image that represents the distribution of epidermal cells, and an initial image that represents the distribution of atypical cells. Store in the storage unit 113. For example, when it is determined that the divided image is a dermal cell, in the initial image representing the distribution of the dermal cell, all the pixels in the region of the position of the divided image are “1”, otherwise “0”. Is set as follows. That is, these initial images are monochrome images. In such a case, the size of these initial images is the same as the pathological image.

The second preprocessing unit 115 generates, from the image stored in the second image storage unit 113, an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, and an image representing the distribution of atypical cells. The image is stored in the image storage unit 117. In the present embodiment, a process of reducing each initial image to an image size that can be processed by the second CNN 119 (for example, vertical and horizontal 200 × 200 pixels) is executed. For example, in this process, a process of converting each pixel to a predetermined gradation (256 gradations) is also performed. That is, the image representing the distribution of dermal cells, the image representing the distribution of epidermal cells, and the image representing the distribution of atypical cells are grayscale images.

The second CNN 119 outputs whether or not it is malignant melanoma from the image stored in the third image storage unit 117. For example, the likelihood of malignant melanoma, the likelihood of pigmented nevus, and other likelihoods are output. Note that the likelihood of malignant melanoma and other likelihoods may be output.

The second post-processing unit 121 stores the discrimination result in the result storage unit 123 based on the output from the second CNN 119, and outputs the discrimination result to the display device or other output device.

In this way, when a pathological image is input, it is possible to automatically obtain a discrimination result as to whether or not the patient is malignant melanoma.

In this embodiment, the distribution of epidermal cells, the distribution of dermal cells, and the distribution of atypical cells are represented as separate images, so that the relationship between the distributions can be easily evaluated. This makes it possible to perform appropriate discrimination using a convolutional neural network.

The first learning processing unit 131 executes learning processing for the first CNN 109 using training data including a large number of sets of cell images and their types. Since the learning process method is the same as the conventional method, a detailed description is omitted.

The second learning processing unit 133 is training data including a large number of sets of images representing the distribution of dermal cells, images representing the distribution of epidermal cells, images representing the distribution of atypical cells, and discrimination results. Execute learning process. Since the learning process method is the same as the conventional method, a detailed description is omitted.

Next, processing contents of the information processing apparatus according to the present embodiment will be described with reference to FIGS. 6 to 8C.

First, the input unit 101 receives an input of a pathological image to be processed from a user and stores it in the pathological image storage unit 103 (step S1). Although it is a grayscale image due to a problem in expression, an image (color image) as shown in FIGS. 1A and 1B is input as a pathological image. The input pathological image may be stored in a storage unit of the information processing apparatus, a storage unit of another apparatus connected to the network, or the like. In this case, the input unit 101 reads out from the storage unit. And stored in the pathological image storage unit 103.

Then, the first preprocessing unit 105 performs the first preprocessing on the pathological image to be processed stored in the pathological image storage unit 103, and stores the processing result in the first image storage unit 107 (step). S3). In this first preprocessing, as schematically shown in FIG. 7, the pathological image 1001 is divided into a large number of divided images 1002 of N × N pixels in the vertical and horizontal directions. N is, for example, 32 or 64. Note that a divided image may be generated by performing processing for recognizing a cell nucleus to identify a cell and specifying an image region including the cell.

Thereafter, the first CNN 109 reads out each divided image stored in the first image storage unit 107, and executes a classification process on them (step S5). The classification process is a process for determining whether the divided image to be processed corresponds to a dermal cell, an epidermal cell, an atypical cell, or another cell. For example, the likelihood for each is output.

Then, the first post-processing unit 111 executes the first post-processing based on the output from the first CNN 109, and stores the processing result in the second image storage unit 113 (step S7). In the first post-processing, as schematically shown in FIG. 8A, in the initial image representing the distribution of dermal cells, the pixel values of all the pixels in the region 1002a at the position of the divided image determined to be dermal cells are “1”. To "". For the other pixels, the pixel value is set to “0”. Similarly, as schematically shown in FIG. 8B, in the initial image representing the distribution of epidermal cells, the pixel values of all the pixels in the region 1002b at the position of the divided image determined to be epidermal cells are set to “1”. To do. For the other pixels, the pixel value is set to “0”. Further, as schematically shown in FIG. 8C, in the initial image representing the distribution of atypical cells, the pixel values of all the pixels in the region 1002c at the position of the divided image determined to be atypical cells are set to “1”. . For the other pixels, the pixel value is set to “0”.

Note that such first post-processing is merely an example, and the size of the initial image representing the distribution may be reduced. For example, one pixel may be associated with each divided image, and the pixel value of one pixel at the position of the divided image may be set to “1” in the initial image representing the distribution. In this way, the size is 1 / N both vertically and horizontally with respect to the pathological image.

Next, the second preprocessing unit 115 performs second preprocessing on the image stored in the second image storage unit 113, and stores the processing result in the third image storage unit 117 (step S9). ). In the second preprocessing, image reduction processing is executed. In the image reduction process, the pixel value of the pixel after reduction is calculated by calculating the average of the pixel values of neighboring pixels. Since the pixel value in the initial image representing the distribution is “0” or “1”, the average is calculated to be a real number from “0” to “1”. This real value is linearly mapped so as to be an integer of, for example, “0” to “255”. As a result, the initial image (monochrome image) representing the distribution is converted into an image (grayscale image) representing the distribution. Note that a non-average method may be employed in the reduction process, but since it is the same as the prior art, it will not be described in detail here.

By executing such processing, an image representing the distribution of dermal cells as shown in FIGS. 3A and 4A, an image representing the distribution of epidermal cells as shown in FIGS. 3B and 4B, and FIGS. 3C and 4C An image representing the distribution of atypical cells as represented is obtained.

In the present embodiment, the reduction process is performed on the initial image representing the distribution due to the problem such as the load of the convolutional neural network. However, if there is no problem such as the load, the reduction process is not performed. good.

Then, the second CNN 119 reads out the images stored in the third image storage unit 117, and executes a discrimination process on them (step S11). In the present embodiment, a grayscale image group as shown in FIGS. 3A to 3C or 4A to 4C is processed as a grayscale image of each channel of a color image. The discrimination process is here whether or not it is malignant melanoma. For example, the likelihood of malignant melanoma, the likelihood of pigmented nevus, and other likelihoods are output. As mentioned above, it is also possible to differentiate malignant melanoma from others.

The second post-processing unit 121 stores the discrimination result in the result storage unit 123 based on the output from the second CNN 119, and outputs it to the display device or other output device (step S13). The output device may be another device connected to the network.

As described above, based on the distribution of atypical cells relative to the distribution of epidermal cells and the distribution of dermal cells in the skin tissue to be treated, the malignancy of the pathological tissue is learned, and an appropriate discrimination result can be obtained. become.

It should be noted that the examination can be simplified and the burden on the patient can be reduced.

[Modification A of Embodiment 1]
In some cases, auxiliary information such as the contour of the skin tissue and the distribution of other portions in the skin tissue in the images as shown in FIGS. 3A to 3C and FIGS. 4A to 4C may be useful. In such a case, the pathological image may be reduced to a gray scale image so as to match the size of the image representing the distribution.

That is, the second preprocessing unit 115 reads out the pathological image from the pathological image storage unit 103, reduces it to a predetermined size, and stores it in the third image storage unit 117.

The second CNN 119 is further learned by using a reduced grayscale image of a pathological image in addition to an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, and an image representing the distribution of atypical cells.

When the second CNN 119 performs the discrimination process, in addition to the image representing the distribution of dermal cells, the image representing the distribution of epidermal cells, and the image representing the distribution of atypical cells, a reduced grayscale image of the pathological image is input. To determine whether or not it is malignant melanoma.

This improves the discrimination accuracy.

[Modification B of Embodiment 1]
Furthermore, when attention is paid to the phenomenon that immune cells increase when the degree of progression of malignant melanoma increases, an image representing the distribution of immune cells may be further used as the input of the second CNN 119.

That is, the first CNN 109 is configured to determine whether each divided image is a dermal cell, an epidermal cell, an immune cell, an atypical cell, or the like, and an image of the dermal cell or an image of the epidermal cell. Learning is performed using images of immune cells and images of atypical cells. Then, the first CNN 109 outputs the likelihood for each of dermal cells, epidermal cells, immune cells, atypical cells, and the like.

Based on the output from the first CNN 109, the first post-processing unit 111 is an initial image representing the distribution of dermal cells, an initial image representing the distribution of epidermal cells, an initial image representing the distribution of immune cells, and an initial image representing the distribution of atypical cells. Generate an image. The initial image representing the distribution of immune cells is generated in the same manner as the initial image representing the other distribution.

Also, the second preprocessing unit 115 generates an image representing the distribution of immune cells from the initial image representing the distribution of immune cells in the same manner as an image representing other distributions. An image representing the distribution of immune cells in the skin tissue known as malignant melanoma is, for example, an image as shown in FIG. Further, an image representing the distribution of immune cells in the skin tissue known to be pigmented nevus is an image as shown in FIG. 10, for example. In these figures, the white cells indicate that the cells of interest exist. In FIG. 9, there is an overlap with the distribution of atypical cells.

Furthermore, the second CNN 119 is further learned using an image representing the distribution of immune cells in addition to an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, and an image representing the distribution of atypical cells.

When the second CNN 119 performs the discrimination process, in addition to the image representing the distribution of dermal cells, the image representing the distribution of epidermal cells, and the image representing the distribution of atypical cells, an image representing the distribution of immune cells is input. And discriminate between malignant melanoma, pigmented nevus and others.

This improves the discrimination accuracy.

[Modification C of Embodiment 1]
Furthermore, since it may be presumed that the pathological condition including the degree of progression of malignant melanoma and the distribution of cells forming the lumen are related, an image representing the distribution of the cells forming the lumen is displayed in the second CNN119. It may be further used as an input.

In this case, the cells are deformed to use cells that form a lumen instead of the immune cells in the modified example B.

That is, the first CNN 109 is configured to determine whether each divided image is a dermal cell, an epidermal cell, a luminal cell, an atypical cell, or the like. Learning is performed using the image of the cell, the image of the cell forming the lumen, and the image of the atypical cell. If it does so, 1st CNN109 will output the likelihood about each of a dermal cell, an epidermal cell, a cell which forms a lumen, an atypical cell, and others.

The first post-processing unit 111, based on the output from the first CNN 109, an initial image representing the distribution of dermal cells, an initial image representing the distribution of epidermal cells, an initial image representing the distribution of cells forming a lumen, and the distribution of atypical cells An initial image representing is generated. The initial image representing the distribution of the cells forming the lumen is generated in the same manner as the initial image representing the other distribution.

Also, the second preprocessing unit 115 generates an image representing the distribution of the cells forming the lumen from the initial image representing the distribution of the cells forming the lumen, similarly to the image representing the other distribution. An image representing the distribution of cells forming a lumen in skin tissue known as malignant melanoma is, for example, an image as shown in FIG. Moreover, the image showing the distribution of the cells forming the lumen in the skin tissue known as pigmented nevus is an image as shown in FIG. 12, for example. In these figures, the white cells indicate that the cells of interest exist. In FIG. 11, there are cells that form many lumens adjacent to a portion where the distribution of atypical cells is large.

Furthermore, the second CNN 119 is further learned using an image representing the distribution of cells forming the lumen in addition to an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, and an image representing the distribution of atypical cells. Keep it.

In the case of performing the discrimination process by the second CNN 119, in addition to an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of atypical cells, an image representing the distribution of cells forming the lumen To identify whether it is malignant melanoma, pigmented nevus, or others.

This improves the discrimination accuracy.

[Modification D of Embodiment 1]
Modifications A to C can be arbitrarily combined. That is, at least one of a reduced gray scale image of a pathological image, an image representing the distribution of immune cells, and an image representing the distribution of cells forming a lumen may be used.

Since there are various types of immune cells such as neutrophils, lymphocytes, and macrophages, an image representing any type of distribution may be used.

[Embodiment 2]
In addition to the differentiation of malignant melanoma as in the first embodiment, it is modified so as to perform prognosis prediction (life expectancy), stage classification prediction indicating the degree of progression, prediction of drug effect on a specific drug, etc. Also good.

As for the prognosis prediction (life expectancy), a real number (for example, 5 years) may be predicted, or less than 1 year, 1 year to 3 years, 3 years to 5 years, 5 years or more. You may also predict the survival probability over time as represented by the remaining life range or survival rate curve.

Staging prediction predicts how bad malignant melanoma is in a predetermined number of stages and predicts that stage.

Prediction of drug effect predicts the effect when a specific drug (for example, expensive nivolumab) is administered to malignant melanoma, and predicts whether or not there is an effect.

For the prediction regarding such a disease state, the distribution of epidermal cells, the distribution of dermal cells, and the distribution of atypical cells, which are the main information in the first embodiment, are effective. This is the distribution of atypical cells relative to the distribution of dermal cells and epidermal cells, that is, in what part and in what distribution the atypical cells are present, and to what extent the pathological tissue is malignant. This is because if the malignancy is high, the prognosis is predicted to be poor.

In addition, when the progression of malignant melanoma increases, there is a phenomenon that immune cells increase, and there is a risk of metastasis when atypical cells are present in the cells that make up the lumen, The distribution of these immune cells and the cells that make up the lumen is also related to prognosis.

In addition, immune cells, especially neutrophils and lymphocytes, may be involved in the medication effects of certain drugs.

Accordingly, although the basic configuration is almost the same as that of the first embodiment, in addition to the second CNN 119 that performs the discrimination process, a convolutional neural network for predicting medication effects, a convolutional neural network for predicting prognosis, and a disease Introduce a convolutional neural network for period classification prediction.

Furthermore, the first CNN 109 is also modified so that an image used in the discrimination process and the prediction process related to the medical condition can be generated. In this embodiment, dermal cells, epidermal cells, immune cells, cells that form lumens, and the like are discriminated. However, in the case of predicting the dosing effect for a specific drug, the type of immune cells (neutrophils, lymphocytes, macrophages) can be further discriminated.

FIG. 13 shows a configuration example of the information processing apparatus according to this embodiment. In addition, the same reference number is attached | subjected when it has the same function as the component of the information processing apparatus which concerns on 1st Embodiment.

The information processing apparatus according to the present embodiment includes an input unit 201, a pathological image storage unit 103, a first preprocessing unit 105, a first image storage unit 107, first CNNs 209a and 209b, and a first post processing unit. 211, the second image storage unit 113, the second preprocessing unit 115, the third image storage unit 117, the second CNNs 219a to 219d, the second post-processing unit 221, the result storage unit 123, and the first learning. A processing unit 231 and a second learning processing unit 233 are included.

The input unit 201 receives an input of a pathological image to be processed from a user and stores it in the pathological image storage unit 103. The input unit 201 also accepts instructions for processing details. That is, it accepts designation of one or a plurality of processes among differentiation of malignant melanoma, prognosis prediction (life expectancy), staging classification prediction indicating the degree of progression, and prediction of medication effect for a specific drug.

The processing content of the first preprocessing unit 105 is the same as that of the first embodiment.

The first CNN 209a determines whether each divided image stored in the first image storage unit 107 corresponds to a dermal cell, epidermal cell, immune cell, luminal cell, atypical cell, or other cell. It is a convolutional neural network learned as follows. Accordingly, the first CNN 209a outputs, for example, the respective likelihoods for the divided images related to the processing.

The first CNN 209b is a convolutional neural network learned so as to determine whether each divided image stored in the first image storage unit 107 corresponds to a neutrophil, lymphocyte, macrophage, or other cell. is there. Accordingly, the first CNN 209b outputs, for example, respective likelihoods for the divided images related to the processing.

The first CNN 209a and the first CNN 209b operate in response to an instruction from the input unit 201, for example. For example, when predicting the dosing effect for a specific drug, the first CNN 209a and the first CNN 209b are used. In other cases, only the first CNN 209a is used.

The first post-processing unit 211, based on the output from the first CNN 209a, an initial image that represents the distribution of dermal cells, an initial image that represents the distribution of epidermal cells, an initial image that represents the distribution of immune cells, and the distribution of cells that make up the lumen And an initial image representing the distribution of atypical cells are generated and stored in the second image storage unit 113. For example, when it is determined that the divided image is an epidermal cell, in the initial image representing the distribution of the epidermal cell, all the pixels in the region of the position of the divided image are “1” and “0” otherwise. Is set as follows. That is, these initial images are monochrome images. In such a case, the size of these initial images is the same as the pathological image.

Similarly, the first post-processing unit 211 generates an initial image representing a neutrophil distribution, an initial image representing a lymphocyte distribution, and an initial image representing a macrophage distribution based on the output from the first CNN 209b. The image is stored in the second image storage unit 113. The specific processing contents are the same as in the case of output from the first CNN 209a.

From the image stored in the second image storage unit 113, the second preprocessing unit 115 is configured to display an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of immune cells, and a cell forming a lumen. And an image representing the distribution of atypical cells are generated and stored in the third image storage unit 117. The processing contents are the same as in the first embodiment. When there is an output from the first CNN 209b, the second preprocessing unit 115 stores an initial image representing the neutrophil distribution stored in the second image storage unit 113, an initial image representing the macrophage distribution, lymph From the initial image representing the sphere distribution, an image representing the neutrophil distribution, an image representing the macrophage distribution, and an image representing the lymphocyte distribution are generated and stored in the third image storage unit 117.

The second CNN 219a uses an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of immune cells, an image representing the distribution of cells making up the lumen, and an image representing the distribution of atypical cells. It is a convolutional neural network that is trained to output any of tumor, pigmented nevus, and others. The second CNN 219a outputs, for example, the likelihood of malignant melanoma, the likelihood of pigmented nevus, and other likelihoods from the image stored in the third image storage unit 117. However, as in the first embodiment, discrimination may be performed without using an image representing the distribution of immune cells or an image representing the distribution of cells forming a lumen.

The second CNN 219b predicts the prognosis from an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of immune cells, an image representing the distribution of cells forming the lumen, and an image representing the distribution of atypical cells. This is a convolutional neural network that is trained to output a predicted value of (life expectancy) (real number (year), likelihood for each life expectancy range, or survival probability over time as represented by a survival rate curve). The second CNN 219b outputs a predicted value of prognosis prediction from the image stored in the third image storage unit 117.

The second CNN 219c is based on an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of immune cells, an image representing the distribution of cells that form a lumen, and an image representing the distribution of atypical cells. It is a convolutional neural network learned to output classification prediction (for example, the likelihood of each stage). The second CNN 219c outputs the staging classification prediction (for example, the likelihood of each stage) for the image stored in the third image storage unit 117.

The second CNN 219d is an image showing the distribution of dermal cells, an image showing the distribution of epidermal cells, an image showing the distribution of neutrophils, an image showing the distribution of macrophages, an image showing the distribution of lymphocytes, and the cells that make up the lumen It is a convolutional neural network learned to output a medication effect (for example, likelihood of being present and likelihood of being absent) of a specific drug from an image representing a distribution and an image representing a distribution of atypical cells. The second CNN 219d outputs the medication effect of the specific medicine from the image stored in the third image storage unit 117.

The second post-processing unit 221 stores the discrimination result in the result storage unit 123 based on the output from the second CNN 219a, and outputs the discrimination result to the display device or other output device. Furthermore, when the prediction regarding the medical condition is performed, the second post-processing unit 221 stores the prediction result regarding the medical condition in the result storage unit 123 based on the output from at least one of the second CNNs 219b to 219d. The prediction result is output to a display device or other output device.

In this way, when a pathological image is input, it is possible to automatically obtain a discrimination result as to whether it is malignant melanoma or pigmented nevus, and further a prediction result regarding a disease state.

In this embodiment, the distribution between epidermal cells, dermal cells, atypical cells, immune cells, and luminal cells are represented as separate images, thereby evaluating the relationship between the distributions. It is easy to do. This makes it possible to perform appropriate discrimination and prediction regarding a medical condition using a convolutional neural network.

In particular, by evaluating the distribution of immune cells and the cells that make up the lumen, it becomes possible to evaluate the degree of malignancy of the tumor and the possibility of metastasis in the case of malignant melanoma. Further, if the distribution of immune cells estimated to have a relationship with a specific drug is further evaluated, it is possible to predict whether or not the effect of the specific drug is effective. For example, it is possible to determine whether or not the administration of an expensive drug is appropriate.

Note that the first learning processing unit 231 executes learning processing for the first CNNs 209a and 209b with training data including a large number of sets of cell images and their types. Since the learning process method is the same as the conventional method, a detailed description is omitted.

In addition, the second learning processing unit 233 displays an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of atypical cells, an image representing the distribution of cells forming a lumen, and the distribution of immune cells. A training process is performed on the second CNN 219a with training data including a large number of sets of images to be represented and identification results. Similarly, the second learning processing unit 233 includes an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of atypical cells, an image representing the distribution of cells forming a lumen, and the distribution of immune cells. The training process is executed on the second CNN 219b or 219c with training data including a large number of sets of images representing the prognosis and prognosis prediction (life expectancy) or staging classification prediction (stage number, etc.). Further, the second learning processing unit 233 includes an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of atypical cells, an image representing the distribution of cells forming a lumen, and the distribution of neutrophils. Training data including a large number of sets of images representing a macrophage distribution, an image representing a macrophage distribution, a lymphocyte distribution, and a medication effect (such as the presence or absence of an effect) of a specific drug, and executing learning processing on the second CNN 219d . Since the learning process method is the same as the conventional method, a detailed description is omitted.

Next, processing contents of the information processing apparatus according to the present embodiment will be described with reference to FIG.

First, the input unit 201 receives an input of a pathological image to be processed from a user and stores it in the pathological image storage unit 103 (step S31). This is the same as step S1 in FIG.

In addition, the input unit 201 receives an input of a prediction process instruction regarding a medical condition to be executed in addition to the discrimination process from the user (step S33). For example, the user gives an instruction by giving at least one of prognosis prediction, staging classification prediction, and medication effect of a specific drug. However, the prediction regarding the medical condition may not be performed, and when the discrimination result is not malignant melanoma, the prediction regarding the medical condition is not performed.

Then, the first preprocessing unit 105 performs the first preprocessing on the pathological image to be processed stored in the pathological image storage unit 103, and stores the processing result in the first image storage unit 107 (step). S35). This first preprocessing is the same as step S3 in FIG.

Thereafter, the first CNN 209a reads out each divided image stored in the first image storage unit 107, and executes a classification process on them (step S37). This classification process is a process for determining whether the divided image to be processed corresponds to a dermal cell, epidermal cell, immune cell, luminal cell, atypical cell, or other cell. Output likelihood.

In addition, when the prediction of the medication effect of a specific medicine is instructed, the input unit 201 instructs the first CNN 209b to execute the process. Then, the first CNN 209b also executes the classification process on each divided image stored in the first image storage unit 107. This classification process is a process for determining whether the divided image to be processed corresponds to a neutrophil, a macrophage, a lymphocyte, or another cell. For example, the likelihood for each is output.

The first post-processing unit 211 executes the first post-processing based on the output from the first CNN 209a and / or the first CNN 209b, and stores the processing result in the second image storage unit 113 (step S39).

The first post-process according to the present embodiment is the same as the first post-process according to the first embodiment. That is, in the initial image representing the distribution of dermal cells, the pixel values of all the pixels in the region at the position of the divided image determined as dermal cells are set to “1”. For the other pixels, the pixel value is set to “0”. Similarly, in the initial image representing the distribution of epidermal cells, the pixel values of all the pixels in the region at the position of the divided image determined to be epidermal cells are set to “1”. For the other pixels, the pixel value is set to “0”. Further, in the initial image representing the distribution of atypical cells, the pixel values of all the pixels in the region at the position of the divided image determined as the atypical cell are set to “1”. For the other pixels, the pixel value is set to “0”.

Further, in the initial image representing the distribution of immune cells, the pixel values of all the pixels in the region at the position of the divided image determined as immune cells are set to “1”. For the other pixels, the pixel value is set to “0”. In the initial image representing the distribution of the cells forming the lumen, the pixel values of all the pixels in the region at the position of the divided image determined as the cell forming the lumen are set to “1”. For the other pixels, the pixel value is set to “0”.

In addition, when there are divided images determined to be neutrophils, divided images determined to be macrophages, and divided images determined to be lymphocytes, the above-described processing is performed for each, An initial image representing the distribution of the macrophages, an initial image representing the distribution of macrophages, and an initial image representing the distribution of lymphocytes are also generated.

Next, the second preprocessing unit 115 performs second preprocessing on the image stored in the second image storage unit 113, and stores the processing result in the third image storage unit 117 (step S41). ). The second preprocessing is the same as in the first embodiment.

Then, the second CNN 219a reads out the images stored in the third image storage unit 117, and executes a discrimination process on them (step S43). The discrimination process according to the present embodiment is the same as step S11 in FIG. 6 except that an image representing the distribution of immune cells and an image representing the distribution of cells forming a lumen are further used. That is, in the present embodiment, an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of immune cells, an image representing the distribution of cells forming a lumen, and an image representing the distribution of atypical cells Are processed as a gray scale image of each channel of the color image. The discrimination process is here whether or not it is malignant melanoma. For example, the likelihood of malignant melanoma, the likelihood of pigmented nevus, and other likelihoods are output. As mentioned above, it is also possible to differentiate malignant melanoma from others.

The second post-processing unit 221 specifies the discrimination result based on the output from the second CNN 219a, and determines whether or not to predict a medical condition according to an instruction from the input unit 201 (step S45). If the user instructs to perform only the discrimination process, and the discrimination result is not malignant melanoma, the process proceeds to step S49.

When performing a prediction regarding a medical condition, at least one of the second CNNs 219b to 219d according to the instruction performs a prediction process according to the instruction on the image stored in the third image storage unit 117 (step S47).

Specifically, the second CNN 219b represents an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of immune cells, an image representing the distribution of cells forming the lumen, and the distribution of atypical cells. From the image, a predicted value of prognosis prediction (life expectancy) (real number (year), likelihood for each life expectancy range, or survival probability over time as represented by a survival rate curve) is output.

The second CNN 219c is based on an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of immune cells, an image representing the distribution of cells that form a lumen, and an image representing the distribution of atypical cells. The classification prediction (for example, the likelihood of each stage) is output.

The second CNN 219d is an image showing the distribution of dermal cells, an image showing the distribution of epidermal cells, an image showing the distribution of neutrophils, an image showing the distribution of macrophages, an image showing the distribution of lymphocytes, and the cells that make up the lumen From the image representing the distribution and the image representing the distribution of the atypical cells, the medication effect (for example, the likelihood of being present and the likelihood of being absent) of a specific drug is output.

When at least one of the second CNNs 219b to 219d is used, the second post-processing unit 221 specifies a prediction result related to the medical condition based on the output from the second CNN 219b to 219d, and stores the prediction result together with the discrimination result in the result storage unit 123 for display. The data is output to the other output device (step S49). The output device may be another device connected to the network.

As described above, in addition to the discrimination result that the disease is any one of malignant melanoma, pigmented nevus, and the like, it is possible to execute a predetermined prediction regarding the disease state.

Note that at least one of the second CNNs 219b to 219d may be executed in parallel with the process of the second CNN 219a.

Further, at least one of the second CNNs 219b to 219d may be executed without performing the process of the second CNN 219a. For example, when it is known by other means that it is malignant melanoma, only the prediction process related to the disease state may be performed.

In this way, in addition to simplifying the examination and reducing the burden on the patient, it becomes easier to explain the medical condition to the patient. It also makes it easier to set guidelines for treatment.

In addition, although the example which uses two CNN in the classification | category of a division | segmentation cell was shown, for example, it classify | categorizes into any one of a dermal cell, an epidermal cell, a cell which makes a lumen, an atypical cell, and an immune cell for every kind In the post-processing, when the distribution of the immune cells for each type becomes unnecessary, the discrimination results of the immune cells for each type may be integrated.

Furthermore, in the above, the cells that make up the lumen are discriminated, and the image showing the distribution of the cells that make up the lumen is used. For example, the prediction of the medication effect of a specific drug and the staging classification prediction may not be used. Further, the prediction regarding the medical condition may be implemented so that at least one of them can be performed at the same time, instead of performing the three types simultaneously.

[Embodiment 3]
The differentiation process described above focused on malignant melanoma. However, if an image representing the distribution as described above is used, it is possible to differentiate skin diseases related to other tumors.

For example, a convolutional neural network is trained so that a malignant melanoma can be differentiated from a disease related to a predetermined number of tumors (predetermined number is a relatively small number such as 10) that is frequently subjected to pathological examination in dermatology. You may do it.

Specifically, the convolutional neural network that performs the discrimination process is different from the image representing the distribution of dermal cells, the image representing the distribution of epidermal cells, the image representing the distribution of immune cells, and the image representing the distribution of cells forming the lumen. An image representing a cell distribution is used as an input, and learning is performed so as to output which of a plurality of predetermined diseases to be differentiated and others.

Then, instead of the second CNN 119 in the first embodiment, using a convolutional neural network in which such learning is performed, an output of a corresponding disease or others can be obtained.

This will make the examination easier and reduce the burden on the patient.

[Embodiment 4]
It may be modified so that various skin diseases related to tumors other than malignant melanoma are differentiated.

Specifically, it can be modified to cope with tumor diseases as listed in FIGS. 15A to 15C.

In FIG. 15A to FIG. 15C, the disease name column lists the disease names to be identified, the atypical cell origin column indicates which cell is the atypical cell, and epidermal cells. The column from to the immune cell indicates which cell distribution is necessary to distinguish the disease (O or X).

For example, when discriminating all the diseases listed in FIGS. 15A to 15C, the convolutional neural network that classifies the divided images includes cells that form epithelial cells, dermal cells, lumens, and immune cells. Atypical cells for pigment cells, atypical cells for spiny cells, atypical cells for apocrine gland cells, atypical cells for Merkel cells, atypical cells for basal cells, atypical cells for hair follicle cells, etc. The convolutional neural network learned to distinguish and output the atypical cells for each classification of the cells from which the atypical cells are derived, listed in the column of atypical cells. Thus, the convolutional neural network that performs classification outputs the cell type or others (for example, their likelihood).

Thus, an image showing the distribution of atypical cells for pigment cells, an image showing the distribution of atypical cells for spiny cells, an image showing the distribution of atypical cells for apocrine gland cells, and an image showing the distribution of atypical cells for Merkel cells An image representing the distribution of atypical cells is generated for each cell that is derived, such as an image representing the distribution of atypical cells for basal cells and an image representing the distribution of atypical cells for hair follicle cells.

Then, the convolutional neural network that performs the differentiation process includes an image representing the distribution of epidermal cells, an image representing the distribution of dermal cells, an image representing the distribution of cells that form a lumen, and an image representing the distribution of immune cells, This is a convolutional neural network that has been trained to input an image representing the distribution of atypical cells for each cell to be derived and output any disease name or others. Therefore, the convolutional neural network that performs the discrimination process outputs one of the disease names or others (for example, their likelihood) to the image representing the distribution.

Note that some of the diseases listed in FIGS. 15A to 15C may be distinguished. For example, atypical cells for pigment cells, atypical cells for apocrine gland cells, atypical cells for sebaceous cells, atypical cells for eccrine gland cells, epidermal cells, dermal cells, luminal cells, immune cells By dividing the image into cells, malignant melanoma, pigmented nevus, dermal pigment cell nevus, Paget's disease of breast, apocrine adenocarcinoma, extramammary Paget's disease, eccrine nevus, Eccrine sweat cysts, syringoma, eccrine sweat pores, skin mixed tumors (ecline type), eccrine spiral tumors, apocrine nevus, apocrine sweat cysts, papillary sweat adenoma, papillary sweat ducts It can be configured to differentiate cystadenoma, tubular apocrine adenoma, papillary adenoma, skin mixed tumor (apocrine type), sebaceous carcinoma, sebaceous adenoma, sebaceous hyperplasia, and sebaceous adenoma good.

In other words, for a disease to be differentiated, it is possible to classify atypical cells from cells derived from atypical cells, and to learn a convolutional neural network so that differentiation can be performed using an image representing the distribution of those atypical cells. Good.

Note that when distinguishing only benign lymphoma and malignant lymphoma, it is not necessary to classify the cells forming the lumen.

In this way, it will be possible to automatically identify a disease associated with a tumor.

In addition, although discrimination was performed here, you may further deform | transform so that the prediction regarding a medical condition may be performed like 2nd Embodiment.

[Embodiment 5]
In the embodiment described above, it has been described that a skin disease related to a tumor is differentiated, but the present invention can also be applied to a non-neoplastic disease.

For example, for the diseases listed in FIG. 16, the divided images are classified into epidermal cells, dermal cells, luminal cells, and immune cells, and an image representing their distribution is generated. Then, using the images representing these distributions as input, the disease names are identified using a convolutional neural network learned to output any of the disease names listed in FIG. 16 or others.

In this way, non-neoplastic diseases can be automatically identified.

In addition, in this embodiment, in addition to the discrimination, it may be further modified so as to make a prediction regarding a medical condition as in the second embodiment.

Although the embodiment of the present invention has been described above, the present invention is not limited to this. For example, depending on the purpose, any technical feature in the above-described embodiment may be deleted, or any technical feature may be deleted after combining the embodiments.

The functional block configuration of the information processing apparatus described above is an example, and may not match the program module configuration. As for the processing flow, as long as the processing result does not change, the processing order may be changed or a plurality of steps may be executed in parallel. Further, when the output of CNN is likelihood, the likelihood may be output together.

In FIG. 5, the first CNN 109 and the second CNN 119 have been described on the assumption of the CNN, but the CNN is an example of a learned model in which supervised machine learning is performed. That is, the first CNN 109 may be such a learned model 1000. Similarly, the second CNN 119 may be such a learned model 2000. For supervised machine learning, a support vector machine (Support Vector Vector), a perceptron or other neural network can be employed. In this case, the first learning processing unit 131 and the second learning processing unit 133 are employed learning processing units for machine learning.

The first CNNs 209a and 209b in FIG. 13 are also examples of learned models in which supervised machine learning is performed, as in FIG. That is, the first CNN 209a and 209b may be such a learned model or a set 3000 of learned models. The second CNNs 219a to 219d may also be such a learned model or a set 4000 of learned models. Also in this case, as the supervised machine learning, a support vector machine, a perceptron or other neural network can be employed. In this case, the first learning processing unit 231 and the second learning processing unit 233 serve as a learning processing unit for machine learning that is employed.

The information processing apparatus described above is a computer apparatus, and includes a memory, a CPU (Central Processing Unit), a hard disk drive (HDD: Hard Disk Drive), a display control unit connected to the display device, and a removable disk. And a communication control unit for connecting to a network are connected by a bus. An operating system (OS: Operating System) and an application program for performing the processing in this embodiment are stored in the HDD, and are read from the HDD to the memory when executed by the CPU. The CPU controls the display control unit, the communication control unit, and the drive device in accordance with the processing content of the application program, and performs a predetermined operation. Further, data in the middle of processing is mainly stored in the memory, but may be stored in the HDD. In the embodiment of the present invention, an application program for performing the above-described processing is stored and distributed on a computer-readable removable disk, and installed from the drive device to the HDD. In some cases, the HDD is installed via a network such as the Internet and a communication control unit. Such a computer apparatus realizes various functions as described above by organically cooperating hardware such as the CPU and memory described above with programs such as the OS and application programs.

Note that the convolutional neural network may be implemented by a program, or a high-speed process may be performed by incorporating a dedicated arithmetic device (for example, Graphics Processing Unit) into the computer.

The above-described embodiment can be summarized as follows.

The disease discrimination processing method according to the first aspect includes (A) an image representing the distribution of dermal cells in the skin tissue, an image representing the distribution of epidermal cells in the skin tissue, and an image representing the distribution of atypical cells in the skin tissue. A first image representing the distribution of dermis cells in the skin tissue to be processed, and the epidermis in the skin tissue to be processed by a learned convolutional neural network that receives the identification of a specific skin disease related to the tumor as an input The first image, the second image, and the third image read from the storage device that stores the second image representing the distribution of cells and the third image representing the distribution of atypical cells in the skin tissue to be processed. And (B) based on the output from the learned convolutional neural network, the discrimination result of the specific skin disease is obtained. And a step of force.

In this way, the relationship between the distribution of atypical cells and the presence or absence of a specific skin disease with respect to the distribution of dermal cells and the distribution of epidermal cells is learned in advance, and an image representing the distribution of dermal cells and the distribution of epidermal cells If an image representing the distribution and an image representing the distribution of atypical cells are input and evaluated as separate images, a specific skin disease related to the tumor (for example, malignant melanoma) can be differentiated.

The learned convolutional neural network further inputs at least one of a grayscale image of the skin tissue, an image representing the distribution of immune cells in the skin tissue, and an image representing the distribution of cells forming the lumen in the skin tissue. And the storage device described above is a fourth image that is a grayscale image of the skin tissue to be processed and a fifth image that represents the distribution of immune cells in the skin tissue to be processed. There may be a case where at least one of the sixth image representing the distribution of the cells forming the lumen in the skin tissue of the subject is stored. In the execution step described above, at least one of the fourth image, the fifth image, and the sixth image is further read from the storage device, and a predetermined calculation is executed by the learned convolutional neural network. You may do it. In this way, the accuracy of discrimination is improved.

Further, the disease discrimination processing method according to the first aspect generates (C) at least one of the first to third images and the fourth to sixth images from the image of the skin tissue to be processed. You may make it further contain the step stored in a memory | storage device. For such processing, another convolutional neural network may be used, or another cell type discrimination technique may be used.

The disease state prediction processing method according to the second aspect includes (A) an image representing the distribution of dermal cells in the skin tissue, an image representing the distribution of epidermal cells in the skin tissue, and an image representing the distribution of atypical cells in the skin tissue. The distribution of the dermal cells in the skin tissue to be processed by the trained convolutional neural network that takes as input the image representing the distribution of immune cells in the skin tissue and outputs a prediction regarding the pathology of the specific skin disease related to the tumor. A first image representing, a second image representing a distribution of epidermal cells in the skin tissue to be processed, a third image representing a distribution of atypical cells in the skin tissue to be treated, and immunity in the skin tissue to be treated A predetermined image for the first image, the second image, the third image, and the fourth image read from the storage device that stores the fourth image representing the distribution of cells; An executing step of executing a calculation based on the output from the (B) trained convolutional neural network, and outputting the predicted results for pathology of the specific skin diseases.

In this way, the relationship between the distribution of atypical cells and the distribution of immune cells with respect to the distribution of dermal cells and the distribution of epidermal cells and the correspondence between the determination results regarding the pathology of a specific skin disease related to a tumor are learned in advance, and the dermis If an image representing the distribution of cells, an image representing the distribution of epidermal cells, an image representing the distribution of atypical cells, and an image representing the distribution of immune cells are input and evaluated as separate images, a specific skin disease related to a tumor It becomes possible to make a prediction regarding the pathology of (for example, malignant melanoma).

In some cases, a plurality of images representing the distribution of immune cells in the skin tissue and a fourth image are used for each type of immune cells. In addition, the prediction regarding the pathology of a specific skin disease may be any one of prediction of life expectancy, prediction of staging, and prediction of administration effect of a specific drug.

In addition, the learned convolutional neural network described above is learned by further using an image representing the distribution of the cells forming the lumen in the skin tissue as an input, and the storage device described above is used for the skin to be processed. A fifth image representing the distribution of cells that make up the lumen in the tissue may be stored. In that case, in the execution step described above, the fifth image may be further read from the storage device, and a predetermined calculation may be executed by the learned convolutional neural network. This improves the accuracy of prediction.

The disease discrimination processing method according to the third aspect includes (A) an image representing the distribution of dermal cells in the skin tissue, an image representing the distribution of epidermal cells in the skin tissue, and an image representing the distribution of atypical cells in the skin tissue. A learned convolutional neural network that receives an image representing the distribution of immune cells in the skin tissue and an image representing the distribution of cells that form lumens in the skin tissue, and outputs the differentiation of multiple types of skin diseases related to the tumor. The network displays a first image representing the distribution of dermal cells in the skin tissue to be processed, a second image representing the distribution of epidermal cells in the skin tissue to be processed, and the distribution of atypical cells in the skin tissue to be processed. A third image representing, a fourth image representing a distribution of immune cells in the skin tissue to be processed, and a distribution of cells forming a lumen in the skin tissue to be processed. A step of performing a predetermined calculation on the first to fifth images read from the storage device storing the fifth image, and (B) a plurality of types based on the output from the learned convolutional neural network. Outputting a discrimination result of the skin disease.

It will be possible to distinguish multiple types of skin diseases related to tumors.

Note that an image representing the distribution of atypical cells in the skin tissue and a third image may be prepared for each cell from which the atypical cells are derived. As a result, the accuracy can be improved.

The disease discrimination processing method according to the fourth aspect includes (A) an image representing the distribution of dermal cells in the skin tissue, an image representing the distribution of epidermal cells in the skin tissue, and an image representing the distribution of immune cells in the skin tissue. Dermal cells in the skin tissue to be processed by a trained convolutional neural network that receives an image representing the distribution of cells that form lumens in the skin tissue and outputs a discrimination of a plurality of non-neoplastic skin diseases A first image representing the distribution of the skin, a second image representing the distribution of epidermal cells in the skin tissue to be processed, a third image representing the distribution of immune cells in the skin tissue to be processed, and the skin to be processed Performing a predetermined operation on the first to fourth images read from a storage device storing a fourth image representing a distribution of cells that form a lumen in the tissue; B) based on the output from the learned convolutional neural network, and outputting the discrimination results of nonneoplastic plurality of types of skin disorders.

The diagnosis support system according to the fifth aspect includes (A) an image representing the distribution of dermal cells in the skin tissue, an image representing the distribution of epidermal cells in the skin tissue, and the distribution of atypical cells in the skin tissue. A learned model of machine learning using as input an image representing the image, (B) a first image representing a distribution of dermal cells in the skin tissue to be processed, and a first image representing a distribution of epidermal cells in the skin tissue to be processed An output data generation unit that generates output data related to a skin disease based on an output from a learned model of machine learning for the image of 2 and a third image representing the distribution of atypical cells in the skin tissue to be processed Have.

In this way, by preparing a machine learning learned model that uses a plurality of images each representing the distribution of specific cells in the skin tissue included in the pathological image as input, various output data related to skin diseases can be generated, Support for diagnosis with high accuracy.

Note that the output of the machine learning learned model described above is a discrimination of a specific skin disease related to a tumor, and the output data related to the skin disease described above may include a discrimination result of a specific skin disease. is there. It is useful for identifying specific skin diseases related to tumors.

In addition, the machine learning learned model described above can be input from a grayscale image of skin tissue, an image representing the distribution of immune cells in the skin tissue, and an image representing the distribution of cells forming the lumen in the skin tissue. It may further include at least one of them. In this case, the output data generation unit described above includes a fourth image that is a grayscale image of the skin tissue to be processed, a fifth image that represents the distribution of immune cells in the skin tissue to be processed, and the skin to be processed. Based on the output from the learned model of machine learning for at least one of the sixth image representing the distribution of the cells forming the lumen in the tissue and the first to third images, the discrimination result of the specific skin disease is obtained. You may make it produce | generate the output data containing. In this way, the accuracy of discrimination is improved.

Furthermore, the diagnosis support system according to the fifth aspect may further include (C) an image generation unit that generates first to third images from the image of the skin tissue to be processed. Note that the image generation unit may generate at least one of the fourth to sixth images.

Also, the input of the machine learning learned model described above may further include an image representing the distribution of immune cells in the skin tissue. Further, the output of the machine learning learned model described above may be a prediction related to the pathology of a specific skin disease related to a tumor. In such a case, the output data generation unit described above outputs the learned model of machine learning for the first to third images and the fourth image representing the distribution of immune cells in the skin tissue to be processed. On the basis of the above, output data including a prediction result related to a medical condition of a specific skin disease may be generated. For example, the same effect as the disease state prediction processing method according to the second aspect can be obtained.

In some cases, a plurality of images representing the distribution of immune cells in the skin tissue and a fourth image are used for each type of immune cells. In addition, the prediction regarding the pathology of a specific skin disease may be any one of prediction of life expectancy, prediction of staging, and prediction of administration effect of a specific drug.

Furthermore, the input of the learned model of machine learning described above may further include an image representing the distribution of cells that form a lumen in the skin tissue. In this case, the output data generation unit described above is the machine learning learned model for the first to fourth images and the fifth image representing the distribution of cells forming the lumen in the skin tissue to be processed. Based on the output, output data including a prediction result related to a medical condition of a specific skin disease may be generated. This improves the accuracy of prediction.

The input of the machine learning learned model described above may further include an image representing the distribution of immune cells in the skin tissue and an image representing the distribution of cells forming the lumen in the skin tissue. In addition, the output of the learned model of machine learning may be a discrimination between a plurality of types of skin diseases related to a tumor. In such a case, the output data generation unit described above includes the first to third images, the fourth image representing the distribution of immune cells in the skin tissue to be processed, and the lumen in the skin tissue to be processed. Based on the output of the learned model of machine learning with respect to the fifth image representing the distribution of the cells that make up the output, output data including the discrimination results of a plurality of types of skin diseases may be generated.

Note that an image representing the distribution of atypical cells and a third image may be prepared for each cell from which the atypical cells are derived. As a result, the accuracy can be improved.

The diagnosis support system according to the sixth aspect includes (A) an image representing the distribution of dermal cells in the skin tissue, an image representing the distribution of epidermal cells in the skin tissue, and the distribution of immune cells in the skin tissue. A learned model of machine learning that receives an image representing the distribution of the cells forming the lumen in the skin tissue and an output representing the differentiation of a plurality of non-neoplastic skin diseases, and (B) processing A first image representing the distribution of dermal cells in the subject skin tissue, a second image representing the distribution of epidermal cells in the skin tissue to be treated, and a third image representing the distribution of immune cells in the skin tissue to be treated. Based on the output of the learned model of machine learning for the image and the fourth image representing the distribution of the cells that make up the lumen in the skin tissue to be processed, it is possible to differentiate multiple types of non-neoplastic skin diseases. Generating output data including an output data generating unit.

The learned model of machine learning in the diagnosis support system according to the fifth and sixth aspects may be a neural network (particularly a learned convolutional neural network) or a support vector machine.

In the present application, the term “system” includes one or a plurality of information processing apparatuses. That is, a case where a plurality of information processing apparatuses connected via a network operate as one system in cooperation with each other and a case where the information processing apparatuses operate with one information processing apparatus are included.

In addition, a program for executing the above processing can be created, and the program can be read by a computer such as a flexible disk, an optical disk (CD-ROM, DVD-ROM, etc.), a magneto-optical disk, a semiconductor memory, a hard disk, etc. Stored in a storage medium or storage device. The intermediate processing result is temporarily stored in a storage device such as a main memory.

Claims (17)

1. Learning of machine learning using as input an image representing a distribution of dermal cells in a skin tissue included in a pathological image, an image representing a distribution of epidermal cells in the skin tissue, and an image representing a distribution of atypical cells in the skin tissue Finished model,
   A first image representing the distribution of dermal cells in the skin tissue to be processed, a second image representing the distribution of epidermal cells in the skin tissue to be processed, and the distribution of atypical cells in the skin tissue to be processed An output data generation unit that generates output data related to a skin disease based on the output from the learned model of the machine learning for the third image;
   A diagnostic support system.
2. The output of the learned model of machine learning is a differentiation of a specific skin disease related to the tumor,
   The output data relating to the skin disease includes a discrimination result of the specific skin disease,
   The diagnosis support system according to claim 1.
3. The machine learning learned model input is at least one of a grayscale image of the skin tissue, an image representing a distribution of immune cells in the skin tissue, and an image representing a distribution of cells forming a lumen in the skin tissue. Further including
   The output data generation unit
   A fourth image that is a grayscale image of the skin tissue to be processed, a fifth image that represents a distribution of immune cells in the skin tissue to be processed, and a distribution of cells that form a lumen in the skin tissue to be processed. Generating output data including a discrimination result of the specific skin disease based on an output from the learned model of the machine learning for at least one of the sixth image to be represented and the first to third images. Item 3. The diagnosis support system according to Item 2.
4. The diagnosis support system according to claim 1, further comprising an image generation unit configured to generate the first to third images from the image of the skin tissue to be processed.
5. The machine learning learned model input further includes an image representing a distribution of immune cells in the skin tissue;
   The output of the learned model of machine learning is a prediction regarding the pathology of a particular skin disease related to the tumor,
   The output data generation unit
   Prediction regarding the pathology of the specific skin disease based on the output of the learned model of the machine learning for the first to third images and the fourth image representing the distribution of immune cells in the skin tissue to be processed The diagnosis support system according to claim 1, wherein output data including a result is generated.
6. The diagnosis support system according to claim 5, wherein a plurality of images representing the distribution of immune cells in the skin tissue and the fourth image are used for each type of the immune cells.
7. The machine learning learned model input further includes an image representing a distribution of cells that form lumens in the skin tissue;
   The output data generation unit
   Based on the output of the learned model of the machine learning for the first image to the fourth image and the fifth image representing the distribution of cells forming a lumen in the skin tissue to be processed, the specific skin disease The diagnosis support system according to claim 5 or 6, wherein output data including a prediction result related to a medical condition is generated.
8. The prediction regarding the pathology of the specific skin disease is:
   The diagnosis support system according to any one of claims 5 to 7, which is any one of prediction of life expectancy, prediction of staging classification, and prediction of administration effect of a specific drug.
9. The input of the learned model of the machine learning further includes an image representing a distribution of immune cells in the skin tissue and an image representing a distribution of cells forming a lumen in the skin tissue,
   The output of the learned model of the machine learning is a differentiation of multiple types of skin diseases related to a tumor,
   The output data generation unit
   With respect to the first to third images, the fourth image representing the distribution of immune cells in the skin tissue to be processed, and the fifth image representing the distribution of cells forming a lumen in the skin tissue to be processed, The diagnosis support system according to claim 1, wherein output data including discrimination results of the plurality of types of skin diseases is generated based on an output of the learned model of the machine learning.
10. The diagnosis support system according to claim 9, wherein an image representing the distribution of atypical cells in the skin tissue and the third image are prepared for each cell from which the atypical cells are derived.
11. An image representing the distribution of dermal cells in the skin tissue included in the pathological image, an image representing the distribution of epidermal cells in the skin tissue, an image representing the distribution of immune cells in the skin tissue, and a lumen in the skin tissue A machine learning learned model that takes as input an image representing the distribution of cells to be made and outputs an identification of multiple types of non-neoplastic skin diseases,
    A first image representing the distribution of dermal cells in the skin tissue to be processed, a second image representing the distribution of epidermal cells in the skin tissue to be processed, and the distribution of immune cells in the skin tissue to be processed Based on the output of the learned model of the machine learning with respect to the third image and the fourth image representing the distribution of cells that form a lumen in the skin tissue to be processed, the plurality of types of non-neoplastic skin An output data generation unit for generating output data including disease discrimination results;
    A diagnostic support system.
12. Learning of machine learning using as input an image representing a distribution of dermal cells in a skin tissue included in a pathological image, an image representing a distribution of epidermal cells in the skin tissue, and an image representing a distribution of atypical cells in the skin tissue A first image representing the distribution of dermal cells in the skin tissue to be treated, a second image representing the distribution of epidermal cells in the skin tissue to be treated, and the atypical cells in the skin tissue to be treated. Obtaining an output for a third image representing a distribution of
    Generating output data relating to skin disease based on the output from the learned model of machine learning;
    A diagnostic support method executed by a computer.
13. The output of the learned model of machine learning is a differentiation of a specific skin disease related to the tumor,
    The output data relating to the skin disease includes a discrimination result of the specific skin disease,
    The diagnosis support method according to claim 12.
14. The machine learning learned model input further includes an image representing a distribution of immune cells in the skin tissue;
    The output of the learned model of machine learning is a prediction regarding the pathology of a particular skin disease related to the tumor,
    In the step of generating the output data,
    Prediction regarding the pathology of the specific skin disease based on the output of the learned model of the machine learning for the first to third images and the fourth image representing the distribution of immune cells in the skin tissue to be processed The diagnosis support method according to claim 12, wherein output data including a result is generated.
15. The input of the learned model of the machine learning further includes an image representing a distribution of immune cells in the skin tissue and an image representing a distribution of cells forming a lumen in the skin tissue,
    The output of the learned model of the machine learning is a differentiation of multiple types of skin diseases related to a tumor,
    In the step of generating the output data,
    With respect to the first to third images, the fourth image representing the distribution of immune cells in the skin tissue to be processed, and the fifth image representing the distribution of cells forming a lumen in the skin tissue to be processed, The diagnosis support method according to claim 12, wherein output data including a discrimination result of the plurality of types of skin diseases is generated based on an output of the learned model of the machine learning.
16. An image representing the distribution of dermal cells in the skin tissue included in the pathological image, an image representing the distribution of epidermal cells in the skin tissue, an image representing the distribution of immune cells in the skin tissue, and a lumen in the skin tissue A first learning model representing a distribution of dermal cells in a skin tissue to be processed from a machine learning learned model that receives an image representing a distribution of cells to be created and outputs a discrimination of a plurality of non-neoplastic skin diseases as an output. An image; a second image representing a distribution of epidermal cells in the skin tissue to be treated; a third image representing a distribution of immune cells in the skin tissue to be treated; and a lumen in the skin tissue to be treated. Obtaining an output for a fourth image representing a distribution of cells that make up
    Generating output data including discrimination results of the plurality of non-neoplastic skin diseases based on the output of the learned model of the machine learning;
    A diagnostic support method executed by a computer.
17. A program to be executed for causing one or more processors to execute the diagnosis support method according to claim 12 to 16.

Images