JPWO2018221625A1 - System and method for diagnosis support using pathological images of skin tissue - Google Patents

Abstract

(A) An image showing the distribution of dermal cells in the skin tissue included in the pathological image, an image showing the distribution of epidermal cells in the skin tissue, and a variant in the skin tissue. A learned model of machine learning that inputs an image representing the distribution of cells, (B) a first image representing the distribution of dermal cells in the skin tissue to be processed, and a distribution of epidermal cells in the skin tissue to be processed And an output data generation unit that generates output data related to a skin disease based on the output from the learned model with respect to the second image that represents the distribution of the atypical cells in the skin tissue to be processed. Have.

Description

  The present invention relates to a technique for supporting diagnosis using a pathological image of skin tissue.

  One paper describes the use of a two-stage convolutional neural network to detect Mitosis cells from pathological images of breast cancer pathology. Here, it is sufficient to detect only a specific type of cells, and there is no problem in diagnosing in further consideration of the distribution of specific types of cells.

  Other papers use a convolutional neural network to detect the tumor locally from pathological images of prostate cancer, and detect metastatic sites of breast cancer in sentinel lymph nodes locally Using a convolutional neural network is described. In this paper, in the analysis of pathological images of prostate cancer, it is possible to obtain the possibility of cancer from the entire pathological image by using the shape of a histogram of the likelihood of whether the tumor is a tumor, and to obtain the presence or absence of breast cancer metastasis locally. The determination is made by scoring the maximum likelihood of whether or not there is metastasis, and there is no problem in diagnosing in consideration of the distribution of specific types of cells.

  In addition, some literature considers that changes in cell nuclei and their surrounding tissues are important in discriminating the nature of tumors, and based on sub-images centering on cell nuclei, cavities, cytoplasm, stroma, etc. It is disclosed that by extracting a sub-image as a learning pattern and an input pattern, the presence / absence of a tumor and benign / malignant of the tumor are determined with high accuracy based on the sub-image. This document considers that the tissue collected in the pathological examination is stained (stained with hematoxylen, eosin, etc.), so that the cell nucleus and its surrounding tissues are stained in unique colors. By extracting sub-images centering on cell nuclei, cavities, cytoplasm, stroma, etc. from pathological images, extracting color information of cell nuclei and storing both as feature candidates, tumor tumors can be extracted with higher accuracy. It is also disclosed to determine the presence / absence and whether the tumor is benign or malignant. However, a characteristic element of the pathological image in the skin tissue is not considered.

JP 2006-153742 A

Hao Chen, QiDou, XiWang, Jing Qin & Pheng-Ann Heng, "Mitosis Detection in Breast Cancer Histology Images via Deep Cascaded Networks", Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-16) Geert Litjens, Clara I. Sanchez, NadyaTimofeeva, Meyke Hermsen, Iris Nagtegaal, Iringo Kovacs, Christina Hulsbergen-van de Kaa, Peter Bult, Bram van Ginneken and Jeroen van der Laak, "Deep learning as a tool for increased accuracy and efficiency of efficiency histopathological diagnosis ", Scientific Reports, 6: 26286, 2016

  An object of the present invention, according to one aspect, is to provide a novel technique for supporting diagnosis of a skin disease.

  The first diagnostic support system according to the present invention includes: (A) an image representing distribution of dermal cells in skin tissue included in a pathological image, an image representing distribution of epidermal cells in skin tissue, and distribution of atypical cells in skin tissue. And (B) a first image representing the distribution of dermal cells in the skin tissue to be processed and a second image representing the distribution of epidermal cells in the skin tissue to be processed. An output data generation unit that generates output data relating to a skin disease based on the output from the trained model for the second image and the third image representing the distribution of atypical cells in the skin tissue to be processed.

  The second diagnosis support system according to the present invention comprises: (A) an image representing distribution of dermal cells in skin tissue included in a pathological image, an image representing distribution of epidermal cells in skin tissue, and a distribution of immune cells in skin tissue. And an image representing distribution of cells forming a lumen in skin tissue as input, and a learned model of machine learning that outputs a discrimination of a plurality of non-neoplastic skin diseases, and (B) processing A first image representing the distribution of dermal cells in the skin tissue of the subject, a second image representing the distribution of epidermal cells in the skin tissue of the subject, and a third image representing the distribution of immune cells in the skin tissue of the subject. Based on the output of the machine-learned model for the image and the fourth image representing the distribution of cells forming the lumen in the skin tissue to be processed, the discrimination result of a plurality of non-neoplastic skin diseases is obtained. And an output data generating unit for generating a non-output data.

FIG. 1A is a diagram illustrating an example of a pathological image of skin tissue including malignant melanoma. FIG. 1B is a diagram illustrating an example of a pathological image of skin tissue including pigmented nevus. FIG. 2 is a diagram illustrating an example of a convolutional neural network. FIG. 3A is a diagram showing an example of an image showing the distribution of dermal cells in skin tissue known to be malignant melanoma. FIG. 3B is a diagram showing an example of an image representing the distribution of epidermal cells in skin tissue known to be malignant melanoma. FIG. 3C is a diagram showing an example of an image showing the distribution of atypical cells in skin tissue known to be malignant melanoma. FIG. 4A is a diagram showing an example of an image representing the distribution of dermal cells in skin tissue known to be pigmented nevus. FIG. 4B is a diagram showing an example of an image representing the distribution of epidermal cells in skin tissue known to be pigmented nevus. FIG. 4C is a diagram showing an example of an image representing the distribution of atypical cells in skin tissue known to be pigmented nevus. FIG. 5 is a diagram illustrating a functional block configuration of the information processing apparatus according to the first embodiment. FIG. 6 is a diagram illustrating a processing flow executed by the information processing apparatus according to the first embodiment. FIG. 7 is a diagram for explaining a divided image. FIG. 8A is a diagram for explaining an initial image of the distribution of dermal cells. FIG. 8B is a diagram for explaining an initial image of the distribution of epidermal cells. FIG. 8C is a diagram for explaining an initial image of the distribution of atypical cells. FIG. 9 is a diagram showing an example of an image representing the distribution of immune cells in skin tissue known to be malignant melanoma. FIG. 10 is a diagram showing an example of an image representing the distribution of immune cells in skin tissue known to be pigmented nevus. FIG. 11 is a diagram showing an example of an image representing the distribution of cells forming a lumen in skin tissue known to be malignant melanoma. FIG. 12 is a diagram showing an example of an image representing the distribution of cells forming a lumen in skin tissue known to be pigmented nevus. FIG. 13 is a diagram illustrating a functional block configuration of the information processing apparatus according to the second embodiment. FIG. 14 is a diagram illustrating a processing flow executed by the information processing apparatus according to the second embodiment. FIG. 15A is a diagram showing the relationship between the name of the disease to be differentiated, the name of the cell from which the atypical cell associated with the tumor is derived, and the type of cell used in the processing. FIG. 15B is a diagram showing the relationship between the name of the disease to be differentiated, the name of the cell from which the atypical cell associated with the tumor is derived, and the type of cell used for the processing. FIG. 15C is a diagram showing the relationship between the name of the disease to be differentiated, the name of the cell from which the atypical cell associated with the tumor is derived, and the type of cell used in the processing. FIG. 16 is a diagram showing the relationship between the name of a non-neoplastic disease and the type of cells used for processing.

[Embodiment 1]
The purpose of this embodiment is to distinguish between malignant melanoma and pigmented nevus. Malignant melanoma is a cancer of pigment cells present in the epidermis. On the other hand, pigmented nevus is a benign tumor of pigment cells, and it is often difficult to distinguish between them.

  FIG. 1A shows an example (however, a grayscale image) of a pathological image of skin tissue containing malignant melanoma. FIG. 1B shows an example of a pathological image of a skin tissue including a pigmented nevus (however, a grayscale image). Thus, it is difficult to discriminate between the two. Such a pathological image is a digital image obtained by scanning a preparation of a pathological tissue.

  In the present embodiment, as the first stage processing, in such a pathological image, the distribution of epidermal cells, the distribution of dermal cells, and the distribution of atypical cells in the skin tissue are specified. The atypical cells are either pigmented cells of malignant melanoma or pigmented cells of pigmented nevus, which is a benign tumor. Hereinafter, atypical cells refer to cells that have become tumorous or unusual.

  Next, as the second stage processing, a malignant image is obtained from an image representing the distribution of epidermal cells (an image representing the distribution is also referred to as a heat map), an image representing the distribution of dermal cells, and an image representing the distribution of atypical cells. Distinguish between melanoma and benign tumor pigmented nevus and others. However, you may make it distinguish between malignant melanoma and others.

  In the present embodiment, it is intended to estimate the degree of malignancy of a pathological tissue based on the distribution of atypical cells with respect to the distribution of epidermal cells and the distribution of dermal cells. That is, it is intended to predict the degree of malignancy of a pathological tissue based on where and what distribution of atypical cells are present in the epidermis or dermis. In other words, in a dermal cell group, an epidermal cell group, or the like that represents the structure of the skin and exists from normal times, it evaluates how the atypical cell group enters. Such prediction and evaluation are realized by causing a convolutional neural network (CNN) to perform learning.

  The convolutional neural network has, for example, a structure as shown in FIG. This has a structure similar to a well-known convolutional neural network called AlexNet, and includes a plurality of combinations of a convolutional layer and a sub-sampling layer (pooling layer) between an input layer and an output layer. In the figure, 2) a set is provided, and two total coupling layers are further added. However, there are various variations in the structure of the convolutional neural network, and these may be used. For example, the number of combinations of the convolutional layer and the sub-sampling layer may be increased.

  As described above, an image representing the distribution of epidermal cells, an image representing the distribution of dermal cells, and an image representing the distribution of atypical cells are input to the input layer. On the other hand, the output layer outputs, for example, the likelihood of malignant melanoma, the likelihood of pigmented nevus, and the likelihood in other cases.

  As is well known, the convolutional layer has parameters for the filter and the activation function, the subsampling layer also has parameters for the subsampling, and the fully connected layer has parameters for the activation function. Appropriate values and the like are set for them by learning. The algorithm for learning uses a known method.

  FIG. 3A shows an image showing the distribution of dermal cells in skin tissue known to be malignant melanoma. FIG. 3B shows an image showing the distribution of epidermal cells in skin tissue known to be malignant melanoma. FIG. 3C shows an image showing the distribution of atypical cells in skin tissue known to be malignant melanoma. When such an image is input to the input layer, learning is performed for the purpose of outputting from the output layer that the likelihood of malignant melanoma is one. In these figures, white cells indicate that the cell of interest is present.

  On the other hand, FIG. 4A shows an image showing the distribution of dermal cells in skin tissue known to be pigmented nevus. FIG. 4B shows an image showing the distribution of epidermal cells in skin tissue known to be pigmented nevus. FIG. 4C shows an image showing the distribution of atypical cells in skin tissue known to be pigmented nevus. When such an image is input to the input layer, learning is performed for the purpose of outputting an output having a likelihood of 1 as a pigmented nevus from the output layer. Note that, also in these figures, white cells indicate that the cell of interest is present.

  Note that a convolutional neural network may be used in the first stage processing. In this case, a partial image of a pathological image including a skin tissue (a partial image having a size similar to one cell or a partial image including one extracted cell) is input, and dermis cells and epidermis are obtained. A learned convolutional neural network that outputs the likelihood of cells, atypical cells, and others (glass parts, etc.) is used. However, in the first stage processing, the distribution of epidermal cells, the distribution of dermal cells, and the distribution of atypical cells may be extracted from the pathological image by using another technique such as region extraction.

  Hereinafter, the configuration of the information processing apparatus that performs such processing and the details of the processing will be described.

  FIG. 5 shows a functional configuration example of the information processing apparatus according to the present embodiment.

  The information processing apparatus according to the present embodiment includes an input unit 101, a pathological image storage unit 103, a first preprocessing unit 105, a first image storage unit 107, a first CNN 109, and a first post-processing unit 111. , A second image storage unit 113, a second pre-processing unit 115, a third image storage unit 117, a second CNN 119, a second post-processing unit 121, a result storage unit 123, and a first learning processing unit 131. , A second learning processing unit 133.

  The input unit 101 receives an input of a pathological image to be processed from a user, and stores the input in the pathological image storage unit 103.

  As described above, the pathological image is a digital image obtained by scanning a preparation of a pathological tissue. Here, the image is, for example, an image scanned by magnifying 20 times the objective, and is, for example, an RGB image of 1 M × 1 M pixels vertically and horizontally.

  The first pre-processing unit 105 divides the pathological image stored in the pathological image storage unit 103 into a size of about one cell for calculation in the first CNN 109, and divides the pathological image (hereinafter, referred to as a divided image). ) Is stored in the first image storage unit 107. For example, a square divided image of 32 × 32 pixels in length and width is generated.

  The first CNN 109 is a convolutional neural network that has been learned to determine whether each of the divided images stored in the first image storage unit 107 corresponds to a dermal cell, an epidermal cell, an atypical cell, or another cell. is there. Therefore, the first CNN 109 outputs, for example, the likelihood of each of the divided images related to the processing.

  The first post-processing unit 111 generates, based on the output from the first CNN 109, an initial image representing the distribution of dermal cells, an initial image representing the distribution of epidermal cells, and an initial image representing the distribution of atypical cells, and the second image It is stored in the storage unit 113. For example, if the divided image is determined to be a dermal cell, in the initial image representing the distribution of the dermal cells, all the pixels in the region at the position of the divided image are “1”, and otherwise “0”. It is set as follows. That is, these initial images are monochrome images. In such a case, the size of these initial images is the same as the pathological image.

  The second preprocessing unit 115 generates, from the images stored in the second image storage unit 113, an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, and an image representing the distribution of atypical cells. It is stored in the image storage unit 117. In the present embodiment, a process of reducing each initial image to an image size (for example, 200 × 200 pixels) that can be processed by the second CNN 119 is executed. For example, in this process, a process of converting each pixel to a predetermined gradation (256 gradations) is also performed. That is, an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, and an image representing the distribution of atypical cells are grayscale images.

  The second CNN 119 outputs, from the image stored in the third image storage unit 117, whether or not the cell is a malignant melanoma. For example, the likelihood of malignant melanoma, the likelihood of pigmented nevus, and other likelihoods are output. The likelihood of malignant melanoma and other likelihoods may be output.

  The second post-processing unit 121 stores the discrimination result in the result storage unit 123 based on the output from the second CNN 119, and outputs the discrimination result to a display device or another output device.

  In this way, by inputting a pathological image, it is possible to automatically obtain a result of discriminating whether or not the subject is a malignant melanoma.

  In the present embodiment, the distribution of the epidermal cells, the distribution of the dermal cells, and the distribution of the atypical cells are represented as separate images, so that the relationship between the distributions can be easily evaluated. This enables appropriate discrimination by the convolutional neural network.

  The first learning processing unit 131 executes a learning process on the first CNN 109 with training data including a large number of sets of cell images and cell types. The method of the learning process is the same as the conventional one, so a detailed description will be omitted.

  Further, the second learning processing unit 133 is a training data including a large number of sets of an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of atypical cells, and a discrimination result. Execute the learning process. The method of the learning process is the same as the conventional one, so a detailed description will be omitted.

  Next, processing contents of the information processing apparatus according to the present embodiment will be described with reference to FIGS. 6 to 8C.

  First, the input unit 101 receives an input of a pathological image to be processed from a user, and stores the pathological image in the pathological image storage unit 103 (step S1). Although the image is a grayscale image due to a problem in expression, an image (color image) as shown in FIGS. 1A and 1B is input as a pathological image. The input pathological image may be stored in a storage unit of the present information processing apparatus, a storage unit of another apparatus connected to the network, or the like. In such a case, the input unit 101 reads out from the storage unit. Stored in the pathological image storage unit 103.

  Then, the first preprocessing unit 105 executes the first preprocessing on the processing target pathological image stored in the pathological image storage unit 103, and stores the processing result in the first image storage unit 107 (step S3). In the first preprocessing, as schematically shown in FIG. 7, the pathological image 1001 is divided into a large number of divided images 1002 of N × N pixels. N is, for example, 32 or 64. Note that a divided image may be generated by recognizing a nucleus of a cell, specifying the cell, and performing a process of specifying an image region including the cell.

  After that, the first CNN 109 reads out each divided image stored in the first image storage unit 107 and executes a classification process on them (step S5). The classification process is a process of determining whether the divided image to be processed corresponds to a dermal cell, an epidermal cell, an atypical cell, or another cell, and outputs, for example, the likelihood for each of the cells.

  Then, the first post-processing unit 111 executes the first post-processing based on the output from the first CNN 109, and stores the processing result in the second image storage unit 113 (Step S7). In the first post-processing, as shown schematically in FIG. 8A, in the initial image representing the distribution of the dermal cells, the pixel values of all the pixels in the region 1002a at the position of the divided image determined to be the dermal cells are set to “1”. To "." For the other pixels, the pixel values are set to “0”. Similarly, as schematically shown in FIG. 8B, in the initial image representing the distribution of the epidermal cells, the pixel values of all the pixels in the area 1002b at the position of the divided image determined to be the epidermal cells are set to “1”. I do. For the other pixels, the pixel values are set to “0”. Further, as schematically shown in FIG. 8C, in the initial image representing the distribution of the atypical cells, the pixel values of all the pixels in the region 1002c at the position of the divided image determined to be the atypical cells are set to “1”. . For the other pixels, the pixel values are set to “0”.

  Note that such first post-processing is an example, and the size of the initial image representing the distribution may be reduced. For example, one pixel may be associated with each divided image, and in the initial image representing the distribution, the pixel value of one pixel at the position of the divided image may be set to “1”. In this way, the size of the pathological image is 1 / N both vertically and horizontally.

  Next, the second preprocessing unit 115 performs the second preprocessing on the image stored in the second image storage unit 113, and stores the processing result in the third image storage unit 117 (step S9). ). In the second preprocessing, the image is reduced. In the image reduction processing, the pixel value of the pixel after reduction is calculated by calculating the average of the pixel values of the neighboring pixels. Since the pixel value is “0” or “1” in the initial image representing the distribution, when the average is calculated, it becomes a real number of “0” to “1”. This real value is linearly mapped to be an integer from “0” to “255”, for example. As a result, the initial image (monochrome image) representing the distribution is converted to an image (grayscale image) representing the distribution. In some cases, a non-average method is used in the reduction processing. However, since the method is the same as that of the related art, a detailed description will not be given here.

  By performing such processing, an image representing the distribution of dermal cells as shown in FIGS. 3A and 4A, an image representing the distribution of epidermal cells as shown in FIGS. 3B and 4B, and FIGS. 3C and 4C. As a result, an image representing the distribution of atypical cells can be obtained.

  In the present embodiment, the reduction processing is performed on the initial image representing the distribution due to the load and the like of the convolutional neural network. However, if there is no load and the like, the reduction processing may be performed without reduction. good.

  Then, the second CNN 119 reads out the images stored in the third image storage unit 117 and executes the discrimination processing on them (step S11). In the present embodiment, a group of grayscale images as shown in FIGS. 3A to 3C or FIGS. 4A to 4C is processed as a grayscale image of each channel of a color image. The discrimination processing here is whether or not the cell is a malignant melanoma. For example, the likelihood of a malignant melanoma, the likelihood of a pigmented nevus, and other likelihoods are output. As mentioned above, it is also possible to distinguish malignant melanoma from others.

  The second post-processing unit 121 stores the discrimination result in the result storage unit 123 based on the output from the second CNN 119, and outputs the result to a display device or another output device (step S13). The output device may be another device connected to the network.

  According to the above, based on the distribution of atypical cells relative to the distribution of epidermal cells and the distribution of dermal cells in the skin tissue to be treated, the malignancy of the pathological tissue is learned, and an appropriate discrimination result can be obtained. become.

  In addition, simplification of the examination and reduction of the burden on the patient can be expected.

[Modification A of Embodiment 1]
Auxiliary information such as the contour of the skin tissue and the distribution of other parts in the skin tissue in images such as FIGS. 3A to 3C and FIGS. 4A to 4C may be useful. In such a case, the pathological image may be reduced to a grayscale image so as to match the size of the image representing the distribution.

  That is, the second preprocessing unit 115 reads the pathological image from the pathological image storage unit 103, reduces the size to a predetermined size, and stores the reduced size in the third image storage unit 117.

  The second CNN 119 is learned by further using a reduced grayscale image of a pathological image in addition to an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, and an image representing the distribution of atypical cells.

  When performing the discrimination processing by the second CNN 119, a reduced grayscale image of a pathological image is input in addition to an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, and an image representing the distribution of atypical cells. Then, it is determined whether or not the subject is a malignant melanoma.

  Thereby, the discrimination accuracy is improved.

[Modification B of Embodiment 1]
Further, when attention is paid to the phenomenon that the number of immune cells increases as the degree of progress of malignant melanoma increases, an image representing the distribution of immune cells may be further used as an input of the second CNN 119.

  That is, the first CNN 109 is configured so as to determine whether each of the divided images is a dermal cell, an epidermal cell, an immune cell, an atypical cell, or the like, and an image of the dermal cell and an image of the epidermal cell Learning is performed using an image of an immune cell and an image of an atypical cell. Then, the first CNN 109 outputs the likelihood for each of the dermal cells, epidermal cells, immune cells, atypical cells, and others.

  Based on the output from the first CNN 109, the first post-processing unit 111 generates an initial image indicating the distribution of dermal cells, an initial image indicating the distribution of epidermal cells, an initial image indicating the distribution of immune cells, and an initial image indicating the distribution of atypical cells. Generate an image. The initial image representing the distribution of immune cells is generated in the same manner as the initial images representing other distributions.

  In addition, the second preprocessing unit 115 generates an image representing the distribution of immune cells from the initial image representing the distribution of immune cells, in the same manner as the images representing other distributions. An image showing the distribution of immune cells in skin tissue known to be malignant melanoma is, for example, an image shown in FIG. An image representing the distribution of immune cells in skin tissue known to be pigmented nevus is, for example, an image as shown in FIG. Note that, also in these figures, white cells indicate that the cell of interest is present. FIG. 9 shows an overlap with the distribution of atypical cells.

  Further, the second CNN 119 is learned by further using an image representing the distribution of immune cells in addition to an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, and an image representing the distribution of atypical cells.

  In the case of performing the differentiation processing by the second CNN 119, an image representing the distribution of immune cells is input in addition to an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, and an image representing the distribution of atypical cells. Then, a distinction is made between malignant melanoma, pigmented nevus, and others.

  Thereby, the discrimination accuracy is improved.

[Modification C of First Embodiment]
Further, since it may be estimated that the pathology including the degree of progression of malignant melanoma is related to the distribution of cells forming the lumen, an image showing the distribution of cells forming the lumen is obtained from the second CNN119. It may be further used as an input.

  In this case, a modification is made such that the cells forming the lumen are used instead of the immune cells in the modification B.

  That is, the first CNN 109 is configured so as to determine whether each of the divided images is a dermal cell, an epidermal cell, a cell forming a lumen, an atypical cell, or the like. Learning is performed using the image of the cell, the image of the cell forming the lumen, and the image of the atypical cell. Then, the first CNN 109 outputs the likelihood for each of the dermal cells, the epidermal cells, the cells forming the lumen, the atypical cells, and the like.

  Based on the output from the first CNN 109, the first post-processing unit 111 generates an initial image indicating the distribution of dermal cells, an initial image indicating the distribution of epidermal cells, an initial image indicating the distribution of cells forming a lumen, and the distribution of atypical cells. Generate an initial image representing. The initial image representing the distribution of cells forming the lumen is generated in the same manner as the initial images representing other distributions.

  In addition, the second preprocessing unit 115 generates an image representing the distribution of cells forming the lumen from the initial image representing the distribution of cells forming the lumen, like the images representing the other distributions. An image representing the distribution of cells forming the lumen in skin tissue known to be malignant melanoma is, for example, an image as shown in FIG. An image showing the distribution of cells forming a lumen in skin tissue known to be pigmented nevus is, for example, an image as shown in FIG. Note that, also in these figures, white cells indicate that the cell of interest is present. In FIG. 11, there are cells forming many lumens adjacent to a portion where the distribution of atypical cells is large.

  Further, for the second CNN 119, in addition to an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, and an image representing the distribution of atypical cells, learning is further performed using an image representing the distribution of cells forming the lumen. Keep it.

  When the second CNN 119 performs the discrimination processing, in addition to the image representing the distribution of the dermal cells, the image representing the distribution of the epidermal cells, the image representing the distribution of the atypical cells, and the image representing the distribution of the cells forming the lumen. Is input to make the discrimination between malignant melanoma, pigmented nevus, and others.

  Thereby, the discrimination accuracy is improved.

[Modification D of Embodiment 1]
Modifications A to C can be arbitrarily combined. That is, at least one of a reduced grayscale image of a pathological image, an image representing the distribution of immune cells, and an image representing the distribution of cells forming a lumen may be used.

  In addition, since there are various types of immune cells such as neutrophils, lymphocytes, and macrophages, an image representing any type of distribution may be used.

[Embodiment 2]
As in the first embodiment, the present invention is modified not only to differentiate malignant melanoma, but also to perform prognosis prediction (life expectancy), stage classification prediction indicating the degree of progression, prediction of medication effect on a specific drug, and the like. Is also good.

  As for the prognosis prediction (life expectancy), a real number (for example, 5 years) may be predicted, or a predetermined value such as less than 1 year, 1 year or more, 3 years, 3 years to 5 years, 5 years or more, etc. It is also possible to predict the survival probability over time as represented by the life expectancy range and the survival rate curve.

  The staging classification prediction is to predict how bad a malignant melanoma is by expressing it in a predetermined number of stages.

  The drug effect prediction predicts the effect when a specific drug (eg, expensive nivolumab) is administered to malignant melanoma, and predicts whether or not there is an effect.

  The distribution of the epidermal cells, the distribution of the dermal cells, and the distribution of the atypical cells, which are the main information in the first embodiment, are effective in predicting such a disease state. This indicates that the distribution of atypical cells with respect to the distribution of dermal cells and the distribution of epidermal cells, that is, the distribution of atypical cells and the distribution of atypical cells indicates the extent to which pathological tissue is malignant. This is because the higher the degree of malignancy, the worse the prognosis is expected.

  In addition, focusing on the fact that when the degree of progression of malignant melanoma increases, there is a phenomenon that immune cells increase, and there is a risk of metastasis when atypical cells exist in the cells that make up the lumen. The distribution of these immune cells and the cells that make up the lumen is also relevant for prognosis.

  In addition, immune effects, particularly neutrophils and lymphocytes, may be involved in the effect of certain drugs.

  Therefore, although the basic configuration is almost the same as that of the first embodiment, in addition to the second CNN 119 for performing the discriminating process, a convolutional neural network for predicting the medication effect, a convolutional neural network for predicting the prognosis, A convolutional neural network for period classification prediction is additionally introduced.

  Further, the first CNN 109 is also modified so as to generate an image used in the discrimination processing and the prediction processing relating to the medical condition. In the present embodiment, dermal cells, epidermal cells, immune cells, cells forming a lumen, and others are determined. However, when predicting the effect of medication for a specific drug, the type of immune cell (neutrophil, lymphocyte, macrophage) can be further distinguished.

  FIG. 13 shows a configuration example of the information processing apparatus according to the present embodiment. Note that the same reference numerals are given to components having the same functions as those of the information processing apparatus according to the first embodiment.

  The information processing apparatus according to the present embodiment includes an input unit 201, a pathological image storage unit 103, a first preprocessing unit 105, a first image storage unit 107, first CNNs 209a and 209b, and a first post-processing unit. 211, a second image storage unit 113, a second pre-processing unit 115, a third image storage unit 117, second CNNs 219a to 219d, a second post-processing unit 221, a result storage unit 123, a first learning unit It has a processing unit 231 and a second learning processing unit 233.

  The input unit 201 receives an input of a pathological image to be processed from a user and stores the input in the pathological image storage unit 103. The input unit 201 also receives an instruction regarding the processing content. That is, designation of one or a plurality of processes of differentiation of malignant melanoma, prediction of prognosis (life expectancy), prediction of stage classification indicating the degree of progression, and prediction of medication effect on a specific drug are accepted.

  The processing content of the first preprocessing unit 105 is the same as that of the first embodiment.

  The first CNN 209a determines whether each of the divided images stored in the first image storage unit 107 corresponds to a dermal cell, an epidermal cell, an immune cell, a cell forming a lumen, an atypical cell, or another cell. Is a convolutional neural network learned as follows. Therefore, the first CNN 209a outputs, for example, the likelihood of each of the divided images related to the processing.

  The first CNN 209b is a convolutional neural network that has been learned to determine whether each of the divided images stored in the first image storage unit 107 corresponds to a neutrophil, a lymphocyte, a macrophage, or another cell. is there. Therefore, the first CNN 209b outputs, for example, the likelihood of each of the divided images related to the processing.

  The first CNN 209a and the first CNN 209b operate, for example, in response to an instruction from the input unit 201. For example, when predicting the medication effect for a specific drug, the first CNN 209a and the first CNN 209b are used. In other cases, only the first CNN 209a is used.

  Based on the output from the first CNN 209a, the first post-processing unit 211 generates an initial image representing a distribution of dermal cells, an initial image representing a distribution of epidermal cells, an initial image representing a distribution of immune cells, and a distribution of cells forming a lumen. And an initial image representing the distribution of atypical cells are generated and stored in the second image storage unit 113. For example, when the divided image is determined to be an epidermal cell, in the initial image representing the distribution of the epidermal cells, all the pixels in the region at the position of the divided image are “1”, and otherwise “0”. It is set as follows. That is, these initial images are monochrome images. In such a case, the size of these initial images is the same as the pathological image.

  Similarly, based on the output from the first CNN 209b, the first post-processing unit 211 generates an initial image representing the distribution of neutrophils, an initial image representing the distribution of lymphocytes, and an initial image representing the distribution of macrophages, It is stored in the second image storage unit 113. The specific processing contents are the same as in the case of the output from the first CNN 209a.

  The second pre-processing unit 115, from the images stored in the second image storage unit 113, an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of immune cells, and cells forming the lumen And an image representing the distribution of atypical cells are generated and stored in the third image storage unit 117. The processing content is the same as in the first embodiment. If there is an output from the first CNN 209b, the second pre-processing unit 115 generates an initial image representing the distribution of neutrophils stored in the second image storage unit 113, an initial image representing the distribution of macrophages, From the initial image representing the sphere distribution, an image representing the distribution of neutrophils, an image representing the distribution of macrophages, and an image representing the distribution of lymphocytes are generated and stored in the third image storage unit 117.

  The second CNN 219a is obtained from an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of immune cells, an image representing the distribution of cells forming the lumen, and an image representing the distribution of atypical cells. A convolutional neural network trained to output either a tumour, a pigmented nevus, and others. The second CNN 219a outputs, for example, the likelihood of malignant melanoma, the likelihood of pigmented nevus, and other likelihoods from the image stored in the third image storage unit 117. However, as in the first embodiment, discrimination may be performed without using an image representing the distribution of immune cells or an image representing the distribution of cells forming a lumen.

  The second CNN 219b predicts prognosis from an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of immune cells, an image representing the distribution of cells forming the lumen, and an image representing the distribution of atypical cells. This is a convolutional neural network that has been learned to output a predicted value of (life expectancy) (real number (year), likelihood for each life expectancy range, or survival probability over time as represented by a survival rate curve). The second CNN 219b outputs a predicted value of prognosis prediction from the image stored in the third image storage unit 117.

  The second CNN 219c is obtained from an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of immune cells, an image representing the distribution of cells forming the lumen, and an image representing the distribution of atypical cells. This is a convolutional neural network that has been learned to output classification prediction (eg, likelihood of each stage). The second CNN 219c outputs a stage classification prediction (for example, likelihood of each stage) for an image stored in the third image storage unit 117.

  The second CNN 219d is an image showing distribution of dermal cells, an image showing distribution of epidermal cells, an image showing distribution of neutrophils, an image showing distribution of macrophages, an image showing distribution of lymphocytes, and an image showing distribution of lymphocytes. This is a convolutional neural network that has been learned so as to output a medication effect (for example, likelihood of existence and absence likelihood) of a specific drug from an image representing distribution and an image representing distribution of atypical cells. The second CNN 219d outputs the medication effect of the specific medicine from the image stored in the third image storage unit 117.

  The second post-processing unit 221 stores the discrimination result in the result storage unit 123 based on the output from the second CNN 219a, and outputs the discrimination result to a display device or another output device. Further, in the case where the prediction regarding the medical condition is performed, the second post-processing unit 221 stores the prediction result regarding the medical condition in the result storage unit 123 based on the output from at least one of the second CNNs 219b to 219d. And outputs the prediction result to a display device or other output device.

  In this way, by inputting a pathological image, it is possible to automatically obtain a result of discriminating whether or not the subject is a malignant melanoma or a pigmented nevus, and a prediction result regarding a medical condition.

  In this embodiment, the distribution of epidermal cells, the distribution of dermal cells, the distribution of atypical cells, the distribution of immune cells, and the distribution of cells forming the lumen are represented as separate images, thereby evaluating the relationship between distributions. It is easy to do. This enables appropriate discrimination and prediction of a medical condition by using a convolutional neural network.

  In particular, by evaluating the distribution of immune cells and the distribution of cells forming the lumen, it becomes possible to evaluate the degree of malignancy of the tumor and the possibility of metastasis in the case of malignant melanoma. Further, by further evaluating the distribution of immune cells that are presumed to have a relationship with a specific drug, the presence or absence of the effect of the specific drug can be predicted. For example, it is possible to determine whether or not administration of an expensive drug is appropriate.

  The first learning processing unit 231 performs a learning process on the first CNNs 209a and 209b using training data including a large number of sets of cell images and their types. The method of the learning process is the same as the conventional one, so a detailed description will be omitted.

  Further, the second learning processing unit 233 generates an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of atypical cells, an image representing the distribution of cells forming the lumen, and the distribution of immune cells. A learning process is performed on the second CNN 219a using training data including a large number of sets of images and discrimination results. Similarly, the second learning processing unit 233 includes an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of atypical cells, an image representing the distribution of cells forming the lumen, and the distribution of immune cells. The learning process is performed on the second CNN 219b or 219c using training data including a large number of sets of images indicating prognosis (life expectancy) or stage classification prediction (stage number or the like). The second learning processing unit 233 further includes an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of atypical cells, an image representing the distribution of cells forming the lumen, and the distribution of neutrophils. The learning process is performed on the second CNN 219d with training data including a large number of images representing the distribution of macrophages, images representing the distribution of lymphocytes, and images representing the distribution of lymphocytes, and a number of medication effects (presence or absence of effects) of the specific drug. . The method of the learning process is the same as the conventional one, so a detailed description will be omitted.

  Next, the processing content of the information processing apparatus according to the present embodiment will be described using FIG.

  First, the input unit 201 receives an input of a pathological image to be processed from a user, and stores the pathological image in the pathological image storage unit 103 (step S31). This is the same as step S1 in FIG.

  Further, the input unit 201 receives an input of an instruction of a prediction process regarding a medical condition to be executed in addition to the discrimination process from the user (step S33). For example, the user gives an instruction of at least one of prognosis prediction, staging classification prediction, and medication effect of a specific drug, and receives the instruction. However, the prediction regarding the medical condition may not be performed, and the prediction regarding the medical condition is not performed when the discrimination result is not malignant melanoma.

  Then, the first preprocessing unit 105 executes the first preprocessing on the processing target pathological image stored in the pathological image storage unit 103, and stores the processing result in the first image storage unit 107 (step S35). This first preprocessing is the same as step S3 in FIG.

  After that, the first CNN 209a reads out the divided images stored in the first image storage unit 107 and executes a classification process on them (step S37). This classification process is a process of determining whether the divided image to be processed corresponds to a dermal cell, an epidermal cell, an immune cell, a cell forming a lumen, an atypical cell, or another cell. Output likelihood.

  When the prediction of the medication effect of the specific medicine is instructed, the input unit 201 instructs the first CNN 209b to execute the process. Then, the first CNN 209b also performs a classification process on each of the divided images stored in the first image storage unit 107. This classification processing is processing for determining whether the divided image to be processed corresponds to a neutrophil, a macrophage, a lymphocyte, or another cell, and outputs, for example, the likelihood of each.

  Then, the first post-processing unit 211 executes the first post-processing based on the output from the first CNN 209a and / or the first CNN 209b, and stores the processing result in the second image storage unit 113 (Step S39).

  The first post-processing according to the present embodiment is basically the same as the first post-processing according to the first embodiment. That is, in the initial image representing the distribution of the dermal cells, the pixel values of all the pixels in the region at the position of the divided image determined to be the dermal cells are set to “1”. For the other pixels, the pixel values are set to “0”. Similarly, in the initial image representing the distribution of the epidermal cells, the pixel values of all the pixels in the region at the position of the divided image determined to be the epidermal cells are set to “1”. For the other pixels, the pixel values are set to “0”. Further, in the initial image representing the distribution of the atypical cells, the pixel values of all the pixels in the region at the position of the divided image determined to be the atypical cells are set to “1”. For the other pixels, the pixel values are set to “0”.

  Further, in the initial image representing the distribution of the immune cells, the pixel values of all the pixels in the region at the position of the divided image determined to be the immune cells are set to “1”. For the other pixels, the pixel values are set to “0”. In the initial image representing the distribution of cells forming the lumen, the pixel values of all pixels in the region at the position of the divided image determined to be cells forming the lumen are set to “1”. For the other pixels, the pixel values are set to “0”.

  If there are divided images determined to be neutrophils, divided images determined to be macrophages, and divided images determined to be lymphocytes, the processing described above is performed for each of them, , An initial image indicating the distribution of macrophages, and an initial image indicating the distribution of lymphocytes.

  Next, the second preprocessing unit 115 executes the second preprocessing on the image stored in the second image storage unit 113, and stores the processing result in the third image storage unit 117 (Step S41). ). The second pre-processing is the same as in the first embodiment.

  Then, the second CNN 219a reads out the images stored in the third image storage unit 117 and executes the discrimination processing on them (step S43). The discrimination processing according to the present embodiment is the same as step S11 in FIG. 6 except that an image representing the distribution of immune cells and an image representing the distribution of cells forming the lumen are further used. That is, in the present embodiment, an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of immune cells, an image representing the distribution of cells forming the lumen, and an image representing the distribution of atypical cells Is processed as a grayscale image of each channel of the color image. The discrimination processing here is whether or not the cell is a malignant melanoma. For example, the likelihood of a malignant melanoma, the likelihood of a pigmented nevus, and other likelihoods are output. As mentioned above, it is also possible to distinguish malignant melanoma from others.

  The second post-processing unit 221 specifies the discrimination result based on the output from the second CNN 219a, and determines whether the prediction regarding the medical condition is to be performed according to the instruction from the input unit 201 (Step S45). If the user has instructed to execute only the discrimination process, or if the discrimination result is not malignant melanoma, the process proceeds to step S49.

  When performing prediction regarding a medical condition, at least one of the second CNNs 219b to 219d according to the instruction executes a prediction process according to the instruction on the image stored in the third image storage unit 117 (step S47).

  Specifically, the second CNN 219b represents an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of immune cells, an image representing the distribution of cells forming the lumen, and representing the distribution of atypical cells. From the image, a predicted value of prognosis prediction (life expectancy) (real number (year), likelihood for each life expectancy range, or survival probability over time as represented by a survival rate curve) is output.

  The second CNN 219c is obtained from an image representing the distribution of dermal cells, an image representing the distribution of epidermal cells, an image representing the distribution of immune cells, an image representing the distribution of cells forming the lumen, and an image representing the distribution of atypical cells. A classification prediction (for example, likelihood of each stage) is output.

  The second CNN 219d is an image showing distribution of dermal cells, an image showing distribution of epidermal cells, an image showing distribution of neutrophils, an image showing distribution of macrophages, an image showing distribution of lymphocytes, and an image showing distribution of lymphocytes. From the image representing the distribution and the image representing the distribution of atypical cells, the medication effect of a specific drug (for example, the likelihood of existence and the likelihood of absence) is output.

  When at least one of the second CNNs 219b to 219d is used, the second post-processing unit 221 specifies a prediction result regarding the medical condition based on an output from the second CNN 219b to 219d, stores the prediction result together with the discrimination result in the result storage unit 123, and displays the result. The output is output to the device or another output device (step S49). The output device may be another device connected to the network.

  According to the above-described method, it is possible to execute a predetermined prediction regarding a medical condition in addition to a result of discriminating that the subject is a malignant melanoma, a pigmented nevus, or the like.

  Note that at least one of the second CNNs 219b to 219d may be executed in parallel with the processing of the second CNN 219a.

  Further, at least one of the second CNNs 219b to 219d may be executed without performing the process of the second CNN 219a. For example, if it is known by other means that the disease is malignant melanoma, only the prediction process relating to the disease state may be performed.

  This makes it easier to explain the medical condition to the patient, in addition to simplifying the examination and reducing the burden on the patient. It also makes it easier to set guidelines for treatment.

  Although an example in which two CNNs are used to classify the divided cells has been described, for example, one CNN may be used to classify dermal cells, epidermal cells, cells forming lumens, atypical cells, or immune cells of each type. In the post-processing, when the distribution of immune cells for each type becomes unnecessary, the determination results of the immune cells for each type may be integrated.

  Further, in the above, the cells forming the lumen are discriminated, and the image representing the distribution of the cells forming the lumen is used. However, it may not be used for the prediction of the medical condition. For example, it may not be used for the prediction of the medication effect of a specific drug or the prediction of the stage classification. In addition, it is also possible to implement so that at least one of the three types of prediction regarding the medical condition can be performed at the same time, instead of performing the prediction at the same time.

[Embodiment 3]
The above-described discrimination processing focused on malignant melanoma. However, if an image representing the distribution as described above is used, it is possible to distinguish skin diseases related to other tumors.

  For example, a convolutional neural network is trained so that malignant melanoma can be distinguished from a disease related to a predetermined number of tumors (a relatively small number such as 10, for example) frequently subject to pathological examination in dermatology. You may do it.

  Specifically, the convolutional neural network that performs the discrimination process has an atypical image that represents the distribution of dermal cells, an image that represents the distribution of epidermal cells, an image that represents the distribution of immune cells, and an image that represents the distribution of cells that make up the lumen. An image representing the distribution of cells is input, and learning is performed so as to output any of a plurality of predetermined diseases to be distinguished and others.

  If a convolutional neural network on which such learning has been performed is used instead of the second CNN 119 in the first embodiment, it is possible to obtain an output of a corresponding disease or other.

  This will further simplify the examination and reduce the burden on the patient.

[Embodiment 4]
Modifications may be made so as to discriminate various skin diseases related to tumors other than malignant melanoma.

  Specifically, it can be modified to deal with tumor diseases as listed in FIGS. 15A to 15C.

  In FIG. 15A to FIG. 15C, the disease name column lists disease names to be differentiated, and the column derived from atypical cells indicates which cells are atypical cells, and epidermal cells. The column from to the immune cell indicates which cell distribution is necessary (○ or ×) to distinguish the disease.

  For example, when discriminating all the diseases listed in FIG. 15A to FIG. 15C, a convolutional neural network that performs classification of divided images uses epidermal cells, dermal cells, cells that form lumens, and immune cells for cell images. Plus atypical cells for pigment cells, atypical cells for spiny cells, atypical cells for apocrine gland cells, atypical cells for Merkel cells, atypical cells for basal cells, atypical cells for hair follicle cells, etc. 2 is a convolutional neural network that is trained so as to distinguish and output atypical cells for each classification of cells from which atypical cells are derived, which is listed in the column of origins of atypical cells. Therefore, the convolutional neural network that performs the classification outputs the type of cell or the like (eg, their likelihood).

  Thus, an image showing the distribution of atypical cells for pigment cells, an image showing the distribution of atypical cells for spiny cells, an image showing the distribution of atypical cells for apocrine gland cells, and an image showing the distribution of atypical cells for Merkel cells For example, an image representing the distribution of atypical cells for each origin cell, such as an image representing the distribution of atypical cells for basal cells and an image representing the distribution of atypical cells for hair follicle cells, is generated.

  Then, the convolutional neural network performing the discrimination process, in addition to the image representing the distribution of epidermal cells, the image representing the distribution of dermal cells, the image representing the distribution of cells forming the lumen, and the image representing the distribution of immune cells, This is a convolutional neural network that has been trained to output any disease name or others by using an image representing the distribution of atypical cells for each cell as an input. Therefore, the convolutional neural network performing the discrimination processing outputs any disease name or others (for example, their likelihood) to the images representing their distribution.

  In addition, you may comprise so that a part of disease listed in FIG. 15A thru | or 15C can be distinguished. For example, atypical cells for pigment cells, atypical cells for apocrine gland cells, atypical cells for sebaceous cells, atypical cells for eccrine gland cells, epidermal cells, dermal cells, cells that form lumens, By classifying the divided image into cells, malignant melanoma, pigmented nevus, dermal pigment cell nevus, Paget's disease of the breast, apocrine adenocarcinoma, extramammary Paget's disease, and eccrine nevi, Eccrine sweat cyst, syringoma, eccrine syringoma, mixed skin tumor (eccrine type), eccrine spiral tumor, apocrine nevus, apocrine sweat cyst, papillary sweat adenoma, papillary sweat duct Cystic adenoma, tubular apocrine adenoma, papillary adenoma, mixed skin tumor (apocrine type), sebaceous carcinoma, sebaceous adenoma, sebaceous hyperplasia, and sebaceous adenoma good.

  In other words, for the disease to be differentiated, it is possible to classify the atypical cells of the cells from which the atypical cells are derived, and to train the convolutional neural network so that it can be distinguished using an image representing the distribution of the atypical cells. Good.

  When discriminating only between benign lymphoma and malignant lymphoma, it is not necessary to classify cells forming the lumen.

  This makes it possible to automatically discriminate a disease associated with a tumor.

  Note that the discrimination is performed here, but it may be further modified so as to perform prediction regarding a medical condition as in the second embodiment.

[Embodiment 5]
In the above-described embodiment, discrimination about skin diseases related to tumors has been described. However, the present invention is also applicable to non-tumor diseases.

  For example, for the diseases listed in FIG. 16, the divided images are classified into epidermal cells, dermal cells, cells forming lumens, and immune cells, and an image representing their distribution is generated. Then, using the convolutional neural network learned to output any one of the disease names listed in FIG. 16 or the other with the images representing the distributions as inputs, the disease names are identified.

  This makes it possible to automatically distinguish non-neoplastic diseases.

  Note that, in the present embodiment, in addition to the discrimination, a further modification may be made so as to predict the medical condition as in the second embodiment.

  Although the embodiment of the present invention has been described above, the present invention is not limited to this. For example, depending on the purpose, any technical feature in the above-described embodiment may be deleted, or any technical feature may be deleted after combining the embodiments.

  The functional block configuration of the information processing apparatus described above is an example, and may not match the program module configuration. As for the processing flow, as long as the processing result does not change, the processing order may be changed or a plurality of steps may be executed in parallel. If the output of the CNN is likelihood, the likelihood may be output together.

  In FIG. 5, the first CNN 109 and the second CNN 119 have been described on the assumption that the CNN is used, but the CNN is an example of a learned model in which supervised machine learning has been performed. That is, the first CNN 109 may be such a learned model 1000. Similarly, the second CNN 119 may be such a learned model 2000. As the supervised machine learning, a support vector machine (Support Vector Machine), a perceptron, or another neural network can be used. In this case, the first learning processing unit 131 and the second learning processing unit 133 are learning processing units for the adopted machine learning.

  The first CNNs 209a and 209b in FIG. 13 are also examples of the learned model in which the supervised machine learning has been performed, as in FIG. That is, the first CNN 209a and 209b may be such a trained model or a set 3000 of trained models. The second CNNs 219a to 219d may also be such a learned model or a set 4000 of learned models. Also in this case, as the supervised machine learning, a support vector machine, a perceptron, or another neural network can be used. In this case, the first learning processing unit 231 and the second learning processing unit 233 are learning processing units for the adopted machine learning.

  The information processing device described above is a computer device, and includes a memory, a CPU (Central Processing Unit), a hard disk drive (HDD: Hard Disk Drive), a display control unit connected to a display device, and a removable disk. Drive device, an input device, and a communication control unit for connecting to a network are connected by a bus. An operating system (OS) and an application program for executing the processing in the present embodiment are stored in the HDD, and are read from the HDD to the memory when executed by the CPU. The CPU controls the display control unit, the communication control unit, and the drive device in accordance with the processing content of the application program to perform a predetermined operation. The data being processed is mainly stored in the memory, but may be stored in the HDD. In the embodiment of the present invention, an application program for performing the above-described processing is distributed and stored in a computer-readable removable disk, and is installed in a HDD from a drive device. It may be installed in the HDD via a network such as the Internet and a communication control unit. Such a computer device realizes the above-described various functions by the hardware such as the CPU and the memory described above and the programs such as the OS and the application program cooperate organically.

  The convolutional neural network may be implemented by a program, or a high-speed processing may be performed by incorporating a dedicated arithmetic device (for example, a Graphics Processing Unit) into a computer.

  The above-described embodiment is summarized as follows.

  The disease discrimination processing method according to the first aspect comprises: (A) an image representing distribution of dermal cells in skin tissue, an image representing distribution of epidermal cells in skin tissue, and an image representing distribution of atypical cells in skin tissue. A first image representing the distribution of dermal cells in the skin tissue to be processed by the learned convolutional neural network that receives as input and outputs the identification of a specific skin disease related to the tumor, and the epidermis in the skin tissue to be processed A first image, a second image, and a third image read from a storage device that stores a second image representing the distribution of cells and a third image representing the distribution of atypical cells in the skin tissue to be processed. An execution step of executing a predetermined operation with respect to the image of (b) and (B) outputting the discrimination result of the specific skin disease based on the output from the learned convolutional neural network. And a step of.

  Thus, the relationship between the distribution of atypical cells to the distribution of dermal cells and the distribution of epidermal cells and the correspondence between the presence or absence of a specific skin disease are learned in advance, and the image representing the distribution of dermal cells and the distribution of epidermal cells By inputting and evaluating the image representing the image and the image representing the distribution of atypical cells as separate images, it becomes possible to discriminate a specific skin disease (for example, malignant melanoma) related to a tumor.

  In addition, the learned convolutional neural network further inputs at least one of a grayscale image of skin tissue, an image showing distribution of immune cells in skin tissue, and an image showing distribution of cells forming a lumen in skin tissue. The storage device described above is used to store a fourth image, which is a grayscale image of the skin tissue to be processed, and a fifth image representing the distribution of immune cells in the skin tissue to be processed. A sixth image representing the distribution of cells forming the lumen in the skin tissue of the subject may be stored. Then, in the execution step described above, at least one of the fourth image, the fifth image, and the sixth image is further read from the storage device, and a predetermined operation is executed by the learned convolutional neural network. You may do it. This improves the accuracy of the discrimination.

  Further, the disease identification processing method according to the first aspect generates (C) first to third images and at least one of fourth to sixth images from the image of the skin tissue to be processed. The method may further include the step of storing the information in a storage device. For such processing, another convolutional neural network may be used, or another cell type discrimination technology may be used.

  The pathological condition prediction processing method according to the second aspect comprises: (A) an image representing distribution of dermal cells in skin tissue, an image representing distribution of epidermal cells in skin tissue, and an image representing distribution of atypical cells in skin tissue. An image representing the distribution of immune cells in skin tissue is input, and a learned convolutional neural network is used to output a prediction regarding the pathology of a specific skin disease relating to a tumor. A first image, a second image representing the distribution of epidermal cells in the skin tissue to be treated, a third image representing the distribution of atypical cells in the skin tissue to be treated, and immunity in the skin tissue to be treated. A predetermined operation is performed on the first image, the second image, the third image, and the fourth image read from the storage device that stores the fourth image representing the cell distribution. An execution step of executing, based on the output from the (B) trained convolutional neural network, and outputting the predicted results for pathology of the specific skin diseases.

  Thus, the relationship between the distribution of atypical cells and the distribution of immune cells with respect to the distribution of dermal cells and the distribution of epidermal cells, and the corresponding relationship between the determination results regarding the pathology of a specific skin disease relating to a tumor and the dermis If an image representing the distribution of cells, an image representing the distribution of epidermal cells, an image representing the distribution of atypical cells, and an image representing the distribution of immune cells are input as separate images and evaluated, a specific skin disease related to a tumor can be obtained. (For example, malignant melanoma).

  In some cases, a plurality of images and fourth images representing the distribution of immune cells in skin tissue are used for each type of immune cell. In addition, the prediction of the condition of a specific skin disease may be any one of prediction of life expectancy, prediction of staging, and prediction of administration effect of a specific drug.

  In addition, the learned convolutional neural network described above has been learned by further using an image representing the distribution of cells forming the lumen in the skin tissue as an input, and the storage device described above stores the skin to be processed. A fifth image representing the distribution of cells forming the lumen in the tissue may be stored. In that case, in the above-described execution step, the fifth image may be further read from the storage device, and a predetermined operation may be executed by the learned convolutional neural network. This improves the accuracy of the prediction.

  The disease discrimination processing method according to the third aspect comprises: (A) an image representing distribution of dermal cells in skin tissue, an image representing distribution of epidermal cells in skin tissue, and an image representing distribution of atypical cells in skin tissue. , A trained convolutional neural network that receives as input an image representing the distribution of immune cells in skin tissue and an image representing the distribution of cells that make up the lumen in skin tissue and outputs the identification of multiple types of skin diseases related to tumors By the network, a first image representing the distribution of dermal cells in the skin tissue to be treated, a second image representing the distribution of epidermal cells in the skin tissue to be treated, and the distribution of atypical cells in the skin tissue to be treated are represented by: A third image representing the distribution of immune cells in the skin tissue to be treated, and a fourth image representing the distribution of cells forming the lumen in the skin tissue to be treated. Performing a predetermined operation on the first to fifth images read from the storage device storing the first and fifth images, and (B) a plurality of types of skin based on outputs from the learned convolutional neural network. Outputting a disease identification result.

  It becomes possible to distinguish a plurality of types of skin diseases related to a tumor.

  Note that an image representing the distribution of atypical cells in skin tissue and a third image may be prepared for each cell from which the atypical cells are derived. As a result, accuracy can be improved.

  The disease discrimination processing method according to the fourth aspect includes (A) an image representing the distribution of dermal cells in skin tissue, an image representing distribution of epidermal cells in skin tissue, and an image representing distribution of immune cells in skin tissue. Dermis cells in the skin tissue to be processed by a learned convolutional neural network, which takes as input an image representing the distribution of cells forming the lumen in the skin tissue and outputs the discrimination of a plurality of non-neoplastic skin diseases , A second image representing the distribution of epidermal cells in the skin tissue to be treated, a third image representing the distribution of immune cells in the skin tissue to be treated, and the skin to be treated Performing a predetermined operation on the first to fourth images read from the storage device storing the fourth image representing the distribution of cells forming the lumen in the tissue; and ) Based on the output from the learned convolutional neural network, and outputting the discrimination results of nonneoplastic plurality of types of skin disorders.

  The diagnosis support system according to the fifth aspect includes (A) an image representing distribution of dermal cells in skin tissue included in a pathological image, an image representing distribution of epidermal cells in skin tissue, and distribution of atypical cells in skin tissue. And (B) a first image representing the distribution of dermal cells in the skin tissue to be processed and a second image representing the distribution of epidermal cells in the skin tissue to be processed. An output data generation unit that generates output data relating to a skin disease based on the output from the machine learning-learned model for the second image and the third image that represents the distribution of atypical cells in the skin tissue to be processed. Have.

  By preparing a machine-learned model using a plurality of images each representing the distribution of specific cells in the skin tissue included in the pathological image as input in this way, it becomes possible to generate various output data related to skin diseases, High-precision diagnosis support can be performed.

  Note that the output of the learned model of machine learning described above is a discrimination of a specific skin disease related to a tumor, and the output data related to the skin disease described above may include a discrimination result of a specific skin disease. is there. It is useful for discriminating a specific skin disease related to a tumor.

  In addition, the input of the learned model of machine learning described above is a grayscale image of skin tissue, an image showing distribution of immune cells in skin tissue, and an image showing distribution of cells forming a lumen in skin tissue. It may further include at least one of them. In this case, the output data generation unit described above includes a fourth image that is a grayscale image of the skin tissue to be processed, a fifth image representing the distribution of immune cells in the skin tissue to be processed, and the skin to be processed. Based on at least one of the sixth image representing the distribution of cells forming the lumen in the tissue and the first to third images, the discrimination result of the specific skin disease is determined based on the output from the trained model of the machine learning. Alternatively, output data including the output data may be generated. This improves the accuracy of the discrimination.

  Furthermore, the diagnosis support system according to the fifth aspect may further include (C) an image generation unit that generates first to third images from an image of the skin tissue to be processed. Note that the image generation unit may generate at least one of the fourth to sixth images.

  In addition, the input of the machine-learned model described above may further include an image representing the distribution of immune cells in the skin tissue. Also, the output of the machine-learned model described above may be a prediction about the pathology of a particular skin disease involving a tumor. In such a case, the output data generation unit described above outputs the learned model of the machine learning model for the first to third images and the fourth image representing the distribution of immune cells in the skin tissue to be processed. May be used to generate output data including a prediction result regarding a specific skin disease condition. For example, the same effects as those of the disease condition prediction processing method according to the second aspect can be obtained.

  In some cases, a plurality of images and fourth images representing the distribution of immune cells in skin tissue are used for each type of immune cell. In addition, the prediction of the condition of a specific skin disease may be any one of prediction of life expectancy, prediction of staging, and prediction of administration effect of a specific drug.

  Further, the input of the machine-learned model described above may further include an image representing a distribution of cells forming a lumen in the skin tissue. In this case, the output data generation unit described above generates a machine-learned learned model of the first to fourth images and the fifth image representing the distribution of cells forming the lumen in the skin tissue to be processed. Based on the output, output data including a prediction result related to a specific skin disease condition may be generated. This improves the accuracy of the prediction.

  The input of the learned model of machine learning described above may further include an image representing the distribution of immune cells in the skin tissue and an image representing the distribution of cells forming a lumen in the skin tissue. Also, the output of the machine-learned model may be a discrimination of a plurality of types of skin diseases related to the tumor. In such a case, the output data generation unit described above outputs the first to third images, the fourth image representing the distribution of immune cells in the skin tissue to be processed, and the lumen in the skin tissue to be processed. May be generated based on the output of the machine learning-learned model with respect to the fifth image representing the distribution of the cells that make up the image.

  Note that an image representing the distribution of atypical cells and a third image may be prepared for each cell from which the atypical cells are derived. As a result, accuracy can be improved.

  The diagnosis support system according to the sixth aspect includes (A) an image representing distribution of dermal cells in skin tissue included in a pathological image, an image representing distribution of epidermal cells in skin tissue, and a distribution of immune cells in skin tissue. And an image representing distribution of cells forming a lumen in skin tissue as input, and a learned model of machine learning that outputs a discrimination of a plurality of types of non-neoplastic skin diseases, and (B) processing A first image representing the distribution of dermal cells in the skin tissue of the subject, a second image representing the distribution of epidermal cells in the skin tissue of the subject, and a third image representing the distribution of immune cells in the skin tissue of the subject. Based on the output of the machine-learned model for the image and the fourth image representing the distribution of cells forming the lumen in the skin tissue to be processed, a result of discriminating a plurality of non-neoplastic skin diseases And an output data generating unit for generating output data including.

  The learned model of machine learning in the diagnosis support systems according to the fifth and sixth aspects may be a neural network (particularly a learned convolutional neural network) or a support vector machine.

  In the present application, the term “system” includes one or more information processing devices. In other words, it includes a case where a plurality of information processing apparatuses connected via a network operate as one system in cooperation with each other, and a case where the information processing apparatuses operate with one information processing apparatus.

  In addition, a program for executing the above processing can be created. The program can be a computer readable program such as a flexible disk, an optical disk (CD-ROM, DVD-ROM, etc.), a magneto-optical disk, a semiconductor memory, and a hard disk. Stored in a suitable storage medium or storage device. The intermediate processing result is temporarily stored in a storage device such as a main memory.

Claims (17)

Machine learning learning that receives as input the image representing the distribution of dermal cells in the skin tissue included in the pathological image, the image representing the distribution of epidermal cells in the skin tissue, and the image representing the distribution of atypical cells in the skin tissue Model and
A first image representing the distribution of dermal cells in the skin tissue to be treated, a second image representing the distribution of epidermal cells in the skin tissue to be treated, and a distribution of atypical cells in the skin tissue to be treated An output data generation unit that generates output data related to a skin disease based on an output from the machine-learned model with respect to a third image,
Diagnostic support system having The output of the learned model of machine learning is the identification of a specific skin disease related to the tumor,
The output data on the skin disease includes a result of identifying the specific skin disease,
The diagnosis support system according to claim 1. The input of the learned model of the machine learning is at least one of a grayscale image of the skin tissue, an image representing the distribution of immune cells in the skin tissue, and an image representing the distribution of cells forming a lumen in the skin tissue. Or further including
The output data generator,
A fourth image, which is a grayscale image of the skin tissue to be processed, a fifth image representing the distribution of immune cells in the skin tissue to be processed, and a distribution of cells forming a lumen in the skin tissue to be processed. Generating output data including a result of discrimination of the specific skin disease, based on an output from the machine-learned model for at least one of the sixth image to be represented and the first to third images. Item 3. The diagnosis support system according to Item 2. The diagnosis support system according to claim 1, further comprising: an image generation unit configured to generate the first to third images from an image of the skin tissue to be processed. The input of the machine learning learned model further includes an image representing a distribution of immune cells in the skin tissue,
The output of the learned model of machine learning is a prediction of the pathology of a particular skin disease associated with a tumor,
The output data generator,
A prediction of the pathology of the specific skin disease based on the output of the machine learning-learned model with respect to the first to third images and a fourth image representing a distribution of immune cells in the skin tissue to be processed; The diagnosis support system according to claim 1, wherein output data including a result is generated. The diagnosis support system according to claim 5, wherein a plurality of images representing the distribution of immune cells in the skin tissue and the fourth image are used for each type of the immune cells. The input of the machine learning learned model further includes an image representing a distribution of cells forming a lumen in the skin tissue,
The output data generator,
Based on the output of the machine-learned model for the first to fourth images and the fifth image representing the distribution of cells forming the lumen in the skin tissue to be processed, the specific skin disease The diagnosis support system according to claim 5, wherein output data including a prediction result regarding a medical condition is generated. The prediction regarding the condition of the specific skin disease includes:
The diagnosis support system according to any one of claims 5 to 7, wherein the prediction is one of prediction of life expectancy, prediction of stage classification, and prediction of administration effect of a specific drug. The input of the learned model of the machine learning further includes an image representing the distribution of immune cells in the skin tissue, and an image representing the distribution of cells forming a lumen in the skin tissue,
The output of the learned model of the machine learning is a discrimination of a plurality of skin diseases related to the tumor,
The output data generator,
With respect to the first to third images, a fourth image representing the distribution of immune cells in the skin tissue to be treated, and a fifth image representing the distribution of cells forming a lumen in the skin tissue to be treated, The diagnosis support system according to claim 1, wherein output data including a result of discriminating the plurality of types of skin diseases is generated based on an output of the learned model of the machine learning. The diagnosis support system according to claim 9, wherein an image representing the distribution of atypical cells in the skin tissue and the third image are prepared for each cell from which the atypical cells are derived. An image representing the distribution of dermal cells in the skin tissue included in the pathological image, an image representing the distribution of epidermal cells in the skin tissue, an image representing the distribution of immune cells in the skin tissue, and a lumen in the skin tissue. An image representing the distribution of cells to be made as an input, and a learned model of machine learning having an output of discriminating a plurality of non-neoplastic skin diseases,
A first image representing the distribution of dermal cells in the skin tissue to be treated, a second image representing the distribution of epidermal cells in the skin tissue to be treated, and a distribution of immune cells in the skin tissue to be treated The non-neoplastic plural types of skin based on the output of the machine-learned model with respect to a third image and a fourth image representing a distribution of cells forming a lumen in the skin tissue to be processed. An output data generation unit that generates output data including a disease identification result,
Diagnostic support system having Machine learning learning that receives as input the image representing the distribution of dermal cells in the skin tissue included in the pathological image, the image representing the distribution of epidermal cells in the skin tissue, and the image representing the distribution of atypical cells in the skin tissue A first image representing the distribution of dermal cells in the skin tissue to be treated, a second image representing the distribution of epidermal cells in the skin tissue to be treated, and an atypical cell in the skin tissue to be treated Obtaining an output for a third image representing the distribution of
Based on the output from the learned model of the machine learning, generating output data related to skin diseases,
And a computer-implemented diagnosis support method. The output of the learned model of machine learning is the identification of a specific skin disease related to the tumor,
The output data on the skin disease includes a result of identifying the specific skin disease,
The diagnosis support method according to claim 12. The input of the machine learning learned model further includes an image representing a distribution of immune cells in the skin tissue,
The output of the learned model of machine learning is a prediction of the pathology of a particular skin disease associated with a tumor,
In the step of generating the output data,
A prediction of the pathology of the specific skin disease based on the output of the machine learning-learned model for the first to third images and a fourth image representing the distribution of immune cells in the skin tissue to be processed; The diagnostic support method according to claim 12, wherein output data including a result is generated. The input of the learned model of the machine learning further includes an image representing the distribution of immune cells in the skin tissue, and an image representing the distribution of cells forming a lumen in the skin tissue,
The output of the learned model of the machine learning is a discrimination of a plurality of skin diseases related to the tumor,
In the step of generating the output data,
With respect to the first to third images, a fourth image representing the distribution of immune cells in the skin tissue to be treated, and a fifth image representing the distribution of cells forming a lumen in the skin tissue to be treated, The diagnosis support method according to claim 12, wherein output data including a result of discriminating the plurality of types of skin diseases is generated based on an output of the machine-learned model. An image representing the distribution of dermal cells in the skin tissue included in the pathological image, an image representing the distribution of epidermal cells in the skin tissue, an image representing the distribution of immune cells in the skin tissue, and a lumen in the skin tissue. A first image representing the distribution of dermal cells in the skin tissue to be processed from a machine-learned model in which an image representing the distribution of cells to be made is input and the output of discriminating a plurality of non-neoplastic skin diseases is output. An image, a second image representing the distribution of epidermal cells in the skin tissue to be treated, a third image representing the distribution of immune cells in the skin tissue to be treated, and a lumen in the skin tissue to be treated Obtaining an output for a fourth image representing the distribution of cells making
Based on the output of the trained model of the machine learning, the step of generating output data including a discrimination result of the non-neoplastic multiple types of skin diseases,
And a computer-implemented diagnosis support method.   A program to be executed by one or more processors to execute the diagnosis support method according to claim 12.

Images