JP2019148473A - Method for image analysis, image analyzer, program, method for manufacturing learned deep learning algorithm, and learned deep learning algorithm - Google Patents

Abstract

To generate data showing whether the region in analysis data is a tumor region.SOLUTION: The above objective is attained by a method for image analysis which analyzes an image of a tissue using a deep learning algorithm 60 of a neutral network structure. The method for image analysis includes: generating analysis data 80 from an analysis target image 78 including the tissue to be analyzed; inputting the analysis data 80 into the deep learning algorithm 60; and generating data showing whether the region in the analysis data is a tumor region by the deep learning algorithm 60.SELECTED DRAWING: Figure 6

Description

  The present specification discloses an image analysis method, an image analysis device, a program, a manufacturing method of a learned deep learning algorithm, and a learned deep learning algorithm. More specifically, for an image of a tissue to be analyzed, an image analysis method, an image analysis device, a program, and a learned deep layer including generating data indicating whether or not the region included in the analysis data is a tumor region A learning algorithm manufacturing method, a learned deep learning algorithm, and the like are disclosed.

  Patent Document 1 discloses an image diagnosis support apparatus that classifies and determines a tissue image in a pathological tissue image into four groups: normal, benign tumor, precancerous state, and cancerous state. The image classification means extracts a gaze area from the image data, calculates a feature amount indicating the feature of the gaze area, and classifies the group based on the calculated feature amount. The feature amount includes a density of a lump per unit area in the cell nucleus, a density of a lump area, a lump area, a lump thickness, a lump length, and the like. The image determination unit learns the relationship between the feature amount and the determination result, and performs determination based on the learned parameter that has been learned. In the learning, machine learning is performed using a learning algorithm such as a support vector machine.

JP 2010-203949 A

  When making a definitive diagnosis as to whether or not the tumor is a malignant tumor, a histopathological diagnosis using a histopathological specimen is performed. Further, histopathological diagnosis is often performed as an intraoperative rapid diagnosis for determining the excision site of a tissue containing a malignant tumor during an operation. Intraoperative rapid diagnosis involves waiting while the affected area of the patient is incised during the operation to determine whether the tumor is malignant, whether there is no tumor at the stump of the excised tissue, or whether there is lymph node metastasis. This is performed by histopathological diagnosis. The result of the intraoperative rapid diagnosis determines the direction of the subsequent operation of the waiting patient.

  The pathological diagnosis is performed by a doctor, particularly a pathologist, by observing a tissue specimen with a microscope or the like. Originally, it is necessary to repeat observation of tissue specimens of various cases, and it takes a lot of time to train pathologists.

  The shortage of pathologists is serious, and as a result of the shortage of pathologists, there is a concern that the diagnosis of the patient's malignant tumor is delayed, the start of treatment is delayed, or the treatment starts without waiting for a definitive diagnosis Yes. In addition, since both normal tissue diagnosis and intraoperative rapid diagnosis are concentrated on a small number of pathologists, the amount of work for one pathologist is enormous, and the labor status of the pathologist itself is also a problem. However, no solution to this problem has been found at present.

  Therefore, it is considered that the fact that the apparatus can support histopathological diagnosis greatly contributes to the elimination of the shortage of pathologists and the improvement of the working conditions of the pathologists, especially when the diagnosis is closer to the judgment by human eyes.

  In that the apparatus supports pathological diagnosis, the invention described in Patent Document 1 described above performs pathological determination of a sample tissue based on image analysis by machine learning. In this method, it is necessary to create a feature value by hand. There is a problem in the method of creating the feature quantity by hand that the ability of the person greatly affects the performance of image analysis. Furthermore, in the invention described in Patent Document 1, it is determined whether or not each cell nucleus in the tissue specimen image strongly magnified with a microscope is a cancer cell nucleus. Therefore, there is a problem that it takes a long time to analyze the entire sample.

  Therefore, before determining whether or not each cell is the nucleus of a cancer cell by a pathologist or a machine learning algorithm, the entire tissue contained in one specimen is observed, and the tumor tissue is located in which part of the specimen. Screening for inclusion helps to reduce the time required for pathological diagnosis of one specimen.

  The present invention relates to an image analysis method, an image analysis apparatus, a program, a learned deep learning algorithm manufacturing method, and learning for generating data indicating whether or not a region included in analysis data is a tumor region. It is an object to provide a deep learning algorithm.

  An embodiment relates to an image analysis method for analyzing an image of a tissue using a deep learning algorithm (60) having a neural network structure. In the image analysis method, the analysis data (80) is generated from the analysis target image (78) including the tissue to be analyzed, the analysis data (80) is input to the deep learning algorithm (60), and the deep learning algorithm is input. By (60), data indicating whether or not the region included in the analysis data is a tumor region is generated. According to the present invention, it is possible to generate data indicating whether or not the region included in the analysis data in the analysis target tissue is a tumor region.

  In the embodiment, preferably, the data indicating whether or not the region included in the analysis data is a tumor region is data indicating the region of the tumor region. According to this embodiment, the user can grasp the tumor region in the tissue to be analyzed at a glance.

  In the embodiment, preferably, the analysis target image is an image of a specimen for tissue diagnosis, and includes a hue obtained by combining two or more primary colors. According to this embodiment, pathological diagnosis can be assisted.

  In the embodiment, the analysis target image is preferably an image obtained by enlarging the analysis target tissue from 3 to 20 times. In the embodiment, the size of the region included in the analysis target image is 200 μm × 200 μm or more and 400 μm × 400 μm or less. According to these embodiments, more accurate determination can be performed.

  In the embodiment, preferably, data for separately presenting the tumor cell region and other regions is generated based on data indicating whether or not the region included in the analysis data is a tumor region. To do. According to this embodiment, the user can grasp the tumor region in the tissue to be analyzed at a glance.

  In the embodiment, preferably, data indicating a boundary between a tumor cell region and other regions is generated based on data indicating whether or not the region included in the analysis data is a tumor region. According to this embodiment, the user can grasp the boundary between the tumor region and the non-tumor region in the tissue to be analyzed at a glance.

  In the embodiment, preferably, data indicating the content rate of the tumor region in the analysis target tissue is generated based on data indicating whether or not the region included in the analysis data is a tumor region. According to this embodiment, the user can grasp the content rate of the tumor region in the tissue to be analyzed.

  In the embodiment, preferably, data indicating a ratio of a tumor region to a non-tumor region in the analysis target tissue is generated based on data indicating whether or not the region included in the analysis data is a tumor region. . According to this embodiment, the user can grasp the content rate of the tumor region in the tissue to be analyzed. In addition, it is possible to know whether a sample used for genetic testing or the like is appropriately collected.

  According to the above embodiment, preferably, data for separately presenting a tumor region, a non-tumor region, and a region not including a tissue is generated for one analysis target image. According to this embodiment, the user can grasp at a glance whether or not there is a tumor region in the tissue to be analyzed.

  In the embodiment, preferably, a plurality of analysis data corresponding to each region having a predetermined number of pixels is generated for one analysis target image. According to this embodiment, it is possible to accurately analyze a wide range of tissues to be analyzed.

  In the embodiment, preferably, the deep learning algorithm (60) indicates that the region having the predetermined number of pixels includes at least one selected from a tumor region and a non-tumor region with respect to the input analysis data. Is generated.

  In the embodiment, preferably, the number of nodes in the input layer (50a) of the neural network (50) corresponds to the product of the predetermined number of pixels of the analysis data (80) and the number of combined primary colors. . According to this embodiment, a highly accurate analysis can be performed.

  In the embodiment, preferably, the specimen is a stained specimen, and the analysis target image is an image obtained by imaging the stained specimen in a bright field of a microscope. According to this embodiment, pathological diagnosis can be assisted.

  In the above embodiment, preferably, the training data used for the learning of the deep learning algorithm is a staining of a specimen prepared by performing staining for bright field observation on a specimen of a tissue including a tumor region collected from an individual. The image is generated based on a bright field image obtained by capturing the image under a bright field of a microscope. This embodiment can help with tissue diagnosis.

  In the embodiment, preferably, the staining for bright field observation uses hematoxylin for nuclear staining. More preferably, the bright field observation staining is hematoxylin-eosin staining (HE staining). According to this embodiment, examinations with high versatility can be covered as tissue diagnosis.

  In the embodiment, preferably, the training data includes a label value indicating a tumor region determined from the bright field image. More preferably, the training data includes the label value for each region of a predetermined number of pixels of the bright field image. According to this embodiment, training data for a tumor region can be generated.

  In the embodiment, preferably, the training data is generated for each region having a predetermined number of pixels in the bright field image. According to this aspect, the learning efficiency of the deep learning algorithm (50) can be increased.

  In the embodiment, preferably, the deep learning algorithm classifies the analysis data into a class indicating that a tissue included in the analysis target image is a tumor region.

  In the embodiment, preferably, the output layer (50b) of the neural network (50) is a node having a softmax function as an activation function. According to this embodiment, learning efficiency and analysis accuracy can be increased.

  In the embodiment, it is preferable that the deep learning algorithm sets data indicating that the tissue included in the analysis target image is a tumor region every time the analysis data (78) is input to a region having a predetermined number of pixels. Generate every time. According to this embodiment, the analysis efficiency can be increased.

  In the embodiment, the deep learning algorithm (60) is preferably generated according to the type of the tissue. Further, preferably, the analysis data is processed using the deep learning algorithm corresponding to the type of tissue to be analyzed, selected from the plurality of deep learning algorithms according to the type of tissue. . According to this embodiment, the analysis accuracy can be increased.

  An embodiment relates to an image analysis apparatus (100) that analyzes an image of a tissue using a deep learning algorithm (60) having a neural network structure. The analysis device generates analysis data (80) from the analysis target image (78) including the tissue to be analyzed, inputs the analysis data (80) to the deep learning algorithm (50), and inputs the deep learning algorithm ( 50), a processing unit (10) for generating data indicating whether or not the region included in the analysis data is a tumor region is provided.

  One embodiment relates to a computer program for analyzing tissue images using a neural network structured deep learning algorithm (60). Processing for generating analysis data (80) from the analysis target image (78) including the tissue to be analyzed, processing for inputting the analysis data (80) to the deep learning algorithm (60), and A process for generating data indicating whether or not the region included in the analysis data is a tumor region is executed by a learning algorithm (60).

  One embodiment generates training data used for learning of the deep learning algorithm (50) based on a training image obtained from a tissue sample for training, and causes the neural network (50) to learn the training data. Learning, and the generating step includes a first acquisition step of acquiring first training data (74R1) corresponding to a first training image (70R1) obtained by imaging a tumor region, and a non-tumor A second acquisition step of acquiring second training data (74R2) corresponding to the second training image (70R2) that images the region, and a third training image (70R3) that images the region not including the tissue ) Acquiring third training data (74R3) corresponding to the first training data (74R3), and the learning step includes the first training data (74R3). ) Causes the neural network (50) to learn that the neural network (50) is a tumor region and the second training data (74R2) is a non-tumor region. A learned deep learning algorithm comprising: a second learning step; and a third learning step for causing the neural network (50) to learn that the third training data (74R3) is a region not including tissue. Relates to the generation method. Preferably, the first training data (74R1), the second training data (74R2), and the third training data (74R3) are used as an input layer (50a) of the neural network (50), and in the tumor region The first training data (74R1), the second training data (74R2), and the third training data (the non-tumor region and the region not including the tissue). 74R3) and the output layer (50b) of the neural network (50). More preferably, before the first acquisition step, before generating the first training data (74R1) from the first training image (70R1) and before the second acquisition step. From the second training image (70R3), the step of generating the second training data (74R2) from the second training image (70R2), and the third training image (70R3) before the third acquisition step, Generating the third training data (74R3). According to the present embodiment, it is possible to generate a learned deep learning algorithm that determines that a tissue to be analyzed includes at least one selected from a tumor region and a non-tumor region.

  In the generation method, preferably, the training image is a bright field image obtained by imaging a stained image of a specimen prepared by performing staining for bright field observation on a tissue collected from an individual under a bright field of a microscope. It is.

  In one embodiment, the first training data (74R1), the second training data (74R2), and the third training data (74R3) serve as an input layer (50a) of the neural network (50), and a tumor region , A non-tumor region, and a region that does not include tissue, the first training data (74R1), the second training data (74R2), and the third training data, respectively. A deep learning algorithm learned as an output layer (50b) of a neural network (50) corresponding to (74R3), wherein the first training data (74R1) images a tumor region of a tissue for training The second training data (74R2) is generated from the first training image (70R1), and the second training data (74R2) is generated from the second training image (70R2) obtained by imaging the non-tumor region of the training tissue. Made is, the third training data (74R3) is generated from the third training image of the captured area does not include the tissue (70R3), a learned deep learning algorithm (60).

  Data for screening the whole tissue contained in the specimen and screening which part of the specimen contains the tumor tissue can be generated.

It is a schematic diagram for demonstrating the outline | summary of the deep learning method. It is a schematic diagram for demonstrating the outline | summary of the deep learning method. It is a schematic diagram for demonstrating the outline | summary of the deep learning method. It is a schematic diagram for demonstrating the detail of training data. It is a schematic diagram for demonstrating the detail of training data. It is a schematic diagram for demonstrating the outline | summary of an image analysis method. 1 is a schematic configuration diagram of an image analysis system according to a first embodiment. 2 is a block diagram showing a hardware configuration of a vendor-side device 100. FIG. 3 is a block diagram showing a hardware configuration of a user side device 200. FIG. It is a block diagram for demonstrating the function of 100 A of deep learning apparatuses which concern on 1st Embodiment. It is a flowchart which shows the procedure of a deep learning process. It is a schematic diagram for demonstrating the detail of the learning by a neural network. It is a block diagram for demonstrating the function of image analysis apparatus 200A which concerns on 1st Embodiment. It is a flowchart which shows the procedure of an image analysis process. It is a schematic block diagram of the image analysis system which concerns on 2nd Embodiment. It is a block diagram for demonstrating the function of the integrated image analysis apparatus 200B which concerns on 2nd Embodiment. It is a schematic block diagram of the image analysis system which concerns on 3rd Embodiment. It is a block diagram for demonstrating the function of the integrated image analysis apparatus 100B which concerns on 3rd Embodiment. It is the analysis result of the tissue sample acquired from the stomach cancer tissue. (A) is a hole slide image of a bright-field image obtained by HE-staining gastric cancer tissue and is an image to be analyzed. (B) is an image in which a pathologist designates a tumor region of a hole slide image and is surrounded by a solid line. (C) shows an analysis result using an image for training and an image for analysis having a magnification of 1 ×. (D) shows the analysis result using the image for training and the image for analysis of 5 times magnification. (E) shows the analysis result using the image for training and the image for analysis of 40 times magnification. In FIGS. 19C to 19E, the tumor area obtained by the analysis process is displayed in white, the non-tumor area is shown in gray, and the non-tissue area is shown in black. It is the analysis result of the tissue sample acquired from the stomach cancer tissue. (A) is a hole slide image of a bright-field image obtained by HE-staining gastric cancer tissue and is an image to be analyzed. (B) is an image in which a pathologist designates a tumor region of a hole slide image and is surrounded by a solid line. (C) shows an analysis result using an image for training and an image for analysis having a magnification of 5 times. (D) shows the analysis result using the image for training and the image for analysis of 16 times magnification. In FIGS. 19C and 19D, the tumor area obtained by the analysis process is displayed in white, the non-tumor area is shown in gray, and the non-tissue area is shown in black. It is a figure which shows the difference of the discrimination precision by the magnification factor of a tissue sample, and the number of pixels of window size. (A) shows the sensitivity of each magnification, and (b) shows the positive predictive value of each magnification. (C) shows the sensitivity due to the difference in the number of pixels in the window size, and (d) shows the change in the positive predictive value.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, an overview and embodiments of the invention will be described in detail with reference to the accompanying drawings. In the following description and drawings, the same reference numerals indicate the same or similar components, and thus descriptions of the same or similar components are omitted.

  The image analysis method analyzes an image of a tissue. The image analysis method uses a deep learning algorithm having a neural network structure, preferably a convolutional neural network structure. The image analysis method includes generating data indicating whether or not the region included in the analysis data is a tumor region.

  In the present invention, an image of a tissue or cell is an image obtained from a specimen of a tissue sample or a specimen of a sample containing cells. Samples of tissue samples or samples containing cells are taken from an individual. The individual is not particularly limited, but is preferably a mammal, more preferably a human. When a sample is collected from the individual, it does not matter whether the individual is alive or dead. The tissue is not limited as long as it exists within an individual. Examples of the tissue collected from the individual include surgically excised tissue and biopsy tissue. The tumor may be epithelial or non-epithelial. The tumor is preferably a malignant epithelial tumor. The malignant tumor is not particularly limited. Examples of the malignant tumor include respiratory malignant tumors arising from the trachea, bronchi, lungs, etc .; nasopharynx, esophagus, stomach, duodenum, jejunum, ileum, cecum, appendix, ascending colon Gastrointestinal malignant tumor arising from the transverse colon, sigmoid colon, rectum or anus; liver cancer; pancreatic cancer; urinary malignant tumor arising from the bladder, ureter or kidney; ovary, fallopian tube, uterus, etc. Female genital malignant tumors that occur; breast cancer; prostate cancer; skin cancer; endocrine malignant tumors such as the hypothalamus, pituitary gland, thyroid gland, parathyroid gland, adrenal glands; central nervous system malignant tumors; malignant tumors that arise from bone and soft tissue Solid tumors. More preferably, respiratory epithelial malignant tumors such as lung cancer (squamous cell cancer, small cell cancer, large cell cancer, adenocarcinoma); gastric cancer, duodenal cancer, colon cancer (sigmoid colon cancer, rectal cancer, etc.) Gastrointestinal epithelial malignant tumors; liver cancer; pancreatic cancer; bladder cancer; thyroid cancer; ovarian cancer; breast cancer; Most preferred is gastric cancer.

  The specimen is intended to be a processed state so that the tissue can be observed with a microscope or the like, for example, a preparation. The specimen can be prepared according to a known method. For example, in the case of a tissue specimen, after the tissue is collected from the individual, the tissue is fixed with a predetermined fixing solution (formalin fixation or the like), the fixed tissue is embedded in paraffin, and the paraffin-embedded tissue is sliced. . Place thin slices on a slide glass. The slide glass on which the section is placed is stained for observation with an optical microscope, that is, for bright field observation, and a predetermined encapsulation process is performed to complete a specimen. A typical example of the tissue specimen is a specimen for tissue diagnosis (pathological specimen), and the staining is hematoxylin-eosin (HE) staining.

  Nuclear staining for HE staining is hematoxylin. Hematoxylin is widely used as a nuclear stain in tissue cell staining (for example, immunostaining, lectin staining, sugar staining, fat staining, collagen fiber staining, etc.). Therefore, the present invention can be applied to all specimens using such hematoxylin for nuclear staining.

  For the image analysis, a deep learning algorithm trained using a training image is used. In the image analysis, analysis data is generated from an analysis target image including a tissue to be analyzed acquired from the specimen. The analysis data is input to the deep learning algorithm, and data indicating whether or not a region in which the tissue present in the analysis target image is included in the analysis data is a tumor region is generated. The data indicating whether or not the region included in the analysis data is a tumor region is the possibility that the region including a plurality of target pixels included in the tissue to be discriminated includes a tumor tissue and / or a non-tumor region It is data indicating the possibility of Data indicating whether or not the region included in the analysis data is a tumor region can be a label value, a display, or the like that can distinguish between including a tumor region and a non-tumor region.

  The training image is acquired from a plurality of regions of a sample to be learned. Preferably, the training image is acquired from a plurality of layer structures included in a tissue to be learned from a region including a tumor region, a region including a non-tumor region, and a region not including a tissue for each layer structure.

  For example, the first training image 70R1 is an image acquired from a region including a tumor region included in a tissue collected from an individual. This image is acquired from a sample stained so that the tissue structure can be recognized by microscopic observation. The staining is not limited as long as the tissue structure can be recognized, but is preferably staining for bright field observation. The staining for bright field observation is not limited as long as at least a cell nucleus and a site other than the cell nucleus can be stained so as to be distinguishable by hue. When the specimen is a mammalian tissue specimen, for example, HE staining can be mentioned.

  From the first training image 70R1, first training data 74R1 and first preliminary training data 70R1L are generated. The first training data 74R1 is information relating to a single color image obtained by separating the hues included in the first training image 70R1 for each primary color. The first preliminary training data 70R1L is generated as binary data (label value) indicating that the tissue region included in the first training image 70R1 includes a tumor region. Whether or not the tissue region included in the first training image 70R1 includes a tumor region is determined by, for example, specimen observation by a pathologist or the like. Here, in the tissue region included in the training image, tumor cells, non-tumor cells, and tissue regions other than the cells may be mixed. In this case, when the area of the tissue included in the training image is 100% of the area of the entire tissue area included in the training image, for example, the tumor cells are about 10% or more of the entire area, Including tumor area when occupying about 20% or more, about 30% or more, about 40% or more, about 50% or more, about 60% or more, about 70% or more, about 80% or more, or about 90% or more Can be determined. Preferably, it can be determined to include a tumor region if it accounts for about 40% or more, about 50% or more, about 60% or more, about 70% or more, about 80% or more, or about 90% or more. Ratio in which the tumor cells occupy the entire tissue area by subtracting the ratio of non-tumor cells and tissues other than cells (non-tumor areas) from the area (100%) of the entire tissue area included in the training image May be determined as

  The second training image 70R2 included in the training image is an image acquired from a tissue region including a non-tumor region (preferably a region including non-tumor cells and a tissue region other than cells). . This image is acquired from a sample stained so that the tissue structure can be recognized by microscopic observation. The staining is not limited as long as the tissue structure can be recognized, but is preferably staining for bright field observation. The staining for bright field observation is not limited as long as at least a cell nucleus and a site other than the cell nucleus can be stained so as to be distinguishable by hue. When the specimen is a mammalian tissue specimen, for example, HE staining can be mentioned. The staining is preferably the same as the specimen from which the first training image 70R1 has been acquired.

  From the second training image 70R2, second training data 74R2 and second preliminary training data 70R2L are generated. The second training data 74R2 is information related to a single color image obtained by separating the hue included in the second training image 70R2 for each primary color. The second preliminary training data 70R2L is binarized data (label value) indicating that the tissue region included in the second training image 70R2 includes a non-tumor region. Whether or not the tissue region included in the second training image 70R2 includes a non-tumor region is determined by, for example, specimen observation by a pathologist or the like. Here, in the tissue region included in the training image, tumor cells, non-tumor cells, and tissue regions other than the cells may coexist. In this case, when the area of the tissue included in the training image is, for example, 100% of the total area of the tissue included in the training image, the tissue area other than the non-tumor cells (non-tumor) Region) is about 10% or more, about 20% or more, about 30% or more, about 40% or more, about 50% or more, about 60% or more, about 70% or more, about 80% or more, about When it occupies 90% or more, or 100%, it can be determined that a non-tumor region is included. Preferably, when it accounts for about 40% or more, about 50% or more, about 60% or more, about 70% or more, about 80% or more, about 90% or more, or 100%, it is determined to include a non-tumor region Can do. The area obtained by subtracting the ratio of the tumor area from the area (100%) of the entire tissue area included in the second training image 70R2 is the non-tumor cell and the area of the tissue other than the cells. You may determine as a ratio to occupy.

  The third training image 70R3 included in the training image is a region (also referred to as a non-tissue region) that does not include a tissue in the specimen from which the first training image 70R1 and / or the second training image 70R2 is acquired. ).

  From the third training image, third training data 74R3 and third preliminary training data 70R3L are generated. The third training data 74R3 is information related to a single color image obtained by separating the hue included in the third training image 70R3 for each primary color. The third preliminary training data 70R2L is binarized data (label value) indicating that the region included in the third training image 70R3 is a region not including a tissue. The region included in the third training image 70R3 is determined by specimen observation by a pathologist or the like, for example. Here, the training image may include a tissue region and a background region (for example, a glass portion of a preparation). In this case, when the area of the background included in the training image is, for example, 100% of the area of the entire region included in the one training image, the background region is approximately 90% or more and approximately 91%. More than about 92%, about 93% or more, about 94% or more, about 95% or more, about 96% or more, about 97% or more, about 98% or more, or about 99% or more, or background area Occupies 100%, it can be determined that the region does not include the tissue. Preferably, if the background area occupies about 95% or more, about 96% or more, about 97% or more, about 98% or more, or about 99% or more, or if the background area occupies 100% It can be determined that the area is not included.

  Each training image is preferably acquired as an image obtained by enlarging a training tissue sample, for example, 5 to 20 times. From the viewpoint of discrimination accuracy, an image enlarged 30 times or more, particularly 40 times or more is unsuitable as a training image.

  Tumor region training data is generated from the first training data 74R1 and the first preliminary training data 70R1L, and non-tumor region training data is generated from the second training data 74R2 and the second preliminary training data 70R2L. Non-tissue region training data is generated from the third training data 74R3 and the third preliminary training data 70R2L. The tumor area training data, non-tumor area training data, and non-tissue area training data may be collectively referred to as training data.

  In the outline and the embodiment of the present invention, an example in which the deep learning algorithm determines that the tissue included in the image obtained by imaging the HE-stained tissue specimen includes a tumor region, a non-tumor region, and a non-tissue region. Will be described.

[Outline of deep learning method and image analysis method]
First, the deep learning method will be described.

Outline of Deep Learning Method The outline of the first training data 74R1, the first preliminary training data 70R1L, and the tumor region training data 75R1 will be described with reference to FIG. FIG. 1 shows an example of input of training data to a neural network using a hole slide image 70W1 of a tissue specimen acquired by a slide image scanner. A hole slide image 70W1 shows a slide image of a tissue specimen obtained by imaging a specimen prepared by performing HE staining as bright field observation staining in a bright field. The hole slide image 70W2 is an image specified by the pathologist enclosing the portion including the tumor region in the hole slide image 70W1 with a solid line by the pathologist. The hole slide image 70W3 indicates an image obtained by dividing the hole slide image 70W2 as a training image (for example, 512 divisions). A region surrounded by a square frame indicated by symbol R1 in the hole slide image 70W3 indicates a region used as the first training image 70R1 shown in FIG. A region surrounded by a square frame indicated by a symbol R2 indicates a region used as the second training image 70R2 illustrated in FIG. A region surrounded by a square frame indicated by a symbol R3 indicates a region used as the third training image 70R3 illustrated in FIG. The first training image 70R1 includes a tumor region, the second training image 70R2 includes a non-tumor region, and the third training image 70R3 includes a tissue-free region. You may determine before acquiring a training image, and you may determine after acquiring each training image.

  In FIG. 1, the first training image 70R1 is obtained by capturing a HE-stained specimen as a color image, for example, under bright field observation. Therefore, the first training image 70R1 includes a plurality of hues.

  The first training image 70R1 can be acquired in advance using an image acquisition device such as a known optical microscope, fluorescence microscope, or virtual slide scanner. Illustratively, the color imaging acquired from the image acquisition device in the present embodiment is preferably a 24-bit color with a color space of RGB. In the RGB 24-bit color, it is preferable to express the red, green, and blue densities (color densities) with 8-bit (256 levels) gradation. The first training image 70R1 may be an image including one or more primary colors. Corresponding to the first training image 70R1, first training data 74R1 is generated.

  In the present invention, the hue is illustratively defined by a combination of three primary colors of light or a combination of three primary colors. The first training data 74R1 is generated for each primary color by separating the hue appearing in the first training image 70R1 generated from the first training image 70R1 for each primary color, and a code corresponding to the density It is data represented by. In FIG. 1, an image (hereinafter also referred to as “single color image”) 72R1R represented by shades of a single color separated for each of the primary colors of light (red (R), green (G), and blue (B)). , 72R1G, 72R1B are obtained.

  The color density of each color is encoded for each pixel on the single color image 72R1R, 72R1G, 72R1B, and the entire image is encoded corresponding to the color density of each pixel for each image of R, G, B (hereinafter, “ 72R1r, 72R1g, and 72R1b are also generated. The color density may be encoded with a numerical value indicating 256 levels of each color. Further, the color density may be further pre-processed with respect to the numerical value indicating the 256 levels of each color, and the color density in each pixel may be encoded with, for example, a number indicating 8 levels from 0 to 7. Color density encoding diagrams 72R1r, 72R1g, and 72R1b generated from single color images of R, G, and B colors exemplarily shown in FIG. 1 show the color density values of 0 to 7 for each pixel for convenience of explanation. It is represented by a stage code. First training data 74R1 in which R, G, and B color density values are combined for each pixel is generated from the color density encoding diagrams 72R1r, 72R1g, and 72R1b shown in FIG. The code indicating the color density is also referred to as a color density value in this specification. In addition, instead of the color density coding diagram, a matrix of color density values corresponding to each pixel may be generated as the first training data 74R1.

  In FIG. 1, the first preliminary training data 70R1L is data indicating that the tissue included in the first training image 70R1 includes a tumor region. For example, since the first training image 70R1 is determined to be a tumor region by the determination of the pathologist, the first training image 70R1 includes a tumor for each region having a predetermined number of pixels to be described later. The same numerical value, for example, “1” is assigned as a label value indicating that the region is included, and the first preliminary training data 70R1L is obtained.

  In the deep learning method, tumor region training data 75R1 is generated from the first training data 74R1 and the first preliminary training data 70R1L shown in FIG. Then, the color density value data 76 (obtained from the first training data 74R1) of each pixel of the tumor region training data 75R1 is used as the input layer 50a, and a label value 77R1 (obtained from the first preliminary training data 70R1L) is output. The neural network 50 as the layer 50b is trained.

  Next, the outline of the second training data 74R2, the second preliminary training data 70R2L, and the non-tumor region training data 75R2 will be described with reference to FIG. In FIG. 2, since the hole slide image 70W1, the hole slide image 70W2, and the hole slide image 70W3 are the same as those in FIG. 1, the second training image 70R2 includes a plurality of images similar to the first training image 70R1. Hue is included.

  The second training image 70R2 is acquired by the same method as the first training image 70R1 except that a tissue including a non-tumor region is used instead of a tissue including a tumor region. Corresponding to the second training image 70R2, second training data 74R2 is generated.

  The second training data 74R2 is generated in the same manner as the first training data 74R1, except that a tissue including a non-tumor region is used instead of a tissue including a tumor region. Second training data 74R2 in which R, G, and B color density values are combined for each pixel is generated from the color density encoding diagrams 72R2r, 72R2g, and 72R2b shown in FIG.

  In FIG. 2, the second preliminary training data 70R2L is data indicating that the second training image 70R2 includes a non-tumor region. Since the entire area of the second training image 70R2 is determined to be a non-tumor area by the determination of the pathologist, the second training image 70R2 includes a non-tumor for each area having a predetermined number of pixels to be described later. The same numerical value, for example, “2” is assigned as the label value indicating that the region is included, and the second preliminary training data 70R2L is obtained. The label value indicating the region including the non-tumor region is not limited as long as it can be distinguished from the label value indicating the region including the tumor region and the region not including the tissue.

  In the deep learning method, non-tumor region training data 75R2 is generated from the second training data 74R2 and the second preliminary training data 70R2L shown in FIG. The color density value data 76 (obtained from the second training data 74R2) of each pixel of the non-tumor region training data 75R2 is used as the input layer 50a, and the label value 77R2 (obtained from the second preliminary training data 70R2L) is used. The neural network 50 serving as the output layer 50b is trained.

  Next, the outline of the third training data 74R3, the third preliminary training data 70R3L, and the non-tissue region training data 75R3 will be described with reference to FIG. In FIG. 3, since the hole slide image 70W1, the hole slide image 70W2, and the hole slide image 70W3 are the same as those in FIG. 1, the third training image 70R3 includes a plurality of images similar to the first training image 70R1. Hue is included.

  The third training image 70R3 is acquired by the same method as the first training image 70R1 except that a region not including tissue is used instead of a region including a tumor region. Corresponding to the third training image 70R3, third training data 74R3 is generated.

  The third training data 74R3 is generated in the same manner as the first training data 74R1, except that a region not including tissue is used instead of a region including a tumor region. The third training data 74R3 in which R, G, and B color density values are combined for each pixel is generated from the color density encoding diagrams 72R3r, 72R3g, and 72R3b shown in FIG.

  In FIG. 3, the third preliminary training data 70R3L is data indicating that the region included in the third training image 70R3 is a region not including the tissue. Since all the regions of the third training image 70R3 are determined to be non-tissue regions by the pathologist's determination, the third training image 70R3 includes a tissue for each region having a predetermined number of pixels to be described later. The same numerical value, for example, “0” is added as a label value indicating that the region does not include, and the third preliminary training data 70R3L is obtained. A label value indicating that the region does not include a tissue is distinguished from a numerical value indicating a region including a tumor region and a region including a non-tumor region.

  In the deep learning method, the non-tissue region training data 75R3 is generated from the third training data 74R3 and the third preliminary training data 70R3L shown in FIG. 3, and the color density value data 76 of each pixel of the non-tissue region training data 75R3. The neural network 50 having the input layer 50a as the input layer 50a (obtained from the third training data 74R3) and the output layer 50b as the label value 77R3 (obtained from the third preliminary training data 70R3L) is trained.

With reference to FIG. 4 (a), (b), the production | generation method of the tumor area | region training data 75R1 is demonstrated. The first training data 74R1 shown in FIG. 4A is data obtained by combining the color density values of the color density encoding diagrams 72R1r, 72R1g, and 72R1b of the tumor region having a predetermined number of pixels. In this specification, the position of each pixel of the first training data 74R1 is represented by _{m 1, m} 2 ··· _{m j} from above for convenience _l 1 columns from the _left, l 2 · · · l _i, the line . In FIG. 4A, the first training data 74R1 has its image size (size per one training data) simplified for convenience of explanation, and the first training data 74R1 is It has a total of 81 pixels, 9 pixels in the direction and 9 pixels in the horizontal direction. That is, the position of each pixel represents _l 1 columns from the _left, l 2 · · · _{l 9,} from the top row in _{m 1, m 2 ··· m 9} . The three values shown in FIG. 4A are the color density values of the R, G, and B colors in each pixel. Illustratively, the three values are stored in the order of red (R), green (G) and blue (B) from the left. For convenience, the color density value of each pixel in the first training data 74R1 is shown in 8 levels from 0 to 7. As an example of image pre-processing, the brightness of each color image 72R1R, 72R1G, 72R1B represented in 256 levels when captured is converted into 8-level color density values. As the color density value, for example, the lowest brightness (a gradation group having a low brightness value when expressed in 256 colors of RGB color) is set to a color density value 0, and a gradually higher value is assigned as the degree of brightness increases. Then, the highest brightness (a gradation group having a high luminance value when expressed in 256 steps of RGB color) is set as a color density value 7.

The tumor area training data 75R1 shown in FIG. 1 and FIG. 4 (b) is an area (hereinafter referred to as “window size”) composed of a predetermined number of pixels from the first training data 74R1 shown in FIG. 4 (a). The cut out data is attached with a label value 77R1 corresponding to the label value of the first preliminary training data 70R1L. The window-sized tumor region training data 75R1 is also shown in a simplified form of 3 × 3 pixels (l ₁ : l ₃ , m ₁ : m ₃ ) for convenience of explanation. from represented by _{l 1, l 2 ··· l i} , m 1 from the top _{row, m} 2 ··· m _j. The actual preferred window size is illustratively 125 × 125 pixels (i = 125, j = 125), 150 × 150 pixels (i = 150, j = 150), 166 × 166 pixels (i = 166, j = 166), 180 × 180 pixels (i = 180, j = 180), 200 × 200 pixels (i = 200, j = 200), 220 × 220 pixels (i = 220, j = 220), 250 × 250 pixels ( i = 250, j = 250) is preferable from the viewpoint of learning efficiency. More preferably, the window size is about 200 × 200 pixels. The window size is, for example, in a range of 60 μm × 60 μm to 500 μm × 500 μm. Preferably, they are 60 micrometers x 60 micrometers, 100 micrometers x 100 micrometers, 150 micrometers x 150 micrometers, 200 micrometers x 200 micrometers, 250 micrometers x 250 micrometers, 300 micrometers x 300 micrometers, 350 micrometers x 350 micrometers, 400 micrometers x 400 micrometers, 450 micrometers x 450 micrometers, 500 micrometers x 500 micrometers. More preferably, they are 200 micrometers x 200 micrometers-400 micrometers x 400 micrometers. For example, as shown in FIG. 4B, a window W1 (l ₁ : l ₃ , m ₁ : m ₃ ) of 3 × 3 pixels is set from the left end of the first training data 74R1, and the first training data The window W1 is moved from left to right with respect to 74R1. The window W1 cuts out the tumor region training data 75R1 of (l ₁ : l ₃ , m ₁ : m ₃ ), and then converts the label value 77R1 corresponding to the label value of the first preliminary training data 70R1L to the tumor region training data 75R1. Attached. The window W1 moves to the next window (l ₄ : l ₆ , m ₄ : m ₆ ) of the tumor area training data 74R1 indicated by a dotted line, and (l ₄ : l ₆ , m ₄ : m ₆ ) is a window size tumor. Cut out as area training data 75R1. A label value 77R1 corresponding to the label value of the first preliminary training data 70R1L is attached to the newly cut out tumor region training data 75R1. This operation is repeated to cut out tumor region training data 75R1 having a plurality of window sizes from the first training data 74R1. The extracted tumor region training data 75R1 of the window size is assigned a label value 77R1 corresponding to the label value of the first preliminary training data 70R1L, and is used for learning of the neural network 50 shown in FIG.

  As shown in FIG. 1, the number of nodes of the input layer 50a of the neural network 50 is the number of pixels of the tumor area training data 75R1 of the input window size and the number of primary colors included in the image (for example, the three primary colors of light, R, G, and B). The color density value data 76 of each pixel of the tumor area training data 75R1 of the window size is used as the neural network input layer 50a, and the label value 77R1 corresponding to the first preliminary training data 70R1L is used as the neural network output layer 50b. Let 50 learn. The color density value data 76 of each pixel is set data of color density values of R, G, and B colors of each pixel of the tumor region training data 75R1. For example, if the window size tumor region training data 75R1 is 3 × 3 pixels, one color density value is given for each of R, G, and B for each pixel. The number of color density values is “27” (3 × 3 × 3 = 27), and the number of nodes in the input layer 50a of the neural network 50 is also “27”.

  In this way, the window-sized tumor region training data 75R1 input to the neural network 50 can be automatically created by the computer without being created by the user. Thereby, efficient deep learning of the neural network 50 is promoted.

  As shown in FIG. 4B, in the initial state, the window W1 is located at the upper left corner of the first training data 74R1. Thereafter, the window-sized tumor region training data 75R1 is cut out by the window W1, and the position of the window W1 is moved each time the neural network 50 is learned. Specifically, the window W1 is moved so that the window W1 scans, for example, all the pixels of the first training data 74R1. Thereby, the tumor area training data 75R1 having a window size cut out from all the pixels of the first training data 74R1 is used for learning of the neural network 50. Therefore, the degree of learning of the neural network 50 can be improved, and as a result of deep learning, a deep learning algorithm having the structure of the neural network 60 shown in FIG. 6 is obtained.

  FIG. 5A shows the second training data 74R2. The method of generating the non-tumor region training data 75R2 from the second training data 74R2 replaces the first training data 74R1 and the first preliminary training data 70R1L with the second training data 74R2 and the second preliminary training data. Similar to the tumor area training data 75R1, except that 70R2L is used. A label value 77R2 corresponding to the label value of the second preliminary training data 70R2L attached to the non-tumor region training data 75R2 is input to the output layer 50b shown in FIG.

  FIG. 5B shows the third training data 74R3. The method for generating the non-tissue region training data 75R3 from the third training data 74R3 replaces the first training data 74R1 and the first preliminary training data 70R1L with the third training data 74R3 and the third preliminary training data. Similar to the tumor area training data 75R1, except that 70R3L is used. A label value 77R3 corresponding to the label value of the third preliminary training data 70R3L attached to the non-tissue region training data 75R3 is input to the output layer 50b shown in FIG.

  Each area training data is preferably generated using 10 or more, 20 or more, 50 or more, or 100 or more areas of training images.

Outline of Image Analysis Method As shown in FIG. 6, in the image analysis method, analysis data 80 is generated from an analysis target image (bright field image) 78 obtained by imaging a sample including a tissue to be analyzed. The specimen is preferably stained the same as the first training image. The analysis target image 78 can also be acquired as, for example, a color image using, for example, a known microscope or virtual slide scanner. The analysis target image 78 is preferably acquired at the same magnification or the same magnification as the training image. The analysis target image (bright field image) 78 may be an image including one or more primary colors. When the color analysis target image 78 is encoded with the color density values of the R, G, and B colors for each pixel, the entire image can be represented as an encoded diagram of the color density values for each pixel for each of R, G, and B. (Analysis color density coding diagram 79r, 79g, 79b). Analysis object data (not shown) in which R, G, and B color density values are combined for each pixel is generated from the analysis color density encoding diagrams 79r, 79g, and 79b. Color density encoding diagrams 79r, 79g, and 79b showing the color density codes in the single color images of R, G, and B shown in FIG. 6 exemplarily, instead of the images 79R, 79G, and 79B of the three primary colors, A color density value represented by a code of 8 levels from 0 to 7 is displayed. In this specification, the position of each pixel of the analysis target data represents for convenience columns from left _{l 1, l 2 ··· l i} , m 1 from the top _{row, m} 2 ··· m _j.

  The analysis data 80 is data obtained by cutting out the analysis target data in an area of a predetermined number of pixels (that is, the same window size as each area training data), and the tissue color density value included in the analysis target image 78. It is data including. In FIG. 6, the window size analysis data 80 is also simplified to 3 × 3 pixels for convenience of explanation, similarly to the tumor region training data 75R1, the non-tumor region training data 75R2, and the non-tissue region training data 75R3. Although shown, the actual preferred window size is similar to the tumor area training data 75R1, non-tumor area training data 75R2, and non-tissue area training data 75R3. For example, a 3 × 3 pixel window W2 is set, and the window W2 is moved with respect to the analysis target data. The window W2 includes a predetermined number of pixels in the analysis target data. When the analysis target data is cut out by the window W2 indicated by, for example, a 3 × 3 pixel black frame in the same manner as each region training data, the window size Analysis data 80 is obtained. In the same manner as each area training data, a plurality of analysis data 80 is generated for each area having a predetermined number of pixels corresponding to the window size from the analysis target data. Similarly to the tumor area training data 75R1, the non-tumor area training data 75R2, and the non-tissue area training data 75R3 in the analysis target data and window size analysis data 80, the color density value is red (R) for each pixel. They are stored in the order of green (G) and blue (B).

  The image analysis method uses a deep learning algorithm 60 having a neural network trained using the window size tumor region training data 75R1, non-tumor region training data 75R2 and non-tissue region training data 75R3 shown in FIGS. Then, the analysis data 80 is processed. By processing the analysis data 80, data 83 indicating whether or not the region included in the analysis data in the analysis target tissue is a tumor region is generated.

  Referring to FIG. 6 again, the neural network 60 in which the analysis data 80 cut out from the analysis target data generated based on the color density encoding diagrams 79r, 79g, and 79b of the R, G, and B colors constitutes the deep learning algorithm. Is input. The number of nodes in the input layer 60a of the neural network 60 corresponds to the product of the number of input pixels and the number of primary colors included in the image. When the color density value data 81 of each pixel of the analysis data 80 is input to the neural network 60, an estimated value 82 (three values) of the analysis data 80 is output from the output layer 60b. For example, when the estimated value is 1, it indicates that the region (pixel region) having a predetermined number of pixels constituting the analysis data 80 includes a tumor region, and when the estimated value is 2, the pixel region includes a non-tumor region. When the estimated value is 0, it indicates that the pixel area is a non-tissue area. That is, the estimated value 82 output from the output layer 60b of the neural network 60 is a label value generated for each pixel region of the analysis target image, and the region included in the analysis data in the analysis target image is a tumor region. It is data indicating whether or not there is. The estimated value 82 is also referred to as a class in the following description regarding the neural network. Based on the color density value data 81 of a predetermined number of pixels included in the analysis data 80, the neural network 60 generates a label value indicating that the tumor area is included, and a non-tumor region from the input analysis data 80. A label value indicating inclusion or a label value indicating a non-tissue region is generated. In other words, the neural network 60 classifies the analysis data 80 into a class indicating the state of the pixel region of interest included in the analysis target image (whether it is tumorized, not tumorized, or not tissue). Here, the color density value data 81 of each pixel is set data of color density values of R, G, and B colors of each pixel of the analysis data 80.

  Thereafter, the analysis data 80 is cut out by the window size while moving the window W2 in units of a predetermined number of pixels so that the window W2 scans all the pixels of the analysis target data. The extracted analysis data 80 is input to the neural network 60. Thereby, the label value 83 is obtained based on the data indicating whether or not the region included in the tissue analysis data in the analysis target image is a tumor region. In the example illustrated in FIG. 6, a region enhancement image 84 indicating a region is obtained by performing region detection processing on the label value 83. Specifically, the area detection process is, for example, a process of detecting each area in accordance with the estimated value 82. A pixel area with an estimated value 82 of “1” is actually a tumor area, and an estimated value 82 of “2”. The pixel area is determined as a non-tumor area, and the pixel area whose estimated value 82 is 0 is determined as a non-tissue area. The region-enhanced image 84 is a diagram in which the region detected by the image analysis process is represented by the label value 83 in color (tumor region is white, non-tumor region is gray, and non-tissue region is black). In addition, after each region is identified, a process of causing the display device to display a tumor region and other regions (that is, a non-tumor region and / or a non-tissue region) in a distinguishable manner may be performed. For example, processing such as painting the tumor region with a color, drawing a line between the tumor region and the other region, and the like are displayed on the display device in an identifiable manner. Moreover, you may perform the process which displays on a display device also about a non-tumor area | region and / or a non-tissue area | region so that identification is possible.

<First Embodiment>
In the first embodiment, the configuration of a system that implements the deep learning method and the image analysis method described in the above outline will be specifically described.
[Configuration overview]

  Referring to FIG. 7, the image analysis system according to the first embodiment includes a deep learning device 100A and an image analysis device 200A. The vendor-side device 100 operates as the deep learning device 100A, and the user-side device 200 operates as the image analysis device 200A. The deep learning device 100A causes the neural network 50 to learn using the training data, and provides the user with a deep learning algorithm 60 trained with the training data. The deep learning algorithm composed of the learned neural network 60 is provided from the deep learning device 100A to the image analysis device 200A through the storage medium 98 or the network 99. The image analysis apparatus 200 </ b> A analyzes an image to be analyzed using a deep learning algorithm composed of the learned neural network 60.

  The deep learning device 100A is configured by, for example, a general-purpose computer, and performs deep learning processing based on a flowchart described later. The image analysis apparatus 200A is configured by, for example, a general-purpose computer, and performs image analysis processing based on a flowchart described later. The storage medium 98 is a non-temporary tangible storage medium that can be read by a computer, such as a DVD-ROM or a USB memory.

  The deep learning device 100A is connected to the imaging device 300. The imaging apparatus 300 includes an imaging element 301 and a fluorescence microscope 302, and captures a bright field image and a fluorescence image of a learning sample 308 set on the stage 309. The learning specimen 308 is stained as described above. The deep learning device 100 </ b> A acquires the training image captured by the imaging device 300.

  The image analysis device 200A is connected to the imaging device 400. The imaging apparatus 400 includes an imaging element 401 and a fluorescence microscope 402, and captures a bright field image of the specimen 408 to be analyzed set on the stage 409. The specimen 408 to be analyzed is pre-stained as described above. The image analysis device 200 </ b> A acquires the analysis target image 78 captured by the imaging device 400.

  As the imaging devices 300 and 400, a known fluorescence microscope or virtual slide scanner having a function of imaging a specimen can be used. The imaging apparatus 400 may be an optical microscope as long as it has a function of imaging a specimen.

[Hardware configuration]
Referring to FIG. 8, the vendor-side device 100 (100A, 100B) includes a processing unit 10 (10A, 10B), an input unit 16, and an output unit 17.

  The processing unit 10 includes a CPU (Central Processing Unit) 11 that performs data processing, which will be described later, a memory 12 that is used as a data processing work area, a storage unit 13 that stores programs and processing data, which will be described later, and each unit. A bus 14 for transmitting data, an interface unit 15 for inputting / outputting data to / from an external device, and a GPU (Graphics Processing Unit) 19 are provided. The input unit 16 and the output unit 17 are connected to the processing unit 10. Illustratively, the input unit 16 is an input device such as a keyboard or a mouse, and the output unit 17 is a display device such as a liquid crystal display. The GPU 19 functions as an accelerator that assists arithmetic processing (for example, parallel arithmetic processing) performed by the CPU 11. That is, in the following description, the process performed by the CPU 11 means that the process performed by the CPU 11 using the GPU 19 as an accelerator is included.

  Further, the processing unit 10 stores in advance the program according to the present invention and the neural network 50 before learning in the storage unit 13 in an execution format, for example, in order to perform the processing of each step described in FIG. 10 below. . The execution format is, for example, a format generated by being converted from a programming language by a compiler. The processing unit 10 performs processing using the program stored in the storage unit 13 and the neural network 50 before learning.

  In the following description, unless otherwise specified, the processing performed by the processing unit 10 means processing performed by the CPU 11 based on the program stored in the storage unit 13 or the memory 12 and the neural network 50. The CPU 11 temporarily stores necessary data (such as intermediate data during processing) using the memory 12 as a work area, and appropriately stores data to be stored for a long period of time, such as calculation results, in the storage unit 13.

  Referring to FIG. 9, the user device 200 (200A, 200B, 200C) includes a processing unit 20 (20A, 20B, 20C), an input unit 26, and an output unit 27.

  The processing unit 20 includes a CPU (Central Processing Unit) 21 that performs data processing (to be described later), a memory 22 that is used as a data processing work area, a storage unit 23 that stores programs and processing data (to be described later), and each unit. A bus 24 for transmitting data, an interface unit 25 for inputting / outputting data to / from an external device, and a GPU (Graphics Processing Unit) 29 are provided. The input unit 26 and the output unit 27 are connected to the processing unit 20. Illustratively, the input unit 26 is an input device such as a keyboard or a mouse, and the output unit 27 is a display device such as a liquid crystal display. The GPU 29 functions as an accelerator that assists arithmetic processing (for example, parallel arithmetic processing) performed by the CPU 21. That is, in the following description, the processing performed by the CPU 21 means that the processing performed by the CPU 21 using the GPU 29 as an accelerator is included.

  Further, the processing unit 20 stores the program according to the present invention and the learned deep learning algorithm 60 of the neural network structure in the storage unit 23 in an execution format, for example, in order to perform the processing of each step described in FIG. 14 below. Pre-stored. The execution format is, for example, a format generated by being converted from a programming language by a compiler. The processing unit 20 performs processing using the program stored in the storage unit 23 and the deep learning algorithm 60.

In the following description, unless otherwise specified, the processing performed by the processing unit 20 is actually performed by the CPU 21 of the processing unit 20 based on the program stored in the storage unit 23 or the memory 22 and the deep learning algorithm 60. Means. The CPU 21 temporarily stores necessary data (such as intermediate data during processing) using the memory 22 as a work area, and appropriately stores long-term data such as calculation results in the storage unit 23.
[Function blocks and processing procedures]
・ Deep learning processing

  Referring to FIG. 10, the processing unit 10A of the deep learning device 100A according to the first embodiment includes a training data generation unit 101, a training data input unit 102, and an algorithm update unit 103. These functional blocks are realized by installing a program for causing a computer to execute a deep learning process in the storage unit 13 or the memory 12 of the processing unit 10A and executing the program by the CPU 11. The window size database 104 and the algorithm database 105 are stored in the storage unit 13 or the memory 12 of the processing unit 10A.

  The first training image 70R1, the second training image 70R2, and the third training image 70R3 of the learning sample are captured in advance by the imaging device 300 and stored in advance in the storage unit 13 or the memory 12 of the processing unit 10A. It shall be remembered. The neural network 50 is stored in advance in the algorithm database 105 in association with, for example, the type of organization (for example, organization name) from which the sample to be analyzed is derived.

  The processing unit 10A of the deep learning device 100A performs the process shown in FIG. If it demonstrates using each functional block shown in FIG. 10, the training data generation part 101 will perform the process of step S10 to S12, S17, and S18. The training data input unit 102 performs the process of step S14. The algorithm update unit 103 performs the processes in steps S15 and S16.

  In steps S10 to S18 described below, a deep learning process using the first training image 70R1, the second training image 70R2, and the third training image 70R3 will be described.

  The processing unit 10A performs a deep learning process for generating the tumor region training data 75R1, the non-tumor region training data 75R2, and the non-tissue region training data 75R3 according to the method described in the outline of the deep learning method. The processing unit 10A displays, for example, a wide area image (hole slide image 70W1) including the area of the first training image 70R1 on the output unit 17 by an operation from the input unit 16 by a pathologist or the like. The pathologist or the like who makes the determination visually confirms the image of the hole slide image 70W1 displayed on the output unit 17. A pathologist or the like designates an area in the hole slide image 70W1 determined to include the tumor area via, for example, the input unit 16, and surrounds the hole slide image 70W1 with a solid line such as red. Similarly, for a non-tumor region and a non-tissue region, a pathologist or the like surrounds the region in the hole slide image 70W1 with, for example, solid lines such as blue and green different from red. Instead of displaying the hole slide image 70W1 on the output unit 17 and having the pathologist determine it, the processing unit 10A receives the determined hole slide image 70W2 via the I / F unit 15 through the network 99, for example. You may get through. A pathologist or the like may divide the hole slide image 70W2 into pixels preferable as training images.

  The processing unit 10A includes a predetermined number of pixels for the first training image 70R1 acquired from the region surrounded by the red solid line in the hole slide image 70W2 by the designation from the input unit 16 such as a pathologist. Cut out as follows. Similarly, each of the second training image 70R2 and the third training image 70R3 is cut out from the portions determined as the non-tumor region and the non-tissue region so as to include a predetermined pixel region.

  In step S10, the processing unit 10A generates color density encoded diagrams 72R1r, 72R1g, and 72R1b for each color of R, G, and B from the cut out first training image 70R1. The color density encoding diagrams 72R1r, 72R1g, and 72R1b are generated by assigning to each pixel a code that represents the color density values of the R, G, and B colors of each pixel of the first training image 70R1 step by step. In the present embodiment, color density coding diagrams 72R1r, 72R1g, and 72R1b are generated for each of the R, G, and B gradation images, assuming that the color density value is expressed by 256 gradations from value 0 to value 255. For the color density value assignment, for example, the lowest brightness is set to a color density value 0, a higher value is assigned gradually as the brightness level becomes higher, and the highest brightness is set to a color density value 255. The processing unit 10A generates first training data 74R1 with R, G, and B color density values for each pixel from the color density coding diagrams 72R1r, 72R1g, and 72R1b. The processing unit 10A generates first preliminary training data 70R1L in which each pixel of the first training image 70R1 is associated with “1” that is a label value indicating a tumor region.

  In step S11, the processing unit 10A generates second training data 74R2 from the second training image 70R2 by the same method as in step S10. The processing unit 10A generates second preliminary training data 70R2L in which each pixel of the second training image 70R2 is associated with “2” that is a label value indicating a non-tumor region.

  In step S12, the processing unit 10A generates third training data 74R3 from the third training image 70R3 by the same method as in step S10. The processing unit 10A generates third preliminary training data 70R3L in which each pixel of the third training image 70R3 is associated with “0” that is a label value indicating a non-tissue region.

  Step S10 to step S12 may be performed simultaneously or in any order.

  In step S <b> 13, the processing unit 10 </ b> A receives an input of a learning tissue type from the user on the deep learning device 100 </ b> A side through the input unit 16. The processing unit 10A refers to the window size database 104 (window size DB 104) based on the input organization type, sets the window size, refers to the algorithm database 105 (algorithm DB 105), and uses the neural network for learning. Set up the network 50. The window size is, for example, 200 × 200 pixels. The window size is a unit of training data input to the neural network 50 at one input, and the product of the number of pixels of the training data corresponding to each region of the window size and the number of primary colors included in the image, This corresponds to the number of nodes in the input layer 50a. The window size is associated with the organization type and stored in advance in the window size database 104.

  In step S14, the processing unit 10A generates window-sized tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3 from the training data corresponding to each region. Specifically, in the above-mentioned “Outline of Deep Learning Method”, as described with reference to FIGS. 4A and 4B, from the first training data 74R1 and the first preliminary training data 70R1L, The window size tumor area training data 75R1 is created by the window W1. The processing unit 10A generates non-tumor region training data 75R2 having a window size from the second training data 74R2 and the second preliminary training data 70R2L. Specifically, as described with reference to FIGS. 4A, 4B, and 5A in the above-described “Outline of Deep Learning Method”, the second training data 74R2 and the second training data 74R2 From the preliminary training data 70R2L, window size non-tumor region training data 75R2 is created by the window W1. The processing unit 10A generates non-tissue region training data 75R3 having a window size from the third training data 74R3 and the third preliminary training data 70R3L. Specifically, as described with reference to FIGS. 4A, 4B, and 5B in the above-mentioned “Outline of Deep Learning Method”, the third training data 74R3 and the third training data 74R3 From the preliminary training data 70R3L, window size non-tissue region training data 75R3 is created by the window W1.

  In step S15 shown in FIG. 11, the processing unit 10A causes the neural network 50 to learn using the window-sized tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3. The learning result of the neural network 50 is accumulated every time the neural network 50 is learned using the window size tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3.

  In the image analysis method according to the embodiment, a convolutional neural network is used, and the stochastic gradient descent method is used. Therefore, in step S16, the processing unit 10A accumulates learning results for a predetermined number of trials. It is judged whether it is done. When the learning result is accumulated for the predetermined number of trials, the processing unit 10A performs the process of step S17. When the learning result is not accumulated for the predetermined number of trials, the processing unit 10A performs the process of step S18.

  When learning results are accumulated for a predetermined number of trials, in step S17, the processing unit 10A updates the connection weight w of the neural network 50 using the learning results accumulated in step S15. In the image analysis method according to the embodiment, since the stochastic gradient descent method is used, the connection weight w of the neural network 50 is updated when the learning results for a predetermined number of trials are accumulated. Specifically, the process of updating the connection weight w is a process of performing calculation by the gradient descent method shown in (Expression 11) and (Expression 12) described later.

  In step S <b> 17, the processing unit 10 </ b> A determines whether or not a predetermined number of pixels in each area training data has been processed for each of the first training data 74 </ b> R <b> 1, the second training data 74 </ b> R <b> 2, and the third training data 74 </ b> R <b> 3. To do. When the series of processing from step S15 to step S17 is performed for the prescribed number of pixels of each area training data, the deep learning process is terminated. The learning of the neural network does not necessarily have to be performed for all the pixels in each training data, and the processing unit 10A learns by processing a part of the pixels in each training data, that is, a prescribed number of pixels. It can be performed. The prescribed number of pixels may be all the pixels in each area training data.

  When the prescribed number of pixels in each area training data is not processed, the processing unit 10A, as shown in FIG. 4B, the second training data in the first training data 74R1 in step S19. The position of the window is moved in 74R2 or in the third training data 74R3. Thereafter, the processing unit 10A performs a series of processes from step S14 to step S16 at the new window position after the movement. That is, in step S14, the processing unit 10A sets each region size training data 75R1 of the window size for each of the first training data 74R1, the second training data 74R2, and the third training data 74R3 at the new window position after the movement. , 75R2 and 75R3, and label values indicating the respective regions are attached. Subsequently, in step S15, the processing unit 10A causes the neural network 50 to learn using the area training data 75R1, 75R2, and 75R3 of the newly cut window size. If learning results for a predetermined number of trials are accumulated in step S16, the processing unit 10A updates the connection weight w of the neural network 50 in step S17. Such learning of the neural network 50 for each window size is performed on a prescribed number of pixels for each of the first training data 74R1, the second training data 74R2, and the third training data 74R3.

  The above-described deep learning process in steps S10 to S19 is repeated for the first training image 70R1, the second training image 70R2, and the third training image 70R3 that are separately acquired. Thus, the degree of learning of the neural network 50 is improved. Thereby, the deep learning algorithm 60 having the neural network structure shown in FIG. 5 is obtained.

-Neural Network Structure As shown in FIG. 12A, in the first embodiment, a deep learning type neural network is used. As in the neural network 50 shown in FIG. 12, the deep learning type neural network includes an input layer 50a, an output layer 50b, and an intermediate layer 50c between the input layer 50a and the output layer 50b. It consists of multiple layers. The number of layers constituting the intermediate layer 50c can be, for example, five or more.

In the neural network 50, a plurality of nodes 89 arranged in layers are coupled between layers. As a result, information propagates from the input side layer 50a to the output side layer 50b only in one direction indicated by an arrow D in the figure. In the present embodiment, the number of nodes of the input layer 50a corresponds to the product of the number of pixels of the input image, that is, the number of pixels of the window W1 shown in FIG. 3C and the number of primary colors included in each pixel. ing. Since the pixel data (color density value) of the image can be input to the input layer 50a, the user can input the input image to the input layer 50a without separately calculating the feature amount from the input image.
・ Operations at each node

  FIG. 12B is a schematic diagram showing the calculation at each node. Each node 89 receives a plurality of inputs and calculates one output (z). In the example shown in FIG. 12B, the node 89 receives four inputs. The total input (u) received by the node 89 is expressed by the following (Equation 1).

  Each input is given a different weight. In (Expression 1), b is a value called a bias. The output (z) of the node is an output of a predetermined function f with respect to the total input (u) represented by (Expression 1), and is expressed by the following (Expression 2). The function f is called an activation function.

FIG. 12C is a schematic diagram illustrating computation between nodes. In the neural network 50, nodes that output the result (z) represented by (Expression 2) are arranged in layers with respect to the total input (u) represented by (Expression 1). The output of the node of the previous layer becomes the input of the node of the next layer. In the example shown in FIG. 12C, the output of the node 89a in the left layer in the drawing becomes the input of the node 89b in the right layer in the drawing. Each node 89b in the right layer receives the output from the node 89a in the left layer. Each connection between each node 89a in the left layer and each node 89b in the right layer is given a different weight. When the respective outputs of the plurality of nodes 89a on the left side of the layers and _x 1 ~x _4, inputs to each of the three nodes 89b of the right side layer, the following equation (3-1) to (Equation 3-3) It is represented by

  When these (Formula 3-1) to (Formula 3-3) are generalized, (Formula 3-4) is obtained. Here, i = 1,... I, j = 1,.

Applying (Equation 3-4) to the activation function gives an output. The output is expressed by the following (Equation 4).

-Activation function In the image analysis method according to the embodiment, a normalized linear unit function is used as the activation function. The normalized linear function is expressed by the following (Formula 5).

  (Expression 5) is a function that sets u = 0 for a portion of u <0 in a linear function of z = u. In the example shown in FIG. 12C, the output of the node with j = 1 is expressed by the following equation using (Equation 5).

-Learning of a neural network If a function expressed using a neural network is set to y (x: w), the function y (x: w) will change if the parameter w of a neural network is changed. Adjusting the function y (x: w) so that the neural network selects a more suitable parameter w for the input x is called neural network learning. Assume that a plurality of pairs of input and output of a function expressed using a neural network are given. If a desired output for a certain input x is d, an input / output set is given as {(x ₁ , d ₁ ), (x ₂ , d ₂ ),..., (X _n , d _n )}. A set of each group represented by (x, d) is called training data. Specifically, a set of a set of color density values for each pixel and a label of a true value image in a single color image of each of R, G, and B shown in FIG. 3B is shown in FIG. It is the training data shown.

Neural network learning means that for any input / output pair (x _n , d _n ), the output y (x _n : w) of the neural network when the input x _n is given is the output d _n. This means that the weight w is adjusted so as to be as close as possible. The error function is the proximity between the function expressed using a neural network and the training data.

It is a scale to measure. The error function is also called a loss function. The error function E (w) used in the image analysis method according to the embodiment is expressed by the following (formula 6). (Equation 6) is called cross entropy.

The calculation method of the cross entropy of (Formula 6) is demonstrated. In the output layer 50b of the neural network 50 used in the image analysis method according to the embodiment, that is, in the final layer of the neural network, an activation function for classifying the input x into a finite number of classes according to the contents is used. The activation function is called a softmax function and is expressed by the following (Equation 7). It is assumed that the same number of nodes as the number of classes k are arranged in the output layer 50b. It is assumed that the total input u of each node k (k = 1,... K) of the output layer L is given by u _k ^(L) from the output of the previous layer L-1. Thereby, the output of the k-th node of the output layer is expressed by the following (formula 7).

(Equation 7) is a softmax function. The sum of the outputs y ₁ ,..., Y _K determined by (Equation 7) is always 1.

If each class is represented as C ₁ ,..., C _K , the output y _k (ie, u _k ^(L) ) of the node k in the output layer L represents the probability that the given input x belongs to the class C _k. . See (Equation 8) below. The input x is classified into a class having the maximum probability represented by (Equation 8).

  In neural network learning, the function represented by the neural network is regarded as a model of the posterior probability of each class, and the likelihood (likelihood) of the weight w for the training data under such a probability model. Evaluate and select a weight w that maximizes the likelihood.

The target output d _n by softmax function (Equation 7), the output is only 1 if a class of correct, the output is to be a 0 otherwise. When the target output is expressed in a vector format of d _n = [d _n1 ,..., D _nK ], for example, when the correct class of the input x _n is C ₃ , only the target output d _n3 becomes 1, and the others The target output is zero. When encoded in this way, the posterior distribution (posterior) is expressed by the following (Equation 9).

The likelihood L (w) of the weight w for the training data {(x _n , d _n )} (n = 1,..., N) is expressed by the following (Equation 10). When the logarithm of the likelihood L (w) is taken and the sign is inverted, the error function of (Expression 6) is derived.

  Learning means that the error function E (w) calculated based on the training data is minimized with respect to the parameter w of the neural network. In the image analysis method according to the embodiment, the error function E (w) is expressed by (Expression 6).

Minimizing the error function E (w) with respect to the parameter w is equivalent to finding the local minimum of the function E (w). The parameter w is a connection weight between nodes. The minimum point of the weight w is obtained by iterative calculation in which the parameter w is repeatedly updated with an arbitrary initial value as a starting point. An example of such a calculation is the gradient descent method.
In the gradient descent method, a vector represented by the following (formula 11) is used.

In the gradient descent method, the process of moving the value of the current parameter w in the negative gradient direction (that is, −∇E) is repeated many times. When the current weight is w ^(t) and the weight after movement is w ^{(t + 1)} , the calculation by the gradient descent method is expressed by the following (Equation 12). The value t means the number of times the parameter w has been moved.

symbol
Is a constant that determines the amount of update of the parameter w and is called a learning coefficient. By repeating the calculation represented by (Expression 12), the error function E (w ^(t) ) decreases as the value t increases, and the parameter w reaches the minimum point.

  The calculation according to (Equation 12) may be performed on all training data (n = 1,..., N), or may be performed only on a part of the training data. The gradient descent method performed only on a part of the training data is called stochastic gradient descent. In the image analysis method according to the embodiment, the stochastic gradient descent method is used.

Image Analysis Processing Referring to FIG. 13, the processing unit 20A of the image analysis apparatus 200A according to the first embodiment includes an analysis data generation unit 201, an analysis data input unit 202, an analysis unit 203, and a region detection. Unit 204. These functional blocks are realized by installing a program that causes a computer according to the present invention to execute an image analysis process in the storage unit 23 or the memory 22 of the processing unit 20A and executing the program by the CPU 21. The window size database 104 and the algorithm database 105 are provided from the deep learning device 100A through the storage medium 98 or the network 99, and are stored in the storage unit 23 or the memory 22 of the processing unit 20A.

  Assume that the analysis target image 78 of the analysis target tissue is captured in advance by the imaging device 400 and stored in advance in the storage unit 23 or the memory 22 of the processing unit 20A. The deep learning algorithm 60 including the learned connection weight w is stored in the algorithm database 105 in association with the tissue type from which the tissue sample to be analyzed is derived, and is a program that causes the computer to execute image analysis processing. Functions as part of a program module. In other words, the deep learning algorithm 60 is used in a computer having a CPU and a memory, and executes calculation or processing of specific information according to the purpose of use, such as outputting data indicating each region in the tissue to be analyzed. Make the computer work. Specifically, the CPU 21 of the processing unit 20A performs the calculation of the neural network 60 based on the learned connection weight w according to the algorithm defined in the deep learning algorithm 60 stored in the storage unit 23 or the memory 22. The CPU 21 of the processing unit 20A performs calculation on the analysis target image 78 obtained by imaging the analysis target tissue and is input to the input layer 60a, and is data indicating each region in the analysis target tissue from the output layer 60b. A ternary image 83 is output.

  Referring to FIG. 14, the processing unit 20A of the image analysis apparatus 200A performs the process shown in FIG. If it demonstrates using each functional block shown in FIG. 13, the data generation part 201 for an analysis will perform the process of step S21 and S22. The processing of steps S23, S24, S26 and S27 is performed by the analysis data input unit 202. The processing of steps S25 and S28 is performed by the analysis unit 203. The region detection unit 204 performs the process in step S29.

  In step S21, the processing unit 20A, for example, in response to a processing start command input from the input unit 26 by the user or the like, from the input analysis target image 78, color density encoding diagrams 79r, 79g, 79g, 79g, 79b is generated. Data to be analyzed is generated by combining color density values of R, G, and B colors for each pixel from the color density encoding diagrams 79r, 79g, and 79b. The generation method of the color density encoding diagrams 79r, 79g, and 79b is the same as the generation method in step S10 in the deep learning process shown in FIG.

  In step S22 illustrated in FIG. 14, the processing unit 20A accepts an input of a tissue type from the user on the image analysis apparatus 200A side as an analysis condition through the input unit 26. The processing unit 20A refers to the window size database 104 and the algorithm database 105 based on the input tissue type, sets the window size used for the analysis, and acquires the deep learning algorithm 60 used for the analysis. The window size is a unit of analysis data input to the neural network 60 at the time of one input, and the product of the number of pixels of the window size analysis data 80 and the number of primary colors included in the image is the input layer. This corresponds to the number of nodes 60a. The window size is associated with the organization type and stored in advance in the window size database 104. The window size is, for example, 3 × 3 pixels like the window W2 shown in FIG. The deep learning algorithm 60 is also stored in advance in the algorithm database 105 shown in FIG. 13 in association with the type of organization.

  In step S23 shown in FIG. 14, the processing unit 20A generates analysis target data combining color density values of R, G, and B colors for each pixel from the color density encoding diagrams 79r, 79g, and 79b, and further displays the window size. The analysis data 80 is generated.

  In step S <b> 24, the processing unit 20 </ b> A inputs the analysis data 80 shown in FIG. 5 to the deep learning algorithm 60. The initial position of the window is, for example, a pixel located at 3 × 3 pixels in the window corresponding to the upper left corner of the analysis target data, as in step S14 during the deep learning process. When the processing unit 20A inputs the data 81 of the total 27 color density values of 3 × 3 pixels × 3 primary colors included in the window size analysis data 80 to the input layer 60a, the deep learning algorithm 60 The discrimination result 82 is output to 60b.

  In step S25 illustrated in FIG. 14, the processing unit 20A stores the determination result 82 output to the output layer 60b illustrated in FIG. The discrimination result 82 is an estimated value (any one of three values 0, 1, and 2) for each pixel region of the analysis target data.

  In step S <b> 26 shown in FIG. 14, the processing unit 20 </ b> A determines whether all the pixels in the input image have been processed. The input image is the color density encoding diagram 79r, 79g, 79b shown in FIG. 6, and when a series of processing from step S23 to step S25 shown in FIG. The process of step S28 is performed.

  When all the pixels in the analysis target data have not been processed, the processing unit 20A, in step S27, similarly to step S20 during the deep learning process, the color density encoding diagrams 79r, 79g, and 79b shown in FIG. The window W2 is moved in units of a predetermined number of pixels. Thereafter, the processing unit 20A performs a series of processing from step S23 to step S25 at the position of the new window W2 after movement. In step S25, the processing unit 20A stores the determination result 82 corresponding to the new window position after the movement. By storing the determination result 82 for each window size for all the pixels in the analysis target image, an analysis result ternary image 83 is obtained. The image size of the ternary image 83 as the analysis result is the same as the image size of the analysis target image. Here, the ternary image 83 may be numerical data in which the estimated value 2, 1, and 0 are assigned to each pixel, and instead of the estimated value 2, 2, 1 and 0. For example, an image indicated by display colors associated with value 2, value 1, and value 0 may be used.

  In step S <b> 28 illustrated in FIG. 14, the processing unit 20 </ b> A outputs the ternary image 83 of the analysis result to the output unit 27.

  In step S29, following step S28, the processing unit 20A further performs a region detection process on the ternary image 83 as the analysis result. In the ternary image 83, the tumor area, the non-tumor area, and the non-tissue area are represented by ternary values.

  Although optional, the processing unit 20A creates the region-enhanced image 84 by superimposing the obtained regions on the analysis target image 78 to be analyzed. The processing unit 20A outputs the created region emphasized image 84 to the output unit 27, and ends the image analysis process.

  Although optional, the processing unit 20A may calculate how much a tumor region is included in the tissue included in the analysis target image 78. For example, when the number of pixels corresponding to the tissue included in the analysis target image 78, that is, the total number of pixels determined as a tumor region and a non-tumor region (total number of pixels in the tissue region) is 100%, it is determined as a tumor region. By calculating how many percent of the number of pixels is occupied, the content rate of the tumor region can be calculated. In addition, by subtracting the number of pixels determined to be a tumor area from the total number of pixels or assuming that the total number of pixels in a tissue area is 100%, the percentage of pixels determined to be a non-tumor area By calculating, the content rate of the non-tumor region can be calculated. And you may calculate the ratio of the content rate of a tumor area | region, and the content rate of a non-tumor area | region. The calculated value may be output to the output unit 27. In addition, the calculated value may be output to the output unit 27 together with the region emphasized image 84.

  The calculation of the content of the tumor region in this way is useful for screening whether a tissue used for genetic analysis or the like of a cancer tissue contains a tumor tissue suitable for the examination.

  As described above, the user of the image analysis device 200A can acquire the ternary image 83 as the analysis result by inputting the analysis target image 78 of the tissue to be analyzed to the image analysis device 200A. The ternary image 83 represents a tumor region, a non-tumor region, and a non-tissue region in a sample to be analyzed, and the user can discriminate each region in the sample to be analyzed.

  Furthermore, the user of the image analysis device 200A can acquire the region emphasized image 84 as the analysis result. The region-enhanced image 84 is generated by, for example, painting each region with a color on the analysis target image 78 to be analyzed. Moreover, in another aspect, it is produced | generated by overlapping the boundary line of each area | region. As a result, the user can grasp each region at a glance in the analysis target organization.

  Displaying a tumor region and / or a non-tumor region in a specimen to be analyzed helps to increase the efficiency of specimen observation by screening the tumor area in the tissue to be analyzed prior to tissue diagnosis.

<Second Embodiment>
Hereinafter, the image analysis system according to the second embodiment will be described with respect to differences from the image analysis system according to the first embodiment.

[Configuration overview]
Referring to FIG. 15, the image analysis system according to the second embodiment includes a user device 200, and the user device 200 operates as an integrated image analysis device 200B. The image analysis apparatus 200B is configured by a general-purpose computer, for example, and performs both the deep learning process and the image analysis process described in the first embodiment. That is, the image analysis system according to the second embodiment is a stand-alone system that performs deep learning and image analysis on the user side. In the image analysis system according to the second embodiment, the integrated image analysis device 200B installed on the user side functions as both the deep learning device 100A and the image analysis device 200A according to the first embodiment. However, it differs from the image analysis system according to the first embodiment.

  The image analysis device 200B is connected to the imaging device 400. The imaging device 400 acquires a training image of a learning tissue during the deep learning process, and acquires an analysis target image 78 of the analysis target tissue during the image analysis processing.

[Hardware configuration]
The hardware configuration of the image analysis device 200B is the same as the hardware configuration of the user side device 200 shown in FIG.

[Function blocks and processing procedures]
Referring to FIG. 16, the processing unit 20B of the image analysis apparatus 200B according to the second embodiment includes a training data generation unit 101, a training data input unit 102, an algorithm update unit 103, an analysis data generation unit 201, and the like. , An analysis data input unit 202, an analysis unit 203, and a region detection unit 204. These functional blocks are realized by installing a program for causing a computer to execute a deep learning process and an image analysis process in the storage unit 23 or the memory 22 of the processing unit 20B and executing the program by the CPU 21. The window size database 104 and the algorithm database 105 are stored in the storage unit 23 or the memory 22 of the processing unit 20B, and both are commonly used during deep learning and image analysis processing. The learned neural network 60 is stored in advance in the algorithm database 105 in association with the type of tissue or the type of sample including cells, and the connection weight w is updated by the deep learning process, and the deep learning algorithm 60 is updated. Is stored in the algorithm database 105. Note that the first training image 70R1, the second training image 70R2, and the third training image 70R3, which are training images, are captured in advance by the imaging device 400 and stored in the storage unit 23 or the memory 22 of the processing unit 20B. In advance. The analysis target image 78 of the sample to be analyzed is also captured in advance by the imaging device 400 and is stored in advance in the storage unit 23 or the memory 22 of the processing unit 20B.

  The processing unit 20B of the image analysis device 200B performs the process illustrated in FIG. 11 during the deep learning process, and performs the process illustrated in FIG. 14 during the image analysis process. If it demonstrates using each functional block shown in FIG. 16, the training data generation part 101 will perform the process of step S10 to S12, S14, S18, and S19 at the time of a deep learning process. The training data input unit 102 performs the process of step S13. The algorithm update unit 103 performs the processes in steps S16 and S17. During the image analysis process, the analysis data generation unit 201 performs the processes of steps S21 and S22. The processing of steps S23, S24, S26 and S27 is performed by the analysis data input unit 202. The processing of steps S25 and S28 is performed by the analysis unit 203. The region detection unit 204 performs the process in step S29.

  The procedures of the deep learning process and the image analysis process performed by the image analysis device 200B according to the second embodiment are the same as the procedures performed by the deep learning device 100A and the image analysis device 200A according to the first embodiment, respectively. . The image analysis device 200B according to the second embodiment is different from the deep learning device 100A and the image analysis device 200A according to the first embodiment in the following points.

  In step S <b> 13 during the deep learning process, the processing unit 20 </ b> B receives an input of the type of organization for learning from the user of the image analysis device 200 </ b> B through the input unit 26. The processing unit 20B refers to the window size database 104 based on the input tissue type, sets the window size, and refers to the algorithm database 105 to set the neural network 50 used for learning.

  As described above, the user of the image analysis device 200B can acquire the ternary image 83 as the analysis result by inputting the analysis target image 78 to the image analysis device 200B. Furthermore, the user of the image analysis device 200B can acquire the region emphasized image 84 as the analysis result.

  According to the image analysis apparatus 200B according to the second embodiment, the user can use the type of tissue selected by the user as a learning organization. This means that learning of the neural network 50 is not left to the vendor, and the user himself can improve the degree of learning of the neural network 50.

<Third Embodiment>
Hereinafter, the image analysis system according to the third embodiment will be described with respect to differences from the image analysis system according to the second embodiment.

[Configuration overview]
Referring to FIG. 17, the image analysis system according to the third embodiment includes a vendor-side device 100 and a user-side device 200. The vendor apparatus 100 operates as an integrated image analysis apparatus 100B, and the user apparatus 200 operates as a terminal apparatus 200C. The image analysis apparatus 100B is configured by, for example, a general-purpose computer, and is an apparatus on the cloud server side that performs both the deep learning process and the image analysis process described in the first embodiment. The terminal device 200C is configured by a general-purpose computer, for example, and transmits an analysis target image to the image analysis device 100B via the network 99, and receives an analysis result image from the image analysis device 100B via the network 99. Terminal device.

  In the image analysis system according to the third embodiment, the integrated image analysis device 100B installed on the vendor side has the functions of both the deep learning device 100A and the image analysis device 200A according to the first embodiment. FIG. 5 is the same as the image analysis system according to the second embodiment. On the other hand, the image analysis system according to the third embodiment includes a terminal device 200C, and provides an input interface for an image to be analyzed and an output interface for an image of an analysis result to the terminal device 200C on the user side. Different from the image analysis system according to the second embodiment. That is, the image analysis system according to the third embodiment is a cloud service type in which the vendor side that performs the deep learning process and the image analysis process provides an input / output interface of the analysis target image and the analysis result image to the user side. System.

  The image analysis apparatus 100 </ b> B is connected to the imaging apparatus 300, and acquires a training image of a learning tissue that is captured by the imaging apparatus 300.

  The terminal device 200 </ b> C is connected to the imaging device 400, and acquires the analysis target image 78 of the analysis target tissue imaged by the imaging device 400.

[Hardware configuration]
The hardware configuration of the image analysis apparatus 100B is the same as the hardware configuration of the vendor-side apparatus 100 shown in FIG. The hardware configuration of the terminal device 200C is the same as the hardware configuration of the user-side device 200 shown in FIG.

[Function blocks and processing procedures]
Referring to FIG. 18, the processing unit 10B of the image analysis apparatus 100B according to the third embodiment includes a training data generation unit 101, a training data input unit 102, an algorithm update unit 103, an analysis data generation unit 201, and the like. , An analysis data input unit 202, an analysis unit 203, and a region detection unit 204. These functional blocks are realized by installing a program for causing a computer to execute a deep learning process and an image analysis process in the storage unit 13 or the memory 12 of the processing unit 10B and executing the program by the CPU 11. The window size database 104 and the algorithm database 105 are stored in the storage unit 13 or the memory 12 of the processing unit 10B, and both are commonly used during deep learning and image analysis processing. The neural network 50 is associated with the type of organization and stored in advance in the algorithm database 105, the connection weight w is updated by the deep learning process, and is stored in the algorithm database 105 as the deep learning algorithm 60.

  Note that the first training image 70R1, the second training image 70R2, and the third training image 70R3, which are training images, are captured in advance by the imaging device 300, and are stored in the storage unit 13 or the memory 12 of the processing unit 10B. In advance. The analysis target image 78 of the analysis target tissue is also captured in advance by the imaging device 400 and is stored in advance in the storage unit 23 or the memory 22 of the processing unit 20C of the terminal device 200C.

  The processing unit 10B of the image analysis apparatus 100B performs the process illustrated in FIG. 11 during the deep learning process, and performs the process illustrated in FIG. 14 during the image analysis process. If it demonstrates using each functional block shown in FIG. 18, the training data generation part 101 will perform the process of step S10 to S12, S14, S18, and S19 at the time of a deep learning process. The training data input unit 102 performs the process of step S13. The algorithm update unit 103 performs the processes in steps S16 and S17. During the image analysis process, the analysis data generation unit 201 performs the processes of steps S21 and S22. The processing of steps S23, S24, S26 and S27 is performed by the analysis data input unit 202. The processing of steps S25 and S28 is performed by the analysis unit 203. The region detection unit 204 performs the process in step S29.

  The procedures of the deep learning process and the image analysis process performed by the image analysis device 100B according to the third embodiment are the same as the procedures performed by the deep learning device 100A and the image analysis device 200A according to the first embodiment, respectively. . The image analysis apparatus 100B according to the third embodiment is different from the deep learning apparatus 100A and the image analysis apparatus 200A according to the first embodiment in the following four points.

  In step S21 at the time of the image analysis processing shown in FIG. 14, the processing unit 10B receives the analysis target image 78 of the analysis target tissue from the terminal device 200C on the user side, and R, G, and R from the received analysis target image 78. Color density encoding diagrams 79r, 79g, and 79b for each color B are generated. The method of generating the color density encoding diagrams 79r, 79g, and 79b and the analysis target data is the same as the generation method in step S10 in the deep learning process shown in FIG.

  In step S22 at the time of the image analysis processing shown in FIG. 14, the processing unit 10B receives an input of the tissue type from the user of the terminal device 200C as an analysis condition through the input unit 26 of the terminal device 200C. The processing unit 10B refers to the window size database 104 and the algorithm database 105 based on the input tissue type, sets the window size used for the analysis, and acquires the deep learning algorithm 60 used for the analysis.

  In step S28 during the image analysis process, the processing unit 10B transmits the ternary image 83 of the analysis result to the terminal device 200C on the user side. In the terminal device 200C on the user side, the processing unit 20C outputs the received three images 83 of the analysis result to the output unit 27.

  In step S29 during the image analysis process, the processing unit 10B further performs a nuclear region detection process on the ternary image 83 as the analysis result, following step S28. As an optional step, the processing unit 10B creates the region-enhanced image 84 by superimposing the obtained regions on the analysis target image 78 to be analyzed. The processing unit 10B transmits the created region emphasized image 84 to the terminal device 200C on the user side. In the user-side terminal device 200C, the processing unit 20C outputs the received region-emphasized image 84 to the output unit 27, and ends the image analysis process.

  As described above, the user of the terminal device 200C can acquire the ternary image 83 as the analysis result by transmitting the analysis target image 78 of the analysis target tissue to the image analysis device 100B. Furthermore, the user of the terminal device 200C can acquire the region emphasized image 84 as an analysis result.

According to the image analysis device 100B according to the third embodiment, the user can enjoy the result of the image analysis processing without acquiring the window size database 104 and the algorithm database 105 from the deep learning device 100A. Thereby, as a service for analyzing a tissue to be analyzed, a service for discriminating a tumor region and / or a non-tumor region and screening a tumor region in each tissue can be provided as a cloud service.
[Computer program]
The embodiment of the present invention includes a computer program and a product thereof that cause the processing units 10A, 20B, and 10B to execute the steps S10 to S20 to generate a learned deep learning algorithm. In addition, in the embodiment of the present invention, there is provided a computer program and a product for causing the processing units 10A, 20B, and 10B to execute the steps S21 to S29 and causing the computer to function to analyze an image of a tissue collected from an individual. included.

[Other forms]
As described above, the present invention has been described with the outline and specific embodiments, but the present invention is not limited to the above outline and each embodiment.

  In the first to third embodiments, in step S13, the processing units 10A, 20B, and 10B refer to the window size database 104 and set the number of pixels of the window size. You may set the size directly. In this case, the window size database 104 becomes unnecessary.

  In the first to third embodiments, in step S13, the processing units 10A, 20B, and 10B set the number of pixels of the window size based on the input tissue type. Instead of the input, the size of the tissue may be input. The processing units 10A, 20B, and 10B may set the number of pixels of the window size with reference to the window size database 104 based on the input tissue size. In step S22, as in step S13, the size of the tissue may be input instead of the input of the tissue type. The processing units 20A, 20B, and 10B may obtain the neural network 60 by setting the number of pixels of the window size with reference to the window size database 104 and the algorithm database 105 based on the input tissue size.

  As for the mode of inputting the size of the organization, the size may be directly input as a numerical value. For example, the user interface of the input is used as a pull-down menu, and a predetermined numerical range corresponding to the size to be input by the user is given to the user. You may select and input.

  Further, in step S13 and step S22, in addition to the type of tissue or the size of the tissue, an imaging magnification when the training image and the analysis target image 78 are captured may be input. As for the mode of inputting the imaging magnification, the magnification may be directly input as a numerical value. For example, the user can select a predetermined numerical range corresponding to the magnification to be input by the user using a pull-down menu of the input user interface. May be entered.

  In the first to third embodiments, the window size is set to 3 × 3 pixels for the convenience of explanation during the deep learning process and the image analysis process. It is not limited. The window size may be set according to the type of organization, for example. In this case, the product of the number of pixels of the window size and the number of primary colors included in the image only needs to correspond to the number of nodes of the input layers 50a and 60a of the neural networks 50 and 60.

  In the first to third embodiments, in step S17, the processing units 10A, 20B, and 10B store the deep learning algorithm 60 in the algorithm database 105 in a one-to-one correspondence with the type of organization. Instead, in step S17, the processing units 10A, 20B, and 10B may store a plurality of tissue types in the algorithm database 105 in association with one deep learning algorithm 60.

  In the first to third embodiments, the hue is defined by a combination of the three primary colors of light or a combination of the three primary colors of colors. However, the number of hues is not limited to three. The number of hues may be four primary colors obtained by adding yellow (Y) to red (R), green (G), and blue (B), or 3 colors of red (R), green (G), and blue (B). Two primary colors obtained by reducing any one hue from the primary colors may be used. Alternatively, only one of the three primary colors of red (R), green (G), and blue (B) (for example, green (G)) may be used. For example, the training images 70R1, 70R2, 70R3 and the analysis target image 78 acquired using a known microscope or virtual slide scanner are also color images of the three primary colors red (R), green (G), and blue (B). However, the present invention is not limited to this, and may be a color image of two primary colors as long as the image includes one or more primary colors.

  In the first to third embodiments, in step S10, the processing units 10A, 20B, and 10B each color coding diagram (color density coding diagram 70R1r, 70R1g, 70R1b, color density coding diagram) of the training image. 70R2r, 70R2g, 70R2b, and color density coding diagrams 70R3r, 70R3g, and 70R3b) are generated as a single color image of 8 bits (256 steps) for each primary color. The gradation is not limited to 8 bits. The gradation of each color density encoding diagram may depend on the imaging conditions of the slide image scanner. For example, it may be an image of 1 bit or more and 2 bits, 4 bits, 16 bits, or 24 bits. Similarly, in step S21, the processing units 20A, 20B, and 10B are generated as single color images of the primary colors of the color density encoded diagrams 79r, 79g, and 79b, but the primary colors used when creating the color density encoded diagrams are shown. The gradation is not limited to three gradations. The primary color when creating a color density coding diagram is not limited to 8 bits. The gradations of the color density encoding diagrams 79r, 79g, and 79b may depend on the imaging conditions of the slide image scanner. For example, it may be an image of 1 bit or more and 2 bits, 4 bits, 16 bits, or 24 bits. It is preferable that the gradations of the color density coding diagrams and the color density coding diagrams 79r, 79g, and 79b of the training image are all the same.

  In the first to third embodiments, in step S10, the processing units 10A, 20B, and 10B generate each color density coding diagram from the training image. An encoded matrix table may be used as a training image. The processing units 10A, 20B, and 10B may directly acquire the training image as each color density encoded diagram, for example, from a virtual slide scanner or the like. Similarly, in step S21, the processing units 20A, 20B, and 10B generate R, G, and B color density encoding diagrams 79r, 79g, and 79b from the analysis target image 78. A matrix table encoded according to the key may be used as the analysis image. That is, the processing units 20A, 20B, and 10B may directly acquire the color density encoding diagrams 79r, 79g, and 79b from, for example, a virtual slide scanner.

In the first to third embodiments, RGB is used as the color space when generating each color density coding diagram from the training image, but the color space is not limited to RGB. In addition to RGB, various color spaces such as YUV, CMY, and CIE L ^* a ^* b ^* can be used.

  In the first to third embodiments, in each area training data and analysis data 80, density values are stored in the order of red (R), green (G), and blue (B) for each pixel. The order of storing and handling density values is not limited to this. For example, the density values may be stored in the order of blue (B), green (G), and red (R), the order of density values in each area training data, and the density value in analysis data 80. The order may be the same.

  In the first to third embodiments, the processing units 10A and 10B are realized as an integrated device, but the processing units 10A and 10B do not have to be an integrated device, and the CPU 11, the memory 12, and the storage unit 13 Etc. may be arranged in different places, and these may be connected via a network. The processing units 10A and 10B, the input unit 16, and the output unit 17 are not necessarily arranged in one place, and may be arranged in different places and connected to each other via a network. The processing units 20A, 20B, and 20C are the same as the processing units 10A and 10B.

  In the first to third embodiments, the training data generation unit 101, the training data input unit 102, the algorithm update unit 103, the analysis data generation unit 201, the analysis data input unit 202, the analysis unit 203, and the region detection unit 204 Each functional block is executed by a single CPU 11 or a single CPU 21, but these functional blocks do not necessarily need to be executed by a single CPU, and are distributed and executed by a plurality of CPUs. Also good. Each of these functional blocks may be distributed and executed by a plurality of GPUs, or may be distributed and executed by a plurality of CPUs and a plurality of GPUs.

  In the second and third embodiments, a program for performing the processing of each step described with reference to FIGS. 11 and 14 is stored in the storage units 13 and 23 in advance. Alternatively, the program may be installed in the processing units 10B and 20B from a computer-readable non-temporary tangible storage medium 98 such as a DVD-ROM or a USB memory. Alternatively, the processing units 10B and 20B may be connected to the network 99, and the program may be downloaded and installed from, for example, an external server (not shown) via the network 99.

  In the first to third embodiments, the input units 16 and 26 are input devices such as a keyboard or a mouse, and the output units 17 and 27 are realized as display devices such as a liquid crystal display. Alternatively, the input units 16 and 26 and the output units 17 and 27 may be integrated to be realized as a touch panel display device. Alternatively, the output units 17 and 27 may be configured by a printer or the like, and the ternary image 83 or the region emphasized image 84 as an analysis result may be printed and output.

  In the first to third embodiments, the imaging device 300 is directly connected to the deep learning device 100A or the image analysis device 100B, but the imaging device 300 is connected to the deep learning device 100A or the image analysis via the network 99. It may be connected to the device 100B. Similarly for the imaging device 400, the imaging device 400 is directly connected to the image analysis device 200A or the image analysis device 200B. However, the imaging device 400 is connected to the image analysis device 200A or the image analysis device 200B via the network 99. It may be connected.

<Verification of learned deep learning algorithm>
The deep learning process and the image analysis process were performed in the stand-alone system shown in the second embodiment. As a tissue to be learned, human stomach cancer tissue including a tumor region and a normal region was used. In creating a deep learning algorithm that has already been trained, change the magnification of the tissue sample to 1x, 3x, 5x, 10x, 20x, and 40x, and take training images. Training data using training images was learned by a neural network, and the accuracy of the analysis results was confirmed.
The details of learning data and analysis data were as follows.
Learning data: 106 hole slide images Analysis data for verification: 45 hole slide images

[Create and learn training data]
A virtual slide scanner (NanoZoomer-XR, (Hamamatsu Photonics; scan resolution: 0.46 μm / pixel at 20x mode scan, 0.23 μm / pixel at 40x mode scan) ) Was used for color imaging. The imaging magnification was 1 ×, 3 ×, 5 ×, 10 ×, 20 ×, and 40 ×. A pathologist designated a tumor area, a non-tumor area, and a non-tissue area on a hole slide image including a tissue to be learned. A hole slide image in which each region was designated was divided under the above conditions to obtain a training image. It was determined for each training image whether it was a tumor area, a non-tumor area or a non-tissue area. At that time, for a tissue region in which a tumor region and a non-tumor region are mixed, out of the number of pixels corresponding to the tissue region in the training image, the number of pixels designated as a tumor region occupies 50% or more. Tumor area. In addition, when the ratio of the number of pixels designated as the non-tumor region out of the number of pixels corresponding to the tissue region in the training image is 100%, the non-tumor region is determined. A training image in which a non-tissue region and a tissue region are mixed is defined as a non-tissue region when the tissue region is 0% of the number of pixels of the training image, and the rest is a tissue region. For each pixel of each training image determined to be a tumor area, a non-tumor area, or a non-tissue area, a label value that distinguishes each area ("1" for a tumor area, "2" for a non-tumor area, Was assigned “0”), and the first preliminary training data 70R1L, the second preliminary training data 70R2L, and the third preliminary training data 70R2L were generated. The training image determined as the tumor region is the first training image, the training image determined as the non-tumor region is the second training image 70R2, and the training image determined as the non-tissue region is the first training image. 3 training images 70R3. For each training image, the color density value of each color of R, G, B is gradationized by 8 bits to generate a color density coding diagram of each color of R, G, B, and the color of each created R, G, B color Each of the first training data 74R1, the second training data 74R2, and the third training data 74R3 is generated by combining the density coding diagrams.

  Thereafter, tumor region training data was created by combining the first training data 74R1 and the first preliminary training data 70R1L. Non-tumor region training data was created by combining the second training data 74R2 and the second preliminary training data 70R2L. The non-tissue region training data was created by combining the third training data 74R3 and the third preliminary training data 70R2L. Each created region training data was divided into a window size of 200 × 200 pixels, and a neural network was trained using the training data having the divided window size as an input layer.

  Each area training data was generated for each magnification of the tissue sample, and a neural network was trained for each magnification.

[Create analysis target image]
Similar to the training data, a hole slide image of a bright field image of HE-stained human gastric cancer tissue was color-imaged using a virtual slide scanner to obtain an image for analysis. The imaging magnification was 1 ×, 3 ×, 5 ×, 10 ×, 20 ×, and 40 ×. After that, based on the captured analysis image, a color density coding diagram for each of the R, G, and B colors is created, and analysis target data is generated by combining the created color density coding diagrams for each of the R, G, and B colors. . The analysis target data was generated for each enlargement magnification.

[Analysis result]
Analysis data having a window size of 200 × 200 pixels was created from the analysis target data, and the analysis data was input to a learned neural network corresponding to the magnification. Based on the analysis result output from the neural network, it is classified into tumor area, non-tumor area, and non-tissue area (background), and the tumor area obtained by the analysis process is displayed in white. Was displayed in gray, and the non-tissue region was displayed in black. The analysis result is shown in FIG.

  In FIG. 19, (a) is a hole slide image of a bright field image obtained by HE-staining gastric cancer tissue and is an image to be analyzed. (B) is an image in which a pathologist designates a tumor region of a hole slide image and is surrounded by a solid line. (C) shows an analysis result using an image for training and an image for analysis having a magnification of 1 ×. Each area designated by the pathologist could not be accurately shown. (D) shows the analysis result using the image for training and the image for analysis of 5 times magnification. The tumor area designated by the pathologist was generally discriminated as the tumor area. In addition, the non-tumor area designated by the pathologist could be generally discriminated as a non-tumor area. The non-tissue area designated by the pathologist could be generally discriminated as a non-tissue area. (E) shows the analysis result using the image for training and the image for analysis of 40 times magnification. The tumor area designated by the pathologist could be identified to some extent as a tumor area. However, in the region designated by the pathologist as a non-tumor region, the number of regions determined to be tumor regions has increased, and the rate of erroneously determining non-tumor regions as tumor regions has increased.

  From the above results, it has been shown that the enlargement magnification when acquiring the training image and the analysis image is too low or too high can cause erroneous determination.

  In FIG. 20, (a) is a hole slide image of a bright-field image obtained by HE staining a gastric cancer tissue different from that in FIG. 19, and is an image to be analyzed. (B) is an image in which a pathologist designates a tumor region of a hole slide image and is surrounded by a solid line. (C) shows an analysis result using an image for training and an image for analysis having a magnification of 5 times. The tumor area designated by the pathologist was generally discriminated as the tumor area. In addition, the non-tumor area designated by the pathologist could be generally discriminated as a non-tumor area. The non-tissue area designated by the pathologist could be generally discriminated as a non-tissue area. (D) shows the analysis result using the image for training and the image for analysis of 20 times magnification. As in the case of an enlargement magnification of 5 times, each region designated by the pathologist could be roughly discriminated.

  For the analysis image with a magnification of 5 times, the content of the tumor region in the tissue region present in one hole slide image was calculated according to the following formula.

  Content ratio (%) of tumor region = {[number of pixels determined to be a tumor region] / [(number of pixels determined to be a tumor region) + (number of pixels determined to be a non-tumor region)]} × 100

  The content of the tumor area was 54.5%.

  FIG. 21 shows sensitivity (a) and positive predictive value (b) at each magnification. From this result, it was considered that 5 to 20 times was appropriate as the magnification of the tissue specimen when acquiring the training image and the analysis image. FIG. 21 shows changes in sensitivity (c) and positive predictive value (d) due to differences in the number of pixels in the window size. The analysis image was taken at a magnification of 5 times. The number of pixels of the window size was considered to be preferably 125 × 125 pixels to 200 × 200 pixels.

  This time, the analysis time for one hole slide image was several minutes to 30 minutes.

10 (10A, 10B) Processing unit 20 (20A, 20B, 20C) Processing unit 11, 21 CPU
12, 22 Memory 13, 23 Recording unit 14, 24 Bus 15, 25 Interface unit 16, 26 Input unit 17, 27 Output unit 19, 29 GPU
50 Neural network (deep learning algorithm)
50a input layer 50b output layer 50c intermediate layer 60 learned neural network (learned deep learning algorithm)
60a Input layer 60b Output layer 60c Intermediate layer 70W1 Hole slide image 70W2 Hole slide image 70W3 Hole slide image 70R1L First preliminary training data 70R2L Second preliminary training data 70R3L Third preliminary training data 72R1r, 72R1g, 72R1b Color density code 72R2r, 72R2g, 72R2b Color density coding diagram 72R3r, 72R3g, 72R3b Color density coding diagram 74R1, 74R2, 74R3 Training data 75R1, 75R2, 75R3 Training data for window size 76 Color density value 77R1, 77R2, 77R3 True value Image label value 78 Bright field images 79r, 79g, 79b to be analyzed Color density encoding chart 80 to be analyzed 80 Analysis data 81 Color density value 82 Discrimination result (pixel estimated value)
83 Three-value image of analysis result 84 Area-enhanced image 89 (89a, 89b) Node 98 Recording medium 99 Network 100 Vendor side device 100A Deep learning device 100B Integrated image analysis device 101 Training data generation unit 102 Training data input unit 103 Algorithm Update unit 104 Window size database 105 Algorithm database 200 User side device 200A Image analysis device 200B Integrated image analysis device 200C Terminal device 201 Analysis data generation unit 202 Analysis data input unit 203 Analysis unit 204 Area detection unit 300, 400 Apparatus 301, 401 Image sensor 308, 408 Sample structure 309, 409 Stage W1 Window W2 Window

Claims (32)

An image analysis method for analyzing an image of a tissue using a deep learning algorithm having a neural network structure,
Generate analysis data from the analysis target image including the analysis target tissue,
The analysis data is input to the deep learning algorithm,
The deep learning algorithm generates data indicating whether the region included in the analysis data is a tumor region,
An image analysis method. Furthermore, by the deep learning algorithm,
Generating data indicating that the region included in the analysis data is a non-tumor region;
The image analysis method according to claim 1. Further, the deep learning algorithm generates data indicating that the region included in the analysis data is a region that does not include tissue.
The image analysis method according to claim 1 or 2. The analysis target image is an image of a tissue diagnostic specimen, and the analysis target image includes a hue obtained by combining two or more primary colors;
The image analysis method according to claim 1. The analysis target image is an image obtained by enlarging the analysis target tissue from 3 times to 20 times,
The image analysis method according to claim 1. The size of the region included in the analysis target image is 200 μm × 200 μm or more and 400 μm × 400 μm or less.
The image analysis method according to claim 1. Based on the data indicating whether the region included in the analysis data is a tumor region, to generate data for distinguishing and presenting the region of the tumor cell and other regions,
The image analysis method according to claim 1. Based on the data indicating whether the region included in the analysis data is a tumor region, to generate data indicating the boundary between the region of the tumor cell and the other region,
The image analysis method according to claim 1. Based on the data indicating whether the region included in the analysis data is a tumor region, to generate data indicating the content of the tumor region in the tissue to be analyzed,
The image analysis method according to claim 1. Based on the data indicating whether or not the region included in the analysis data is a tumor region, to generate data indicating the ratio of the tumor region and the non-tumor region in the tissue to be analyzed,
The image analysis method according to claim 1. Generating a plurality of data for analysis corresponding to each region of a predetermined number of pixels for one analysis target image;
The image analysis method according to claim 1.   The image analysis method according to any one of claims 1 to 11, further comprising: generating data for distinguishing and presenting a tumor region, a non-tumor region, and a region not including a tissue for one analysis target image. . The number of nodes in the input layer of the neural network corresponds to the product of the predetermined number of pixels of the analysis data and the number of combined primary colors.
The image analysis method according to claim 1. The specimen is a stained specimen, and the analysis target image is an image obtained by imaging the stained specimen in a bright field of a microscope.
The image analysis method according to claim 4. The training data used for the learning of the deep learning algorithm is a stained image of a specimen prepared by performing staining for bright field observation on a tissue specimen including a tumor region collected from an individual under a bright field of a microscope. Generated based on the captured bright field image,
The image analysis method according to claim 1. The staining for bright field observation uses hematoxylin for nuclear staining,
The image analysis method according to claim 15. The bright field observation staining is hematoxylin-eosin staining,
The image analysis method according to claim 15. The training data includes a label value determined from the bright field image indicating a tumor region;
The image analysis method according to claim 15. The training data includes the label value for each region of a predetermined number of pixels of the bright field image.
The image analysis method according to claim 18. The training data is generated for each region of a predetermined number of pixels in the bright field image,
The image analysis method according to claim 15. The deep learning algorithm classifies the analysis data into a class indicating that the tissue included in the analysis target image is a tumor region.
The image analysis method according to claim 1. The output layer of the neural network is a node having a softmax function as an activation function.
The image analysis method according to any one of claims 1 to 21. The deep learning algorithm generates data indicating whether or not the predetermined region includes a tumor region for each region of a predetermined number of pixels each time the analysis data is input.
The image analysis method according to any one of claims 1 to 22. The deep learning algorithm is generated according to the type of the tissue;
The image analysis method according to any one of claims 1 to 23. Furthermore, the analysis data is processed using the deep learning algorithm corresponding to the type of tissue to be analyzed, selected from among the plurality of deep learning algorithms according to the type of the tissue,
The image analysis method according to claim 24. An image analysis device that analyzes an image of a tissue using a deep learning algorithm having a neural network structure,
Generate analysis data from the analysis target image including the analysis target tissue,
The analysis data is input to the deep learning algorithm,
A processing unit that generates data indicating whether or not the region included in the analysis data is a tumor region by the deep learning algorithm,
An image analysis apparatus comprising: A computer program for analyzing an image of a tissue using a neural network structure deep learning algorithm,
On the computer,
Processing for generating analysis data from the analysis target image including the tissue to be analyzed;
Processing for inputting the analysis data to the deep learning algorithm;
Processing to generate data indicating whether or not the region included in the analysis data is a tumor region by the deep learning algorithm;
A program that executes Generating training data used for learning of a deep learning algorithm based on a training image acquired from a tissue sample for training; and causing the neural network to learn the training data,
The generating step includes
A first acquisition step of acquiring first training data corresponding to a first training image obtained by imaging a tumor region;
A second acquisition step of acquiring second training data corresponding to a second training image obtained by imaging a non-tumor region;
A third acquisition step of acquiring third training data corresponding to a third training image obtained by imaging a region not including a tissue;
Including
The learning step includes
A first learning step for causing the neural network to learn that the first training data is a tumor region;
A second learning step for causing the neural network to learn that the second training data is a non-tumor region;
A third learning step for causing the neural network to learn that the third training data is a region not including tissue;
including,
How to generate a trained deep learning algorithm. The first training data, the second training data, and the third training data are used as an input layer of a neural network, and are a tumor region, a non-tumor region, and a region not including a tissue. A neural network output layer corresponding to the first training data, the second training data, and the third training data, respectively.
The method for generating a learned deep learning algorithm according to claim 28. Generating the first training data from the first training image before the first obtaining step;
Generating the second training data from the second training image before the second obtaining step;
Generating the third training data from the third training image before the third obtaining step;
Further including
30. A method of manufacturing the learned deep learning algorithm of claim 28 or 29. The training image is a bright-field image obtained by capturing a stained image of a specimen prepared by performing bright-field observation staining on a tissue collected from an individual under a bright field of a microscope.
A method for generating a learned deep learning algorithm according to any one of claims 28 to 30. The first training data, the second training data, and the third training data are input layers of the neural network, and are a tumor region, a non-tumor region, and a region that does not include tissue. A deep learning algorithm that learns as an output layer of a neural network corresponding to the first training data, the second training data, and the third training data, respectively,
The first training data is generated from a first training image obtained by imaging a tumor region of training tissue;
The second training data is generated from a second training image obtained by imaging a non-tumor region of training tissue;
The third training data is generated from a third training image obtained by imaging a region not including tissue.
Trained deep learning algorithm.