JP7458519B2 - Image analysis method, image analysis device, program, method for producing trained deep learning algorithm, and trained deep learning algorithm - Google Patents

Description

This specification discloses an image analysis method, an image analysis device, a program, a method for manufacturing a trained deep learning algorithm, and a trained deep learning algorithm. More specifically, an image analysis method, an image analysis device, a program, and a learned deep layer include generating data indicating whether or not a region included in analysis data is a tumor region for an image of a tissue to be analyzed. A method for manufacturing a learning algorithm, a trained deep learning algorithm, and the like are disclosed.

Patent Document 1 discloses an image diagnosis support device that classifies and judges tissue images in a pathological tissue image into four groups: normal, benign tumor, precancerous state, and cancerous state. The image classification means extracts a region of interest from image data, calculates feature values that indicate the characteristics of the region of interest, and classifies the group based on the calculated feature values. The feature values include the density of clumps per unit area in the cell nucleus, the density of the clump area, the clump area, the clump thickness, and the clump length. The image judgment means learns the relationship between such feature values and the judgment result, and makes a judgment based on the learned learning parameters. The learning is performed using a learning algorithm such as a support vector machine,
Perform machine learning.

JP 2010-203949 A

When definitively diagnosing whether or not it is a malignant tumor, histopathological diagnosis using a histopathological specimen is performed. Furthermore, histopathological diagnosis is often performed as a quick intraoperative diagnosis to determine the site of resection of tissue containing a malignant tumor during surgery. Rapid intraoperative diagnosis involves having the patient stand by while an incision is made in the affected area during surgery, and determining whether the tumor is malignant, whether there is any tumor remaining in the resected tissue margin, and whether there is lymph node metastasis. This is done through histopathological diagnosis. The results of the intraoperative rapid diagnosis will determine the direction of subsequent surgery for the waiting patient.

Histopathological diagnosis is performed by a doctor, especially a pathologist, by observing tissue specimens using a microscope, etc.
In order to be able to make accurate definitive diagnoses through observation of tissue specimens, it is necessary to repeatedly observe tissue specimens from various cases under the supervision of a skilled pathologist over a long period of time, which requires a huge amount of training for pathologists. It takes a lot of time.

There is a serious shortage of pathologists, and as a result of the shortage, there are concerns that a definitive diagnosis of a patient's malignant tumor will be delayed, and the start of treatment will be delayed, or that treatment may be started without waiting for a definitive diagnosis. There is. In addition, because both normal tissue diagnosis and intraoperative rapid diagnosis are concentrated on a small number of pathologists, the workload of each pathologist becomes enormous, and the working conditions of the pathologists themselves are also becoming a problem. However, no solution to this problem has been found so far.

Therefore, it is thought that the ability of a device to support pathological tissue diagnosis will greatly contribute to resolving the shortage of pathologists and improving the working conditions of pathologists, especially as the diagnosis is closer to judgment made by the human eye.

In terms of the device supporting pathological tissue diagnosis, the invention described in Patent Document 1 described above performs pathological determination of sample tissue based on image analysis using machine learning. This method requires the feature values to be created manually. The problem with methods of manually creating feature amounts is that the ability of the person greatly affects the performance of image analysis. Further, the invention described in Patent Document 1 determines whether each cell nucleus in an image of a tissue specimen strongly magnified with a microscope is a nucleus of a cancer cell. Therefore, there is a problem in that it takes a long time to analyze an entire specimen.

Therefore, before a pathologist or machine learning algorithm determines whether an individual cell is the nucleus of a cancer cell, it is necessary to observe the entire tissue contained in a single specimen and determine which part of the specimen contains tumor tissue. Screening for the presence of such substances helps reduce the time required for pathological diagnosis of a single specimen.

The present invention relates to an image analysis method, an image analysis device, a program, a method for manufacturing a trained deep learning algorithm, and a learning method for generating data indicating whether or not a region included in analysis data of a tissue to be analyzed is a tumor region. The goal is to provide advanced deep learning algorithms.

One embodiment relates to an image analysis method for analyzing an image of tissue using a deep learning algorithm (60) having a neural network structure. The image analysis method includes generating analysis data (80) from an analysis target image (78) including a tissue to be analyzed, and
) is input to a deep learning algorithm (60), and the deep learning algorithm (60) generates data indicating whether or not a region included in the analysis data is a tumor region. According to the present invention, data indicating whether or not a region included in the analysis data in the tissue to be analyzed is a tumor region can be generated.

In the embodiment, preferably, the data indicating whether the area included in the analysis data is a tumor area is data indicating the area of the tumor area. This embodiment allows the user to grasp the tumor region in the tissue to be analyzed at a glance.

In the embodiment, preferably, the image to be analyzed is an image of a specimen for tissue diagnosis, and includes a hue that is a combination of two or more primary colors. This embodiment can assist in pathological diagnosis.

In the above embodiment, the analysis target image is preferably an image of the tissue to be analyzed magnified by 3 to 20 times. In the above embodiment, the size of the area included in the analysis target image is 200 μm×200 μm or more and 400 μm×400 μm or less. These embodiments allow for more accurate determination.

In the embodiment, preferably, data for distinguishing and presenting tumor cell regions and other regions is generated based on data indicating whether or not a region included in the analysis data is a tumor region. do. This embodiment allows the user to grasp the tumor region in the tissue to be analyzed at a glance.

In the embodiment, preferably, data indicating a boundary between a tumor cell area and other areas is generated based on data indicating whether the area included in the analysis data is a tumor area. This embodiment allows the user to grasp at a glance the boundary between the tumor region and non-tumor region in the tissue to be analyzed.

In the embodiment, preferably, data indicating the content rate of tumor regions in the tissue to be analyzed is generated based on data indicating whether a region included in the analysis data is a tumor region, allowing a user to grasp the content rate of tumor regions in the tissue to be analyzed.

In the embodiment, preferably, data indicating a ratio of a tumor area to a non-tumor area in the tissue to be analyzed is generated based on data indicating whether a region included in the analysis data is a tumor area. . This embodiment allows the user to grasp the content rate of tumor regions in the tissue to be analyzed. In addition, it is possible to know whether samples used for genetic testing, etc. are being collected appropriately.

According to the above-mentioned embodiment, data for distinguishing and presenting a tumor region, a non-tumor region, and a region not containing tissue is preferably generated for one analysis target image. This embodiment allows the user to grasp at a glance the presence or absence of a tumor region in the tissue to be analyzed.

In the embodiment, preferably, a plurality of pieces of analysis data are generated for each region of a predetermined number of pixels for one image to be analyzed. According to this embodiment, it is possible to accurately analyze a wide range of tissues to be analyzed.

In the embodiment, preferably, the deep learning algorithm (60) applies a label to the input analysis data indicating that the region of the predetermined number of pixels includes at least one selected from a tumor region and a non-tumor region. generate.

In the embodiment, preferably the input layer (50) of the neural network (50)
The number of nodes in a) corresponds to the product of the predetermined number of pixels of the analysis data (80) and the number of combined primary colors. This embodiment allows highly accurate analysis.

In the embodiment, preferably, the tissue diagnosis specimen is a stained specimen, and the analysis target image is an image of the stained specimen taken under a bright field of a microscope. This embodiment can assist in pathological diagnosis.

In the embodiment, preferably, the training data used for learning the deep learning algorithm is staining of a specimen prepared by applying bright field observation staining to a tissue specimen including a tumor region collected from an individual. The image is generated based on a bright field image taken under the bright field of a microscope. This embodiment can aid in tissue diagnosis.

In the embodiment, preferably, the staining for bright field observation uses hematoxylin for nuclear staining. More preferably, the staining for bright field observation is hematoxylin and eosin staining (HE staining). This embodiment makes it possible to cover highly versatile tests for tissue diagnosis.

In the embodiment, preferably the training data includes a label value determined from the bright field image indicating a tumor region. More preferably, the training data includes the label value for each area of a predetermined number of pixels of the bright field image. This embodiment allows the generation of training data for tumor regions.

In the embodiment, preferably, the training data is generated for each region of a predetermined number of pixels in the bright field image. This aspect makes it possible to increase the learning efficiency of the deep learning algorithm (50).

In the embodiment, preferably, the deep learning algorithm classifies the analysis data into a class indicating that the tissue included in the image to be analyzed is a tumor region.

In the embodiment, preferably, the output layer (
50b) is a node that uses a softmax function as an activation function.
It is possible to improve learning efficiency and analysis accuracy.

In the embodiment, preferably, the deep learning algorithm uses analysis data (7
Every time 8) is input, data indicating that the tissue included in the image to be analyzed is a tumor region is generated for each region of a predetermined number of pixels. This embodiment allows for increased analysis efficiency.

In said embodiment, preferably a deep learning algorithm (60) is generated depending on the type of tissue. Furthermore, preferably, the analysis data is processed using the deep learning algorithm corresponding to the type of tissue to be analyzed, which is selected from the plurality of deep learning algorithms according to the type of tissue. . This embodiment can improve analysis accuracy.

One embodiment relates to an image analysis device (100) that analyzes tissue images using a deep learning algorithm (60) with a neural network structure. The analysis device generates analysis data (80) from an analysis target image (78) including a tissue to be analyzed, and
80) into a deep learning algorithm (50), and the deep learning algorithm (50) generates data indicating whether the region included in the analysis data is a tumor region or not. Be prepared.

Certain embodiments relate to a computer program that analyzes images of tissue using a deep learning algorithm (60) in a neural network structure. The computer includes a process of generating analysis data (80) from an analysis target image (78) including a tissue to be analyzed, a process of inputting the analysis data (80) into a deep learning algorithm (60), and a process of inputting the analysis data (80) into a deep learning algorithm (60). The learning algorithm (60) executes a process of generating data indicating whether or not the region included in the analysis data is a tumor region.

An embodiment includes the step of generating training data used for learning the deep learning algorithm (50) based on a training image obtained from a training tissue sample, and causing the neural network (50) to learn the training data. a learning step, and the generation step includes a first acquisition step of acquiring first training data (74R1) corresponding to a first training image (70R1) that captures a tumor region; A second acquisition step of acquiring second training data (74R2) corresponding to a second training image (70R2) that captures a region, and a third training image (70R3) that captures a region that does not include tissue. ), the learning step includes: a third acquisition step of acquiring third training data (74R3) corresponding to
The first training data (74R1) is a tumor region using a neural network (
50), a second learning step in which the neural network (50) learns that the second training data (74R2) is a non-tumor region, and the third training step. The present invention relates to a method for generating a trained deep learning algorithm, including a third learning step of causing a neural network (50) to learn that the data (74R3) is a region that does not include tissue. Preferably, the first training data (74R1) and the second
The training data (74R2) and the third training data (74R3) are used as the input layer (50a) of the neural network (50), and the training data (74R2) and the third training data (74R3) are used as the input layer (50a) of the neural network (50).
The first training data (74R1) and the second training data indicate that the region does not contain tissue, respectively.
The output layer (50b) of the neural network (50) corresponds to the training data (74R2) and the third training data (74R3). More preferably, before the first acquisition step, the first training data (74R1) is extracted from the first training image (70R1).
), and before the second acquisition step, the second training image (70
R2), the second training data (74R2) is generated from the third training image (70R3), and before the third acquisition step, the third training data (74R2) is generated from the third training image (70R3).
R3). According to this embodiment, the tissue to be analyzed is
A trained deep learning algorithm can be generated that determines whether the tumor region includes at least one selected from a tumor region and a non-tumor region.

In the generation method, preferably, the training image is a bright field image obtained by capturing a stained image of a specimen prepared by performing bright field observation staining on a tissue collected from an individual under the bright field of a microscope. It is.

Some embodiments provide first training data (74R1), second training data (74R2), and third training data (74R3) to the input layer (50a) of the neural network (50).
), and the first training data (74R1) and second training data (74R2) respectively indicate that it is a tumor region, that it is a non-tumor region, and that it is a region that does not contain tissue. A deep learning algorithm trained as an output layer (50b) of a neural network (50) corresponding to third training data (74R3),
4R1) is generated from a first training image (70R1) that captures a tumor region of the training tissue, and the second training data (74R2) is generated from a first training image (74R2) that captures a non-tumor region of the training tissue. 2
The third training data (74R3) is generated from a trained deep learning algorithm (70R2) that is generated from a third training image (70R3) that captures a region that does not include tissue. 60).

It is possible to observe the entire tissue contained in a specimen and generate data for screening which parts of the specimen contain tumor tissue.

FIG. 2 is a schematic diagram for explaining an overview of a deep learning method. FIG. 2 is a schematic diagram for explaining an overview of a deep learning method. FIG. 2 is a schematic diagram for explaining an overview of a deep learning method. FIG. 3 is a schematic diagram for explaining details of training data. FIG. 3 is a schematic diagram for explaining details of training data. FIG. 2 is a schematic diagram for explaining an overview of an image analysis method. FIG. 1 is a schematic configuration diagram of an image analysis system according to a first embodiment. 1 is a block diagram showing the hardware configuration of a vendor-side device 100. FIG. 2 is a block diagram showing the hardware configuration of a user-side device 200. FIG. FIG. 2 is a block diagram for explaining the functions of a deep learning device 100A according to the first embodiment. It is a flowchart which shows the procedure of deep learning processing. FIG. 2 is a schematic diagram for explaining details of learning by a neural network. FIG. 2 is a block diagram for explaining the functions of an image analysis device 200A according to the first embodiment. It is a flowchart which shows the procedure of image analysis processing. FIG. 2 is a schematic configuration diagram of an image analysis system according to a second embodiment. FIG. 2 is a block diagram for explaining the functions of an integrated image analysis device 200B according to a second embodiment. FIG. 2 is a schematic configuration diagram of an image analysis system according to a third embodiment. FIG. 13 is a block diagram for explaining functions of an integrated image analyzing device 100B according to a third embodiment. 19 shows the analysis results of a tissue specimen obtained from a stomach cancer tissue. (a) is a whole slide image of a bright field image obtained by HE staining of stomach cancer tissue, and is an image to be analyzed. (b) is an image in which a pathologist specifies a tumor area in the whole slide image and surrounds it with a solid line. (c) shows the analysis results using a training image and an analysis image with a magnification of 1x. (d) shows the analysis results using a training image and an analysis image with a magnification of 5x. (e) shows the analysis results using a training image and an analysis image with a magnification of 40x. In Fig. 19(c) to (e), the tumor area obtained by the analysis process is displayed in white, the non-tumor area is displayed in gray, and the non-tissue area is displayed in black. 19 shows the analysis results of a tissue specimen obtained from a stomach cancer tissue. (a) is a whole slide image of a bright field image obtained by HE staining of stomach cancer tissue, and is the image to be analyzed. (b) is an image in which a pathologist specifies a tumor area in the whole slide image and surrounds it with a solid line. (c) shows the analysis results using a training image and an analysis image with a magnification of 5 times. (d) shows the analysis results using a training image and an analysis image with a magnification of 16 times. In Fig. 19(c) and (d), the tumor area obtained by the analysis process is displayed in white, the non-tumor area is displayed in gray, and the non-tissue area is displayed in black. FIG. 7 is a diagram showing the difference in discrimination accuracy depending on the magnification of the tissue sample and the number of pixels of the window size. (a) shows the sensitivity of each magnification, and (b) shows the positive predictive value of each magnification. (c) shows the sensitivity due to the difference in the number of pixels of the window size, and (d) shows the change in the positive predictive value.

DESCRIPTION OF THE PREFERRED EMBODIMENTS An outline and embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that in the following description and drawings, the same reference numerals indicate the same or similar components, and therefore, explanations regarding the same or similar components will be omitted.

The image analysis method analyzes images of tissues. The image analysis method uses a deep learning algorithm of a neural network structure, preferably a convolutional neural network structure. The image analysis method includes generating data indicating whether a region included in the analysis data is a tumor region.

In the present invention, the image of a tissue or cell is an image obtained from a specimen of a tissue sample or a specimen of a sample containing cells. The specimen of a tissue sample or a specimen of a sample containing cells is taken from an individual. The individual is not particularly limited, but is preferably a mammal, more preferably a human. When the sample is taken from the individual, it does not matter whether the individual is alive or dead. The tissue is not limited as long as it is present within the individual. Examples of the tissue taken from the individual include surgically resected tissue, biopsy tissue, etc. The tumor may be either epithelial or non-epithelial. The tumor is preferably a malignant epithelial tumor. The malignant tumor is not particularly limited, but examples of the malignant tumor include respiratory system malignant tumors arising from the trachea, bronchi, or lungs, etc.; nasopharynx, esophagus, stomach, duodenum, jejunum, ileum,
Examples of the solid tumor include malignant tumors of the digestive tract that occur in the cecum, appendix, ascending colon, transverse colon, sigmoid colon, rectum, anus, etc.; liver cancer; pancreatic cancer; malignant tumors of the urinary system that occur in the bladder, ureter, or kidney; malignant tumors of the female reproductive system that occur in the ovaries, fallopian tubes, and uterus; breast cancer; prostate cancer; skin cancer; malignant tumors of the endocrine system such as the hypothalamus, pituitary gland, thyroid gland, parathyroid gland, and adrenal gland; malignant tumors of the central nervous system; and malignant tumors that occur in bone and soft tissue. More preferred are malignant tumors of the respiratory tract epithelium such as lung cancer (squamous cell carcinoma, small cell carcinoma, large cell carcinoma, and adenocarcinoma); malignant tumors of the digestive tract epithelium such as gastric cancer, duodenal cancer, and large intestine cancer (sigmoid colon cancer, rectal cancer, etc.); liver cancer; pancreatic cancer; bladder cancer; thyroid cancer; ovarian cancer; breast cancer; and prostate cancer. Most preferred is gastric cancer.

The specimen is intended to be a state in which the tissue is processed so that it can be observed under a microscope or the like, for example, a slide. The specimen can be prepared according to a known method. For example, in the case of a tissue specimen, after collecting tissue from the individual, the tissue is fixed with a specified fixative (formalin fixation, etc.), the fixed tissue is embedded in paraffin, and the paraffin-embedded tissue is thin-sliced. The thin slice is placed on a slide glass. The slide glass on which the slice is placed is stained for observation under an optical microscope, i.e., for bright-field observation, and a specified mounting process is performed to complete the specimen. A typical example of a tissue specimen is a tissue diagnostic specimen (pathological specimen), and the stain is hematoxylin-eosin (HE) staining.

The nuclear stain for HE staining is hematoxylin. Hematoxylin is widely used as a nuclear stain in tissue cell staining (eg, immunostaining, lectin staining, sugar staining, fat staining, collagen fiber staining, etc.). Therefore, the present invention can be applied to all kinds of specimens in which hematoxylin is used for nuclear staining.

The image analysis uses a deep learning algorithm trained using a training image. The image analysis generates analysis data from an analysis target image including a tissue to be analyzed, acquired from the specimen. The analysis data is input to the deep learning algorithm, and data is generated indicating whether a region of the tissue present in the analysis target image included in the analysis data is a tumor region. The data indicating whether a region included in the analysis data is a tumor region is data indicating the possibility that a region including a plurality of pixels of interest included in the tissue to be discriminated includes tumor tissue and/or is a non-tumor region. The data indicating whether a region included in the analysis data is a tumor region may be a label value, display, or the like that can distinguish whether the region includes a tumor region from a non-tumor region.

The training images are acquired from a plurality of regions of a sample to be studied. Preferably, the training images are acquired from a region including a tumor region, a region including a non-tumor region, and a region not including tissue for each layer structure from a plurality of layer structures included in the tissue to be learned.

For example, the first training image 70R1 is an image acquired from a region including a tumor region included in tissue collected from an individual. This image is obtained by microscopic observation from a specimen that has been stained so that its tissue structure can be recognized. The staining is not limited as long as the tissue structure can be recognized, but staining for bright field observation is preferable. The staining for bright field observation is not limited as long as at least the cell nucleus and a region other than the cell nucleus can be stained in a distinguishable manner by hue. When the specimen is a mammalian tissue specimen, for example, HE staining can be used.

First training data 74R1 and first preliminary training data 70R1L are generated from the first training image 70R1. The first training data 74R1 is information regarding a single color image in which the hues included in the first training image 70R1 are separated for each primary color. The first preliminary training data 70R1L is generated as binarized data (label value) indicating that the tissue region included in the first training image 70R1 includes a tumor region. Whether or not the tissue region included in the first training image 70R1 includes a tumor region is determined by, for example, specimen observation by a pathologist or the like. Here, in the tissue region included in the training image, tumor cells, non-tumor cells, and regions of tissue other than cells may coexist. In this case, the tissue region included in the training image is such that, for example, when the area of the entire tissue region included in the training image is 100%, tumor cells occupy approximately 10% or more of the total area; About 20% or more, about 30% or more, about 40% or more, about 50% or more, about 60% or more, about 70% or more, about 80% or more, or about 90% or more
% or more, it can be determined that a tumor area is included. Preferably about 40%
If the area occupies about 50% or more, about 60% or more, about 70% or more, about 80% or more, or about 90% or more, it can be determined that a tumor area is included. The ratio of the tumor cells to the total area of the tissue is calculated by subtracting the ratio of non-tumor cells and tissues other than cells (non-tumor area) from the area (100%) of the entire tissue region included in the training image. It may be determined as

The second training image 70R2 included in the training images is an image acquired from a tissue region including a non-tumor region (preferably a region including non-tumor cells and a region of tissue other than cells). . This image is obtained by microscopic observation from a specimen that has been stained so that its tissue structure can be recognized. The staining is not limited as long as the tissue structure can be recognized, but staining for bright field observation is preferable. The staining for bright field observation is not limited as long as at least the cell nucleus and a region other than the cell nucleus can be stained in a distinguishable manner by hue. When the specimen is a mammalian tissue specimen, for example, HE staining can be used. The staining is performed on the first training image 70R.
It is preferable that the specimen be the same as the specimen from which No. 1 was obtained.

Second training data 74R2 and second preliminary training data 70R2L are generated from the second training image 70R2. The second training data 74R2 is information regarding a single color image in which the hues included in the second training image 70R2 are separated for each primary color. The second preliminary training data 70R2L is binarized data (label value) indicating that the tissue region included in the second training image 70R2 includes a non-tumor region. Whether or not the tissue region included in the second training image 70R2 includes a non-tumor region is determined by, for example, specimen observation by a pathologist or the like. Here, in the tissue region included in the training image, tumor cells, non-tumor cells,
Areas of tissue other than cells may be present. In this case, the area of tissue included in the training image is defined as non-tumor cells and areas of tissue other than cells (non-tumor area) is about 10% or more, about 20% or more, about 30% or more, about 40% or more, about 50% or more, about 60% or more, about 70% of the entire tissue area.
If the area occupies about 80% or more, about 90% or more, or 100%, it can be determined that a non-tumor area is included. Preferably, it is determined that a non-tumor region is included if it occupies about 40% or more, about 50% or more, about 60% or more, about 70% or more, about 80% or more, about 90% or more, or 100%. Can be done. The area of the entire tissue including the non-tumor cells and non-cell tissue is calculated by subtracting the proportion occupied by the tumor region from the area (100%) of the entire tissue region included in the second training image 70R2. It may also be determined as a percentage.

The third training image 70R3 included in the training images is the first training image 70R1,
and/or a region (
This is an image obtained from a non-tissue area (also called a non-tissue area).

From the third training image, third training data 74R3 and third preliminary training data 70R3 are obtained.
L is generated. The third training data 74R3 is information regarding a single color image in which the hues included in the third training image 70R3 are separated for each primary color. Third preliminary training data 70R2L
is binarized data (label value) indicating that the region included in the third training image 70R3 is a region that does not include tissue. The area included in the third training image 70R3 is determined by, for example, specimen observation by a pathologist or the like. Here, the training image may include a tissue region and a background region (for example, a glass portion of a slide). In this case, the background area included in the training image is about 90% or more, about 91%, for example, when the area of the entire area included in the one training image is 100%. or more, about 92% or more, about 93% or more, about 94% or more, about 95% or more, about 96% or more, about 97% or more
% or more, approximately 98% or more, or approximately 99% or more, or the background area is 100% or more
%, it can be determined that the area does not contain tissue. Preferably, the background area is about 95% or more, about 96% or more, about 97% or more, about 98% or more, or about 99%
If the background area occupies 100% or more, or if the background area occupies 100%, it can be determined that the area does not contain tissue.

It is preferable that each training image is obtained as an image of a training tissue specimen enlarged, for example, from 5 times to 20 times. From the viewpoint of discrimination accuracy, images enlarged 30 times or more, particularly 40 times or more, are not suitable as training images.

Tumor region training data is generated from the first training data 74R1 and the first preliminary training data 70R1L, non-tumor region training data is generated from the second training data 74R2 and the second preliminary training data 70R2L, and the first training data 74R1 and the first preliminary training data 70R1L generate tumor region training data. Non-organized region training data is generated from the third training data 74R3 and the third preliminary training data 70R2L. Tumor region training data, non-tumor region training data, and non-tissue region training data may be collectively referred to as training data.

In the overview and embodiments of the present invention, an example is described in which a deep learning algorithm is used to determine whether the tissue contained in an image of an HE-stained tissue specimen includes tumorous regions, non-tumorous regions, and non-tissue regions.

[Overview of deep learning methods and image analysis methods]
First, we will explain the deep learning method.

- Outline of deep learning method The outline of the first training data 74R1, the first preliminary training data 70R1L, and the tumor region training data 75R1 will be explained using FIG. FIG. 1 shows an example of inputting training data to a neural network using a whole slide image 70W1 of a tissue specimen obtained by a slide image scanner. The whole slide image 70W1 is a slide image of a tissue specimen obtained by imaging a specimen prepared by applying HE staining as a bright field observation stain in a bright field. The whole slide image 70W2 was created by a pathologist.
This is an image in which the pathologist specified the portion including the tumor region in No. 1 by surrounding it with a solid line. The whole slide image 70W3 is an image obtained by dividing the whole slide image 70W2 as a training image (for example, into 512 parts). An area surrounded by a rectangular frame indicated by the symbol R1 in the hole slide image 70W3 indicates an area used as the first training image 70R1 shown in FIG. 1. The area surrounded by a rectangular frame indicated by symbol R2 is the second training image 70 shown in FIG.
The area used as R2 is shown. The area surrounded by a rectangular frame indicated by symbol R3 indicates an area used as the third training image 70R3 shown in FIG. 3. The fact that the first training image 70R1 includes a tumor region, that the second training image 70R2 includes a non-tumor region, and that the third training image 70R3 is a region that does not include tissue means that each The determination may be made before acquiring the training images, or after acquiring each training image.

In FIG. 1, the first training image 70R1 is obtained by capturing an HE-stained specimen as a color image under bright-field observation, and therefore includes a plurality of hues.

The first training image 70R1 can be acquired in advance using an image acquisition device such as a known optical microscope, a fluorescent microscope, or a virtual slide scanner.
In this embodiment, the color image captured by the image capture device is in the RGB color space.
In the 24-bit color of RGB, it is preferable that the saturation (color density) of each of the red, green, and blue colors is expressed by 8-bit (256 levels) gradation. The first training image 70R1 may be an image including one or more primary colors. First training data 74R1 is generated corresponding to the first training image 70R1.

In the present invention, hue is illustratively defined by a combination of three primary colors of light or a combination of three primary colors of color. The first training data 74R1 is generated from the first training image 70R1 by separating the hue appearing in the first training image 70R1 into individual primary colors, and generating a code for each primary color according to the density. This is the data expressed as . In Figure 1, red (R
), green (G), and blue (B), images (hereinafter also referred to as "single color images") 72R1R, 72R1G, and 72R1B are obtained, which are shown in shading of a single color separated for each primary color.

The color density of each color is encoded for each pixel on the single color images 72R1R, 72R1G, and 72R1B, and the entire image is converted into an encoding diagram (hereinafter referred to as " 72R1r, 72R1g, and 72R1b are generated. The color density may be encoded as a numerical value indicating 256 levels for each color. In addition, the color density is 25 for each color.
The numerical values indicating six levels may be further subjected to preprocessing, and the color density at each pixel may be encoded as a number indicating eight levels from value 0 to value 7, for example. For convenience of explanation, the color density encoding diagrams 72R1r, 72R1g, and 72R1b generated from single-color images of R, G, and B shown as an example in FIG. It is expressed by a stage code. First training data 74R1 in which R, G, and B color density values are combined for each pixel is generated from the color density encoding diagrams 72R1r, 72R1g, and 72R1b shown in FIG. The code indicating color density is also referred to as color density value in this specification. Further, as the first training data 74R1, a matrix of color density values corresponding to each pixel may be generated instead of the color density encoded diagram.

In FIG. 1, the first preliminary training data 70R1L is data indicating that the tissue included in the first training image 70R1 includes a tumor region. For example, the first training image 7
In 0R1, since the entire region is determined to be a tumor region by a pathologist, the first training image 70R1 includes a label value indicating that a tumor region is included in each region of a predetermined number of pixels, which will be described later. The same numerical value, for example, "1" is assigned as the first preliminary training data 70R1L.

In the deep learning method, tumor region training data 75R1 is generated from the first training data 74R1 and the first preliminary training data 70R1L shown in FIG.
The neural network 50 is trained using the color density value data 76 of each pixel of 5R1 (obtained from the first training data 74R1) as the input layer 50a and the label value 77R1 (obtained from the first preliminary training data 70R1L) as the output layer 50b.

Next, the second training data 74R2, the second preliminary training data 70R2L, and the non-tumor region training data 75R2 will be outlined with reference to FIG. 2. In FIG. 2, the hole slide image 70W1, the hole slide image 70W2, and the hole slide image 70
Since W3 is the same as in FIG. 1, the second training image 70R2 has the same width as the first training image 70R
Like 1, it contains multiple hues.

The second training image 70R2 is obtained in the same manner as the first training image 70R1, except that tissue including a non-tumor region is used instead of tissue including a tumor region. Second training image 70
Second training data 74R2 is generated corresponding to R2.

The second training data 74R2 is generated in the same manner as the first training data 74R1, except that tissue including a non-tumor region is used instead of tissue including a tumor region. Second training data 74R2 in which R, G, and B color density values are combined for each pixel is generated from the color density encoding diagrams 72R2r, 72R2g, and 72R2b shown in FIG.

In FIG. 2, the second preliminary training data 70R2L is data indicating that the second training image 70R2 includes a non-tumor region. Since the entire region of the second training image 70R2 is determined to be a non-tumor region by the pathologist, the second training image 70R2 includes non-tumor regions for each region of a predetermined number of pixels, which will be described later. The same numerical value, for example, "2" is assigned as a label value indicating that the area is included, and the data becomes second preliminary training data 70R2L. A label value indicating a region including a non-tumor region is not limited as long as it can be distinguished from a label value indicating a region including a tumor region and a region not including tissue.

In the deep learning method, non-tumor region training data 75R2 is generated from second training data 74R2 and second preliminary training data 70R2L shown in FIG. Then, the color density value data 76 (obtained from the second training data 74R2) of each pixel of the non-tumor region training data 75R2 is used as the input layer 50a, and the label value 77R2 (obtained from the second preliminary training data 70R2L)
The neural network 50 with the output layer 50b is made to learn.

Next, an outline of the third training data 74R3, the third preliminary training data 70R3L, and the non-organized region training data 75R3 will be explained using FIG. In FIG. 3, a hole slide image 70W1, a hole slide image 70W2, and a hole slide image 70
Since W3 is the same as in FIG. 1, the third training image 70R3 includes the first training image 70R.
Similar to 1, multiple hues are included.

The third training image 70R3 is acquired in a manner similar to that of the first training image 70R1, except that a tissue-free region is substituted for the region containing the tumor region.
Corresponding to R3, a third training data 74R3 is generated.

The third training data 74R3 is generated in the same manner as the first training data 74R1, except that a tissue-free region is used instead of a tumor-containing region. The third training data 74R3 is generated by combining R, G, and B color density values for each pixel from the color density coding diagrams 72R3r, 72R3g, and 72R3b shown in FIG.

In FIG. 3, the third preliminary training data 70R3L is data indicating that the region included in the third training image 70R3 is a region that does not include tissue. Since the entire region of the third training image 70R3 has been determined by the pathologist to be a non-tissue region, the third training image 70R3 includes tissues in each region of a predetermined number of pixels, which will be described later. The same numerical value, for example, "0" is attached as a label value indicating that the region does not include the third preliminary training data 70R.
It becomes 3L. A label value indicating a region that does not contain tissue is distinguished from a numerical value indicating a region containing a tumor region and a region containing a non-tumor region.

In the deep learning method, non-organized region training data 75R3 is generated from the third training data 74R3 and third preliminary training data 70R3L shown in FIG. (obtained from the third training data 74R3) as the input layer 50a, and the label value 77R3 (obtained from the third preliminary training data 70R3L) as the output layer 50b.

A method of generating the tumor region training data 75R1 will be described with reference to FIGS. 4(a) and 4(b).
The first training data 74R1 shown in FIG. 4A is data that combines the color density values of the color density encoded diagrams 72R1r, 72R1g, and 72R1b of a tumor region with a predetermined number of pixels. In this specification, for convenience, the position of each pixel in the first training data 74R1 is expressed by l ₁ , l _{2 .} . . l _i in columns from the left, and m ₁ , _{m 2} . . First training data 74
In FIG. 4A, the image size (size per training data) of R1 is simplified for convenience of explanation, and the first training data 74R1 has 9 pixels in the vertical direction and 9 pixels in the horizontal direction. It has a total of 81 pixels, including 9 pixels in the direction. In other words, the position of each pixel is from the left in columns l ₁ , l ₂ .
・l ₉ , the rows are represented by m ₁ , m ₂ . . . m ₉ from the top. The three values shown in FIG. 4A are the color density values of R, G, and B colors in each pixel. Illustratively, the three values are, from the left, red (R)
, green (G) and blue (B). The color density value of each pixel of the first training data 74R1 is shown in eight levels from value 0 to value 7 for convenience. This is an example of image pre-processing.
, 72R1B into eight levels of color density values. As for the color density value, for example, the lowest brightness (a gradation group with a low brightness value when expressed in 256 RGB color levels) is set to a color density value of 0, and gradually higher values are assigned as the degree of brightness increases. Then, the highest brightness (gradation group with a high brightness value when expressed in 256 RGB color levels) is set to a color density value of 7.

The tumor region training data 75R1 shown in FIGS. 1 and 4(b) is based on the first training data 74R1 shown in FIG. A label value 77R1 corresponding to the label value of the first preliminary training data 70R1L is attached to the cut-out data. The window-sized tumor region training data 75R1 is also shown simplified to 3×3 pixels (l ₁ :l ₃ , m ₁ :m ₃ ) for convenience of explanation, but the position of each pixel is shown with the column on the left for convenience. From l ₁ , l ₂ ... l _i , the rows are represented by m ₁ , m ₂ ... m _j from the top. The actual preferred window size is illustratively 125 x 125 pixels (i=125,
j = 125), 150 x 150 pixels (i = 150, j = 150), 166 x 166 pixels (
i=166, j=166), 180×180 pixels (i=180, j=180), 200×
200 pixels (i=200, j=200), 220×220 pixels (i=220, j=220
), about 250×250 pixels (i=250, j=250) is preferable from the point of view of learning efficiency. More preferably, the window size is about 200×200 pixels. The window size is, for example, in a range of 60 μm×60 μm or more and 500 μm×500 μm or less. Preferably, 60 μm x 60 μm, 100 μm x 100 μm, 150 μm x 150 μm, 20
0μm×200μm, 250μm×250μm, 300μm×300μm, 350μm×
350μm, 400μm×400μm, 450μm×450μm, 500μm×500μm
It is m. More preferably, the size is 200 μm×200 μm to 400 μm×400 μm.
For example, as shown in FIG. 4(b), a 3×3 pixel window W1 (l ₁ :l ₃ , m ₁ :m ₃ ) is set from the left end of the first training data 74R1, and the first training data The window W1 is moved from left to right with respect to 74R1. Window W1 is (l ₁ :l ₃ , m ₁ :m ₃ )
After cutting out the tumor region training data 75R1, a label value 77R1 corresponding to the label value of the first preliminary training data 70R1L is attached to the tumor region training data 75R1. Window W1
is the next window of tumor region training data 74R1 indicated by a dotted line (l ₄ :l ₆ , m ₄ :m ₆ )
, and set (l ₄ :l ₆ ,m ₄ :m ₆ ) to the tumor region training data of window size 75R.
Cut out as 1. A label value 77R1 corresponding to the label value of the first preliminary training data 70R1L is attached to the newly extracted tumor region training data 75R1. Repeat this operation,
Tumor region training data 75R1 of multiple window sizes are cut out from the first training data 74R1. The tumor region training data 75R1 of the cut out window size is given a label value 77R1 corresponding to the label value of the first preliminary training data 70R1L, and is used for learning of the neural network 50 shown in FIG. 1.

As shown in FIG. 1, the number of nodes in the input layer 50a of the neural network 50 is determined by the number of pixels of the tumor region training data 75R1 of the input window size and the number of primary colors included in the image (for example, in the case of the three primary colors of light, R, G, and B). The color density value data 76 of each pixel of the window-sized tumor region training data 75R1 is used as the input layer 50a of the neural network, and the label value 77R1 corresponding to the first preliminary training data 70R1L is
The neural network 50 is made to learn by using it as the output layer 50b of the neural network. The color density value data 76 of each pixel includes R, G,
B is a collection of color density values for each color. As an example, if the window size tumor region training data 75R1 is 3 x 3 pixels, one color density value is given for each R, G, and B for each pixel, so the color density value data 76 is The number of color density values is "27" (3 x 3 x 3 = 2
7), and the number of nodes in the input layer 50a of the neural network 50 is also "27".

In this way, the window-sized tumor region training data 75R1 to be input to the neural network 50 can be automatically created by the computer without being created by the user. This facilitates efficient deep learning of the neural network 50.

As shown in FIG. 4(b), in the initial state, the window W1 contains the first training data 7.
It is located in the upper left corner of 4R1. Thereafter, the tumor region training data 75R1 of the window size is cut out using the window W1, and the position of the window W1 is moved every time the neural network 50 is trained. Specifically, the window W1 is the first training data 74.
The window W1 is moved so as to scan, for example, all pixels of R1. This results in
Tumor region training data 75R1 of a window size cut out from all pixels of the first training data 74R1 is used for learning of the neural network 50. Therefore, the degree of learning of the neural network 50 can be improved, and as a result of deep learning, a deep learning algorithm having the structure of the neural network 60 shown in FIG. 6 can be obtained.

FIG. 5(a) shows second training data 74R2. The method of generating non-tumor region training data 75R2 from the second training data 74R2 is to replace the first training data 74R1 and the first preliminary training data 70R1L with the second training data 74R2 and the second preliminary training data. This is similar to the tumor region training data 75R1 except that 70R2L is used. A label value 77R2 corresponding to the label value of the second preliminary training data 70R2L attached to the non-tumor region training data 75R2 is input to the output layer 50b shown in FIG. 2.

FIG. 5(b) shows third training data 74R3. The method of generating the non-organized region training data 75R3 from the third training data 74R3 is to replace the first training data 74R1 and the first preliminary training data 70R1L with the third training data 74R3 and the third preliminary training data. This is similar to the tumor region training data 75R1 except that 70R3L is used. A label value 77R3 corresponding to the label value of the third preliminary training data 70R3L attached to the non-organized region training data 75R3 is input to the output layer 50b shown in FIG. 3.

It is preferable that the training data for each region is generated using 10 or more, 20 or more, 50 or more, or 100 or more training images for each region.

- Outline of image analysis method As shown in FIG. 6, in the image analysis method, analysis data 80 is generated from an analysis target image (bright field image) 78 obtained by capturing a specimen including a tissue to be analyzed. Preferably, the specimen is stained with the same staining as the first training image. Analysis target image 78
It is also possible to obtain, for example, a color image using, for example, a known microscope or virtual slide scanner. It is preferable that the analysis target image 78 is acquired at the same degree or the same magnification as the training image. The image to be analyzed (bright field image) 78 may be any image containing one or more primary colors. When the color analysis target image 78 is encoded using the color density values of R, G, and B for each pixel, the entire image can be represented as an encoded diagram of the color density values of each pixel for each R, G, and B. (Analytical color density encoding diagrams 79r, 79g, 79b). Analysis target data (not shown) in which R, G, and B color density values are combined for each pixel is generated from the analytical color density coding diagrams 79r, 79g, and 79b. The color density coding diagrams 79r, 79g, and 79b showing the signs of color density in the single color images of each of R, G, and B, exemplarily shown in FIG. Color density values are displayed using codes in eight stages from value 0 to value 7. In this specification, for convenience, the position of each pixel of the data to be analyzed is represented by columns l ₁ , l ₂ . . . l _i from the left, and rows m ₁ , _{m 2} .

The analysis data 80 is data obtained by cutting out the analysis target data into a region with a predetermined number of pixels (that is, the same window size as the training data for each region described above), and the color density value of the tissue included in the analysis target image 78. This is data that includes. In FIG. 6, window size analysis data 80 is also simplified to 3×3 pixels for convenience of explanation, similar to tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3. As shown, the actual preferred window size is similar to tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3. For example, a 3×3 pixel window W2 is set and the window W2 is moved relative to the data to be analyzed. The window W2 includes a predetermined number of pixels in the data to be analyzed, and if the data to be analyzed is cut out using the window W2 indicated by a black frame of, for example, 3 x 3 pixels in the same manner as the training data for each region, the window size is Analysis data 80 is obtained. In the same way as each region training data,
A plurality of analysis data 80 are generated from the analysis target data for each region of a predetermined number of pixels corresponding to the window size. In the analysis target data and window size analysis data 80, the color density value of each pixel is red (R), similarly to the tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3. They are stored in the order of green (G) and blue (B).

In the image analysis method, the tumor region training data 75R of the window size shown in Figs.
1. Analysis data 80 is processed using deep learning algorithm 60 having a neural network trained using non-tumor region training data 75R2 and non-tissue region training data 75R3. By processing analysis data 80, data 83 is generated that indicates whether a region included in the analysis data in the tissue to be analyzed is a tumor region.

6 again, analysis data 80 extracted from the analysis target data generated based on the color density coding maps 79r, 79g, and 79b for each of the R, G, and B colors is input to a neural network 60 constituting a deep learning algorithm. The number of nodes in the input layer 60a of the neural network 60 corresponds to the product of the number of input pixels and the number of primary colors contained in the image. When color density value data 81 of each pixel of the analysis data 80 is input to the neural network 60, an estimated value 82 (three values) of the analysis data 80 is output from the output layer 60b. For example, if the estimated value is 1, then the area (pixel area) of a predetermined number of pixels constituting the analysis data 80 is 1.
indicates that the pixel region includes a tumor region, an estimated value of 2 indicates that the pixel region includes a non-tumor region, and an estimated value of 0 indicates that the pixel region is a non-tissue region. That is, the estimated value 82 output from the output layer 60b of the neural network 60 is a label value generated for each pixel region of the image to be analyzed, and is data indicating whether or not a region included in the analysis data in the image to be analyzed is a tumor region. The estimated value 82 is also referred to as a class in the description of neural networks to be given later. The neural network 60
For the input analysis data 80, a label value indicating the inclusion of a tumorous region, a label value indicating the inclusion of a non-tumorous region, or a label value indicating a non-tissue region is generated based on color density value data 81 of a predetermined number of pixels contained in the analysis data 80. In other words, the neural network 60 classifies the analysis data 80 into classes indicating the state of the pixel region of interest contained in the image to be analyzed (tumorous, non-tumorous, or non-tissue).
Here, the color density value data 81 of each pixel is a set of data of the color density values of each of the R, G, and B colors of each pixel of the analysis data 80 .

Thereafter, the window W2 scans all pixels of the data to be analyzed.
2 in units of a predetermined number of pixels, the analysis data 80 is cut out in a window size. The extracted analysis data 80 is input to the neural network 60. As a result, a label value 83 is obtained based on data indicating whether the region included in the tissue analysis data in the image to be analyzed is a tumor region. In the example shown in FIG. 6, by further performing region detection processing on the label value 83, a region-enhanced image 84 indicating the region is obtained. Specifically, the area detection process is a process of detecting each area according to the estimated value 82, for example, and actually identifies a pixel area where the estimated value 82 is "1" as a tumor area, and a pixel area where the estimated value 82 is "2". This is a process for determining a pixel region as a non-tumor region and a pixel region for which the estimated value 82 is 0 as a non-tissue region. Area emphasis image 8
4 is a diagram showing regions detected by image analysis processing using label values 83 and colors (white for tumor regions, gray for non-tumor regions, and black for non-tissue regions). Further, after each region is discriminated, processing may be performed to display the tumor region and other regions (namely, non-tumor region and/or non-tissue region) on the display device in a distinguishable manner. For example, processing such as filling the tumor region with color or drawing lines between the tumor region and other regions is performed, and these are displayed on the display device in a distinguishable manner. Further, a process may be performed to display the non-tumor region and/or the non-tissue region in a distinguishable manner on the display device.

<First embodiment>
In the first embodiment, the configuration of a system that implements the deep learning method and image analysis method described in the above overview will be specifically described.
[Configuration overview]

Referring to FIG. 7, the image analysis system according to the first embodiment includes a deep learning device 100A.
and an image analysis device 200A. The vendor device 100 operates as a deep learning device 100A, and the user device 200 operates as an image analysis device 200A. Deep learning device 1
00A causes the neural network 50 to learn using training data, and provides the user with a deep learning algorithm 60 trained using the training data. The deep learning algorithm composed of the trained neural network 60 is provided from the deep learning device 100A to the image analysis device 200A via the storage medium 98 or the network 99. The image analysis device 200A analyzes the image to be analyzed using a deep learning algorithm configured by the trained neural network 60.

The deep learning device 100A is composed of, for example, a general-purpose computer, and performs deep learning processing based on a flowchart described later. The image analysis device 200A is composed of, for example, a general-purpose computer, and performs image analysis processing based on a flowchart described later. The storage medium 98 is a computer-readable non-transitory tangible storage medium such as a DVD-ROM or a USB memory.

The deep learning device 100A is connected to the imaging device 300. The imaging device 300 includes an imaging element 301 and a fluorescence microscope 302, and captures a bright field image and a fluorescence image of a learning specimen 308 set on a stage 309. The learning specimen 308 is stained as described above. The deep learning device 100A acquires training images captured by the imaging device 300.

Image analysis device 200A is connected to imaging device 400. The imaging device 400 includes an imaging device 401 and a fluorescence microscope 402, and captures a bright field image of a specimen 408 to be analyzed set on a stage 409. The specimen 408 to be analyzed has been stained in advance as described above. The image analysis device 200A analyzes an analysis target image 78 captured by the imaging device 400.
get.

As the imaging devices 300 and 400, a known fluorescence microscope, virtual slide scanner, or the like, which has a function of imaging a specimen, can be used. The imaging device 400 may be an optical microscope as long as it has the function of imaging a specimen.

[Hardware configuration]
Referring to FIG. 8, the vendor device 100 (100A, 100B) has a processing unit 10 (100B).
A, 10B), an input section 16, and an output section 17.

The processing unit 10 includes a CPU (Central Processing Unit) 11 that performs data processing, which will be described later, a memory 12 that is used as a work area for data processing, and a storage unit 13 that stores programs and processing data that will be described later. It includes a bus 14 for transmitting data, an interface section 15 for inputting and outputting data to and from external devices, and a GPU (Graphics Processing Unit) 19. The input section 16 and the output section 17 are connected to the processing section 10. Illustratively, the input unit 16 is an input device such as a keyboard or a mouse, and the output unit 17 is a display device such as a liquid crystal display. The GPU 19 functions as an accelerator that assists the arithmetic processing (for example, parallel arithmetic processing) performed by the CPU 11. In other words, in the following explanation, CPU1
The processing performed by the CPU 11 includes the processing performed by the CPU 11 using the GPU 19 as an accelerator.

Further, the processing unit 10 stores in advance the program according to the present invention and the pre-learning neural network 50 in the storage unit 13 in an executable format, for example, in order to process each step described in FIG. 10 below. . The executable format is, for example, a format generated by converting a programming language with a compiler. The processing unit 10 performs processing using the program stored in the storage unit 13 and the pre-learning neural network 50.

In the following description, unless otherwise specified, the processing performed by the processing unit 10 is based on the program stored in the storage unit 13 or the memory 12 and the neural network 50.
It means the process performed by PU11. The CPU 11 uses the memory 12 as a work area to temporarily store necessary data (intermediate data during processing, etc.), and appropriately stores long-term storage data such as calculation results in the storage unit 13.

Referring to FIG. 9, the user device 200 (200A, 200B, 200C) includes a processing section 20 (20A, 20B, 20C), an input section 26, and an output section 27.

The processing unit 20 includes a CPU (Central Processing Unit) 21 that performs data processing, which will be described later, a memory 22, which is used as a work area for data processing, and a storage unit 23, which stores programs and processing data, which will be described later. It includes a bus 24 for transmitting data, an interface section 25 for inputting and outputting data to and from external devices, and a GPU (Graphics Processing Unit) 29. The input section 26 and the output section 27 are connected to the processing section 20. Illustratively, the input unit 26 is an input device such as a keyboard or a mouse, and the output unit 27 is a display device such as a liquid crystal display. The GPU 29 functions as an accelerator that assists the arithmetic processing (for example, parallel arithmetic processing) performed by the CPU 21. That is, in the following explanation, CPU2
The processing performed by CPU 1 includes processing performed by CPU 21 using GPU 29 as an accelerator.

Further, the processing unit 20 stores the program according to the present invention and the deep learning algorithm 60 of the learned neural network structure in the storage unit 23 in an executable format, for example, in order to process each step described in FIG. 14 below. I remember it in advance. The executable format is, for example, a format generated by converting a programming language with a compiler. The processing unit 20 performs processing using the program stored in the storage unit 23 and the deep learning algorithm 60.

In the following description, unless otherwise specified, the processing performed by the processing unit 20 is actually the processing performed by the CPU 21 of the processing unit 20 based on the program and deep learning algorithm 60 stored in the storage unit 23 or memory 22. means. The CPU 21 uses the memory 22 as a work area to temporarily store necessary data (intermediate data during processing, etc.), and appropriately stores long-term storage data such as calculation results in the storage unit 23.

[Functional blocks and processing procedures]
-Deep learning processing Referring to FIG. 10, the processing unit 10A of the deep learning device 100A according to the first embodiment is
It includes a training data generation section 101, a training data input section 102, and an algorithm update section 103. These functional blocks install a program that causes a computer to execute deep learning processing in the storage unit 13 or memory 12 of the processing unit 10A, and install this program in the C
This is realized by the PU 11 executing it. The window size database 104 and the algorithm database 105 are stored in the storage unit 13 or memory 12 of the processing unit 10A.

The first training image 70R1, second training image 70R2, and third training image 70R3 of the learning sample are captured in advance by the imaging device 300 and stored in the storage unit 13 or memory 12 of the processing unit 10A in advance. It is assumed that it is remembered. The neural network 50 is stored in advance in the algorithm database 105 in association with, for example, the type of tissue (eg, tissue name) from which the sample to be analyzed comes from.

The processing unit 10A of the deep learning device 100A performs the processing shown in FIG. 11. Explaining using each functional block shown in FIG. 10, the training data generation unit 101 performs the processing from steps S10 to S12, S17, and S18. The process in step S14 is performed by the training data input unit 102.
will do. The processing in steps S15 and S16 is performed by the algorithm updating unit 103.

In steps S10 to S18 described below, deep learning processing using the first training image 70R1, the second training image 70R2, and the third training image 70R3 will be described.

The processing unit 10A generates the tumor region training data 75 according to the method described in the overview of the deep learning method.
R1, deep learning processing is performed to generate non-tumor region training data 75R2 and non-tissue region training data 75R3. The processing unit 10A displays, on the output unit 17, an image of a wide area (whole slide image 70W1) including the area of the first training image 70R1, based on an operation from the input unit 16 by a pathologist or the like, for example. A pathologist or the like who makes the determination visually confirms the whole slide image 70W1 displayed on the output unit 17. The pathologist or the like specifies, for example, via the input unit 16, an area in the whole slide image 70W1 that has been determined to include a tumor area, and surrounds the area in the whole slide image 70W1 with a solid line of, for example, red. Similarly, the pathologist and the like surround non-tumor areas and non-tissue areas in the whole slide image 70W1 with solid lines in blue, green, etc., which are different from red, for example. Instead of displaying the whole slide image 70W1 on the output unit 17 for judgment by a pathologist, the processing unit 10A displays the judged whole slide image 70W2 via the I/F unit 15, for example, on the network 99. It may also be obtained via A pathologist or the like may divide the whole slide image 70W2 into pixels preferable as a training image.

The processing unit 10A processes the whole slide image 7 based on the instructions from the input unit 16 such as a pathologist.
The first training image 70R1 acquired from the area surrounded by the solid red line during 0W2 is cut out to include a predetermined number of pixels. Similarly, each of the second training image 70R2 and the third training image 70R3 is cut out from the portion determined to be a non-tumor region and a non-tissue region so as to include a predetermined pixel region.

In step S10, the processing unit 10A generates color density encoded diagrams 72R1r, 72R1g, and 72R1b for each of R, G, and B colors from the extracted first training image 70R1. The color density encoding diagrams 72R1r, 72R1g, and 72R1b are generated by attaching codes to each pixel that stepwise represent the color density values of R, G, and B of each pixel of the first training image 70R1.
In this embodiment, color density encoded diagrams 72R1r, 72R1g, and 72R1b are generated for each R, G, and B gradation image, assuming that color density values are represented by 256 gradations from value 0 to value 255. As for the assignment of color density values, for example, the lowest brightness is assigned a color density value of 0, and as the level of brightness increases, gradually higher values are assigned, and the highest brightness is assigned a color density value of 255. Processing section 10
A generates first training data 74R1 in which R, G, and B color density values are assigned to each pixel from the color density encoded diagrams 72R1r, 72R1g, and 72R1b. The processing unit 10A generates a first training image 70R1 in which each pixel of the first training image 70R1 is associated with a label value "1" indicating that it is a tumor region.
Preliminary training data 70R1L is generated.

In step S11, the processing unit 10A processes step S from the second training image 70R2.
The second training data 74R2 is generated by the same method as in 10. The processing unit 10A generates second preliminary training data 70R2L in which each pixel of the second training image 70R2 is associated with a label value "2" indicating that it is a non-tumor region.

In step S12, the processing unit 10A performs step S
The processing unit 10A generates third training data 74R3 in a manner similar to that of the processing unit 10. The processing unit 10A generates third preliminary training data 70R3L in which each pixel of the third training image 70R3 is associated with a label value of “0” indicating a non-tissue region.

Steps S10 to S12 may be performed simultaneously or in random order.

In step S13, the processing unit 10A inputs the deep learning device 100 through the input unit 16.
Receives input of the type of organization for learning from the user on the A side. The processing unit 10A stores the window size database 104 (window size DB) based on the input tissue type.
104), the window size is set, and the algorithm database 105 (algorithm DB 105) is referenced to set the neural network 50 used for learning. The window size is, for example, 200×200 pixels. The window size is a unit of training data input to the neural network 50 at one time of input, and the product of the number of pixels of the training data corresponding to each region of the window size and the number of primary colors included in the image is: This corresponds to the number of nodes in the input layer 50a. The window size is stored in advance in the window size database 104 in association with the type of organization.

In step S14, the processing unit 10A generates window-sized tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3 from the training data corresponding to each region. Specifically, as explained in the above-mentioned "Overview of deep learning method" with reference to FIGS. 4(a) and 4(b), from the first training data 74R1 and the first preliminary training data 70R1L, Tumor region training data 75R1 of window size is created using window W1. The processing unit 10A generates window-sized non-tumor region training data 75R2 from the second training data 74R2 and the second preliminary training data 70R2L.
generate. Specifically, in the above “Overview of deep learning method”, Figure 4(a) and (b)
And as explained with reference to FIG. 5(a), non-tumor region training data 75R2 having a window size of window W1 is created from the second training data 74R2 and the second preliminary training data 70R2L. The processing unit 10A generates window-sized non-organized region training data 75R3 from the third training data 74R3 and the third preliminary training data 70R3L. Specifically, as explained with reference to FIGS. 4(a), (b) and FIG. 5(b) in the above-mentioned "Outline of deep learning method", the third training data 74R3 and the third From the preliminary training data 70R3L, non-organized region training data 75R3 having a window size of window W1 is created.

In step S15 shown in FIG. 11, the processing unit 10A stores the tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R of the window size.
3 to train the neural network 50. The learning results of the neural network 50 include window size tumor region training data 75R1 and non-tumor region training data 7.
5R2 and the non-organized region training data 75R3 are accumulated each time the neural network 50 is trained.

The image analysis method according to the embodiment uses a convolutional neural network,
Since the stochastic gradient descent method is used, in step S16, the processing unit 10A determines whether learning results for a predetermined number of trials have been accumulated. If the learning results have been accumulated for the predetermined number of trials, the processing unit 10A performs the process of step S17, and if the learning results have not been accumulated for the predetermined number of trials, the processing unit 10A performs the process of step S18.

If the learning results have been accumulated for the predetermined number of trials, in step S17, the processing unit 1
0A updates the connection weight w of the neural network 50 using the learning results accumulated in step S15. Since the image analysis method according to the embodiment uses the stochastic gradient descent method, the connection weight w of the neural network 50 is updated at the stage when learning results for a predetermined number of trials have been accumulated. Specifically, the process of updating the connection weight w is performed using (Equation 1
1) and (Equation 12), this is a process of performing calculations using the gradient descent method.

In step S17, the processing unit 10A judges whether or not a prescribed number of pixels in each of the first training data 74R1, the second training data 74R2, and the third training data 74R3 have been processed.
When a series of processes from step S15 to step S17 have been performed, the deep learning process is terminated. The neural network does not necessarily need to be trained on all pixels in each training data, and the processing unit 10A can process and train a portion of the pixels in each training data, i.e., a specified number of pixels. The specified number of pixels may be all pixels in each area training data.

If the specified number of pixels in each area training data have not been processed, the processing unit 10A, in step S19, processes the first training data 74R1, the second training data The position of the window is moved within 74R2 or within the third training data 74R3. Thereafter, the processing unit 10A performs a series of processes from step S14 to step S16 at the new window position after the movement. That is, the processing unit 10A performs step S
14, the first training data 74R1 and the second training data 74R1 are displayed at the new window position after the movement.
For each of the training data 74R2 and the third training data 74R3, each region training data 75R1, 75R2, 75R3 of the window size is cut out, and a label value indicating each region is attached. Subsequently, in step S15, the processing unit 10A causes the neural network 50 to learn using the training data 75R1, 75R2, and 75R3 for each region of the newly cut out window size. If the learning results for the predetermined number of trials have been accumulated in step S16, the processing unit 10A updates the connection weight w of the neural network 50 in step S17. Such learning of the neural network 50 for each window size is performed for a specified number of pixels for each of the first training data 74R1, the second training data 74R2, and the third training data 74R3.

The deep learning processing from steps S10 to S19 described above is repeatedly performed on the separately acquired first training image 70R1, second training image 70R2, and third training image 70R3. This improves the degree of learning of the neural network 50. As a result, a deep learning algorithm 60 having a neural network structure shown in FIG. 5 is obtained.

- Structure of Neural Network As shown in FIG. 12(a), the first embodiment uses a deep learning type neural network. A deep learning type neural network, like a neural network 50 shown in FIG. 12, includes an input layer 50a, an output layer 50b, and an intermediate layer 50c between the input layer 50a and the output layer 50b. Consists of multiple layers. The number of layers constituting the intermediate layer 50c can be, for example, five or more.

In the neural network 50, a plurality of nodes 89 arranged in layers are connected between the layers. As a result, information propagates from the input side layer 50a to the output side layer 50b in only one direction shown by arrow D in the figure. In this embodiment, the number of nodes in the input layer 50a corresponds to the product of the number of pixels of the input image, that is, the number of pixels of the window W1 shown in FIG. 3(c), and the number of primary colors included in each pixel. ing. Since pixel data (color density values) of an image can be input to the input layer 50a, the user can input the input image to the input layer 50a without separately calculating feature amounts from the input image.

-Calculation at each node FIG. 12(b) is a schematic diagram showing the calculation at each node. Each node 89 receives multiple inputs and calculates one output (z). In the example shown in FIG. 12(b), node 89 receives four inputs. The total input (u) received by the node 89 is expressed by the following (Equation 1).

Each input is given a different weight. In (Formula 1), b is a value called bias. The output (z) of the node is the output of a predetermined function f for the total input (u) expressed by (Formula 1), and is expressed by (Formula 2) below. The function f is called an activation function.

FIG. 12(c) is a schematic diagram showing operations between nodes. In the neural network 50, nodes that output a result (z) expressed by (Formula 2) with respect to a total input (u) expressed by (Formula 1) are arranged in a layered manner. The output of the node in the previous layer becomes the input to the node in the next layer. In the example shown in FIG. 12(c), the output of the node 89a on the left layer in the figure becomes the input to the node 89b on the right layer in the figure. Each node 89b in the right layer receives an output from node 89a in the left layer. Each connection between each node 89a in the left layer and each node 89b in the right layer is given a different weight. If the outputs of the plurality of nodes 89a in the left layer are x ₁ to x ₄ , the inputs to each of the three nodes 89b in the right layer are as follows (Equation 3-1) to (Equation 3-3) It is expressed as

When these (Formula 3-1) to (Formula 3-3) are generalized, (Formula 3-4) is obtained. Here, i
=1,...I, j=1,...J.

（式３－４）を活性化関数に適用すると出力が得られる。出力は以下の（式４）で表さ
れる。

Applying (Equation 3-4) to the activation function yields an output. The output is expressed by the following (Equation 4).

・Activation function In the image analysis method according to the embodiment, a rectified linear function is used as the activation function.
near unit function). The normalized linear function is expressed by the following (Equation 5).

(Equation 5) is a function in which the part where u<0 is set to u=0 among the linear functions of z=u. Figure 1
In the example shown in 2(c), the output of the node with j=1 is expressed by the following equation using (Equation 5).

・Learning of neural networks If the function expressed using a neural network is y(x:w), then the function y(x:w)
:w) changes when the parameter w of the neural network is changed. The function y(x:
Adjusting w) is called neural network learning. Assume that multiple pairs of input and output of a function expressed using a neural network are given. If the desired output for a certain input x is d, the input/output pair is {(x ₁ , d ₁ ), (x ₂ , d ₂ ),
..., (x _n , d _n )}. Each set of pairs represented by (x, d) is called training data. Specifically, a set of pairs of color density values and true value image labels for each pixel in a single color image of R, G, and B shown in FIG. 3(b) is shown in FIG. 3(a). This is the training data shown.

Neural network learning means that for any input/output pair (x _n , d _n ), the output y (x _n :w) of the neural network when input x _n is given is the output d _n This means adjusting the weight w so that it is as close as possible. Error function is the closeness between the function expressed using a neural network and the training data.

It is a scale to measure. The error function is also called a loss function. The error function E(w) used in the image analysis method according to the embodiment is expressed by the following (Equation 6). (Formula 6
) is called cross entropy.

A method for calculating the cross entropy of (Equation 6) will be explained. In the output layer 50b of the neural network 50 used in the image analysis method according to the embodiment, that is, in the final layer of the neural network, an activation function is used to classify the input x into a finite number of classes according to the content. The activation function is called a softmax function and is expressed by the following (Equation 7). It is assumed that the same number of nodes as the number of classes k are arranged in the output layer 50b. It is assumed that the total input u of each node k (k=1, . . . K) of the output layer L is given by u _k ^(L) from the output of the previous layer L-1. As a result, the output of the k-th node of the output layer is expressed by the following (Equation 7).

(Equation 7) is the softmax function. The sum of the outputs y ₁ , . . . , y _K determined by (Equation 7) is always 1.

If each class is represented as C ₁ , ..., C _K , then the output y _k (i.e. u
_k ^(L) ) represents the probability that a given input x belongs to class C _k . Please refer to (Equation 8) below. The input x is classified into the class in which the probability expressed by (Equation 8) is maximized.

In neural network training, the function represented by the neural network is regarded as a model of the posterior probability of each class, and under such a probability model, the likelihood of the weight w for the training data is calculated as follows: evaluate and select the weight w that maximizes the likelihood.

It is assumed that the target output d _n by the softmax function in (Equation 7) is set to 1 only when the output is in the correct class, and is set to 0 otherwise. The target output is d _n =[d _n1 ,...
. , d _nK ], for example, if the correct class of input x _n is C ₃ , only the target output d _n3 becomes 1, and the other target outputs become 0. When encoded in this way, the posterior distribution is expressed by the following (Equation 9).

Likelihood L(w) of weight w for training data {(x _n , d _n )} (n=1,...,N)
is expressed by the following (Formula 10). If we take the logarithm of the likelihood L(w) and invert the sign, we get (Equation 6
) is derived.

Learning means minimizing the error function E(w) calculated based on training data with respect to the parameter w of the neural network. In the image analysis method according to the embodiment, the error function E(w) is expressed by (Equation 6).

Minimizing the error function E(w) with respect to the parameter w has the same meaning as finding a local minimum point of the function E(w). The parameter w is the weight of connections between nodes. The minimum point of the weight w is determined by an iterative calculation in which the parameter w is repeatedly updated using an arbitrary initial value as a starting point. An example of such a calculation is gradient descent met
hod).
The gradient descent method uses a vector expressed by the following (Equation 11).

In the gradient descent method, the process of moving the current value of the parameter w in the negative gradient direction (i.e., −∇E) is repeated many times. If the current weight is w ^(t) and the weight after the movement is w ^(t+1) , the calculation by the gradient descent method is expressed by the following (Equation 12). The value t is the parameter w
This refers to the number of times the character has been moved.

は、パラメータｗの更新量の大きさを決める定数であり、学習係数と呼ばれる。（式１２
）で表される演算を繰り返すことにより、値ｔの増加に伴って誤差関数Ｅ（ｗ^（ｔ））が
減少し、パラメータｗは極小点に到達する。 symbol

is a constant that determines the amount of update of the parameter w, and is called a learning coefficient. (Formula 12
), the error function E(w ^(t) ) decreases as the value t increases, and the parameter w reaches its minimum point.

Note that the calculation according to (Equation 12) may be performed on all training data (n=1, . . . , N), or may be performed on only some training data. Gradient descent performed only on a portion of the training data is called stochastic gradient descent. The image analysis method according to the embodiment uses stochastic gradient descent.

- Image analysis processing Referring to FIG. 13, the processing unit 20A of the image analysis device 200A according to the first embodiment is
It includes an analysis data generation section 201, an analysis data input section 202, an analysis section 203, and a region detection section 204. These functional blocks are realized by installing a program that causes a computer according to the present invention to execute image analysis processing in the storage unit 23 or memory 22 of the processing unit 20A, and having the CPU 21 execute this program. The window size database 104 and the algorithm database 105 are provided from the deep learning device 100A via the storage medium 98 or the network 99, and are stored in the storage unit 23 or memory 22 of the processing unit 20A.

It is assumed that an analysis target image 78 of a tissue to be analyzed is captured in advance by the imaging device 400 and stored in advance in the storage unit 23 or memory 22 of the processing unit 20A. The deep learning algorithm 60 including the learned connection weights w is stored in the algorithm database 105 in association with the type of tissue from which the tissue sample to be analyzed comes from, and is used as a program for causing a computer to perform image analysis processing. Functions as a part of a program module.
That is, the deep learning algorithm 60 is used in a computer equipped with a CPU and memory, and is designed to perform calculations or processing of information specific to the purpose of use, such as outputting data indicating each region in the tissue to be analyzed. , make the computer work. Specifically, the CPU 21 of the processing unit 20A performs calculations of the neural network 60 based on the learned connection weights w according to the algorithm defined in the deep learning algorithm 60 stored in the storage unit 23 or the memory 22. The CPU 21 of the processing unit 20A has an input layer 60a.
The calculation is performed on the analysis target image 78, which is input to the analysis target tissue, and outputs a ternary image 83, which is data indicating each region in the analysis target tissue, from the output layer 60b.

Referring to FIG. 14, the processing unit 20A of the image analysis device 200A performs the processing shown in FIG. 13. Explaining using each functional block shown in FIG. 13, the processing of steps S21 and S22 is performed by the analysis data generation unit 201. Steps S23, S24, S26 and S27
This process is performed by the analysis data input unit 202. The processing in steps S25 and S28 is performed by the analysis unit 203. The process in step S29 is performed by the area detection unit 204.

In step S21, the processing unit 20A, in response to a command to start processing input by a user or the like from the input unit 26, generates color density encoded diagrams 79r, 79g of R, G, and B colors from the input analysis target image 78, for example. Generate 79b. Data to be analyzed is generated by combining the color density values of R, G, and B colors for each pixel from the color density encoded diagrams 79r, 79g, and 79b. The method of generating the color density encoded diagrams 79r, 79g, and 79b is the same as the method of generating them in step S10 during the deep learning process shown in FIG. 11.

In step S22 shown in FIG. 14, the processing unit 20A receives an input of the type of tissue from the user of the image analysis device 200A as an analysis condition through the input unit 26. Based on the input tissue type, the processing unit 20A refers to the window size database 104 and the algorithm database 105, sets a window size to be used for analysis, and obtains a deep learning algorithm 60 to be used for analysis. The window size is a unit of analysis data input to the neural network 60 at one time of input, and the product of the number of pixels of the window size analysis data 80 and the number of primary colors included in the image is 60a
corresponds to the number of nodes. The window size is stored in advance in the window size database 104 in association with the type of organization. The window size is, for example, 3×3 pixels, as shown in window W2 shown in FIG. The deep learning algorithm 60 is also stored in advance in the algorithm database 105 shown in FIG. 13 in association with the type of organization.

In step S23 shown in FIG. 14, the processing unit 20A performs color density encoding
Data to be analyzed is generated by combining the color density values of R, G, and B colors for each pixel from g and 79b, and furthermore, analysis data 80 of the window size is generated.

In step S24, the processing unit 20A inputs the analysis data 80 shown in FIG. 5 to the deep learning algorithm 60. The initial position of the window is, for example, a position where a 3×3 pixel within the window corresponds to the upper left corner of the data to be analyzed, as in step S14 during deep learning processing. The processing unit 20A generates window size analysis data 80.
When the data 81 of a total of 27 color density values of 3×3 pixels×3 primary colors included in the input layer 60a is input to the input layer 60a, the deep learning algorithm 60 outputs a discrimination result 82 to the output layer 60b.

In step S25 shown in FIG. 14, the processing unit 20A stores the determination result 82 output to the output layer 60b shown in FIG. The determination result 82 is an estimated value (one of the three values 0, 1, and 2) for each pixel region of the data to be analyzed.

In step S26 shown in FIG. 14, the processing unit 20A determines whether all pixels in the input image have been processed. The input images are color density encoded diagrams 79r, 79g, 79 shown in FIG.
b, and the series of processes from step S23 to step S25 shown in FIG. 14 have been performed for all pixels in the data to be analyzed, the process of step S28 is performed.

If all the pixels in the data to be analyzed have not been processed, in step S27, the processing unit 20A converts the color density encoding diagrams 79r, 79g, 79b shown in FIG. The window W2 is moved in units of a predetermined number of pixels. Thereafter, the processing unit 20A performs a series of processes from step S23 to step S25 at the new position of the window W2 after the movement. In step S25, the processing unit 20A stores the determination result 82 corresponding to the new window position after the movement. By storing such determination results 82 for each window size for all pixels in the image to be analyzed, a ternary image 83 as an analysis result is obtained. The image size of the ternary image 83 as the analysis result is the same as the image size of the analysis target image. Here, the ternary image 83 may include numerical data in which estimated values 2, 1, and 0 are attached to each pixel;
For example, instead of the value 0, the image may be displayed in a display color corresponding to each of the values 2, 1, and 0.

In step S28 shown in FIG. 14, the processing unit 20A outputs the ternary image 83 as the analysis result to the output unit 27.

In step S29, following step S28, processing unit 20A further performs region detection processing on the analysis result ternary image 83. In ternary image 83, the tumor region, non-tumor region, and non-tissue region are represented by being distinguished by three values.

Although optional, the processing unit 20A creates a region-enhanced image 84 by superimposing each obtained region on the analysis target image 78 to be analyzed. The processing unit 20A outputs the created region-enhanced image 84 to the output unit 27, and ends the image analysis process.

Although optional, the processing unit 20A may calculate how much of the tumor region is included in the tissue included in the analysis target image 78. For example, if the number of pixels corresponding to the tissue included in the analysis target image 78, that is, the total number of pixels determined to be a tumor region and a non-tumor region (the total number of pixels in the tissue region), is determined to be a tumor region. By calculating the percentage occupied by the number of pixels, the content rate of the tumor region can be calculated. In addition, by subtracting the number of pixels determined to be tumor areas from the total number of pixels, or the total number of pixels in tissue areas) can be reduced to 100%.
By calculating the percentage of pixels determined to be non-tumor areas,
The content of non-tumor areas can be calculated. Then, the ratio between the content rate of the tumor region and the content rate of the non-tumor region may be calculated. The calculated value may be output to the output section 27. Further, the calculated value may be output to the output unit 27 together with the region-enhanced image 84.

Calculating the content rate of tumor regions in this manner is useful in screening whether tissue to be used for genetic analysis of cancer tissue or the like contains tumor tissue suitable for testing.

As described above, the user of the image analysis apparatus 200A can obtain the ternary image 83 as an analysis result by inputting the analysis target image 78 of the tissue to be analyzed into the image analysis apparatus 200A. The ternary image 83 represents a tumor region, a non-tumor region, and a non-tissue region in the specimen to be analyzed, and allows the user to distinguish each region in the specimen to be analyzed.

Further, the user of the image analysis device 200A can obtain a region-enhanced image 84 as an analysis result. The region-enhanced image 84 is generated, for example, by filling each region of the analysis target image 78 with a color. In another aspect, the area is generated by overlapping the boundaries of each area. This allows the user to grasp each area at a glance in the tissue to be analyzed.

Indicating the tumor region and/or non-tumor region in the specimen to be analyzed helps to screen the tumor region in the tissue to be analyzed prior to tissue diagnosis and increases sample observation efficiency.

<Second embodiment>
Hereinafter, the image analysis system according to the second embodiment will be described with respect to the differences from the image analysis system according to the first embodiment.

[Configuration Overview]
Referring to FIG. 15, the image analysis system according to the second embodiment includes a user device 200.
The image analysis system according to the second embodiment is a stand-alone system that performs deep learning and image analysis on the user side. The image analysis system according to the second embodiment is different from the image analysis system according to the first embodiment in that the integrated image analysis device 200B installed on the user side performs the functions of both the deep learning device 100A and the image analysis device 200A according to the first embodiment.

Image analysis device 200B is connected to imaging device 400. The imaging device 400 acquires a training image of a tissue for learning during deep learning processing, and acquires an analysis target image 78 of a tissue to be analyzed during image analysis processing.

[Hardware configuration]
The hardware configuration of the image analysis device 200B is similar to the hardware configuration of the user-side device 200 shown in FIG.

[Functional blocks and processing procedures]
Referring to FIG. 16, the processing unit 20B of the image analysis device 200B according to the second embodiment is
A training data generation unit 101, a training data input unit 102, an algorithm update unit 103,
It includes an analysis data generation section 201, an analysis data input section 202, an analysis section 203, and a region detection section 204. These functional blocks are realized by installing a program that causes a computer to execute deep learning processing and image analysis processing in the storage unit 23 or memory 22 of the processing unit 20B, and having the CPU 21 execute this program. The window size database 104 and the algorithm database 105 are stored in the storage unit 23 or memory 22 of the processing unit 20B, and both are used in common during deep learning and image analysis processing. The trained neural network 60 is stored in advance in the algorithm database 105 in association with the type of tissue or the type of sample containing cells, and the connection weights w are updated by deep learning processing, and the deep learning algorithm 60 It is stored in the algorithm database 105 as follows. Note that the first training image 7 which is a training image
It is assumed that 0R1, the second training image 70R2, and the third training image 70R3 are captured in advance by the imaging device 400 and stored in the storage unit 23 or memory 22 of the processing unit 20B. It is assumed that an analysis target image 78 of a specimen to be analyzed is also captured in advance by the imaging device 400 and stored in advance in the storage unit 23 or the memory 22 of the processing unit 20B.

The processing unit 20B of the image analysis device 200B performs the processing shown in FIG. 11 during deep learning processing, and performs the processing shown in FIG. 14 during image analysis processing. To explain using each functional block shown in FIG. 16, during deep learning processing, the training data generation unit 101 performs the processing from steps S10 to S12, S14, S18, and S19. The process of step S13 is performed by the training data input unit 102. The processing in steps S16 and S17 is carried out by the algorithm updating unit 103.
will do. During image analysis processing, the processing of steps S21 and S22 is performed by the analysis data generation unit 201. The processing of steps S23, S24, S26, and S27 is performed by the analysis data input unit 202. The processing in steps S25 and S28 is performed by the analysis unit 203. The process in step S29 is performed by the area detection unit 204.

The deep learning processing procedure and the image analysis processing procedure performed by the image analysis device 200B according to the second embodiment are similar to those of the deep learning device 100A and the image analysis device 200A according to the first embodiment.
The procedure is the same as that for each. Note that the image analysis device 200B according to the second embodiment is different from the deep learning device 100A and the image analysis device 200 according to the first embodiment in the following points.
Different from A.

In step S13 during deep learning processing, the processing unit 20B receives an input of the type of tissue for learning from the user of the image analysis device 200B through the input unit 26. Processing section 2
Based on the input tissue type, 0B refers to the window size database 104 to set the window size, and refers to the algorithm database 105 to set the neural network 50 used for learning.

As described above, the user of the image analysis device 200B can obtain the ternary image 83 as an analysis result by inputting the analysis target image 78 into the image analysis device 200B. moreover,
The user of the image analysis device 200B can obtain a region-enhanced image 84 as an analysis result.

According to the image analysis device 200B according to the second embodiment, the user can use the type of tissue selected by the user as a learning tissue. This is neural network 5
This means that the learning of 0 is not left to the vendor side, but that the user himself can improve the learning level of the neural network 50.

<Third embodiment>
Hereinafter, the image analysis system according to the third embodiment will be described with respect to the differences from the image analysis system according to the second embodiment.

[Configuration overview]
Referring to FIG. 17, the image analysis system according to the third embodiment includes a vendor-side device 100
and a user-side device 200. The vendor device 100 is an integrated image analysis device 100
The user device 200 operates as a terminal device 200C. The image analysis device 100B is configured of, for example, a general-purpose computer, and is a device on the cloud server side that performs both the deep learning processing and the image analysis processing described in the first embodiment. The terminal device 200C is configured with a general-purpose computer, for example, and transmits the image to be analyzed to the image analysis device 100B through the network 99.
This is a user-side terminal device that receives an image of an analysis result from the image analysis device 100B.

In the image analysis system according to the third embodiment, an integrated image analysis device 100B installed on the vendor side performs the deep learning device 100A and the image analysis device 200A according to the first embodiment.
In terms of having both functions, the image analysis system according to the second embodiment is similar to the image analysis system according to the first embodiment.
On the other hand, the image analysis system of the third embodiment differs from the image analysis system of the second embodiment in that it is equipped with a terminal device 200C and provides an input interface for the image to be analyzed and an output interface for the image of the analysis result to the user's terminal device 200C.
In other words, the image analysis system according to the third embodiment is a cloud service type system in which the vendor that performs the deep learning processing and image analysis processing provides the user with an input/output interface for the images to be analyzed and the images of the analysis results.

The image analyzing device 100B is connected to the imaging device 300, and acquires training images of the learning tissue captured by the imaging device 300.

The terminal device 200C is connected to the imaging device 400, and acquires an analysis target image 78 of the tissue to be analyzed, which is captured by the imaging device 400.

[Hardware configuration]
The hardware configuration of the image analyzing device 100B is similar to that of the vendor-side device 100 shown in Fig. 7. The hardware configuration of the terminal device 200C is similar to that of the user-side device 200 shown in Fig. 9.

[Function blocks and processing procedures]
Referring to FIG. 18, a processing unit 10B of an image analyzing device 100B according to the third embodiment
A training data generation unit 101, a training data input unit 102, an algorithm update unit 103,
The computer-readable storage device includes an analysis data generating unit 201, an analysis data input unit 202, an analysis unit 203, and an area detection unit 204. These functional blocks are realized by installing a program for causing a computer to execute deep learning processing and image analysis processing in the storage unit 13 or memory 12 of the processing unit 10B, and executing the program by the CPU 11. The window size database 104 and the algorithm database 105 are stored in the storage unit 13 or memory 12 of the processing unit 10B, and both are used in common during deep learning and image analysis processing. The neural network 50 is associated with the type of tissue and stored in advance in the algorithm database 105, and the connection weight w is updated by the deep learning processing and stored in the algorithm database 105 as a deep learning algorithm 60.

The first training image 70R1, the second training image 70R2, and the third training image 70R3 are training images.
The training image 70R3 is assumed to be captured in advance by the imaging device 300 and stored in advance in the storage unit 13 or memory 12 of the processing unit 10B. The analysis target image 78 of the tissue to be analyzed is also assumed to be captured in advance by the imaging device 400 and stored in advance in the storage unit 23 or memory 22 of the processing unit 20C of the terminal device 200C.

The processing unit 10B of the image analysis device 100B performs the processing shown in FIG. 11 during deep learning processing, and performs the processing shown in FIG. 14 during image analysis processing. To explain using each functional block shown in FIG. 18, during deep learning processing, the training data generation unit 101 performs the processing from steps S10 to S12, S14, S18, and S19. The process of step S13 is performed by the training data input unit 102. The processing in steps S16 and S17 is carried out by the algorithm updating unit 103.
will do. During image analysis processing, the processing of steps S21 and S22 is performed by the analysis data generation unit 201. The processing of steps S23, S24, S26, and S27 is performed by the analysis data input unit 202. The processing in steps S25 and S28 is performed by the analysis unit 203. The process in step S29 is performed by the area detection unit 204.

The deep learning processing procedure and the image analysis processing procedure performed by the image analysis device 100B according to the third embodiment are similar to those of the deep learning device 100A and the image analysis device 200A according to the first embodiment.
The procedure is the same as that for each. Note that the image analysis device 100B according to the third embodiment differs from the deep learning device 100A and the image analysis device 200A according to the first embodiment in the following four points.

In step S21 during the image analysis process shown in FIG. 14, the processing unit 10B receives the analysis target image 78 of the tissue to be analyzed from the user's terminal device 200C, and from the received analysis target image 78, R, G, Color density encoded diagrams 79r, 79g, and 79b for each B color are generated. The method of generating the color density encoding diagrams 79r, 79g, and 79b and the data to be analyzed is the same as the method of generating them in step S10 during the deep learning process shown in FIG. 11.

In step S22 during the image analysis process shown in FIG.
Through the input unit 26 of 00C, input of the type of organization from the user of the terminal device 200C is accepted as an analysis condition. Based on the input tissue type, the processing unit 10B refers to the window size database 104 and the algorithm database 105, sets the window size to be used for analysis, and obtains the deep learning algorithm 60 used for the analysis.

In step S28 at the time of image analysis processing, the processing unit 10B uses the ternary image 83 as the analysis result.
is transmitted to the user's terminal device 200C. In the user's terminal device 200C, the processing unit 20C outputs the three images 83 of the received analysis results to the output unit 27.

In step S29 during the image analysis process, the processing unit 10B performs a process of detecting a nucleus region on the ternary image 83 of the analysis result following step S28. As an optional step, the processing unit 10B creates a region-enhanced image 84 by superimposing each of the obtained regions on the analysis target image 78 of the analysis target. The processing unit 10B converts the created region-enhanced image 84 into
The user terminal device 200C transmits the request to the processing unit 20
C outputs the received area-enhanced image 84 to the output unit 27, and ends the image analysis process.

As described above, the user of the terminal device 200C can obtain the ternary image 83 as an analysis result by transmitting the analysis target image 78 of the tissue to be analyzed to the image analysis apparatus 100B. Further, the user of the terminal device 200C can obtain the region-enhanced image 84 as an analysis result.

According to the image analyzing device 100B of the third embodiment, a user can enjoy the results of the image analysis process without acquiring the window size database 104 and the algorithm database 105 from the deep learning device 100A. This makes it possible to provide, as a cloud service, a service for analyzing a tissue to be analyzed, for discriminating between tumorous regions and/or non-tumorous regions and screening the tumorous region in each tissue.

[Computer program]
In the embodiment of the present invention, steps S10 to S20 are performed by the processing units 10A, 20B, 10.
This includes a computer program that is executed by B and generates a trained deep learning algorithm, and products thereof. Further, an embodiment of the present invention includes a computer program and product thereof that causes the processing units 10A, 20B, and 10B to execute the steps S21 to S29 and causes the computer to function to analyze images of tissues collected from an individual. included.

[Other forms]
Although the present invention has been described above with reference to the outline and specific embodiments, the present invention is not limited to the outline and each embodiment described above.

In the first to third embodiments described above, in step S13, the processing units 10A, 20B,
10B sets the number of pixels of the window size with reference to the window size database 104, but the operator or user may directly set the window size.
In this case, the window size database 104 becomes unnecessary.

In the first to third embodiments described above, in step S13, the processing units 10A, 20B,
10B sets the number of pixels of the window size based on the input tissue type, but the tissue size may be input instead of the tissue type. Processing parts 10A, 20B
, 10B is a window size database 104 based on the input tissue size.
You can set the number of pixels for the window size by referring to . In step S22, as in step S13, instead of inputting the type of organization, the size of the organization may be input. The processing units 20A, 20B, and 10B refer to the window size database 104 and the algorithm database 105 based on the input tissue size, set the number of pixels of the window size, and acquire the neural network 60.

Regarding the mode of inputting the size of the organization, the size may be input directly as a numerical value, or
For example, the input user interface may be a pull-down menu, and the user may select and input a predetermined numerical range corresponding to the size the user wants to input.

Furthermore, in step S13 and step S22, in addition to the tissue type or tissue size, the imaging magnification at which the training image and the analysis target image 78 were captured may be input. Regarding the mode of inputting the imaging magnification, the magnification may be directly input as a numerical value, or, for example, the input user interface may be set as a pull-down menu and the user can select a predetermined range of numerical values corresponding to the magnification that the user wants to input. You can also input it by

In the first to third embodiments above, the window size is set to 3 x 3 pixels for the convenience of explanation during deep learning processing and image analysis processing, but the number of pixels in the window size is set to 3 x 3 pixels for convenience of explanation. Not limited. The window size may be set depending on the type of organization, for example. In this case, the product of the number of pixels in the window size and the number of primary colors included in the image is
It is sufficient that the number of nodes corresponds to the number of nodes in the input layers 50a and 60a of the neural networks 50 and 60.

In the first to third embodiments described above, in step S17, the processing units 10A, 20B,
10B stores the deep learning algorithm 60 in the algorithm database 105 in one-to-one correspondence with the type of organization. Alternatively, in step S17, the processing units 10A, 20B, and 10B may store one deep learning algorithm 60 in association with a plurality of tissue types in the algorithm database 105.

In the first to third embodiments described above, the hue is defined by a combination of three primary colors of light or a combination of three primary colors of color, but the number of hues is not limited to three. The number of hues is red (
It is also possible to use four primary colors such as R), green (G), blue (B) plus yellow (Y), or red (R), green (G).
), blue (B), and blue (B). Alternatively, only one of the three primary colors red (R), green (G), and blue (B) (for example, green (G)) may be used. For example, the training images 70R1, 70R2, 70R3 and the analysis target image 78 obtained using a known microscope or virtual slide scanner are also red (R),
The image is not limited to a color image of three primary colors, green (G) and blue (B), and may be a color image of two primary colors, as long as it includes one or more primary colors.

In the first to third embodiments described above, in step S10, the processing units 10A, 20B,
10B is each color density coding diagram (color density coding diagram 70R1r, 70R1g, 7
0R1b, color density encoding diagram 70R2r, 70R2g, 70R2b, color density encoding diagram 70R
3r, 70R3g, 70R3b) are generated as a single color image of 8 bits (256 levels) of each primary color, but the gradation of the primary color when creating each color density encoded diagram is not limited to 8 bits. The gradation of each color density encoded map may depend on the imaging conditions of the slide image scanner. For example, the image may be one bit or more, and may be a 2-bit, 4-bit, 16-bit, or 24-bit image. Similarly, in step S21, the processing units 20A, 20B, and 10B generate the color density encoded diagrams 79r, 79g, and 79b as single color images of each primary color. The gradations are not limited to three gradations. The primary colors used when creating a color density encoded diagram are not limited to 8 bits. The gradations of the color density encoding diagrams 79r, 79g, and 79b are as follows:
It may also depend on the imaging conditions of the slide image scanner. For example, 1 bit or more and 2
It may be a bit, 4 bit, 16 bit, or 24 bit image. It is preferable that the gradations of each color density encoded diagram and the color density encoded diagrams 79r, 79g, and 79b of the training image are all the same gradation.

In the first to third embodiments, in step S10, the processing units 10A, 20B,
The processing units 10A, 20B generate each color density coding map from the training image, but a matrix table in which each pixel is coded according to a gray level in advance may be used as the training image.
B may directly acquire the training images as the color density coding maps from, for example, a virtual slide scanner. Similarly, in step S21, the processing units 20A, 20B, and 10B
However, a matrix table in which each pixel is coded in advance according to its gradation may be used as the analysis image.
79b may be acquired directly, for example from a virtual slide scanner.

In the first to third embodiments described above, RGB is used as the color space when generating each color density encoded diagram from the training image, but the color space is not limited to RGB. RG
In addition to B, various color spaces such as YUV, CMY, and CIE L ^* a ^* b ^* can be used.

In the first to third embodiments described above, in each region training data and analysis data 80, the density values for each pixel are stored in the order of red (R), green (G), and blue (B). , the order in which the density values are stored and handled is not limited to this. For example, the density values are blue (B), green (
G) and red (R), as long as the order of density values in each region training data is the same as the order of density values in analysis data 80.

In the first to third embodiments described above, the processing units 10A and 10B are realized as an integrated device, but the processing units 10A and 10B do not need to be an integrated device, and the CPU 11 and the memory 1
2. The storage unit 13 and the like may be located elsewhere, and these may be connected via a network. The processing units 10A, 10B, the input unit 16, and the output unit 17 do not necessarily need to be placed in one place, but may be placed in separate places and connected to each other so as to be communicable via a network. The processing units 20A, 20B, and 20C are also similar to the processing units 10A and 10B.

In the first to third embodiments described above, the training data generation unit 101, the training data input unit 102
, algorithm update section 103, analysis data generation section 201, analysis data input section 202,
Each functional block of the analysis unit 203 and the area detection unit 204 is executed in a single CPU 11 or a single CPU 21, but each of these functional blocks does not necessarily need to be executed in a single CPU, and is executed in a plurality of CPUs. It may also be executed in a distributed manner by the CPUs. Further, each of these functional blocks may be executed in a distributed manner by a plurality of GPUs, or may be executed in a distributed manner by a plurality of CPUs and a plurality of GPUs.

In the second and third embodiments described above, programs for performing the processing of each step explained in FIGS. 11 and 14 are stored in advance in the storage units 13 and 23. Alternatively, the program may be installed in the processing unit 10B, 20B from a computer-readable, non-transitory tangible storage medium 98, such as a DVD-ROM or a USB memory. Alternatively, the processing units 10B and 20B may be connected to the network 99, and a program may be downloaded and installed from, for example, an external server (not shown) via the network 99.

In the first to third embodiments described above, the input units 16 and 26 are input devices such as a keyboard or a mouse, and the output units 17 and 27 are realized as display devices such as a liquid crystal display. Alternatively, the input sections 16, 26 and the output sections 17, 27 may be integrated into a touch panel type display device. Alternatively, the output units 17 and 27 may be configured with a printer or the like, and the ternary image 83 or the region-enhanced image 84 resulting from the analysis may be printed and output.

In the first to third embodiments described above, the imaging device 300 is directly connected to the deep learning device 100A or the image analysis device 100B; It may be connected to the device 100B. Similarly, the imaging device 400 is directly connected to the image analysis device 200A or the image analysis device 200B, but the imaging device 400 is connected to the image analysis device 200A or the image analysis device 200B via the network 99. May be connected.

<Verification of trained deep learning algorithm>
Deep learning processing and image analysis processing were performed using the stand-alone system shown in the second embodiment. Human gastric cancer tissue, which includes a tumor region and a normal region, was used as the tissue to be studied. To create a trained deep learning algorithm, training images were taken with the tissue sample magnified at different magnifications of 1x, 3x, 5x, 10x, 20x, and 40x. The neural network was trained on training data using training images, and the accuracy of the analysis results was confirmed.
The details of the learning data and analysis data were as follows.
Learning data: 106 hole slide images Verification analysis data: 45 hole slide images

[Creating and learning training data]
Bright-field whole-slide images of HE-stained gastric cancer tissues were color-imaged using a virtual slide scanner (NanoZoomer-XR, (Hamamatsu Photonics; scan resolution: 0.46 μm/pixel in 20x mode scan, 0.23 μm/pixel in 40x mode scan). The imaging magnifications were 1x, 3x, 5x, 10x, 20x, and 40x. A pathologist designated tumor regions, non-tumor regions, and non-tissue regions on the whole-slide images containing the tissues to be studied. The whole-slide images in which each region was designated were divided under the above conditions to be used as training images. Each training image was determined to be a tumor region, a non-tumor region, or a non-tissue region. In this case, for tissue regions in which tumor regions and non-tumor regions were mixed, if the number of pixels designated as tumor regions accounted for 50% or more of the number of pixels corresponding to tissue regions in the training image, the region was designated as a tumor region. In addition, if the ratio of the number of pixels designated as non-tumor regions to the number of pixels corresponding to tissue regions in the training image was 100%, the region was designated as a non-tumor region. For training images containing a mixture of non-tissue and tissue regions, if the tissue region accounts for 0% of the number of pixels in the training image, it is determined to be a non-tissue region, and otherwise it is determined to be a tissue region. For each pixel in each training image determined to be a tumor region, a non-tumor region, or a non-tissue region, a label value that distinguishes each region (tumor region is "1", non-tumor region is "2", and non-tissue region is "0") is assigned, and the first preliminary training data 70R1L, the second preliminary training data 70R2L, and the third preliminary training data 70R3L are generated.
The training images determined to be tumor regions were designated as first training images, the training images determined to be non-tumor regions were designated as second training images 70R2, and the training images determined to be non-tissue regions were designated as third training images 70R3. The color density values of R, G, and B for each training image were converted to 8-bit gradation to generate color density coded maps for R, G, and B, and the created color density coded maps for R, G, and B were combined to generate the first training data 70R2L.
The first training data 74R1, the second training data 74R2, and the third training data 74R3 were generated.

Thereafter, tumor region training data was created by combining the first training data 74R1 and the first preliminary training data 70R1L. Second training data 74R2 and second preliminary training data 7
Non-tumor region training data was created by combining with 0R2L. Third training data 74R3
and the third preliminary training data 70R2L were combined to create non-organized region training data.
The created training data for each region was divided into a window size of 200 x 200 pixels, and a neural network was trained using the training data of the divided window size as an input layer.

Training data for each region was generated for each magnification of the tissue specimen, and the neural network was trained for each magnification.

[Creating an image to be analyzed]
Similar to the training data, whole slide images of bright field images of HE-stained human gastric cancer tissues were color-imaged using a virtual slide scanner to obtain images for analysis. The imaging magnifications were 1x, 3x, 5x, 10x, 20x, and 40x. Then, based on the images for analysis, color density coding maps for each of the colors R, G, and B were created, and the data to be analyzed were generated by combining the color density coding maps for each of the colors R, G, and B that were created. The data to be analyzed was generated for each magnification.

[Analysis result]
Create analysis data with a window size of 200 x 200 pixels from the analysis target data,
The analysis data was input to a trained neural network corresponding to the magnification. Based on the analysis results output from the neural network, the image was classified into tumor regions, non-tumor regions, and non-tissue regions (background), and the tumor regions obtained by the analysis process were displayed in white, the non-tumor regions in gray, and the non-tissue regions in black. The analysis results are shown in FIG. 19.

In FIG. 19, (a) is a whole slide image of a bright field image obtained by staining gastric cancer tissue with HE, and is an image to be analyzed. (b) is a whole slide image in which a tumor region is designated by a pathologist and surrounded by a solid line. (c) shows an analysis result using a training image and an analysis image with a magnification of 1x. It was not possible to accurately indicate each area designated by the pathologist. (d) shows an analysis result using a training image and an analysis image with a magnification of 5 times. The tumor area designated by the pathologist could generally be determined as a tumor area. Furthermore, the non-tumor region specified by the pathologist could generally be determined as a non-tumor region. The non-tissue area designated by the pathologist could generally be determined as a non-tissue area. (e) shows an analysis result using a training image and an analysis image with a magnification of 40 times. The tumor area specified by the pathologist could be determined to some extent as a tumor area. However, in areas designated as non-tumor areas by pathologists, the number of areas determined to be tumor areas was increasing, and the rate of erroneously determining non-tumor areas as tumor areas was high.

The above results indicate that either too low or too high a magnification when acquiring training images and analysis images causes erroneous determinations.

In FIG. 20, (a) is a whole slide image of a bright field image obtained by staining gastric cancer tissue with HE, which is different from that shown in FIG. 19, and is the image to be analyzed. (b) is a whole slide image in which a tumor region is designated by a pathologist and surrounded by a solid line. (c) shows an analysis result using a training image and an analysis image with a magnification of 5 times. The tumor area designated by the pathologist could generally be determined as a tumor area. Furthermore, the non-tumor region specified by the pathologist could generally be determined as a non-tumor region. The non-tissue area designated by the pathologist could generally be determined as a non-tissue area. (d) shows an analysis result using a training image and an analysis image with a magnification of 20 times. As with the case of 5x magnification, each region specified by the pathologist could be generally distinguished.

Regarding the image for analysis at a magnification of 5 times, the content rate of the tumor region in the tissue region present in one whole slide image was calculated according to the following formula.

Content rate of tumor area (%) = {[Number of pixels determined to be a tumor area]/[(Number of pixels determined to be a tumor area) + (Number of pixels determined to be a non-tumor area)]}×100

The tumor area content was 54.5%.

Furthermore, FIG. 21 shows the sensitivity (a) and positive predictive value (b) of each magnification. From this result, it was considered that a magnification of 5 to 20 times is appropriate for the tissue sample when acquiring images for training and images for analysis. In addition, Figure 21 shows the sensitivity (c
) and positive predictive value (d) are shown. Images for analysis were taken at a magnification of 5 times. It was considered that the number of pixels of the window size is preferably from 125 x 125 pixels to 200 x 200 pixels.

The analysis time for one whole slide image was approximately several to 30 minutes.

10 (10A, 10B) Processing unit 20 (20A, 20B, 20C) Processing unit 11, 21 CPU
12, 22 Memory 13, 23 Recording section 14, 24 Bus 15, 25 Interface section 16, 26 Input section 17, 27 Output section 19, 29 GPU
50 Neural network (deep learning algorithm)
50a Input layer 50b Output layer 50c Middle layer 60 Trained neural network (trained deep learning algorithm)
60a Input layer 60b Output layer 60c Intermediate layer 70W1 Hole slide image 70W2 Hole slide image 70W3 Hole slide image 70R1L First preliminary training data 70R2L Second preliminary training data 70R3L Third preliminary training data 72R1r, 72R1g, 72R1b Color density code Color density encoding diagrams 72R2r, 72R2g, 72R2b Color density encoding diagrams 72R3r, 72R3g, 72R3b Color density encoding diagrams 74R1, 74R2, 74R3 Training data 75R1, 75R2, 75R3 Window size training data 76 Color density values 77R1, 77R2, 77R3 True value Image label value 78 Bright field images to be analyzed 79r, 79g, 79b Color density encoding diagram for analysis target 80 Data for analysis 81 Color density value 82 Discrimination result (estimated value of pixel)
83 Analysis result ternary image 84 Region-enhanced image 89 (89a, 89b) Node 98 Recording medium 99 Network 100 Vendor device 100A Deep learning device 100B Integrated image analysis device 101 Training data generation unit 102 Training data input unit 103 Algorithm Update unit 104 Window size database 105 Algorithm database 200 User device 200A Image analysis device 200B Integrated image analysis device 200C Terminal device 201 Analysis data generation unit 202 Analysis data input unit 203 Analysis unit 204 Region detection unit 300, 400 Imaging Apparatus 301, 401 Image sensor 308, 408 Sample tissue 309, 409 Stage W1 Window W2 Window

Claims (21)

An image analysis method for analyzing tissue images using a deep learning algorithm with a neural network structure,
Generating data to be analyzed from an image to be analyzed that includes the tissue to be analyzed, cutting out multiple pixel regions each consisting of a plurality of pixels from the data to be analyzed, and generating data for analysis for each of the cut out pixel regions;
inputting each of the analysis data to the deep learning algorithm;
generating data indicating whether the pixel region is a tumor region by the deep learning algorithm;
including that
The number of pixels in the pixel area is set according to the type of tissue to be analyzed.
Image analysis method. The image analysis method according to claim 1, wherein the data indicating whether the pixel region is a tumor region indicates that the pixel region is a tumor region, that the pixel region is a non-tumor region, or that the pixel region is a region that does not contain tissue. The image to be analyzed is an image of a specimen for tissue diagnosis, and the image to be analyzed includes a hue that is a combination of two or more primary colors.
The image analysis method according to claim 1 or 2. The image to be analyzed is an image obtained by enlarging the tissue to be analyzed from 3 times to 20 times.
The image analysis method according to any one of claims 1 to 3. The pixel area is square, and the size of the pixel area is 200 μm x 200 μm or more and 400 μm x 400 μm or less,
The image analysis method according to any one of claims 1 to 4. generating data indicating a boundary between a tumor cell area and other areas based on data indicating whether the pixel area is a tumor area;
The image analysis method according to any one of claims 1 to 5. generating data indicating a content rate of a tumor area in the tissue to be analyzed based on data indicating whether the pixel area is a tumor area;
The image analysis method according to any one of claims 1 to 6. Generating data indicating a ratio of a tumor area to a non-tumor area in the tissue to be analyzed based on data indicating whether the pixel area is a tumor area;
The image analysis method according to any one of claims 1 to 6. The number of nodes in the input layer of the neural network corresponds to the product of the number of pixels in the pixel area and the number of primary colors of each pixel.
The image analysis method according to any one of claims 1 to 8. The tissue diagnostic specimen is a stained specimen, and the image to be analyzed is an image of the stained specimen taken under a bright field of a microscope.
The image analysis method according to claim 3 . The training data used for learning the deep learning algorithm is a stained image of a specimen prepared by performing bright field observation staining on a tissue specimen including a tumor region collected from an individual under the bright field of a microscope. Generated based on the captured bright field image,
The image analysis method according to any one of claims 1 to 10. The bright field observation staining uses hematoxylin for nuclear staining.
The image analysis method according to claim 11. The bright field observation staining is hematoxylin and eosin staining.
The image analysis method according to claim 11. The training data includes a label value determined from the bright field image indicating a tumor region.
The image analysis method according to any one of claims 11 to 13. the training data includes the label value for each region of a predetermined number of pixels of the bright field image;
The image analysis method according to claim 14. The image analysis method according to any one of claims 11 to 15, wherein the training data is generated for each region of a predetermined number of pixels in the bright field image. the output layer of the neural network is a node whose activation function is a softmax function;
The image analysis method according to any one of claims 1 to 16. The deep learning algorithm generates data indicating whether the pixel region is a tumor region each time the analysis data is input.
The image analysis method according to any one of claims 1 to 17. and processing the analysis data using a deep learning algorithm that corresponds to the type of tissue to be analyzed and is selected from among a plurality of deep learning algorithms according to the type of tissue.
19. The image analysis method according to any one of claims 1 to 18. An image analysis device that analyzes tissue images using a deep learning algorithm with a neural network structure,
Generating data to be analyzed from an image to be analyzed that includes the tissue to be analyzed, cutting out a plurality of pixel regions made up of a plurality of pixels cut out from the data to be analyzed, generating data for analysis for each of the cut out pixel regions,
inputting each of the analysis data to the deep learning algorithm;
a processing unit that uses the deep learning algorithm to generate data indicating whether the pixel region is a tumor region;
Equipped with
The number of pixels in the pixel region is set according to the type of tissue to be analyzed. A computer program for analyzing tissue images using a deep learning algorithm having a neural network structure, comprising:
On the computer,
A process of generating analysis target data from an analysis target image including a tissue to be analyzed, extracting a plurality of pixel regions each consisting of a plurality of pixels from the analysis target data, and generating analysis data for each of the extracted pixel regions;
inputting each of the analysis data into the deep learning algorithm;
and generating data indicating whether the pixel region is a tumor region using the deep learning algorithm;
The number of pixels in the pixel region is set according to the type of tissue to be analyzed.