JP2024038396A - Image analysis method, image analysis device, and program - Google Patents

Abstract

An object of the present invention is to generate data indicating whether a region included in analysis data is a tumor region.
The problem is solved by an image analysis method that analyzes tissue images using a deep learning algorithm (60) with a neural network structure. In the image analysis method, a deep learning algorithm (60) corresponding to the type of tissue to be analyzed is selected from a plurality of deep learning algorithms (60) generated according to the type of tissue, and a deep learning algorithm (60) that includes the tissue to be analyzed is selected. Generate analysis data (80) from the analysis target image (78), input the analysis data (80) to a deep learning algorithm (60), and based on the analysis result output from the deep learning algorithm (60), The image of the tissue to be analyzed is displayed so that tumor areas and other areas can be distinguished.
[Selection diagram] Figure 6

Description

This specification discloses an image analysis method, an image analysis device, a program, a method for manufacturing a trained deep learning algorithm, and a trained deep learning algorithm. More specifically, an image analysis method, an image analysis device, a program, and a learned deep layer include generating data indicating whether or not a region included in analysis data is a tumor region for an image of a tissue to be analyzed. A method for manufacturing a learning algorithm, a trained deep learning algorithm, and the like are disclosed.

Patent Document 1 discloses an image diagnosis support device that classifies and determines tissue images in pathological tissue images into four groups: normal, benign tumor, precancerous state, and cancerous state. The image classification means extracts a region of interest from the image data, calculates a feature amount representing the feature of the region of interest, and classifies the image into groups based on the calculated feature amount. The feature amounts include the density of the clump per unit area in the cell nucleus, the density of the clump area, the area of the clump, the thickness of the clump, and the length of the clump. The image determination means learns the relationship between such feature amounts and determination results, and performs determination based on learned learning parameters. Machine learning is performed using a learning algorithm such as support vector machine.

Japanese Patent Application Publication No. 2010-203949

When definitively diagnosing whether a tumor is malignant or not, histopathological diagnosis using a histopathological specimen is performed. Furthermore, histopathological diagnosis is often performed as a quick intraoperative diagnosis to determine the resection site of tissue containing a malignant tumor during surgery. Rapid intraoperative diagnosis involves having the patient stand by while an incision is made in the affected area during surgery, and determining whether the tumor is malignant, whether there is any tumor remaining in the resected tissue margin, and whether there is lymph node metastasis. This is done through histopathological diagnosis. The results of the intraoperative rapid diagnosis will determine the direction of subsequent surgery for the waiting patient.

Histopathological diagnosis is performed by a doctor, especially a pathologist, by observing tissue specimens using a microscope, etc., but in order to be able to make an accurate and definitive diagnosis by observing tissue specimens, it takes a long period of time for a skilled pathologist to make a diagnosis. Initially, it is necessary to repeatedly observe tissue specimens from various cases, and it takes an enormous amount of time to train pathologists.

There is a serious shortage of pathologists, and as a result of the shortage, there are concerns that a definitive diagnosis of a patient's malignant tumor will be delayed, and the start of treatment will be delayed, or that treatment may be started without waiting for a definitive diagnosis. There is. In addition, because both normal tissue diagnosis and intraoperative rapid diagnosis are concentrated on a small number of pathologists, the workload of each pathologist becomes enormous, and the working conditions of the pathologists themselves are also becoming a problem. However, no solution to this problem has been found so far.

Therefore, it is thought that the ability of a device to support pathological tissue diagnosis will greatly contribute to resolving the shortage of pathologists and improving the working conditions of pathologists, especially as the diagnosis is closer to judgment made by the human eye.

In terms of the device supporting pathological tissue diagnosis, the invention described in Patent Document 1 described above performs pathological determination of sample tissue based on image analysis using machine learning. This method requires the feature values to be created manually. The problem with methods of manually creating feature amounts is that the ability of the person greatly affects the performance of image analysis. Further, the invention described in Patent Document 1 determines whether each cell nucleus in an image of a tissue specimen strongly magnified with a microscope is a nucleus of a cancer cell. Therefore, there is a problem in that it takes a long time to analyze an entire specimen.

Therefore, before a pathologist or machine learning algorithm determines whether an individual cell is the nucleus of a cancer cell, it is necessary to observe the entire tissue contained in a single specimen and determine which part of the specimen contains tumor tissue. Screening for the presence of such substances helps reduce the time required for pathological diagnosis of a single specimen.

SUMMARY OF THE INVENTION An object of the present invention is to provide an image analysis method, an image analysis device, and a program that generate data indicating whether or not a region of a tissue to be analyzed that is included in analysis data is a tumor region.

In order to solve the above problems, an image analysis method according to one aspect of the present invention is an image analysis method that analyzes an image of a tissue using a deep learning algorithm (60) having a neural network structure. The deep learning algorithm (60) corresponding to the type of tissue to be analyzed is selected from a plurality of deep learning algorithms (60) generated according to the type, and an analysis target image (78) containing the tissue to be analyzed is created. generate analysis data (80) from the analysis data (80), input the analysis data (80) to the selected deep learning algorithm (60), and input the analysis data (80) to the deep learning algorithm (60). 60), displaying an image of the tissue to be analyzed in such a manner that a tumor region and other regions can be distinguished from each other.

The plurality of deep learning algorithms (60) are stored in the database (105), and the deep learning algorithm (60) corresponding to the type of tissue to be analyzed is one of the plurality of deep learning algorithms (60) stored in the database (105). deep learning algorithms (60).

Each of the plurality of deep learning algorithms (60) may be stored in the database (105) in association with a type of tissue.

At least one of the plurality of deep learning algorithms (60) may be stored in the database (105) in association with one tissue type.

The deep learning algorithm (60) corresponding to the type of tissue to be analyzed may be selected based on a user's operation via the input unit (26).

The image analysis method performs a drawing process on the analysis target image (78) to display a tumor region and other regions in a distinguishable manner based on the analysis result output from the deep learning algorithm (60). It may further include.

The drawing process may include a process of drawing the tumor area and other areas in different colors.

The analysis target image (78) may be an image of a specimen for tissue diagnosis, and the analysis target image (78) may include a hue that is a combination of two or more primary colors.

The image to be analyzed (78) may be an image obtained by enlarging the tissue to be analyzed from 3 times to 20 times.

The analysis data (80) corresponds to an image cut out from the analysis target image, and may be generated in plural from the analysis target image (78).

The image to be analyzed (78) is an image of a specimen for tissue diagnosis, the specimen for tissue diagnosis is a stained specimen, and the image to be analyzed (78) is an image of the stained specimen in the bright field of a microscope. It may be an image taken below.

The training data used for learning the deep learning algorithm (60) is a stained image of a specimen prepared by performing bright field observation staining on a tissue specimen including a tumor region collected from an individual. It may be generated based on a bright field image captured under a visual field.

The output layer (50b) of the neural network (50) may be a node whose activation function is a softmax function.

In order to solve the above problems, an image analysis device (200) according to one aspect of the present invention is an image analysis device that analyzes tissue images using a deep learning algorithm (60) having a neural network structure. , selects the deep learning algorithm (60) corresponding to the type of tissue to be analyzed from a plurality of deep learning algorithms (60) generated according to the type of tissue, and selects the deep learning algorithm (60) corresponding to the type of tissue to be analyzed. Analysis data (80) is generated from the analysis target image (78) including the tissue, the analysis data (80) is input to the selected deep learning algorithm (60), and the analysis data (80) is ), an image of the tissue to be analyzed is displayed on the display unit (27) in such a way that tumor regions and other regions can be distinguished from each other based on the analysis results output from the deep learning algorithm (60) into which , a control section (20).

The image analysis device (200) further includes a storage unit (23) that stores the plurality of deep learning algorithms (60), and the control unit (20) is configured to store the deep learning algorithms (23) corresponding to the type of tissue to be analyzed. The learning algorithm (60) may be selected from the plurality of deep learning algorithms (60) stored in the storage unit (23).

The storage unit (23) may store each of the plurality of deep learning algorithms (60) in association with the type of tissue.

The storage unit (23) may store at least one of the plurality of deep learning algorithms (60) in association with one tissue type.

The image analysis device (200) further includes an input section (26) operated by a user, and the control section (20) controls the analysis target tissue based on the operation via the input section (26). The deep learning algorithm (60) corresponding to the type may be selected.

The control unit (20) may perform a drawing process on the analysis target image (78) to display a tumor region and other regions in a distinguishable manner based on the generated analysis data.

The drawing process may include a process of drawing the tumor area and other areas in different colors.

The analysis data (80) corresponds to an image cut out from the analysis target image, and may be generated in plural from the analysis target image (78).

In order to solve the above problems, a computer program according to one aspect of the present invention is a computer program that analyzes an image of a tissue using a deep learning algorithm (60) having a neural network structure. A process of selecting the deep learning algorithm (60) corresponding to the type of tissue to be analyzed from a plurality of deep learning algorithms (60) generated according to the type of the tissue, and an analysis target image including the tissue to be analyzed. (78) to generate analysis data (80); input the analysis data (80) into the selected deep learning algorithm (60); and input the analysis data (80). Based on the analysis result output from the deep learning algorithm (60), a process of displaying an image of the tissue to be analyzed in such a manner that a tumor region and other regions can be distinguished is executed.

It is possible to observe the entire tissue contained in a specimen and generate data for screening which part of the specimen contains tumor tissue.

FIG. 2 is a schematic diagram for explaining an overview of a deep learning method. FIG. 2 is a schematic diagram for explaining an overview of a deep learning method. FIG. 2 is a schematic diagram for explaining an overview of a deep learning method. FIG. 3 is a schematic diagram for explaining details of training data. FIG. 3 is a schematic diagram for explaining details of training data. FIG. 2 is a schematic diagram for explaining an overview of an image analysis method. FIG. 1 is a schematic configuration diagram of an image analysis system according to a first embodiment. 1 is a block diagram showing the hardware configuration of a vendor-side device 100. FIG. 2 is a block diagram showing the hardware configuration of a user-side device 200. FIG. FIG. 2 is a block diagram for explaining the functions of a deep learning device 100A according to the first embodiment. It is a flowchart which shows the procedure of deep learning processing. FIG. 2 is a schematic diagram for explaining details of learning by a neural network. FIG. 2 is a block diagram for explaining the functions of an image analysis device 200A according to the first embodiment. It is a flowchart which shows the procedure of image analysis processing. FIG. 2 is a schematic configuration diagram of an image analysis system according to a second embodiment. FIG. 2 is a block diagram for explaining the functions of an integrated image analysis device 200B according to a second embodiment. FIG. 2 is a schematic configuration diagram of an image analysis system according to a third embodiment. It is a block diagram for explaining the function of integrated type image analysis device 100B concerning a 3rd embodiment. These are the analysis results of tissue specimens obtained from gastric cancer tissue. (a) is a whole slide image of a bright field image obtained by staining gastric cancer tissue with HE, and is an image to be analyzed. (b) is a whole slide image in which a tumor region is designated by a pathologist and surrounded by a solid line. (c) shows an analysis result using a training image and an analysis image with a magnification of 1x. (d) shows an analysis result using a training image and an analysis image with a magnification of 5 times. (e) shows an analysis result using a training image and an analysis image with a magnification of 40 times. In FIGS. 19(c) to (e), the tumor region obtained by the analysis process is shown in white, the non-tumor region is shown in gray, and the non-tissue region is shown in black. These are the analysis results of tissue specimens obtained from gastric cancer tissue. (a) is a whole slide image of a bright field image obtained by staining gastric cancer tissue with HE, and is an image to be analyzed. (b) is a whole slide image in which a tumor region is designated by a pathologist and surrounded by a solid line. (c) shows an analysis result using a training image and an analysis image with a magnification of 5 times. (d) shows an analysis result using a training image and an analysis image with a magnification of 16 times. In FIGS. 19(c) and 19(d), the tumor region obtained by the analysis process is shown in white, the non-tumor region is shown in gray, and the non-tissue region is shown in black. FIG. 7 is a diagram showing the difference in discrimination accuracy depending on the magnification of the tissue sample and the number of pixels of the window size. (a) shows the sensitivity of each magnification, and (b) shows the positive predictive value of each magnification. (c) shows the sensitivity due to the difference in the number of pixels of the window size, and (d) shows the change in the positive predictive value.

DESCRIPTION OF THE PREFERRED EMBODIMENTS An outline and embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that in the following description and drawings, the same reference numerals indicate the same or similar components, and therefore, explanations regarding the same or similar components will be omitted.

The image analysis method analyzes images of tissue. The image analysis method uses a deep learning algorithm of a neural network structure, preferably a convolutional neural network structure. The image analysis method includes generating data indicating whether a region included in the analysis data is a tumor region.

In the present invention, an image of a tissue or a cell is an image obtained from a tissue sample specimen or a sample specimen containing cells. A specimen of a tissue sample or a specimen containing cells is taken from an individual. Although the individual is not particularly limited, it is preferably a mammal, more preferably a human. It does not matter whether the individual is alive or dead when the sample is taken from the individual. The tissue is not limited as long as it exists within an individual. Examples of the tissue collected from the individual include surgically excised tissue, biopsy tissue, and the like. The tumor may be either epithelial or non-epithelial. The tumor is preferably a malignant epithelial tumor. Although malignant tumors are not particularly limited, examples of malignant tumors include respiratory malignant tumors arising from the trachea, bronchi, or lungs; nasopharynx, esophagus, stomach, duodenum, jejunum, ileum, cecum, appendix, and ascending colon. , gastrointestinal malignant tumors arising from the transverse colon, sigmoid colon, rectum, or anus; liver cancer; pancreatic cancer; urinary system malignant tumors arising from the bladder, ureter, or kidney; ovaries, fallopian tubes, uterus, etc. Malignant tumors of the female reproductive system that occur; breast cancer; prostate cancer; skin cancer; malignant tumors of the endocrine system such as the hypothalamus, pituitary gland, thyroid, parathyroid gland, and adrenal gland; malignant tumors of the central nervous system; malignant tumors that arise from bone and soft tissue, etc. solid tumors. More preferably, respiratory epithelial malignant tumors such as lung cancer (squamous cell carcinoma, small cell carcinoma, large cell carcinoma, adenocarcinoma); gastric cancer, duodenal cancer, colorectal cancer (sigmoid colon cancer, rectal cancer, etc.), etc. Examples include gastrointestinal epithelial malignant tumors; liver cancer; pancreatic cancer; bladder cancer; thyroid cancer; ovarian cancer; breast cancer; and prostate cancer. Most preferred is gastric cancer.

The specimen is intended to be one in which the tissue has been processed so that it can be observed with a microscope, for example, a preparation. The specimen can be prepared according to a known method. For example, in the case of a tissue specimen, after collecting the tissue from the individual, the tissue is fixed with a predetermined fixative (formalin fixation, etc.), the fixed tissue is embedded in paraffin, and the paraffin-embedded tissue is sliced. . Place the thin section on a glass slide. The glass slide on which the section is mounted is stained for observation with an optical microscope, that is, for bright field observation, and the specimen is completed by a predetermined mounting process. A typical example of a tissue specimen is a specimen for tissue diagnosis (pathological specimen), and the staining is hematoxylin and eosin (HE) staining.

The nuclear stain for HE staining is hematoxylin. Hematoxylin is widely used as a nuclear stain in tissue cell staining (eg, immunostaining, lectin staining, sugar staining, fat staining, collagen fiber staining, etc.). Therefore, the present invention can be applied to all kinds of specimens in which hematoxylin is used for nuclear staining.

The image analysis uses a deep learning algorithm trained using training images. The image analysis generates analysis data from an analysis target image that includes a tissue to be analyzed and is acquired from the specimen. The analysis data is input to the deep learning algorithm to generate data indicating whether or not a region in which the tissue present in the analysis target image is included in the analysis data is a tumor region. Data indicating whether or not a region included in the analysis data is a tumor region means that there is a possibility that the region including multiple pixels of interest included in the tissue to be determined contains tumor tissue and/or a non-tumor region. This is data indicating the possibility that The data indicating whether or not a region included in the analysis data is a tumor region may be a label value, a display, etc. that can distinguish whether a region includes a tumor region or a non-tumor region.

The training images are acquired from a plurality of regions of a sample to be studied. Preferably, the training images are acquired from a region including a tumor region, a region including a non-tumor region, and a region not including tissue for each layer structure from a plurality of layer structures included in the tissue to be learned.

For example, the first training image 70R1 is an image acquired from a region including a tumor region included in tissue collected from an individual. This image is obtained by microscopic observation from a specimen that has been stained so that its tissue structure can be recognized. The staining is not limited as long as the tissue structure can be recognized, but staining for bright field observation is preferable. The staining for bright field observation is not limited as long as at least the cell nucleus and a region other than the cell nucleus can be stained in a distinguishable manner by hue. When the specimen is a mammalian tissue specimen, for example, HE staining can be used.

First training data 74R1 and first preliminary training data 70R1L are generated from the first training image 70R1. The first training data 74R1 is information regarding a single color image in which the hues included in the first training image 70R1 are separated for each primary color. The first preliminary training data 70R1L is generated as binarized data (label value) indicating that the tissue region included in the first training image 70R1 includes a tumor region. Whether or not the tissue region included in the first training image 70R1 includes a tumor region is determined by, for example, specimen observation by a pathologist or the like. Here, in the tissue region included in the training image, tumor cells, non-tumor cells, and regions of tissue other than cells may coexist. In this case, the tissue region included in the training image is such that, for example, when the area of the entire tissue region included in the training image is 100%, tumor cells occupy approximately 10% or more of the total area; Includes a tumor region if it occupies about 20% or more, about 30% or more, about 40% or more, about 50% or more, about 60% or more, about 70% or more, about 80% or more, or about 90% or more can be determined. Preferably, it can be determined that a tumor region is included when it occupies about 40% or more, about 50% or more, about 60% or more, about 70% or more, about 80% or more, or about 90% or more. The ratio of the tumor cells to the total area of the tissue is calculated by subtracting the ratio of non-tumor cells and tissues other than cells (non-tumor area) from the area (100%) of the entire tissue region included in the training image. It may be determined as

The second training image 70R2 included in the training images is an image acquired from a tissue region including a non-tumor region (preferably a region including non-tumor cells and a region of tissue other than cells). . This image is obtained by microscopic observation from a specimen that has been stained so that its tissue structure can be recognized. The staining is not limited as long as the tissue structure can be recognized, but staining for bright field observation is preferable. The staining for bright field observation is not limited as long as at least the cell nucleus and a region other than the cell nucleus can be stained in a distinguishable manner by hue. When the specimen is a mammalian tissue specimen, for example, HE staining can be used. Preferably, the staining is the same as that of the specimen from which the first training image 70R1 was obtained.

Second training data 74R2 and second preliminary training data 70R2L are generated from the second training image 70R2. The second training data 74R2 is information regarding a single color image in which the hues included in the second training image 70R2 are separated for each primary color. The second preliminary training data 70R2L is binarized data (label value) indicating that the tissue region included in the second training image 70R2 includes a non-tumor region. Whether or not the tissue region included in the second training image 70R2 includes a non-tumor region is determined by, for example, specimen observation by a pathologist or the like. Here, in the tissue region included in the training image, tumor cells, non-tumor cells, and regions of tissue other than cells may coexist. In this case, the area of tissue included in the training image is defined as non-tumor cells and areas of tissue other than cells (non-tumor area) is about 10% or more, about 20% or more, about 30% or more, about 40% or more, about 50% or more, about 60% or more, about 70% or more, about 80% or more, about If it occupies 90% or more or 100%, it can be determined that a non-tumor area is included. Preferably, it is determined that a non-tumor region is included if it occupies about 40% or more, about 50% or more, about 60% or more, about 70% or more, about 80% or more, about 90% or more, or 100%. Can be done. The area of the entire tissue including the non-tumor cells and non-cell tissue is calculated by subtracting the proportion occupied by the tumor region from the area (100%) of the entire tissue region included in the second training image 70R2. It may also be determined as a percentage.

The third training image 70R3 included in the training images is an area that does not include tissue (also referred to as a non-tissue area) in the sample from which the first training image 70R1 and/or the second training image 70R2 was obtained. This is an image obtained from ).

Third training data 74R3 and third preliminary training data 70R3L are generated from the third training image. The third training data 74R3 is information regarding a single color image in which the hues included in the third training image 70R3 are separated for each primary color. The third preliminary training data 70R2L is binarized data (label value) indicating that the region included in the third training image 70R3 is a region that does not include tissue. The area included in the third training image 70R3 is determined by, for example, specimen observation by a pathologist or the like. Here, the training image may include a tissue region and a background region (for example, a glass portion of a slide). In this case, the background area included in the training image is about 90% or more, about 91%, for example, when the area of the entire area included in the one training image is 100%. or more than about 92%, about 93% or more, about 94% or more, about 95% or more, about 96% or more, about 97% or more, about 98% or more, or about 99% or more, or the background area occupies 100%, it can be determined that the area does not contain tissue. Preferably, the tissue is grown when the background area occupies about 95% or more, about 96% or more, about 97% or more, about 98% or more, or about 99% or more, or when the background area occupies 100%. It can be determined that the area does not include the above.

It is preferable that each training image is obtained as an image of a training tissue specimen enlarged, for example, from 5 times to 20 times. From the viewpoint of discrimination accuracy, images enlarged 30 times or more, particularly 40 times or more, are not suitable as training images.

Tumor region training data is generated from the first training data 74R1 and the first preliminary training data 70R1L, non-tumor region training data is generated from the second training data 74R2 and the second preliminary training data 70R2L, and the first training data 74R1 and the first preliminary training data 70R1L generate tumor region training data. Non-organized region training data is generated from the third training data 74R3 and the third preliminary training data 70R2L. Tumor region training data, non-tumor region training data, and non-tissue region training data may be collectively referred to as training data.

In the outline and embodiments of the present invention, an example will be described in which a deep learning algorithm is used to determine whether tissues included in an image obtained by capturing an HE-stained tissue specimen include a tumor region, a non-tumor region, and a non-tissue region. It will be explained as follows.

[Overview of deep learning methods and image analysis methods]
First, we will explain the deep learning method.

- Outline of deep learning method The outline of the first training data 74R1, the first preliminary training data 70R1L, and the tumor region training data 75R1 will be explained using FIG. FIG. 1 shows an example of inputting training data to a neural network using a whole slide image 70W1 of a tissue specimen obtained by a slide image scanner. The whole slide image 70W1 is a slide image of a tissue specimen obtained by imaging a specimen prepared by applying HE staining as a bright field observation stain in a bright field. The whole slide image 70W2 is an image in which the pathologist has designated a portion of the whole slide image 70W1 that includes the tumor region by surrounding it with a solid line. The whole slide image 70W3 is an image obtained by dividing the whole slide image 70W2 as a training image (for example, into 512 parts). An area surrounded by a rectangular frame indicated by the symbol R1 in the hole slide image 70W3 indicates an area used as the first training image 70R1 shown in FIG. 1. The area surrounded by a rectangular frame indicated by the symbol R2 indicates an area used as the second training image 70R2 shown in FIG. 2. The area surrounded by a rectangular frame indicated by symbol R3 indicates an area used as the third training image 70R3 shown in FIG. 3. The fact that the first training image 70R1 includes a tumor region, that the second training image 70R2 includes a non-tumor region, and that the third training image 70R3 is a region that does not include tissue means that each The determination may be made before acquiring the training images, or after acquiring each training image.

In FIG. 1, the first training image 70R1 is obtained by capturing an HE-stained specimen as a color image under bright-field observation, and therefore includes a plurality of hues.

The first training image 70R1 can be acquired in advance using, for example, an image acquisition device such as a known optical microscope, fluorescence microscope, or virtual slide scanner. Illustratively, in the present embodiment, it is preferable that the color image acquired from the image acquisition device is a 24-bit color image with a color space of RGB. In 24-bit RGB color, it is preferable that the depth (color density) of each of red, green, and blue be represented by 8-bit (256 levels) gradations. The first training image 70R1 may be an image containing one or more primary colors. First training data 74R1 is generated corresponding to the first training image 70R1.

In the present invention, hue is illustratively defined by a combination of three primary colors of light or a combination of three primary colors of color. The first training data 74R1 is generated from the first training image 70R1 by separating the hue appearing in the first training image 70R1 into individual primary colors, and generating a code for each primary color according to the density. This is the data expressed as . In Figure 1, an image (hereinafter also referred to as a "single-color image") 72R1R is shown in shading of a single color separated for each of the three primary colors of light, red (R), green (G), and blue (B). , 72R1G and 72R1B are obtained.

The color density of each color is encoded for each pixel on the single color images 72R1R, 72R1G, and 72R1B, and the entire image is converted into an encoding diagram (hereinafter referred to as " 72R1r, 72R1g, and 72R1b are generated. The color density may be encoded as a numerical value indicating 256 levels for each color. Furthermore, the color density may be further preprocessed on the numerical values indicating 256 levels of each color, and the color density at each pixel may be encoded by numbers indicating 8 levels from value 0 to value 7, for example. For convenience of explanation, the color density encoding diagrams 72R1r, 72R1g, and 72R1b generated from single-color images of R, G, and B shown as an example in FIG. It is expressed by a stage code. First training data 74R1 in which R, G, and B color density values are combined for each pixel is generated from the color density encoding diagrams 72R1r, 72R1g, and 72R1b shown in FIG. The code indicating color density is also referred to as color density value in this specification. Further, as the first training data 74R1, a matrix of color density values corresponding to each pixel may be generated instead of the color density encoded diagram.

In FIG. 1, the first preliminary training data 70R1L is data indicating that the tissue included in the first training image 70R1 includes a tumor region. For example, in the first training image 70R1, since the entire region is determined to be a tumor region by a pathologist, the first training image 70R1 includes a tumor region for each region of a predetermined number of pixels, which will be described later. The same numerical value, for example, "1" is assigned as a label value indicating that the region is included, and the data becomes first preliminary training data 70R1L.

In the deep learning method, tumor region training data 75R1 is generated from first training data 74R1 and first preliminary training data 70R1L shown in FIG. Then, the color density value data 76 (obtained from the first training data 74R1) of each pixel of the tumor region training data 75R1 is used as the input layer 50a, and the label value 77R1 (obtained from the first preliminary training data 70R1L) is output. The neural network 50 serving as the layer 50b is caused to learn.

Next, an outline of the second training data 74R2, the second preliminary training data 70R2L, and the non-tumor region training data 75R2 will be explained using FIG. In FIG. 2, the whole slide image 70W1, the whole slide image 70W2, and the whole slide image 70W3 are the same as those in FIG. Contains hue.

The second training image 70R2 is obtained in the same manner as the first training image 70R1, except that tissue including a non-tumor region is used instead of tissue including a tumor region. Second training data 74R2 is generated corresponding to the second training image 70R2.

The second training data 74R2 is generated in the same manner as the first training data 74R1, except that tissue including a non-tumor region is used instead of tissue including a tumor region. Second training data 74R2 in which R, G, and B color density values are combined for each pixel is generated from the color density encoding diagrams 72R2r, 72R2g, and 72R2b shown in FIG.

In FIG. 2, the second preliminary training data 70R2L is data indicating that the second training image 70R2 includes a non-tumor region. Since the entire region of the second training image 70R2 is determined to be a non-tumor region by the pathologist, the second training image 70R2 includes non-tumor regions for each region of a predetermined number of pixels, which will be described later. The same numerical value, for example, "2" is assigned as a label value indicating that the area is included, and the data becomes second preliminary training data 70R2L. A label value indicating a region including a non-tumor region is not limited as long as it can be distinguished from a label value indicating a region including a tumor region and a region not including tissue.

In the deep learning method, non-tumor region training data 75R2 is generated from second training data 74R2 and second preliminary training data 70R2L shown in FIG. Then, the color density value data 76 (obtained from the second training data 74R2) of each pixel of the non-tumor region training data 75R2 is used as the input layer 50a, and the label value 77R2 (obtained from the second preliminary training data 70R2L) is set as the input layer 50a. The neural network 50 serving as the output layer 50b is made to learn.

Next, an outline of the third training data 74R3, the third preliminary training data 70R3L, and the non-organized region training data 75R3 will be explained using FIG. In FIG. 3, the hole slide image 70W1, the hole slide image 70W2, and the hole slide image 70W3 are the same as in FIG. Contains hue.

The third training image 70R3 is obtained in the same manner as the first training image 70R1, except that a region that does not include tissue is used instead of a region that includes a tumor region. Third training data 74R3 is generated corresponding to the third training image 70R3.

The third training data 74R3 is generated in the same manner as the first training data 74R1, except that a region containing no tissue is used instead of a region containing a tumor region. Third training data 74R3 in which R, G, and B color density values are combined for each pixel is generated from the color density encoding diagrams 72R3r, 72R3g, and 72R3b shown in FIG.

In FIG. 3, the third preliminary training data 70R3L is data indicating that the region included in the third training image 70R3 is a region that does not include tissue. Since the entire region of the third training image 70R3 has been determined by the pathologist to be a non-tissue region, the third training image 70R3 includes tissues in each region of a predetermined number of pixels, which will be described later. The same numerical value, for example "0", is assigned as a label value indicating that the area is not included, and becomes third preliminary training data 70R3L. A label value indicating that the region does not contain tissue is distinguished from a numerical value indicating a region containing a tumor region and a region containing a non-tumor region.

In the deep learning method, non-organized region training data 75R3 is generated from the third training data 74R3 and third preliminary training data 70R3L shown in FIG. (obtained from the third training data 74R3) as the input layer 50a, and the label value 77R3 (obtained from the third preliminary training data 70R3L) as the output layer 50b.

A method of generating the tumor region training data 75R1 will be described with reference to FIGS. 4(a) and 4(b). The first training data 74R1 shown in FIG. 4A is data that combines the color density values of the color density encoded diagrams 72R1r, 72R1g, and 72R1b of a tumor region with a predetermined number of pixels. In this specification, for convenience, the position of each pixel in the first training data 74R1 is expressed by l ₁ , l _{2 .} . . l _i in columns from the left, and m ₁ , _{m 2} . . In FIG. 4A, the image size (size per training data) of the first training data 74R1 is simplified for convenience of explanation, and the first training data 74R1 is vertically It has a total of 81 pixels, 9 pixels in the direction and 9 pixels in the horizontal direction. That is, the position of each pixel is represented by columns l ₁ , l _{2 .} . . l ₉ from the left, and rows m ₁ , m _{2 .} . . m ₉ from the top. The three values shown in FIG. 4A are the color density values of R, G, and B in each pixel. Illustratively, the three values are stored in the order of red (R), green (G), and blue (B) from the left. The color density value of each pixel of the first training data 74R1 is shown in eight levels from value 0 to value 7 for convenience. This is an example of image preprocessing, in which the brightness of each color image 72R1R, 72R1G, 72R1B, which is expressed in 256 levels when captured, is converted into color density values in 8 levels. As for the color density value, for example, the lowest brightness (a gradation group with a low brightness value when expressed in 256 RGB color levels) is set to a color density value of 0, and gradually higher values are assigned as the degree of brightness increases. Then, the highest brightness (gradation group with a high brightness value when expressed in 256 RGB color levels) is set to a color density value of 7.

The tumor region training data 75R1 shown in FIGS. 1 and 4(b) is based on the first training data 74R1 shown in FIG. A label value 77R1 corresponding to the label value of the first preliminary training data 70R1L is attached to the cut-out data. The window-sized tumor region training data 75R1 is also shown simplified to 3×3 pixels (l ₁ :l ₃ , m ₁ :m ₃ ) for convenience of explanation, but the position of each pixel is shown with the column on the left for convenience. From l ₁ , l ₂ ... l _i , the rows are represented by m ₁ , m ₂ ... m _j from the top. Actual preferred window sizes are illustratively 125 x 125 pixels (i = 125, j = 125), 150 x 150 pixels (i = 150, j = 150), 166 x 166 pixels (i = 166, j = 166), 180 x 180 pixels (i = 180, j = 180), 200 x 200 pixels (i = 200, j = 200), 220 x 220 pixels (i = 220, j = 220), 250 x 250 pixels ( i=250, j=250) is preferable in terms of learning efficiency. More preferably, the window size is about 200×200 pixels. The window size is, for example, in a range of 60 μm×60 μm or more and 500 μm×500 μm or less. Preferably 60 μm x 60 μm, 100 μm x 100 μm, 150 μm x 150 μm, 200 μm x 200 μm, 250 μm x 300 μm, 300 μm x 300 μm, 350 μm x 350 μm, 400 μm x 400 μm, 450 μm x 450 μm x 450 μm. , 500 μm x 500 μm. More preferably, the size is 200 μm×200 μm to 400 μm×400 μm. For example, as shown in FIG. 4(b), a 3×3 pixel window W1 (l ₁ :l ₃ , m ₁ :m ₃ ) is set from the left end of the first training data 74R1, and the first training data The window W1 is moved from left to right with respect to 74R1. After cutting out the tumor region training data 75R1 of (l ₁ :l ₃ , m ₁ :m ₃ ) in the window W1, the label value 77R1 corresponding to the label value of the first preliminary training data 70R1L is added to the tumor region training data 75R1. attach The window W1 is moved to the next window (l ₄ :l ₆ ,m ₄ :m ₆ ) of the tumor region training data 74R1 shown by the dotted line, and the window size of (l ₄ :l ₆ ,m ₄ :m ₆ ) is changed to the tumor area of the window size. It is extracted as region training data 75R1. A label value 77R1 corresponding to the label value of the first preliminary training data 70R1L is attached to the newly extracted tumor region training data 75R1. This operation is repeated to cut out tumor region training data 75R1 of a plurality of window sizes from the first training data 74R1. The tumor region training data 75R1 of the cut out window size is given a label value 77R1 corresponding to the label value of the first preliminary training data 70R1L, and is used for learning of the neural network 50 shown in FIG. 1.

As shown in FIG. 1, the number of nodes in the input layer 50a of the neural network 50 is determined by the number of pixels of the tumor region training data 75R1 of the input window size and the number of primary colors included in the image (for example, in the case of the three primary colors of light, R, G, and B). The color density value data 76 of each pixel of the window-sized tumor region training data 75R1 is used as the input layer 50a of the neural network, and the label value 77R1 corresponding to the first preliminary training data 70R1L is used as the output layer 50b of the neural network. 50 to learn. The color density value data 76 of each pixel is a collection of color density values of each color of R, G, and B of each pixel of the tumor region training data 75R1. As an example, if the window size tumor region training data 75R1 is 3 x 3 pixels, one color density value is given for each R, G, and B for each pixel, so the color density value data 76 is The number of color density values is "27" (3x3x3=27), and the number of nodes in the input layer 50a of the neural network 50 is also "27".

In this way, the window-sized tumor region training data 75R1 to be input to the neural network 50 can be automatically created by the computer without being created by the user. This facilitates efficient deep learning of the neural network 50.

As shown in FIG. 4(b), in the initial state, the window W1 is located at the upper left corner of the first training data 74R1. Thereafter, the tumor region training data 75R1 of the window size is cut out using the window W1, and the position of the window W1 is moved every time the neural network 50 is trained. Specifically, the window W1 is moved so that the window W1 scans, for example, all pixels of the first training data 74R1. Thereby, the tumor region training data 75R1 of the window size cut out from all pixels of the first training data 74R1 is used for learning of the neural network 50. Therefore, the degree of learning of the neural network 50 can be improved, and as a result of deep learning, a deep learning algorithm having the structure of the neural network 60 shown in FIG. 6 can be obtained.

FIG. 5(a) shows second training data 74R2. The method of generating non-tumor region training data 75R2 from the second training data 74R2 is to replace the first training data 74R1 and the first preliminary training data 70R1L with the second training data 74R2 and the second preliminary training data. This is similar to the tumor region training data 75R1 except that 70R2L is used. A label value 77R2 corresponding to the label value of the second preliminary training data 70R2L attached to the non-tumor region training data 75R2 is input to the output layer 50b shown in FIG. 2.

FIG. 5(b) shows third training data 74R3. The method of generating the non-organized region training data 75R3 from the third training data 74R3 is to replace the first training data 74R1 and the first preliminary training data 70R1L with the third training data 74R3 and the third preliminary training data. This is similar to the tumor region training data 75R1 except that 70R3L is used. A label value 77R3 corresponding to the label value of the third preliminary training data 70R3L attached to the non-organized region training data 75R3 is input to the output layer 50b shown in FIG. 3.

It is preferable that each region training data is generated using 10 or more, 20 or more, 50 or more, or 100 or more training images of each region.

- Outline of image analysis method As shown in FIG. 6, in the image analysis method, analysis data 80 is generated from an analysis target image (bright field image) 78 obtained by capturing a specimen including a tissue to be analyzed. Preferably, the specimen is stained with the same staining as the first training image. The image to be analyzed 78 can also be obtained as a color image using, for example, a known microscope or virtual slide scanner. It is preferable that the analysis target image 78 is acquired at the same degree or the same magnification as the training image. The image to be analyzed (bright field image) 78 may be any image containing one or more primary colors. When the color analysis target image 78 is encoded using the color density values of R, G, and B for each pixel, the entire image can be represented as an encoded diagram of the color density values of each pixel for each R, G, and B. (Analytical color density encoding diagrams 79r, 79g, 79b). Analysis target data (not shown) in which R, G, and B color density values are combined for each pixel is generated from the analytical color density coding diagrams 79r, 79g, and 79b. The color density coding diagrams 79r, 79g, and 79b showing the signs of color density in the single color images of each of R, G, and B, exemplarily shown in FIG. Color density values are displayed using codes in eight stages from value 0 to value 7. In this specification, for convenience, the position of each pixel of the data to be analyzed is represented by columns l ₁ , l ₂ . . . l _i from the left, and rows m ₁ , _{m 2} .

The analysis data 80 is data obtained by cutting out the analysis target data into a region with a predetermined number of pixels (that is, the same window size as the training data for each region described above), and the color density value of the tissue included in the analysis target image 78. This is data that includes. In FIG. 6, window size analysis data 80 is also simplified to 3×3 pixels for convenience of explanation, similar to tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3. As shown, the actual preferred window size is similar to tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3. For example, a 3×3 pixel window W2 is set and the window W2 is moved relative to the data to be analyzed. The window W2 includes a predetermined number of pixels in the data to be analyzed, and if the data to be analyzed is cut out using the window W2 indicated by a black frame of, for example, 3 x 3 pixels in the same manner as the training data for each region, the window size is Analysis data 80 is obtained. A plurality of pieces of analysis data 80 are generated from the data to be analyzed for each area of a predetermined number of pixels corresponding to the window size in the same manner as the training data for each area. In the analysis target data and window size analysis data 80, the color density value of each pixel is red (R), similarly to the tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3. They are stored in the order of green (G) and blue (B).

The image analysis method uses a deep learning algorithm 60 having a neural network trained using tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3 with window sizes shown in FIGS. 1 to 3. Then, the analysis data 80 is processed. By processing the analysis data 80, data 83 indicating whether or not a region included in the analysis data in the tissue to be analyzed is a tumor region is generated.

Referring again to FIG. 6, the analysis data 80 cut out from the analysis target data generated based on the color density encoding diagrams 79r, 79g, and 79b for each of R, G, and B colors forms a neural network 60 that constitutes a deep learning algorithm. is input. The number of nodes in the input layer 60a of the neural network 60 corresponds to the product of the number of input pixels and the number of primary colors included in the image. When the color density value data 81 of each pixel of the analysis data 80 is input to the neural network 60, the estimated value 82 (ternary value) of the analysis data 80 is output from the output layer 60b. For example, an estimated value of 1 indicates that a region (pixel region) of a predetermined number of pixels constituting the analysis data 80 includes a tumor region, and an estimated value of 2 indicates that the pixel region includes a non-tumor region. , and when the estimated value is 0, it indicates that the pixel region is a non-tissue region. That is, the estimated value 82 outputted from the output layer 60b of the neural network 60 is a label value generated for each pixel region of the image to be analyzed, and the estimated value 82 is a label value generated for each pixel region of the image to be analyzed, and the region included in the analysis data in the image to be analyzed is a tumor region. This is data indicating whether or not there is. The estimated value 82 is also referred to as a class in the following description of the neural network. The neural network 60 assigns a label value indicating that a tumor region is included and a non-tumor region to the input analysis data 80 based on color density value data 81 of a predetermined number of pixels included in the analysis data 80. Generate a label value to indicate containment or a label value to indicate non-tissue region. In other words, the neural network 60 classifies the analysis data 80 into classes indicating the state of the pixel region of interest included in the analysis target image (tumor, non-tumor, tissue). Here, the color density value data 81 of each pixel is a collection of color density values of each of the R, G, and B colors of each pixel of the analysis data 80.

Thereafter, the analysis data 80 is cut out at the window size while moving the window W2 in units of a predetermined number of pixels so that the window W2 scans all pixels of the data to be analyzed. The extracted analysis data 80 is input to the neural network 60. As a result, a label value 83 is obtained based on data indicating whether the region included in the tissue analysis data in the image to be analyzed is a tumor region. In the example shown in FIG. 6, by further performing region detection processing on the label value 83, a region-enhanced image 84 indicating the region is obtained. Specifically, the area detection process is a process of detecting each area according to the estimated value 82, for example, and actually identifies a pixel area where the estimated value 82 is "1" as a tumor area, and a pixel area where the estimated value 82 is "2". This is a process for determining a pixel region as a non-tumor region and a pixel region for which the estimated value 82 is 0 as a non-tissue region. The region-enhanced image 84 is a diagram in which regions detected by the image analysis process are represented by label values 83 in colors (white for tumor regions, gray for non-tumor regions, and black for non-tissue regions). Further, after each region is discriminated, processing may be performed to display the tumor region and other regions (namely, non-tumor region and/or non-tissue region) on the display device in a distinguishable manner. For example, processing such as filling the tumor region with color or drawing lines between the tumor region and other regions is performed, and these are displayed on the display device in a distinguishable manner. Further, a process may be performed to display the non-tumor region and/or the non-tissue region in a distinguishable manner on the display device.

<First embodiment>
In the first embodiment, the configuration of a system that implements the deep learning method and image analysis method described in the above overview will be specifically described.

[Configuration overview]
Referring to FIG. 7, the image analysis system according to the first embodiment includes a deep learning device 100A and an image analysis device 200A. The vendor device 100 operates as a deep learning device 100A, and the user device 200 operates as an image analysis device 200A. The deep learning device 100A causes the neural network 50 to learn using training data, and provides the user with a deep learning algorithm 60 trained using the training data. The deep learning algorithm composed of the trained neural network 60 is provided from the deep learning device 100A to the image analysis device 200A via the storage medium 98 or the network 99. The image analysis device 200A analyzes the image to be analyzed using a deep learning algorithm configured by the trained neural network 60.

The deep learning device 100A is composed of, for example, a general-purpose computer, and performs deep learning processing based on a flowchart described later. The image analysis device 200A is composed of, for example, a general-purpose computer, and performs image analysis processing based on a flowchart described later. The storage medium 98 is a computer-readable non-transitory tangible storage medium such as a DVD-ROM or a USB memory.

The deep learning device 100A is connected to the imaging device 300. The imaging device 300 includes an imaging element 301 and a fluorescence microscope 302, and captures a bright field image and a fluorescence image of a learning specimen 308 set on a stage 309. The learning specimen 308 is stained as described above. The deep learning device 100A acquires training images captured by the imaging device 300.

Image analysis device 200A is connected to imaging device 400. The imaging device 400 includes an imaging device 401 and a fluorescence microscope 402, and captures a bright field image of a specimen 408 to be analyzed set on a stage 409. The specimen 408 to be analyzed has been stained in advance as described above. The image analysis device 200A acquires an analysis target image 78 captured by the imaging device 400.

As the imaging devices 300 and 400, a known fluorescence microscope, virtual slide scanner, or the like, which has a function of imaging a specimen, can be used. The imaging device 400 may be an optical microscope as long as it has the function of imaging a specimen.

[Hardware configuration]
Referring to FIG. 8, the vendor device 100 (100A, 100B) includes a processing section 10 (10A, 10B), an input section 16, and an output section 17.

The processing unit 10 includes a CPU (Central Processing Unit) 11 that performs data processing, which will be described later, a memory 12 that is used as a work area for data processing, and a storage unit 13 that stores programs and processing data that will be described later. It includes a bus 14 for transmitting data, an interface section 15 for inputting and outputting data to and from external devices, and a GPU (Graphics Processing Unit) 19. The input section 16 and the output section 17 are connected to the processing section 10. Illustratively, the input unit 16 is an input device such as a keyboard or a mouse, and the output unit 17 is a display device such as a liquid crystal display. The GPU 19 functions as an accelerator that assists the arithmetic processing (for example, parallel arithmetic processing) performed by the CPU 11. That is, in the following description, the processing performed by the CPU 11 includes the processing performed by the CPU 11 using the GPU 19 as an accelerator.

Further, the processing unit 10 stores in advance the program according to the present invention and the pre-learning neural network 50 in the storage unit 13 in an executable format, for example, in order to process each step described in FIG. 10 below. . The executable format is, for example, a format generated by converting a programming language with a compiler. The processing unit 10 performs processing using the program stored in the storage unit 13 and the pre-learning neural network 50.

In the following description, unless otherwise specified, the processing performed by the processing unit 10 means the processing performed by the CPU 11 based on the program stored in the storage unit 13 or the memory 12 and the neural network 50. The CPU 11 uses the memory 12 as a work area to temporarily store necessary data (intermediate data during processing, etc.), and appropriately stores long-term storage data such as calculation results in the storage unit 13.

Referring to FIG. 9, the user device 200 (200A, 200B, 200C) includes a processing section 20 (20A, 20B, 20C), an input section 26, and an output section 27.

The processing unit 20 includes a CPU (Central Processing Unit) 21 that performs data processing, which will be described later, a memory 22, which is used as a work area for data processing, and a storage unit 23, which stores programs and processing data, which will be described later. It includes a bus 24 for transmitting data, an interface section 25 for inputting and outputting data to and from external devices, and a GPU (Graphics Processing Unit) 29. The input section 26 and the output section 27 are connected to the processing section 20. Illustratively, the input unit 26 is an input device such as a keyboard or a mouse, and the output unit 27 is a display device such as a liquid crystal display. The GPU 29 functions as an accelerator that assists the arithmetic processing (for example, parallel arithmetic processing) performed by the CPU 21. That is, in the following description, the processing performed by the CPU 21 includes the processing performed by the CPU 21 using the GPU 29 as an accelerator.

Further, the processing unit 20 stores the program according to the present invention and the deep learning algorithm 60 of the learned neural network structure in the storage unit 23 in an executable format, for example, in order to process each step described in FIG. 14 below. I remember it in advance. The executable format is, for example, a format generated by converting a programming language with a compiler. The processing unit 20 performs processing using the program stored in the storage unit 23 and the deep learning algorithm 60.

In the following description, unless otherwise specified, the processing performed by the processing unit 20 is actually the processing performed by the CPU 21 of the processing unit 20 based on the program and deep learning algorithm 60 stored in the storage unit 23 or memory 22. means. The CPU 21 uses the memory 22 as a work area to temporarily store necessary data (intermediate data during processing, etc.), and appropriately stores long-term storage data such as calculation results in the storage unit 23.

[Functional blocks and processing procedures]
- Deep Learning Processing Referring to FIG. 10, the processing unit 10A of the deep learning device 100A according to the first embodiment includes a training data generation unit 101, a training data input unit 102, and an algorithm update unit 103. These functional blocks are realized by installing a program that causes a computer to execute deep learning processing in the storage unit 13 or memory 12 of the processing unit 10A, and having the CPU 11 execute this program. The window size database 104 and the algorithm database 105 are stored in the storage unit 13 or memory 12 of the processing unit 10A.

The first training image 70R1, second training image 70R2, and third training image 70R3 of the learning sample are captured in advance by the imaging device 300 and stored in the storage unit 13 or memory 12 of the processing unit 10A in advance. It is assumed that it is remembered. The neural network 50 is stored in advance in the algorithm database 105 in association with, for example, the type of tissue (eg, tissue name) from which the sample to be analyzed comes from.

The processing unit 10A of the deep learning device 100A performs the processing shown in FIG. 11. Explaining using each functional block shown in FIG. 10, the training data generation unit 101 performs the processing from steps S10 to S12, S17, and S18. The process of step S14 is performed by the training data input unit 102. The processing in steps S15 and S16 is performed by the algorithm updating unit 103.

In steps S10 to S18 described below, deep learning processing using the first training image 70R1, the second training image 70R2, and the third training image 70R3 will be described.

The processing unit 10A performs deep learning processing to generate tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3 according to the method described in the overview of deep learning methods. The processing unit 10A displays, on the output unit 17, an image of a wide area (whole slide image 70W1) including the area of the first training image 70R1, based on an operation from the input unit 16 by a pathologist or the like, for example. A pathologist or the like who makes the determination visually confirms the whole slide image 70W1 displayed on the output unit 17. The pathologist or the like specifies, for example, via the input unit 16, an area in the whole slide image 70W1 that has been determined to include a tumor area, and surrounds the area in the whole slide image 70W1 with a solid line of, for example, red. Similarly, the pathologist and the like surround non-tumor areas and non-tissue areas in the whole slide image 70W1 with solid lines in blue, green, etc., which are different from red, for example. Instead of displaying the whole slide image 70W1 on the output unit 17 for judgment by a pathologist, the processing unit 10A displays the judged whole slide image 70W2 via the I/F unit 15, for example, on the network 99. It may also be obtained via A pathologist or the like may divide the whole slide image 70W2 into pixels preferable as a training image.

The processing unit 10A, based on the designation from the input unit 16 of a pathologist or the like, converts the first training image 70R1 acquired from the area surrounded by the solid red line in the whole slide image 70W2 to include a predetermined number of pixels. Cut it out like this. Similarly, the second training image 70R2 and the third training image 70R3 are each cut out from the portion determined to be a non-tumor region and a non-tissue region so as to include a predetermined pixel region.

In step S10, the processing unit 10A generates color density encoded diagrams 72R1r, 72R1g, and 72R1b for each of R, G, and B colors from the extracted first training image 70R1. The color density encoding diagrams 72R1r, 72R1g, and 72R1b are generated by attaching codes to each pixel that stepwise represent the color density values of R, G, and B of each pixel of the first training image 70R1. In this embodiment, color density encoded diagrams 72R1r, 72R1g, and 72R1b are generated for each R, G, and B gradation image, assuming that color density values are represented by 256 gradations from value 0 to value 255. As for the assignment of color density values, for example, the lowest brightness is assigned a color density value of 0, and as the level of brightness increases, gradually higher values are assigned, and the highest brightness is assigned a color density value of 255. The processing unit 10A generates first training data 74R1 in which R, G, and B color density values are attached to each pixel from the color density encoded diagrams 72R1r, 72R1g, and 72R1b. The processing unit 10A generates first preliminary training data 70R1L in which each pixel of the first training image 70R1 is associated with a label value "1" indicating a tumor region.

In step S11, the processing unit 10A generates second training data 74R2 from the second training image 70R2 using a method similar to step S10. The processing unit 10A generates second preliminary training data 70R2L in which each pixel of the second training image 70R2 is associated with a label value "2" indicating that it is a non-tumor region.

In step S12, the processing unit 10A generates third training data 74R3 from the third training image 70R3 using the same method as in step S10. The processing unit 10A generates third preliminary training data 70R3L in which each pixel of the third training image 70R3 is associated with a label value "0" indicating that it is a non-tissue region.

Steps S10 to S12 may be performed simultaneously or in random order.

In step S13, the processing unit 10A receives an input of the type of organization for learning from the user of the deep learning device 100A through the input unit 16. Based on the input tissue type, the processing unit 10A refers to the window size database 104 (window size DB 104) to set the window size, refers to the algorithm database 105 (algorithm DB 105), and sets the neural Set up the network 50. The window size is, for example, 200×200 pixels. The window size is a unit of training data input to the neural network 50 at one time of input, and the product of the number of pixels of the training data corresponding to each region of the window size and the number of primary colors included in the image is: This corresponds to the number of nodes in the input layer 50a. The window size is stored in advance in the window size database 104 in association with the type of organization.

In step S14, the processing unit 10A generates window-sized tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3 from the training data corresponding to each region. Specifically, as explained in the above-mentioned "Overview of deep learning method" with reference to FIGS. 4(a) and 4(b), from the first training data 74R1 and the first preliminary training data 70R1L, Tumor region training data 75R1 of window size is created using window W1. The processing unit 10A generates window-sized non-tumor region training data 75R2 from the second training data 74R2 and the second preliminary training data 70R2L. Specifically, as explained in the above-mentioned "Outline of deep learning method" with reference to FIGS. 4(a), (b), and FIG. 5(a), the second training data 74R2 and the second From the preliminary training data 70R2L, non-tumor region training data 75R2 having a window size of window W1 is created. The processing unit 10A generates window-sized non-organized region training data 75R3 from the third training data 74R3 and the third preliminary training data 70R3L. Specifically, as explained with reference to FIGS. 4(a), (b) and FIG. 5(b) in the above-mentioned "Outline of deep learning method", the third training data 74R3 and the third From the preliminary training data 70R3L, non-organized region training data 75R3 having a window size of window W1 is created.

In step S15 shown in FIG. 11, the processing unit 10A causes the neural network 50 to learn using the window-sized tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3. The learning results of the neural network 50 are accumulated each time the neural network 50 is trained using the window-sized tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3.

Since the image analysis method according to the embodiment uses a convolutional neural network and uses a stochastic gradient descent method, in step S16, the processing unit 10A accumulates learning results for a predetermined number of trials. Determine whether or not. If the learning results have been accumulated for the predetermined number of trials, the processing unit 10A performs the process of step S17, and if the learning results have not been accumulated for the predetermined number of trials, the processing unit 10A performs the process of step S18.

If the learning results have been accumulated for the predetermined number of trials, in step S17, the processing unit 10A updates the connection weight w of the neural network 50 using the learning results accumulated in step S15. Since the image analysis method according to the embodiment uses the stochastic gradient descent method, the connection weight w of the neural network 50 is updated at the stage when learning results for a predetermined number of trials have been accumulated. Specifically, the process of updating the connection weight w is a process of performing calculations using the gradient descent method, as shown in (Equation 11) and (Equation 12), which will be described later.

In step S17, the processing unit 10A determines whether a specified number of pixels in each region training data have been processed for each of the first training data 74R1, the second training data 74R2, and the third training data 74R3. do. If the series of processes from step S15 to step S17 have been performed on the specified number of pixels of each region training data, the deep learning process is ended. Learning of the neural network does not necessarily need to be performed on all pixels in each training data, and the processing unit 10A performs learning by processing some pixels in each training data, that is, a specified number of pixels. It can be performed. The specified number of pixels may be all pixels within each region training data.

If the specified number of pixels in each area training data have not been processed, the processing unit 10A, in step S19, processes the first training data 74R1, the second training data The position of the window is moved within 74R2 or within the third training data 74R3. Thereafter, the processing unit 10A performs a series of processes from step S14 to step S16 at the new window position after the movement. That is, in step S14, the processing unit 10A sets each area training data 75R1 of the window size for each of the first training data 74R1, the second training data 74R2, and the third training data 74R3 at the new window position after the movement. , 75R2, and 75R3 are cut out, and a label value indicating each area is attached. Subsequently, in step S15, the processing unit 10A causes the neural network 50 to learn using the training data 75R1, 75R2, and 75R3 for each region of the newly cut out window size. If the learning results for the predetermined number of trials have been accumulated in step S16, the processing unit 10A updates the connection weight w of the neural network 50 in step S17. Such learning of the neural network 50 for each window size is performed for a specified number of pixels for each of the first training data 74R1, the second training data 74R2, and the third training data 74R3.

The deep learning processing from steps S10 to S19 described above is repeatedly performed on the separately acquired first training image 70R1, second training image 70R2, and third training image 70R3. This improves the degree of learning of the neural network 50. As a result, a deep learning algorithm 60 having a neural network structure shown in FIG. 5 is obtained.

- Structure of Neural Network As shown in FIG. 12(a), the first embodiment uses a deep learning type neural network. A deep learning type neural network, like a neural network 50 shown in FIG. 12, includes an input layer 50a, an output layer 50b, and an intermediate layer 50c between the input layer 50a and the output layer 50b. Consists of multiple layers. The number of layers constituting the intermediate layer 50c can be, for example, five or more.

In the neural network 50, a plurality of nodes 89 arranged in layers are connected between the layers. As a result, information propagates from the input side layer 50a to the output side layer 50b in only one direction shown by arrow D in the figure. In this embodiment, the number of nodes in the input layer 50a corresponds to the product of the number of pixels of the input image, that is, the number of pixels of the window W1 shown in FIG. 3(c), and the number of primary colors included in each pixel. ing. Since pixel data (color density values) of an image can be input to the input layer 50a, the user can input the input image to the input layer 50a without separately calculating feature amounts from the input image.

-Calculation at each node FIG. 12(b) is a schematic diagram showing the calculation at each node. Each node 89 receives multiple inputs and calculates one output (z). In the example shown in FIG. 12(b), node 89 receives four inputs. The total input (u) received by the node 89 is expressed by the following (Equation 1).

Each input is given a different weight. In (Formula 1), b is a value called bias. The output (z) of the node is the output of a predetermined function f for the total input (u) expressed by (Formula 1), and is expressed by (Formula 2) below. The function f is called an activation function.

FIG. 12(c) is a schematic diagram showing operations between nodes. In the neural network 50, nodes that output a result (z) expressed by (Formula 2) with respect to a total input (u) expressed by (Formula 1) are arranged in a layered manner. The output of the node in the previous layer becomes the input to the node in the next layer. In the example shown in FIG. 12(c), the output of the node 89a on the left layer in the figure becomes the input to the node 89b on the right layer in the figure. Each node 89b in the right layer receives an output from node 89a in the left layer. Each connection between each node 89a in the left layer and each node 89b in the right layer is given a different weight. If the outputs of the plurality of nodes 89a in the left layer are x ₁ to x ₄ , the inputs to each of the three nodes 89b in the right layer are as follows (Equation 3-1) to (Equation 3-3) It is expressed as

When these (Formula 3-1) to (Formula 3-3) are generalized, (Formula 3-4) is obtained. Here, i=1,...I, and j=1,...J.

（式３－４）を活性化関数に適用すると出力が得られる。出力は以下の（式４）で表される。

Applying (Equation 3-4) to the activation function yields an output. The output is expressed by the following (Equation 4).

- Activation Function In the image analysis method according to the embodiment, a rectified linear unit function is used as the activation function. The normalized linear function is expressed by the following (Equation 5).

(Equation 5) is a function in which the part where u<0 is set to u=0 among the linear functions of z=u. In the example shown in FIG. 12(c), the output of the node with j=1 is expressed by the following equation using (Equation 5).

-Learning of a neural network Letting y(x:w) be a function expressed using a neural network, the function y(x:w) changes when the parameter w of the neural network is changed. Adjusting the function y(x:w) so that the neural network selects a more suitable parameter w for the input x is called learning of the neural network. Assume that multiple pairs of input and output of a function expressed using a neural network are given. When the desired output for a certain input x is d, the input/output pair is given as {(x ₁ , d ₁ ), (x ₂ , d ₂ ), . . . , (x _n , d _n )}. Each set of pairs represented by (x, d) is called training data. Specifically, a set of pairs of color density values and true value image labels for each pixel in a single color image of R, G, and B shown in FIG. 3(b) is shown in FIG. 3(a). This is the training data shown.

Neural network learning means that for any input/output pair (x _n , d _n ), the output y (x _n :w) of the neural network when input x _n is given is the output d _n This means adjusting the weight w so that it is as close as possible. The error function is the closeness between the function expressed using a neural network and the training data.

It is a scale to measure. The error function is also called a loss function. The error function E(w) used in the image analysis method according to the embodiment is expressed by the following (Equation 6). (Formula 6) is called cross entropy.

A method for calculating the cross entropy of (Equation 6) will be explained. In the output layer 50b of the neural network 50 used in the image analysis method according to the embodiment, that is, in the final layer of the neural network, an activation function is used to classify the input x into a finite number of classes according to the content. The activation function is called a softmax function and is expressed by the following (Equation 7). It is assumed that the same number of nodes as the number of classes k are arranged in the output layer 50b. It is assumed that the total input u of each node k (k=1, . . . K) of the output layer L is given by u _k ^(L) from the output of the previous layer L-1. As a result, the output of the k-th node of the output layer is expressed by the following (Equation 7).

(Formula 7) is the softmax function. The sum of the outputs y ₁ , . . . , y _K determined by (Equation 7) is always 1.

If each class is represented as C ₁ , ..., _CK , then the output y _k (i.e. u _k ^(L) ) of node k of output layer L represents the probability that a given input x belongs to class C _k . Please refer to (Equation 8) below. The input x is classified into the class in which the probability expressed by (Equation 8) is maximized.

In learning a neural network, the function represented by the neural network is regarded as a model of the posterior probability of each class, and under such a probability model, the likelihood of the weight w for the training data is calculated as follows: evaluate and select the weight w that maximizes the likelihood.

It is assumed that the target output d _n by the softmax function in (Equation 7) is set to 1 only when the output is in the correct class, and is set to 0 otherwise. If the target output is expressed in a vector format d _n = [d _n1 ,..., d _nK ], for example, if the correct class of input x _n is C ₃ , only the target output d _n3 will be 1, and the other The target output becomes 0. When encoded in this way, the posterior distribution is expressed by the following (Equation 9).

The likelihood L(w) of the weight w for the training data {(x _n , d _n )} (n=1, . . . , N) is expressed by the following (Equation 10). By taking the logarithm of the likelihood L(w) and inverting its sign, the error function of (Equation 6) is derived.

Learning means minimizing the error function E(w) calculated based on training data with respect to the parameter w of the neural network. In the image analysis method according to the embodiment, the error function E(w) is expressed by (Equation 6).

Minimizing the error function E(w) with respect to the parameter w has the same meaning as finding a local minimum point of the function E(w). The parameter w is the weight of connections between nodes. The minimum point of the weight w is determined by an iterative calculation in which the parameter w is repeatedly updated using an arbitrary initial value as a starting point. An example of such a calculation is a gradient descent method.

The gradient descent method uses a vector expressed by the following (Equation 11).

In the gradient descent method, the process of moving the current value of the parameter w in the negative gradient direction (ie, −∇E) is repeated many times. When the current weight is w ^(t) and the weight after movement is w ^(t+1) , the calculation by the gradient descent method is expressed by the following (Equation 12). The value t means the number of times the parameter w has been moved.

は、パラメータｗの更新量の大きさを決める定数であり、学習係数と呼ばれる。（式１２）で表される演算を繰り返すことにより、値ｔの増加に伴って誤差関数Ｅ（ｗ^（ｔ））が減少し、パラメータｗは極小点に到達する。 symbol

is a constant that determines the amount of update of the parameter w, and is called a learning coefficient. By repeating the calculation expressed by (Equation 12), the error function E(w ^(t) ) decreases as the value t increases, and the parameter w reaches the minimum point.

Note that the calculation according to (Equation 12) may be performed on all training data (n=1, . . . , N), or may be performed on only some training data. Gradient descent performed only on a portion of the training data is called stochastic gradient descent. The image analysis method according to the embodiment uses stochastic gradient descent.

- Image analysis processing Referring to FIG. 13, the processing unit 20A of the image analysis apparatus 200A according to the first embodiment includes an analysis data generation unit 201, an analysis data input unit 202, an analysis unit 203, and a region detection unit 20A. 204. These functional blocks are realized by installing a program that causes a computer according to the present invention to execute image analysis processing in the storage unit 23 or memory 22 of the processing unit 20A, and having the CPU 21 execute this program. The window size database 104 and the algorithm database 105 are provided from the deep learning device 100A via the storage medium 98 or the network 99, and are stored in the storage unit 23 or memory 22 of the processing unit 20A.

It is assumed that an analysis target image 78 of a tissue to be analyzed is captured in advance by the imaging device 400 and stored in advance in the storage unit 23 or memory 22 of the processing unit 20A. The deep learning algorithm 60 including the learned connection weights w is stored in the algorithm database 105 in association with the type of tissue from which the tissue sample to be analyzed comes from, and is used as a program for causing a computer to perform image analysis processing. Functions as a part of a program module. That is, the deep learning algorithm 60 is used in a computer equipped with a CPU and memory, and is designed to perform calculations or processing of information specific to the purpose of use, such as outputting data indicating each region in the tissue to be analyzed. , make the computer work. Specifically, the CPU 21 of the processing unit 20A performs calculations of the neural network 60 based on the learned connection weights w according to the algorithm defined in the deep learning algorithm 60 stored in the storage unit 23 or the memory 22. The CPU 21 of the processing unit 20A performs calculations on the analysis target image 78 of the tissue to be analyzed, which is input to the input layer 60a, and outputs data indicating each region in the tissue to be analyzed from the output layer 60b. A ternary image 83 is output.

Referring to FIG. 14, the processing unit 20A of the image analysis device 200A performs the processing shown in FIG. 13. Explaining using each functional block shown in FIG. 13, the processing of steps S21 and S22 is performed by the analysis data generation unit 201. The processing of steps S23, S24, S26, and S27 is performed by the analysis data input unit 202. The processing in steps S25 and S28 is performed by the analysis unit 203. The process in step S29 is performed by the area detection unit 204.

In step S21, the processing unit 20A, in response to a command to start processing input by a user or the like from the input unit 26, generates color density encoded diagrams 79r, 79g of R, G, and B colors from the input analysis target image 78, for example. Generate 79b. Data to be analyzed is generated by combining the color density values of R, G, and B colors for each pixel from the color density encoded diagrams 79r, 79g, and 79b. The method of generating the color density encoded diagrams 79r, 79g, and 79b is the same as the method of generating them in step S10 during the deep learning process shown in FIG. 11.

In step S22 shown in FIG. 14, the processing unit 20A receives an input of the type of tissue from the user of the image analysis device 200A as an analysis condition through the input unit 26. Based on the input tissue type, the processing unit 20A refers to the window size database 104 and the algorithm database 105, sets a window size to be used for analysis, and obtains a deep learning algorithm 60 to be used for analysis. The window size is a unit of analysis data input to the neural network 60 at one time of input, and the product of the number of pixels of the window size analysis data 80 and the number of primary colors included in the image is This corresponds to the number of nodes of 60a. The window size is stored in advance in the window size database 104 in association with the type of organization. The window size is, for example, 3×3 pixels, as shown in window W2 shown in FIG. The deep learning algorithm 60 is also stored in advance in the algorithm database 105 shown in FIG. 13 in association with the type of organization.

In step S23 shown in FIG. 14, the processing unit 20A generates data to be analyzed by combining the color density values of R, G, and B for each pixel from the color density encoding diagrams 79r, 79g, and 79b, and further The analysis data 80 is generated.

In step S24, the processing unit 20A inputs the analysis data 80 shown in FIG. 5 to the deep learning algorithm 60. The initial position of the window is, for example, a position where a 3×3 pixel within the window corresponds to the upper left corner of the data to be analyzed, as in step S14 during deep learning processing. When the processing unit 20A inputs a total of 27 color density value data 81 of 3 x 3 pixels x 3 primary colors included in the window size analysis data 80 to the input layer 60a, the deep learning algorithm 60 inputs the data 81 to the output layer. The determination result 82 is output to 60b.

In step S25 shown in FIG. 14, the processing unit 20A stores the determination result 82 output to the output layer 60b shown in FIG. The determination result 82 is an estimated value (one of the three values 0, 1, and 2) for each pixel region of the data to be analyzed.

In step S26 shown in FIG. 14, the processing unit 20A determines whether all pixels in the input image have been processed. The input images are color density encoded diagrams 79r, 79g, and 79b shown in FIG. 6, and if all pixels in the data to be analyzed have been subjected to the series of processes from step S23 to step S25 shown in FIG. , performs the process of step S28.

If all the pixels in the data to be analyzed have not been processed, in step S27, the processing unit 20A converts the color density encoding diagrams 79r, 79g, 79b shown in FIG. The window W2 is moved in units of a predetermined number of pixels. Thereafter, the processing unit 20A performs a series of processes from step S23 to step S25 at the new position of the window W2 after the movement. In step S25, the processing unit 20A stores the determination result 82 corresponding to the new window position after the movement. By storing such determination results 82 for each window size for all pixels in the image to be analyzed, a ternary image 83 as an analysis result is obtained. The image size of the ternary image 83 as the analysis result is the same as the image size of the analysis target image. Here, the ternary image 83 may be numerical data in which the estimated values 2, 1, and 0 are attached to each pixel, and instead of the estimated values 2, 1, and 0. For example, it may be an image shown in display colors corresponding to value 2, value 1, and value 0, respectively.

In step S28 shown in FIG. 14, the processing unit 20A outputs the ternary image 83 as the analysis result to the output unit 27.

In step S29, subsequent to step S28, the processing unit 20A further performs area detection processing on the ternary image 83 as the analysis result. In the ternary image 83, a tumor region, a non-tumor region, and a non-tissue region are distinguished and expressed using ternary values.

Although optional, the processing unit 20A creates a region-enhanced image 84 by superimposing each obtained region on the analysis target image 78 to be analyzed. The processing unit 20A outputs the created region-enhanced image 84 to the output unit 27, and ends the image analysis process.

Although optional, the processing unit 20A may calculate how much of the tumor region is included in the tissue included in the analysis target image 78. For example, if the number of pixels corresponding to the tissue included in the analysis target image 78, that is, the total number of pixels determined to be a tumor region and a non-tumor region (total number of pixels in the tissue region), is determined to be a tumor region. By calculating the percentage occupied by the number of pixels, the content rate of the tumor region can be calculated. In addition, by subtracting the number of pixels determined to be tumor regions from the total number of pixels, or when the total number of pixels determined to be tissue regions is set to 100%, the percentage of pixels determined to be non-tumor regions can be calculated. By calculating, the content rate of the non-tumor region can be calculated. Then, the ratio between the content rate of the tumor region and the content rate of the non-tumor region may be calculated. The calculated value may be output to the output section 27. Further, the calculated value may be output to the output unit 27 together with the region-enhanced image 84.

Calculating the content of the tumor region in this way is useful in screening whether the tissue used for genetic analysis of cancer tissue, etc. contains tumor tissue suitable for testing.

As described above, the user of the image analysis apparatus 200A can obtain the ternary image 83 as an analysis result by inputting the analysis target image 78 of the tissue to be analyzed into the image analysis apparatus 200A. The ternary image 83 represents a tumor region, a non-tumor region, and a non-tissue region in the specimen to be analyzed, and allows the user to distinguish each region in the specimen to be analyzed.

Further, the user of the image analysis device 200A can obtain a region-enhanced image 84 as an analysis result. The region-enhanced image 84 is generated, for example, by filling each region of the analysis target image 78 with a color. In another aspect, the area is generated by overlapping the boundaries of each area. This allows the user to grasp each area at a glance in the tissue to be analyzed.

Indicating the tumor region and/or non-tumor region in the specimen to be analyzed helps to screen the tumor region in the tissue to be analyzed prior to tissue diagnosis and increases sample observation efficiency.

<Second embodiment>
Hereinafter, the image analysis system according to the second embodiment will be described with respect to the differences from the image analysis system according to the first embodiment.

[Configuration overview]
Referring to FIG. 15, the image analysis system according to the second embodiment includes a user device 200, and the user device 200 operates as an integrated image analysis device 200B. The image analysis device 200B is composed of, for example, a general-purpose computer, and performs both the deep learning processing and the image analysis processing described in the first embodiment. In other words, the image analysis system according to the second embodiment is a stand-alone system in which deep learning and image analysis are performed on the user side. In the image analysis system according to the second embodiment, an integrated image analysis device 200B installed on the user side has the functions of both the deep learning device 100A and the image analysis device 200A according to the first embodiment. This is different from the image analysis system according to the first embodiment.

Image analysis device 200B is connected to imaging device 400. The imaging device 400 acquires a training image of a tissue for learning during deep learning processing, and acquires an analysis target image 78 of a tissue to be analyzed during image analysis processing.

[Hardware configuration]
The hardware configuration of the image analysis device 200B is similar to the hardware configuration of the user-side device 200 shown in FIG.

[Functional blocks and processing procedures]
Referring to FIG. 16, the processing unit 20B of the image analysis device 200B according to the second embodiment includes a training data generation unit 101, a training data input unit 102, an algorithm update unit 103, and an analysis data generation unit 201. , an analysis data input section 202, an analysis section 203, and a region detection section 204. These functional blocks are realized by installing a program that causes a computer to execute deep learning processing and image analysis processing in the storage unit 23 or memory 22 of the processing unit 20B, and having the CPU 21 execute this program. The window size database 104 and the algorithm database 105 are stored in the storage unit 23 or memory 22 of the processing unit 20B, and both are used in common during deep learning and image analysis processing. The trained neural network 60 is stored in advance in the algorithm database 105 in association with the type of tissue or the type of sample containing cells, and the connection weights w are updated by deep learning processing, and the deep learning algorithm 60 It is stored in the algorithm database 105 as follows. Note that the first training image 70R1, second training image 70R2, and third training image 70R3, which are training images, are captured in advance by the imaging device 400, and are stored in the storage unit 23 or memory 22 of the processing unit 20B. is stored in advance. It is assumed that an analysis target image 78 of a specimen to be analyzed is also captured in advance by the imaging device 400 and stored in advance in the storage unit 23 or the memory 22 of the processing unit 20B.

The processing unit 20B of the image analysis device 200B performs the processing shown in FIG. 11 during deep learning processing, and performs the processing shown in FIG. 14 during image analysis processing. Explaining using each functional block shown in FIG. 16, during deep learning processing, the training data generation unit 101 performs the processing from steps S10 to S12, S14, S18, and S19. The process of step S13 is performed by the training data input unit 102. The processing in steps S16 and S17 is performed by the algorithm updating unit 103. During image analysis processing, the processing of steps S21 and S22 is performed by the analysis data generation unit 201. The processing of steps S23, S24, S26, and S27 is performed by the analysis data input unit 202. The processing in steps S25 and S28 is performed by the analysis unit 203. The process in step S29 is performed by the area detection unit 204.

The deep learning processing procedure and the image analysis processing procedure performed by the image analysis device 200B according to the second embodiment are similar to the procedures performed by the deep learning device 100A and the image analysis device 200A according to the first embodiment, respectively. . Note that the image analysis device 200B according to the second embodiment differs from the deep learning device 100A and the image analysis device 200A according to the first embodiment in the following points.

In step S13 during deep learning processing, the processing unit 20B receives an input of the type of tissue for learning from the user of the image analysis device 200B through the input unit 26. Based on the input tissue type, the processing unit 20B refers to the window size database 104 to set the window size, and refers to the algorithm database 105 to set the neural network 50 used for learning.

As described above, the user of the image analysis device 200B can obtain the ternary image 83 as an analysis result by inputting the analysis target image 78 into the image analysis device 200B. Furthermore, the user of the image analysis device 200B can obtain a region-enhanced image 84 as an analysis result.

According to the image analysis device 200B according to the second embodiment, the user can use the type of tissue selected by the user as a learning tissue. This means that the learning of the neural network 50 is not left to the vendor, but the user himself can improve the learning level of the neural network 50.

<Third embodiment>
Hereinafter, the image analysis system according to the third embodiment will be described with respect to the differences from the image analysis system according to the second embodiment.

[Configuration overview]
Referring to FIG. 17, the image analysis system according to the third embodiment includes a vendor device 100 and a user device 200. The vendor device 100 operates as an integrated image analysis device 100B, and the user device 200 operates as a terminal device 200C. The image analysis device 100B is configured of, for example, a general-purpose computer, and is a device on the cloud server side that performs both the deep learning processing and the image analysis processing described in the first embodiment. The terminal device 200C is composed of, for example, a general-purpose computer, and is a user side that transmits an image to be analyzed to the image analysis device 100B through the network 99, and receives an image resulting from the analysis from the image analysis device 100B through the network 99. This is a terminal device.

In the image analysis system according to the third embodiment, an integrated image analysis device 100B installed on the vendor side has the functions of both the deep learning device 100A and the image analysis device 200A according to the first embodiment. This is the same as the image analysis system according to the second embodiment. On the other hand, the image analysis system according to the third embodiment includes a terminal device 200C, and provides the user-side terminal device 200C with an input interface for an image to be analyzed and an output interface for an image as an analysis result. This is different from the image analysis system according to the second embodiment. In other words, the image analysis system according to the third embodiment is a cloud service type in which a vendor side that performs deep learning processing and image analysis processing provides an input/output interface for images to be analyzed and images of analysis results to the user side. It is a system.

The image analysis device 100B is connected to the imaging device 300, and acquires training images of the learning tissue captured by the imaging device 300.

The terminal device 200C is connected to the imaging device 400, and acquires an analysis target image 78 of the tissue to be analyzed, which is captured by the imaging device 400.

[Hardware configuration]
The hardware configuration of the image analysis device 100B is similar to the hardware configuration of the vendor-side device 100 shown in FIG. The hardware configuration of the terminal device 200C is similar to the hardware configuration of the user side device 200 shown in FIG.

[Functional blocks and processing procedures]
Referring to FIG. 18, the processing unit 10B of the image analysis apparatus 100B according to the third embodiment includes a training data generation unit 101, a training data input unit 102, an algorithm update unit 103, and an analysis data generation unit 201. , an analysis data input section 202, an analysis section 203, and a region detection section 204. These functional blocks are realized by installing a program that causes a computer to execute deep learning processing and image analysis processing in the storage unit 13 or memory 12 of the processing unit 10B, and having the CPU 11 execute this program. The window size database 104 and the algorithm database 105 are stored in the storage unit 13 or the memory 12 of the processing unit 10B, and both are used in common during deep learning and image analysis processing. The neural network 50 is stored in advance in the algorithm database 105 in association with the type of tissue, and the connection weights w are updated by deep learning processing and stored in the algorithm database 105 as the deep learning algorithm 60.

Note that the first training image 70R1, second training image 70R2, and third training image 70R3, which are training images, are captured in advance by the imaging device 300, and are stored in the storage unit 13 or memory 12 of the processing unit 10B. is stored in advance. It is assumed that an analysis target image 78 of a tissue to be analyzed is also captured in advance by the imaging device 400 and stored in advance in the storage unit 23 or memory 22 of the processing unit 20C of the terminal device 200C.

The processing unit 10B of the image analysis device 100B performs the processing shown in FIG. 11 during deep learning processing, and performs the processing shown in FIG. 14 during image analysis processing. To explain using each functional block shown in FIG. 18, during deep learning processing, the training data generation unit 101 performs the processing from steps S10 to S12, S14, S18, and S19. The process of step S13 is performed by the training data input unit 102. The processing in steps S16 and S17 is performed by the algorithm updating unit 103. During image analysis processing, the processing of steps S21 and S22 is performed by the analysis data generation unit 201. The processing of steps S23, S24, S26, and S27 is performed by the analysis data input unit 202. The processing in steps S25 and S28 is performed by the analysis unit 203. The process in step S29 is performed by the area detection unit 204.

The deep learning processing procedure and the image analysis processing procedure performed by the image analysis device 100B according to the third embodiment are similar to the procedures performed by the deep learning device 100A and the image analysis device 200A according to the first embodiment, respectively. . Note that the image analysis device 100B according to the third embodiment differs from the deep learning device 100A and the image analysis device 200A according to the first embodiment in the following four points.

In step S21 during the image analysis process shown in FIG. 14, the processing unit 10B receives the analysis target image 78 of the tissue to be analyzed from the user's terminal device 200C, and from the received analysis target image 78, R, G, Color density encoded diagrams 79r, 79g, and 79b for each B color are generated. The method of generating the color density encoding diagrams 79r, 79g, and 79b and the data to be analyzed is the same as the method of generating them in step S10 during the deep learning process shown in FIG. 11.

In step S22 during the image analysis process shown in FIG. 14, the processing unit 10B receives an input of the type of organization from the user of the terminal device 200C as an analysis condition through the input unit 26 of the terminal device 200C. Based on the input tissue type, the processing unit 10B refers to the window size database 104 and the algorithm database 105, sets the window size to be used in the analysis, and obtains the deep learning algorithm 60 to be used in the analysis.

In step S28 during the image analysis process, the processing unit 10B transmits the ternary image 83 as the analysis result to the user's terminal device 200C. In the user's terminal device 200C, the processing unit 20C outputs the three images 83 of the received analysis results to the output unit 27.

In step S29 during image analysis processing, the processing unit 10B further performs nuclear region detection processing on the ternary image 83 as the analysis result, subsequent to step S28. As an optional step, the processing unit 10B creates a region-enhanced image 84 by superimposing each obtained region on the analysis target image 78 to be analyzed. The processing unit 10B transmits the created region-enhanced image 84 to the user's terminal device 200C. In the user's terminal device 200C, the processing unit 20C outputs the received region-enhanced image 84 to the output unit 27, and ends the image analysis process.

As described above, the user of the terminal device 200C can obtain the ternary image 83 as an analysis result by transmitting the analysis target image 78 of the tissue to be analyzed to the image analysis apparatus 100B. Further, the user of the terminal device 200C can obtain the region-enhanced image 84 as an analysis result.

According to the image analysis device 100B according to the third embodiment, the user can enjoy the results of image analysis processing without acquiring the window size database 104 and the algorithm database 105 from the deep learning device 100A. Thereby, as a service for analyzing a tissue to be analyzed, a service for determining tumor regions and/or non-tumor regions and screening tumor regions in each tissue can be provided as a cloud service.

[Computer program]
Embodiments of the present invention include a computer program that causes the processing units 10A, 20B, and 10B to execute steps S10 to S20 and generates a trained deep learning algorithm, and products thereof. Further, an embodiment of the present invention includes a computer program and product thereof that causes the processing units 10A, 20B, and 10B to execute the steps S21 to S29 and causes the computer to function to analyze images of tissues collected from an individual. included.

[Other forms]
Although the present invention has been described above using the outline and specific embodiments, the present invention is not limited to the above-described outline and each embodiment.

In the first to third embodiments described above, in step S13, the processing units 10A, 20B, and 10B refer to the window size database 104 to set the number of pixels of the window size. You can also set the size directly. In this case, the window size database 104 becomes unnecessary.

In the first to third embodiments described above, in step S13, the processing units 10A, 20B, and 10B set the number of pixels of the window size based on the input tissue type. Instead of the input, the size of the organization may be input. The processing units 10A, 20B, and 10B may refer to the window size database 104 and set the number of pixels of the window size based on the input tissue size. In step S22, as in step S13, instead of inputting the type of organization, the size of the organization may be input. The processing units 20A, 20B, and 10B refer to the window size database 104 and the algorithm database 105 based on the input tissue size, set the number of pixels of the window size, and acquire the neural network 60.

Regarding the mode of inputting the size of the organization, the size may be input directly as a numerical value, or, for example, the input user interface may be set as a pull-down menu to prompt the user for a predetermined range of numerical values corresponding to the size that the user is trying to input. You may select and input.

Furthermore, in step S13 and step S22, in addition to the tissue type or tissue size, the imaging magnification at which the training image and the analysis target image 78 were captured may be input. Regarding the mode of inputting the imaging magnification, the magnification may be directly input as a numerical value, or, for example, the input user interface may be set as a pull-down menu and the user can select a predetermined range of numerical values corresponding to the magnification that the user wants to input. You can also input it by

In the first to third embodiments above, the window size is set to 3 x 3 pixels for the convenience of explanation during deep learning processing and image analysis processing, but the number of pixels in the window size is set to 3 x 3 pixels for convenience of explanation. Not limited. The window size may be set depending on the type of organization, for example. In this case, it is sufficient that the product of the number of pixels of the window size and the number of primary colors included in the image corresponds to the number of nodes in the input layers 50a and 60a of the neural networks 50 and 60.

In the first to third embodiments described above, in step S17, the processing units 10A, 20B, and 10B store the deep learning algorithm 60 in the algorithm database 105 in one-to-one correspondence with the type of organization. Alternatively, in step S17, the processing units 10A, 20B, and 10B may store one deep learning algorithm 60 in association with a plurality of tissue types in the algorithm database 105.

In the first to third embodiments described above, the hue is defined by a combination of three primary colors of light or a combination of three primary colors of color, but the number of hues is not limited to three. The number of hues may be the four primary colors of red (R), green (G), blue (B) plus yellow (Y), or the three primary colors of red (R), green (G), and blue (B). It is also possible to use two primary colors obtained by subtracting one of the hues from the primary colors. Alternatively, only one of the three primary colors red (R), green (G), and blue (B) (for example, green (G)) may be used. For example, the training images 70R1, 70R2, 70R3 and the analysis target image 78 acquired using a known microscope or virtual slide scanner are also color images of the three primary colors of red (R), green (G), and blue (B). The image is not limited to , and may be a color image of two primary colors, as long as it includes one or more primary colors.

In the first to third embodiments described above, in step S10, the processing units 10A, 20B, and 10B each color density encoded diagram (color density encoded diagram 70R1r, 70R1g, 70R1b, color density encoded diagram 70R1b, color density encoded diagram 70R2r, 70R2g, 70R2b, color density encoded diagrams 70R3r, 70R3g, 70R3b) are generated as 8-bit (256 steps) single color images of each primary color, but the primary color when creating each color density encoded diagram is Gradation is not limited to 8 bits. The gradation of each color density encoded map may depend on the imaging conditions of the slide image scanner. For example, the image may be one bit or more, and may be a 2-bit, 4-bit, 16-bit, or 24-bit image. Similarly, in step S21, the processing units 20A, 20B, and 10B generate the color density encoded diagrams 79r, 79g, and 79b as single color images of each primary color. The gradations are not limited to three gradations. The primary colors used when creating a color density encoded diagram are not limited to 8 bits. The gradations of the color density encoded diagrams 79r, 79g, and 79b may depend on the imaging conditions of the slide image scanner. For example, the image may be one bit or more, and may be a 2-bit, 4-bit, 16-bit, or 24-bit image. It is preferable that the gradations of each color density encoded diagram and the color density encoded diagrams 79r, 79g, and 79b of the training image are all the same gradation.

In the first to third embodiments described above, in step S10, the processing units 10A, 20B, and 10B generate each color density encoded diagram from the training image. A coded matrix table may be used as a training image. The processing units 10A, 20B, and 10B may directly acquire training images as color density encoded images from, for example, a virtual slide scanner. Similarly, in step S21, the processing units 20A, 20B, and 10B generate color density encoded diagrams 79r, 79g, and 79b for each color of R, G, and B from the analysis target image 78, but each pixel is A matrix table encoded according to the key may be used as an analysis image. That is, the processing units 20A, 20B, and 10B may directly acquire the color density encoded diagrams 79r, 79g, and 79b from, for example, a virtual slide scanner.

In the first to third embodiments described above, RGB is used as the color space when generating each color density encoded diagram from the training image, but the color space is not limited to RGB. In addition to RGB, various color spaces such as YUV, CMY, and CIE L ^* a ^* b ^* can be used.

In the first to third embodiments described above, in each region training data and analysis data 80, the density values for each pixel are stored in the order of red (R), green (G), and blue (B). , the order in which the density values are stored and handled is not limited to this. For example, the density values may be stored in the order of blue (B), green (G), and red (R), and the order of arrangement of density values in each region training data and the arrangement of density values in analysis data 80 It is sufficient if the order is the same.

In the first to third embodiments described above, the processing units 10A and 10B are realized as an integrated device, but the processing units 10A and 10B do not need to be integrated devices, and include the CPU 11, memory 12, storage unit 13, etc. may be placed in separate locations and connected via a network. The processing units 10A, 10B, the input unit 16, and the output unit 17 do not necessarily need to be placed in one place, but may be placed in separate places and connected to each other so as to be communicable via a network. The processing units 20A, 20B, and 20C are also similar to the processing units 10A and 10B.

In the first to third embodiments described above, a training data generation section 101, a training data input section 102, an algorithm update section 103, an analysis data generation section 201, an analysis data input section 202, an analysis section 203, and a region detection section 204. Each functional block is executed on a single CPU 11 or a single CPU 21, but each of these functional blocks does not necessarily need to be executed on a single CPU, but may be executed in a distributed manner on multiple CPUs. Good too. Further, each of these functional blocks may be executed in a distributed manner by a plurality of GPUs, or may be executed in a distributed manner by a plurality of CPUs and a plurality of GPUs.

In the second and third embodiments described above, programs for performing the processing of each step explained in FIGS. 11 and 14 are stored in advance in the storage units 13 and 23. Alternatively, the program may be installed in the processing unit 10B, 20B from a computer-readable, non-transitory tangible storage medium 98, such as a DVD-ROM or a USB memory. Alternatively, the processing units 10B and 20B may be connected to the network 99, and a program may be downloaded and installed from, for example, an external server (not shown) via the network 99.

In the first to third embodiments described above, the input units 16 and 26 are input devices such as a keyboard or a mouse, and the output units 17 and 27 are realized as display devices such as a liquid crystal display. Alternatively, the input sections 16, 26 and the output sections 17, 27 may be integrated into a touch panel type display device. Alternatively, the output units 17 and 27 may be configured with a printer or the like, and the ternary image 83 or the region-enhanced image 84 resulting from the analysis may be printed and output.

In the first to third embodiments described above, the imaging device 300 is directly connected to the deep learning device 100A or the image analysis device 100B; It may be connected to the device 100B. Similarly, the imaging device 400 is directly connected to the image analysis device 200A or the image analysis device 200B, but the imaging device 400 is connected to the image analysis device 200A or the image analysis device 200B via the network 99. May be connected.

<Verification of trained deep learning algorithm>
Deep learning processing and image analysis processing were performed using the stand-alone system shown in the second embodiment. Human gastric cancer tissue, which includes a tumor region and a normal region, was used as the tissue to be studied. To create a trained deep learning algorithm, training images were taken with the tissue sample magnified at different magnifications of 1x, 3x, 5x, 10x, 20x, and 40x. The neural network was trained on training data using training images, and the accuracy of the analysis results was confirmed.
The details of the learning data and analysis data were as follows.
Learning data: 106 hole slide images Verification analysis data: 45 hole slide images

[Creating training data and learning]
A whole slide image of a bright field image of HE-stained gastric cancer tissue was scanned using a virtual slide scanner (NanoZoomer-XR, (Hamamatsu Photonics; scanning resolution: 0.46 μm/pixel when scanning in 20x mode, 0.23 μm/pixel when scanning in 40x mode). ) was used to capture color images. The imaging magnification was 1x, 3x, 5x, 10x, 20x, and 40x. A pathologist designated a tumor region, a non-tumor region, and a non-tissue region on a whole slide image containing the tissue to be studied. A whole slide image in which each region was designated was divided according to the above conditions and used as a training image. It was determined whether each training image was a tumor region, a non-tumor region, or a non-tissue region. At that time, for tissue regions where tumor regions and non-tumor regions coexist, if the number of pixels designated as tumor regions occupies 50% or more of the number of pixels corresponding to tissue regions in the training image, It was defined as the tumor area. Furthermore, if the number of pixels designated as a non-tumor region accounts for 100% of the number of pixels corresponding to a tissue region in the training image, the region is determined to be a non-tumor region. For training images in which non-tissue areas and tissue areas coexist, if the tissue area is 0% of the number of pixels of the training image, it is considered a non-tissue area, and other areas are considered tissue areas. For each pixel of each training image determined to be a tumor region, non-tumor region, or non-tissue region, a label value that distinguishes each region (“1” for a tumor region, “2” for a non-tumor region, a non-tissue region was assigned "0"), and the first preliminary training data 70R1L, the second preliminary training data 70R2L, and the third preliminary training data 70R2L were generated. The training image determined to be a tumor region is set as the first training image, the training image determined to be a non-tumor region is set as the second training image 70R2, and the training image determined to be a non-tissue region is set as the first training image. 3 as the training image 70R3. For each training image, the color density values of each color of R, G, and B are gradated using 8 bits to generate a color density encoded diagram of each color of R, G, and B. The density coded maps were combined to generate first training data 74R1, second training data 74R2, and third training data 74R3, respectively.

Thereafter, tumor region training data was created by combining the first training data 74R1 and the first preliminary training data 70R1L. Non-tumor region training data was created by combining the second training data 74R2 and the second preliminary training data 70R2L. Non-organized region training data was created by combining the third training data 74R3 and the third preliminary training data 70R2L. The created training data for each region was divided into a window size of 200 x 200 pixels, and a neural network was trained using the training data of the divided window size as an input layer.

Training data for each region was generated for each magnification of the tissue specimen, and the neural network was trained for each magnification.

[Create image to be analyzed]
Similar to the training data, a whole slide image of a bright field image of HE-stained human gastric cancer tissue was imaged in color using a virtual slide scanner to obtain an image for analysis. The imaging magnification was 1x, 3x, 5x, 10x, 20x, and 40x. Thereafter, color density coding diagrams for each color of R, G, and B were created based on the captured analysis images, and data to be analyzed was generated by combining the created color density coding diagrams for each color of R, G, and B. . Data to be analyzed was generated for each magnification.

[Analysis result]
Data for analysis with a window size of 200 x 200 pixels was created from the data to be analyzed, and the data for analysis was input into a trained neural network corresponding to the magnification. Based on the analysis results output from the neural network, it is classified into tumor areas, non-tumor areas, and non-tissue areas (background), and the tumor areas obtained through the analysis process are displayed in white, and the non-tumor areas are are displayed in gray, and non-tissue areas are displayed in black. The analysis results are shown in FIG. 19.

In FIG. 19, (a) is a whole slide image of a bright field image obtained by staining gastric cancer tissue with HE, and is an image to be analyzed. (b) is a whole slide image in which a tumor region is designated by a pathologist and surrounded by a solid line. (c) shows an analysis result using a training image and an analysis image with a magnification of 1x. It was not possible to accurately indicate each area designated by the pathologist. (d) shows an analysis result using a training image and an analysis image with a magnification of 5 times. The tumor area designated by the pathologist could generally be determined as a tumor area. Furthermore, the non-tumor region specified by the pathologist could generally be determined as a non-tumor region. The non-tissue area designated by the pathologist could generally be determined as a non-tissue area. (e) shows an analysis result using a training image and an analysis image with a magnification of 40 times. The tumor area specified by the pathologist could be determined to some extent as a tumor area. However, in areas designated as non-tumor areas by pathologists, the number of areas determined to be tumor areas was increasing, and the rate of erroneously determining non-tumor areas as tumor areas was high.

The above results indicate that either too low or too high a magnification when acquiring training images and analysis images causes erroneous determinations.

In FIG. 20, (a) is a whole slide image of a bright field image obtained by staining gastric cancer tissue with HE, which is different from that shown in FIG. 19, and is the image to be analyzed. (b) is a whole slide image in which a tumor region is designated by a pathologist and surrounded by a solid line. (c) shows an analysis result using a training image and an analysis image with a magnification of 5 times. The tumor area designated by the pathologist could generally be determined as a tumor area. Furthermore, the non-tumor region specified by the pathologist could generally be determined as a non-tumor region. The non-tissue area designated by the pathologist could generally be determined as a non-tissue area. (d) shows an analysis result using a training image and an analysis image with a magnification of 20 times. As with the case of 5x magnification, each region specified by the pathologist could be generally distinguished.

Regarding the image for analysis at a magnification of 5 times, the content rate of the tumor region in the tissue region present in one whole slide image was calculated according to the following formula.

Content rate of tumor area (%) = {[Number of pixels determined to be a tumor area]/[(Number of pixels determined to be a tumor area) + (Number of pixels determined to be a non-tumor area)]}×100

The tumor area content was 54.5%.

Furthermore, FIG. 21 shows the sensitivity (a) and positive predictive value (b) of each magnification. From this result, it was considered that a magnification of 5 to 20 times is appropriate for the tissue sample when acquiring images for training and images for analysis. Furthermore, FIG. 21 shows changes in sensitivity (c) and positive predictive value (d) due to differences in the number of pixels in the window size. Images for analysis were taken at a magnification of 5 times. It was considered that the number of pixels of the window size is preferably from 125 x 125 pixels to 200 x 200 pixels.

The analysis time for one whole slide image was approximately several to 30 minutes.

<Additional notes>
An embodiment relates to an image analysis method for analyzing tissue images using a deep learning algorithm (60) having a neural network structure. In the image analysis method, analysis data (80) is generated from an analysis target image (78) including a tissue to be analyzed, and the analysis data (80) is input to a deep learning algorithm (60). (60) may generate data indicating whether the region included in the analysis data is a tumor region. According to the present invention, it is possible to generate data indicating whether a region included in analysis data in a tissue to be analyzed is a tumor region.

In the embodiment, preferably, the data indicating whether the area included in the analysis data is a tumor area is data indicating an area of the tumor area. This embodiment allows the user to grasp the tumor region in the tissue to be analyzed at a glance.

In the embodiment, preferably, the image to be analyzed is an image of a specimen for tissue diagnosis, and includes a hue that is a combination of two or more primary colors. This embodiment can assist in pathological diagnosis.

In the embodiment, preferably, the image to be analyzed is an image obtained by enlarging the tissue to be analyzed from 3 times to 20 times. In the embodiment, the size of the area included in the image to be analyzed is greater than or equal to 200 μm×200 μm and less than or equal to 400 μm×400 μm. These embodiments enable more accurate discrimination.

In the embodiment, preferably, data for distinguishing and presenting tumor cell regions and other regions is generated based on data indicating whether or not a region included in the analysis data is a tumor region. do. This embodiment allows the user to grasp the tumor region in the tissue to be analyzed at a glance.

In the embodiment, preferably, data indicating the boundary between the tumor cell area and other areas is generated based on data indicating whether the area included in the analysis data is a tumor area. This embodiment allows the user to grasp at a glance the boundary between the tumor region and non-tumor region in the tissue to be analyzed.

In the embodiment, preferably, data indicating the content rate of the tumor region in the tissue to be analyzed is generated based on data indicating whether the region included in the analysis data is a tumor region. This embodiment allows the user to grasp the content rate of tumor regions in the tissue to be analyzed.

In the embodiment, preferably, data indicating a ratio of a tumor area to a non-tumor area in the tissue to be analyzed is generated based on data indicating whether a region included in the analysis data is a tumor area. . This embodiment allows the user to grasp the content rate of tumor regions in the tissue to be analyzed. In addition, it is possible to know whether samples used for genetic testing, etc. are being collected appropriately.

According to the above-mentioned embodiment, data for distinguishing and presenting a tumor region, a non-tumor region, and a region not containing tissue is preferably generated for one analysis target image. This embodiment allows the user to grasp at a glance the presence or absence of a tumor region in the tissue to be analyzed.

In the embodiment, preferably, a plurality of pieces of analysis data are generated for each region of a predetermined number of pixels for one image to be analyzed. According to this embodiment, it is possible to accurately analyze a wide range of tissues to be analyzed.

In the embodiment, preferably, the deep learning algorithm (60) applies a label to the input analysis data indicating that the region of the predetermined number of pixels includes at least one selected from a tumor region and a non-tumor region. generate.

In the embodiment, preferably, the number of nodes in the input layer (50a) of the neural network (50) corresponds to the product of the predetermined number of pixels of the analysis data (80) and the number of combined primary colors. . This embodiment allows highly accurate analysis to be performed.

In the embodiment, preferably, the specimen is a stained specimen, and the image to be analyzed is an image of the stained specimen taken under a bright field of a microscope. This embodiment can assist in pathological diagnosis.

In the embodiment, preferably, the training data used for learning the deep learning algorithm is staining of a specimen prepared by applying bright field observation staining to a tissue specimen including a tumor region collected from an individual. The image is generated based on a bright field image taken under the bright field of a microscope. This embodiment can aid in tissue diagnosis.

In the embodiment, preferably, the staining for bright field observation uses hematoxylin for nuclear staining. More preferably, the staining for bright field observation is hematoxylin and eosin staining (HE staining). This embodiment makes it possible to cover highly versatile tests for tissue diagnosis.

In the embodiment, preferably the training data includes a label value determined from the bright field image indicating a tumor region. More preferably, the training data includes the label value for each area of a predetermined number of pixels of the bright field image. This embodiment allows the generation of training data for tumor regions.

In the embodiment, preferably, the training data is generated for each region of a predetermined number of pixels in the bright field image. This aspect makes it possible to increase the learning efficiency of the deep learning algorithm (50).

In the embodiment, preferably, the deep learning algorithm classifies the analysis data into a class indicating that the tissue included in the analysis target image is a tumor region.

In the embodiment, preferably, the output layer (50b) of the neural network (50) is a node whose activation function is a softmax function. This embodiment can improve learning efficiency and analysis accuracy.

In the embodiment, preferably, the deep learning algorithm extracts data indicating that the tissue included in the analysis target image is a tumor region into an area of a predetermined number of pixels each time the analysis data (78) is input. generated every time. This embodiment allows for increased analysis efficiency.

In said embodiment, preferably a deep learning algorithm (60) is generated depending on the type of tissue. Furthermore, preferably, the analysis data is processed using the deep learning algorithm corresponding to the type of tissue to be analyzed, which is selected from the plurality of deep learning algorithms according to the type of tissue. . This embodiment can improve analysis accuracy.

One embodiment relates to an image analysis device (100) that analyzes tissue images using a deep learning algorithm (60) with a neural network structure. The analysis device generates analysis data (80) from an analysis target image (78) including a tissue to be analyzed, inputs the analysis data (80) to a deep learning algorithm (50), and inputs the analysis data (80) to a deep learning algorithm (50). 50) may include a processing unit (10) that generates data indicating whether or not a region included in the analysis data is a tumor region.

Certain embodiments relate to a computer program that analyzes images of tissue using a deep learning algorithm (60) in a neural network structure. The computer includes a process of generating analysis data (80) from an analysis target image (78) including a tissue to be analyzed, a process of inputting the analysis data (80) into a deep learning algorithm (60), and a process of inputting the analysis data (80) into a deep learning algorithm (60). The learning algorithm (60) may perform a process of generating data indicating whether the region included in the analysis data is a tumor region.

An embodiment includes the step of generating training data used for learning the deep learning algorithm (50) based on a training image obtained from a training tissue sample, and causing the neural network (50) to learn the training data. a learning step, and the generation step includes a first acquisition step of acquiring first training data (74R1) corresponding to a first training image (70R1) that captures a tumor region; A second acquisition step of acquiring second training data (74R2) corresponding to a second training image (70R2) that captures a region, and a third training image (70R3) that captures a region that does not include tissue. ), the learning step includes a third acquisition step of acquiring third training data (74R3) corresponding to 50), a second learning step in which the neural network (50) learns that the second training data (74R2) is a non-tumor region, and the third training step. The present invention relates to a method for generating a trained deep learning algorithm, including a third learning step of causing a neural network (50) to learn that the data (74R3) is a region that does not include tissue. Preferably, the first training data (74R1), the second training data (74R2), and the third training data (74R3) are used as the input layer (50a) of the neural network (50), and The first training data (74R1), the second training data (74R2), and the third training data (74R2) respectively indicate that the area is a non-tumor area, and that the area does not contain tissue. 74R3) is the output layer (50b) of the neural network (50). More preferably, before the first acquisition step, a step of generating the first training data (74R1) from the first training image (70R1), and before the second acquisition step, , from the second training image (70R2), before the step of generating the second training data (74R2), and before the third acquisition step, from the third training image (70R3), The method may further include a step of generating the third training data (74R3). According to the present embodiment, it is possible to generate a trained deep learning algorithm that determines that the tissue to be analyzed includes at least one selected from a tumor region and a non-tumor region.

In the generation method, preferably, the training image is a bright field image obtained by capturing a stained image of a specimen prepared by performing bright field observation staining on a tissue collected from an individual under the bright field of a microscope. It is.

In one embodiment, the first training data (74R1), the second training data (74R2), and the third training data (74R3) are used as the input layer (50a) of the neural network (50), and the tumor region The first training data (74R1), the second training data (74R2), and the third training data indicate that the area is a non-tumor area, the area is a non-tumor area, and the area does not contain tissue, respectively. (74R3) is a deep learning algorithm trained as an output layer (50b) of a neural network (50) corresponding to The second training data (74R2) is generated from the first training image (70R1), and the second training data (74R2) is generated from the second training image (70R2) capturing a non-tumor region of the training tissue. The third training data (74R3) may be a trained deep learning algorithm (60) generated from a third training image (70R3) that captures a region that does not include tissue.

10 (10A, 10B) Processing unit 20 (20A, 20B, 20C) Processing unit 11, 21 CPU
12, 22 Memory 13, 23 Recording section 14, 24 Bus 15, 25 Interface section 16, 26 Input section 17, 27 Output section 19, 29 GPU
50 Neural network (deep learning algorithm)
50a Input layer 50b Output layer 50c Middle layer 60 Trained neural network (trained deep learning algorithm)
60a Input layer 60b Output layer 60c Intermediate layer 70W1 Hole slide image 70W2 Hole slide image 70W3 Hole slide image 70R1L First preliminary training data 70R2L Second preliminary training data 70R3L Third preliminary training data 72R1r, 72R1g, 72R1b Color density code Color density encoding diagrams 72R2r, 72R2g, 72R2b Color density encoding diagrams 72R3r, 72R3g, 72R3b Color density encoding diagrams 74R1, 74R2, 74R3 Training data 75R1, 75R2, 75R3 Window size training data 76 Color density values 77R1, 77R2, 77R3 True value Image label value 78 Bright field images to be analyzed 79r, 79g, 79b Color density encoding diagram for analysis target 80 Data for analysis 81 Color density value 82 Discrimination result (estimated value of pixel)
83 Analysis result ternary image 84 Region-enhanced image 89 (89a, 89b) Node 98 Recording medium 99 Network 100 Vendor device 100A Deep learning device 100B Integrated image analysis device 101 Training data generation unit 102 Training data input unit 103 Algorithm Update unit 104 Window size database 105 Algorithm database 200 User device 200A Image analysis device 200B Integrated image analysis device 200C Terminal device 201 Analysis data generation unit 202 Analysis data input unit 203 Analysis unit 204 Region detection unit 300, 400 Imaging Apparatus 301, 401 Image sensor 308, 408 Sample tissue 309, 409 Stage W1 Window W2 Window

Claims (22)

An image analysis method for analyzing tissue images using a deep learning algorithm with a neural network structure,
Selecting the deep learning algorithm corresponding to the type of tissue to be analyzed from a plurality of deep learning algorithms generated according to the type of tissue,
Generating analysis data from the analysis target image including the analysis target tissue,
inputting the analysis data to the selected deep learning algorithm;
Displaying an image of the tissue to be analyzed so that a tumor region and other regions can be distinguished based on an analysis result output from the deep learning algorithm into which the analysis data has been input;
Image analysis method. the plurality of deep learning algorithms are stored in a database;
The image analysis method according to claim 1, wherein the deep learning algorithm corresponding to the type of tissue to be analyzed is selected from the plurality of deep learning algorithms stored in the database. The image analysis method according to claim 2, wherein each of the plurality of deep learning algorithms is stored in the database in association with a tissue type. The image analysis method according to claim 3, wherein at least one of the plurality of deep learning algorithms is stored in the database in association with one tissue type. The image analysis method according to any one of claims 1 to 4, wherein the deep learning algorithm corresponding to the type of tissue to be analyzed is selected based on a user's operation via an input unit. Any one of claims 1 to 5, further comprising subjecting the image to be analyzed to a drawing process for displaying a tumor region and other regions in a distinguishable manner based on an analysis result output from the deep learning algorithm. The image analysis method according to item 1. 7. The image analysis method according to claim 6, wherein the drawing process includes a process of drawing the tumor area and other areas in different colors. 8. The image analysis method according to claim 1, wherein the image to be analyzed is an image of a specimen for tissue diagnosis, and the image to be analyzed includes a hue that is a combination of two or more primary colors. The image analysis method according to any one of claims 1 to 8, wherein the image to be analyzed is an image obtained by enlarging the tissue to be analyzed from 3 times to 20 times. The image analysis method according to any one of claims 1 to 9, wherein the analysis data corresponds to an image cut out from the analysis target image, and a plurality of the analysis data are generated from the analysis target image. The image to be analyzed is an image of a specimen for tissue diagnosis, the specimen for tissue diagnosis is a stained specimen, and the image to be analyzed is an image obtained by imaging the stained specimen under a bright field of a microscope. The image analysis method according to any one of claims 1 to 10. The training data used for learning the deep learning algorithm is a stained image of a specimen prepared by performing bright field observation staining on a tissue specimen including a tumor region collected from an individual under the bright field of a microscope. The image analysis method according to claim 11, wherein the image analysis method is generated based on a captured bright field image. 13. The image analysis method according to claim 1, wherein the output layer of the neural network is a node whose activation function is a softmax function. An image analysis device that analyzes tissue images using a deep learning algorithm with a neural network structure,
A display section;
The deep learning algorithm corresponding to the type of tissue to be analyzed is selected from a plurality of deep learning algorithms generated according to the type of tissue, and data for analysis is generated from the image to be analyzed including the tissue to be analyzed. , the analysis data is input to the selected deep learning algorithm, and based on the analysis result output from the deep learning algorithm into which the analysis data is input, tumor regions and other regions can be distinguished. a control unit configured to display an image of the tissue to be analyzed on the display unit;
Image analysis device. further comprising a storage unit that stores the plurality of deep learning algorithms,
The image analysis apparatus according to claim 14, wherein the control unit selects the deep learning algorithm corresponding to the type of tissue to be analyzed from the plurality of deep learning algorithms stored in the storage unit. The image analysis device according to claim 15, wherein the storage unit stores each of the plurality of deep learning algorithms in association with a tissue type. The image analysis device according to claim 16, wherein the storage unit stores at least one of the plurality of deep learning algorithms in association with one tissue type. further comprising an input section operated by a user,
The image analysis device according to any one of claims 14 to 17, wherein the control unit selects the deep learning algorithm corresponding to the type of tissue to be analyzed based on an operation via the input unit. . Any one of claims 14 to 18, wherein the control unit subjects the analysis target image to a drawing process for displaying a tumor region and other regions in a distinguishable manner based on the generated analysis data. The image analysis device described in section. The image analysis apparatus according to claim 19, wherein the drawing process includes a process of drawing the tumor area and other areas in different colors. The image analysis apparatus according to any one of claims 14 to 20, wherein the analysis data corresponds to an image cut out from the analysis target image, and a plurality of pieces of analysis data are generated from the analysis target image. A computer program that analyzes tissue images using a deep learning algorithm with a neural network structure,
to the computer,
a process of selecting the deep learning algorithm corresponding to the type of tissue to be analyzed from a plurality of deep learning algorithms generated according to the type of tissue;
A process of generating analysis data from the analysis target image including the analysis target tissue;
a process of inputting the analysis data to the selected deep learning algorithm;
Displaying an image of the tissue to be analyzed in such a way that a tumor region and other regions can be distinguished based on an analysis result output from the deep learning algorithm into which the analysis data has been input;
computer program.