WO2022260380A1 - Lymph node metastasis prediction method using endoscopic resection sample image of early colon cancer, and analysis device - Google Patents

Abstract

A lymph node metastasis prediction method using an endoscopic resection sample image of early colon cancer, comprises steps in which: an analysis device receives a stained sample image of the colon area of a patient with colon cancer; the analysis device inputs the sample image into a deep learning model trained in advance; and the analysis device predicts, on the basis of a value output by the deep learning model having received the sample image, whether lymph node metastasis has occurred in the patient with colon cancer. The deep learning model receives patch images in which the whole sample image is divided into a plurality of areas, and outputs, on the basis of the feature about the whole sample image to which weights for the patch images are provided, a prediction value about whether lymph node metastasis has occurred in the patient.

Description

Method and analysis device for predicting lymph node metastasis using endoscopic resection specimen images of early colorectal cancer

The technique described below is a technique for predicting whether or not metastases to the lymph nodes in colorectal cancer patients who have undergone endoscopic resection.

Early colon cancer refers to colorectal cancer in which cancer cells have invaded the mucosa or submucosa. Early colorectal cancer confined to the mucosal layer can be cured by complete resection of only the primary tumor because cancer cells remain only in the primary site without distant metastasis, including lymph nodes. On the other hand, about 10% of early colorectal cancers infiltrating the submucosa show lymph node metastasis.

The current universal guideline is to perform endoscopic resection first for early colorectal cancer with indications for endoscopic resection, and if mucosal invasion of the tumor exceeds a certain standard after endoscopic resection, the risk of lymph node metastasis is high, and additional resection surgery is recommended. have. However, even when additional surgery is performed according to the guidelines, 75 to 98% of the surgical specimen test results are not confirmed as actual lymph node metastasis.

Clinical judgment on endoscopically resected specimens utilizes staining images of the specimen. However, it is very difficult to predict whether colorectal cancer has metastasized to the lymph nodes based only on the staining image of the endoscopically resected specimen.

The technique described below is intended to provide a technique for predicting whether or not lymph node metastasis of colorectal cancer is based on an endoscopic resection specimen. The technique to be described below is intended to provide a technique for predicting whether colorectal cancer has metastasized to a lymph node using only a staining image of a specimen.

A method for predicting lymph node metastasis using endoscopically resected specimen images of early colorectal cancer includes the step of receiving a stained specimen image of the colon of a colorectal cancer patient as an input to an analysis device, the specimen image to a deep learning model trained in advance by the analysis device and predicting whether or not lymph node metastasis to the colorectal cancer patient is present based on a value output by the deep learning model having received the specimen image by the analysis device. The deep learning model receives patch images in which the entire sample image is divided into a plurality of regions, and determines whether the patient has metastasis to the lymph nodes based on the characteristics of the entire sample image to which weights are assigned to the patch images. output the predicted value for

An analysis device for predicting lymph node metastasis of early colorectal cancer includes an input device that receives a stained sample image of the large intestine of a colorectal cancer patient, a storage device that stores a deep learning model, and the deep learning model that receives the sample image. and an arithmetic unit for predicting whether the colorectal cancer patient has metastasis to a lymph node based on an output value. The deep learning model receives patch images in which the entire sample image is divided into a plurality of regions, and determines whether the patient has metastasis to the lymph nodes based on the characteristics of the entire sample image to which weights are assigned to the patch images. output the predicted value for

The technology described below accurately predicts lymph node metastasis for patients who have undergone endoscopic resection using an artificial intelligence model. The technology described below predicts the need for additional surgery quickly and with high accuracy using only the staining image of the specimen.

1 is an example of a system for predicting lymph node metastasis for a patient with early colorectal cancer.

2 is an example of a deep learning model predicting lymph node metastasis of early colorectal cancer.

3 is an example of a learning process of a deep learning model.

4 is an example of an analysis device.

5 is an AUC result of a deep learning model.

Since the technology to be described below can have various changes and various embodiments, specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the technology described below to specific embodiments, and it should be understood to include all modifications, equivalents, or substitutes included in the spirit and scope of the technology described below.

Terms such as first, second, A, B, etc. may be used to describe various elements, but the elements are not limited by the above terms, and are merely used to distinguish one element from another. used only as For example, without departing from the scope of the technology described below, a first element may be referred to as a second element, and similarly, the second element may be referred to as a first element. The terms and/or include any combination of a plurality of related recited items or any of a plurality of related recited items.

In terms used in this specification, singular expressions should be understood to include plural expressions unless clearly interpreted differently in context, and terms such as “comprising” refer to the described features, numbers, steps, operations, and components. , parts or combinations thereof, but it should be understood that it does not exclude the possibility of the presence or addition of one or more other features or numbers, step-action components, parts or combinations thereof.

Prior to a detailed description of the drawings, it is to be clarified that the classification of components in the present specification is merely a classification for each main function in charge of each component. That is, two or more components described below may be combined into one component, or one component may be divided into two or more for each more subdivided function. In addition, each component to be described below may additionally perform some or all of the functions of other components in addition to its main function, and some of the main functions of each component may be performed by other components. Of course, it may be dedicated and performed by .

In addition, in performing a method or method of operation, each process constituting the method may occur in a different order from the specified order unless a specific order is clearly described in context. That is, each process may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

The technique described below is a technique for predicting lymph node metastasis by analyzing a specimen of a colorectal cancer or early colorectal cancer patient. Specimens from early colorectal cancer patients can be obtained in a variety of ways. For example, a patient's specimen may be obtained during an endoscopic resection procedure.

Clinical examination of tissues generally uses stained specimens. Staining can use a variety of reagents. Representatively, hematoxylin and eosin (hereinafter referred to as H&E) can be used for specimen staining. The technique described below uses stained specimen images to predict lymph node metastasis. Hereinafter, for convenience of description, the H&E image will be described as a standard. Since the sample is stained in slide units, it is also called WSI (whole slide image). Therefore, the image used for analysis can be named H&E WSI.

The technology to be described below analyzes H&E images using a learning model to predict metastasis to the lymph nodes. The learning model means a machine learning model. A learning model is meant to include various types of models. For example, a learning model includes a decision tree, a random forest, a K-nearest neighbor (KNN), a Naive Bayes, a support vector machine (SVM), an artificial neural network, and the like.

An artificial neural network is a statistical learning algorithm that mimics a biological neural network. Various neural network models are being studied. Recently, deep learning networks (DNNs) are attracting attention. DNN is an artificial neural network model consisting of several hidden layers between an input layer and an output layer. DNNs, like general artificial neural networks, can model complex non-linear relationships. Various types of DNN models have been studied. For example, there are a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a Restricted Boltzmann Machine (RBM), a Deep Belief Network (DBN), a Generative Adversarial Network (GAN), and Relation Networks (RL).

The technology described below predicts lymph node metastasis using a deep learning model. The structure of a specific deep learning model will be described later.

On the other hand, a device that analyzes whether or not metastases to the lymph node using the H&E image of the specimen is called an analysis device. The analysis device is a device capable of processing data and may be in the form of a PC, a smart device, or a server.

1 is an example of a system 100 for predicting lymph node metastasis for a patient with early colorectal cancer. 1 illustrates an example in which the analysis device is a computer terminal 150 and a server 250 .

1(A) is a system 100 in which a user R predicts lymph node metastasis using a computer terminal 150. The dyed image generating device 110 is a device for generating a dyed image (H&E WSI) by staining a tissue slide and scanning the stained result. Alternatively, the dyed image generation device 110 is a device that generates H&E WSI for a slide stained by a researcher.

The dyed image generating device 110 transmits the H&E WSI to the computer terminal 150 through a wired or wireless network. The computer terminal 150 predicts whether the sample has metastasis to the lymph nodes based on the H&E WSI. The computer terminal 150 predicts metastasis to the lymph nodes using a pre-learned deep learning model. The deep learning model receives the H&E WSI and outputs a predicted probability value for lymph node metastasis for the sample. The deep learning model will be described later. The computer terminal 150 provides the analysis result to the user R.

1 shows the dyed image generating device 110 and the computer terminal 150 as separate objects. However, the dyed image generating device 110 and the computer terminal 150 may be physically implemented as one device or a connected device.

1(B) is a system 200 in which a user accesses an analysis server 250 through a user terminal 270 and predicts lymph node metastasis.

The dyed image generation device 210 is a device for generating a dyed image (H&E WSI) by staining a tissue slide and scanning the stained result. Alternatively, the dyed image generation device 210 is a device that generates H&E WSI for a slide stained by a researcher. The dyed image generating device 210 transmits the H&E WSI to the analysis server 250 through a wired or wireless network.

The analysis server 250 predicts whether the sample has metastasis to the lymph nodes based on the H&E WSI. The analysis server 250 predicts whether lymph node metastasis is present using a pre-learned deep learning model. The deep learning model receives the H&E WSI and outputs a predicted probability value for lymph node metastasis for the sample. The deep learning model will be described later.

The analysis server 250 may transmit the analysis result to the user terminal 270 . In addition, the analysis server 250 may transmit the analysis result to the EMR system 260 of the hospital.

2 is an example of a deep learning model 300 for predicting lymph node metastasis of early colorectal cancer. The deep learning model 300 may include a feature extraction layer 310, an attention layer 320, and a classification layer 330. That is, the deep learning model 330 may be referred to as an attention-based CNN model.

When the H&E WSI of a specific patient is input, the analysis device divides the tissue region of the H&E WSI to extract patch images. Hereinafter, one divided (classified) image is called an H&E WSI patch.

The analysis device inputs each H&E WSI patch to the deep learning model 300, and the deep learning model 300 outputs a classification result for each H&E WSI patch. The deep learning model 300 extracts features of the input data and outputs a value of lymph node metastasis positive or lymph node metastasis negative based on the extracted features.

The feature extraction layer 310 may be implemented in various forms. For example, the feature extraction layer 310 may include a plurality of convolution layers and pooling layers. The feature extraction layer 310 may include various activation functions.

The feature extraction layer 310 outputs a feature vector for one H&E WSI patch.

The attention layer 320 may receive the feature vector and assign an attention score to the corresponding H&E WSI patch. The attention layer 320 adds a relative importance contributing to the prediction of lymph node metastasis to each H&E WSI patch. The higher the value of the attention score, the more it affects the determination of lymph node metastasis.

The analysis device may calculate a feature vector for one H&E WSI by performing a weighted average on the feature vectors of all H&E WSI patches based on the attention score. Then, the analysis device may input feature vectors for the entire H&E WSI to the classification layer 330.

A feature vector for K patches of H&E WSI is defined as H = {h ₁ ,...,h _k ). K may be a different value for each H&E WSI. The H&E WSI feature value h _wsi can be expressed as in Equation 1 below.

α is a weight given to the feature of the H&E WSI patch in the attention layer 320. α corresponds to the aforementioned attention score. α has a value between 0 and 1. The sum of the weights for all H&E WSI patches is 1.

W ∈ R ^L×1 and V ∈ R ^L×M are parameters of the attention layer 320 . M is the dimensional length of the feature vector of each H&E WSI patch, and L may be set to half of M. tanh (hyperbolic tangent) is the activation function. tanh provides non-linear information about the feature vector, and tanh scales the feature vector to a value between -1 and 1 in order to detect similarities and dissimilarity between patches in H&E WSI.

The attention layer 320 may be composed of two fully connected layers (FNs). The first FN may have a tanh activation function, and the second FN may have a softmax function. Finally, the attention layer 320 performs attention-based pooling on all input patch feature vectors h _k as shown in Equation 1, and outputs an average WSI feature h _wsi for one WSI.

The classification layer 330 may be implemented in various forms. For example, the classification layer 330 may include a fully connected network (FN) widely used in general CNNs. The classification layer 330 may include various activation functions. Also, the classification layer 330 may include a classification function (eg, softmax) for final classification. The classification layer 330 may receive a feature vector for the entire H&E WSI and output a predicted probability value for lymph node metastasis for a corresponding sample.

Hereinafter, a process of learning the deep learning model 300 will be described. The researcher explains the process of building a deep learning model.

3 is an example of a learning process 400 of a deep learning model. For deep learning model learning, a researcher or developer learns a deep learning model using a computer device such as a PC or server. Therefore, it will be described below that the computer device performs the learning process of the deep learning model.

The deep learning model 300 of FIG. 2 was prepared through a two-step learning process. In the first learning process (A), the computer device prepares the feature extraction layer 310 of the deep learning model 300. Then, in the second learning process (B), the computer device prepares the entire deep learning model 300 including the feature extraction layer 310 prepared in the first learning process.

The first learning process (A) is a process of learning a CNN-based first model that extracts and classifies features of a patch-level input image. The CNN-based first model can be divided into a feature extraction layer (a) for extracting features and a classification layer (b) for classifying based on the extracted features. The feature extraction layer (a) may be implemented in various forms. For example, the feature extraction layer (a) may include a plurality of convolution layers and pooling layers. The feature extraction layer (a) may include various activation functions. The classification layer (b) may also be implemented in various forms. For example, the classification layer (b) may include FNs widely used in general CNNs. The classification layer (b) may also include various activation functions. Also, the classification layer (b) may include a classification function (eg, softmax) for final classification.

The researcher chose the first model with the ResNet 18 structure. Patches were extracted from non-overlapping tissue regions with a size of 256 × 152 (2 μm/pixel) magnified 5 times in all H&E WSIs. A tissue region was defined as a case where more than 70% of pixel values had values less than 200 in the gray scale color space.

The researcher prepared H&E WSI (positive WSI) for the positive group in advance (410). The positive group refers to patients who underwent endoscopic resection and were confirmed to have lymph node metastasis as a result of the sample test.

In addition, the researcher prepared H&E WSI (negative WSI) for the negative group (420). The negative group refers to patients who underwent endoscopic resection but no lymph node metastasis was confirmed.

On the other hand, the researcher prepared learning data (H&E WSIs for the specimen) in the following process.

Specimen data are data of 691 patients who underwent additional surgery within 3 months after endoscopic mucosal resection (EMR) or endoscopic submucosal dissection (ESD) at Samsung Seoul Hospital between 2010 and 2018. was used. The researcher excluded patients whose interpretation of H&E images was not clear, patients who had not undergone lymph node dissection, or patients with synchronous invasive carcinoma from the sample data. Through this process, the researcher finally extracted the data of 400 patients. For actual learning data, H&E WSI is used, and 430 H&E WSI were prepared for 400 people. Of the 430 H&E WSIs, 82 were positive for lymph node metastasis and 348 were negative for lymph node metastasis. 430 H&E WSIs were randomly selected and training data and verification data were prepared at a 4:1 ratio, respectively.

The researcher used only the patient's H&E image, and did not use additional clinical information (age, gender, family history, smoking status, tumor type, tumor location, etc.) of the patient as learning data.

Specimens obtained by endoscopic resection were fixed in formaldehyde and paraffin blocks were made. After H&E staining of the slides, H&E WSI was acquired using a VENTANA iScan HT scanner (using 20x magnification). The researcher selected only images that convey pathologically clear information among the acquired H&E WSI as learning data.

The computer device generates an input data group by dividing the positive group H&E WSI and the negative group H&E WSI into patch units, respectively (430). The computer device may pre-process the size, direction, and color of the H&E WSI patch for learning to be advantageous for learning.

Each learning data may be composed of patch unit images and information on the clinical results (whether or not metastases to the lymph nodes) of the patient of the corresponding images. Lymph node metastasis was labeled as 1 if metastasis positive, and as 0 if metastasis negative.

The computer device classifies the input H&E WSI by inputting the patch unit H&E WSI to the CNN-based first model. The first model extracts features of the input data and outputs a value of positive lymph node metastasis or a value of negative lymph node metastasis based on the extracted features. The first model outputs a value between 0 and 1 corresponding to the prediction result of lymph node metastasis.

The computer device compares the label value of the currently input H&E WSI patch with the value currently output by the first model, and adjusts the parameter of the first model if the output value is incorrect. The computer device optimizes the parameters of the model while repeating this process for a plurality of training data (440). The researcher performed learning to minimize cross entropy loss with a learning rate of 0.0001 using the RAdam optimizer.

The learning process (B) is a process of learning the second model based on attention. The second model corresponds to the deep learning model 300 described above. The second model includes a feature extraction layer 310, an attention layer 320, and a classification layer 330. In the learning process (B), the feature extraction layer 310 imports and uses the feature extraction layer (a) that has been learned in the learning process (A).

That is, the computer device removes the classification layer (FN) from the ResNet 18 learned in the learning process (A) and connects the remaining learned feature extraction layer (a = 310) with the new attention layer 320 and classification layer 330 to construct the second model.

The computer device inputs the H&E WSI patch for training to the second model. Meanwhile, the researcher used a feature extraction layer 310 that outputs a 512-dimensional feature vector for each H&E WSI patch. Based on this, it is explained. The feature vector output from the feature extraction layer 310 is input to the attention layer 320. The attention layer 320 may output WSI features h _wsi ∈ R ⁵¹² and concatenated 11-dimensional clinical features. The classification layer 330 may output the final binary classification result by receiving WSI features and clinical features output from the attention layer 320 through 512 channels and 256 channels, respectively. The computer device repeats the process of optimizing the parameters of the model based on the output result of inputting the training H&E WSI patches to the second model (450).

The researcher trained the second attention-based model in the direction of minimizing the cross-entropy loss with a learning rate of 0.0001 using the RAdam optimizer.

In the learning process (B), the computer device may set only the remaining attention layer 320 and the classification layer 330 as learning targets, excluding the feature extraction layer 310 learned in the learning process (A) among the second models. have. Alternatively, in the learning process (B), the computer device may optimize parameters by taking all layers including the feature extraction layer 310 as learning targets.

4 is an example of the analysis device 500. The analysis device 500 corresponds to the above-described analysis devices ( 150 and 250 in FIG. 1 ). The analysis device 500 may be physically implemented in various forms. For example, the analysis device 500 may have a form of a computer device such as a PC, a network server, and a data processing dedicated chipset.

The analysis device 500 may include a storage device 510, a memory 520, an arithmetic device 530, an interface device 540, a communication device 550, and an output device 560.

The storage device 510 may store a deep learning model for predicting lymph node metastasis based on the H&E WSI of a colorectal cancer patient.

The storage device 510 may store the input H&E WSI.

The storage device 510 may store images obtained by processing input H&E WSI in patch units.

The storage device 510 may store other programs or codes for image processing.

In addition, the storage device 510 may store instructions or program codes for a process of predicting metastasis to a lymph node through the process described above.

The storage device 510 may store analysis results.

The memory 520 may store data and information generated in the course of the analysis device 500 predicting metastasis to a lymph node.

The interface device 540 is a device that receives certain commands and data from the outside. The interface device 540 may receive the H&E WSI of the analysis target from a physically connected input device or an external storage device.

The communication device 550 refers to a component that receives and transmits certain information through a wired or wireless network. The communication device 550 may receive the H&E WSI of the analysis target from the external object. Alternatively, the communication device 550 may transmit the analysis result to an external object such as a user terminal.

Since the interface device 540 and the communication device 550 are configured to receive certain information and images from a user or other physical object, they can also be collectively referred to as input devices. Alternatively, the input device may mean an interface of a path that transfers the H&E WSI or request received from the communication device 550 to the inside of the analysis device 500.

The output device 560 is a device that outputs certain information. The output device 560 may output interfaces and analysis results necessary for data processing.

The arithmetic device 530 may predict metastasis to the lymph node for the analysis target using instructions or program codes stored in the storage device 510 .

The arithmetic device 530 may predict metastasis to a lymph node using a deep learning model stored in the storage device 510 .

The computing device 530 may predict whether metastases to the lymph nodes based on output values of the deep learning model to which the H&E WSI patches are input.

The computing device 530 may divide the H&E WSI into a plurality of H&E WSI patches.

The arithmetic device 530 may perform certain image processing on the plurality of H&E WSI patches classified (divided).

The computing device 530 may calculate features for the entire H&E WSI by collecting features weighted for each of the H&E WSI patches by the attention layer of the deep learning model.

The computing device 530 may transfer the characteristics of the entire H&E WSI to the classification layer to obtain a final prediction value for metastasis to the lymph nodes.

The arithmetic device 530 may be a device such as a processor, an AP, or a chip in which a program is embedded that processes data and performs certain arithmetic operations.

Hereinafter, the effect of predicting lymph node metastasis in patients with early colorectal cancer using the aforementioned deep learning model will be described. The researcher prepared data for verification in the process of preparing learning data, and tested the effect of the learned model using the verification data.

Table 1 below is information on all specimens (patients) described in the learning process together with FIG. 4 .

		Total (n=400)	Negative LN metastasis (n=329)	Positive LN metastasis (N=71)	p value
Age at diagnosis	Year (IQR)	59.0 (52.0-65.0)	59.0 (52.0-65.0)	60.0 (52.0-68.0)	0.337
Gender	Male (%) Female (%)	239 161	193 (58.7) 136 (41.3)	46 (64.8) 25 (35.2)	0.354
Body mass index	Kg/m 2 (IQR)	24.1 (22.2-26.3)	23.9 (22.0-26.1)	24.8 (23.3-27.4)	0.006
Presence of comorbidity	No (%) Yes (%)	253 147	217 (66.0) 112 (34.0)	36 (50.7) 35 (49.3)	0.021
Family history of CRC	No (%) yes (%)	345 55	294 (86.3) 45 (13.7)	61 (85.9) 10 (14.1)	1.000
Smoking status	No (%) Ex-smoker (%) Yes (%)	257 69 74	214 (65.0) 59 (17.9) 56 (17.0)	43 (60.6) 10 (14.1) 18 (25.4)	0.238
Alcohol consumption	No (%) Ex-drinker (%) Yes (%)	223 141 36	192 (58.4) 110 (33.4) 27 (8.2)	31 (43.7) 31 (43.7) 9 (12.7)	0.071
Endoscopic morphology of tumor	Sessile (%) Semi-pedunculated (%) Pedunculated (%)	48 316 36	38 (11.6) 264 (80.2) 27 (8.2)	10 (14.1) 52 (73.2) 9 (12.7)	0.373
Tumor location	Left side (%) Right side (%)	291 109	241 (73.3) 88 (26.7)	50 (70.4) 21 (29.6)	0.660

The researcher prepared data for model construction based on the data of 400 patients, excluding 291 patients out of a total of 691 patients with early colorectal cancer. For 400 patients, 430 H&E WSIs were prepared. Of the 430 H&E WSIs, 82 were positive for lymph node metastasis and 348 were negative for lymph node metastasis. 430 H&E WSIs were randomly selected and training data and verification data were prepared at a 4:1 ratio, respectively. The training data consisted of data from 57 patients with lymph node metastasis and data from 263 patients without lymph node metastasis. The verification data consisted of data from 14 patients with lymph node metastasis and data from 66 patients without lymph node metastasis. The researcher constructed models for various cases and performed verification. The verification results are shown in Table 2 below. The values in Table 2 are AUC (area under the ROC).

Cross-validation on train Set	Only Clinical Features	Only Step-1 Alone	Step-2 with Clinical Features	Step-2 without Clinical Features
One	0.612	0.783	0.758	0.772
2	0.455	0.745	0.786	0.781
3	0.685	0.676	0.685	0.683
4	0.441	0.694	0.764	0.780
5	0.611	0.663	0.719	0.724
Average of five-folds	0.561	0.712	0.742	0.747
Test set	0.367	0.705	0.755	0.764

Based on the average of 5 trials, the case of using only clinical information of patients without using H&E WSI (only clinical features) has AUC of 0.561. In the case of using the deep learning model without attention based on the H&E WSI (only step 1 alone), the AUC is 0.705. The attention-based deep learning model (Step 2 with Clinical Feature) using the patient's clinical information and H&E WSI has an AUC of 0.742. The attention-based deep learning model (Step 2 without Clinical Feature) using only the H&E WSI without the patient's clinical information has an AUC of 0.747. As a result, it can be said that the performance is the best when only the H&E WSI is used for the attention-based deep learning model described in FIG. 3 . Therefore, the above-described deep learning model can be said to sufficiently predict lymph node metastasis in early colorectal cancer patients using only the H&E WSI without clinical information of the patient. FIG. 5 shows the AUC results of the deep learning model. 5 is the performance of an attention-based deep learning model using only H&E WSI. Referring to FIG. 5 , the average AUC was 0.747 as a result of 5-fold cross verification on the verification data. In addition, as a result of ensembling the models created in each fold and experimenting on the verification data, AUC was 0.764. In addition, since false negatives cannot be tolerated in actual clinical applications, when 100% sensitivity is guaranteed, the specificity for the training data is 28.1% and the specificity for the validation data is 45.1%. was

In addition, the above-described method for analyzing medical images and predicting lymph node metastasis of colorectal cancer may be implemented as a program (or application) including an executable algorithm that can be executed on a computer. The program may be stored and provided in a temporary or non-transitory computer readable medium.

A non-transitory readable medium is not a medium that stores data for a short moment, such as a register, cache, or memory, but a medium that stores data semi-permanently and can be read by a device. Specifically, the various applications or programs described above are CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM (read-only memory), PROM (programmable read only memory), EPROM (Erasable PROM, EPROM) Alternatively, it may be stored and provided in a non-transitory readable medium such as EEPROM (Electrically EPROM) or flash memory.

Temporary readable media include static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), and enhanced SDRAM (Enhanced SDRAM). SDRAM, ESDRAM), Synchronous DRAM (Synclink DRAM, SLDRAM) and Direct Rambus RAM (DRRAM).

This embodiment and the drawings accompanying this specification clearly represent only a part of the technical idea included in the foregoing technology, and those skilled in the art can easily understand it within the scope of the technical idea included in the specification and drawings of the above technology. It will be obvious that all variations and specific examples that can be inferred are included in the scope of the above-described technology.

Claims (10)

1. receiving a stained sample image of a colon of a colon cancer patient by an analysis device;

   inputting the sample image to a deep learning model learned in advance by the analysis device; and

   Predicting whether the analysis device has metastasized to the lymph nodes of the colorectal cancer patient based on a value output by the deep learning model receiving the sample image,

   The deep learning model receives patch images in which the entire sample image is divided into a plurality of regions, and determines whether the patient has metastasis to the lymph nodes based on the characteristics of the entire sample image to which weights are assigned to the patch images. A method for predicting lymph node metastasis using early colorectal cancer endoscopic resection specimen images that output predictive values for
2. According to claim 1,

   The sample image is a method for predicting lymph node metastasis using an endoscopic resection sample image of early colorectal cancer, which is an H&E (Haematoxylin and eosin) staining image for a sample detected by endoscopic resection.
3. According to claim 1,

   The deep learning model is

   a feature extraction layer extracting features for each of the patch images;

   For each of the patch images, the feature output by the feature extraction layer is weighted by the degree of contribution of each patch image to the prediction of lymph node metastasis, and the features in which the degree of contribution is weighted for each of the patch images are averaged to obtain the specimen image an attention layer that calculates features for the whole; and

   A method for predicting lymph node metastasis using an endoscopic resection specimen image of early colorectal cancer, comprising a classification layer receiving characteristics of all of the specimen images output from the attention layer and outputting the prediction value.
4. According to claim 1,

   The deep learning model includes a feature extraction layer for extracting features for each of the patch images,

   The feature extraction layer is calculated by removing a classification layer from a separate convolutional neural network (CNN) model learned using separate training data,

   The learning data is a method for predicting lymph node metastasis using endoscopic resection specimen images of early colorectal cancer, including stained specimen images of separate colorectal cancer patients and label values for whether or not lymph node metastasis of the separate colorectal cancer patients.
5. According to claim 4,

   The deep learning model is

   The feature extraction layer, an attention layer that outputs features of the entire sample image by assigning weights to each of the patch images, and classification that outputs a prediction value for whether or not the lymph nodes are located based on features output by the attention layer. contains layers,

   The attention layer and the classification layer are learned again using separate learning data,

   The learning data is a method for predicting lymph node metastasis using endoscopic resection specimen images of early colorectal cancer, including stained specimen images of separate colorectal cancer patients and label values for whether or not lymph node metastasis of the separate colorectal cancer patients.
6. an input device that receives a stained sample image of a colon of a colon cancer patient;

   a storage device for storing a deep learning model; and

   Including an arithmetic device for predicting whether or not metastasis to a lymph node for the colorectal cancer patient based on a value output by the deep learning model receiving the sample image,

   The deep learning model receives patch images in which the entire sample image is divided into a plurality of regions, and determines whether the patient has metastasis to the lymph nodes based on the characteristics of the entire sample image to which weights are assigned to the patch images. An analysis device that predicts lymph node metastasis of early colorectal cancer that outputs a predictive value for
7. According to claim 6,

   The sample image is an analysis device for predicting lymph node metastasis of early colorectal cancer, which is an H&E (Haematoxylin and eosin) staining image for a sample detected by endoscopic resection.
8. According to claim 6,

   The deep learning model is

   a feature extraction layer extracting features for each of the patch images;

   For each of the patch images, the feature output by the feature extraction layer is weighted by the degree of contribution of each patch image to the prediction of lymph node metastasis, and the features in which the degree of contribution is weighted for each of the patch images are averaged to obtain the specimen image an attention layer that calculates features for the whole; and

   An analysis device for predicting lymph node metastasis of early colorectal cancer, including a classification layer receiving characteristics of the entire sample image output from the attention layer and outputting the prediction value.
9. According to claim 6,

   The deep learning model includes a feature extraction layer for extracting features for each of the patch images,

   The feature extraction layer is calculated by removing a classification layer from a separate convolutional neural network (CNN) model learned using separate training data,

   The learning data is an analysis device for predicting lymph node metastasis of early colorectal cancer, including a stained sample image for separate colorectal cancer patients and a label value for whether or not the separate colorectal cancer patients have metastasized lymph nodes.
10. According to claim 9,

    The deep learning model is

    The feature extraction layer, an attention layer that outputs features of the entire sample image by assigning weights to each of the patch images, and a classification that outputs a prediction value for whether or not metastases to the lymph nodes based on features output by the attention layer. contains layers,

    The attention layer and the classification layer are learned again using separate learning data,

    The learning data is an analysis device for predicting lymph node metastasis of early colorectal cancer, including a stained sample image for separate colorectal cancer patients and a label value for whether or not the separate colorectal cancer patients have metastasized lymph nodes.

Images