CN116936091A - Hepatocellular carcinoma microvascular invasion prediction method and model - Google Patents

Abstract

The invention discloses a hepatic cell cancer microvascular invasion prediction model, which comprises the following steps: a feature extraction module, a modal attention module, and a two-channel example feature aggregation module. The invention also discloses a method for predicting the invasion of the hepatic cell cancer microvessels. Compared with the previous two-dimensional-based deep learning model, the two-dimensional-based deep learning model based on modal attention and double channels has the advantages that all two-dimensional slices of the three-dimensional tumor are input, information is not lost, and therefore the accuracy is higher. Compared with a three-dimensional model, the method has the advantages that the two-dimensional feature extractor is used for extracting the features of all slices, and meanwhile, the double-channel multi-example aggregator can be used for obtaining better feature representation, so that the accuracy is higher, the required calculation resources are less, and the training is easier.

Description

Hepatocellular carcinoma microvascular invasion prediction method and model

Technical Field

The invention relates to the technical fields of image medicine and image histology, in particular to a method and a model for predicting hepatic cell cancer microvascular invasion.

Background

Hepatocellular carcinoma (HCC) is one of the most common malignant tumors of the liver in the world, and has the characteristics of high morbidity and mortality. Means for treating HCC include liver resection, arterial chemoembolization, radiofrequency ablation, systemic treatment, and liver transplantation. However, the recurrence rate after HCC surgery is high, with three and five year recurrence rates of about 50% and 70%, respectively. Postoperative recurrence is the leading cause of death. Thus, identifying risk factors for HCC recurrence may provide a more aggressive treatment strategy for the clinician.

Studies have shown that microvascular invasion (MVI) is a key prognostic factor for postoperative recurrence in HCC patients. After surgery or liver transplantation, MVI patients had earlier relapse times, lower overall five-year survival, and poorer prognosis than MVI-free patients. Currently, MVI can only be identified by post-operative histopathological examination. However, for HCC patients, if their MVI status can be accurately predicted preoperatively, by developing more rational treatment strategies (e.g., enlarging tumor resection margin for high-risk MVI patients), their postoperative recurrence rate will be reduced. Therefore, prediction of MVI by preoperative medical imaging is highly desirable.

Many studies in the prior art predict MVI through image histology and have achieved better prediction results. Because the method based on image histology relies on expert knowledge, the characteristics need to be manually extracted, time and effort are wasted, and the prediction result is unstable. Therefore, the end-to-end prediction method by deep learning is becoming the mainstream. The prediction method of the hepatic cell cancer microvascular invasion based on deep learning is divided into two types: one is based on a two-dimensional deep learning method, which inputs a slice of a three-dimensional tumor into a model to obtain a prediction result. The other is based on a three-dimensional deep learning method, which inputs the whole three-dimensional tumor into a model to obtain a prediction result.

The model in the two-dimensional deep learning method in the prior art only takes one slice of the tumor, and ignores the context information of the focus. The model based on the three-dimensional deep learning method inputs the whole tumor, so that the calculation resource consumption is high, the training is difficult, and meanwhile, the fitting is easy due to the small data volume.

Disclosure of Invention

The invention aims to provide a prediction method and a model for hepatic cell cancer microvascular invasion, which can solve the defect of low accuracy in the prior art.

The invention adopts the following technical scheme:

a hepatocellular carcinoma microvascular invasion prediction model comprising: a feature extraction module, a modal attention module, and a two-channel example feature aggregation module.

The feature extraction module consists of five convolution layers and a full connection layer, wherein each convolution layer is respectively followed by a Relu activation function, a Batch normalization layer and a pooling layer, the Batch normalization layer is used for accelerating model convergence, the Relu layer is used for increasing the nonlinearity degree of the model, and the pooling layer is used for reducing the size of a feature map.

The modal attention module converts the channel in senet into an MRI sequence number, which comprises three operations, namely compression, excitation and re-weighting, wherein the compression operation compresses all two-dimensional slices in the phase three-dimensional tumor MRI into one modal vector, the excitation operation maps the vector into a group of weights, and the re-weighting operation multiplies the weights with the original five modal vectors for re-weighting.

The two-channel multi-example feature aggregation module is provided with two channels, the upper-layer channel obtains a maximum score value through multi-example maximization and finds a key example, the lower-layer channel calculates similarity between other examples and the key example through calculating cosine similarity between the other examples and the key example, the more similar examples are given larger weight, then the examples are added through the weight to obtain the feature of the package, finally the feature of the package is obtained through a classifier to obtain the prediction score, and finally the prediction score of the package is added with the prediction score of the key example to obtain the total prediction score.

A method for predicting microvascular invasion of hepatocellular carcinoma, comprising the steps of:

s1: collecting a hepatocellular carcinoma MRI dataset, each MRI data comprising five sequences of arterial phase, portal phase, delay phase, diffuse sequence and T2 weighting;

s2: drawing out the region of interest of the tumor in each arterial MRI slice by using a rectangular frame at a position 5-10mm away from the edge of the tumor;

s3: registering the MRI of the arterial phase with the portal phase, the delay phase, the diffusion sequence and the T2 weighted MRI, and then cutting the three-dimensional tumor of each phase;

s4: the dataset was read as per 4:1 is divided into a training set and a verification set, and five-fold cross verification is carried out;

s5: consider a single sequence 3D MRI lesion as one multi-example learning package, and consider all slices within the package as examples;

s6, before five sequences of packets are input into a model, uniformly scaling the slices to S.times.64.times.64, wherein S represents the number of the slices, and 64.times.64 represents the length and width of the slices;

s7, firstly, obtaining S-128-dimensional features of the MRI slices of each mode through a shared feature extractor;

s8: after the characteristics of five MRI modes are spliced, the characteristics of example dimensions are changed into 5 x 128 dimensions, and then the characteristics are input into a mode attention module, and a module with larger prediction assistance is given more important weight through the mode attention module;

s9, inputting the example features output from the modal attention module into a dual-channel example aggregator;

s10, in the first channel of the aggregator, finding a key example through multi-example maximization pooling, and obtaining a prediction score c of the key example _m (B) The formula is as follows:

c _m (B)＝max{W ₀ h ₀ ,…,W ₀ h _n-1 }；

s11: in the second channel of the aggregator, the similarity coefficient between the other examples and the key example is obtained by calculating the cosine similarity between the key example and the other examples, and the formula is as follows:

s12: weighting all examples according to the similarity coefficient, obtaining the characteristics of a packet:

s13, obtaining a prediction score based on the characteristics of the packet through a full connection layer and an activation function, wherein the formula is as follows:

c _b (B)＝W _b b；

s14: the example-based prediction and the packet-based prediction score are added to obtain a final prediction score, the formula is as follows:

the invention has the advantages that: compared with the previous two-dimensional-based deep learning model, the two-dimensional-based deep learning model based on modal attention and double channels has the advantages that all two-dimensional slices of the three-dimensional tumor are input, information is not lost, and therefore accuracy is higher. Compared with a three-dimensional model, the two-dimensional feature extractor is used for extracting the features of all slices, and meanwhile, the two-channel multi-example aggregator can be used for obtaining better feature representation, so that the accuracy is higher, the required computing resources are less and the training is easier.

Drawings

The invention is described in detail below with reference to examples and figures, wherein:

fig. 1: arterial phase, portal phase, delayed phase, diffuse sequence and T2 weighted five sequences of hepatocyte MRI images; wherein, (A) arterial phase, (B) delayed phase, (C) portal phase, (D) T2 weighting, (E) hepatocyte MRI image of diffuse sequence;

fig. 2: a processing flow of the MRI image of the hepatocellular carcinoma;

fig. 3: the architecture diagram of the feature extractor of the present invention;

fig. 4: the invention discloses a feature extractor structure of a two-dimensional slice in the implementation;

fig. 5: the present invention is implemented in a two-channel multi-instance aggregator architecture.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The terms "first," "second," "third," "fourth" and the like in the description and in the claims and in the above drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein.

As used herein, the word "if" may be interpreted as "at … …", depending on the context

Or when … … or in response to a determination or in response to a detection.

When a doctor diagnoses a disease by means of medical images, he examines all two-dimensional slices of a medical image (e.g. MRI) and decides that the patient is problematic when at least one of them is found to be problematic. When all slices are free of problems, then it is determined that there are no problems. The invention constructs a method for predicting hepatic cell cancer microvascular invasion based on modal attention and a two-channel multi-example deep learning model.

The invention discloses a hepatic cell cancer microvascular invasion prediction model, which comprises the following steps: a feature extraction module, a modal attention module, and a two-channel example feature aggregation module.

1. Feature extraction module

The feature extraction module structure consists of five convolution layers and a full connection layer, wherein each convolution layer is respectively followed by a Relu activation function, batch normalization layers and a pooling layer, the Batch normalization layers are used for accelerating model convergence, the Relu layers are used for increasing the nonlinearity degree of the model, and the pooling layer is used for reducing the size of a feature map. As shown in fig. 3, the convolution kernel size of each convolution layer in the feature extraction module is 3×3, and the maximum pool is 2×2. The number of channels in each convolutional layer is 16, 32, 64, 128, 256. The number of neurons in the fully connected layer is 128. As shown in fig. 4, (s, 1, 64, 64) represent the number of slices, the channel, the length and the width of the slice, respectively, in MRI. In the feature extractor, the input is a1×64×64 MRI slice, and after five convolution layers, the feature dimension is 256×2×2. Upon planarization, the feature map from the convolutional layer has dimensions 4096, which is reduced to 128 by the fully connected layer.

2. Modal attention module

The modal attention module is modified from senet, and the model focuses on the MRI of one or several modalities by changing channels in senet to MRI sequence numbers. The modal attention module includes three operations, compression, excitation and re-weighting, respectively. The compression operation compresses all two-dimensional slices within the phase-three-dimensional tumor MRI into one modal vector. The excitation operation maps the vector into a set of weights. The re-weighting operation re-weights the weights by multiplying the original five modal vectors.

Here, let m= { M1, M2, M3, M4, M5} be a characteristic of five MRI modalities.(k=1, 2,3,4, 5) represents a mode characteristic tensor of the kth mode, where s represents the number of slices and 128 represents a characteristic tensor of each slice. The global average pool is used to compress the modality descriptor as follows:

where mk (i, j) represents the j-th element in the i-th slice of mk. The excitation operator maps descriptors to a set of modal weights. SENet was simulated by two fully connected layers and the relu structure. The description is as follows:

a＝W ₂ relu(W ₁ d)

where d= { d1, d2, d3, d4, d5} is the modality descriptor vector, W1 and W2 are weight vectors of two fully connected layers, and a= { a1, a2, a3, a4, a5} is the attention vector.

The re-weighting operator ensures that a particular MRI modality is emphasized by multiplying the attention vector a= { a1, a2, a3} with the original multi-modality feature m= { M1, M2, M3, M4, M5 }. The description is as follows:

wherein the method comprises the steps ofCharacterization of five re-weighted MRI modalities

3. Dual-channel multi-example feature aggregation module

The dual channel multi-example feature aggregation module has two channels. The upper layer channel is pooled to a maximum score value through multiple instance maximization and finds the key instance. The lower layer channel calculates the similarity between other examples and the key examples by calculating the cosine similarity between the other examples and the key examples, the more similar examples are given a larger weight, the characteristics of the packets are obtained by adding the examples through the weight, and finally the characteristics of the packets are obtained by a classifier. And finally, adding the prediction score of the packet and the prediction score of the key example to obtain the total prediction score.

The dual-channel multi-example feature aggregation module is used for operating the pathological image patch. Let b= { x1, …, xn } denote a set of MRI slices of a particular modality, where x _i Example i is shown. X is x _i After passing through the feature extraction module and the modal attention module, it can be projected to the embedded h _i ∈R ^128×1 After the feature extraction module and the modality attention module. The first channel of the module uses an instance classifier on each instance embedding and performs multi-instance learning max pooling on the scores to obtain the key instance h _m And highest score c _m :

c _m (B)＝max{W ₀ h ₀ ,…,W ₀ h _n-1 }；

Wherein W is ₀ Is a weight vector and represents a classifier. W (W) ₀ h _i Representing h obtained by classifier _i Is a predictive score of (a). Then max { W ₀ h ₀ ,…,W ₀ h _n-1 Represented at W ₀ h _i The maximum score is obtained. The second branch aggregates all example features into one packet feature, after which the prediction score of the packet is obtained by the packet classifier. Specifically, each example feature h _i (including h _m ) Is converted into two vectors: q _i ∈R ^640×1 And v _i ∈R ^640×1 The transformation procedure is as follows: q _i ＝W _q h _i ,v _i ＝W _v h _i ,i＝0,...,n-1；

Wherein W is _q And W is _v Represents a weight matrix, q _i And v _i Is h _i Is a linear representation of the other.

The feature extractor is responsible for extracting example features, while the modal attention module simulates the physician's decision process and helps the model focus on the most important MRI sequences. The dual stream instance feature aggregation module aggregates the features of all instances into bag features enabling the model to focus on critical MRI slices. Each MRI modality is first processed by a feature-sharing extractor that produces feature tensors of dimensions sx 128. After connecting the feature tensors of the five MRI modalities, the dimension of the example feature tensor becomes 5×128. The tensor is then input to a modality attention module which assigns a higher weight to the MRI modality with the highest predicted effect. The features of the examples are then aggregated to form bag features, which are used to produce a final prediction result.

The invention also discloses a method for predicting the invasion of the hepatic cell cancer microvessels, which comprises the following steps:

s1: a set of MRI data of hepatocellular carcinoma is collected, each MRI data comprising five sequences of arterial phase, portal phase, delay phase, diffuse sequence and T2 weighting.

S2: an imaging physician who uses 5 years old draws a rectangular frame to draw out the interested area of the tumor in each arterial MRI slice at a position 5-10mm away from the edge of the tumor, and another imaging physician who uses 20 years old confirms the interested area;

s3: registering the MRI of the arterial phase with the portal phase, the delay phase, the diffusion sequence and the T2 weighted MRI, and then cutting the three-dimensional tumor of each phase;

s4: the dataset was read as per 4:1 is divided into a training set and a verification set, and five-fold cross verification is carried out;

s5: consider a single sequence 3D MRI lesion as one multi-example learning package, and consider all slices within the package as examples;

s6, before five sequences of packets are input into a model, uniformly scaling the slices to S.times.64.times.64, wherein S represents the number of the slices, and 64.times.64 represents the length and width of the slices;

s7, firstly, obtaining S-128-dimensional features of the MRI slices of each mode through a shared feature extractor;

s8: after stitching the features of the five MRI modalities, the features of the example dimensions become 5 x 128 dimensions, before entering the modality attention module. By the modality attention module, modules that help the prediction are more heavily weighted;

s9, inputting the example features output from the modal attention module into a dual-channel example aggregator; s10, in the first channel of the two-channel example aggregator, key examples are found through multi-example maximization pooling, and a prediction score c of the key examples is obtained _m (B) The formula is as follows:

c _m (B)＝max{W ₀ h ₀ ,…,W ₀ h _n-1 }；

wherein c _m (B) Predictive score, h, representing key examples ₀ Representing example features, W ₀ Representing the matrix coefficients of the example classifier.

S11: in the second channel of the aggregator, the similarity coefficient between the other examples and the key example is obtained by calculating the cosine similarity between the key example and the other examples, and the formula is as follows:

U(h _i ,h _m ) The coefficient of similarity is represented by a coefficient of similarity,<q _i ,q _m >representation example q _i And key example q _m And also the cosine similarity between them, U (h) is determined by a softmax function _i ,h _m ) Normalize, let all U (h _i ,h _m ) And 1.

S12: weighting all examples according to the similarity coefficient, obtaining the characteristics of a packet:

b represents the characteristics of the bag, v _i Conversion vectors representing examples, all examples v are calculated according to similarity coefficients to key examples _i Weighting is carried out, and the feature b of the package is obtained.

S13, obtaining a prediction score based on the characteristics of the packet through a full connection layer and an activation function, wherein the formula is as follows:

c _b (B)＝W _b b；

c _b (B) Representing a packet-based predictive score, W _b Representing the matrix coefficients of the example classifier.

S14: the example-based prediction and the packet-based prediction score are added to obtain the final prediction score c (B), as follows:

wherein c _m (B) Representing a predictive score based on a key example, c _b (B) Represents a packet-based prediction score, and c (B) represents a final prediction score.

In this implementation, the model is implemented by a Pytorch open source deep learning framework. All experiments were performed on a dell T640 workstation with two independent NVIDIA GeForce RTX 2080Ti graphic cards and two Intel Xeon Silver4110cpu. In the experiment, a cross entropy loss function and adam optimizer were employed. In the present invention, s denotes the number of slices in the MRI volume, and 64 denotes the width and height of the slices.

Example 1

The experiment of the method verifies the model through indexes such as Accuracy, sensitivity, specificity, negative predictive value NPV, positive predictive value PPV, area under curve AUC and the like.

1) First, in order to verify the effectiveness of each module of the method, ablation experiments were performed on the modules, respectively. Assuming that the feature extracted by the feature extractor is directly classified and predicted to be used as a benmark, then a modal attention module and a dual-channel multi-example aggregation module are respectively added, and the prediction performance of the model is verified. The experimental results are shown in table 1:

TABLE 1

As shown in the figure, when only bechmark is used, the model performance is the worst, the accuracy is 67.14%, the sensitivity is 73.89%, the specificity is 60.3%, the NPV is 67.38%, the PPV is 67.77, the AUC is 0.6710, and when the modal attention module and the dual-channel multi-example aggregation module are added, the prediction performance is improved. When all modules were added, the model performed best, with an accuracy of 76.43%, sensitivity of 71.41%, specificity of 80.36%, NPV of 74.16%, PPV of 81.74%, and 7AUC of 0.7422.

2) Because the influence of the feature extraction on the model effect is large, the model is easy to be overfitted when the number of layers of the feature extractor is deep, and the feature extraction is possibly insufficient when the number of layers of the feature extractor is shallow, and in order to determine the number of convolution layers of the feature extractor, experiments are carried out on the feature extractor with the number of convolution layers of 4,5,6,7 and 8 respectively. The results are shown in the following table:

TABLE 2

As can be seen from the above table, the model is best predicted when the number of convolution layers of the feature extractor is 5, and is poor when the number of convolution layers is less than 5 or more than 5.

3) Physicians often focus on certain important MRI modalities in diagnosing disease, and therefore have more accurate diagnostic results based on these modalities. To verify the effectiveness of the model, we tested the predictive performance of the model in a single MRI modality, the results were as follows:

TABLE 3 Table 3

As can be seen from the above table, the overall predictive performance of the model is best when only Arterial phase (Arterial) images are input, where the accuracy is 72.50%, sensitivity is 76.73%, specificity is 68.81%, NPV is 76.19%, PPV is 72.30%, and AUC is 0.69.74. When only the gate period (venosus) and the diffusion period (ADC) are input, the predicted performance is inferior. The effect is the worst when only Delay and T2 weighted phase images are input. These results are consistent with the clinical diagnostic experience of the physician, which also demonstrates the effectiveness of the model.

4) Since the data enhancement policy also has a large impact on model predictive performance, we verify the impact of both the data enhancement policy and the data enhancement policy on the model. For the data enhancement strategy we performed a random 90 ° vertical flip and rotation of the MRI slices within each packet. The results are shown in the following table:

TABLE 4 Table 4

From the table, when there is no random enhancement, the accuracy is 70.71%, the sensitivity is 70.26%, the specificity is 70.22%, the NPV is 69.75%, the PPV is 72.30%, and the AUC is 0.6804. When data enhancement is performed, the predictive performance of the model is greatly improved compared to when data enhancement is not performed. The above structure can be seen that the enhancement combination of each slice in the packet greatly improves the data increase of the packet, and thus the prediction performance of the model is improved more than the conventional data enhancement.

The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, and alternatives falling within the spirit and principles of the invention.

Claims (5)

1. A model for predicting microvascular invasion of hepatocellular carcinoma, comprising: a feature extraction module, a modal attention module, and a two-channel example feature aggregation module.

2. The model of claim 1, wherein the feature extraction module consists of five convolution layers, one full connection layer, each convolution layer being followed by a Relu activation function, batch normalization layers and a pooling layer, respectively, wherein Batch normalization layers are used to accelerate model convergence, relu layers are used to increase the degree of model nonlinearity, and pooling layers are used to reduce the size of the feature map.

3. The model of claim 1, wherein the modal attention module converts the channel in senet to MRI sequence number, which includes three operations, compression, excitation and weighting, respectively, the compression operation compresses all two-dimensional slices within the phase three-dimensional tumor MRI into one modal vector, the excitation operation maps the vector into a set of weights, and the weighting operation re-weights the weights by multiplying the weights with the five modal vectors.

4. The model of claim 1, wherein the two-channel multi-instance feature aggregation module has two channels, the upper channel is maximized and pooled to obtain a maximum score value through multiple instances and find a key instance, the lower channel is used for calculating cosine similarity between other instances and the key instance, the similarity between the other instances and the key instance is calculated, the more similar instances are given a larger weight, the instances are added to obtain the feature of the package through the weight, the feature of the package is finally obtained through a classifier, and finally the prediction score of the package is added to the prediction score of the key instance to obtain the total prediction score.

5. A method for predicting microvascular invasion of hepatocellular carcinoma, comprising the steps of:

s1: collecting a hepatocellular carcinoma MRI dataset, each MRI data comprising five sequences of arterial phase, portal phase, delay phase, diffuse sequence and T2 weighting;

s2: drawing out the region of interest of the tumor in each arterial MRI slice by using a rectangular frame at a position 5-10mm away from the edge of the tumor;

s3: registering the MRI of the arterial phase with the portal phase, the delay phase, the diffusion sequence and the T2 weighted MRI, and then cutting the three-dimensional tumor of each phase;

s4: the dataset was read as per 4:1 is divided into a training set and a verification set, and five-fold cross verification is carried out;

s5: consider a single sequence 3D MRI lesion as one multi-example learning package, and consider all slices within the package as examples;

s6, before five sequences of packets are input into a model, uniformly scaling the slices to S.times.64.times.64, wherein S represents the number of the slices, and 64.times.64 represents the length and width of the slices;

s7, firstly, obtaining S-128-dimensional features of the MRI slices of each mode through a shared feature extractor;

s8: after the characteristics of five MRI modes are spliced, the characteristics of example dimensions are changed into 5 x 128 dimensions, and then the characteristics are input into a mode attention module, and a module with larger prediction assistance is given more important weight through the mode attention module;

s9, inputting the example features output from the modal attention module into a dual-channel example aggregator;

s10, in the first channel of the aggregator, finding a key example through multi-example maximization pooling, and obtaining a prediction score c of the key example _m (B) The formula is as follows:

c _m (B)＝max{W ₀ h ₀ ,…,W ₀ h _n-1 }；

s11: in the second channel of the aggregator, the similarity coefficient between the other examples and the key example is obtained by calculating the cosine similarity between the key example and the other examples, and the formula is as follows:

s12: weighting all examples according to the similarity coefficient, obtaining the characteristics of a packet:

s13, obtaining a prediction score based on the characteristics of the packet through a full connection layer and an activation function, wherein the formula is as follows:

c _b (B)＝W _b b；

s14: the example-based prediction and the packet-based prediction score are added to obtain a final prediction score, the formula is as follows: