JP7411126B2 - Multitemporal CT image classification system and construction method based on spatiotemporal attention model - Google Patents

Description

The present invention relates to the field of medical image processing technology, and particularly to a multi-temporal CT image classification system and construction method based on a spatio-temporal attention model.

CT (Computed Tomography) is a type of computer tomography that uses precisely collimated X-ray beams, gamma rays, ultrasound waves, etc., to take cross-sectional images of certain parts of the human body one after another using extremely sensitive detectors. CT image scanning has the characteristics of short scanning time and clear images, and as treatment methods have improved, CT image scanning has become popular for diagnosing various tumors (for example, liver cancer). The site, size and range of the tumor can be quickly found, the presence or absence of changes such as necrosis or bleeding in the lesion can be directly observed, and the presence or absence of tumor metastasis can be detected, increasing the tumor detection rate. There is.

Although plain CT scans can quickly locate lesions and even detect some diseases, some lesions, such as vascular malformations, early cancers, and metastatic tumors, cannot be diagnosed with plain CT scans. Contrast-enhanced CT (CECT) is required to enhance the visualization of the lesion and determine its extent and clinical stage. For example, in the case of brain CT examination, the accuracy rate of simple CT diagnosis is 82%, and the accuracy rate of contrast-enhanced CT scan has increased to 92-95%, which shows that contrast-enhanced CT is useful for improving the diagnostic rate. In contrast-enhanced CT scans, a contrast agent is generally injected intravenously, and there are currently two common intravenous injection methods: one is manual injection, and the other is injection using a high-pressure syringe. When a contrast agent is injected, contrast-enhanced CT can provide more information than plain CT and can observe blood flow in the arterial phase, portal venous phase, and delayed phase, which is very useful for diagnosis. Treatment methods for different subtypes of tumors are different, and currently, multi-phase contrast-enhanced CT is an important tool for preoperative diagnosis of tumor subtypes.

The application of deep learning to medical image processing is also a major direction, and by introducing it to machine learning, we can get closer to the original goal of artificial intelligence, and understand the inherent laws and expression level of sample data. The information gained through these learning processes is highly useful in interpreting data such as text, images, and audio. The ultimate goal is for machines to have the ability to analyze and learn like humans, and to be able to recognize data such as text, images, and audio. Deep learning is a complex machine learning algorithm, and the effects it has achieved in voice and image recognition far exceed those of related technologies to date. They have all achieved much success in multimedia learning, audio, recommendation, personalization technology, and other related fields. Deep learning enables machines to imitate human activities such as audiovisual and thinking, solving many complex pattern recognition challenges, and is bringing major advances in related technologies of artificial intelligence. With the development of deep learning, the updating of convolutional neural networks is repeated, and more and more applications are realized in the field of image recognition, which does not require too much human intervention, can automatically extract image features, and improves its learning ability. Its advantages include high competitive performance, especially in medical image analysis tasks such as cancer classification and lesion detection.

However, the determination and diagnosis of malignant tumors remains difficult, preoperative misdiagnosis can lead to poor treatment decisions, and the increasing complexity of tumor imaging reporting and data systems makes implementation in large-scale practices difficult. As the difficulty increases, the clinical need for computational decision support tools has expanded to improve work efficiency, and traditional convolutional neural networks have certain advantages in local feature extraction of CT images, and Although it is possible to quickly check the situation, since images of multiple phases of contrast-enhanced CT cannot be used, the temporal information linkage is weakened, the information is not fully utilized, and the final diagnosis result is affect.

Chinese patent application CN110443268A discloses a method for classifying liver cancer CT images into benign and malignant based on deep learning.The method is designed and improved based on the existing Resnet34 network model, and uses patient liver information. Select the largest slice of the data and put it into the model for classification through data processing and enhancement. However, since CT images are 3D, the spatial features extracted by this method are incomplete, and the situation of multi-temporal CT images is not taken into account, so lesions in multiple temporal phases of a patient can be treated effectively. The accuracy and precision of diagnostic results are reduced.

Therefore, in order to solve the above-mentioned problem, there is a need for a method that can process multi-temporal CT in combination to improve the accuracy and speed of classification. Based on the current state of development of existing medical image processing methods and deep learning, it is conceivable to use an encoder configured with an attention mechanism and a transformer. A transformer is a model originally proposed in the natural language processing (NLP) field in 2017, and was first used in the visual field in 2020. It is similar to NLP and is used to serialize images. can successfully perform the image classification task, the final classification result is comparable to the best convolutional neural network, and the required computational resources are significantly reduced, improving the classification efficiency and accuracy rate. It's increasing.

The present invention proposes a multi-temporal CT image classification system and construction method based on a spatiotemporal attention model, taking into account that there is no major change in the lesion structure of a patient during normal CT scans and contrast-enhanced CT scans, Based on the conventional convolutional neural network, the problem that multi-temporal CT images cannot be combined and processed is solved.

In the present invention, a professional medical imaging doctor first labels a multi-phase CT image, then pre-processes the image to segment the lesion and adjust the image size to fit the model input. The input is a normal simple scan CT image and a multi-phase contrast enhanced CT image after contrast injection, and the output is a normal simple scan CT image and a multi-phase contrast CT image after contrast injection. It is an embedding vector of a multi-phase contrast-enhanced CT image, and a spatial attention network is constructed, and the input of the network model is the embedding vector of the above CT image. The spatial features of phase CT images can be output respectively, and the above spatial features can be combined to construct a temporal attention network, and the input of the network model is the combined spatial features, and the temporal attention network is constructed by combining the above spatial features. It can output a vector that combines features and spatial features, and then outputs the final classification result through a classification layer, and finally calculates the label to obtain the loss, and repeats training and optimization to calculate the loss. is minimized, an optimal classification model is obtained, and a multi-temporal CT image classification system based on the spatiotemporal attention model is obtained.

The technical solution adopted in the present invention is specifically as follows.
A multi-temporal CT image classification system based on a spatio-temporal attention model, comprising:
a data acquisition unit for acquiring CT images of s time phases of a patient to be classified;
A first embedded layer network unit including s first embedded layer networks, each of which divides a CT image of each time phase into a plurality of image blocks, and divides each image block into an image block. a first embedding layer network unit that expands into block vectors, combines all image block vectors and class label vectors, and then adds them to a position vector of the same dimension to obtain an embedding vector of a CT image of a corresponding time phase;
a spatial attention unit comprising s spatial attention networks, each spatial attention network comprising a first multi-head attention network MSA in the L1 layer, a first multi-layer perceptron in the L1 layer, and a first multi-head perceptron in the L1 layer; a normalization layer, a first multi-head attention network MSA of the L1 layer and a first multi-layer perceptron of the L1 layer are interleaved in turn, and the first multi-head attention network MSA includes a plurality of self-attention modules SA; The self-attention module SA transforms the normalized input vector into three different matrices: a query matrix Q _1i , a keyword matrix K _1i and a value matrix V _1i , a query matrix Q _1i , a keyword matrix It is used to generate the attention function between each vector in the input vectors based on three different matrices of K _1i and value matrix V _1i , where i=1, 2... is the i-th one in the spatial attention unit. Representing a self-attention module SA, the splicing layer is used to splice the output attention function of each self-attention module SA to obtain the final spatial attention function, and the splicing layer is used to splice the output attention function of each self-attention module SA to obtain the final spatial attention function, and the splicing layer is used to splice the output attention function of each self-attention module SA to obtain the final spatial attention function. Let what is added be the input vector corresponding to the first multilayer perceptron of the next layer (the network can compare the connections between different vectors with each other by the multi-head attention module and strengthen the important parts), and The first multi-layer perceptron encodes the normalized input vector and adds it to the input vector, and uses the addition result as an input corresponding to the first multi-head attention network MSA of the next layer. The input vector of the attention network MSA is an embedded vector, and the first normalization layer normalizes the first dimension vector of the vector obtained by adding the vector output from the first multilayer perceptron in the final layer and its input vector. , a spatial attention unit that is a spatial feature of a CT image of a corresponding time phase;
A second embedding layer network unit including one second embedding layer network, wherein the class label vector is obtained after combining the spatial features of s corresponding temporal CT images output from the s spatial attention networks. a second embedding layer network unit for obtaining an embedding layer vector together with
A temporal attention unit including one temporal attention network, wherein the temporal attention network includes a second multi-head attention network MSA in the L2 layer, a second multi-layer perceptron in the L2 layer, and a second normalization in the first layer. a second multi-head attention network MSA of the L2 layer and a second multi-layer perceptron of the L2 layer are interleaved in sequence, and the second multi-head attention network MSA includes a plurality of self-attention modules SA and a splicing The self-attention module SA transforms the normalized input vector into three different matrices: a query matrix Q _2j , a keyword matrix K _2j and a value matrix V _2j , a query matrix Q _2j , a keyword matrix K _2j The splicing layer is used to generate the attention function between each vector in the input vectors based on three different matrices of and value matrix _V2j , and the splicing layer splices the attention function output from each self-attention module SA to form the final is used to obtain the final temporal attention function, where j = 1, 2... represents the j-th self-attention module SA in the temporal attention unit, and add the final temporal attention function and the input vector. is an input vector corresponding to the second multilayer perceptron of the next layer, and the second multilayer perceptron encodes the normalized input vector and adds it to the input vector, and adds the addition result to the second multilayer perceptron of the next layer. The input vector of the second multi-head attention network MSA in the first layer is the embedding layer vector output from the second embedding layer network unit, and the second normalization layer is the input vector of the second multi-head attention network MSA in the first layer. a temporal attention unit that normalizes the first dimension vector of the vector obtained by adding the output vector of the second multilayer perceptron and its input vector to obtain a vector having spatial characteristics and temporal characteristics;
A classification layer unit including a classification layer for obtaining a classification result based on a vector having spatial and temporal features.

Furthermore, s is 2 or more, and the CT images of s time phases are specifically a simple scan phase CT image, an arterial phase CT image, a portal venous phase CT image, and a delayed phase CT image. Contains at least two.

Furthermore, the embedding vector specifically includes:
X ₀ = [X _class ; X ¹ _p ; X ² _p ...X ^N _p ]+X _pos ,
However, X _class represents a class label vector, X _pos represents a position vector, X _p represents an image block vector after linearization, and N represents the number of image blocks after division.

Furthermore, generating the attention function between each vector in the input vectors based on three different matrices, query matrix Q _1i , keyword matrix K _1i and value matrix V _1i specifically,
and
However, _dk represents the dimension of each keyword vector k in the keyword matrix _K1i , and softmax() is a softmax function.

Similarly, generating the attention function between each vector in the input vectors based on three different matrices: query matrix Q _2j , keyword matrix K _2j and value matrix V _2j specifically,
and
However, _dk represents the dimension of each keyword vector k in the keyword matrix _K2j , and softmax() is a softmax function.

Furthermore, the input vectors of the first multi-head attention network MSA and the second multi-head attention network MSA are:
and
LN represents the normalization method, x _l represents the input vector of the first multi-head attention network MSA or the second multi-head attention network MSA, and MLP() represents the output of the corresponding first multi-layer perceptron or second multi-layer perceptron. where x' _l-1 represents the input vector of the first multilayer perceptron or the second multilayer perceptron of the l-1th layer.

Furthermore, the input vectors of the first multilayer perceptron and the second multilayer perceptron are as follows:
and
LN represents the normalization method, _x'l represents the input vector of the first multi-layer perceptron or the second multi-layer perceptron, and MSA() represents the output of the corresponding first multi-head attention network MSA or second multi-head attention network MSA. , and x _l represents the input vector of the first multi-head attention network MSA or the second multi-head attention network MSA of the l-th layer.

A method for constructing a multi-temporal CT image classification system based on a spatio-temporal attention model, the method comprising:
Collecting samples to construct a dataset, each sample of the dataset including s temporal CT images of one patient;
The goal is to construct a multi-temporal CT image classification system based on the above spatio-temporal attention model, and to minimize the error between the classification result output by the system and the classification label, using each sample in the dataset as input to the system. and obtaining a multi-temporal CT image classification system based on the spatio-temporal attention model.

The beneficial effects of the present invention are as follows.
(1) The present invention proposes a multi-temporal CT image classification system based on a spatio-temporal attention model, which includes two types of attention networks: a spatial attention network and a temporal attention network. The spatial attention network can extract the spatial features of CT images, and the temporal attention network can extract the connections between CT images of different temporal phases, and the global attention between CT images of each temporal phase can be extracted. Strengthen.

(2) The present invention has universality to various diseases that need to be diagnosed based on multi-phase CT images, and more effectively utilizes the characteristics of lesions in different temporal phases to By strengthening connections and abandoning the traditional convolutional neural network-based design, the attention mechanism allows you to put more computations into important areas and get more details about the target you want to focus on. , This suppresses other unnecessary information, reduces calculation redundancy and delay, makes it easier to diagnose CT images in a shorter time, increases diagnosis accuracy, and makes the diagnosis effect more stable. do.

FIG. 2 is a structural diagram of a multi-temporal CT image classification system based on the spatio-temporal attention model of the present invention. 2 is a flowchart of classification performed by the multi-temporal CT image classification system based on the spatio-temporal attention model of the present invention. 1 is a flowchart of a method for constructing a multi-temporal CT image classification system for liver cancer based on the spatio-temporal attention model of the present invention.

Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. Where the following description refers to drawings, the same numbers in different drawings indicate the same or similar elements, unless stated otherwise. The embodiments described in the illustrative examples below do not represent all embodiments consistent with the present invention. On the contrary, they are merely examples of apparatus and methods consistent with certain aspects of the invention, as detailed in the appended claims.

The terminology used in the present invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.

As used in this invention and the appended claims, the singular forms "a", "a", and "the" are intended to include the plural as well, unless the context clearly dictates otherwise. . It should also be understood that the term "and/or" as used herein refers to and encompasses any or all possible combinations of one or more of the associated listed items.

Note that in the present invention, various information may be explained using terms such as "first," "second," and "third," but these information are not limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, first information may also be referred to as second information, and similarly, second information may be referred to as first information without departing from the scope of the invention. Depending on the context, the word "if" as used herein may be interpreted as "when" or "with" or "as determined."

The purpose of the present invention is to propose a multi-temporal CT image classification system and construction method based on a spatio-temporal attention model, and to solve the problem that multi-temporal CT images cannot be combined and processed based on a conventional convolutional neural network. Solve a problem. Note that the multi-phase CT images of the present invention include CT images scanned normally in clinical practice and contrast-enhanced CT images scanned after contrast agent injection; the CT images scanned normally are simple scan phase CT images; Contrast CT images scanned after injection include arterial phase, portal venous phase, and delayed phase CT images.

The multi-temporal CT image classification system based on the spatio-temporal attention model of the present invention, as shown in FIG. 1, includes the following:
a data acquisition unit, which is used to acquire s time phase CT images of a patient to be classified;
A first embedded layer network unit including s first embedded layer networks, each of which divides a CT image of each time phase into a plurality of image blocks, and divides each image block into an image block. It is expanded into a block vector, combined with all image block vectors and class label vectors, and then added to the position vector of the same dimension to obtain the embedded vector of the CT image of the corresponding time phase. The size of the image is
, H and W are the length and width of one CT image, and C is the number of layers of the CT image. The size of the image block after division is P×P×C, P is the length and width of the image block after division, each image block is expanded into an image block vector through a convolution layer, and the embedding vector ₀ and the embedding vector X ₀ is
_{X 0} ⁼ _[ ^X _{class ; X} ¹ _p ^; _{_} ^_
and
However, X _class represents a class label vector, X _pos represents a position vector, X _p represents an image block vector after linearization, N represents the number of image blocks after division, and N=HW/P ² It is. D is the number of convolution kernels in the convolution layer, and by combining the image block vector that has passed through the convolution layer and the learnable class label vector, it is possible to collect the expression information of the entire label vector, and furthermore, it is possible to collect the expression information of the entire label vector. Data information can be emphasized by adding it to the position vector of .

The multi-temporal CT image classification system also includes a spatial attention unit including s spatial attention networks, each spatial attention network comprising a first multi-head attention network MSA of the L1 layer and a first multi-head attention network MSA of the L1 layer. The first multi-head attention network MSA of the L1 layer and the first multi-layer perceptron of the L1 layer are sequentially interleaved, and the first multi-head attention network MSA includes a multi-layer perceptron and a first normalization layer. It includes a plurality of self-attention modules SA and one splicing layer, and the self-attention module SA transforms the normalized input vector into three different matrices: a query matrix Q _1i , a keyword matrix K _1i and a value matrix V _1i Specifically, the input vector is first converted into three different vectors: a query vector q, a keyword vector k, and a value vector v, of which the query vector q is used to match other vectors. , the keyword vector k is matched, the value vector v represents the information to be extracted, and three types of vectors, q, k, and v, are obtained by multiplying the learnable matrix and the input vector. Considering that the embedding vector is multidimensional, expressed from a global perspective,
Q _1i =XW _1i ^Q , K _1i =XW _1i ^K , V _1i =XW _1i ^V (2)
Then,
Here, W _1i ^Q , W _1i ^K , W _1i ^V represent the i-th trainable weight matrix, and X represents the input vector.

An attention function between each vector in the input vectors is generated based on three different matrices: a query matrix Q _1i , a keyword matrix K _1i , and a value matrix V _1i , and specifically, an attention function between each vector in the input vectors is generated. Find the dot product, divide the dot product by the square root of the dimension of the keyword vector k, and multiply by the value vector v through the softmax layer to find the sum, and the softmax function is used to map the input value to the interval (0, 1). It will be done. The formula for the attention function between input vectors is
(3)
and
Here, d _k represents the dimension of each keyword vector k in the keyword matrix K _1i , softmax( ) is a softmax function, and head _1i represents the output of the i-th self-attention module SA.

The splicing layer is used to splice the attention function output from each self-attention module SA to obtain a final spatial attention function,
MSA()=Concat(head ₁₁ ,..., head _1i ,...)W ₁ ^O (4)
It is expressed as,
MSA( ) is the output of the spatial attention network and W ₁ ^O is the trainable weight matrix.

The network can compare the connections between different vectors with each other and strengthen the important parts by the multi-head attention module. The first multi-layer perceptron MLP is used based on the first multi-head attention network MSA, MLP represents a multi-layer perceptron with Gelu function as a nonlinear layer, Gelu function is a high-performance neural network activation function, and its nonlinear This is because the change is a random regular transformation method that matches expectations. Specifically, the sum of the final spatial attention function and the input vector is set as the input vector corresponding to the first multilayer perceptron of the next layer,
(5)
LN represents the normalization method, x' _l represents the input vector of the first multi-layer perceptron, MSA() represents the output of the first multi-head attention network, x _l represents the l-th layer first multi-head attention network represents the input vector of .

The first multi-layer perceptron encodes the normalized input vector and adds it to the input vector, and uses the addition result as an input vector corresponding to the first multi-head attention network MSA of the next layer;
(6)
MLP() represents the output of the first multilayer perceptron, and x' _l-1 represents the input vector of the first multilayer perceptron at the l-1th layer.

The input vector of the first multi-head attention network MSA in the first layer is an embedding vector, that is, x _{1 =} It is used to normalize the first dimension vector of the vector obtained by adding the vectors and use it as a spatial feature of the CT image of the corresponding time phase,
(7)
x ⁰ _L represents the first dimension data of x _L after passing through all encoding layers, and L=2L1.

For the CT images of the simple scan phase, arterial phase, portal venous phase, and delayed phase, spatial features of the corresponding CT images of the simple scan phase, arterial phase, portal venous phase, and delayed phase are obtained, respectively.
A second embedding layer network unit including one second embedding layer network, wherein the class label vector is obtained after combining the spatial features of s corresponding temporal CT images output from the s spatial attention networks. is used to obtain the embedding layer vector x,
x = [X _class ; x _space ], x _space ∈R ^s×D , X _class ∈R ^1×D (8)
However, x _space represents the combined spatial feature.

The multi-temporal CT image classification system also includes a temporal attention unit including one temporal attention network, the structure and function of the temporal attention network are the same as the structure of the spatial attention network, specifically: The second multi-head attention network MSA of the L2 layer and the second multi-layer of the L2 layer include a second multi-head attention network MSA of the L2 layer, a second multi-layer perceptron of the L2 layer and a second normalization layer of one layer. The perceptrons are interleaved in sequence, and the second multi-head attention network MSA includes a plurality of self-attention modules SA and one splicing layer, and the self-attention module SA has a normalized input according to equation (2). Convert the vector into three different matrices: query matrix Q _2j , keyword matrix K _2j and value matrix V _2j , and based on the three different matrices: query matrix Q _2j , keyword matrix K _2j and value matrix V _2j , formulate the equation ( 3), generate the attention function between each vector in the input vectors, j is the index of the self-attention module SA in the temporal attention unit, and the splicing layer generates the output of each self-attention module SA according to equation (4). Splicing the attention function to obtain the final temporal attention function and adding the final temporal attention function and the input vector according to equation (5) is the input vector corresponding to the second multilayer perceptron of the next layer. According to equation (6), the second multi-layer perceptron encodes the normalized input vector, adds it to the input vector, and adds the addition result to the input vector corresponding to the second multi-head attention network MSA in the next layer. The input vector of the second multi-head attention network MSA in the first layer is the embedding layer vector output from the second embedding layer network unit, and the second normalization layer is the vector output from the second multi-layer perceptron in the final layer. is used to normalize the first dimension vector of the vector obtained by adding the input vector and obtain a vector x _time having spatial characteristics and temporal characteristics,
a classification layer unit including a classification layer W, which is used to obtain a classification result Prob based on a vector having spatial features and temporal features;
Prob=W(x _time ^T ) (9)
ProbεR ^C represents the probability distribution of classes, and C represents the total number of classes.

FIG. 2 is a flowchart of classification performed by the multi-temporal CT image classification system based on the spatio-temporal attention model of the present invention, and the details are as follows.
The CT images of s temporal phases of the patient to be classified acquired by the data acquisition unit are input to the first embedding layer network unit, and each first embedding layer network receives the CT images of the corresponding single temporal phase. Divide into multiple image blocks, develop each image block into an image block vector, combine all image block vectors and class label vectors, and then add them to the position vector of the same dimension to calculate the CT image of the corresponding time phase. Get the embedding vector,
inputting the obtained embedding vector of the CT image of the corresponding temporal phase into the corresponding spatial attention network in the spatial attention unit to obtain the spatial features of the CT image of the corresponding temporal phase;
The spatial features of the s CT images of the corresponding temporal phases output by the s spatial attention networks are input to the second embedding layer network unit, and the spatial features of the CT images of the s corresponding temporal phases are input to the second embedding layer network unit. After combining, configure the embedding layer vector by overlapping it with the class label vector,
Input the embedding layer vector into the temporal attention unit to obtain the vector with spatial features and temporal features, and finally input the obtained vector with spatial features and temporal features into the classification layer unit. The final classification result is then output.

The system of the present invention realizes classification of CT images based on differences in CT images of different types of tumors or subtypes, and further realizes diagnostic classification of tumor types/stages. The system of the present invention can be used to classify two or more types of tumors, and is specifically determined by how the system is constructed. For example, liver cancer is generally divided into two types: primary and secondary. Primary liver malignant tumors arise from the epithelial or mesenchymal tissues of the liver, and secondary (also called metastatic) liver cancers are malignant tumors originating from multiple organs throughout the body that invade the liver. refers to Generally, liver metastasis from malignant tumors in organs such as the stomach, biliary tract, pancreas, colorectum, ovary, uterus, lungs, and mammary glands is frequently observed.

FIG. 3 is a flowchart of a method for constructing a liver cancer multi-temporal CT image classification system based on the spatiotemporal attention model of the present invention, and specifically includes the following steps.
(1) collecting samples to construct a dataset, each sample of the dataset including liver cancer CT images of s time phases of one patient;
Taking the binary classification of hepatocellular carcinoma and intrahepatic cholangiocellular carcinoma for liver cancer CT images as an example, hepatocellular carcinoma (HCC) is a primary liver cancer with a high mortality rate. Intrahepatic cholangiocarcinoma (ICC) refers to adenocarcinoma that arises from the epithelium of secondary bile ducts and their branches, and is a primary liver malignant tumor with an incidence rate second only to hepatocellular carcinoma. . A total of 400 samples were collected, including 200 HCC samples and 200 ICC samples, and the labeling of all samples was performed by professional medical imaging physicians, specifically as follows: It is.
(1.1) First, we collect simple scan phase liver CT images and contrast-enhanced CT images (arterial phase, portal venous phase, and delayed phase liver CT images) of liver cancer patients from hospitals, and perform complete data screening. Select patient data with research information, use data masking technology to remove patient's personal private information, help protect patient privacy and improve data confidentiality, and ultimately from HCC and ICC patients. A total of 400 liver CT images and corresponding liver function test reports were collected, of which 200 were for HCC patients and 200 were for ICC patients, and they were labeled according to the class they belong to, HCC patients were labeled as 1, and ICC patients were labeled as 1. Label patient 0.

(1.2) A specialized medical imaging doctor labels and divides the focal areas in the liver CT images of four time phases to construct a data set.

In addition, due to individual differences in patients, the examining physician may set different number of scans for different patients, so the number of slices in the original CT image will be different, and for the convenience of research, the size and size of CT images in each time phase and Numbers are uniformly defined. In this example, the size of the liver CT image of each sample is 64 x 128 x 128 x 4, where 64 represents the number of layers of the liver CT image of each time phase, and 128 and 128 are the length of each liver CT image. 4 represents the four time phases,
In addition, to enhance the data and create more value from the data when the data is insufficient, the input is a 4-phase liver CT image that has completed data preprocessing, random rotation, random inversion. Supplement the dataset with samples.

(2) Based on the above spatiotemporal attention model including a data acquisition unit, a first embedded layer network unit, a spatial attention unit, a second embedded layer network unit, a temporal attention unit, and a classification layer unit. A multi-temporal CT image classification system is constructed, each sample in the dataset is used as input to the system, and trained with the goal of minimizing the error between the classification result output by the system and the classification label, and binary cross-entropy loss is applied. Taking as an example the calculation of the error between the classification result output by the system and the classification label using a function, it is expressed as follows.
Loss=-ylog(Prob)-(1-y)log(1-Prob) (10)
where y∈{0,1}, where 0 represents an ICC patient and 1 represents an HCC patient.

A stochastic gradient descent algorithm is used to optimize the entire system, and the goal is to find the minimum error loss and finally obtain the optimal classification model. In this example, the Adam stochastic optimization algorithm is used to perform gradient backpropagation and optimization, the learning rate is set to 0.0001, and the final result is two-dimensional hepatocellular carcinoma and intrahepatic cholangiocellular carcinoma. A multitemporal CT image classification system based on a spatiotemporal attention model that realizes term classification is obtained.

The method of the present invention is universal to various diseases that need to be diagnosed based on multi-phase CT images, and more effectively utilizes the characteristics of lesions in different temporal phases to establish temporal connections. By strengthening and abandoning the traditional convolutional neural network design as the main model, the attention mechanism allows you to put more computations into important areas and get more details of the target you want to focus on. This suppresses other unnecessary information, reduces calculation redundancy and delay, makes it easier to diagnose CT images in a shorter time, improves the accuracy of diagnosis, and makes the effect of diagnosis more stable.

Note that the examples described in the present invention are not limitations on the embodiments of the present invention, but are merely examples given to clearly explain the present invention. Various other types of modifications or changes may occur to those skilled in the art in light of the above description. It is not necessary or possible to list all embodiments here. All modifications, equivalent substitutions, improvements, etc. within the spirit and principles of the present invention are included within the scope of the claims of the present invention. The scope of protection claimed by the present invention shall be in accordance with the content of the claims, and the description of the detailed description and the like in the specification may be used to interpret the content of the claims. Obvious changes or modifications arising therefrom still fall within the protection scope of the present invention.

Claims (2)

A multi-temporal CT image classification system based on a spatiotemporal attention model that classifies tumors by obtaining spatial and temporal features from CT images, the system comprising:
A data acquisition unit for acquiring CT images of s temporal phases of a patient to be classified, where s is 2 or more, and the CT images of s temporal phases are a simple scan phase CT image and a contrast-enhanced CT image. a data acquisition unit including at least two of an arterial phase CT image in CT, a portal venous phase CT image in contrast-enhanced CT, and a delayed phase CT image in contrast-enhanced CT;
A first embedded layer network unit including s first embedded layer networks, wherein each of the s first embedded layer networks divides a CT image of each time phase into a plurality of image blocks, and divides each image block into a plurality of image blocks. Each block is expanded by a convolution layer to obtain an image block vector after linearization, and after combining all the image block vectors and class token vectors, they are added to the position vector of the same dimension to obtain a CT image of the corresponding time phase. Obtain the embedding vector, where the size of the CT image for each time phase is
, H and W are the length and width of one CT image, C is the number of layers of the CT image, the size of the image block after division is P×P×C, and P is the width after division. _is the length and width _of the image _block , _and ^the _embedding vector is _{X 0} = ^{[ X} _class ^{; (1+N)×D} , where X _class represents the class token vector, X _pos represents the position vector, X _p represents the image block vector after linearization, R represents the set of all real numbers, and N represents the division a first embedding layer network unit representing the number of subsequent image blocks, N=HW/P ² , and D is the number of convolution kernels of the convolution layer;
a spatial attention unit comprising s spatial attention networks, each of the s spatial attention networks comprising: a first multi-head attention network MSA in the L1 layer; a first multi-layer perceptron in the L1 layer; a first normalization layer of layers, L1 is any positive integer, a first multi-head attention network MSA of the L1 layer and a first multilayer perceptron of the L1 layer are interleaved in sequence;
The first multi-head attention network MSA includes a plurality of self-attention modules SA and one splicing layer, and the self-attention module SA receives a normalized input vector from a query matrix Q _1i , a keyword matrix K _1i and a value. It is used to transform the input vector into three different matrices of matrix V _1i , query vector q, keyword vector k and value vector v, among which query vector q is matched with other vectors. The keyword vector k is used for matching, the value vector v represents the information to be extracted, the three types of vectors q, k, and v are obtained by multiplying the input vector by the learnable matrix, and the embedding vector Considering that is multidimensional, expressed from a global perspective,
Q _1i =XW _1i ^Q , K _1i =XW _1i ^K , V _1i =XW _1i ^V (2),
where W _1i ^Q , W _1i ^K , W _1i ^V represent the i-th trainable weight matrix, X represents the input vector,
Generate an attention function between each vector in the input vectors based on three different matrices: a query matrix Q _1i , a keyword matrix K _1i , and a value matrix V _1i , calculate a dot product of the query vector q and each keyword vector k, The dot product is divided by the square root of the dimension of the keyword vector k and multiplied by the value vector v through a softmax layer to find the sum.The softmax function is used to map the input value to the interval (0,1), and the The formula (3) of the attention function is
(3),
where dk represents the dimension of each keyword vector k in the keyword matrix K _1i , softmax() is the softmax function, head _1i represents the output of the i-th self-attention module SA,
The splicing layer is used to splice the attention functions output from each self-attention module SA to obtain the final spatial attention function, and the splicing layer is expressed by the following equation (4),
MSA()=Concat(head ₁₁ ,..., head _1i ,...)W ₁ ^O (4)
where MSA() is the output of the spatial attention network, W ₁ ^O is the trainable weight matrix,
(5)
Here, LN represents the normalization method, x′ _l represents the input vector of the first multi-layer perceptron, MSA() represents the output of the first multi-head attention network, and x _l represents the first multi-head attention network of the l-th layer. represents the input vector of the head attention network,
The sum of the final spatial attention function and the input vector is the input vector corresponding to the first multilayer perceptron of the next layer,
(6)
Here, MLP() represents the output of the first multilayer perceptron, x'l _-1 represents the input vector of the first multilayer perceptron of the l-1th layer,
The first multi-layer perceptron encodes the normalized input vector and adds it to the input vector, and uses the addition result as an input vector corresponding to the first multi-head attention network MSA of the next layer;
The input vector of the first multi-head attention network MSA in the first layer is an embedded vector, and the first normalization layer is a vector obtained by adding the output vector of the first multi-layer perceptron in the final layer and its input vector. a spatial attention unit that normalizes a first dimension vector of and obtains spatial features of a CT image of a corresponding time phase;
a second embedding layer network unit including one second embedding layer network, the second embedding layer network unit forming a class token after combining spatial features of s corresponding temporal CT images output from the s spatial attention networks; Together with the vector, the embedding layer vector x is obtained using the following formula, x=[X _class ; _xspace ], _xspace ∈R ^s×D , X _class ∈R ^1×D (7)
where x _space represents the combined spatial feature, a second embedded layer network unit;
A temporal attention unit comprising one temporal attention network, the temporal attention network comprising a second multi-head attention network MSA in the L2 layer, a second multi-layer perceptron in the L2 layer, and a second normal in one layer. a second multi-head attention network MSA of the L2 layer and a second multilayer perceptron of the L2 layer are interleaved in sequence, L2 is an arbitrary positive integer;
The second multi-head attention network MSA includes a plurality of self-attention modules SA and one splicing layer, and the self-attention module SA has the following formula (8):
Q _2i = XW _2i ^Q , K _2i = XW _2i ^K , V _2i = XW _2i ^V (8) Spread the normalized input vector into three different matrices: query matrix Q _2j , keyword matrix K _2j and value matrix V _2j Based on three different matrices: query matrix Q _2j , keyword matrix K _2j and value matrix V _2j , according to the following formula (9),
head _2i =Attention _2i (Q _2i ,K _2i ,V _2i )=Softmax(Q _2i K _2i ^T /√d _k )V _2i (9) Generate the attention function between each vector in the input vectors, where j is the time is the index of the self-attention module SA in the target attention unit, and the splicing layer is calculated according to the following formula (10).
MSA()=Concat( _head21 ,..., _head2i ,...) _W2O ⁽ 10) The attention function output from each self-attention module SA is spliced to obtain the final temporal attention function, and the following formula ( According to 11),
x′ _l =MSA(LN(x _l ))+x _l , l=1...L2 (11)
Here, LN represents the normalization method, x′ _l represents the input vector of the second multi-layer perceptron, MSA() represents the output of the second multi-head attention network, and x _l represents the second multi- head attention network of the l-th layer. represents the input vector of the head attention network,
The sum of the final temporal attention function and the input vector is set as the input vector corresponding to the second multilayer perceptron of the next layer, and according to the following equation (12),
x _l =M LP (LN(x' _l-1 )) + x' _l-1 , l=2... L2 (12)
Here, MLP() represents the output of the second multilayer perceptron, x'l _-1 represents the input vector of the second multilayer perceptron of the l-1th layer,
The second multi-layer perceptron encodes the normalized input vector, adds it to the input vector, uses the addition result as an input vector corresponding to the second multi-head attention network MSA of the next layer, and uses the second multi-head attention network MSA of the first layer as the input vector. The input vector of the multi-head attention network MSA is the embedding layer vector output from the second embedding layer network unit, and the second normalization layer adds the vector output from the second multilayer perceptron in the final layer and the input vector. a temporal attention unit that normalizes the first dimension vector of the obtained vector to obtain a vector x _time having spatial characteristics and temporal characteristics;
A classification layer unit including a classification layer W, which is used to obtain a classification result Prob according to the following formula (13) based on a vector having spatial characteristics and temporal characteristics,
Prob=W(x _time ^T ) (13)
x _time ^T is a vector having the spatial features and temporal features, Prob∈R ^C represents the probability distribution of classes, R represents the set of all real numbers, C represents the total number of classes, a classification layer unit A multi-temporal CT image classification system based on a spatio-temporal attention model, comprising: Collecting samples to construct a dataset, each sample of the dataset including s temporal phases of CT images of one patient, s being 2 or more, and s temporal phases; The CT image includes at least two of a simple scan phase CT image, an arterial phase CT image in contrast-enhanced CT, a portal venous phase CT image in contrast-enhanced CT, and a delayed phase CT image in contrast-enhanced CT;
A multi-temporal CT image classification system based on the spatio-temporal attention model according to claim 1 is constructed, and each sample in the dataset is input to the system to minimize the error between the classification result output by the system and the classification label. A multi-temporal CT image classification system based on a spatio-temporal attention model, comprising the step of training with the goal of obtaining a multi-temporal CT image classification system based on the spatio-temporal attention model. How to build.