CN115841476A - Method, device, equipment and medium for predicting life cycle of liver cancer patient - Google Patents
Method, device, equipment and medium for predicting life cycle of liver cancer patient Download PDFInfo
- Publication number
- CN115841476A CN115841476A CN202211607508.9A CN202211607508A CN115841476A CN 115841476 A CN115841476 A CN 115841476A CN 202211607508 A CN202211607508 A CN 202211607508A CN 115841476 A CN115841476 A CN 115841476A
- Authority
- CN
- China
- Prior art keywords
- image
- data
- liver
- tumor
- images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Image Analysis (AREA)
- Apparatus For Radiation Diagnosis (AREA)
Abstract
The invention provides a method, a device, equipment and a medium for predicting the life cycle of a liver cancer patient, which relate to the technical field of data processing and comprise the following steps: acquiring historical liver images of a plurality of patients, and preprocessing the acquired data set; the data set comprises a plurality of data packets, each data packet comprises a plurality of tumor images of each modality of a certain patient, and a unique life cycle label is associated with each data packet; establishing an initial prediction model, acquiring any data packet in a data set, and performing multi-mode image fusion on tumor images in the data packet after feature extraction in the initial prediction model to output intermediate feature data; after the attention mechanism learning is carried out by adopting a transformer module; outputting a life cycle prediction result by adopting a classifier; training is completed to obtain a target prediction model; the method comprises the steps of obtaining a liver image of a target patient, preprocessing the liver image, inputting the preprocessed liver image into a target prediction model, generating a target life cycle prediction result, and solving the problem that a liver cancer patient life cycle prediction method in the prior art is lack of.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a method, a device, equipment and a medium for predicting the life cycle of a liver cancer patient.
Background
Primary liver cancer (HCC) refers to a primary malignant tumor originated from hepatocytes, seriously harms human life and health, and is the fourth cause of death due to cancer in the world. Because of its high invasiveness and occult nature, most patients are in the middle and late stages when diagnosed, and the possibility of radical treatment of tumors is greatly reduced. At present, the main treatment mode of HCC is that an experienced doctor judges the malignancy and the stage of a tumor and determines a diagnosis and treatment scheme through CT or MRI images of a patient, but an accurate disease prediction and prognosis evaluation strategy is lacked.
With the development of deep learning, the auxiliary diagnosis technology of artificial intelligence combined medical images is continuously perfected. By means of different neural network models such as ResNet and EfficientNet, artificial intelligence can not only extract superficial layer information such as the size and the shape of a tumor from CT images, ultrasonic images and MRI images, but also mine deep-level association between semantic information contained in the tumor images and target characteristics such as benign and malignant tumors, and provide reference opinions for doctors. At present, people can accurately judge whether tumors are good or not by using different deep learning models, but prognosis evaluation of diseases is still unknown, especially survival prediction of cancer patients is mostly judged according to experience of doctors.
Disclosure of Invention
In order to overcome the technical defects, the invention aims to provide a method, a device, equipment and a medium for predicting the life cycle of a liver cancer patient, and solve the problem that the method for autonomously predicting the life cycle of the liver cancer patient is lacked in the prior art.
The invention discloses a life prediction method for a liver cancer patient, which comprises the following steps:
acquiring historical liver images of a plurality of patients, preprocessing each liver image, and acquiring a data set containing tumor images; wherein the historical liver images comprise liver images of at least three modalities; the data set comprises a plurality of data packets, each data packet comprises a plurality of tumor images of each modality of a certain patient, and a unique life cycle label is associated with each data packet;
establishing an initial prediction model, wherein the initial prediction model comprises a feature extraction module, a transform module and a classifier, and is trained by adopting the data set;
acquiring any data packet in a data set, and in the initial prediction model, respectively performing multi-mode image fusion on tumor images in the data packet after feature extraction by adopting a feature extraction module so as to output intermediate feature data containing all features in each tumor image; performing attention mechanism learning by adopting a transformer module based on the intermediate characteristic data and the pathological information of the patient corresponding to the data packet; outputting a life cycle prediction result by adopting a classifier; training is completed to obtain a target prediction model;
and acquiring a liver image of the target patient, preprocessing the liver image, inputting the preprocessed liver image into the target prediction model, and generating a target life cycle prediction result.
Preferably, the performing multi-modal image fusion on the tumor images in the data packet after feature extraction by using the feature extraction module to output intermediate feature data including all features in each tumor image includes:
in a feature extraction module, feature extraction is carried out on each tumor image by respectively adopting a convolution network based on a plurality of tumor images of each modality in the data packet, so as to obtain at least three first feature data sets comprising a plurality of first feature data, wherein each first feature data set corresponds to a tumor image of one modality;
fusing the first characteristic data in each first characteristic data set by adopting an attention convergence network to obtain at least three second characteristic data;
and performing feature combination on each second feature data by adopting a full connection layer, and outputting intermediate feature data.
Preferably, the fusing the first feature data in each of the first feature data sets by using the attention convergence network to obtain at least three second feature data includes:
for any first feature data set:
obtaining corresponding attention scores by each first characteristic data of the first characteristic data set through an attention mechanism;
converting each attention score into a probability value in a preset range by adopting a softmax function;
and multiplying each first characteristic data by the corresponding probability value respectively and then adding to obtain second characteristic data.
Preferably, the pre-treatment comprises:
acquiring historical liver images of a certain patient, and adjusting each liver image according to a preset size;
for any adjusted liver image:
manually marking the liver image to obtain a tumor region, and generating a mask image;
dividing the liver image into a plurality of image slices, and overlapping each image slice with the mask image to generate three-channel images corresponding to each image slice as tumor images;
and collecting tumor images corresponding to the image slices, namely a plurality of tumor images of a certain modality of the patient.
Preferably, the superimposing each image slice with the mask image to generate a three-channel image corresponding to each image slice includes:
for each image slice, a three-channel image is generated in such a way that the image slice overlays the image slice overlay mask image.
Preferably, after the acquiring the data set containing the tumor image, the method further comprises:
data enhancement is performed based on the data set.
Preferably, before performing attention mechanism learning based on the intermediate feature data and the pathological information of the patient corresponding to the data packet by using a transformer module, the method includes:
acquiring pathological information of a patient corresponding to the data packet;
respectively carrying out one-bit coding and normalization processing on the category data and the numerical data in the pathological information to generate numerical characteristics;
and embedding the digital features into a vector with the same dimension as the intermediate feature data through a multilayer perceptron, and adding the digital features and the intermediate feature data to input into a transformer module.
The invention also provides a device for predicting the life cycle of a liver cancer patient, which comprises:
the acquisition module is used for acquiring historical liver images of a plurality of patients, preprocessing each liver image and acquiring a data set containing tumor images; wherein the historical liver images comprise liver images of at least three modalities; the data set comprises a plurality of data packets, each data packet comprises a plurality of tumor images of each modality of a certain patient, and a unique life cycle label is associated with each data packet;
the system comprises an establishing module, a prediction module and a prediction module, wherein the establishing module is used for establishing an initial prediction model, the initial prediction model comprises a feature extraction module, a transformer module and a classifier, and the data set is adopted for training;
the processing module is used for acquiring any data packet in the data set, and in the initial prediction model, the characteristic extraction module is adopted to perform multi-mode image fusion on the tumor images in the data packet after characteristic extraction so as to output intermediate characteristic data containing all characteristics in each tumor image; performing attention mechanism learning by adopting a transformer module based on the intermediate characteristic data and the pathological information of the patient corresponding to the data packet; outputting a life cycle prediction result by adopting a classifier; training is completed to obtain a target prediction model;
and the execution module is used for acquiring the liver image of the target patient, preprocessing the liver image and inputting the preprocessed liver image into the target prediction model to generate a target life cycle prediction result.
The invention also provides computer equipment which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the steps of the life cycle prediction method of the liver cancer patient when executing the computer program.
The present invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the method for predicting the survival of a liver cancer patient.
After the technical scheme is adopted, compared with the prior art, the method has the following beneficial effects:
the method comprises the steps of obtaining historical liver images of a plurality of patients, wherein the historical liver images comprise liver images of at least three modalities, obtaining a data set after preprocessing, wherein the data set comprises a plurality of data packets, each data packet comprises a plurality of tumor images of each modality of a certain patient, and a unique life cycle label is associated with each data packet; establishing an initial prediction model, and fusing different modal image characteristics after extracting the characteristics of the tumor images in the data packet by adopting a characteristic extraction module; according to the image characteristics and the pathological data, the prediction result of the life cycle of the patient is obtained through the classifier through the attention learning of the transducer block, and the problem that the method for autonomously predicting the life cycle of the liver cancer patient is lacked in the prior art is solved.
Drawings
FIG. 1 is a flowchart illustrating a method for predicting the survival of a liver cancer patient according to a first embodiment of the present invention;
FIG. 2 is a flowchart of outputting intermediate characteristic data according to a first embodiment of the method for predicting the survival time of a liver cancer patient of the present invention;
FIG. 3 is a schematic diagram of image slices and mask images according to a first embodiment of the method for predicting the survival of a liver cancer patient of the present invention;
FIG. 4 is a schematic diagram of a network structure of an initial prediction model in a method for predicting the survival time of a liver cancer patient according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a network structure of an attention convergence network according to a first embodiment of the method for predicting the lifetime of a liver cancer patient of the present invention;
FIG. 6 is a schematic block diagram of a second embodiment of the device for predicting the survival time of a liver cancer patient according to the present invention;
fig. 7 is a schematic diagram of a computer device module according to the present invention.
Reference numerals:
5-a liver cancer patient life prediction device; 51-an acquisition module; 52-a building module; 53-a processing module; 54-an execution module; 6-a computer device; 61-a memory; 62-processor.
Detailed Description
The advantages of the invention are further illustrated in the following description of specific embodiments in conjunction with the accompanying drawings.
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same modality from each other. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present disclosure. The word "if" as used herein may be interpreted as "at" \8230; "or" when 8230; \8230; "or" in response to a determination ", depending on the context.
In the description of the present invention, it is to be understood that the terms "longitudinal", "lateral", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", and the like, indicate orientations or positional relationships based on those shown in the drawings, and are used merely for convenience of description and for simplicity of description, and do not indicate or imply that the referenced devices or elements must have a particular orientation, be constructed in a particular orientation, and be operated, and thus, are not to be construed as limiting the present invention.
In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "mounted," "connected," and "connected" are to be interpreted broadly, and may be, for example, a mechanical connection or an electrical connection, a communication between two elements, a direct connection, or an indirect connection through an intermediate medium, and those skilled in the art will understand the specific meaning of the terms as they are used in the specific case.
In the following description, suffixes such as "module", "part", or "unit" used to indicate elements are used only for facilitating the description of the present invention, and do not have a specific meaning per se. Thus, "module" and "component" may be used in a mixture.
The first embodiment is as follows: the invention discloses a life cycle prediction method for a liver cancer patient, which is used for explaining that a 2D multi-modal multi-example learning transform model is constructed, and the corresponding predicted life cycle can be given only by inputting flat scan CT, two-stage enhanced CT and the current pathological detection data of the patient so as to help a doctor to select a scheme capable of prolonging the life cycle of the patient to the maximum extent from various clinical diagnosis and treatment methods. Specifically, referring to fig. 1-5, the method includes the following steps:
s100: acquiring historical liver images of a plurality of patients, preprocessing each liver image, and acquiring a data set containing tumor images; wherein the historical liver images comprise liver images of at least three modalities; the data set comprises a plurality of data packets, each data packet comprises a plurality of tumor images of each modality of a certain patient, and a unique life cycle label is associated with each data packet;
in the above embodiment, a clear liver image can be acquired by using a CT image acquisition device of a hospital, and can be directly acquired from a hospital database, and in the present application, the lifetime prediction is performed based on a transform network, and in order to predict the lifetime according to the change of a liver tumor, images of multiple historical modalities, such as images of three historical phases, or images of multiple devices are taken into consideration.
The pretreatment specifically comprises the following steps: acquiring historical liver images of a certain patient, and adjusting each liver image according to a preset size; by way of example and not limitation, the reserved size is set to be 20 pixels, and a tumor area in the image is delineated by a professional physician by using annotation software; counting the maximum length and width of a tumor region in all liver images drawn by a doctor, widening by 20 pixels in the x and y directions respectively, and replacing the original complete CT image with the cuboid region to serve as an interested region; that is, the images are adjusted to a uniform size for subsequent model processing.
After the above-mentioned size adjustment, for any adjusted liver image: manually marking the liver image (which may be marked by the professional physician by using software), obtaining a tumor region, and generating a mask image (refer to the right picture of fig. 3); dividing the liver image into a plurality of image slices (for example, set as 224 × 224 pixels, refer to the left picture in fig. 3), and overlapping each image slice with the mask image to generate three-channel images corresponding to the image slices as tumor images; and collecting tumor images corresponding to the image slices, namely a plurality of tumor images of a certain modality of the patient.
It should be noted that the generating a three-channel image corresponding to each image slice by superimposing each image slice with the mask image includes: for each image slice, a three-channel image is generated in such a way that the image slice overlaps the image slice with the mask image, illustratively two image slices overlap the mask image, which two image slices are the same image slice and can be obtained by replication.
For the above processing, as a specific example and not by way of limitation, each patient has a three-phase CT image and a corresponding tumor region, the three-phase CT image is operated, the tumor is regarded as a 2D image according to slice (i.e., the liver image is divided into a plurality of image slices according to the slice), and the gray-scale image corresponding to the slice is changed into a three-channel image by the superposition of the image + the tumor mask.
After the above-mentioned acquiring the data set containing the tumor image, in order to further increase the data subsequently used for model training, thereby further increasing the model processing accuracy, the method may further include: data enhancement is performed based on the data set. For example, according to the task object, the local liver tumor image in the CT image is cut into a gray scale image (i.e., the image slice) with a size of 224 × 224, and data enhancement is performed by using image processing operations such as random horizontal inversion, random vertical inversion, gaussian blur, contrast enhancement, and the like in the pytorch after three channels are superimposed. During data enhancement, consistency of the operations experienced by the image slice and the mask image is maintained. And inputting the image obtained after the data enhancement operation into the following model, training by using a back propagation mechanism, and recording the model accuracy acc and the model cumulative loss.
S200: establishing an initial prediction model, wherein the initial prediction model comprises a feature extraction module, a transform module and a classifier, and is trained by adopting the data set;
in the above steps, the data set may be divided into a training set, a validation set and a test set in a proportion of 7 to 1, the training set is used for training in the following step S300, and the validation set and the test set are used for hyper-parametric adjustment and testing of the model. It should be emphasized that, as described above, each data packet in the data set can be regarded as a packet composed of a plurality of instances (i.e., a plurality of tumor images in each modality is an example), and the packet is labeled (i.e., the lifetime label is described above), but the instance (i.e., on the tumor image) itself does not have the label. All CT images of a patient are classified into one bag by means of the thought of multi-example learning, and the three modes can be images acquired by different equipment, such as flat scan CT, first type enhanced CT and second type enhanced CT; or images acquired at different periods, such as three stages of a tumor. Referring to fig. 4, as an illustration, where n1 is the number of two-dimensional cross-sections containing a tumor in a flat-scan CT, n2 is the number of two-dimensional cross-sections containing a tumor in a first enhanced CT, and n3 is the number of two-dimensional cross-sections containing a tumor in a second enhanced CT.
S300: acquiring any data packet in a data set, and in the initial prediction model, respectively performing multi-mode image fusion on tumor images in the data packet after feature extraction by adopting a feature extraction module so as to output intermediate feature data containing all features in each tumor image; performing attention mechanism learning by adopting a transformer module based on the intermediate characteristic data and the pathological information of the patient corresponding to the data packet; outputting a life cycle prediction result by adopting a classifier; adjusting the initial prediction model based on the comparison between the output lifetime prediction result and the lifetime label associated with the data packet; training is completed to obtain a target prediction model;
in this embodiment, based on the above, the prediction model provided in this embodiment may perform feature extraction and fusion on the tumor images of multiple modalities (and each modality may include multiple tumor images (image slices/local tumor images)) through the feature extraction module, so as to obtain features of the tumor images in different modalities, so as to learn tumor changes in the subsequent transform module, and determine, according to the changes, the probability that the lifetime is in each modality through the classifier, so as to output the predicted lifetime result. The predicted life cycle result is a certain time interval, which can be a month, a week, a day, etc., such as 0-3 months; 3-6 months; more than 6 months, etc.
Specifically, the above-mentioned feature extraction module performs multi-modal image fusion on the tumor images in the data packet after feature extraction, so as to output intermediate feature data including all features in each tumor image, with reference to fig. 2, including:
s311: in a feature extraction module, feature extraction is carried out on each tumor image by respectively adopting a convolution network based on a plurality of tumor images of each modality in the data packet, so as to obtain at least three first feature data sets comprising a plurality of first feature data, wherein each first feature data set corresponds to a tumor image of one modality;
in the above step, for each slice (i.e. the above slice image), the above method is used to superimpose the single-channel pictures into three-channel pictures (i.e. picture + mask), i.e. each tumor image, and the image features (i.e. the above first feature data) are extracted by different simple convolution layers. The convolution network can use common convolution layers such as ResNet18 or ResNet34, and can set a convolution network for each tumor image, or set a convolution network for several tumor images in each mode to execute processing in turn; even a convolution network can be set up for all tumor networks, which can be adjusted according to the actual processing data volume.
S312: fusing the first characteristic data in each first characteristic data set by adopting an attention convergence network to obtain at least three second characteristic data;
in the above step, for images of different phases (modalities), different weights are given to the feature vectors extracted by different slices and summed up, so as to obtain the feature vector corresponding to the phase, that is, the second feature data. It should be noted that, the above-mentioned at least three are exemplified by three modalities (three devices) in the present application, and in an actual processing process, besides a multi-phase CT image, an MRI image may be set, and only a slight adjustment to the model is needed (i.e. the input of the above-mentioned convolution network and attention convergence network is added), the model may be added, so as to implement a multi-modality, multi-example, transform network learning for CT and MRI.
Specifically, the above fusing the first feature data in each first feature data set by using the attention convergence network to obtain at least three second feature data includes:
for any first feature data set: obtaining corresponding attention scores by each first characteristic data of the first characteristic data set through an attention mechanism; converting each attention score into a probability value in a preset range by adopting a softmax function; and multiplying each first characteristic data by the corresponding probability value respectively and then adding to obtain second characteristic data.
Illustratively, the focus-focusing (see fig. 5) embodies the mechanism, which is essentially a weighted sum of n eigenvectors (i.e., the first feature data described above). Each vector is subjected to an attention mechanism to obtain a different attention score alpha j Converted into a probability value sigma between 0 and 1 through a softmax function j The final output (i.e. the second feature data) can be obtained by multiplying the probability value by the original feature vector and then summing.
S313: and performing feature combination on each second feature data by adopting a full connection layer, and outputting intermediate feature data.
In the above steps, features of different phases (modalities) are combined by using a full connection layer, and the output feature vectors (i.e. the above feature data) are regarded as image features extracted from all tumor images of the patient.
Specifically, before performing attention mechanism learning based on the intermediate feature data and the pathological information of the patient corresponding to the data packet by using the transformer module, the method includes:
s321: acquiring pathological information of a patient corresponding to the data packet;
based on the above, in this embodiment, the data packets all correspond to a unique patient, and the pathological information may be directly collected from the database for obtaining the liver image in advance, or may be obtained in the preprocessing process, specifically, for example, the number of pixel points occupied by the three-dimensional tumor region is recorded, and the pixel points are recorded into the pathological information csv file of the patient, where the size of the tumor is similar to the size of the tumor.
S322: respectively carrying out one-bit coding and normalization processing on the category data and the numerical data in the pathological information to generate numerical characteristics;
specifically, one-hot coding is carried out on classified data in the csv file; carrying out normalization processing on numerical data in the csv file; by way of example and not limitation, the classification data in the csv file includes, but is not limited to, age, sex, cp, and the like, and the numerical data is a detection value corresponding to each classification data. After the digital features are generated, the data features and the intermediate feature data are input into a transform module for processing, namely, learning based on pathological data is combined, so that the accuracy of subsequent prediction results is improved.
S323: and embedding the digital features into a vector with the same dimension as the intermediate feature data through a multilayer perceptron, and adding the digital features and the intermediate feature data to input into a transformer module.
In the above step, the intermediate feature data is a feature vector, and in order to realize the summation of the digital feature and the intermediate feature data, the digital feature is converted into a vector form by a multi-layer perceptron, so as to be input into a transform module for attention mechanism learning. And then obtaining the prediction results of different survival times of the patient through the classifier.
S400: and acquiring a liver image of the target patient, preprocessing the liver image, inputting the preprocessed liver image into the target prediction model, and generating a target life cycle prediction result.
In the above step, the target prediction model and the initial prediction model are in a structure consistent with each other, and are trained to adjust and fix model parameters; the preprocessing here is similar to the preprocessing in step S100, and includes processing the picture into a consistent size, dividing each image into a plurality of slices, and converting the slices into a three-channel image in the form of picture + mask to be input into the target prediction model for feature extraction.
It should be noted that, in the process of using the target prediction model to perform auxiliary diagnosis, the number of input modalities of the model can be flexibly adjusted according to the medical imaging scanning technique actually accepted by the patient. If the patient still holds the MRI image, the image is preprocessed, and a local tumor picture with the same size is obtained by interception (namely, the preprocessing) and then the picture can be input into the network.
The method for predicting the life cycle of the liver cancer patient provided by the embodiment can synthesize medical imaging pictures and pathological data of the patient by using a multi-modal and multi-example learning method, acquire the pathological data by fusing image features of different modalities by using attention convergence, assemble the image features, perform attention learning by a transducer block, and finally obtain prediction results of different life cycles of the patient by a classifier. The method has the advantages that the prognosis level of the patient is comprehensively evaluated by using more information, and the method has strong practical and reference values.
Example two: the present invention also provides a liver cancer patient survival prediction device 5, referring to fig. 6, including:
an obtaining module 51, configured to obtain historical liver images of multiple patients, pre-process each liver image, and obtain a data set including a tumor image; wherein the historical liver images comprise liver images of at least three modalities; the data set comprises a plurality of data packets, each data packet comprises a plurality of tumor images of each modality of a certain patient, and a unique life cycle label is associated with each data packet;
in the above-mentioned acquisition module, the preprocessing includes adjusting each liver image by a preset size, that is, adjusting each image to the same size, and for any adjusted liver image: manually marking the liver image (which can be marked by the professional physician by using software) to obtain a tumor area, and generating a mask image; the liver image is segmented into a plurality of image slices (e.g., set to 224 × 224 pixels), and each image slice is superimposed with the mask image to generate a three-channel image, i.e., a tumor image, corresponding to each image slice in the form of image + mask.
The establishing module 52 is configured to establish an initial prediction model, where the initial prediction model includes a feature extraction module, a transform module, and a classifier, and is trained by using the data set;
the data set may be divided into a training set, a validation set and a test set according to a ratio of 7. All CT images of a patient are classified into a bag by means of the multi-example learning idea, and the three modes can be images acquired by different equipment, such as flat scan CT, first enhanced CT and second enhanced CT; or images acquired at different periods, such as three stages of tumor, etc.
A processing module 53, configured to obtain any data packet in a data set, and in the initial prediction model, perform multi-modal image fusion on tumor images in the data packet after feature extraction by using a feature extraction module, so as to output intermediate feature data including all features in each tumor image; performing attention mechanism learning by adopting a transformer module based on the intermediate characteristic data and the pathological information of the patient corresponding to the data packet; outputting a sample life cycle prediction result by adopting a classifier; adjusting the initial prediction model based on the comparison between the sample lifetime prediction result and the lifetime label associated with the data packet; training is completed to obtain a target prediction model;
in the processing module, feature extraction is performed on tumor pictures of multiple modalities (each modality may include multiple tumor pictures (image slices/local tumor pictures)) through the feature extraction module, feature fusion is performed on the tumor pictures under each modality through the attention convergence network, features of the tumor pictures under different modalities can be obtained, attention mechanism learning is performed in the subsequent transform module based on the extracted features and pathological information of a patient to obtain tumor changes, and the probability that the life cycle is in each modality is judged through the classifier according to the changes of the tumor changes, so that a predicted life cycle result is output.
And the execution module 54 is used for acquiring the liver image of the target patient, preprocessing the liver image, inputting the preprocessed liver image into the target prediction model, and generating a target life cycle prediction result.
In the execution module, the target prediction model and the initial prediction model have the same structure, and the preprocessing is similar to the preprocessing of the acquisition module, and includes processing a picture into the same size, dividing each image into a plurality of slices, and converting the slices into a three-channel image (a tumor image corresponding to a liver image of a target patient) in the form of a picture + a mask to be input into the target prediction model for processing.
In the embodiment, an acquisition module is adopted to acquire historical liver images of a plurality of patients, wherein the historical liver images comprise liver images of at least three modalities, a data set comprising tumor images is acquired after preprocessing, the data set comprises a plurality of data packets, each data packet comprises a plurality of tumor images of each modality of a certain patient, and a unique life cycle label is associated with each data packet; after an initial prediction model is established in an establishing module, training is carried out by adopting a data set, and the training comprises the steps of respectively extracting the characteristics of tumor images in the data packet by adopting a characteristic extracting module, and fusing the characteristics of different modal images; according to the image characteristics, pathological data are acquired at the same time, through attention learning of a transformer block, and finally, prediction results of different survival times of the patient are obtained through a classifier, the number of input modes (examples) of the model can be flexibly adjusted according to a medical imaging scanning technology actually accepted by the patient, and the problem that a method for autonomously predicting the survival time of the liver cancer patient based on the tumor state is lacked in the prior art is solved.
Example three: in order to achieve the above object, the present invention further provides a computer device 6, the computer device may include a plurality of computer devices, the components of the liver cancer patient lifetime prediction apparatus 5 according to the second embodiment may be distributed in different computer devices 6, and the computer device 6 may be a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, etc. executing programs. The computer device of the embodiment at least includes but is not limited to: a memory 61, a processor 62 and a liver cancer patient life prediction device 5 which can be mutually connected by a system bus. As shown in FIG. 7, it is noted that FIG. 7 only shows a computer device having components, but it is understood that not all of the shown components are required to be implemented, and that more or fewer components may be implemented instead.
In this embodiment, the memory 61 may include a program storage area and a data storage area, wherein the program storage area may store an application program required for at least one function of the system; the storage data area may store data of a user on the computer device. Further, the memory 61 may include high speed random access memory, and may also include non-volatile memory, and in some embodiments, the memory 61 may optionally include memory 61 located remotely from the processor, which may be connected via a network. Examples of such networks include, but are not limited to, the internet, local area networks, and the like.
Processor 62 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 62 is typically used to control the overall operation of the computer device. In this embodiment, the processor 62 is configured to run program codes or processing data stored in the memory 61, for example, run the liver cancer patient lifetime prediction apparatus 5, so as to implement the liver cancer patient lifetime prediction method of the first embodiment.
It is noted that fig. 7 only shows the computer device 6 with components 61-62, but it is to be understood that not all shown components are required to be implemented, and that more or less components may be implemented instead.
Example four:
to achieve the above objects, the present invention also provides a computer-readable storage medium including a plurality of storage media such as a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a magnetic disk, an optical disk, a server, etc., on which a computer program is stored, which when executed by the processor 62, implements corresponding functions. The computer-readable storage medium of this embodiment is used for storing the lifetime prediction apparatus 5 for the liver cancer patient, and when being executed by the processor 62, the method for assessing bone age of the target detection and the convolution transformation of the first embodiment is implemented.
It should be noted that the embodiments of the present invention have been described in terms of preferred embodiments, and not by way of limitation, and that those skilled in the art can make modifications and variations of the embodiments described above without departing from the spirit of the invention.
Claims (10)
1. A method for predicting the survival time of a liver cancer patient, which is characterized by comprising the following steps:
acquiring historical liver images of a plurality of patients, preprocessing each liver image, and acquiring a data set containing tumor images; wherein the historical liver images comprise liver images of at least three modalities; the data set comprises a plurality of data packets, each data packet comprises a plurality of tumor images of each modality of a certain patient, and a unique life cycle label is associated with each data packet;
establishing an initial prediction model, wherein the initial prediction model comprises a feature extraction module, a transform module and a classifier, and is trained by adopting the data set;
acquiring any data packet in a data set, and in the initial prediction model, respectively performing multi-mode image fusion on tumor images in the data packet after feature extraction by adopting a feature extraction module so as to output intermediate feature data containing all features in each tumor image; performing attention mechanism learning by adopting a transformer module based on the intermediate characteristic data and the pathological information of the patient corresponding to the data packet; outputting a life cycle prediction result by adopting a classifier; training is completed to obtain a target prediction model;
and acquiring a liver image of the target patient, preprocessing the liver image, inputting the preprocessed liver image into the target prediction model, and generating a target life cycle prediction result.
2. The lifetime prediction method of claim 1, wherein the performing multi-modal image fusion on the tumor images in the data packet after feature extraction by using the feature extraction module to output intermediate feature data including all features of each tumor image comprises:
in a feature extraction module, feature extraction is carried out on each tumor image by respectively adopting a convolution network based on a plurality of tumor images of each modality in the data packet, so as to obtain at least three first feature data sets comprising a plurality of first feature data, wherein each first feature data set corresponds to a tumor image of one modality;
fusing the first characteristic data in each first characteristic data set by adopting an attention convergence network to obtain at least three second characteristic data;
and performing feature combination on each second feature data by adopting a full connection layer, and outputting intermediate feature data.
3. The lifetime prediction method of claim 2, wherein the fusing the first feature data in each of the first feature data sets by using an attention convergence network to obtain at least three second feature data comprises:
for any first feature data set:
obtaining corresponding attention scores by each first characteristic data of the first characteristic data set through an attention mechanism;
converting each attention score into a probability value in a preset range by adopting a softmax function;
and multiplying each first characteristic data by the corresponding probability value respectively and then adding to obtain second characteristic data.
4. The method of claim 1, wherein the pre-processing comprises:
acquiring historical liver images of a certain patient, and adjusting each liver image according to a preset size;
for any adjusted liver image:
manually marking the liver image to obtain a tumor region, and generating a mask image;
dividing the liver image into a plurality of image slices, and overlapping each image slice with the mask image to generate a three-channel image corresponding to each image slice as a tumor image;
and collecting tumor images corresponding to the image slices, namely a plurality of tumor images of a certain modality of the patient.
5. The lifetime prediction method of claim 4, wherein the superimposing each image slice with the mask image to generate a three-channel image corresponding to each image slice comprises:
for each image slice, a three-channel image is generated in such a way that the image slice overlays the image slice overlay mask image.
6. The method of claim 1, wherein obtaining the data set comprising the tumor image further comprises:
data enhancement is performed based on the data set.
7. The method of claim 1, wherein before performing attention mechanism learning based on the intermediate feature data and the pathological information of the patient corresponding to the data packet by using a transformer module, the method comprises:
acquiring pathological information of a patient corresponding to the data packet;
respectively carrying out one-bit coding and normalization processing on the category data and the numerical data in the pathological information to generate numerical characteristics;
and embedding the digital features into a vector with the same dimension as the intermediate feature data through a multilayer perceptron, and adding the digital features and the intermediate feature data to input into a transformer module.
8. A device for predicting the survival time of a liver cancer patient, comprising:
the acquisition module is used for acquiring historical liver images of a plurality of patients, preprocessing each liver image and acquiring a data set containing tumor images; wherein the historical liver images comprise liver images of at least three modalities; the data set comprises a plurality of data packets, each data packet comprises a plurality of tumor images of each modality of a certain patient, and a unique life cycle label is associated with each data packet;
the system comprises an establishing module, a prediction module and a prediction module, wherein the establishing module is used for establishing an initial prediction model, the initial prediction model comprises a feature extraction module, a transformer module and a classifier, and the data set is adopted for training;
the processing module is used for acquiring any data packet in the data set, and in the initial prediction model, the characteristic extraction module is adopted to perform multi-mode image fusion on the tumor images in the data packet after characteristic extraction so as to output intermediate characteristic data containing all characteristics in each tumor image; performing attention mechanism learning by adopting a transformer module based on the intermediate characteristic data and the pathological information of the patient corresponding to the data packet; outputting a life cycle prediction result by adopting a classifier; training is completed to obtain a target prediction model;
and the execution module is used for acquiring the liver image of the target patient, preprocessing the liver image and inputting the preprocessed liver image into the target prediction model to generate a target life cycle prediction result.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to perform the steps of the method for predicting the survival of a liver cancer patient according to any one of claims 1 to 8.
10. A computer-readable storage medium, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the method for predicting the survival of a liver cancer patient according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211607508.9A CN115841476A (en) | 2022-12-14 | 2022-12-14 | Method, device, equipment and medium for predicting life cycle of liver cancer patient |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211607508.9A CN115841476A (en) | 2022-12-14 | 2022-12-14 | Method, device, equipment and medium for predicting life cycle of liver cancer patient |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115841476A true CN115841476A (en) | 2023-03-24 |
Family
ID=85578619
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211607508.9A Pending CN115841476A (en) | 2022-12-14 | 2022-12-14 | Method, device, equipment and medium for predicting life cycle of liver cancer patient |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115841476A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116503672A (en) * | 2023-06-26 | 2023-07-28 | 首都医科大学附属北京佑安医院 | Liver tumor classification method, system and storage medium |
-
2022
- 2022-12-14 CN CN202211607508.9A patent/CN115841476A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116503672A (en) * | 2023-06-26 | 2023-07-28 | 首都医科大学附属北京佑安医院 | Liver tumor classification method, system and storage medium |
| CN116503672B (en) * | 2023-06-26 | 2023-08-25 | 首都医科大学附属北京佑安医院 | Liver tumor classification method, system and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11610308B2 (en) | Localization and classification of abnormalities in medical images | |
| Mazurowski et al. | Deep learning in radiology: An overview of the concepts and a survey of the state of the art with focus on MRI | |
| Pinaya et al. | Unsupervised brain imaging 3D anomaly detection and segmentation with transformers | |
| Dangi et al. | A distance map regularized CNN for cardiac cine MR image segmentation | |
| Mall et al. | A comprehensive review of deep neural networks for medical image processing: Recent developments and future opportunities | |
| EP3576020A1 (en) | Methods for generating synthetic training data and for training deep learning algorithms for tumor lesion characterization, method and system for tumor lesion characterization, computer program and electronically readable storage medium | |
| Shukla et al. | AI-DRIVEN novel approach for liver cancer screening and prediction using cascaded fully convolutional neural network | |
| Zhu et al. | A 3d coarse-to-fine framework for automatic pancreas segmentation | |
| WO2021186592A1 (en) | Diagnosis assistance device and model generation device | |
| CN112529834A (en) | Spatial distribution of pathological image patterns in 3D image data | |
| CN115841476A (en) | Method, device, equipment and medium for predicting life cycle of liver cancer patient | |
| Gheorghiță et al. | Improving robustness of automatic cardiac function quantification from cine magnetic resonance imaging using synthetic image data | |
| Tang et al. | Lesion segmentation and RECIST diameter prediction via click-driven attention and dual-path connection | |
| Singh et al. | Attention-guided residual W-Net for supervised cardiac magnetic resonance imaging segmentation | |
| WO2020033594A1 (en) | Interpretable deep machine learning for clinical radiology | |
| CN115965785A (en) | Image segmentation method, device, equipment, program product and medium | |
| Sohail et al. | A modified U-net based framework for automated segmentation of Hippocampus region in brain MRI | |
| Su et al. | Res-DUnet: A small-region attentioned model for cardiac MRI-based right ventricular segmentation | |
| Alshamrani et al. | [Retracted] Automation of Cephalometrics Using Machine Learning Methods | |
| Keshavamurthy et al. | Weakly supervised pneumonia localization in chest X‐rays using generative adversarial networks | |
| Shen et al. | Nodule synthesis and selection for augmenting chest x-ray nodule detection | |
| CN115409837B (en) | Endometrial cancer CTV automatic delineation method based on multi-modal CT image | |
| US11282193B2 (en) | Systems and methods for tumor characterization | |
| Vázquez Romaguera et al. | Personalized respiratory motion model using conditional generative networks for MR-guided radiotherapy | |
| US20230196557A1 (en) | Late Gadolinium Enhancement Analysis for Magnetic Resonance Imaging |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |