CN114359666A - Multi-mode fusion lung cancer patient curative effect prediction method, system, device and medium - Google Patents
Multi-mode fusion lung cancer patient curative effect prediction method, system, device and medium Download PDFInfo
- Publication number
- CN114359666A CN114359666A CN202111628059.1A CN202111628059A CN114359666A CN 114359666 A CN114359666 A CN 114359666A CN 202111628059 A CN202111628059 A CN 202111628059A CN 114359666 A CN114359666 A CN 114359666A
- Authority
- CN
- China
- Prior art keywords
- information
- lung cancer
- feature
- cancer patient
- features
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses a multi-mode fused lung cancer patient curative effect prediction method, a system, a device and a medium, wherein the method comprises the following steps: obtaining first patient information for a first lung cancer patient; carrying out data preprocessing on the first patient information to obtain second patient information; respectively inputting the second patient information into a plurality of preset neural network models for feature extraction to obtain face appearance features, CT image features, pathological features, pulse features and clinical features of the first lung cancer patient; and performing feature fusion on the face feature, the CT image feature, the pathological feature, the pulse feature and the clinical feature according to preset weights to obtain multi-modal fusion features, and further inputting the multi-modal fusion features into a pre-trained lung cancer patient curative effect prediction model to obtain a curative effect prediction result of the first lung cancer patient. The invention improves the comprehensiveness of the curative effect prediction of the lung cancer patient and the accuracy of the prognosis survival time of the lung cancer patient, and can be widely applied to the technical field of artificial intelligence.
Description
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to a multi-mode fusion lung cancer patient curative effect prediction method, system, device and medium.
Background
One implementation mode of modernization of traditional Chinese medicine is to perform integrated feature extraction and fusion on multi-modal clinical data such as CT, pathological images, medical record data, vital sign data, traditional Chinese medicine pulse disease treatment data and the like involved in the process of preventing and treating lung cancer in traditional Chinese medicine by means of new technical means such as artificial intelligence and the like and by combining with an image omics technology, so that superior population features are mined and a curative effect prediction model is constructed.
In the prior art, modeling prediction is mostly performed for a certain data modality of a lung cancer patient, and a method which contains multi-modality information as much as possible and can accurately predict the prognosis survival time of the lung cancer patient is lacked.
Disclosure of Invention
The present invention aims to solve at least to some extent one of the technical problems existing in the prior art.
Therefore, an object of an embodiment of the present invention is to provide a multi-modal fusion lung cancer patient efficacy prediction method, which extracts and performs feature fusion on facial features, CT image features, pathological features, pulse features, and clinical features of a lung cancer patient to obtain multi-modal fusion features, where the multi-modal fusion features include feature information of various aspects of the lung cancer patient, and performs training and prediction on an efficacy prediction model of the lung cancer patient based on the multi-modal fusion features, so as to improve the comprehensiveness of efficacy prediction and the accuracy of prognosis survival of the lung cancer patient.
It is another object of the embodiments of the present invention to provide a system for predicting the efficacy of a multi-modal fused lung cancer patient.
In order to achieve the technical purpose, the technical scheme adopted by the embodiment of the invention comprises the following steps:
in a first aspect, an embodiment of the present invention provides a method for predicting an efficacy of a multi-modal fused lung cancer patient, including the following steps:
acquiring first patient information of a first lung cancer patient, wherein the first patient information comprises first facial appearance information, first CT image information, first pathological slide information, first pulse wave information and first clinical information;
carrying out data preprocessing on the first patient information to obtain second patient information;
inputting the second patient information into a plurality of preset neural network models respectively for feature extraction to obtain face appearance features, CT image features, pathological features, pulse features and clinical features of the first lung cancer patient;
and performing feature fusion on the face feature, the CT image feature, the pathological feature, the pulse feature and the clinical feature according to preset weights to obtain multi-modal fusion features, and inputting the multi-modal fusion features into a pre-trained lung cancer patient curative effect prediction model to obtain a curative effect prediction result of the first lung cancer patient.
Further, in an embodiment of the present invention, the step of obtaining first patient information of a first lung cancer patient specifically includes:
obtaining first facial information of the first lung cancer patient before treatment;
acquiring first CT image information of the first lung cancer patient before treatment;
acquiring a lung pathological slide of the first lung cancer patient, and digitally scanning the lung pathological slide by using a digital slide scanner to obtain first pathological slide information;
acquiring first pulse wave information of the first lung cancer patient before treatment;
acquiring first clinical information of the first lung cancer patient, wherein the first clinical information comprises patient age, patient gender, patient blood pressure, patient weight and treatment scheme.
Further, in an embodiment of the present invention, the second patient information includes second facial information, second CT image information, second pathology slide information, second pulse wave information, and second clinical information, and the step of performing data preprocessing on the first patient information to obtain second patient information specifically includes:
carrying out image normalization processing on the first face information to obtain second face information;
obtaining a CT image according to the first CT image information, and carrying out image normalization processing on the CT image to obtain second CT image information;
performing sliding window sampling on the first pathological slide information according to a preset window width to obtain slide sample information, and performing image normalization on the slide sample information to obtain second pathological slide information;
generating a pulse wave image according to the first pulse wave information, and carrying out image normalization on the pulse wave image to obtain second pulse wave information;
and generating text information according to the first clinical information, and carrying out unique hot coding on the text information to obtain the second clinical information.
Further, in an embodiment of the present invention, the step of inputting the second patient information into a plurality of preset neural network models respectively for feature extraction to obtain facial features, CT image features, pathological features, pulse features, and clinical features of the first lung cancer patient specifically includes:
inputting the second facial information into a preset first convolution neural network model for feature extraction to obtain the facial features;
inputting the second CT image information into a preset second convolutional neural network model for feature extraction to obtain the CT image features;
inputting the second pathological slide information into a preset third convolutional neural network model for feature extraction to obtain the pathological features;
inputting the second pulse wave information into a preset long-short term memory neural network model for feature extraction to obtain the pulse features;
and inputting the second clinical information into a preset fully-connected neural network model for feature extraction to obtain the clinical features.
Further, in an embodiment of the present invention, the step of performing feature fusion on the facial feature, the CT image feature, the pathological feature, the pulse feature and the clinical feature according to a preset weight to obtain a multi-modal fusion feature specifically includes:
vectorizing the face feature, the CT image feature, the pathological feature, the pulse feature and the clinical feature to obtain a face feature vector, a CT image feature vector, a pathological feature vector, a pulse feature vector and a clinical feature vector;
and carrying out weighted summation on the face feature vector, the CT image feature vector, the pathological feature vector, the pulse feature vector and the clinical feature vector according to preset weight to obtain the multi-modal fusion feature.
Further, in an embodiment of the present invention, the method for predicting the efficacy of lung cancer patient further includes a step of pre-training a predictive model of the efficacy of lung cancer patient, which specifically includes:
acquiring third patient information of a second lung cancer patient, and acquiring a second fusion characteristic according to the third patient information;
generating a label according to the curative effect of the second lung cancer patient, and generating a training data set according to the second fusion characteristic and the corresponding label;
and inputting the training data set into a pre-constructed lung cancer patient curative effect prediction model for model training to obtain the trained lung cancer patient curative effect prediction model.
Further, in an embodiment of the present invention, the step of inputting the training data set into a pre-constructed lung cancer patient efficacy prediction model for model training to obtain a trained lung cancer patient efficacy prediction model specifically includes:
inputting the training data set into a pre-constructed lung cancer patient curative effect prediction model to obtain a first prediction result;
determining a loss value of the lung cancer patient efficacy prediction model according to the first prediction result and the label;
updating parameters of the lung cancer patient curative effect prediction model through a back propagation algorithm according to the loss value;
and when the loss value reaches a preset first threshold value or the iteration times reaches a preset second threshold value or the test precision reaches a preset third threshold value, stopping training to obtain the trained lung cancer patient curative effect prediction model.
In a second aspect, an embodiment of the present invention provides a system for predicting an efficacy of a multi-modal fused lung cancer patient, including:
the patient information acquisition module is used for acquiring first patient information of a first lung cancer patient, wherein the first patient information comprises first face information, first CT image information, first pathological slide information, first pulse wave information and first clinical information;
the data preprocessing module is used for preprocessing the first patient information to obtain second patient information;
the feature extraction module is used for inputting the second patient information into a plurality of preset neural network models respectively for feature extraction to obtain face appearance features, CT image features, pathological features, pulse features and clinical features of the first lung cancer patient;
and the feature fusion and prediction module is used for performing feature fusion on the face feature, the CT image feature, the pathological feature, the pulse feature and the clinical feature according to preset weights to obtain multi-modal fusion features, and then inputting the multi-modal fusion features into a pre-trained lung cancer patient curative effect prediction model to obtain a curative effect prediction result of the first lung cancer patient.
In a third aspect, an embodiment of the present invention provides a device for predicting curative effect of a multi-modal fused lung cancer patient, including:
at least one processor;
at least one memory for storing at least one program;
the at least one program, when executed by the at least one processor, causes the at least one processor to implement a method of predicting efficacy of a multi-modal fused lung cancer patient as described above.
In a fourth aspect, the present invention further provides a computer-readable storage medium, in which a processor-executable program is stored, and the processor-executable program is configured to perform the method for predicting the curative effect of the multi-modal fused lung cancer patient.
Advantages and benefits of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention:
according to the embodiment of the invention, the face feature, the CT image feature, the pathological feature, the pulse feature and the clinical feature of the lung cancer patient are extracted and feature fusion is carried out to obtain the multi-mode fusion feature, the multi-mode fusion feature contains feature information of various aspects of the lung cancer patient, the training and prediction of a curative effect prediction model of the lung cancer patient are carried out based on the multi-mode fusion feature, and the comprehensiveness of curative effect prediction and the accuracy of the prognosis survival period of the lung cancer patient are improved.
Drawings
In order to more clearly illustrate the technical solution in the embodiment of the present invention, the following description is made on the drawings required to be used in the embodiment of the present invention, and it should be understood that the drawings in the following description are only for convenience and clarity of describing some embodiments in the technical solution of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a flowchart illustrating steps of a method for predicting the efficacy of a multi-modal fusion lung cancer patient according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of feature extraction and multi-modal fusion provided by an embodiment of the present invention;
FIG. 3 is a block diagram of a system for predicting the efficacy of a multi-modal fused lung cancer patient according to an embodiment of the present invention;
fig. 4 is a block diagram illustrating a device for predicting the curative effect of a multi-modal fused lung cancer patient according to an embodiment of the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided only for convenience of illustration, the order between the steps is not limited at all, and the execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.
In the description of the present invention, the meaning of a plurality is two or more, if there is a description to the first and the second for the purpose of distinguishing technical features, it is not understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or implicitly indicating the precedence of the indicated technical features. Furthermore, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art.
Referring to fig. 1, an embodiment of the present invention provides a method for predicting a curative effect of a multi-modal fused lung cancer patient, which specifically includes the following steps:
s101, first patient information of a first lung cancer patient is obtained, wherein the first patient information comprises first face volume information, first CT image information, first pathological slide information, first pulse wave information and first clinical information.
Specifically, the embodiment of the invention models the feature information of five aspects of the lung cancer patient, including face appearance information, CT image information, pathological slide information, pulse wave information and clinical information. Step S101 specifically includes the following steps:
s1011, obtaining first facial information of a first lung cancer patient before treatment;
s1012, acquiring first CT image information of a first lung cancer patient before treatment;
s1013, acquiring a lung pathological slide of a first lung cancer patient, and digitally scanning the lung pathological slide by using a digital slide scanner to obtain first pathological slide information;
s1014, acquiring first pulse wave information of a first lung cancer patient before treatment;
s1015, acquiring first clinical information of the first lung cancer patient, wherein the first clinical information comprises the age of the patient, the sex of the patient, the blood pressure of the patient, the weight of the patient and the treatment scheme.
Specifically, (1) face information: collecting face-to-face photos of a patient within two weeks before treatment, wherein the resolution is more than 256 and 256, normal illumination is required, a crown is not required, glasses are removed, and other requirements are not required; (2) CT image information: collecting DICOM data of lung CT slices of a patient within two weeks before treatment, wherein the model, the layer thickness, the scanning resolution and the like of CT equipment are not required; (3) pathological slide information: digitally scanning pathological slides of the lung of a patient by using a digital slide scanner under an objective lens 40X to generate digital pathology (WSI), wherein generally one WSI has resolution of one hundred thousand and one hundred thousand, and the size of the WSI is generally larger than 1 GB; (4) pulse wave information: measuring the pulse wave at any wrist of the patient in a calm state for 30 seconds to obtain pulse wave information; (5) clinical information: routine information of the patient's hospitalization is collected, including age, sex, blood pressure, weight, and treatment regimen, etc.
S102, data preprocessing is carried out on the first patient information to obtain second patient information.
Specifically, the second patient information includes second face information, second CT image information, second pathology slide information, second pulse wave information, and second clinical information. Step S102 specifically includes the following steps:
s1021, carrying out image normalization processing on the first face information to obtain second face information;
s1022, obtaining a CT image according to the first CT image information, and performing image normalization processing on the CT image to obtain second CT image information;
s1023, sliding window sampling is carried out on the first pathological slide information according to a preset window width to obtain slide sample information, and image normalization is carried out on the slide sample information to obtain second pathological slide information;
s1024, generating a pulse wave image according to the first pulse wave information, and carrying out image normalization on the pulse wave image to obtain second pulse wave information;
and S1025, generating text information according to the first clinical information, and carrying out one-hot coding on the text information to obtain second clinical information.
Specifically, (1) face information: firstly, adjusting the resolution of a face image to 256 × 256 pixels, then carrying out image normalization, compressing the pixel value from (0-255) to (0-1), and then storing the pixel value as a Python Numpy array; (2) CT image information: opening CT images through RadiAnt DICOM Viewer software, storing the CT images into jpg or other image formats one by one, normalizing the stored images, uniformly adjusting the resolution to 512 x 512 pixels, and storing the pixels as Python Numpy arrays; (3) pathological slide information: sampling WSI (digital pathological slide) by sliding windows through open-source openslide software according to a window width of 256 × 256, and storing the WSI as a Python Numpy array after normalization processing; (4) pulse wave information: reading by using a Padas library of Python, then carrying out normalization processing, and storing as a Python Numpy array; (5) medical record data: and carrying out classification statistics on the character data, then carrying out one-hot coding, and storing the character data as a Python Numpy array.
S103, respectively inputting the second patient information into a plurality of preset neural network models for feature extraction, and obtaining the face feature, the CT image feature, the pathological feature, the pulse feature and the clinical feature of the first lung cancer patient.
Specifically, the embodiment of the invention presets a plurality of neural network models for extracting various features. Step S103 specifically includes the following steps:
s1031, inputting the second facial information into a preset first convolution neural network model for feature extraction to obtain facial features;
s1032, inputting the second CT image information into a preset second convolutional neural network model for feature extraction to obtain CT image features;
s1033, inputting the second pathological slide information into a preset third convolutional neural network model for feature extraction to obtain pathological features;
s1034, inputting the second pulse wave information into a preset long-short term memory neural network model for feature extraction to obtain pulse features;
and S1035, inputting the second clinical information into a preset fully-connected neural network model for feature extraction to obtain clinical features.
Specifically, (1) for the face image, Tensorflow2.0 is adopted, a convolutional neural network architecture model EfficientNet library is introduced into the input, and then a corresponding EfficientNet weight is introduced into the model, wherein the weight is obtained by fully training an ImageNet data set and is used for CNN migration learning. Next, since the last layer of the model carrying the weight is 1000 output neurons, the embodiment of the present invention removes the last layer, adds a fully connected layer containing only 1 neuron as an output feature layer, does not add any activation function, sets the learning rate to 0.0008, sets the loss function to mean square error, and sets the optimizer to Adam.
(2) For the CT image, Tensorflow2.0 is adopted, a convolutional neural network architecture model EfficientNet library is introduced into the input, and then the corresponding EfficientNet weight is introduced into the model, wherein the weight is obtained by fully training an ImageNet data set and is used for CNN migration learning. Next, since the last layer of the model carrying the weight is 1000 output neurons, the embodiment of the present invention removes the last layer, adds a fully connected layer containing only 1 neuron as an output feature layer, does not add any activation function, sets the learning rate to 0.0008, sets the loss function to mean square error, and sets the optimizer to Adam.
(3) For pathological images, a Tensorflow2.0 is adopted, a convolutional neural network model EfficientNet library is introduced into input, and then a corresponding EfficientNet weight is introduced into the model, wherein the weight is obtained by fully training an ImageNet data set and is used for CNN migration learning. Next, since the last layer of the model carrying the weight is 1000 output neurons, the embodiment of the present invention removes the last layer, adds a fully connected layer containing only 1 neuron as an output feature layer, does not add any activation function, sets the learning rate to 0.0008, sets the loss function to mean square error, and sets the optimizer to Adam.
(4) For pulse wave data, Tensorflow2.0 is adopted, a long-term and short-term memory neural network (or other recurrent neural networks) is introduced into an input, a layer containing 50 neurons is arranged, the learning rate is 0.0008, the loss function is mean square error, the optimizer is Adam, a full-connection layer containing only 1 neuron is added to serve as an output characteristic layer, no activation function is added to the layer, the learning rate is 0.0008, the loss function is mean square error, and the optimizer is Adam.
(5) For clinical information, Tensorflow2.0 is adopted, a layer of fully-connected network is introduced into input, the fully-connected network comprises 20 neurons, an activation function is RELU, a fully-connected layer only comprising 1 neuron is added to serve as an output characteristic layer, no activation function is added to the layer, the learning rate is set to be 0.0008, a loss function is mean square error, and an optimizer is Adam.
As shown in fig. 2, after the feature extraction processing, the face feature F1, the CT image feature F2, the pathological feature F3, the pulse feature F4 and the clinical feature F5 of the lung cancer patient can be obtained, and then multi-modal feature fusion can be performed.
And S104, performing feature fusion on the face feature, the CT image feature, the pathological feature, the pulse feature and the clinical feature according to preset weights to obtain multi-mode fusion features, and inputting the multi-mode fusion features into a pre-trained lung cancer patient curative effect prediction model to obtain a curative effect prediction result of the first lung cancer patient.
Specifically, in the embodiment of the present invention, the data of five modalities are input into the corresponding models to obtain five features (F1 to F5), and then the five features are fused according to the preset weights to obtain multi-modal fusion features, so that the multi-modal fusion features can be input into the trained lung cancer patient efficacy prediction model to predict the prognosis survival time.
As a further optional implementation manner, the method includes a step of performing feature fusion on the face feature, the CT image feature, the pathological feature, the pulse feature and the clinical feature according to a preset weight to obtain a multi-modal fusion feature, which specifically includes:
a1, performing vectorization processing on the facial features, the CT image features, the pathological features, the pulse features and the clinical features to obtain facial feature vectors, CT image feature vectors, pathological feature vectors, pulse feature vectors and clinical feature vectors;
and A2, carrying out weighted summation on the face feature vector, the CT image feature vector, the pathological feature vector, the pulse feature vector and the clinical feature vector according to preset weights to obtain the multi-modal fusion feature.
As a further alternative, the method for predicting the curative effect of a lung cancer patient further includes the step of pre-training a curative effect prediction model of the lung cancer patient, which specifically includes:
b1, obtaining third patient information of a second lung cancer patient, and obtaining a second fusion characteristic according to the third patient information;
b2, generating a label according to the curative effect of the second lung cancer patient, and generating a training data set according to the second fusion characteristic and the corresponding label;
and B3, inputting the training data set into a pre-constructed lung cancer patient curative effect prediction model for model training to obtain the trained lung cancer patient curative effect prediction model.
Specifically, at the stage of training a curative effect prediction model of a lung cancer patient, obtaining patient information of a second lung cancer patient with known curative effect (i.e., prognosis survival time), and obtaining a second fusion characteristic by a method similar to the aforementioned characteristic extraction and fusion, which is not repeated again in the specific process; meanwhile, determining a label according to the actual curative effect of the second lung cancer patient, wherein the label corresponds to the second fusion characteristic one by one, and a training data set can be constructed according to the label and the second fusion characteristic; and finally, inputting the training data set into an initialized lung cancer patient curative effect prediction model for model training.
As a further optional implementation manner, the step B3 of inputting the training data set into a pre-constructed lung cancer patient efficacy prediction model for model training to obtain a trained lung cancer patient efficacy prediction model specifically includes:
b31, inputting the training data set into a pre-constructed lung cancer patient curative effect prediction model to obtain a first prediction result;
b32, determining the loss value of the lung cancer patient curative effect prediction model according to the first prediction result and the label;
b33, updating parameters of the lung cancer patient curative effect prediction model through a back propagation algorithm according to the loss value;
and B34, stopping training when the loss value reaches a preset first threshold value or the iteration times reaches a preset second threshold value or the test precision reaches a preset third threshold value, and obtaining the trained lung cancer patient curative effect prediction model.
Specifically, the lung cancer patient curative effect prediction model provided by the embodiment of the invention can be built based on a convolutional neural network, the data in the training data set is input into the initialized lung cancer patient curative effect prediction model, the prediction result output by the model can be obtained, and the accuracy of the lung cancer patient curative effect prediction model can be evaluated by using the prediction result and the label, so that the parameters of the model are updated. For the lung cancer patient efficacy prediction model, the accuracy of the model prediction result can be measured by a Loss Function (Loss Function), the Loss Function is defined on a single training data and is used for measuring the prediction error of the training data, and specifically, the Loss value of the training data is determined by the label of the single training data and the prediction result of the model on the training data. In actual training, a training data set has many training data, so a Cost Function (Cost Function) is generally adopted to measure the overall error of the training data set, and the Cost Function is defined on the whole training data set and is used for calculating the average value of prediction errors of all the training data, so that the prediction effect of the model can be measured better. For a general machine learning model, based on the cost function, and a regularization term for measuring the complexity of the model, the regularization term can be used as a training objective function, and based on the objective function, the loss value of the whole training data set can be obtained. There are many kinds of commonly used loss functions, such as 0-1 loss function, square loss function, absolute loss function, logarithmic loss function, cross entropy loss function, etc. all can be used as the loss function of the machine learning model, and are not described one by one here. In the embodiment of the invention, a loss function can be selected from the loss functions to determine the loss value of the training. And updating the parameters of the model by adopting a back propagation algorithm based on the trained loss value, and iterating for several rounds to obtain the trained point cloud tower identification model. Specifically, the number of iteration rounds may be preset, or training may be considered complete when the test set meets the accuracy requirement.
The method steps of the embodiments of the present invention are described above. It should be appreciated that in the prior art, no artificial intelligence model capable of predicting the prognosis survival period of a lung cancer patient based on multi-modal data exists, and particularly, special face diagnosis data of traditional Chinese medicine can be processed.
According to the embodiment of the invention, the face feature, the CT image feature, the pathological feature, the pulse feature and the clinical feature of the lung cancer patient are extracted and feature fusion is carried out to obtain the multi-mode fusion feature, the multi-mode fusion feature contains feature information of various aspects of the lung cancer patient, the training and prediction of a curative effect prediction model of the lung cancer patient are carried out based on the multi-mode fusion feature, and the comprehensiveness of curative effect prediction and the accuracy of the prognosis survival period of the lung cancer patient are improved.
In addition, the embodiment of the invention also has the following advantages:
(1) the model of the embodiment of the invention supports the mode selection according to clinical actual data, can provide complete 5 modes, can also provide only a few of the mode data, and even only provides any one mode, and can carry out modeling prediction in a 'buffet meal' mode.
(2) The later application of the embodiment of the invention has a prompting function for assisting a clinician in selecting dominant population for revealing macroscopic characteristics of a lung cancer patient and finding out a treatment scheme, and belongs to a novel marker.
(3) According to the embodiment of the invention, the curative effect can be predicted only by providing any one or more data of CT, pathology, face and face, pulse, clinical information and the like of routine examination, so that the method is convenient to use and popularize, and the burden of a patient and a medical institution is not required to be increased.
Referring to fig. 3, an embodiment of the present invention provides a system for predicting the curative effect of a multi-modal fused lung cancer patient, including:
the patient information acquisition module is used for acquiring first patient information of a first lung cancer patient, wherein the first patient information comprises first face information, first CT image information, first pathological slide information, first pulse wave information and first clinical information;
the data preprocessing module is used for preprocessing the first patient information to obtain second patient information;
the characteristic extraction module is used for respectively inputting the information of the second patient into a plurality of preset neural network models for characteristic extraction to obtain the face appearance characteristic, the CT image characteristic, the pathological characteristic, the pulse characteristic and the clinical characteristic of the first lung cancer patient;
and the feature fusion and prediction module is used for performing feature fusion on the face feature, the CT image feature, the pathological feature, the pulse feature and the clinical feature according to preset weights to obtain multi-mode fusion features, and further inputting the multi-mode fusion features into a pre-trained lung cancer patient curative effect prediction model to obtain a curative effect prediction result of a first lung cancer patient.
The contents in the above method embodiments are all applicable to the present system embodiment, the functions specifically implemented by the present system embodiment are the same as those in the above method embodiment, and the beneficial effects achieved by the present system embodiment are also the same as those achieved by the above method embodiment.
Referring to fig. 4, an embodiment of the present invention provides a multi-modal fused device for predicting a curative effect of a lung cancer patient, including:
at least one processor;
at least one memory for storing at least one program;
when the at least one program is executed by the at least one processor, the at least one processor may implement the method for predicting the efficacy of a multi-modal fused lung cancer patient.
The contents in the above method embodiments are all applicable to the present apparatus embodiment, the functions specifically implemented by the present apparatus embodiment are the same as those in the above method embodiments, and the advantageous effects achieved by the present apparatus embodiment are also the same as those achieved by the above method embodiments.
Embodiments of the present invention also provide a computer-readable storage medium, in which a program executable by a processor is stored, and the program executable by the processor is used for executing the multi-modal fusion lung cancer patient efficacy prediction method.
The computer-readable storage medium of the embodiment of the invention can execute the multi-modal fusion lung cancer patient curative effect prediction method provided by the embodiment of the method of the invention, can execute any combination of the implementation steps of the embodiment of the method, and has corresponding functions and beneficial effects of the method.
The embodiment of the invention also discloses a computer program product or a computer program, which comprises computer instructions, and the computer instructions are stored in a computer readable storage medium. The computer instructions may be read by a processor of a computer device from a computer-readable storage medium, and executed by the processor to cause the computer device to perform the method illustrated in fig. 1.
In alternative embodiments, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flow charts of the present invention are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are performed independently.
Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the above-described functions and/or features may be integrated in a single physical device and/or software module, or one or more of the functions and/or features may be implemented in a separate physical device or software module. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary for an understanding of the present invention. Rather, the actual implementation of the various functional modules in the apparatus disclosed herein will be understood within the ordinary skill of an engineer, given the nature, function, and internal relationship of the modules. Accordingly, those skilled in the art can, using ordinary skill, practice the invention as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative of and not intended to limit the scope of the invention, which is defined by the appended claims and their full scope of equivalents.
The above functions, if implemented in the form of software functional units and sold or used as a separate product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the above method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Further, the computer readable medium could even be paper or another suitable medium upon which the above described program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
In the foregoing description of the specification, reference to the description of "one embodiment/example," "another embodiment/example," or "certain embodiments/examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that: various changes, modifications, substitutions and alterations can be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
While the preferred embodiments of the present invention have been illustrated and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. A multi-modal fusion lung cancer patient curative effect prediction method is characterized by comprising the following steps:
acquiring first patient information of a first lung cancer patient, wherein the first patient information comprises first facial appearance information, first CT image information, first pathological slide information, first pulse wave information and first clinical information;
carrying out data preprocessing on the first patient information to obtain second patient information;
inputting the second patient information into a plurality of preset neural network models respectively for feature extraction to obtain face appearance features, CT image features, pathological features, pulse features and clinical features of the first lung cancer patient;
and performing feature fusion on the face feature, the CT image feature, the pathological feature, the pulse feature and the clinical feature according to preset weights to obtain multi-modal fusion features, and inputting the multi-modal fusion features into a pre-trained lung cancer patient curative effect prediction model to obtain a curative effect prediction result of the first lung cancer patient.
2. The method of claim 1, wherein the step of obtaining first patient information of a first lung cancer patient comprises:
obtaining first facial information of the first lung cancer patient before treatment;
acquiring first CT image information of the first lung cancer patient before treatment;
acquiring a lung pathological slide of the first lung cancer patient, and digitally scanning the lung pathological slide by using a digital slide scanner to obtain first pathological slide information;
acquiring first pulse wave information of the first lung cancer patient before treatment;
acquiring first clinical information of the first lung cancer patient, wherein the first clinical information comprises patient age, patient gender, patient blood pressure, patient weight and treatment scheme.
3. The method as claimed in claim 2, wherein the second patient information includes second facial information, second CT image information, second pathology slide information, second pulse wave information and second clinical information, and the step of performing data preprocessing on the first patient information to obtain second patient information includes:
carrying out image normalization processing on the first face information to obtain second face information;
obtaining a CT image according to the first CT image information, and carrying out image normalization processing on the CT image to obtain second CT image information;
performing sliding window sampling on the first pathological slide information according to a preset window width to obtain slide sample information, and performing image normalization on the slide sample information to obtain second pathological slide information;
generating a pulse wave image according to the first pulse wave information, and carrying out image normalization on the pulse wave image to obtain second pulse wave information;
and generating text information according to the first clinical information, and carrying out unique hot coding on the text information to obtain the second clinical information.
4. The method according to claim 3, wherein the step of inputting the second patient information into a plurality of preset neural network models for feature extraction to obtain facial features, CT image features, pathological features, pulse features and clinical features of the first lung cancer patient specifically comprises:
inputting the second facial information into a preset first convolution neural network model for feature extraction to obtain the facial features;
inputting the second CT image information into a preset second convolutional neural network model for feature extraction to obtain the CT image features;
inputting the second pathological slide information into a preset third convolutional neural network model for feature extraction to obtain the pathological features;
inputting the second pulse wave information into a preset long-short term memory neural network model for feature extraction to obtain the pulse features;
and inputting the second clinical information into a preset fully-connected neural network model for feature extraction to obtain the clinical features.
5. The method according to claim 1, wherein the step of feature-fusing the facial features, the CT image features, the pathological features, the pulse features and the clinical features according to a predetermined weight to obtain a multi-modal fused feature comprises:
vectorizing the face feature, the CT image feature, the pathological feature, the pulse feature and the clinical feature to obtain a face feature vector, a CT image feature vector, a pathological feature vector, a pulse feature vector and a clinical feature vector;
and carrying out weighted summation on the face feature vector, the CT image feature vector, the pathological feature vector, the pulse feature vector and the clinical feature vector according to preset weight to obtain the multi-modal fusion feature.
6. The method for predicting the curative effect of a lung cancer patient through multi-modal fusion according to any one of claims 1 to 5, wherein the method for predicting the curative effect of the lung cancer patient further comprises the step of pre-training a curative effect prediction model of the lung cancer patient, which specifically comprises:
acquiring third patient information of a second lung cancer patient, and acquiring a second fusion characteristic according to the third patient information;
generating a label according to the curative effect of the second lung cancer patient, and generating a training data set according to the second fusion characteristic and the corresponding label;
and inputting the training data set into a pre-constructed lung cancer patient curative effect prediction model for model training to obtain the trained lung cancer patient curative effect prediction model.
7. The method according to claim 6, wherein the step of inputting the training data set into a pre-constructed lung cancer patient efficacy prediction model for model training to obtain a trained lung cancer patient efficacy prediction model specifically comprises:
inputting the training data set into a pre-constructed lung cancer patient curative effect prediction model to obtain a first prediction result;
determining a loss value of the lung cancer patient efficacy prediction model according to the first prediction result and the label;
updating parameters of the lung cancer patient curative effect prediction model through a back propagation algorithm according to the loss value;
and when the loss value reaches a preset first threshold value or the iteration times reaches a preset second threshold value or the test precision reaches a preset third threshold value, stopping training to obtain the trained lung cancer patient curative effect prediction model.
8. A system for predicting the efficacy of a multi-modality fused lung cancer patient, comprising:
the patient information acquisition module is used for acquiring first patient information of a first lung cancer patient, wherein the first patient information comprises first face information, first CT image information, first pathological slide information, first pulse wave information and first clinical information;
the data preprocessing module is used for preprocessing the first patient information to obtain second patient information;
the feature extraction module is used for inputting the second patient information into a plurality of preset neural network models respectively for feature extraction to obtain face appearance features, CT image features, pathological features, pulse features and clinical features of the first lung cancer patient;
and the feature fusion and prediction module is used for performing feature fusion on the face feature, the CT image feature, the pathological feature, the pulse feature and the clinical feature according to preset weights to obtain multi-modal fusion features, and then inputting the multi-modal fusion features into a pre-trained lung cancer patient curative effect prediction model to obtain a curative effect prediction result of the first lung cancer patient.
9. A multi-modality fused lung cancer patient efficacy prediction device, comprising:
at least one processor;
at least one memory for storing at least one program;
when executed by the at least one processor, cause the at least one processor to implement a method of multi-modal fused lung cancer patient efficacy prediction according to any one of claims 1-7.
10. A computer readable storage medium having stored therein a processor executable program, wherein the processor executable program when executed by a processor is for performing a method of multi-modal fused lung cancer patient efficacy prediction as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111628059.1A CN114359666A (en) | 2021-12-28 | 2021-12-28 | Multi-mode fusion lung cancer patient curative effect prediction method, system, device and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111628059.1A CN114359666A (en) | 2021-12-28 | 2021-12-28 | Multi-mode fusion lung cancer patient curative effect prediction method, system, device and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114359666A true CN114359666A (en) | 2022-04-15 |
Family
ID=81103793
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111628059.1A Pending CN114359666A (en) | 2021-12-28 | 2021-12-28 | Multi-mode fusion lung cancer patient curative effect prediction method, system, device and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114359666A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114927216A (en) * | 2022-04-29 | 2022-08-19 | 中南大学湘雅医院 | Method and system for predicting treatment effect of PD-1 of melanoma patient based on artificial intelligence |
| CN115132354A (en) * | 2022-07-06 | 2022-09-30 | 哈尔滨医科大学 | Patient type identification method and device, electronic equipment and storage medium |
| CN115131642A (en) * | 2022-08-30 | 2022-09-30 | 之江实验室 | Multi-modal medical data fusion system based on multi-view subspace clustering |
| CN115171888A (en) * | 2022-08-04 | 2022-10-11 | 云南师范大学 | Method, system, electronic device and storage medium for classifying SMPLC and IM |
| CN115410686A (en) * | 2022-08-22 | 2022-11-29 | 哈尔滨医科大学 | Method and device for selecting conversion treatment scheme, electronic equipment and storage medium |
| CN115440386A (en) * | 2022-09-30 | 2022-12-06 | 中国医学科学院北京协和医院 | Prediction of immunotherapy effect of advanced cancer patients based on weighted multi-lesion imagemics features |
| CN115512833A (en) * | 2022-11-22 | 2022-12-23 | 四川省医学科学院·四川省人民医院 | Establishment of long-term cost effectiveness prediction model for lung cancer patient based on deep learning Markov framework |
| CN115861303A (en) * | 2023-02-16 | 2023-03-28 | 四川大学 | EGFR gene mutation detection method and system based on lung CT image |
| CN116705297A (en) * | 2023-06-07 | 2023-09-05 | 广州华科盈医疗科技有限公司 | Carotid artery detector based on multiple information processing |
| CN116994745A (en) * | 2023-09-27 | 2023-11-03 | 中山大学附属第六医院 | Multi-mode model-based cancer patient prognosis prediction method and device |
-
2021
- 2021-12-28 CN CN202111628059.1A patent/CN114359666A/en active Pending
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114927216A (en) * | 2022-04-29 | 2022-08-19 | 中南大学湘雅医院 | Method and system for predicting treatment effect of PD-1 of melanoma patient based on artificial intelligence |
| CN115132354A (en) * | 2022-07-06 | 2022-09-30 | 哈尔滨医科大学 | Patient type identification method and device, electronic equipment and storage medium |
| CN115171888A (en) * | 2022-08-04 | 2022-10-11 | 云南师范大学 | Method, system, electronic device and storage medium for classifying SMPLC and IM |
| CN115410686A (en) * | 2022-08-22 | 2022-11-29 | 哈尔滨医科大学 | Method and device for selecting conversion treatment scheme, electronic equipment and storage medium |
| CN115131642A (en) * | 2022-08-30 | 2022-09-30 | 之江实验室 | Multi-modal medical data fusion system based on multi-view subspace clustering |
| CN115440386A (en) * | 2022-09-30 | 2022-12-06 | 中国医学科学院北京协和医院 | Prediction of immunotherapy effect of advanced cancer patients based on weighted multi-lesion imagemics features |
| CN115512833A (en) * | 2022-11-22 | 2022-12-23 | 四川省医学科学院·四川省人民医院 | Establishment of long-term cost effectiveness prediction model for lung cancer patient based on deep learning Markov framework |
| CN115512833B (en) * | 2022-11-22 | 2023-03-24 | 四川省医学科学院·四川省人民医院 | Establishment of long-term cost effectiveness prediction system for lung cancer patient based on deep learning Markov framework |
| CN115861303A (en) * | 2023-02-16 | 2023-03-28 | 四川大学 | EGFR gene mutation detection method and system based on lung CT image |
| CN115861303B (en) * | 2023-02-16 | 2023-04-28 | 四川大学 | EGFR gene mutation detection method and system based on lung CT image |
| CN116705297A (en) * | 2023-06-07 | 2023-09-05 | 广州华科盈医疗科技有限公司 | Carotid artery detector based on multiple information processing |
| CN116705297B (en) * | 2023-06-07 | 2024-01-23 | 广州华科盈医疗科技有限公司 | Carotid artery detector based on multiple information processing |
| CN116994745A (en) * | 2023-09-27 | 2023-11-03 | 中山大学附属第六医院 | Multi-mode model-based cancer patient prognosis prediction method and device |
| CN116994745B (en) * | 2023-09-27 | 2024-02-13 | 中山大学附属第六医院 | Multi-mode model-based cancer patient prognosis prediction method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114359666A (en) | Multi-mode fusion lung cancer patient curative effect prediction method, system, device and medium | |
| Zhao et al. | Supervised segmentation of un-annotated retinal fundus images by synthesis | |
| CN109919928B (en) | Medical image detection method and device and storage medium | |
| WO2016192612A1 (en) | Method for analysing medical treatment data based on deep learning, and intelligent analyser thereof | |
| JP6793774B2 (en) | Systems and methods for classifying multidimensional time series of parameters | |
| Sun et al. | Intelligent analysis of medical big data based on deep learning | |
| JP2022550094A (en) | Learning method and device for making AI model specialized for user institution | |
| CN113077434B (en) | Method, device and storage medium for lung cancer identification based on multi-modal information | |
| Dezaki et al. | Deep residual recurrent neural networks for characterisation of cardiac cycle phase from echocardiograms | |
| CN113256592B (en) | Training method, system and device of image feature extraction model | |
| Zheng et al. | Diagnostic regions attention network (DRA-Net) for histopathology WSI recommendation and retrieval | |
| CN112164448A (en) | Training method, prediction system, method and medium of immunotherapy efficacy prediction model | |
| Shirazi et al. | Deep learning in the healthcare industry: theory and applications | |
| Pastor-Serrano et al. | A semi-supervised autoencoder framework for joint generation and classification of breathing | |
| Ju et al. | 3D-CNN-SPP: A patient risk prediction system from electronic health records via 3D CNN and spatial pyramid pooling | |
| CN113693611B (en) | Machine learning-based electrocardiogram data classification method and device | |
| Bajor et al. | Embedding complexity in the data representation instead of in the model: A case study using heterogeneous medical data | |
| CN113850796A (en) | Lung disease identification method and device based on CT data, medium and electronic equipment | |
| CN117274185A (en) | Detection method, detection model product, electronic device, and computer storage medium | |
| CN116843995A (en) | Method and device for constructing cytographic pre-training model | |
| Nguyen-Duc et al. | Deep EHR spotlight: a framework and mechanism to highlight events in electronic health records for explainable predictions | |
| CN115547502A (en) | Hemodialysis patient risk prediction device based on time sequence data | |
| CN115458161A (en) | Breast cancer progression analysis method, device, apparatus, and medium | |
| CN114897057A (en) | Intelligent yin deficiency syndrome diagnosis system, method and equipment based on multi-mode data fusion | |
| CN113077901A (en) | Electronic medical record analysis device and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |