CN118553407A - Lung tumor diagnosis and prediction system based on multi-mode deep learning - Google Patents

Abstract

The application provides a lung tumor diagnosis and prediction system based on multi-mode deep learning, which comprises the following steps: an input unit that acquires at least two of lung tumor pathology data, transcription data, and clinical prognosis data of a patient; a preprocessing unit for respectively preprocessing the pathological data, the transcription data and the clinical prognosis data; the layering processing unit inputs the data into the multi-mode prediction model, and predicts the obtained result characteristics; the fusion processing unit predicts and obtains a tumor prediction result through a multi-mode prediction model; and the survival time prediction unit inputs the pathology input image into a survival time prediction model to obtain a final patient survival time prediction result. The application utilizes the deep learning technology to effectively fuse the lung tumor pathology, transcriptome and clinical prognosis data, improves the diagnosis accuracy and the model interpretability, and the survival time prediction can provide a prediction result for doctors in real time and can be effectively fused into the current lung tumor clinical diagnosis and treatment flow.

Description

A lung tumor diagnosis and prediction system based on multimodal deep learning

Technical Field

The present application relates to the field of tumor diagnosis, and in particular to a lung tumor diagnosis and prediction system based on multimodal deep learning.

Background Art

With the advancement of medical imaging technology and genomics, the diagnosis and prognosis prediction of lung tumors are increasingly dependent on multi-dimensional data analysis. Although medical images such as CT and MRI provide important information on the morphology of lung tumors, it is still challenging to distinguish between benign and malignant tumors and predict prognosis only through imaging data. Clinical pathological diagnosis is still used as the gold standard for the diagnosis of benign and malignant lung tumors. In recent years, deep learning has made significant breakthroughs in the field of medical image analysis, but in the field of pathological images, research on the classification of benign and malignant lung tumors is still insufficient, mainly due to the complex tissue structure and texture features of pathological images, as well as the subtle differences between different pathological types. Therefore, combining data from multiple modalities for comprehensive analysis has become a key research direction. In addition to medical imaging omics, omics data such as pathological omics and transcriptomics can also reveal the microstructure and gene expression information in tumors, improving the diagnosis and prognosis prediction effects.

At present, deep learning has been initially applied in the diagnosis and prognosis prediction of lung tumors, such as convolutional neural networks (CNN) for CT image feature extraction and classification, and recurrent neural networks (RNN) for time series medical data modeling to predict lung tumor prognosis. However, these solutions mainly focus on single-modality data and fail to fully utilize the correlation and complementarity between multimodal data. Therefore, there is still room for improvement in the accurate identification of benign and malignant lung tumors and prognosis prediction.

Although some technologies attempt to integrate multimodal data, the heterogeneity and complexity of the data make the integration effect unsatisfactory, limiting its application in the diagnosis and treatment of lung tumors. In addition, due to the difficulty of data collection and annotation, the existing related models perform poorly on new data, limiting their generalization ability and practical application value. In addition, many existing deep learning models have not been effectively integrated into the existing clinical workflow, increasing the workload of doctors and limiting the practical application effect of the models.

Summary of the invention

In view of this, the present application proposes a lung tumor diagnosis and prediction system based on multimodal deep learning to solve at least one of the technical problems described in the above background technology. The specific solution is as follows:

A lung tumor diagnosis and prediction system based on multimodal deep learning, comprising:

An input unit, used to obtain at least two data of a patient including pathological data related to lung tumor pathological images, transcription data related to RNA transcription of lung tumor tissue, and clinical prognosis data related to clinical treatment of lung tumor patients;

A preprocessing unit, configured to perform a preset first preprocessing on the pathological data to obtain a pathological input image, perform a preset second preprocessing on the transcription data to obtain transcription input data, and perform a third preprocessing on the clinical prognosis data to obtain clinical input data;

a hierarchical processing unit, configured to input the pathological input image, the transcription input data and the clinical input data into a multimodal prediction model integrated with a pathological model, a transcription model and a clinical model, so that the pathological model outputs n pathological result features according to the pathological input image, the transcription model outputs n pathological result features according to the transcription input data, and the clinical model outputs n clinical result features according to the clinical input data;

A fusion processing unit, used for fusing the n pathological result features, the n transcription result features and the n clinical result features through the multimodal prediction model to obtain a final tumor prediction result;

The survival time prediction unit is used to input the pathological input image into a preset survival time prediction model to obtain the final predicted survival time of the patient.

Among them, the pathological result characteristics, the transcription result characteristics, the clinical result characteristics and the tumor prediction results all involve predicted tumor types and their probabilities, and n is an integer not less than 2.

In some specific embodiments, the expression of the multimodal prediction model is:

in, represents the output of the multimodal prediction model, W _class is the weight matrix of the output of the pathological model, the transcriptional model and the clinical model, Batch represents the batch normalization layer, Dropout represents the Dropout function, ReLU represents the ReLU activation function, W _fusion is the weight matrix of the fusion layer, b _fusion is the bias term of the fusion layer, f ₁ , f ₂ and f ₃ represent the pathological result feature, the transcriptional result feature and the clinical result feature respectively; μ _batch represents the average value of the batch; represents the variance of the batch; ∈ is a small scalar for numerical stability, γ and β are trainable parameters of the BatchNorm layer; x is the output of the fusion layer and the previous operation (ReLU, Dropout).

In some specific embodiments, in the multimodal prediction model:

The n pathological result features, n transcription result features, and n clinical result features are fused into the low-dimensional space as input features through the fusion layer to obtain fusion features;

Applying ReLU activation function to the fused features introduces nonlinearity;

Apply the Dropout function to the features after ReLU activation to discard a preset proportion of input features to reduce overfitting;

Apply batch normalization layer to the features after Dropout to normalize the features;

The fusion features are mapped to the final tumor category through a preset fully connected layer to obtain the final tumor prediction result.

In some specific embodiments, the first preprocessing includes brightness adjustment, contrast adjustment, saturation adjustment, gamma correction and pixel standardization of the image; the second preprocessing includes standardization of transcription data and extraction of genes with non-zero expression copy number; the third preprocessing includes missing value processing and outlier processing of clinical prognosis data.

In some specific embodiments, the second preprocessing further includes: extracting genes with non-zero expression copy numbers after standardizing the transcription data; reducing the dimension of the extracted gene expression matrix to a dimension DIM that matches the total number of preset tumor type classifications, and extracting m*DIM classification high-contribution genes from the reduced-dimensional data to obtain the transcription input data, where m is an integer not less than 2.

In some specific embodiments, the input layer of the transcription model has m*DIM neurons;

The transcription model has two fully connected layers; a ReLU activation function is introduced after the first fully connected layer to introduce nonlinearity; a Dropout function is applied after the first fully connected layer to randomly discard a preset proportion of outputs to reduce overfitting; and the second fully connected layer has n neurons.

In some specific embodiments, the clinical prognosis data are statistical indicators presented in the form of text descriptions or charts for evaluating possible outcomes of disease development and patient treatment effects and survival prospects;

The third preprocessing further includes: extracting key information including tumor stage and survival time from the clinical prognosis data.

In some specific embodiments, the clinical input data includes numerical features and categorical features;

In the clinical model, an embedding layer is used to process the category features and map them to a high-dimensional space; the numerical features and the category features output by the embedding layer are concatenated to form a single input vector; the input vector is subjected to feature extraction and dimensionality reduction through two fully connected layers, nonlinearity is introduced through the ReLU activation function, and the Dropout function is applied to reduce overfitting.

In some specific embodiments, in the pathological model, the output layer is a fully connected layer having n neurons;

During the pathological model training process, the cross entropy loss function is used to measure the difference between the model output and the training data, and the gradient of the loss relative to the model parameters is calculated by the back propagation algorithm; the SGD optimizer is used to update the model weights according to the calculated gradient, and the learning rate is initially set to 0.001.

In some specific embodiments, in the fusion processing unit, the pathological result features, the transcription result features, and the clinical result features all have corresponding weights, and the multimodal prediction model is used to fuse n of the pathological result features, n of the transcription result features, n of the clinical result features and their corresponding weights to obtain the final tumor prediction result.

In some specific embodiments, in the survival time prediction model, a pre-trained convolutional neural network model is used to extract image features of the pathological input image, and then a multi-layer perceptron is used to perform survival time prediction based on the image features to obtain the final survival time prediction result.

In some specific embodiments, during the training process of the survival time prediction model:

The mean square error (MSE) is used as the loss function and the Adam optimizer is used for parameter optimization;

In each training cycle, predictions are calculated via forward propagation and weights are updated via backpropagation.

Beneficial effects: The present application provides a lung tumor diagnosis and prediction system based on multimodal deep learning. To address the problem of single data in traditional deep learning prediction models, the present application uses multimodal deep learning technology to effectively integrate pathological genomics, transcriptomics, and clinical prognosis data. The present application uses deep learning technology to effectively integrate lung tumor pathological genomics, transcriptomics, and clinical prognosis data, thereby improving the accuracy of diagnosis and the interpretability of the model. The survival time prediction can provide doctors with prediction results in real time, providing doctors with more comprehensive and in-depth diagnostic and prognostic information, and helping doctors to better formulate treatment plans and make decisions.

In view of the heterogeneity between multimodal data, technical means such as data preprocessing and modal fusion are used to eliminate differences and conflicts, ensure the effectiveness and stability of the model, and adaptively adjust the weights and contributions of different modal data. Advanced technologies such as attention mechanism and residual connection are used to optimize model performance. Optimization algorithms and technical means such as gradient descent, regularization, and batch normalization are used to accelerate model convergence. Technical means such as data enhancement and transfer learning are used to expand the training data set to improve model accuracy and generalization ability, so that the model can process data of multiple modalities at the same time and has stronger feature extraction and classification capabilities. The multimodal deep learning model is combined with the clinical workflow to provide doctors with real-time diagnostic suggestions and prognosis prediction results, which can adapt to the needs of different clinical scenarios.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

FIG1 is a schematic diagram of the lung tumor diagnosis and prediction system modules of the present application;

FIG2 is a schematic diagram of the principle of the lung tumor diagnosis and prediction system of the present application;

FIG3 is a brightness comparison diagram before and after the modification of the present application;

FIG4 is a contrast comparison diagram before and after the modification of the present application;

FIG5 is a saturation comparison diagram before and after the modification of the present application;

FIG6 is a schematic diagram of the accuracy of the fusion model of the present application;

FIG. 7 is a schematic diagram of the accuracy of survival time prediction of the present application.

Figure numerals: 1-input unit; 2-preprocessing unit; 3-layered processing unit; 4-fusion processing unit; 5-survival time prediction unit.

DETAILED DESCRIPTION

Hereinafter, various embodiments disclosed in the present application will be described more fully. The present application discloses various embodiments, and adjustments and changes can be made therein. However, it should be understood that there is no intention to limit the various embodiments disclosed in the present application to the specific embodiments disclosed herein, but the present application should be understood to cover all adjustments, equivalents and/or alternatives within the spirit and scope of the various embodiments disclosed in the present application.

The terms used in the various embodiments disclosed in the present application are only used to describe the purpose of specific embodiments and are not intended to limit the various embodiments disclosed in the present application. As used herein, the singular form is intended to also include the plural form, unless the context clearly indicates otherwise. Unless otherwise specified, all terms used here (including technical terms and scientific terms) have the same meaning as the meaning generally understood by ordinary technicians in the field of various embodiments disclosed in the present application. The terms (such as the terms defined in the dictionary generally used) will be interpreted as having the same meaning as the contextual meaning in the relevant technical field and will not be interpreted as having an idealized meaning or an overly formal meaning, unless clearly defined in the various embodiments disclosed in the present application.

This application discloses a lung tumor diagnosis and prediction system based on multimodal deep learning, which integrates pathological genomics, transcriptomics and clinical prognosis data to build a multimodal deep learning model to achieve accurate identification of benign and malignant lung tumors and prognosis prediction. At the same time, the heterogeneity between multimodal data is processed to ensure the effectiveness and stability of the model. The module of the lung tumor diagnosis and prediction system is shown in Figure 1, the principle is shown in Figure 2, and the specific scheme is as follows:

A lung tumor diagnosis and prediction system based on multimodal deep learning, comprising:

Input unit 1, used to obtain at least two data of the patient including pathological data related to lung tumor pathological images, transcription data related to RNA transcription of lung tumor tissue, and clinical prognosis data related to clinical treatment of lung tumor patients;

Preprocessing unit 2, used for performing a preset first preprocessing on the pathological data to obtain a pathological input image, performing a preset second preprocessing on the transcription data to obtain transcription input data, and performing a third preprocessing on the clinical prognosis data to obtain clinical input data;

A hierarchical processing unit 3 is used to input the pathological input image, the transcription input data and the clinical input data into a multimodal prediction model that integrates the pathological model, the transcription model and the clinical model, so that the pathological model outputs n pathological result features according to the pathological input image, the transcription model outputs n transcription result features according to the transcription input data, and the clinical model outputs n clinical result features according to the clinical input data;

The fusion processing unit 4 is used to fuse n pathological result features, n transcription result features and n clinical result features through a multimodal prediction model to obtain a final tumor prediction result.

The survival time prediction unit 5 is used to input the pathological input image into a preset survival time prediction model to obtain the final predicted survival time of the patient.

Among them, the pathological result features, transcription result features, clinical result features and tumor prediction results all involve the predicted tumor type and its probability, and n is an integer not less than 2. The larger the value of n, the more data is fused, the higher the reliability of the final tumor prediction result, and the more complex the data fusion process. Therefore, it is necessary to set the value of n reasonably. In practical applications, the value of n can be adjusted according to the output nodes of the model, and the number of output nodes corresponds to the value of n. For example, the pathological model has 128 neurons and can output 128 pathological result features. Correspondingly, the outputs of the transcription model and the clinical model are also changed to 128, so that the number of pathological result features, transcription result features and clinical result features is the same, which is convenient for the fusion of multimodal prediction models.

The solution of the present application improves the accuracy and reliability of lung tumor diagnosis and prognosis prediction by integrating multimodal data. At the same time, it focuses on the generalization ability of the model and the integration of clinical workflows to provide doctors with convenient, fast and accurate diagnostic support. During the model training process, appropriate hyperparameter settings and optimization strategies are adopted, such as learning rate adjustment, batch size selection and regularization. The multimodal data of the present application includes at least pathological data, transcriptional data and clinical prognosis data, and the data input into the pre-trained multimodal prediction model includes at least two of the foregoing to achieve multimodal data prediction.

Among them, pathological data includes information about disease status obtained by means of histology, cytology, immunohistochemistry, molecular biology, etc. during disease diagnosis, research and treatment. These data are usually derived from the analysis of patient biological samples (such as tissue sections, cell smears, etc.), which are the key basis for understanding the nature of the disease, judging the condition, guiding treatment plans and evaluating prognosis. In this application, pathological data mainly exist in the form of images, such as full-view digital slices (WholeSlide Images, WSIs), ordinary digital pathological images, etc. These image data play a core role in modern medicine, and are used in clinical practice to intuitively display and analyze disease status, and assist doctors in making more accurate diagnosis and treatment decisions. Pathological data is different from transcription data that mainly exists in numerical form and clinical prognosis data that mainly exists in text form. The characteristic information contained in its image is more and the processing process is more complicated.

For pathological data, relevant preprocessing of image data is required. Image features are enhanced through a series of image preprocessing steps to improve the performance of image recognition or classification tasks and adapt the image data to the input requirements of deep learning models. Exemplarily, the first preprocessing includes brightness adjustment, contrast adjustment, saturation adjustment, gamma correction and pixel standardization of the image. Exemplarily, in order to adapt to the PyTorch framework, the image data is converted into a tensor format to ensure that the image data can be processed and analyzed in PyTorch in an appropriate form. Exemplarily, the pixel values of the image are normalized and limited to the range of [-1,1]. This step helps to ensure the stability of the data and the convergence of the model, laying a solid foundation for subsequent model training.

Specifically, the brightness adjustment factor, contrast adjustment factor, gamma correction value and saturation adjustment factor are set to adjust the overall visual effect of the image and reduce the brightness, contrast and saturation differences between images. The comparison of the brightness, contrast and saturation differences before and after the image adjustment is shown in Figures 3-5.

1) Regarding brightness adjustment, the standard parameter of the brightness adjustment factor is parameter a. By changing this adjustment factor, the overall brightness of the image can be controlled. If the brightness adjustment factor is greater than a, the image will become brighter, which helps to highlight the details of the dark area. If the brightness adjustment factor is less than a, the image will become darker, which helps to highlight the details of the bright area. If the brightness adjustment factor is equal to a, the brightness of the image will remain unchanged.

2) Regarding contrast adjustment, the standard parameter of the contrast adjustment factor is parameter b, which is used to control the difference between light and dark areas in the image. If the contrast adjustment factor is greater than b, the contrast of the image will increase, making the features in the image more distinct. If the contrast adjustment factor is less than b, the contrast of the image will decrease, making the image look smoother and with less detail differences. If the contrast adjustment factor is equal to b, the contrast of the image will remain unchanged.

3) About Gamma Correction

Gamma correction is a nonlinear operation used to improve the brightness response of an image. The parameter c is the inverse of the gamma value. When the gamma value is less than c, the dark area becomes brighter, thereby increasing the overall contrast of the image. When the gamma value is greater than c, the bright area becomes darker, thereby reducing the overall contrast of the image. The mathematical expression of gamma correction is i ^(1/) , where i is the pixel value and e is the gamma correction value.

4) About saturation adjustment

The saturation adjustment factor affects the intensity and purity of the colors in the image. Lowering the saturation will make the image color tend to be gray, reducing the difference between colors. The parameter d controls this process. Increasing the saturation will make the image colors more vivid and prominent, enhancing the visual impact. Saturation adjustment is achieved by operating on each RGB channel of the image, involving chroma and brightness adjustments within the color space.

5) About pixel standardization

The pixel values of each channel are normalized by the following formula:

For each pixel value in the RGB image, set the original pixel value to original _pixel , the mean to mean, and the standard deviation to std, then the normalized _pixel value is calculated by the above formula.

6) Parameter visualization before and after adjustment

The histogram visualization scheme is used to intuitively display the brightness and saturation distribution of the image before and after adjustment, so as to compare the effect of the first preprocessing and facilitate the adjustment of the specific values of parameters a, b, and c. Specifically, it includes the following: collecting statistical data for each image; calculating the brightness contrast distribution of the original image, including: the original mean (original brightness) and the original standard deviation (original contrast); calculating the brightness contrast distribution of the adjusted image, including: the adjusted mean (adjusted brightness) and the adjusted standard deviation (adjusted contrast); using a histogram to display the distribution of the mean and standard deviation of the brightness contrast of the image before and after processing.

When using pathological data as training data to train the model, the images can also be expanded so that the model can adapt to images with different parameters. The training data is expanded by random cropping, random horizontal flipping, color jittering (random adjustment of brightness, contrast, saturation and hue), etc. Specifically, the images after visual effect adjustment are batch preprocessed to expand the number of pathological sections and improve the generalization ability of the model. Specifically: randomly resize the image and crop it to 224x224 pixels; randomly flip the image horizontally; randomly adjust the brightness, contrast, saturation and hue of the image; standardize the pixel values of each channel; save the processed image in PNG format.

Regarding the pathological model, deep learning network models such as deep convolutional neural network (CNN), recurrent neural network (RNN) or graph neural network (GNN) can be used as the basic image model to adapt to different data structures and features. Preferably, the ResNet18 model is selected as the basic model of the pathological model. ResNet18 is a classic convolutional neural network model that can effectively extract key features in the image through pre-training on large-scale image datasets. These features have important reference value for tumor diagnosis.

Exemplarily, the construction and training of the image model include: using a pre-trained ResNet-18 model as a feature extractor; replacing the last fully connected layer of the model with a new fully connected layer with n output nodes to adapt to a specific pathological image classification task. During the training process, the parameters of the model are fine-tuned for the tumor diagnosis task to improve its adaptability to the task, and the tumor label and its corresponding probability are output. The cross entropy loss function is used to measure the difference between the probability distribution predicted by the model and the true label, and the gradient of the loss relative to the model parameters is calculated by the back propagation algorithm. The SGD optimizer is used to update the model weights according to the calculated gradient, where the learning rate is initially set to 0.001 and is multiplied by 0.1 after each a epoch for decay. During the training process of b epochs, the learning rate is continuously adjusted to optimize the model performance. An epoch represents the process in which the entire training data set is traversed and learned by the model once. An epoch is the process in which the algorithm completes a complete forward pass and backward pass for all training samples.

By analyzing the difference in RNA transcription data between tumor tissue and normal tissue, biomarkers with diagnostic value can be screened out. Based on transcription data, the progression and treatment effect of tumors can also be monitored. If the expression level of related genes returns to normal or changes in a favorable direction after treatment, it may mean that the treatment is effective. For example, the abnormal expression pattern of certain specific genes may be highly correlated with a specific type of tumor. In this application, transcription data is mainly for RNA, reflecting the gene expression in the human body over a period of time. Transcription data includes a gene expression matrix, which can be obtained by high-throughput sequencing and the like. Exemplarily, in a gene expression matrix, rows represent genes, columns represent samples, and the elements in the matrix represent the expression of the gene in the corresponding sample.

The transcription data is subjected to a second preprocessing, which involves operations such as data cleaning, standardization, and normalization to ensure the accuracy and consistency of the data. In some specific embodiments, during the model training process, the FPKM (Fragments Per Kilobase of exonmodel per Million mapped reads) method is used to standardize the transcription data to eliminate the influence of sample differences and gene length. The RNA sequencing data can be effectively converted into a format suitable for model training.

In some specific embodiments, the second preprocessing also includes: extracting genes with non-zero expression copy numbers after standardizing the transcription data; reducing the extracted gene expression matrix to a dimension DIM that matches the total number of preset tumor type classifications, and extracting m*DIM classified high-contribution genes from the reduced-dimensional data to obtain the transcription input data. A gene with a non-zero copy number refers to a gene with a copy number greater than zero in a genome or a specific region. A high-contribution gene refers to a gene that has a greater impact on the occurrence or phenotype of a certain tumor disease. The variation of these genes or changes in expression levels may be closely related to specific biological phenomena or disease states. The dimension DIM matches the total number of tumor type classifications that the model can predict.

Genes with non-zero copy numbers mean that these genes have at least one copy in the genome. These genes have a certain expression level under specific conditions, that is, their transcription products (such as mRNA) exist in cells and may play an important role in the physiological processes, development, and disease occurrence of cells. Because their expression copy number is non-zero, their presence and expression level can be detected by appropriate technical means (such as transcriptome sequencing, qPCR, etc.).

Preferably, principal component analysis (PCA) is used to reduce the dimension to a dimension that matches the total number of tumor type classifications, so as to reduce the data dimension and reduce the computational complexity. The total number of tumor type classifications can be preset, which represents all tumor types involved in the model.

In some specific embodiments, the input layer of the transcription model has m*DIM neurons; the transcription model has two fully connected layers; a ReLU activation function is introduced after the first fully connected layer to introduce nonlinearity; a Dropout function is applied after the first fully connected layer to randomly discard a preset proportion of outputs to reduce overfitting; the second fully connected layer has n neurons to output n prediction results. Exemplarily, a neural network model containing two fully connected layers is constructed, in which the input layer has Class*50 neurons, the first hidden layer has 512 neurons, the second hidden layer has 128 neurons, and 128 transcription result features are output. A ReLU activation function is introduced after the first hidden layer to introduce nonlinearity and enhance the expressiveness of the model. The Dropout technique is applied to randomly discard the neuron output of the first hidden layer with a probability of 0.5 to reduce overfitting and improve the generalization ability of the model.

Clinical prognosis data are statistical indicators presented in the form of text descriptions or charts to evaluate the possible outcomes of disease development, as well as the effectiveness of patient treatment and survival prospects. They usually include survival time, disease recurrence, recovery status, etc., which help to gain a deeper understanding of the natural course of the disease, influencing factors, etc., and provide a basis for the development of new treatments and drugs. In addition, clinical prognosis data also include patient conditions such as age and gender, which will also affect the patient's tumor prediction to a certain extent. In actual applications, clinical prognosis data can be either a doctor's text description of the patient's prognosis or related diagnosis and treatment data recorded in a table.

In some specific embodiments, the third preprocessing of clinical prognosis data performs missing value processing and outlier processing to ensure the integrity and accuracy of the data. For missing values, filling, deletion or interpolation is used for processing; for outliers, truncation, replacement or elimination is used for processing. In some embodiments, the third preprocessing also includes: extracting key information including tumor stage and survival time from the clinical prognosis data, and the key information has a greater impact on tumor prediction, so it needs to be processed separately. The key information includes at least two indicators, tumor stage and survival time. Tumor stage is an important indicator reflecting the severity and spread of the tumor. Different tumor staging systems may vary, but usually consider factors such as the size of the tumor, the depth of invasion, whether there is lymph node metastasis, and whether there is distant metastasis. Early tumor stage usually means that the tumor is small and localized, while late stage means that the tumor may have spread widely. Survival time refers to the time span from the diagnosis of the disease to death or a specific time point.

In some specific embodiments, the clinical input data contains numerical features and categorical features; in the clinical model, an embedding layer is used to process the categorical features and map them to a high-dimensional space; the numerical features and the categorical features output by the embedding layer are concatenated to form a single input vector; the input vector is subjected to feature extraction and dimensionality reduction through two fully connected layers, nonlinearity is introduced through the ReLU activation function, and the Dropout function is applied to reduce overfitting. The clinical model also exists in the form of a fully connected layer model, which takes the clinical input data as input and outputs a tumor label and its corresponding probability. During the model construction process, the network structure and parameters are also adjusted to improve the performance and robustness of the model.

In order to build a deep learning model that can accurately diagnose tumors, this application adopts a multi-information fusion strategy to solve the heterogeneity of multimodal data. First, three independent deep learning models were trained using pathological data, transcriptional data, and clinical data. These models are each responsible for extracting key information from a specific data set and generating corresponding output results. Subsequently, the output results of these three models are fed into a fully connected layer as input for fusion. The role of the fusion layer is to integrate information from different data sources to generate a more comprehensive and accurate tumor label. The multimodal tumor prediction model can comprehensively utilize multi-source information to improve the accuracy and reliability of diagnosis.

In order to build a deep learning model that can accurately predict patient survival time, this application uses a pre-trained ResNet50 model as a pathology data feature extractor, and replaces the last fully connected layer of the model with a new fully connected layer with 1 output node to adapt to the survival time prediction task. In this way, the model can extract deep features in pathology images and output predicted survival time.

In the training phase, the fusion model constructed above is trained as a whole. In order to optimize the parameters and weights of the model, optimization methods such as the gradient descent algorithm are used. Through continuous iteration and updating, the parameters of the model are gradually adjusted so that the error between the output of the model and the true label is gradually reduced. In this process, it is also necessary to select a suitable loss function to evaluate the performance of the model. Commonly used loss functions for multi-classification problems include cross entropy loss functions. By minimizing the loss function, it is ensured that the model continuously improves its performance during the training process. After sufficient training and optimization, the deep learning fusion model has a high diagnostic ability. In practical applications, this trained model can be used to predict patient samples.

In the prediction stage, the pathological data, transcriptional data and clinical prognosis data of the patient samples are input into the multimodal tumor prediction model. The prediction model will generate corresponding tumor labels based on the knowledge and rules it has learned, thus providing doctors with a powerful auxiliary tool. The multimodal tumor prediction model can provide doctors with diagnostic suggestions and prognosis prediction results in real time, helping doctors to better formulate treatment plans and make decisions. Doctors can formulate more accurate treatment plans based on the prediction results of the model and the clinical information of the patient. At the same time, the model can also be used for early screening and prevention of tumors to improve people's quality of life and health level. Combining the multimodal deep learning model with the clinical workflow provides doctors with a convenient, fast and accurate diagnostic support system. This integration method not only improves the work efficiency of doctors, but also enables the model to better adapt to actual clinical needs.

In view of the heterogeneity between multimodal data, this application proposes an effective processing mechanism, including data preprocessing and modal fusion, etc. These mechanisms ensure the effectiveness and stability of the model, enabling the model to adaptively adjust the weights and contributions of different modal data.

In some specific embodiments, the expression of the multimodal prediction model is:

ReLU(x)＝ax(0,x)

in, represents the output of the multimodal prediction model, W _class is the weight matrix of the output of the pathological model, transcriptional model and clinical model, Batch represents the batch normalization layer, Dropout represents the Dropout function, ReLU represents the ReLU activation function, f ₁ , f ₂ and f ₃ represent the pathological result features, transcriptional result features and clinical result features respectively; μ _batch represents the average value of the batch; represents the variance of the batch; ∈ is a small scalar for numerical stability, γ and β are trainable parameters of the BatchNorm layer; x is the output of the fusion layer and the previous operation (ReLU, Dropout).

In some specific embodiments, in a multimodal prediction model: n pathological result features, n transcriptional result features, and n clinical result features are fused into a low-dimensional space as input features through a first fully connected layer to obtain fused features; nonlinearity is introduced through a ReLU activation function; a Dropout function is applied to discard a preset proportion of input features to reduce overfitting; a batch normalization layer is used to plan features; and the fused features are mapped to the final tumor category through a second fully connected layer.

When fusing data of different modalities, the weight and contribution of each modality data can be adjusted according to the actual situation. In some specific embodiments, in the fusion processing unit, the pathological result features, transcription result features, and clinical result features all have corresponding weights, and the multimodal prediction model is used to fuse n pathological result features, n transcription result features, n clinical result features and their corresponding weights to obtain the final tumor prediction result. For example, in some cases, pathological data may be more important, while in other cases, transcription data may be more critical. By flexibly adjusting the weights, the influence of each modality data can be better balanced.

Data of different modalities often have heterogeneity, that is, differences in data distribution and feature expression. The heterogeneity processing mechanism proposed in this application can eliminate this heterogeneity and ensure the effectiveness and stability of the model through technical means such as data preprocessing and modal fusion. This mechanism enables the model to better adapt to different data environments and application scenarios, and improves the generalization ability of the model. When dealing with heterogeneity, in addition to using technical means such as data preprocessing and modal fusion, other advanced heterogeneity processing methods can also be considered, such as data enhancement based on generative adversarial networks (GAN) or feature transformation based on transfer learning.

In some specific embodiments, in the survival time prediction model, a pre-trained convolutional neural network model is used to extract the image features of the pathological input image, and then a multi-layer perceptron is used to predict the survival time based on the image features to obtain the final survival time prediction result. Exemplarily, after obtaining the patient's pathological input image, it is input into a pre-trained convolutional neural network model to extract image features; the extracted image features are input into a multi-layer perceptron (MLP) model to predict the patient's survival time. Using deep learning technology to effectively analyze pathological image data and more comprehensively reflect the characteristics of the tumor not only improves the accuracy of survival time prediction, but also provides doctors with more comprehensive and in-depth pathological information.

In some specific embodiments, the expression of the survival time prediction model is:

Where X represents the input pathological image, _W1 and _b1 are the weights and biases of the first fully connected layer, _H1 is the hidden layer output after the ReLU activation function. _W2 and _b2 are the weights and biases of the second fully connected layer. is the final survival time prediction value. Batch represents the batch normalization layer, Dropout represents the Dropout function, and ReLU represents the ReLU activation function.

Specifically, the patient's pathological slice image data is preprocessed to obtain a pathological input image. The pathological slice image provides morphological information of the tumor and is the basic data for the model to predict survival time. The preprocessing operations include image brightness adjustment, contrast adjustment, saturation adjustment, Gamma correction and pixel standardization to obtain a standardized pathological input image. The pathological input image is input into a pre-trained convolutional neural network (CNN) model, such as ResNet50, to extract image features. By using the pre-trained model, deep features in the pathological image can be effectively extracted and the feature expression ability of the model can be enhanced. Afterwards, the extracted image features are input into a multi-layer perceptron (MLP) model to predict the patient's survival time. The MLP model includes multiple fully connected layers, ReLU activation functions, Dropout layers, and batch normalization layers. Through the combination of these layers, the model can effectively capture the nonlinear relationship between features and improve the accuracy of prediction.

In some specific embodiments, during the training process of the survival time prediction model: the mean square error (MSE) is used as the loss function, and the Adam optimizer is used for parameter optimization; in each training cycle, the predicted value is calculated by forward propagation, and the weight is updated by back propagation.

Exemplarily, the output layer of the survival time prediction model is a fully connected layer with one neuron, which is used to regress the survival time. During the training process of the survival time prediction model, the mean square error (MSE) loss function is used to measure the difference between the model output and the training data, and the gradient of the loss relative to the model parameters is calculated by the back propagation algorithm. The Adam optimizer is used to update the model weights according to the calculated gradient, where the learning rate is initially set to 0.001 to ensure that the model can converge effectively.

In some specific embodiments, the training process of the survival time prediction model includes: preprocessing the pathological images used for training, including brightness adjustment, contrast adjustment, saturation adjustment, Gamma housekeeping, pixel standardization, etc., to obtain image data. Read the image data and survival time data to ensure that the image and survival time data are consistent. Screen the data with a survival time of less than or equal to five years (1825 days). Process missing values and negative values, convert negative values to positive values, and fill the missing values with the average value. Use the ResNet50 model to extract image features, and then use the multi-layer perceptron (MLP) to predict survival time. Use the mean square error (MSE) as the loss function and the Adam optimizer for parameter optimization. In each training cycle, the model calculates the predicted value through forward propagation and updates the weight through back propagation. Evaluation process: Evaluate the prediction performance of the model and calculate the error between the predicted value and the actual value. Use the inverse logarithmic transformation to restore the predicted value and the actual value to the original scale, and calculate the accuracy within the range of ±180 days. Scatter plots of actual values and predicted values, residual plots, and histograms of actual values and predicted values.

For the survival time prediction model, deep learning network models such as deep convolutional neural network (CNN), recurrent neural network (RNN) or graph neural network (GNN) can be used as the basic image model to adapt to different data structures and features. Preferably, the ResNet50 model is selected as the basic model of the pathological model. ResNet50 is a classic convolutional neural network model that can effectively extract key features in the image through pre-training on large-scale image datasets. These features have important reference value for tumor diagnosis.

Exemplary,model construction: Use the pre-trained ResNet50 model as a feature extractor, and replace the last fully connected layer of the model with a new fully connected layer with 1 output node to adapt to the survival time prediction task. In this way, the model can extract deep features in pathological images and output the predicted survival time.

Model training: During the training process, the parameters of the model were fine-tuned for the survival time prediction task to improve its adaptability to the task. The specific training steps are as follows:

The mean square error (MSE) loss function is used to measure the difference between the survival time predicted by the model and the true label. The gradient of the loss relative to the model parameters is calculated by the back propagation algorithm. The Adam optimizer is used to update the model weights according to the calculated gradient, where the learning rate is initially set to 0.001 and adjusted after every several epochs to optimize the model performance.

During the training process, the specific hyperparameters are set as follows:

Loss function: Mean squared error (MSE) loss function, which measures the difference between the model output and the true survival time.

Optimizer: Adam optimizer, with the initial learning rate set to 0.001.

Learning rate adjustment: After every several epochs, the learning rate is multiplied by 0.1 to decay to improve the convergence effect of the model.

For example, each epoch in the training process represents the process in which the entire training data set is traversed and learned by the model once. An epoch includes a complete forward pass and backward pass, so that the model gradually optimizes its parameters in each iteration.

In the training phase, the deep learning model constructed above is trained as a whole. In order to optimize the parameters and weights of the model, optimization methods such as the gradient descent algorithm are used. Through continuous iteration and updating, the parameters of the model are gradually adjusted so that the error between the output result of the model and the true label is gradually reduced. In this process, it is also necessary to select a suitable loss function to evaluate the performance of the model. Commonly used loss functions for regression problems include mean square error loss function (MSE) and the like. By minimizing the loss function, it is ensured that the model continuously improves its performance during the training process. The deep learning model after sufficient training and optimization has a high ability to predict survival time. In practical applications, this trained model can be used to predict the survival time of patient samples. During the training process, appropriate hyperparameter settings and optimization strategies are adopted, such as learning rate adjustment, batch size selection and regularization. The model of this application extracts image features through pre-trained ResNet50, and then uses a multi-layer perceptron (MLP) to predict survival time, thereby realizing efficient use of single modality data.

In the prediction stage, the pathological input image of the patient sample is input into the survival time prediction model. The prediction model will generate the corresponding survival time prediction results based on the knowledge and rules it has learned, thus providing doctors with a powerful auxiliary tool. The survival time prediction model can provide doctors with prediction results in real time, helping them to better formulate treatment plans and make decisions. Doctors can formulate more accurate treatment plans based on the prediction results of the model and the clinical information of the patient. At the same time, the model can also be used to predict the prognosis of cancer patients and improve people's quality of life and health level. Combining deep learning models with clinical workflows provides doctors with a convenient, fast and accurate diagnostic support system. This integration method not only improves the work efficiency of doctors, but also enables the model to better adapt to actual clinical needs.

The scheme of the present application was applied to lung tumor experiments, and the experimental results and comparative analysis are as follows: After multiple rounds of experiments and parameter adjustments, satisfactory experimental results were obtained. Specifically, in the task of distinguishing benign and malignant lung tumors, the multimodal deep learning model of the present application has higher accuracy and lower misjudgment rate than the traditional single modality model. In addition, by comparing with other advanced lung tumor identification methods, it was found that the present application has achieved significant advantages in multiple evaluation indicators. Figure 6 shows the accuracy of the fusion model of the present application in the training stage and the prediction stage. Figure 7 shows the accuracy of the survival time prediction model at different cutoff times.

In order to verify the actual application effect of this application, it was integrated into the hospital's clinical workflow and tested for several months. During this period, doctors actively provided feedback on the use of this application and suggestions for improvement. According to the feedback from actual applications, doctors generally believe that the diagnostic suggestions and prognosis prediction results provided by this application are accurate and reliable, and can provide strong support for them to formulate treatment plans and make decisions.

Through a series of experiments, simulations and practical applications, the feasibility and effectiveness of this application have been proven. The experimental results show that this application has high accuracy and practicality in the task of distinguishing benign and malignant lung tumors, and is expected to bring substantial improvements to clinical diagnosis and treatment. At the same time, the technical solution of this application will be continuously improved and optimized based on actual application feedback and doctor's suggestions to better serve clinicians and patients.

Although this application is mainly used for lung tumor experiments and verification, its core technology and methods are also applicable to other types of tumors. By adjusting the model structure and parameter settings, it can also be applied to the identification and prognosis prediction of various tumor types such as breast cancer, liver cancer, and gastric cancer.

In practical applications, in addition to pathological omics, transcriptomics and clinical prognosis data, other modalities of data, such as proteomics, metabolomics, etc., can also be considered to be integrated to more comprehensively reflect the characteristics and status of the tumor. This will further improve the accuracy and reliability of tumor identification and prognosis prediction.

This application proposes a lung tumor diagnosis and prediction system based on multimodal deep learning. To address the problem of single data in traditional diagnostic methods, deep learning technology is used to effectively integrate lung tumor pathological genomics, transcriptomics and clinical prognosis data, thereby improving the accuracy of diagnosis and the interpretability of the model. The survival time prediction can provide doctors with prediction results in real time, providing doctors with more comprehensive and in-depth diagnostic and prognostic information, helping doctors to better formulate treatment plans and make decisions, and can be effectively integrated into the current clinical diagnosis and treatment process of lung tumors.

In view of the heterogeneity between multimodal data, technical means such as data preprocessing and modal fusion are used to eliminate differences and conflicts, ensure the effectiveness and stability of the model, and adaptively adjust the weights and contributions of different modal data. Advanced technologies such as attention mechanism and residual connection are used to optimize model performance. Optimization algorithms and technical means such as gradient descent, regularization, and batch normalization are used to accelerate model convergence. Technical means such as data enhancement and transfer learning are used to expand the training data set to improve model accuracy and generalization ability, so that the model can process data of multiple modalities at the same time and has stronger feature extraction and classification capabilities. The multimodal deep learning model is combined with the clinical workflow to provide doctors with real-time diagnostic suggestions and prognosis prediction results, which can adapt to the needs of different clinical scenarios.

Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily necessary for the implementation of the present application. Those skilled in the art will appreciate that the modules in the devices in the implementation scenario can be distributed in the devices of the implementation scenario according to the description of the implementation scenario, or can be changed accordingly and located in one or more devices different from the present implementation scenario. The modules of the above-mentioned implementation scenarios can be combined into one module, or can be further split into multiple sub-modules. The above-mentioned serial numbers of this application are only for description and do not represent the pros and cons of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of the present application, but the present application is not limited thereto, and any changes that can be thought of by a technician in this field should fall within the scope of protection of the present application.

Claims (12)

1. A lung tumor diagnosis and prediction system based on multimodal deep learning, characterized by comprising: An input unit, used to obtain at least two data of a patient including pathological data related to lung tumor pathological images, transcription data related to RNA transcription of lung tumor tissue, and clinical prognosis data related to clinical treatment of lung tumor patients; A preprocessing unit, configured to perform a preset first preprocessing on the pathological data to obtain a pathological input image, perform a preset second preprocessing on the transcription data to obtain transcription input data, and perform a third preprocessing on the clinical prognosis data to obtain clinical input data; a hierarchical processing unit, configured to input the pathological input image, the transcription input data and the clinical input data into a multimodal prediction model integrated with a pathological model, a transcription model and a clinical model, so that the pathological model outputs n pathological result features according to the pathological input image, the transcription model outputs n pathological result features according to the transcription input data, and the clinical model outputs n clinical result features according to the clinical input data; A fusion processing unit, used for fusing the n pathological result features, the n transcription result features and the n clinical result features through the multimodal prediction model to obtain a final tumor prediction result; The survival time prediction unit is used to input the pathological input image into a preset survival time prediction model to obtain the final predicted survival time of the patient. Among them, the pathological result characteristics, the transcription result characteristics, the clinical result characteristics and the tumor prediction results all involve predicted tumor types and their probabilities, and n is an integer not less than 2. 2. The lung tumor diagnosis and prediction system according to claim 1, wherein the expression of the multimodal prediction model is: in, represents the output of the multimodal prediction model, W _class is the weight matrix of the output of the pathological model, the transcriptional model and the clinical model, BatchNorm represents the batch normalization layer, Dropout represents the Dropout function, ReLU represents the ReLU activation function, W _fusion is the weight matrix of the fusion layer, b _fusion is the bias term of the fusion layer, f ₁ , f ₂ and f ₃ represent the pathological result feature, the transcriptional result feature and the clinical result feature respectively; μ _batch represents the average value of the batch; represents the variance of the batch; ∈ is a small scalar for numerical stability, γ and β are trainable parameters of the BatchNorm layer; x is the output of the fusion layer and the previous operation (ReLU, Dropout). 3. The lung tumor diagnosis and prediction system according to claim 2, characterized in that, in the multimodal prediction model: The n pathological result features, n transcription result features, and n clinical result features are fused into the low-dimensional space as input features through the fusion layer to obtain fusion features; Applying ReLU activation function to the fused features introduces nonlinearity; Apply the Dropout function to the features after ReLU activation to discard a preset proportion of input features to reduce overfitting; Apply batch normalization layer to the features after Dropout to normalize the features; The fusion features are mapped to the final tumor category through a preset fully connected layer to obtain the final tumor prediction result. 4. The lung tumor diagnosis and prediction system according to claim 1 is characterized in that the first preprocessing includes brightness adjustment, contrast adjustment, saturation adjustment, gamma correction and pixel standardization of the image; the second preprocessing includes standardization of transcription data and extraction of genes with non-zero expression copy number; the third preprocessing includes missing value processing and outlier processing of clinical prognosis data. 5. The lung tumor diagnosis and prediction system according to claim 4 is characterized in that the second preprocessing also includes: after standardizing the transcription data, extracting genes with non-zero expression copy numbers; reducing the dimension of the extracted gene expression matrix to a dimension DIM that matches the total number of preset tumor type classifications, and extracting m*DIM classification high contribution genes from the reduced-dimensional data to obtain the transcription input data, where m is an integer not less than 2. 6. The lung tumor diagnosis and prediction system according to claim 5, characterized in that the input layer of the transcription model has m*DIM neurons; The transcription model has two fully connected layers; a ReLU activation function is introduced after the first fully connected layer to introduce nonlinearity; a Dropout function is applied after the first fully connected layer to randomly discard a preset proportion of outputs to reduce overfitting; and the second fully connected layer has n neurons. 7. The lung tumor diagnosis and prediction system according to claim 4, characterized in that the clinical prognosis data are statistical indicators presented in the form of text descriptions or charts for evaluating possible outcomes of disease development and patient treatment effects and survival prospects; The third preprocessing further includes: extracting key information including tumor stage and survival time from the clinical prognosis data. 8. The lung tumor diagnosis and prediction system according to claim 1, wherein the clinical input data includes numerical features and categorical features; In the clinical model, an embedding layer is used to process the category features and map them to a high-dimensional space; the numerical features and the category features output by the embedding layer are concatenated to form a single input vector; the input vector is subjected to feature extraction and dimensionality reduction through two fully connected layers, nonlinearity is introduced through the ReLU activation function, and the Dropout function is applied to reduce overfitting. 9. The lung tumor diagnosis and prediction system according to claim 1, characterized in that, in the pathological model, the output layer is a fully connected layer with n neurons; During the pathological model training process, the cross entropy loss function is used to measure the difference between the model output and the training data, and the gradient of the loss relative to the model parameters is calculated by the back propagation algorithm; the SGD optimizer is used to update the model weights according to the calculated gradient, and the learning rate is initially set to 0.001. 10. The lung tumor diagnosis and prediction system according to claim 1 is characterized in that, in the fusion processing unit, the pathological result features, the transcription result features, and the clinical result features all have corresponding weights, and the multimodal prediction model is used to fuse n of the pathological result features, n of the transcription result features, n of the clinical result features and their corresponding weights to obtain a final tumor prediction result. 11. The lung tumor diagnosis and prediction system according to claim 1 is characterized in that, in the survival time prediction model, a pre-trained convolutional neural network model is used to extract image features of the pathological input image, and then a multi-layer perceptron is used to predict the survival time based on the image features to obtain the final survival time prediction result. 12. The lung tumor diagnosis and prediction system according to claim 1, characterized in that during the training process of the survival time prediction model: The mean square error (MSE) is used as the loss function and the Adam optimizer is used for parameter optimization; In each training cycle, predictions are calculated via forward propagation and weights are updated via backpropagation.