TWI726459B - Transfer learning aided prediction system, method and computer program product thereof - Google Patents

Abstract

A transfer learning aided prediction system for analyzing an image data of the pharyngeal cancer tumor of a patient is provided. The system includes a data augmentation module and an analyzing module. The data augmentation module is used to apply a data expansion process to the image data, so as to generate a plurality of slices of the pharyngeal cancer tumor. The analyzing module determines a treatment response according to a pharyngeal cancer prognosis prediction model. The basic structure of the pharyngeal cancer prognosis prediction model can be consistent with that of a uterine cervical cancer prognosis prediction model. The uterine cervical cancer prognosis prediction model is constructed by a deep learning technique and is trained to achieve excellent predictive results, and the pharyngeal cancer prognosis prediction model is transformed from the uterine cervical cancer prognosis prediction model by a transfer learning technique.

Description

Transfer learning auxiliary prediction system, method and computer program product

The present invention belongs to the technical field of auxiliary prediction, in particular to the technical field of using transfer learning to convert a trained cervical cancer deep learning prognosis prediction model into an oropharyngeal/hypopharyngeal cancer prognosis prediction model.

Combination of chemical and radiotherapy (Chemoradiotherapy) is currently one of the conventional treatments for locally advanced oropharyngeal/hypopharyngeal cancer. Because the combination of chemical and radiotherapy will have a certain degree of impact on the health of the patient, and the disease Patients may still develop tumor metastasis or local recurrence after treatment. Therefore, if the prognosis of the treatment can be predicted in advance, and then the treatment strategy can be carefully selected, the quality of the patient's medical treatment can be effectively improved. However, there are currently no predictive technologies or products specifically for oropharyngeal/hypopharyngeal cancer.

In addition, the existing technology, which combines medical image analysis with traditional machine learning or statistical algorithms, still has shortcomings in accuracy, and the probability of prediction errors is very high.

In addition, different predictive models must be developed for different diseases. Since the development of predictive models is expensive and time-consuming, there is still room for improvement in current development methods.

In this regard, the present invention provides a migration learning auxiliary prediction system, method, and computer program product, which can effectively solve the above-mentioned problems.

The present invention provides an auxiliary prediction system, which is based on a cervical cancer prognosis prediction model and is converted into an oropharyngeal/hypopharyngeal cancer prognosis prediction model through migration learning. The prognosis prediction model for oropharyngeal/hypopharyngeal cancer can analyze tumor images of patients with oropharyngeal/hypopharyngeal cancer, and then predict the prognosis of the patient after combined chemical and radiotherapy. Since the cervical cancer prognosis prediction model or the oropharyngeal/hypopharyngeal cancer prognosis prediction model is trained by expanding the image data, it can provide a good prediction effect.

According to an aspect of the present invention, a migration learning assisted prediction system is proposed to analyze the image data of the patient’s oropharyngeal/hypopharyngeal cancer tumors before treatment. The system includes a small sample data expansion module and a prognostic prediction model for oropharyngeal/hypopharyngeal cancer. The small sample data expansion module can perform data expansion processing on the image data to generate multiple slice images. The prognosis prediction model of oropharyngeal/hypopharyngeal cancer can analyze the characteristics of slice images to predict the prognosis of patients receiving treatment.

According to another aspect of the present invention, a migration learning assisted prediction method is provided to analyze the image data of the patient’s oropharyngeal/hypopharyngeal tumors before treatment. The method is implemented through a deep learning assisted prediction system, and The method includes the steps of: performing data expansion processing on the image data by a small sample data expansion module to generate a plurality of slice images; and using an oropharyngeal/hypopharyngeal cancer prognosis prediction model to perform feature analysis on the slice images to predict Whether the patient will have an oropharyngeal/hypopharyngeal cancer treatment response event after receiving treatment, in which the prognosis prediction model for oropharyngeal/hypopharyngeal cancer is transformed from the cervical cancer prognosis prediction model through migration learning.

According to yet another aspect of the present invention, there is provided a computer program product stored in a non-transitory computer readable medium for operating a deep learning assisted prediction system, wherein the deep learning assisted prediction system is used to analyze patients The image data of oropharyngeal/hypopharyngeal cancer tumors before treatment, in which the computer program product includes: through the small sample data expansion module, the image data is expanded to generate multiple slice images of the image data; and The prognosis prediction model for oropharyngeal/hypopharyngeal cancer is used to perform feature analysis on slice images to predict the patient’s response after treatment. Among them, the prognosis prediction model for oropharyngeal/hypopharyngeal cancer is based on the cervical cancer prognosis prediction model through migration learning Converted into.

1: Transfer learning assisted prediction system (prediction system)

11: Data input terminal

12: Small sample data expansion module

14: Analysis module

15: Prognosis prediction model for oropharyngeal/hypopharyngeal cancer

16: Prognosis prediction model for cervical cancer

17: Training model

18: Training module

20: Computer Program Products

152-1: External perception convolutional layer

152-2: The first internal perceptual convolutional layer

152-3: The second inner perceptual convolutional layer

154: Global average pooling layer

156: Loss function layer

22: Normalization operation unit

24: Activation function

26: Maximum pooling layer

28: feature path

29: Featured items

T1: the first threshold

T2: second threshold

S11~S16, S21~S25, S51~S57, S61~S62, S71~S72, S81~S72: steps

Figure 1 is a system architecture diagram of a transfer learning assisted prediction system according to an embodiment of the present invention; Figure 2(A) is a flow chart of the basic steps of a transfer learning assisted prediction method according to an embodiment of the present invention; Figure 2(B) is the present invention Fig. 3 is a schematic diagram of the establishment process of an oropharyngeal/hypopharyngeal cancer prognostic prediction model of an embodiment of the present invention; Fig. 4(A) is a training model used in an embodiment of the present invention. Schematic diagram of the architecture before training; Fig. 4(B) is a schematic diagram of the architecture of the cervical cancer prognosis prediction model according to an embodiment of the present invention; Fig. 4(C) is the prognosis prediction model of oropharyngeal/hypopharyngeal cancer according to an embodiment of the present invention Schematic diagram of the architecture; FIG. 5 is a flowchart of the establishment process of a cervical cancer prognosis prediction model according to an embodiment of the present invention; FIG. 6 is a flowchart of the establishment process of the oropharyngeal/hypopharyngeal cancer prognosis prediction model according to the first embodiment of the present invention; 7 is a flowchart of the process of establishing a prognosis prediction model for oropharyngeal/hypopharyngeal cancer in the second embodiment of the present invention; Fig. 8 is a process flowchart of the process of establishing a prognostic prediction model for oropharyngeal/hypopharyngeal cancer in the third embodiment of the present invention.

The following description will provide a number of embodiments of the present invention. It can be understood that these embodiments are not intended to limit. The features of each embodiment of the present invention can be modified, substituted, combined, separated, and designed to be applied to other embodiments.

Fig. 1 is a system architecture diagram of a migration learning assisted prediction system 1 (hereinafter referred to as prediction system 1) according to an embodiment of the present invention, wherein the prediction system 1 is used to analyze the image of the patient’s oropharyngeal/hypopharyngeal cancer tumor before treatment Data to predict whether the patient will have an oropharyngeal/hypopharyngeal cancer treatment event after treatment. As shown in Fig. 1, the prediction system 1 can include a small sample data expansion module 12, an analysis module 14, an oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 and a training module 18, wherein the oropharyngeal/hypopharyngeal cancer The prognosis prediction model 15 is converted from a cervical cancer prognosis prediction model 16.

In addition, in one embodiment, the prediction system 1 may further include a data input terminal 11 for obtaining image data from the outside, that is, the user can input the image data into the prediction system 1 through the data input terminal 11. It should be noted that for oropharyngeal/hypopharyngeal cancer prognosis prediction model 15, the "image" can be, for example, a positron of an oropharyngeal/hypopharyngeal cancer patient before undergoing a treatment. Transmitting positron emission tomography (PET) images (hereinafter referred to as PET images) or computed tomography (CT) images (hereinafter referred to as CT images), in which treatment can be, for example, but not limited to, combined chemical and radiotherapy, The “image data” can be, for example, the volume of interest (VOI) range of the PET image or CT image, but is not limited to this; similarly, if the cervical cancer prognosis prediction model 16 is used, then “image” It can be, for example, a PET image or CT image of a cervical tumor of a patient with cervical cancer before undergoing a treatment. To make the description clearer, the following paragraphs will use PET images as examples.

In one embodiment, after the prediction system 1 obtains the image data of an oropharyngeal/hypopharyngeal tumor of a patient, the small sample data expansion module 12 may perform a data expansion process on the image data to generate an oropharyngeal/hypopharyngeal tumor. Multiple slice images of imaging data of hypopharyngeal tumors. The analysis module 14 can use the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 to perform a feature analysis on the image data of these details to obtain each slice image corresponding to an oropharyngeal/hypopharyngeal cancer treatment response event (defined as the first The occurrence probability of treatment response event), and the analysis module 14 can determine whether the first treatment response event will occur in each slice image according to a first threshold T1 and the occurrence probability. In addition, the analysis module 14 can predict whether the patient will be treated according to the probability of occurrence of the first treatment response event of each slice image of the same tumor or the number of slice images that will occur the first treatment response event and a second threshold value T2. The first treatment response event will occur. In other words, as long as the image data of the patient’s oropharyngeal/hypopharyngeal tumor before receiving chemical and radiotherapy is input into the prediction system 1, the prediction system 1 can predict the patient’s oral cavity after chemical and radiotherapy. Pharyngeal/hypopharyngeal cancer treatment response events, where oropharyngeal/hypopharyngeal cancer treatment response events may be prognostic events such as the recurrence or metastasis possibility of oropharyngeal/hypopharyngeal tumors, and are not limited thereto.

One of the characteristics of the present invention is that the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 is converted from the cervical cancer prognosis prediction model 16. In addition, in an embodiment, the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 can also be converted back to the cervical cancer prognosis prediction model 16. Therefore, the prediction system 1 can also be used for the prognosis prediction of cervical cancer, but is not limited. .

The details of each element will be described below.

The prediction system 1 can be an image processing device, which can be implemented by any device with a microprocessor, such as a desktop computer, a notebook computer, a smart mobile device, a server, a cloud host, and the like. In one embodiment, the prediction system 1 may have a network communication function to transmit data through the network. The network communication may be a wired network or a wireless network, so the prediction system 1 Image data can also be obtained through the Internet. In one embodiment, the prediction system 1 may be implemented by executing a computer program product 20 in a microprocessor. For example, the computer program product 20 may have a plurality of instructions that can enable the processor to perform special operations, and then the processor to execute The functions of the small sample data expansion module 12, the analysis module 14, the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15, the cervical cancer prognosis prediction model 16, or the training module 18, but are not limited, for example, in another embodiment , These modules can also be implemented through different computer programs. In one embodiment, the computer program product 20 can be stored in a non-transitory computer readable medium, such as a memory, but it is not limited thereto.

In one embodiment, the data input terminal 11 is a physical connection port used to obtain external data. For example, when the prediction system 1 is implemented by a computer, the data input terminal 11 may be a universal serial bus on the computer. bus, USB) interface, various transmission line connectors, etc., but not limited. In addition, the data input terminal 11 can also be integrated with a wireless communication chip, so that data can be received in a wireless transmission manner.

The small sample data expansion module 12 can be a functional module, which can be implemented by a program code. For example, when the microprocessor in the prediction system 1 executes the program code, the program code can enable the microprocessor to execute the program code. The processor executes the functions of the small sample data expansion module 12 described above.

The analysis module 14 can input the image data into the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 or the cervical cancer prognosis prediction model 16, and use the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 or the cervical cancer prognosis prediction model 16 Find out multiple image features in each slice image of the image data, and then use a feature path in the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 or cervical cancer prognosis prediction model 16 to predict the corresponding image of each slice Probability of treatment response events. In a first embodiment, the analysis module 14 also compares the occurrence probability of all images with a first threshold T1, where the first threshold T1 is a probability threshold, and when the occurrence probability is equal to or higher than the first threshold T1 When a threshold value T1, the analysis module 14 slices The imaging is deemed to be a treatment response event. In one embodiment, the analysis module 14 counts the number of slice images in which treatment response events will occur in the slice images, and compares the number of slice images in which treatment response events will occur with the second threshold T2, where the first The second threshold T2 is a quantitative threshold, and when the number is equal to or higher than the second threshold T2, the analysis module 14 determines the image data as a treatment response event, that is, the source patient of the image data is being treated Treatment response events will occur later. In a second embodiment, after the analysis module 14 obtains the probability of a treatment response event in each slice image, it does not compare the probability of each slice image with the first threshold value T1, but directly integrates it. The probability of each slice image (for example, calculating the average of the probabilities), and then compare the average of the probabilities with a third threshold value. When the average of the probabilities is higher than the third threshold value, The analysis module 14 determines that the image data is subject to a treatment response event, that is, the source patient of the image data will have a treatment response event after treatment.

The oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 is an artificial intelligence model that uses deep convolutional neural networks to analyze the image characteristics of oropharyngeal/hypopharyngeal cancer tumors; in particular, the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 is The cervical cancer prognosis prediction model 16 is adjusted and formed based on the completed training, and the cervical cancer prognosis prediction model 16 is formed by training based on a training model. In one embodiment, the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 or the cervical cancer prognosis prediction model 16 is composed of a complex algorithm (such as a code). In addition, in order to distinguish the cervical cancer prognosis prediction model 16 before and after training, the cervical cancer prognosis prediction model 16 before training will be referred to as "training model 17" in this article.

The training module 18 can train a training model 17 so that the training model 17 forms a cervical cancer prognosis prediction model 16. The training module 18 can also adjust the cervical cancer prognosis prediction model 16 to convert the cervical cancer prognosis prediction model 16 into the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15. In addition, the training module 18 can also be used to adjust the first threshold value T1 and the second threshold value T2.

After the cervical cancer prognosis prediction model 16 is adjusted and converted into the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15, the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 can be actually used. Next, the situation when the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 is actually used will be described.

FIG. 2(A) is a flowchart of the basic steps of a deep learning assisted prediction method according to an embodiment of the present invention, which is used to illustrate the steps of the prediction system 1 actually using the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 to analyze the patient's image data Flow, and please refer to Figure 1 at the same time. As shown in FIG. 2(A), first step S11 is executed, and the data input terminal 11 obtains an image data of an oropharyngeal/hypopharyngeal tumor of an oropharyngeal/hypopharyngeal cancer patient before receiving chemical and radiotherapy. After that, step S12 is executed, and the small sample data expansion module 12 performs data expansion processing on the image data to generate a plurality of slice images of the image data. After that, step S13 is executed, and the analysis module 14 uses the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 to perform feature analysis on each slice image data to obtain the occurrence probability of each slice image corresponding to the first treatment response event; Step S14 is executed, and the analysis module 14 determines whether the first treatment response event will occur for every f slice images according to the first threshold value T1 and the occurrence probability of each slice image; then step S15 is executed, and the analysis module 14 counts The number of slice images in which the first treatment response event will occur (defined as a first number); then step S16 is executed, and the analysis module 14 predicts whether the patient will occur after treatment based on the second threshold T2 and the first number The first treatment response event. In another embodiment, after step S13 is executed, step S17 is executed, and the analysis module 14 integrates the probability of each slice image corresponding to the first treatment response event, and according to the integrated data (such as the probability The average value of) and the third threshold value are used to predict whether the patient will have the first treatment response event after treatment. Next, the details of each step will be explained.

Regarding step S11, a user of the system (such as a physician) can input image data into the prediction system 1 through the data input terminal 11, where the image data can be, for example, the volume of interest of the oropharyngeal/hypopharyngeal tumor in the PET image. (VOI), where the VOI range can be obtained by various known methods. In one embodiment, the VOI range must include the entire tumor. In one embodiment, the imaging data is the imaging data of the patient's cervical tumor showing an abnormal metabolic reaction to the tracer after the ^{patient has taken the tracer (for example, 18 F-FDG).} In one embodiment, the image data may have a plurality of voxels, and the pixel value of each voxel refers to the Standardized Uptake Value (SUV) of glucose.

Regarding step S12, after the patient's image data is input to the prediction system 1, the small sample data expansion module 12 can perform data expansion processing on the image data according to the instructions in the computer program product 20. The purpose of step S12 is that if the available image data is limited, the results of the system training will be less than expected. Therefore, the amount of data must be expanded before training.

First, the details of the data expansion processing in step S12 will be described. Please also refer to FIG. 1 to FIG. 2(B), wherein FIG. 2(B) is a schematic diagram of the data expansion processing flow of an embodiment of the present invention, and the data expansion processing It is executed by the small sample data expansion module 12, that is, the entire process can be realized by the execution of the processor in the prediction system 1.

As shown in FIG. 2(B), first step S21 is executed, and the small sample data expansion module 12 interpolates the image data (hereinafter defined as the original image data) input into the prediction system 1. The purpose of this step is to improve the resolution of the image data. After that, step S22 is executed. Since the aforementioned interpolation process changes the image resolution, the small sample data expansion module 12 uses the volume pixel (SUV _max ) with the largest SUV value of the tumor in the spatial range of the original image data as the basis to interpolate The previous SUV _max coordinates are converted to the interpolated coordinates. After that, step S23 is executed, and the small sample data expansion module 12 extracts a volume of interest (VOI) _{with SUV max as the center point from the interpolated image data.} After that, step S24 is executed, and the small sample data expansion module 12 sets the XY plane, XZ plane, and YZ plane _{passing through the VOI center (SUV max) as a basic slice image group.} After that, step S25 is executed, and the small sample data expansion module 12 rotates one of the planes of the basic slice image group counterclockwise in a specific direction to obtain a plurality of extended slice image groups. In this way, step S12 can be completed, and the small sample data expansion module 12 can generate multiple slice images from a single image data.

It should be noted that step S12 is not limited to being executed when the trained oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 or cervical cancer prognosis prediction model 16 is actually used. It is required when the training model 17 is used for training. When the image data is insufficient, step S12 can also be executed to expand the image data.

Please refer to FIG. 2(A) again. Regarding steps S13 to S16 or step S17, when multiple slice images are obtained, the analysis module 14 can use the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 to analyze the slice images . Since each slice image contains the local features of the tumor, the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 can automatically analyze the local tumor features in the slice images, and determine the output results of the slice images through the feature path. In this way, the probability of occurrence of treatment events for oropharyngeal/hypopharyngeal cancer corresponding to each slice image is obtained, and the occurrence of oropharyngeal/hypopharyngeal cancer treatment events is predicted based on the results of all slice images. In this way, the prediction system 1 can predict the prognosis of the patient after receiving radiochemotherapy, so as to assist the user (for example, a physician) to determine whether the treatment mode needs to be adjusted. In addition, the description of steps S13 to S16 or step S17 can be found in the description of the analysis module 14 in the foregoing paragraphs, and therefore will not be described in detail.

The foregoing paragraphs have described the actual use of the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15, and the subsequent paragraphs will describe the establishment process of the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15.

FIG. 3 is a schematic diagram of the establishment process of the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 according to an embodiment of the present invention. Please also refer to FIG. 1. As shown in FIG. 3, the training model 17 can form the cervical cancer prognosis prediction model 16 after training, and the cervical cancer prognosis prediction model 16 can form the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 after adjustment. In an embodiment, the training performed by the training model 17 may be, for example, deep learning or machine learning, but it is not limited. In an embodiment, the adjustment performed by the cervical cancer prognosis prediction model 16 may be, for example, transfer learning, but is not limited.

In one embodiment, the training module 18 can use deep learning technology to train the training model 17, and after the training of the training model 17 is completed, a characteristic path will be generated. The characteristic path can be regarded as a neuron in the artificial intelligence model. Conduction path, in which each neuron can represent an image feature detection, and each image feature detection may have a different weight value, so that after the training model 17 is trained, it can form the prognosis prediction of cervical cancer Model 16. In one embodiment, the training module 18 can use the transfer learning technology to make the cervical cancer prognosis prediction model 16 adjust some parameters, and then convert the cervical cancer prognosis prediction model 16 into an oropharyngeal/hypopharyngeal cancer prognosis prediction model 15.

In one embodiment, the training model 17 needs to go through at least one "training phase" to train and establish a feature path, and the training model 17 needs to go through at least one "test phase" to test the accuracy of the feature path. When the accuracy reaches the requirement, it can be used as the actual cervical cancer prognosis prediction model16. In the present invention, the training model 17 will undergo multiple trainings, and will generate different feature paths after each training, and the feature path with the highest accuracy will be set as the actual feature path of the cervical cancer prognosis prediction model 16. . In addition, for the convenience of the explanation in the subsequent paragraphs, the oropharyngeal/hypopharyngeal cancer image data used in the adjustment of the cervical cancer prognosis prediction model 16 is defined as the "first training data", and the training model 17 used in training The imaging data of cervical cancer tumors is defined as "data for second training."

Next, the basic architectures of training model 17, cervical cancer prognosis prediction model 16, and oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 will be described. Please also refer to Figures 1 to 4(C), of which Figure 4(A) It is a schematic diagram of the architecture of the training model 17 before training according to an embodiment of the present invention. FIG. 4(B) is a schematic diagram of the architecture of the cervical cancer prognosis prediction model 16 according to an embodiment of the present invention. FIG. 4(C) is a schematic diagram of the first embodiment of the present invention. The structure diagram of the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 of the embodiment.

Regarding the training model 17, as shown in Figure 4(A), the architecture of the training model 17 may include an external perceptual convolution layer (mlpconv layer) 152-1, a first internal perceptual convolution layer 152-2, and a second The internal perceptual convolution layer 152-3, a global average pooling layer 154, and a loss function layer 156.

The training performed by the training model 17 is formed by training the training model 17 multiple times using a plurality of training images of cervical cancer tumors (second training data). In one embodiment, each second training data may include a tumor image of a cervical cancer patient before treatment and a cervical cancer treatment response event of the patient after treatment (hereinafter defined as a second treatment response event ). Preferably, the second training data can be expanded in advance through the small sample data expansion module 12 to generate multiple slice images.

In one embodiment, the external perceptual convolutional layer 152-1 can be used to obtain a plurality of image features from the slice images of a second training data. The first inner perceptual convolutional layer 152-2 and the second inner perceptual convolutional layer 152-3 are used to integrate the image features. The global average pooling layer 154 is used to establish a correlation between the image features and a second treatment response event (for example, to establish a characteristic path), and to generate a forward prediction prediction probability (for example, event meeting The probability of occurrence) and a negative prediction probability (such as the probability that the event will not occur), where the positive prediction probability and the negative prediction probability can be integrated into the occurrence probability of the second treatment response event. The loss function layer 156 can be used to adjust the weights of the number of training times for the positive prediction probability and the negative prediction probability, so that the two have similar chances of being selected on the feature path during training, avoiding the result of each training to only be biased towards the positive prediction prediction Probability or negative prediction probability; for example, if the result of each training is "the probability that the event will occur" but there is no "the probability that the event will not occur", the subsequent prediction results may be distorted and the function layer will be lost The role of 156 is to make the chances of both being selected similar.

In an embodiment, the outer perceptual convolutional layer 152-1, the first inner perceptual convolutional layer 152-2, and the second inner perceptual convolutional layer 152-3 may each include a normalization operation unit 22 to perform the normalization operation. Here, the normalization operation may be, for example, but not limited to batch normalization (Batch normalization). The normalization operation unit 22 can normalize the data of the convolution operation result of each multi-layer perceptual convolution layer 152, thereby speeding up the convergence speed of subsequent data processing and making the training process more stable. In addition, in an embodiment, each multi-layer perceptual convolutional layer 152 may each include a pooling unit 26 to perform a pooling operation. Here, the pooling operation may be, for example, Maximum pooling. The function is to reduce the size of the feature map obtained by the multi-layer perceptual convolution layer 152 and to concentrate the features in the reduced feature map. In a broad sense, the role of the pooling layer 26 can be regarded as extracting important features from the feature map. This emphasizes important features. In some embodiments, the maximum pooling layer 26 can also be changed to an average pooling layer structure.

In one embodiment, the outer perceptual convolutional layer 152-1, the first inner perceptual convolutional layer 152-2, the second inner perceptual convolutional layer 152-3, and the global average pooling layer 154 may each include an activation function 24 (Activation function ). The activation function 24 can be used to adjust the output of the outer perceptual convolutional layer 152-1, the first inner perceptual convolutional layer 152-2, the second inner perceptual convolutional layer 152-3 or the global average pooling layer 154, so that the output result is non-linear Effect, and then improve the predictive ability of the training model. The activation function 24 can be a saturated activation function or a non-saturate activation function. When the activation function 24 is a saturated activation function, the activation function 24 can adopt architectures such as tanh and sigmoid. When the function 24 is a non-saturated activation function, the activation function 24 may adopt a linear rectification function (Rectified Linear Unit, ReLU) or its variation architecture (for example, ELU, Leaky ReLU, PReLU, RReLU or other variation architecture). In a preferred embodiment, the activation function 24 of the multi-layer perceptual convolution layer 152 adopts a ReLU architecture, and the activation function 24 of the global average pooling layer 154 adopts a structure other than ReLU.

As shown in FIG. 4(A), since the training model 17 has not been trained yet, the feature path has not been established, and the items of image features that can be used for analysis have not yet been determined. In one embodiment, each "training" performed by the training model 17 includes "finding items related to the second treatment response event from the image features of the slice image", "adjusting the external perception convolutional layer 152-1, The internal parameters of the first internal perceptual convolutional layer 152-2, the second internal perceptual convolutional layer 152-3, the global average pooling layer 154, and a loss function layer 156" and "established with the second The characteristic path between the imaging features related to the treatment response event". In addition, in an embodiment, the training module 18 can also set the first threshold or the second threshold according to the prediction accuracy of the prediction model generated after each training, but it is not limited. In one embodiment, the estimation accuracy can be evaluated through the AUC curve, but it is not limited.

Regarding the cervical cancer prognosis prediction model 16, as shown in Figure 4(B), the architecture of the cervical cancer prognosis prediction model 16 may also include an external perceptual convolution layer 152-1, a first internal perceptual convolution layer 152-2, and a second An internal perceptual convolution layer 152-3, a global average pooling layer 154, and a loss function layer 156. Since the cervical cancer prognosis prediction model 16 is a prediction model formed by the training of the training model 17, compared with the training model 17, the cervical cancer prognosis prediction model 16 already has items 29 that can be used for analysis of image features, and Feature path 28 has been established.

Regarding the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15, as shown in Figure 4(C), the architecture of the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 may also include an external perception convolution layer 152-1 and a first internal perception convolution layer 152-2, a second internal perceptual convolution layer 152-3, a global average pooling layer 154, and a loss function layer 156. Since the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 is formed by adjusting the cervical cancer prognosis prediction model 16, its characteristic path, the internal parameters of the external perception convolution layer 152-1, and the first internal perception convolution layer 152-2 The internal parameters, the internal parameters of the second internal perceptual convolution layer 152-3, the internal parameters of the global average pooling layer 154, and the internal parameters of a loss function layer 156 may be different from the cervical cancer prognosis prediction model 16. One real In the embodiment, the characteristic path 28 of the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 and the characteristic path 28 of the cervical cancer prognosis prediction model 16 may be different, but it is not limited.

Next, the process of training the training model 17 to form the cervical cancer prognosis prediction model 16 will be described. 5 is a flowchart of the establishment process of the cervical cancer prognosis prediction model 16 according to an embodiment of the present invention, which can be executed by the training module 18, where steps S51 to S56 correspond to the "training phase", and step S57 corresponds to the "test phase" ", and please refer to Figures 1 to 4(C) at the same time. First, step S51 is executed, and the basic architecture of the training model 17 is set up, that is, the outer perceptual convolution layer 152-1, the inner perceptual convolution layer 152-2 and 152-3, the global average pooling layer 154, and the loss function layer. The number of 156 is set. The outer perceptual convolutional layer 152-1 and the inner perceptual convolutional layers 152-2 and 152-3 can each include feature detectors, and each feature detector is randomly generated. Then step S52 is executed, the training model 17 obtains a plurality of slice images of the second training data; then step S53 is executed, the outer perception convolution layer 152-1 and the inner perception convolution layer 152-2 and 152-3 The feature detectors perform convolution operations on the slice images to find image features; then step S54 is executed, and the global average pooling layer 154 enhances the image features; then step S55 is executed, the global average pooling layer 154 established a feature path, where the predicted path contains two output results, one of which is the positive prediction probability, and the other output is the negative prediction probability; then step S56 is executed, and steps S52 to S55 are re-executed until completion The preset number of training times (for example, 500 times); then step S57 is executed, and the prediction system 1 uses a plurality of slice images of the image data for testing to test the accuracy of each feature path, and sets the feature path with the highest accuracy It is the characteristic path of cervical cancer prognosis prediction model 16. In this way, the cervical cancer prognosis prediction model 16 can be established, and the probability of each slice image for the second treatment response event can be predicted.

In one embodiment, after the cervical cancer prognosis prediction model 16 is established, the training module 18 may further adjust the first threshold value T1 or the second threshold value T2 according to the accuracy tested in step S57, but it is not limited .

Next, the process of forming the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 by the cervical cancer prognosis prediction model 16 will be described. 6 is a flowchart of the establishment process of the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 according to the first embodiment of the present invention, and please refer to FIGS. 1 to 5 at the same time. As shown in Fig. 6, first step S61 is executed, and the cervical cancer prognosis prediction model 16 is established. After that, step S62 is executed, and the training module 18 directly uses the cervical cancer deep learning prediction model 16 as the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15.

預測模型來預測口咽/下咽癌的預後。 In this embodiment, the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 directly uses the cervical cancer prognosis prediction model 16, that is, the feature path of the cervical cancer prognosis prediction model 16 is not used as the image feature for analysis. The internal parameters of the project and each component (external perceptual convolution layer 152-1, first internal perceptual convolution layer 152-2, second internal perceptual convolution layer 152-3, global average pooling layer 154, and loss function layer 156) are adjusted , And the first threshold value T1 and the second threshold value T2 (or the third threshold value) are not adjusted. In other words, this embodiment directly uses the cervical cancer deep learning

Predictive model to predict the prognosis of oropharyngeal/hypopharyngeal cancer.

FIG. 7 is a flowchart of the establishment process of the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 according to the second embodiment of the present invention, and please refer to FIGS. 1 to 5 at the same time. As shown in Fig. 7, first step S71 is executed, and the cervical cancer prognosis prediction model 16 is established. Then step S72 is executed, and the training module 18 uses the external perception convolution layer 152-1, the first internal perception convolution layer 152-2, the second internal perception convolution layer 152-3, and the global average pool of the cervical cancer prognosis prediction model 16. The transformation layer 154 and the loss function layer 156 are used to readjust the first threshold value T1 and the second threshold value T2 through a large amount of image data of oropharyngeal/hypopharyngeal cancer tumors used for training (for example, including expanded slice images) Or readjust the third threshold.

In this embodiment, the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 uses the feature path of the cervical cancer prognosis prediction model 16, as the image feature items for analysis, and various components (external perception convolution layer 152-1, No. The internal parameters of an internal perceptual convolution layer 152-2, a second internal perceptual convolution layer 152-3, a global average pooling layer 154, and a loss function layer 156) are adjusted, but the first threshold value T1 and the second threshold value are adjusted T2 is adjusted. In addition, in an embodiment, if the prediction system 1 adopts step S17 in FIG. 2(A), the third threshold value can be adjusted instead.

In one embodiment, the adjustment method of the first threshold T1 is to input the oropharyngeal/hypopharyngeal cancer tumor slice image into the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 to obtain the slice image to predict the incidence of treatment response events, and pass The first threshold value T1 is adjusted for the goal of obtaining the best prediction accuracy.

In one embodiment, the second threshold value T2 is adjusted by inputting oropharyngeal/hypopharyngeal cancer tumor slice images into the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 to obtain slice images to predict the incidence of treatment response events and use adjustments The subsequent first threshold T1 determines whether there will be a treatment response event in the slice image, and the second threshold T2 is adjusted with the goal of obtaining the best prediction accuracy.

FIG. 8 is a flowchart of the establishment process of the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 according to the third embodiment of the present invention, and please refer to FIGS. 1 to 5 at the same time. As shown in Fig. 8, first step S81 is executed, and the cervical cancer prognosis prediction model 16 is established. Then step S82 is executed, and the training module 18 uses the external perceptual convolutional layer 152-1, the first internal perceptual convolutional layer 152-2, and the second internal perceptual convolutional layer 152-3 of the cervical cancer prognosis prediction model 16, and a large number of Training image data of oropharyngeal/hypopharyngeal cancer tumors (for example, including expanded slice images) to retrain the global average pooling layer 154 and the loss function layer 156, as well as the oropharyngeal/hypopharyngeal training for a large number of training The image data of the cancer tumor (for example, including the expanded slice image) is used to readjust the first threshold value T1 and the second threshold value T2 or readjust the third threshold value.

In this embodiment, the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 will continue to use the image feature items used for analysis in the cervical cancer prognosis prediction model 16, but will retrain the global average pooling layer 154 to create a new Characteristic path. In addition, the internal parameters of the loss function layer 156, the first threshold value T1 and the second threshold value T2 will also be re-adjusted.

In one embodiment, the parameter that can be adjusted in the global average pooling layer 154 is a parameter that integrates image features into slice images to predict the probability of occurrence of treatment response events.

In one embodiment, the adjustable parameter in the loss function layer 156 is an expression represented by it. In an embodiment, the loss function layer 156 of the cervical cancer prognosis prediction model 16 may be: 1-((2 * sensitivity * Positive Predictive Value)/(sensitivity+Positive Predictive Value)), where sensitivity is the occurrence of all medical events The proportion of slice images that are accurately predicted to be the occurrence of treatment response events, and the Positive Predictive Value is the proportion of slice images that are actually predicted to be the occurrence of treatment response events. In one embodiment, the loss function layer 156 of the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 is adjusted to: sqrt(power(1-specificity,n)+power(1-sensitivity,n)), where specificity is all The proportion of slice images that are predicted to be no treatment response events that are accurately predicted to be no treatment response events. sqrt is an arithmetic function for extracting the square root, and power is a power arithmetic function. In this embodiment, n is set to 2.

In an embodiment, the training module 18 can automatically select the embodiment of FIG. 6, FIG. 7, or FIG. 8 according to some condition thresholds. For example, the training module 18 may use the embodiment of FIG. 6 in advance, and use some image data for testing (information about whether the first treatment response event will occur) is used to verify the accuracy. When a threshold is reached, the embodiment of FIG. 6 is continued to be used. When the accuracy of the embodiment of FIG. 6 does not reach the threshold, the embodiment of FIG. 7 is used instead, and the accuracy is performed again. For verification of the degree of accuracy, when the accuracy reaches the threshold, the embodiment of FIG. 7 is continued to be used. When the accuracy of the embodiment of FIG. 7 does not reach the threshold, the embodiment of FIG. 8 is adopted instead; the present invention is not limited to this.

Thereby, the prognosis prediction model 15 for oropharyngeal/hypopharyngeal cancer of the present invention can be formed.

In an experimental example, the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 of an embodiment of the present invention is verified through multiple actual data, wherein the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 predicts positive (that is, treatment will occur). Response events) and the actual results are consistent with a total of 36 data, and the prediction is positive but the experimental results are opposite there are 16 cases, so the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 has an accuracy of 69% for the positive prediction. It has been able to meet actual needs. In addition, the oropharyngeal/hypopharyngeal cancer prognosis prediction model 15 predicts negative (that is, no treatment response event will occur) and the actual results are consistent with 33 data, while the prediction is negative but the experimental results are opposite, there are 12 data. Therefore, the prognosis prediction model 15 for oropharyngeal/hypopharyngeal cancer has an accuracy of 73% for negative prediction, which can also meet actual needs.

In this way, the cervical tumor image assisted prediction system of the present invention can expand a small amount of image data through the small sample data expansion module, without the need to input huge image data at the beginning. In addition, the oropharyngeal/hypopharyngeal cancer prognosis prediction model of the present invention can directly use the trained cervical cancer prognosis prediction model, or the cancer prognosis prediction model can be obtained by adjusting some parameters, without having to spend a lot of time to establish new predictions. model.

Although the present invention has been illustrated through the above-mentioned embodiments, it is understandable that many modifications and changes are possible according to the spirit of the present invention and the scope of the patent application claimed by the present invention.

S11~S16: steps

Claims (11)

A migration learning auxiliary prediction system for analyzing an image data of an oropharyngeal/hypopharyngeal cancer tumor of a patient before a treatment, including: a small sample data expansion module for the oropharyngeal/hypopharyngeal cancer tumor Perform a data expansion process on the image data to generate a plurality of slice images of the image data of the oropharyngeal/hypopharyngeal cancer tumor; and an analysis module that uses an oropharyngeal/hypopharyngeal cancer prognosis prediction model for each slice The image performs a feature analysis to obtain a probability of occurrence of a first treatment response event corresponding to each slice image, and determines whether the first treatment response event will occur in each slice image according to a first threshold value and the occurrence probability, And based on a second threshold and the number of slice images in which the first treatment response event will occur, predict whether the patient will have the first treatment response event after treatment; wherein, the oropharyngeal/hypopharyngeal cancer The prognosis prediction model is converted from a cervical cancer prognosis prediction model through a transfer learning. The transfer learning assisted prediction system according to claim 1, wherein the cervical cancer prognosis prediction model is formed by a training model undergoing multiple training and testing, wherein the training uses multiple training cervical cancers The image data of the tumor is formed by training the training model multiple times, wherein the training model includes: an external perception convolution layer (mlpconv layer), which is used to obtain the image data of a cervical cancer tumor for training A plurality of image features; at least one internal perceptual convolutional layer to integrate the image features; a global average pooling layer to establish the relationship between the image features and a second treatment response event A correlation, and a positive prediction probability or a negative prediction probability is generated according to the correlation, wherein the positive prediction probability or the negative prediction probability corresponds to a probability of occurrence of the second treatment response event; and A loss function layer for adjusting a weight of training times for the positive prediction probability and the negative prediction probability. The migration learning assisted prediction system according to claim 2, wherein the oropharyngeal/hypopharyngeal cancer prognosis prediction model directly uses the cervical cancer prognosis prediction model. The transfer learning assisted prediction system according to claim 2, wherein the oropharyngeal/hypopharyngeal cancer prognosis prediction model directly uses the external perceptual convolutional layer, the at least one internal perceptual convolutional layer, and the cervical cancer prognostic prediction model. The global average pooling layer and the loss function layer are used to readjust the first threshold value and the second threshold value through a large amount of image data of oropharyngeal/hypopharyngeal cancer tumors used for training. The transfer learning assisted prediction system according to claim 2, wherein the oropharyngeal/hypopharyngeal cancer prognosis prediction model directly uses the external perceptual convolutional layer and the at least one internal perceptual convolutional layer of the cervical cancer prognostic prediction model, and Retrain the global average pooling layer and the loss function layer through a large amount of image data of oropharyngeal/hypopharyngeal cancer tumors used for training, and readjust the first threshold value and the second threshold value. A transfer learning-assisted prediction method is used to analyze an image data of an oropharyngeal/hypopharyngeal tumor of a patient before undergoing a treatment. The method is implemented through a deep learning-assisted prediction system, and the method includes the steps: A small sample data expansion module, the image data of the oropharyngeal/hypopharyngeal cancer tumor undergoes a data expansion process to generate multiple slice images of the image data of the oropharyngeal/hypopharyngeal cancer tumor; through an analysis module Group, using an oropharyngeal/hypopharyngeal cancer prognosis prediction model to perform a feature analysis on each slice image to obtain a probability of occurrence of each slice image corresponding to a first treatment response event; through the analysis module, according to a first The threshold value and the probability of occurrence determine whether the first treatment response event will occur in each slice image; and through the analysis module, according to a second threshold value and the number of slice images in which the first treatment response event will occur , Predict whether the patient will have the first treatment response event after treatment; Among them, the oropharyngeal/hypopharyngeal cancer prognosis prediction model is converted from a cervical cancer prognosis prediction model through a transfer learning. The migration learning assisted prediction method according to claim 6, wherein the cervical cancer prognosis prediction model is formed by a training model undergoing multiple training and testing, wherein the training uses multiple training cervical cancer tumors The training model is formed by performing multiple training on the image data of the training model, wherein the training model includes: an external perception convolution layer (mlpconv layer) for obtaining a plurality of images from the image data of a training cervical cancer tumor Original features; at least one internal perceptual convolution layer to integrate the original features; a global average pooling layer to establish a correlation between the original features and a second treatment response event And generate a positive prediction probability and a negative prediction probability according to the correlation, wherein the positive prediction probability and the negative prediction probability are integrated into an occurrence probability of the second treatment response event; and a loss function The layer is used to adjust a weight of training times for the positive prediction probability and the negative prediction probability. The transfer learning assisted prediction method according to claim 7, wherein the transfer learning includes the step of directly using the cervical cancer prognosis prediction model. The transfer learning-assisted prediction method according to claim 7, wherein the transfer learning includes the step of directly using the external perceptual convolutional layer, the at least one internal perceptual convolutional layer, and the global average pooling layer of the cervical cancer prognosis prediction model And the loss function layer; and readjust the first threshold value and the second threshold value through a large amount of image data of oropharyngeal/hypopharyngeal cancer tumors used for training. The transfer learning assisted prediction method according to claim 7, wherein the transfer learning comprises the steps of: directly using the external perceptual convolutional layer and the at least one internal perceptual convolutional layer of the cervical cancer prognosis prediction model; and using a large amount of training The image data of the oropharyngeal/hypopharyngeal cancer tumor retrains the global average pooling layer and the loss function layer, and readjusts the first threshold value and the second threshold value. A computer program product stored in a non-transitory computer-readable medium for the operation of a transfer learning assisted prediction system, wherein the transfer learning assisted prediction system is used to analyze a patient’s mouth before a treatment An image data of a pharynx/hypopharyngeal cancer tumor, wherein the computer program product includes: a command, through a small sample data expansion module, the image data of the oropharyngeal/hypopharyngeal cancer tumor undergoes a data expansion process to generate A plurality of slice images of the image data of the oropharyngeal/hypopharyngeal cancer tumor; an instruction, through an analysis module, uses an oropharyngeal/hypopharyngeal cancer prognosis prediction model to perform a feature analysis on each slice image to obtain each Each slice image corresponds to a probability of occurrence of a first treatment response event; a command, through the analysis module, determines whether the first treatment response event will occur in each slice image according to a first threshold value and the occurrence probability; and A command to predict whether the patient will have the first treatment response event after treatment based on a second threshold value and the number of slice images in which the first treatment response event will occur through the analysis module. The oropharyngeal/hypopharyngeal cancer prognosis prediction model is converted from a cervical cancer prognosis prediction model through a transfer learning.

Images