WO2023040164A1

WO2023040164A1 - Method and apparatus for training pet/ct-based lung adenocarcinoma and squamous carcinoma diagnosis model

Info

Publication number: WO2023040164A1
Application number: PCT/CN2022/074386
Authority: WO
Inventors: 朱闻韬; 金源; 黄海亮; 薛梦凡
Original assignee: 之江实验室
Priority date: 2021-09-14
Filing date: 2022-01-27
Publication date: 2023-03-23
Also published as: CN113516210A; JP2023544466A; CN113516210B

Abstract

Provided in the present invention are a method and apparatus for training a PET/CT-based lung adenocarcinoma and squamous carcinoma diagnosis model. In the present invention, a multi-task learning method is used, and a pathological feature is extracted by a neural network that is obtained by means of diagnosis classification training based on pathological images, thereby assisting in the training of a PET/CT-image-based diagnosis classification neural network. By means of the present invention, the precision of lung carcinoma diagnosis and classification based on a PET/CT image is improved, and pathological images are only used as a priori knowledge during a training process, and do not need to be used as inputs of a network during practical applications. In the method, the concept of multi-dimension fusion is used, such that the precision of using a PET/CT image for lung carcinoma diagnosis and classification is improved, thereby facilitating the further popularization and application of using the PET/CT image as a means for early lung carcinoma diagnosis, and helping a clinician to provide diagnosis for a patient and helping with a subsequent treatment scheme; in addition, by means of the solution of using pathological images as a priori knowledge assistance, the interpretability of a pathological section is further improved, thereby facilitating a pathologist in further extracting a pathological feature.

Description

A PET/CT-based diagnostic model training method and device for lung adenocarcinoma and squamous cell carcinoma

technical field

The present invention relates to the fields of medical imaging and deep learning, in particular to a fully automatic intelligent diagnosis model training method and device for lung adenocarcinoma and squamous cell carcinoma based on PET/CT and pathological slices.

Background technique

Positron emission tomography (PET) is a functional imaging device at the molecular level. The radioactive tracer needs to be injected into the patient before scanning, and the tracer decays and annihilates in the patient's body, producing a pair of 511keV gamma photons with emission directions about 180° opposite, and the detector will collect these gamma photons to reach the position of the crystal and time information. By using image reconstruction algorithms to reconstruct and post-process the acquired information, the metabolism and uptake of the reaction tracer in the patient can be obtained. According to the imaging results of PET/CT, doctors comprehensively analyze the patient's condition in combination with various clinical indicators, so as to determine the treatment plan.

Pathological examination, that is, a pathological and morphological method used to examine pathological changes in organs, tissues or cells of the body, is the examination method with the highest diagnostic accuracy among all examinations. The pathologist takes out a small piece of tissue from the diseased part of the patient's body (according to different situations, forceps, excision, or puncture suction can be used) or surgically removes the specimen to make pathological sections, and observes the morphological changes of cells and tissues to determine the pathological changes. The nature of the lesion and the pathological diagnosis are called biopsy (biopsy), referred to as living body. Pathological examination is a commonly used and relatively accurate method in the diagnosis of tumors, and is known as the "gold standard" for diagnosis.

Deep convolutional neural network (CNN) is one of the common methods for building medical artificial intelligence models in recent years. It extracts feature information of different dimensions of images through layered convolution processing, and the extracted features are input into subsequent specific Networks perform specific tasks such as classification, segmentation, registration, detection, noise reduction, etc. The advantage of this method is that it can automatically learn high-order features that are significant for specific tasks through samples, but it has certain requirements for the amount of data used for training.

Regarding the existing PET/CT-based lung cancer diagnosis and classification model, due to the limitation of data scale and accuracy, no matter whether the model is trained on single-center data or multi-center data, its diagnosis and classification accuracy cannot meet the practical requirements. Pathological slides, which are regarded as the "gold standard" for cancer diagnosis, are rarely used in early diagnosis because sampling often requires invasive or even invasive examinations on patients. Therefore, the development of a PET/CT-based model that can classify early lung cancer with a high accuracy rate can improve the hospital's diagnosis rate of early lung cancer to a certain extent and help clinicians to carry out follow-up treatment.

Contents of the invention

The purpose of the present invention is to address the deficiencies in the prior art, to provide a PET/CT-based lung adenocarcinoma squamous cell carcinoma diagnostic model training method and device, the present invention is based on multi-objective learning (Multi-Task Learning) algorithm, in order to pass Pathological features are used to optimize the PET/CT-based deep learning network to improve the training efficiency and upper limit of accuracy of the PET/CT network. In the clinical application process, only PET/CT images are needed to obtain the diagnostic classification information of lung adenocarcinoma and squamous cell carcinoma, without the participation of pathological information.

The purpose of the present invention is achieved through the following technical solutions:

A training method based on PET/CT lung adenocarcinoma and squamous cell carcinoma diagnostic classification model, specifically comprising:

Obtain the corresponding PET/CT images, pathological images, and lung adenocarcinoma squamous cell carcinoma diagnostic result data, and input the pathological images into a trained neural network A to obtain pathological features;

An initial neural network model is constructed, with PET/CT images as input, predicted pathological features and lung adenocarcinoma squamous cell carcinoma diagnosis results as output, and the acquired data is used to train the initial neural network model, and a PET/CT based lung adenocarcinoma squamous cell carcinoma is obtained. Cancer diagnosis classification model;

Wherein, the input of the neural network A is a pathological image, and the output is the diagnosis result of lung adenocarcinoma and squamous cell carcinoma, which is obtained through the training of the obtained pathological image data. Since the pathological image has a "gold standard" diagnosis and classification effect for lung cancer, it can be trained The network A makes it have extremely high classification accuracy for the input pathological image, and the diagnostic classification accuracy is required to be greater than or equal to 0.95 in the present invention; the pathological feature is the output of a feature extraction layer of the neural network A. Preferably, it is the input of the previous layer of the output layer of the neural network A.

Preferably, the initial neural network model specifically includes the following features:

1. The network contains two inputs. After the pre-processing convolution layer, the input PET and CT images are normalized in size, superimposed along the channel dimension, and then enter the main convolution layer.

2. The network is a multi-task target network, which is mainly established based on the regularization framework. The target function is:

stU U ^T =I

Where m represents the number of tasks, n _i represents the number of training samples,

Indicates the label of the j-th training sample and the i-th task, l(.,.) represents a loss function, such as cross-entropy loss or mean square error loss, b=(b ₁ ,...b _m ) ^T is Offset compensation in all tasks, U∈R ^d×d is a square transformation matrix, A∈R ^d×m contains the parameters of each task, d is the dimension of the parameter, and ||A|| ² _2,1 is Its L2 regularization matrix, a ⁱ represents the model parameters of the i-th task, I is the identity matrix and λ is the regularization parameter. The first part of the objective function represents the experience loss of all tasks, and the second part uses L2 regularization to ensure that the learned rows are sparse and the orthogonalization of the constraint matrix U, then formula (1) can also be expressed as:

in

Indicates the total training loss in formula (1), tr(.) represents the trace of the matrix, W ⁱ =Ua ⁱ represents the model parameters of the i-th task, and D≥0 indicates that the D matrix is a positive semi-definite matrix. The optimization of the multi-task target network is to solve the covariance matrix D, which decouples multiple task problems and promotes their parallel computing.

3. The main task of the initial neural network model is the diagnosis and classification of lung cancer, and the auxiliary task is the fitting of pathological features. The input image outputs high-dimensional features after passing through the main convolutional layer, and the features output diagnostic classification results after being processed by the fully connected layer of the main task, and output fitting pathological features after being processed by the convolutional layer of the auxiliary task. The two outputs are compared with the real diagnostic results of the case and the pathological features output by network A and the loss is calculated. The two losses jointly determine the parameter update of the initial neural network model.

Further, the parameter quantity of the initial neural network model should be less than or equal to the parameter quantity of the neural network A, so that the initial neural network model can learn the diagnostic classification features output by the neural network A, and avoid overfitting.

Further, the pathological image satisfies:

Among them, S _mask and S _all are the area of the image marked by the doctor's mark with lung cancer information and the whole area including the background, respectively.

Further, the neural network A adopts a ResNet-50 structure.

Further, the lung adenocarcinoma and squamous cell carcinoma diagnostic classification model adopts the DenseNet-121 structure, and the input is the feature of fusion of PET and CT images along the channel dimension.

Further, the initial neural network model is trained using the acquired data, and the PET/CT lung adenocarcinoma and squamous cell carcinoma diagnostic classification model is obtained as follows:

Calculate the error between the output of the initial neural network model and the corresponding true value, update the parameters of the initial neural network model according to the error until the error is minimum, and obtain a diagnostic classification model based on PET/CT lung adenocarcinoma and squamous cell carcinoma; the loss represents for:

Loss＝loss1*(1-φ)+loss2*φ

Among them, loss1 and loss2 are the errors between the diagnostic results and pathological features of lung adenocarcinoma squamous cell carcinoma and the corresponding true value, respectively, and φ is a hyperparameter.

Further, the loss1 uses a cross-entropy loss function, and loss2 uses a mean square loss function.

Based on the same inventive idea, the present invention also provides a training device for a PET/CT lung adenocarcinoma and squamous cell carcinoma diagnostic classification model, including:

A data acquisition unit, configured to acquire corresponding PET/CT images, pathological images, and diagnosis result data of lung adenocarcinoma and squamous cell carcinoma;

A pathological feature acquisition unit, configured to input pathological images to a trained neural network A to obtain pathological features;

The training unit is used to construct the initial neural network model, and the PET/CT image is used as input, the predicted pathological features and lung adenocarcinoma squamous cell carcinoma diagnosis results are output, and the data obtained by the data acquisition unit and the pathological feature acquisition unit are used to train the described The initial neural network model was used to obtain a diagnostic classification model for lung adenocarcinoma and squamous cell carcinoma based on PET/CT.

Further, a data preprocessing unit is also included, which is used to process the corresponding PET/CT images and pathological images into pictures of the same size.

The trained lung adenocarcinoma and squamous cell carcinoma diagnostic classification model can directly output the diagnostic classification results based on PET/CT images without the participation of pathological data. specifically:

A PET/CT diagnostic classification device for lung adenocarcinoma and squamous cell carcinoma, comprising:

A data acquisition module, configured to acquire PET/CT images to be diagnosed;

The diagnosis and classification module of lung adenocarcinoma and squamous cell carcinoma is used to input the PET/CT image to be diagnosed into the PET/CT-based diagnostic classification model of lung adenocarcinoma and squamous cell carcinoma trained by any one of the above training methods to obtain the result of diagnosis and classification.

The present invention uses a multi-task learning method to assist in training a lung adenocarcinoma and squamous cell carcinoma diagnosis and classification model based on PET/CT images through a diagnostic and classification neural network based on pathological images. This method aims to assist the training of the diagnostic classification model of lung adenocarcinoma and squamous cell carcinoma through pathological features, and improve the accuracy of lung cancer diagnostic classification based on PET/CT images. At the same time, pathological images are only used as prior knowledge in the training process, and do not need to be used as the input of the network in practical applications. Through the concept of multi-scale fusion, this method improves the accuracy of PET/CT images used in the diagnosis and classification of lung cancer, which is conducive to its further promotion and application as a means of early lung cancer diagnosis, and provides clinicians with patient diagnosis and follow-up treatment plans. At the same time, as a priori knowledge-assisted solution, pathological images can further improve the interpretability of pathological slices and help pathologists further extract pathological features.

Description of drawings

Fig. 1 is a flow chart of training the lung adenocarcinoma squamous cell carcinoma diagnostic model based on PET/CT;

Fig. 2 is a neural network structure diagram trained on a PET/CT-based lung adenocarcinoma squamous cell carcinoma diagnostic model.

Detailed ways

The following examples illustrate how to specifically apply this method to introduce pathological information into the PET/CT-based lung cancer diagnosis and classification network.

As shown in Figure 1-2, a kind of PET/CT-based lung adenocarcinoma squamous cell carcinoma diagnostic model training method of the present invention is specifically as follows:

Step 1: Obtain the corresponding PET/CT images, pathological images and lung adenocarcinoma squamous cell carcinoma diagnostic result data, establish a single input and output classification convolutional neural network, and combine the pathological images corresponding to PET/CT images and lung adenocarcinoma squamous cell carcinoma The cancer diagnosis results are imported into the classification convolutional neural network. Since the pathological image has the "gold standard" diagnostic classification effect for lung cancer, the classification convolutional neural network can be trained to have a very high classification accuracy for the input pathological image, and then saved Parameters to obtain the trained neural network A. And input the pathological image to a trained neural network A to obtain the pathological features; specifically include the following sub-steps:

(1.1) Establish a classification convolutional neural network. In this example, the classification convolutional neural network adopts the ResNet-50 structure. The specific structure is shown in Table 1:

Table 1: ResNet-50 network structure

(1.2) Establish a pathological slice data set for training. Since the original pathological images used are all whole-field digital slices (Whole Slide Image), they have extremely high resolution. In order to meet the input size requirements of the neural network and the computing resources required for network training, this embodiment cuts all the original pathological images into slices with a size of 224*224 in the preprocessing stage. The conditions to be met for slices are:

Among them, S _mask and S _all are the area of the image marked by the doctor's mark with lung cancer information and the whole area including the background, respectively. The purpose of formula (3) is to ensure that the input pathological slice images contain certain lung cancer classification features. Due to the different areas of lung cancer information representation images on different original pathological images, in order to ensure the balance of the sample size distribution during the training process, the Overlap-tile strategy is used in this example to keep the number of slices cut out from each original image. unanimous.

(1.3) Divide the sample into a training set and a verification set, wherein the verification set needs to ensure that the slice images of all cases are included, using the classification convolutional neural network established in the training set training step (1.1), the trained network needs to be guaranteed to be in the test The concentration has a very high accuracy, greater than or equal to 0.95. In this example, the trained neural network A is required to have a lung cancer diagnosis and classification accuracy of 0.99 for pathological slices.

(1.4) For each group of cases, there are corresponding PET, CT and pathological images. Input the pathological image verification set data established in step (1.3) into the neural network A trained in step 1 to obtain its corresponding diagnosis At the same time, extract the input of the previous layer of the output layer of the neural network A, and save it as the pathological feature of the case. The saved features need to be normalized in advance. In this example, the Sigmoid function (formula 4) is used for normalization. A PET/CT image corresponds to the features extracted from a pathological slice.

S(x) and x represent the input and output of the activation function, respectively.

Step 2: Establish an initial neural network model with multi-input and multi-task target output, use the paired PET and CT images as input, and the corresponding diagnostic classification results and pathological features as output, and train the initial neural network model. Specifically include the following sub-steps:

(2.1) Establish an initial neural network model for lung cancer diagnostic classification based on PET/CT, and include the fitting output of pathological features. In this example, the initial neural network model adopts the DenseNet-121 structure, and the main structure is shown in Table 2.

Table 2: DenseNet-121 network structure

(2.2) In the preprocessing stage, a lung cancer slice of size 224*224 is cut out from the original PET/CT image based on the location of the lung cancer lesion, and the slice is superimposed along the channel layer and then input into the initial neural network model. The target output of the network One is the diagnosis and classification result of lung cancer of the case, and the target output two is the pathological features of the case saved in step two (also normalized using the Sigmoid function). For the two target outputs and their corresponding true values, the errors are calculated separately (in this example, the target output 1 uses the cross-entropy loss function, and the target output 2 uses the mean square loss function), and the actual error of the network is:

Loss＝loss1*(1-φ)+loss2*φ (5)

Among them, loss1 and loss2 are the errors of target output 1 and 2 respectively, and φ is a hyperparameter, that is, the ratio of the error of target output 2 to the total network error, φ∈(0,1). The network parameters are updated and adjusted by the actual error, that is, the network is jointly trained by two target tasks, and a PET/CT-based lung adenocarcinoma and squamous cell carcinoma diagnostic classification model is obtained.

The trained lung adenocarcinoma squamous cell carcinoma diagnostic classification model can directly output diagnostic classification results based on PET/CT images without the participation of pathological data.

Thanks to the prior knowledge brought by the pathological images, the lung adenocarcinoma squamous cell carcinoma diagnostic classification model has higher accuracy for the lung cancer diagnostic classification of PET/CT images, while the pathological images are only used in the training process of the network. Application does not need to provide. Therefore, the trained PET/CT lung cancer diagnosis classification network has higher accuracy and better stability than the network trained solely by PET/CT images, which has clinical practical significance for the early diagnosis of lung cancer.

In addition, as a preferred solution, the training device for the PET/CT lung adenocarcinoma squamous cell carcinoma diagnostic classification model constructed based on the above training method includes:

As a preferred solution, a PET/CT diagnostic classification device for lung adenocarcinoma and squamous cell carcinoma based on the lung adenocarcinoma and squamous cell carcinoma diagnostic classification model obtained through training includes:

A data acquisition module, configured to acquire PET/CT images to be diagnosed;

Apparently, the above-mentioned embodiments are only examples for clear description, rather than limiting the implementation. For those of ordinary skill in the art, other changes or changes in different forms can be made on the basis of the above description. It is not necessary and impossible to exhaustively list all implementation modes here. However, the obvious changes or variations derived therefrom still fall within the protection scope of the present invention.

Claims

A kind of training method based on PET/CT lung adenocarcinoma squamous cell carcinoma diagnostic classification model, it is characterized in that, specifically comprises:

Obtain the corresponding PET/CT images, pathological images, and clinical lung adenocarcinoma and squamous cell carcinoma diagnosis data, and input the pathological images into a trained neural network A to obtain pathological features;

Construct an initial neural network model, take PET/CT images as input, predict pathological features and diagnosis results of lung adenocarcinoma and squamous cell carcinoma as output, and use the acquired PET/CT images, pathological features and clinical diagnosis results of lung adenocarcinoma and squamous cell carcinoma for training The initial neural network model is obtained based on PET/CT lung adenocarcinoma squamous cell carcinoma diagnostic classification model, specifically:

Calculate the error between the output of the initial neural network model and the corresponding true value, update the parameters of the initial neural network model according to the error, until the error is minimum, and obtain a diagnostic classification model based on PET/CT lung adenocarcinoma and squamous cell carcinoma; the loss is expressed as:

Loss＝loss1*(1-φ)+loss2*φ

Among them, loss1 and loss2 are the errors between the diagnostic results of lung adenocarcinoma and squamous cell carcinoma, pathological features and corresponding true values, φ is a hyperparameter, φ∈(0,1);

Wherein, the input of the neural network A is a pathological image, and the output is the diagnosis result of lung adenocarcinoma and squamous cell carcinoma, and the diagnostic classification accuracy is greater than or equal to 0.95; the pathological feature is the output of the feature extraction layer of the neural network A.
The training method according to claim 1, wherein the pathological image satisfies:

Among them, S mask and S all are the area of the image marked by the doctor's mark with lung cancer information and the whole area including the background, respectively.
The training method according to claim 1, wherein the neural network A adopts a ResNet-50 structure.
The training method according to claim 1, wherein the initial neural network model adopts a DenseNet-121 structure, and the input is a feature of fusion of PET and CT images along the channel dimension.
The training method according to claim 1, wherein said loss1 adopts a cross-entropy loss function, and loss2 adopts a mean square loss function.
A training device for a PET/CT lung adenocarcinoma squamous cell carcinoma diagnostic classification model based on any one of the training methods of claims 1-5, characterized in that it comprises:

A data acquisition unit, configured to acquire corresponding PET/CT images, pathological images, and diagnosis result data of lung adenocarcinoma and squamous cell carcinoma;

A pathological feature acquisition unit, configured to input pathological images to a trained neural network A to obtain pathological features;

The training unit is used to construct the initial neural network model, and the PET/CT image is used as input, the predicted pathological features and lung adenocarcinoma squamous cell carcinoma diagnosis results are output, and the data obtained by the data acquisition unit and the pathological feature acquisition unit are used to train the described The initial neural network model was used to obtain a diagnostic classification model for lung adenocarcinoma and squamous cell carcinoma based on PET/CT.
The training device according to claim 6, further comprising a data preprocessing unit for processing the corresponding PET/CT images and pathological images into pictures of the same size.
A PET/CT diagnostic classification device for lung adenocarcinoma and squamous cell carcinoma, characterized in that it comprises:

A data acquisition module, configured to acquire PET/CT images to be diagnosed;

Lung adenocarcinoma and squamous cell carcinoma diagnosis and classification module, for inputting the PET/CT image to be diagnosed into the PET/CT lung adenocarcinoma and squamous cell carcinoma diagnosis and classification model obtained by training according to any one of claims 1-6, to obtain a diagnosis classification results.