CN116152235A

CN116152235A - Cross-modal synthesis method for medical image from CT (computed tomography) to PET (positron emission tomography) of lung cancer

Info

Publication number: CN116152235A
Application number: CN202310409782.3A
Authority: CN
Inventors: 郑强; 陈莹钰
Original assignee: Yingruiyun Medical Technology Yantai Co ltd
Current assignee: Yingruiyun Medical Technology Yantai Co ltd
Priority date: 2023-04-18
Filing date: 2023-04-18
Publication date: 2023-05-23

Abstract

The invention relates to the field of medical images, in particular to a cross-modal synthesis method of medical images from lung cancer CT to PET, which comprises the following steps: performing data preprocessing on the CT data and the PET data, converting the CT image and the PET image in NIFIT format into two-dimensional slices in npy format, and manufacturing paired CT data sets and PET data sets; the frame model is designed and obtained under the regCAN mode based on the preprocessed CT data set and the PET data set, so that artificial intelligent synthesis of a PET image can be realized under the condition of not carrying out PET scanning, and a high-quality PET image containing lesion information is obtained.

Description

Cross-modal synthesis method for medical image from CT (computed tomography) to PET (positron emission tomography) of lung cancer

Technical Field

The invention relates to a cross-mode synthesis method of a medical image from CT (computed tomography) to PET (positron emission tomography) of lung cancer.

Background

PET is a positron emission tomography imaging technique widely used for staging and monitoring treatment of various cancers, and radioactive tracers used in PET are useful markers of many cancers compared with normal tissues, helping to detect and locate malignant tumors; while PET imaging has several advantages, the radioactive components used in PET can be somewhat dangerous and expensive overall, and thus PET is not widely used.

With the development of the deep learning algorithm in the field of medical image processing, the cross-modal medical image synthesis is realized by adopting the deep learning algorithm, so that the artificial intelligent synthesis of the PET image can be realized under the condition of not carrying out PET scanning, and a doctor is assisted in diagnosing lung cancer diseases.

Existing methods of medical image cross-modality synthesis generally include two modes based on generating antagonistic network developments: a supervised Pix2Pix model and an unsupervised cyclic consistency model; however, both modes are not ideal in practical applications; the Pix2Pix model, while having excellent performance, requires paired or well-aligned images as a basis; although the cyclic consistency model is less stringent to the training data, its best output is not the only solution; further, convolutional neural network-based generation of the countermeasure network is known as the most advanced model in many medical image synthesis tasks, however, the generator designed based on the convolutional neural network performs local processing by using a compact filter, and this induced deviation impairs learning of the context features, and for medical images of complex anatomy, there is still a great defect in capturing global features and remote interactive modeling and displaying lesion areas.

Disclosure of Invention

The embodiment of the invention provides a cross-modal synthetic method of a lung cancer CT-PET medical image, which is reasonable in design, adopts CT and PET data sets after data preprocessing to train a model based on a convolutional neural network, a full convolutional transducer and a regGAN technology, can capture local information and global context information of the image in the training process, learn the mapping relation of the CT domain image to the PET domain image, further accurately and rapidly complete the synthesis of the lung CT image to the PET image, and can better capture information of lesion tissues related to healthy tissues, and the synthetic PET image can clearly feed back the position of the lesion under the condition that PET scanning is not needed, so that the PET image with high quality and containing the lesion information can be obtained, the risk potential of specific crowd caused by radioactive components is greatly reduced, the economic burden of users is lightened, good clinical application is displayed, and the problems existing in the prior art are solved.

The technical scheme adopted by the invention for solving the technical problems is as follows:

a method for cross-modal synthesis of medical images of lung cancer CT to PET, the synthesis method comprising the steps of:

s1, carrying out data preprocessing on CT data and PET data, converting a NIFIT format CT image and a PET image into npy format two-dimensional slices, and manufacturing a paired CT data set and a paired PET data set;

s2, designing a frame model based on the preprocessed CT data set and the PET data set in a regCAN mode, wherein the frame model comprises a generator, a discriminator and a registration network; the generator is used for learning the mapping relation from the source domain image to the target domain image by adopting a mixed CNN-transducer structure, and referring to the real image distribution, so that the self-generated image is more real to cheat the discriminator; the discriminator is used for judging the received image generated by the generator and the real image by adopting a convolutional neural network structure; the registration network is used for correcting the output of the generator by adopting a deformable registration based on Resune to obtain a random deformation field;

the generator adopts a structure of an encoder-an intermediate information bottleneck layer-a decoder, and convolution blocks are respectively arranged in the encoder and the decoder so as to utilize the local precision advantage of convolution operation; the intermediate information bottleneck layer adopts a mixed structure of a full convolution transducer module and a residual convolution block so as to capture the positions and the structures of lesions related to healthy tissues and related to pathological distribution, and can realize high localization and improve the context sensitivity and high sense of reality in a synthetic image; the encoder and decoder are used for maintaining local precision and induced bias in the learning structure through the convolution layer;

s3, inputting paired CT data sets and PET data sets into the generated frame model for training, and jointly evolving the generator and the discriminator in the alternate iterative game process so as to enable the generator to learn the data distribution closest to the real sample;

and S4, storing the trained model, and inputting the npy two-dimensional slice into the trained model to obtain a corresponding PET image.

The data preprocessing of the CT data and the PET data comprises the following steps:

s1.1, converting CT data and PET data in an original DICOM format acquired from a hospital into NIFIT format by utilizing a simpleITK library;

s1.2, using an FSL tool to linearly register the PET image to the CT image;

s1.3, adjusting the window width of the CT image to 4000, adjusting the window level to 1000, and normalizing the CT image and the PET image to be between [ -1,1];

s1.4, converting the CT image and the PET image in NIFIT format into two-dimensional slices in npy format, and manufacturing to obtain paired CT data sets and PET data sets.

Converting the original DICOM format CT data and PET data acquired from the hospital into the nifet format using the SimpleITK library comprises the steps of:

s1.1.1, constructing a DICOM sequence file reader, and converting data into an array format after packaging and integration;

s1.1.2, obtaining DICOM sequence file basic information, converting an array format into an img format and storing the img format into a NIFIT format.

The use of an FSL tool to linearly register PET images to CT images includes the steps of:

s1.2.1, using an FSL tool to set the CT image as a fixed image and the PET image as a floating image;

s1.2.2 the degree of freedom of the CT image and the PET image is set to 12, interaction information is set by adopting a cost function, and tri-linear interpolation is selected by adopting an interpolation algorithm, so that the PET image is configured on the CT image.

The full convolution transducer module includes a convolution attention module for learning a remote semantic context and a view focusing module for learning local and global contexts using multi-resolution hole convolution;

in the convolution attention module, the input is first mapped into a specified number of feature embeddings by a convolution embedding layer; second convolution projection projects each feature embedding by depth separable convolution to generate Q, K and V; finally, calculating a multi-head self-attention mechanism for Q, K and V embedded by each feature;

in the view focusing module, a multi-branch air conditioner convolution layer is adopted to acquire a large amount of spatial context information by utilizing receptive fields with different sizes, so that fine granularity information of medical images is extracted, and characteristics of the multi-branch convolution layer are fused through summation.

After the generator G, a registration network R is added as a label noise model to improve the quality of the synthesized image G (x), and specifically, the correction loss is as follows:

where x is the CT image of the input source domain,

PET image for target region, +.>

Is a deformation field +.>

Representing a resampling operation;

further, the smoothness of the deformation field can be estimated and the gradient of the deformation field minimized by a loss function formula:

where x is the CT image of the input source domain,

PET image for target region, +.>

Is a deformation field。

The antagonism objective function between the generator G and the discriminator D is:

where x is the CT image of the input source domain,

PET image for target region, +.>

Is the deformation field.

The dataset was represented at 8: the ratio of 2 is randomly divided into a training set and a test set.

By adopting the structure, the CT data and the PET data are subjected to data preprocessing, and a CT image and a PET image in NIFIT format are converted into two-dimensional slices in npy format, so that paired CT data sets and PET data sets are manufactured; capturing local information and global context information of an image through the designed frame model to learn the mapping relation from the CT domain image to the PET domain image; training the frame model to enable the generator to learn the data distribution closest to the real sample; the frame model after training is stored, and the corresponding PET image is obtained by combining the two-dimensional slice with the npy format, so that the method has the advantages of accuracy, practicability, economy and safety.

Drawings

FIG. 1 is a schematic diagram of a cross-modality medical image synthesis model of the present invention.

Detailed Description

In order to clearly illustrate the technical features of the present solution, the present invention will be described in detail below with reference to the following detailed description and the accompanying drawings.

As shown in fig. 1, a method for cross-modal synthesis of a medical image of lung cancer CT to PET, the synthesis method comprising the steps of:

s1.2, using an FSL tool to linearly register the PET image to the CT image;

where x is the CT image of the input source domain,

PET image for target region, +.>

Is a deformation field +.>

Representing a resampling operation;

where x is the CT image of the input source domain,

PET image for target region, +.>

Is the deformation field.

where x is the CT image of the input source domain,

PET image for target region, +.>

Is the deformation field.

The working principle of the medical image cross-mode synthesis method from lung cancer CT to PET in the embodiment of the invention is as follows: based on convolutional neural network, full convolutional transducer and RegGAN technology, CT and PET data sets after data preprocessing are adopted for training, local information and global context information of images can be captured in the training process, the mapping relation from CT domain images to PET domain images is learned, further, synthesis from lung CT images to PET images is accurately and rapidly completed, unlike a medical image synthesis model based on convolutional neural network and a generation countermeasure network mode based on a generator and a discriminator, information of lesion tissues related to healthy tissues can be better captured, the synthesized PET images can be clearly fed back to the positions of lesions, so that artificial intelligent synthesis of PET images can be realized without PET scanning, high-quality PET images containing lesion information can be obtained, risks caused by radioactive components to specific people are greatly reduced, economic burden of users is lightened, and good clinical application potential and popularization application range are displayed.

Further, it is an object of the present application to construct a deep learning network that takes advantage of the benefits of convolutional neural networks and full convolutional transformers in computer vision and incorporates RegGAN modes to improve medical image synthesis quality.

In the overall scheme, the synthesis method mainly comprises the following steps: data preprocessing, model design, model training and picture synthesis.

The data preprocessing step is to perform data preprocessing on CT data and PET data, convert a CT image and a PET image in NIFIT format into a two-dimensional slice in npy format, and manufacture paired CT data sets and PET data sets; specifically, firstly, CT data and PET data in an original DICOM format acquired from a hospital are converted into NIFIT format by utilizing a simpleITK library; then using the FSL tool to linearly register the PET image to the CT image; again adjusting the window width of the CT image to 4000, the window level to 1000, and normalizing the CT image and the PET image to between [ -1,1]; and finally converting the CT image and the PET image in NIFIT format into two-dimensional slices in npy format, and making into paired CT data sets and PET data sets.

The DICOM format is a medical digital imaging and communication format and is the original data collected in a hospital; the NIFIT format is a neuroimaging informatics technical initiative format, which is converted and patient privacy data is deleted from the original data.

After data preprocessing, designing a frame model in a RegCAN mode, wherein the frame model comprises a generator, a discriminator and a registration network; the generator is used for learning the mapping relation from the source domain image to the target domain image by adopting a mixed CNN-transducer structure, and referring to the real image distribution, so that the self-generated image is more real to cheat the discriminator; the discriminator is used for judging the received image generated by the generator and the real image by adopting a convolutional neural network structure; the registration network is used to correct the output of the generator using a resune-based deformable registration to obtain a random deformation field.

Wherein the full convolution transform module-based generator adopts a structure of an encoder-an intermediate information bottleneck layer-a decoder, specific task information is extracted through the intermediate information bottleneck layer, and the encoder and the decoder use a convolution layer to keep local precision and induced bias in the learning structure representation.

Preferably, the full convolution transform module includes a convolution attention module for learning the remote semantic context and a view focusing module for learning the local and global context using multi-resolution hole convolution.

In the convolution attention module, the input is first mapped into a specified number of feature embeddings by a convolution embedding layer; second convolution projection projects each feature embedding by depth separable convolution to generate Q, K and V; finally, the calculation of a multi-head self-attention mechanism is carried out on Q, K and V embedded by each feature.

Further, the discriminator of the present application uses a classifier consisting of five convolutional layers and one fully-connected layer.

For the registration network, the result synthesized by the generator is corrected by adopting the deformable registration based on Resune, and the deformable vector field is predicted by training a convolutional neural network model, so that a time-consuming gradient derivation process is omitted, and the calculation efficiency can be improved; therefore, in order to obtain a better synthesis effect, a registration network R is added as a label noise model after the generator G, so that the quality of the synthesized image G (x) is improved.

Specifically, the correction loss is:

where x is the CT image of the input source domain,

PET image for target region, +.>

Is a deformation field +.>

Representing a resampling operation.

Further, and by the following loss function formula, to evaluate the smoothness of the deformation field and minimize the gradient of the deformation field:

the underlying network framework still involves the GAN's null and game ideas, with the generator G and discriminator D being trained continuously to compete with each other during the training process and ultimately be introduced into the desired ideal state. In this process, a target domain image G (x) is generated from the input CT image x by a training generator G, and a training discriminator D can distinguish a real PET image y from a synthesized PET image G (x). The objective function of the resistance loss is as follows:

wherein the generator G and the discriminator D oppose each other, G minimizes the objective function, and D maximizes the objective function.

For the experimental setup of the present application, the dataset was set at 8:2 into training and testing sets, selecting lung CT and PET two-dimensional slices of each patient from scanned lung cancer patient whole body CT/PET data, resampling the slices to 256 x 256 size, and normalizing to [ -1,1]; the deep learning model is trained in a specific graphic processor by adopting a corresponding language, and an optimizer is used, so that a corresponding medical image is obtained.

The frame model can better capture the information of pathological tissues related to healthy tissues, and the synthesized PET image can clearly see the positions of the focuses.

In general, three objective indexes are used to quantitatively evaluate the synthesized PET image, including normalized average absolute error, peak signal-to-noise ratio and structural similarity indexes; the normalized average absolute error is based on the difference between the pixel measurement composite image and the real image, and the lower the value is, the better the quality of the composite image is indicated; the peak signal-to-noise ratio is an approximate value of human reconstruction quality perception, can quantify the quality of image synthesis, and is better as the value is higher; the structural similarity quantifies the overall similarity between the composite image and the real image, with a value closer to 1 indicating that the composite image is more similar to the real target image.

After quantitative evaluation, it can be seen from the quantitative evaluation result that compared with the real PET image, the synthesized PET image has lower normalized average absolute error, higher peak signal-to-noise ratio and structure similarity index approaching 1, which indicates that the generated PET image has higher quality and similar shape with the real PET image.

It is specifically stated that the present application also has obvious innovations and improvements compared to the prior art, and is now specifically described in terms of the characteristics of each technology:

in the application file named as a cross-mode medical image synthesis method, a system, a terminal and a storage medium, the method is improved on the basis of 3D CGAN, and a generator is U-Net; the loss function used by the generator is MAE, and the loss function used by the discriminator is MSE; the method is older and lacks innovation, does not add a step of image preprocessing, does not add quantitative evaluation and qualitative evaluation of image generation, and can only improve the synthesis effect of the multi-mode images of the specific subjects under the condition of limited paired data to a certain extent.

In the application file named as a multi-mode medical image synthesis method based on a generated type countermeasure network, the method is only suitable for the problem of single-mode input, the step of image preprocessing is not added, the quantitative evaluation of experimental results is only added, the qualitative evaluation of experimental results is not added, fine adjustment is not carried out aiming at a single task, and the clinical application prospect is limited to a certain extent.

In the application file named as a medical image cross-mode synthesis method, a system and a readable storage medium, a swin-transducer block is added in jump connection of the lower three layers of a generator to extract global features, the improvement is simpler, and only experimental result quantitative evaluation is added and experimental result qualitative evaluation is not added.

Compared with the three application documents, the method is only aimed at synthesizing lung cancer CT to PET images, has more pertinence, designs a model on the basis of regGAN, and adds a registration network in addition to a generator and a discriminator in the whole framework; the registration network is used for solving a random deformation field to correct the output of the generator; the generator uses the structure of the encoder-intermediate information bottleneck layer-decoder. The encoder and decoder contain only convolution blocks to take advantage of the local accuracy of the convolution operation; the intermediate information bottleneck layer adopts a mixed structure of a full convolution transducer and a residual convolution block, so that the positions and structures of lesions related to healthy tissues, which are related to pathological distribution, can be captured better.

In summary, the cross-modal synthesis method for the lung cancer CT to PET medical image in the embodiment of the present invention is based on the convolutional neural network, the full convolutional transducer and the RegGAN technology, and uses the CT and PET data sets after data preprocessing to train the model, so that the local information and the global context information of the image can be captured in the training process, the mapping relationship from the CT domain image to the PET domain image can be learned, further, the synthesis from the lung CT image to the PET image can be completed accurately and rapidly, unlike the medical image synthesis model based on the convolutional neural network and the generation countermeasure network mode based on the generator and the discriminator, the synthesized PET image can better capture the information of the lesion tissue related to the healthy tissue, and can also feed back the position of the lesion more clearly, thereby realizing the artificial intelligent synthesis of the PET image without performing PET scanning, obtaining the PET image with high quality, greatly reducing the risk of specific crowd caused by the radioactive component, reducing the economic burden of the user, and displaying good clinical application potential and popularization application range.

The above embodiments are not to be taken as limiting the scope of the invention, and any alternatives or modifications to the embodiments of the invention will be apparent to those skilled in the art and fall within the scope of the invention.

The present invention is not described in detail in the present application, and is well known to those skilled in the art.

Claims

1. The cross-modal synthesis method of the medical image from lung cancer CT to PET is characterized by comprising the following steps of:

2. The method of cross-modality synthesis of a medical image of lung cancer CT to PET of claim 1, wherein the data preprocessing of the CT data and the PET data comprises the steps of:

s1.2, using an FSL tool to linearly register the PET image to the CT image;

3. The method of cross-modality synthesis of a lung cancer CT to PET medical image of claim 2, wherein converting the original DICOM format CT data and PET data acquired from the hospital to the NIFIT format using a SimpleITK library comprises the steps of:

4. A method of cross-modality synthesis of a lung cancer CT to PET medical image according to claim 2, wherein the use of FSL tools to linearly register the PET image onto the CT image comprises the steps of:

5. The method for cross-modal synthesis of a medical image of lung cancer CT to PET of claim 1, wherein: the full convolution transducer module includes a convolution attention module for learning a remote semantic context and a view focusing module for learning local and global contexts using multi-resolution hole convolution;

6. The method for cross-modal synthesis of a medical image of lung cancer CT to PET of claim 1, wherein: after the generator G, a registration network R is added as a label noise model to improve the quality of the synthesized image G (x), and specifically, the correction loss is as follows:

wherein x is CT image of input source domain, < >>

PET image for target region, +.>

Is a deformation field +.>

Representing a resampling operation;

wherein x is CT image of input source domain, < >>

PET image for target region, +.>

Is the deformation field.

7. The method of cross-modality synthesis of medical images of lung cancer CT to PET of claim 6, wherein the antagonistic objective function between generator G and discriminator D is:

wherein x is CT image of input source domain, < >>

PET image for target region, +.>

Is the deformation field.

8. The method for cross-modal synthesis of a medical image of lung cancer CT to PET of claim 1, wherein: the dataset was represented at 8: the ratio of 2 is randomly divided into a training set and a test set.