CN117115046A - Method, system and device for enhancing sparse sampling image of radiotherapy CBCT - Google Patents

Method, system and device for enhancing sparse sampling image of radiotherapy CBCT Download PDF

Info

Publication number
CN117115046A
CN117115046A CN202311378973.4A CN202311378973A CN117115046A CN 117115046 A CN117115046 A CN 117115046A CN 202311378973 A CN202311378973 A CN 202311378973A CN 117115046 A CN117115046 A CN 117115046A
Authority
CN
China
Prior art keywords
cbct
image
radiotherapy
sampling
sparse
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202311378973.4A
Other languages
Chinese (zh)
Other versions
CN117115046B (en
Inventor
夏启胜
邹尧
王海
熊强
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ruishi Wisdom Beijing Medical Technology Co ltd
China Japan Friendship Hospital
Original Assignee
Ruishi Wisdom Beijing Medical Technology Co ltd
China Japan Friendship Hospital
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ruishi Wisdom Beijing Medical Technology Co ltd, China Japan Friendship Hospital filed Critical Ruishi Wisdom Beijing Medical Technology Co ltd
Priority to CN202311378973.4A priority Critical patent/CN117115046B/en
Publication of CN117115046A publication Critical patent/CN117115046A/en
Application granted granted Critical
Publication of CN117115046B publication Critical patent/CN117115046B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/003—Reconstruction from projections, e.g. tomography
    • G06T11/008—Specific post-processing after tomographic reconstruction, e.g. voxelisation, metal artifact correction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0475—Generative networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/094—Adversarial learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/0985—Hyperparameter optimisation; Meta-learning; Learning-to-learn
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G06T2207/10081—Computed x-ray tomography [CT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2211/00—Image generation
    • G06T2211/40—Computed tomography
    • G06T2211/424—Iterative
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2211/00—Image generation
    • G06T2211/40—Computed tomography
    • G06T2211/428—Real-time
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Biophysics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a method, a system and a device for enhancing a sparse sampling image of radiotherapy CBCT, and belongs to the technical field of medical imaging. Comprising the following steps: constructing a residual intensive generation countermeasure (RDN-GAN) deep learning network model; collecting and processing spiral CT data to obtain paired CBCT sparse sampling-full sampling pre-training data sets; finishing the pre-training of the RDN-GAN model; collecting and processing radiotherapy CBCT two-dimensional projection data to obtain a paired CBCT sparse sampling-full sampling optimization training data set; completing the optimization training of the RDN-GAN model; the optimized RDN-GAN model is used for carrying out image enhancement on the radiotherapy CBCT sparse sampling image to obtain a high-quality radiotherapy CBCT image which is used for the follow-up procedure of radiotherapy.

Description

Method, system and device for enhancing sparse sampling image of radiotherapy CBCT
Technical Field
The invention relates to the technical field of medical imaging, in particular to a method, a system and a device for enhancing a CBCT sparse sampling image for radiotherapy.
Background
Radiation therapy is one of the important means for treating malignant tumors, and improvement of radiation therapy accuracy is always a research hot spot in the field of radiation therapy, and accurate patient positioning and planning are key factors for ensuring radiation therapy effects. In recent years, cone Beam CT (CBCT) imaging technology is widely applied to radiotherapy, and CBCT can provide high-quality three-dimensional anatomical structure images, help doctors to accurately finish positioning verification, tumor and organ positioning, and accordingly achieve individuation and precision of radiotherapy. The development and application of CBCT imaging technology provides new ideas and technical means for further improvement of radiotherapy.
However, conventional radiotherapy CBCT imaging is typically based on full dose and full sampling modes, i.e. three-dimensional image reconstruction using higher bulb currents and full angle two-dimensional projection data. However, this mode has many disadvantages, firstly, the scan mode uses a large amount of radiation to ensure that a high quality image is obtained, which adversely affects the health of the patient; second, the full sampling mode requires a long imaging time, which may cause inconvenience to the patient and increase the complexity of the operation; in addition, the full sampling mode may introduce motion artifacts and noise during imaging, reducing the quality and accuracy of the imaging. Therefore, on the premise of ensuring the imaging quality, the radiation dosage of imaging is obviously reduced, the imaging time is shortened, the precision and the safety of radiotherapy are improved, and the radiation imaging method and the radiation imaging device are the technical problems to be solved in the field of radiotherapy CBCT imaging.
In order to overcome the above problems, some sparse sampling-based radiotherapy CBCT imaging methods have emerged in recent years. These methods reduce the amount of data required for imaging by selectively sampling two-dimensional projection data of a patient, thereby reducing imaging time. In addition, sparse sampling can also reduce the exposure time of a patient to radiation dose, and improve the safety of the patient. However, due to sparsity of data, sparse sampled radiotherapy CBCT imaging often results in severe degradation of imaging quality, affecting the accuracy of radiotherapy.
Therefore, how to improve the sharpness of CBCT reconstructed images in a sparse sampling state becomes a key issue. Many groups have developed this, compressed sensing technology is a technique often used in CBCT sparse sampling, but has limited effectiveness. In recent years, many more advanced methods and techniques have been introduced into the field of medical imaging, including image enhancement techniques based on deep learning, artificial intelligence deep learning is a generic term for a type of pattern analysis method, and deep learning techniques are increasingly applied to various fields due to rapid development in recent years, and Convolutional Neural Network (CNN) model is one of the most important methods in the deep learning techniques. The CNN is composed of multiple layers of neurons, so that the CNN has strong characteristic learning capability, the basic principle is that a neural network is used for automatically learning beneficial image characteristics, then the characteristic representation of a sample in an original space is transformed into another new characteristic space through layer-by-layer characteristic transformation, and the image enhancement task is carried out after continuous characteristic extraction and transformation. Compared with the traditional machine learning method, the deep learning model has more parameters and stronger expression capability. The existing CNN-based image enhancement technology has greatly advanced, such as patent number CN113706409a, and the disclosed artificial intelligence-based CBCT image enhancement method, device and storage medium, but are limited by the limitations of the existing algorithm technology and the difficulty in acquiring high-quality large sample paired image data for training, and the existing technology still cannot meet the accuracy requirement of clinical diagnosis.
Therefore, how to provide a method, system and device for sparse sampling image enhancement of radiotherapy CBCT is a problem that needs to be solved by those skilled in the art.
Disclosure of Invention
In view of the above, the present invention provides a method, a system and a device for enhancing sparse sampling image of CBCT for radiotherapy, which are used for solving the technical problems in the prior art.
In order to achieve the above object, the present invention provides the following technical solutions:
in one aspect, the invention provides a method for enhancing a sparse sampled image of radiotherapy CBCT, comprising the following steps:
step 1: constructing an RDN-GAN deep learning network model;
step 2: and collecting spiral CT image data of a patient, generating multi-angle two-dimensional projection data by utilizing orthographic projection, and splitting the two-dimensional projection data into sparse sampling (30-180 projections) and full sampling (300-900 projections) data. The projection parameters are consistent with the existing radiotherapy positioning CBCT imaging parameters, the SDD (source to detector distance) is 1500 mm, and the SOD (source to object center distance) is 1000 mm.
Step 3: and respectively reconstructing the split two-dimensional projection data into three-dimensional tomographic image data by using a CBCT three-dimensional reconstruction algorithm to form paired sparse sampling CBCT reconstruction data and full sampling CBCT reconstruction data, wherein the paired data are used as a pre-training data set of a deep learning algorithm model.
Step 4: and (3) utilizing the paired data set to complete the pre-training of the residual intensive generation countermeasure (RDN-GAN) deep learning network model, and obtaining a pre-training model.
Step 5: and collecting patient radiotherapy positioning CBCT two-dimensional projection data, splitting the projection data into sparse sampling data (30-180 projections) and full sampling projection data (300-900 projections), respectively reconstructing the sparse sampling two-dimensional projection data and the full sampling two-dimensional projection data into three-dimensional tomographic image data by using a CBCT three-dimensional reconstruction algorithm to form paired sparse sampling CBCT reconstruction data and full sampling CBCT reconstruction data, and taking the paired sparse sampling CBCT reconstruction data and full sampling CBCT reconstruction data as an additional training data set of the deep learning algorithm model to finish optimization training of the model and obtain an optimized RDN-GAN deep learning network model, namely a deep learning-based sparse sampling image enhancement algorithm model.
Alternatively, if the original two-dimensional projection data is difficult to acquire, the radiotherapy CBCT fully sampled reconstruction data can be utilized for carrying out orthographic projection to acquire the two-dimensional projection data.
Step 6: and taking the sparse sampled CBCT three-dimensional reconstructed image picture as input, performing image enhancement processing on the input image through the RDN-GAN deep learning network model which is well pre-trained and optimized, and outputting to obtain a high-quality radiotherapy CBCT image so as to improve the image quality and details, and using the CBCT image to follow-up procedures of radiotherapy, thereby improving the accuracy of radiotherapy positioning and tumor positioning.
Step 7: and comparing the CBCT image after image enhancement with a full-sampled high-quality CBCT image (group trunk), and evaluating the image quality and the enhancement effect.
In the steps 3 and 5, an FDK (Feldkamp-Davis-Kress) reconstruction algorithm or an iterative reconstruction algorithm is adopted to perform CBCT three-dimensional reconstruction, and in general, the iterative reconstruction algorithm has better reconstruction definition than the conventional FDK reconstruction algorithm, especially under the condition of sparse sampling, in some cases, such as the case that the original two-dimensional projection data of the CBCT cannot be acquired, when the input sparse sampling reconstruction image of the radiotherapy CBCT during image enhancement is obtained by FDK reconstruction, the FDK reconstruction algorithm can also be selected.
In step 4, the deep learning network model employs a residual error density block RDB (Residual Dense Block) to improve the performance and accuracy of the network. Meanwhile, the invention adopts the countermeasure network (GAN) to realize image enhancement, thereby improving the image quality and detail.
Further, the RDN-GAN deep learning network model includes a residual dense network including a convolution layer, a residual dense block, a cascade network, up-sampling, and deconvolution layer, and an countermeasure network including a convolution layer, an activation function, a normalization layer, and a density network. The residual dense network is used as a generator, the residual dense block is composed of a convolution layer, an activation function and a cascade layer and is used for improving the performance and the precision of the network, the up-sampling layer is used for up-sampling a CBCT image with low resolution, and the countermeasure network is used as a discriminator and is used for judging whether the enhanced image is a real CBCT image or not. By processing the input CBCT image with a plurality of residual error density blocks, the quality and detail of the image can be gradually improved. Meanwhile, the discriminator is used for judging whether the high-quality CBCT image generated by the generator approximates to the real CBCT image. And through repeated training of the generator and the discriminator, the model is continuously optimized, and finally, the high-quality image enhancement of CBCT sparse sampling imaging is realized.
Further, the residual dense generation countermeasure (RDN-GAN) network includes:
the system comprises a residual dense network and an countermeasure network, wherein the residual dense network is used as a generation network of a network architecture and comprises a convolution layer, a residual dense block, a cascade network, up-sampling and deconvolution layers, and the residual dense block consists of the convolution layer, an activation function and the cascade layer. The countermeasure network comprises a convolution layer, an activation function, a normalization layer and a density network;
wherein, the input and output relation of each density network is as follows:
wherein,output of layer c convolutional network for layer d density network, +.>And->Let go of->Layer density networkInput and output of>For RELU activation function, +.>Weights for each convolution layer, +.>Cascading feature map generated for layer-1 density network, ++>Then the output of the 1 st convolutional network for the d-th density network,the output of the c-1 th convolutional network is the d-th density network.
Constructing features and introducing perceptual loss functions prior to RELU activation functionsThe specific calculation formula is as follows:
wherein,for the content loss function->To combat the loss function;
wherein,representing reconstructed pictures->Is a true pictureProbability of->Output representing a residual dense network, +. >Input picture representing countermeasure network, < >>Representing the weights and biases of the generated network,Nin order to train the input total sample,nindexing each training sample;
calculation using MSE and PSNRIs calculated by the following formula:
；
；
wherein,at the maximum value of the pixel,m,nthe number of horizontal pixels and the number of vertical pixels respectively representing the resolution of the input image,i,jand the coordinate index number corresponding to each pixel is given, and I and K respectively represent the image after image enhancement and the full-sampling image.
It is also possible to define the loss function of the residual dense generation countermeasure network (RDN-GAN) as:
wherein,for the perceptual loss function, it is composed of a plurality of loss function weights,Nin order to train the total number of samples entered,nindex for each training sample, +.>Output representing a residual dense network, +.>Input samples for some sparse sample of training set, +.>For a certain full sample of the training set, +.>Weights and biases representing the network, +.>The optimal solution of the network model is obtained through training.
Wherein the training set refers to either a paired CBCT sparse sampling-full sampling optimized training data set or a paired CBCT sparse sampling-full sampling pre-training data set.
Thus RDN-GAN deep learning network model is aimed to be trained and obtained So that the perceptual loss function between the sparsely sampled samples and the fully sampled samples is minimized.
Further, the step of inputting the paired data set into a pre-constructed RDN-GAN network for training to obtain a radiotherapy CBCT sparse sampling image enhancement network model comprises the following steps:
performing data expansion on the sparse sampling-full sampling paired data set, and dividing the normalized image set into a training set, a verification set and a test set;
and inputting the training set into a residual dense network and an countermeasure network, and obtaining a training model according to the loss function of the residual dense network and the loss function of the countermeasure network.
And verifying and testing the verification model by using the verification set and the test set, if the verification and test results do not meet the prediction probability threshold, training the generated network and the countermeasure network again, and if the verification and test results meet the prediction probability threshold, completing training of the RDN-GAN network model.
A system for sparse sampled image enhancement of radiotherapy CBCT, comprising:
a projection acquisition module 201 for acquiring multi-angle two-dimensional projection data of a patient;
a three-dimensional reconstruction module 202, configured to reconstruct two-dimensional projection data of the patient into CBCT three-dimensional tomographic image data;
The image enhancement module 203 is configured to input a low-quality image reconstructed by sparse sampling of the radiotherapy CBCT into a pre-trained and constructed RDN-GAN network model, so as to obtain an image-enhanced radiotherapy CBCT image;
the image display and output module 204 is used for displaying and outputting the image-enhanced radiotherapy CBCT image in a Dicom format for the subsequent procedure of radiotherapy.
A radiation therapy CBCT sparse sampled image enhancement device, comprising: the device comprises a radiotherapy CBCT imaging assembly, a three-dimensional reconstruction assembly, an image enhancement assembly and an image display and output assembly;
the radiotherapy CBCT imaging component comprises an X-ray source, a flat panel detector, a bracket and a motion platform and is used for CBCT imaging; the three-dimensional reconstruction component is used for reconstructing two-dimensional projection data obtained by the radiotherapy CBCT imaging system into a three-dimensional CBCT image; the image enhancement component stores a pre-training and constructed radiotherapy CBCT sparse sampling image enhancement system program based on deep learning, and enhances the low-quality CBCT image transmitted by the three-dimensional reconstruction component into a high-quality radiotherapy CBCT image; the image display and output component is used for displaying the image after the image enhancement and outputting the image in a Dicom format.
Compared with the prior art, the invention discloses a method, a system and a device for enhancing the sparse sampling image of the radiotherapy CBCT, which can obviously reduce the radiation dosage of imaging, quicken the imaging speed and improve the precision and the safety of the radiotherapy on the premise of guaranteeing the imaging quality, and has wide clinical application prospect in the field of tumor radiotherapy. The method has the following specific beneficial effects:
(1) According to the invention, a large number (500-1000 or more) of spiral CT data are firstly adopted to generate CBCT sparse sampling and full sampling paired data through orthographic projection, sparse sampling CBCT imaging image enhancement algorithm model pre-training is carried out, then small sample (200-500) radiotherapy CBCT two-dimensional projection data paired data are used for model optimization training, and an algorithm model training mode of big sample pre-training and small sample additional training is adopted, so that the difficulty in acquiring a large number of high-quality radiotherapy CBCT samples is solved, the robustness of an algorithm model is increased, and the problems of difficult acquisition of radiotherapy CBCT data and poor image quality are solved;
(2) The three-dimensional reconstruction of the CBCT two-dimensional projection data is carried out by adopting an iterative reconstruction algorithm, and the iterative reconstruction algorithm has better reconstruction definition than the traditional FDK reconstruction algorithm, particularly has higher sparse sampling tolerance under the condition of sparse sampling, and can realize high-definition imaging under lower sampling quantity;
(3) The invention adopts the latest improved deep learning algorithm model based on residual dense generation countermeasure (RDN-GAN) network architecture, the algorithm shows the effect exceeding that of the prior deep learning algorithm in the traditional computer vision field, and the improved deep learning algorithm is applied to the radiation treatment CBCT sparse sampling image enhancement field after being improved and perfected, so that the image quality can be effectively improved and the image detail can be enhanced under the sparse sampling condition; compared with the prior CNN deep learning image enhancement algorithm, the RDN-GAN can better maintain the spatial resolution of imaging, and can obtain a high-quality CBCT image under the condition of extremely sparse sampling CBCT imaging;
(4) According to the radiation treatment CBCT sparse sampling image enhancement method, the sampling quantity is greatly reduced, the stand can scan at a faster rotation speed, the imaging speed can be increased, the imaging time can be reduced, the radiation dose of a patient can be reduced, meanwhile, the high imaging definition can be maintained, the safety of the patient during radiation treatment can be improved, the artifacts related to the movement of the patient can be reduced, and the accuracy and the treatment effect of the radiation treatment can be improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of the method of the present invention;
FIG. 2 is a schematic diagram of a flow chart for constructing an RDN-GAN deep learning network model according to the invention;
fig. 3 is a schematic view of sparse sampling scanning imaging of radiotherapy CBCT provided in embodiment 1;
fig. 4 is a schematic diagram of a result of sparse sampling imaging reconstruction of radiotherapy CBCT provided in example 1;
Fig. 5 is a logic block diagram of a deep learning-based radiotherapy CBCT sparse sampling image enhancement method provided in embodiment 1;
FIG. 6 is a schematic diagram of the RDN-GAN deep learning network model according to the present invention;
FIG. 7 is a graph showing the result of enhancing the head imaging image according to the embodiment;
FIG. 8 is a graph showing the result of image enhancement of chest imaging provided by an embodiment of the present invention;
fig. 9 is a system schematic diagram of a sparse sampled image enhancement method of radiotherapy CBCT based on deep learning according to embodiment 2;
fig. 10 is a schematic structural diagram of a device for enhancing CBCT sparse sampling image by applying deep learning in embodiment 3.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention aims to overcome the defects of the prior art and provides a method, a system and a device for enhancing a radiotherapy CBCT sparse sampling image. And generating paired sparse sampling and full sampling data sets by using the collected radiotherapy CBCT original projection data to perform optimization training of the algorithm model, so as to obtain an optimized sparse sampling image enhancement algorithm model. And performing sparse sampling radiotherapy CBCT image enhancement through the optimized RDN-GAN deep learning network model. The invention can greatly improve the definition of the reconstructed image under the sparse sampling state of the CBCT of the dilute radiation therapy, greatly reduce the radiation dose of CBCT imaging of the radiation therapy, reduce the imaging time of the CBCT and improve the safety and the accuracy of the radiation therapy of a patient.
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.
Referring to fig. 1, the embodiment of the invention discloses a method for enhancing a radiotherapy CBCT sparse sampling image, which comprises the following steps:
the method comprises the steps of constructing a radiotherapy CBCT sparse sampling image enhancement algorithm model based on deep learning, and naming the model as follows: (residual dense generation countermeasure) RDN-GAN deep learning network model;
collecting spiral CT image data, and processing the spiral CT data to obtain a paired radiotherapy CBCT sparse sampling-full sampling imaging pre-training data set; completing the pre-training of the RDN-GAN deep learning network model based on the sparse sampling-full sampling paired data set;
collecting radiotherapy CBCT two-dimensional projection data of the same imaging part, and processing the radiotherapy CBCT two-dimensional projection data to obtain a sparse sampling-full sampling imaging optimization training data set of CBCT pairing; completing optimization training of the RDN-GAN deep learning network model based on the paired CBCT sparse sampling-full sampling imaging data set;
and inputting the sparse sampling reconstructed low-quality radiotherapy CBCT image data into an optimally trained RDN-GAN deep learning network model, and outputting to obtain a radiotherapy CBCT image with high imaging quality.
Referring to fig. 2, a schematic flow chart of constructing an RDN-GAN deep learning network model provided by the invention is shown, wherein the principle of constructing the RDN-GAN deep learning network model comprises two stages of pre-training and optimization training, and the method specifically comprises the following steps:
in a specific embodiment, the pre-training phase of the RDN-GAN deep learning network model includes: the helical CT image data of the patient is collected, in this example, craniocerebral CT and thoracic CT, and the DRR orthographic projection is utilized to generate CBCT multi-angle two-dimensional projection data, which is divided into sparse sampling (30-180 projections) and full sampling (300-900 projections). The projection parameters are consistent with the existing radiotherapy positioning CBCT imaging parameters, the SDD (radiation source to detector distance) is 1500 mm, the SOD (radiation source to object center distance) is 1000 mm, and the imaging size of the flat panel detector is set to be 43cm multiplied by 43cm. And reconstructing the two-dimensional projection data into three-dimensional fault data by using a CBCT iterative reconstruction algorithm, wherein a reconstructed image is 512 pixels multiplied by 512 pixels, the pixel size is 0.5mm multiplied by 0.5mm, paired sparse sampling CBCT reconstruction data and full sampling CBCT reconstruction data are formed, and the paired sparse sampling CBCT reconstruction data and full sampling CBCT reconstruction data are used as a pre-training data set of a deep learning algorithm model to finish pre-training of the deep learning-based CBCT sparse sampling image enhancement algorithm model.
Specifically, a residual dense generation countermeasure network model (RDN-GAN) is adopted as a backbone network for enhancing image quality, and the method comprises a residual dense network and a countermeasure network, wherein the residual dense network comprises a convolution layer, a residual dense block, a cascade network, up-sampling and deconvolution lamination, and the countermeasure network comprises the convolution layer, an activation function, a normalization layer and a density network. The residual dense network is used as a generator, the residual dense block is composed of a convolution layer, an activation function and a cascade layer and is used for improving the performance and the precision of the network, the up-sampling layer is used for up-sampling a CBCT image with low resolution, and the countermeasure network is used as a discriminator and is used for judging whether the enhanced image is a real CBCT image or not. By processing the input CBCT image with a plurality of residual error density blocks, the quality and detail of the image can be gradually improved. Meanwhile, the discriminator is used for judging whether the high-quality CBCT image generated by the generator approximates to the real CBCT image. The model is continuously optimized through repeated training of the generator and the discriminator, and finally high-quality reconstruction of ultra-low dose CBCT imaging is realized.
Specifically, the paired radiotherapy CBCT sparse sampling-full sampling data set is used as training set data and is divided into a training set, a verification set and a test set, wherein the training set is used for training the RDN-GAN deep learning network model, the verification set is used for verifying the effect of the network model, and the test set is used for further testing the effect of the network model. In the training process of the RDN-GAN network model, peak signal to noise ratio (PSNR, peak Signal to Noise Ratio) is adopted as a loss function, and an Adam algorithm is adopted as an optimization algorithm. In the training process, training set data are randomly divided into small batches for training, and the training round number is 5000. On the validation set and the test set, PSNR and SSIM (structural similarity index ) after image enhancement are calculated to evaluate the effect of the network model.
Specifically, after it is determined that the loaded CBCT picture is an imaging region including a pre-stored in a database, the picture is normalized to an image of 512 pixels×512 pixels by linear interpolation and affine variation, and then the entire picture is input into an encoder and a decoder for image quality auto-enhancement. The image quality enhancement algorithm of the application can support brain and chest at present, but because the radiotherapy CBCT sparse sampling image enhancement method of the application is universal, the image enhancement model can also support the image enhancement of more radiotherapy CBCT imaging areas according to the requirement as long as more high-quality image data of other imaging areas are collected for model training.
Because most of computer vision algorithms need training, different training data can influence the final effect, in order to achieve better effect, the training data detail used by the radiotherapy CBCT sparse sampling image enhancement model pre-training module is specifically as follows: spiral CT image data comprising 500-1000 patients, each patient comprising 40-400 image pictures for DRR orthographic projection with a projection rotation angle of 360 degrees or 210 degrees, and generating paired sparse sampled (30-180) and full sampled reconstructed (300-900) pre-training datasets.
In a specific embodiment, the optimization phase of the RDN-GAN deep learning network model includes:
in order to further improve the robustness of an image enhancement algorithm model, the invention collects patient radiotherapy CBCT two-dimensional projection data, and utilizes a CBCT iterative reconstruction algorithm to reconstruct sparse sampling two-dimensional projection data (30-180 projections) and full sampling two-dimensional projection data (300-900 projections) into three-dimensional fault data, so as to form paired sparse sampling CBCT reconstruction data and full sampling CBCT reconstruction data, and the paired sparse sampling CBCT reconstruction data and full sampling CBCT reconstruction data are used as an optimization training set of an RDN-GAN algorithm model. And obtaining the optimized sparse sampling image enhancement algorithm model based on the deep learning. Alternatively, if the original two-dimensional projection data is difficult to acquire, the two-dimensional projection data can also be acquired by performing DRR orthographic projection by using the fully sampled radiotherapy CBCT three-dimensional tomographic image data.
The details of training data used by the radiotherapy CBCT sparse sampling image enhancement model optimization training module are specifically as follows: comprises radiotherapy CBCT image data of 200-500 patients, and generates paired sparse sampling (30-180 pieces) reconstruction and full sampling reconstruction (300-900 pieces) optimization training data sets.
The optimized training set setting and training parameters are consistent with the pre-training, and after the training is completed, all modules can only keep the test program and the model obtained by training. In addition, in the implementation, fixed-point implementation is adopted to avoid floating point operation, so that the running speed of the whole system is greatly increased.
Specifically, the pre-trained radiotherapy CBCT sparse sampling image enhancement model can be directly applied to radiotherapy CBCT sparse sampling image enhancement processes of various equipment sources, and under certain conditions, if the enhanced image quality is insufficient or more distortion exists, optimized training can be performed by continuously adding radiotherapy CBCT data, particularly the current radiotherapy CBCT equipment source data, so that the robustness and the accuracy of the algorithm model are improved.
As shown in fig. 3, a schematic drawing of CBCT sparse sampling imaging for radiotherapy is shown according to this embodiment, an X-ray light source bulb tube and a flat panel detector perform 360-degree or 210-degree rotation scanning around an imaging part of a human body, multi-angle two-dimensional projection data are captured, each point of a circle represents a corresponding angle projection, full sampling is to collect 300-900 projections, sparse sampling is to collect 30-180 projections by reducing the number of projections of projection data collected on the rotating circle, and sparse sampling can reduce scanning time and scanning dose, which is a fast low-dose scanning imaging mode.
Specifically, CBCT scanning is divided into a full fan (full fan) mode and a half fan (half fan) mode, the former refers to centering of a flat panel detector when acquiring CBCT images, all objects can be reconstructed on each projection map, and the CBCT scanning is generally applied when an imaging area is smaller, and a scanning angle is more than 180 degrees; the latter refers to the lateral translation of the flat panel detector when acquiring CBCT images, only part of the object can be reconstructed on each projection view, and the Field of view (FOV) is increased by the translation of the flat panel detector, and the scan angle needs to reach 360 degrees. In order to further reduce the CBCT projection time of radiotherapy, the scan angle can be reduced to about 210 degrees when the imaging area is required to be smaller and the flat panel detector can be covered completely, so as to further reduce the imaging time.
However, as shown in fig. 4, the first column is an image before and after 30 projections (360 degrees) of the radiotherapy CBCT head are reconstructed, the second column is an image before and after 66 projections (360 degrees) of the radiotherapy CBCT chest are reconstructed, and due to serious shortage of sampling information, a large number of streak artifacts easily appear in the CBCT image obtained by three-dimensional reconstruction, and the greater the sparse sampling degree, the heavier the artifacts are, so that the image quality is seriously reduced, and the use of the CBCT image in the subsequent radiotherapy process is affected.
Optionally, as shown in fig. 5, in a specific embodiment of the present application, an operation logic block diagram of the above-mentioned radiotherapy CBCT sparse sampling image enhancement method is as follows:
(1) And acquiring CBCT image data obtained by radiotherapy CBCT sparse sampling imaging.
(2) And calling a medical image detection module to detect whether the medical image acquired by the current hardware is an imaging area stored in the database in advance, and if the medical image is not contained, prompting a doctor to input the mismatching image until software can detect the medical image containing the specific organ.
(3) If the medical picture is detected to contain an imaging area stored in the database in advance, a software interface displays a prompt that the image quality enhancement algorithm starts working.
(4) And reading information of the input medical image data, calling a pre-constructed radiotherapy CBCT sparse sampling image enhancement model, enhancing the influence of the input low-quality CBCT and providing the low-quality CBCT as output for a user.
(5) If the information of closing the software by the user is received, closing the software, releasing the memory and exiting.
Specifically, the medical image data processed in this embodiment is sparse sampling imaging data of radiotherapy CBCT, when a loaded medical image is detected, the whole medical image is compared with image data including a specific organ or imaging region stored in advance in a database, if the medical image is detected to include the prestored organ or imaging region, a pre-built image enhancement model for the specific organ or imaging region is started to be called, the CBCT image is subjected to image enhancement, and a high-quality radiotherapy CBCT image is output. It should be noted that, in theory, the image enhancement model may include a plurality of different imaging positions, but in an embodiment, due to the reason of training data collection, CBCT imaging image enhancement of brain and chest radiotherapy is mainly considered at present, and then CBCT sparse sampling image enhancement of more organs and imaging areas, such as abdomen, pelvis, limbs, etc., may be implemented with more training data sets.
In addition, when the loaded medical picture is detected, if the medical picture is detected not to be the organ or the imaging area stored in the database in advance, the follow-up processing is abandoned, and the doctor is prompted that the image picture is not the medical picture containing the specific organ or the imaging area until the doctor loads the radiotherapy CBCT image containing the specific organ or the imaging area.
In one embodiment, the constructed RDN-GAN deep learning network model comprises:
the system comprises a residual dense network and an countermeasure network, wherein the residual dense network comprises a convolution layer, a residual dense block, a cascade network, up-sampling and deconvolution layers, the residual dense block comprises the convolution layer, an activation function and the cascade layer, and the countermeasure network comprises the convolution layer, the activation function, a normalization layer and a density network.
As a specific embodiment, inputting the sparse sampling reconstruction and the full sampling reconstruction data sets into a pre-constructed residual dense generation countermeasure network for training to obtain a radiotherapy CBCT sparse sampling image enhancement model, including:
performing data expansion on the paired data set, and normalizing the expanded registration data set into a 512-pixel by 512-pixel data set;
dividing the normalized image set into a training set, a verification set and a test set;
and inputting the training set into a residual dense network and an countermeasure network, and obtaining a training model according to the loss function of the generated network and the loss function of the countermeasure network.
And verifying and testing the verification model by using the verification set and the test set, if the verification and test results do not meet the prediction probability threshold, training the generation network and the countermeasure network again, and if the results meet the prediction probability threshold, obtaining the radiotherapy CBCT sparse sampling image enhancement algorithm model.
It can be understood that the steps for obtaining the radiotherapy CBCT sparse sampling image enhancement model based on the RDN-GAN network structure are as follows:
(1) Training set construction: the training set comprises a pre-training set and an optimized training set, wherein the pre-training set is a paired sparse sampling and full sampling reconstruction data set obtained by performing CBCT reconstruction after the spiral CT data of a patient are forward projected, and the data volume is 500-1000 patients; the optimized training set is paired sparse sampling and full sampling data sets obtained by reconstructing radiotherapy CBCT two-dimensional projection data, and the number of the paired sparse sampling and full sampling data sets is 200-500 patients.
(2) Training phase: the deep learning network is an improved residual dense network, an countermeasure (RDN-GAN) network structure is generated for residual dense, the radiation treatment CBCT sparse sampling image enhancement is completed through the residual dense network in the RDN-GAN network, and whether the CBCT image enhanced by the image approximates to a full sampling CBCT image is judged through the countermeasure network of the RDN-GAN network.
As shown in FIG. 6, in a particular embodiment of the present application, the training RDN-GAN network structure includes a residual dense network and an countermeasure network; the residual dense network consists of a convolution layer, a residual dense block, a cascade network, up-sampling and deconvolution lamination, wherein the convolution kernel of each layer has a size of k multiplied by k, each layer has c channels, and the residual dense block consists of the convolution layer, an activation function and the cascade layer. The countermeasure network, or discriminator, passes through convolution layers of different parameters for the input pictures, normalizes the layer and activates the function to get whether true or false finally.
The countermeasure network comprises a convolution layer, an activation function, a normalization layer and density networks, wherein the input and output relation of each density network is as follows:
wherein,output of layer c convolutional network for layer d density network, +.>And->Let go of->Input and output of layer density network, +.>For RELU activation function, +.>Weights for each convolution layer, +.>Cascading feature map generated for layer-1 density network, ++>Then the output of the 1 st convolutional network for the d-th density network,the output of the c-1 th convolutional network is the d-th density network.
The RELU activation function is:
；
compared with the general Sigmoid or Tanh-based activation function, the RELU activation function has no problem of derivative disappearance or derivative explosion during training, so that the whole training process is more stable, the RELU activation function is simpler to calculate, floating point operation is not needed, and the processing time is greatly shortened during calculation.
The cascade network links all the characteristics of the density network and adaptively controls the output information through the convolutional network of 1 x 1. And finally, obtaining the output of the whole local countermeasure density network through a residual error learning network.
The loss function of the overall residual dense generation countermeasure network is as follows:
Wherein,for the perceptual loss function, it is composed of a plurality of loss function weights,Nin order to train the total number of samples entered,nindex for each training sample, +.>Output representing a residual dense network, +.>Input samples for some sparse sample of training set, +.>For a certain full sample of the training set, +.>Weights and biases representing the network, +.>The optimal solution of the network model is obtained through training, so that the perception loss function between the sparse sampling sample and the full sampling sample is minimum.
At the same time, to enhance network capacity, features are built and perceptual loss functions are introduced before RELU activation functionsThe specific calculation formula is as follows:
wherein,for the content loss function->To combat the loss function;
wherein,representing reconstructed pictures->Probability of being a real picture, < >>Output representing a residual dense network, +.>Input picture representing countermeasure network, < >>Representing the weights and biases of the generated network,Nin order to train the input total sample,nindexing each training sample;
calculation using MSE and PSNRIs calculated by the following formula:
；
；/>
wherein,at the maximum value of the pixel,m,nthe number of horizontal pixels and the number of vertical pixels respectively representing the resolution of the input image,i,jand the coordinate index number corresponding to each pixel is given, and I and K respectively represent the image after image enhancement and the full-sampling image.
It can be understood that the present application loads CBCT images and normalizes the images to 512 pixels by 512 pixels images by linear interpolation and affine variation, and then inputs the whole images to a residual dense generation countermeasure network (RDN-GAN), thereby obtaining the radiation therapy CBCT images after image enhancement. The radiation therapy CBCT sparse sampling image enhancement model of the application can support the image enhancement of imaging of a plurality of parts, but in the embodiment, the CBCT sparse sampling image enhancement of brain and chest radiation therapy is mainly shown. It can be understood that the sparse sampling image enhancement model of the radiotherapy CBCT can also support the image enhancement of the radiotherapy CBCT images of more organs according to the requirement. Since most computer vision algorithm models need to be trained, different training data sets and data sets with different quality may affect the final model effect, and in order to achieve a better effect, the training data details are as follows: the pre-training of the algorithm model is respectively carried out through spiral CT image data of more than 800 patients 'brains and breasts, and the optimization training of the algorithm model is carried out through CBCT image data of more than 200 patients' radiotherapy.
After training is completed, all modules may only retain the test program and the model obtained by training. In addition, in the implementation, fixed-point implementation is adopted to avoid floating point operation, so that the running speed of the whole system is greatly increased.
(3) And (3) detection: and performing image enhancement on all loaded radiotherapy CBCT sparse sampling imaging pictures by using the radiotherapy CBCT sparse sampling image enhancement model based on deep learning and learned in the training stage, so as to optimize the loss functions of the residual dense network and the countermeasure network, and further obtain a verification model.
(4) Testing: and testing the verification model by using the test set, if the test result does not meet the prediction probability threshold, training the residual dense network and the countermeasure network again, and if the test result meets the prediction probability threshold, obtaining the radiotherapy CBCT sparse sampling image enhancement model.
In some alternative embodiments, the radiotherapy CBCT sparse sampled image enhancement model comprises:
detecting an input CBCT image by using a computer vision model, and judging whether the input CBCT image is an organ or imaging region picture supported by the radiotherapy CBCT sparse sampling image enhancement model;
if yes, the input low-quality sparse sampling reconstructed CBCT image is enhanced into a high-quality radiotherapy CBCT image.
Referring to fig. 7, for the result of image enhancement of the radiotherapy CBCT sparse sampling head provided by the embodiment of the present application, the first column is a low-quality CBCT reconstructed image of the input radiotherapy CBCT sparse sampling head imaging, the number of samples is 30 two-dimensional projections, the second column is an image enhancement result output by the present application, and the third column is a group trunk reconstruction imaging result (group trunk), which can be used as a standard for evaluating the sharpness and fidelity of the image enhancement result.
Referring to fig. 8, for the result of image enhancement of radiotherapy CBCT sparse sampling chest imaging provided by the embodiment of the present application, the first column is a low-quality CBCT reconstructed image of input radiotherapy CBCT sparse sampling chest imaging, the number of samples is 66 projections, the second column is an image enhancement result output by the present application, and the third column is a group trunk imaging result (group trunk), which can be used as a criterion for evaluating the fidelity of the image enhancement result.
The result in the figure shows that the radiotherapy CBCT sparse sampling leads to a large amount of artifacts generated by the reconstructed image, and the definition of the image is seriously reduced, the method is adopted for enhancing the imaging image of the radiotherapy CBCT sparse sampling head, the PSNR value of the enhanced image is improved to 32.2dB from 30.7 dB, the SSIM value is improved to 0.88 from 0.72, the artifacts caused by the sparse sampling can be effectively eliminated, the image quality and the detail are obviously improved, the image quality and the detail are extremely high, the distortion degree are extremely low, the image definition is extremely high, the image definition is extremely close to that of a fully sampled (Groundtruth), and the gray value distribution close to that of the fully sampled CBCT image is provided.
It should be noted that, when the radiation therapy CBCT sparse sampling is imaged, the higher the sparseness, i.e. the smaller the projection number, the more serious artifacts will be generated in the reconstructed image, the image definition will be greatly reduced, the greater the difficulty of implementing high-precision and high-fidelity image enhancement by the algorithm model, after a great deal of trial and exploration study and model optimization, the high-definition image enhancement of 30 projection reconstructed images of the radiation therapy CBCT head and 66 projection reconstructed images of the chest is implemented by the application, on the basis of a great deal of experiments, the team finds that 30 samples are basically close to the limit of the radiation therapy CBCT head imaging sparse sampling, the number of the samples is already at the world leading level, the experimental result is compared with the (group reconstruction) of the fully sampled reconstructed image, the image after image enhancement has very high fidelity, and if the number of the samples is reduced again, serious distortion of the image after enhancement will occur. Of course, a CBCT high definition reconstruction of a lower number of samples may be achieved without excluding the subsequent more advanced algorithms and higher quality data sets.
The hardware platform on which the pre-constructed radiotherapy CBCT sparse sampling image enhancement method depends is the GPU of Nvidia. On Nvidia's 4090 GPU hardware, the deep learning algorithm of the present application can process at least 10 frames of images per second. In summary, the application combines the computer vision technology and the latest artificial intelligence deep learning technology to realize the image quality enhancement of the image after sparse sampling reconstruction, can ensure the imaging definition of the patient, greatly reduce the irradiation dose of the patient, and has good application prospect in the processes of tumor radiotherapy and the like. Overall, the method has the advantages of more accurate algorithm, stronger robustness, more extreme conditions which can be processed, effective elimination of noise caused by various external imaging interferences and the like.
Compared with the closest prior art, the radiotherapy CBCT sparse sampling image enhancement method provided by the application has the following beneficial effects:
(1) The algorithm model training mode of large sample pre-training and small sample additional training is adopted, so that the problem of high acquisition difficulty of a large number of high-quality radiotherapy CBCT samples is solved. The method comprises the steps of generating paired data by adopting a larger number (500-1000 cases) of spiral CT data through orthographic sparse sampling, pre-training a sparse sampling CBCT algorithm model, performing model optimization training by using small sample (200-500 cases) radiotherapy CBCT original two-dimensional projection paired data, increasing the robustness of the algorithm model, and solving the problems of difficult acquisition of radiotherapy CBCT data and poor image quality;
(2) The CBCT iterative reconstruction algorithm is adopted to carry out three-dimensional reconstruction of the CBCT two-dimensional projection data, and has better reconstruction definition than the traditional FDK reconstruction algorithm, particularly has higher sparse sampling tolerance under the condition of sparse sampling, and can realize high-definition imaging under lower sampling quantity.
(3) The latest improved residual intensive generation countermeasure (RDN-GAN) deep learning network model is adopted to realize image enhancement, the algorithm shows an effect which is obviously superior to that of the existing deep learning algorithms in the traditional computer vision field, the algorithm is modified and perfected and then is applied to the radiotherapy CBCT sparse sampling image enhancement field, and the results in the embodiment show that the algorithm model can effectively improve the image quality and enhance the image details; compared with the prior CNN deep learning image enhancement algorithm, the RDN-GAN can better maintain the spatial resolution of imaging, and can obtain a high-quality CBCT image under the condition of sparse sampling CBCT imaging.
(4) The radiation dose of a patient can be greatly reduced, the imaging speed is increased, meanwhile, the high imaging definition is kept, the safety of the patient during radiation therapy is improved, and the artifacts related to the movement of the patient are fewer; the real-time guiding and correcting in the radiotherapy process can be realized, and the effect of accurate radiotherapy is improved.
In summary, aiming at the defects of high imaging quality and low speed of the existing radiotherapy CBCT, the invention provides a method for enhancing the sparse sampling image of the radiotherapy CBCT based on deep learning, which solves the difficult problem of large acquisition difficulty of a large amount of high-quality radiotherapy CBCT image data by adopting a model training mode of pre-training of a large sample and additional training of a small sample, and increases the robustness of an algorithm model; by using the self-grinding CBCT iterative reconstruction algorithm to carry out three-dimensional reconstruction of the CBCT two-dimensional projection data, the iterative reconstruction algorithm has better reconstruction definition than the traditional FDK reconstruction algorithm, has higher sparse sampling tolerance, and can realize high-definition imaging under lower sampling quantity. Image enhancement is achieved by using the latest, more advanced residual dense generation countermeasure (RDN-GAN) deep learning network architecture, which shows a significant effect over all existing deep learning algorithms in the traditional computer vision field, and in embodiments of the present invention, the algorithm model is shown to effectively improve image quality, enhancing image details. Therefore, the invention can greatly reduce the radiation dose of the patient, simultaneously maintain high imaging definition, quicken the imaging speed, improve the safety of the patient during radiotherapy and reduce the artifact related to the patient movement; the method can also realize real-time guiding and correction in the radiotherapy process, and improve the accuracy and treatment effect of the radiotherapy.
As shown in fig. 9, the present embodiment provides a system for applying a deep learning-based radiotherapy CBCT sparse sampling image enhancement method, including:
a projection acquisition module 201 for acquiring multi-angle two-dimensional projection data of a patient;
a three-dimensional reconstruction module 202 for reconstructing two-dimensional projection data of the patient into CBCT three-dimensional tomographic data;
the image enhancement module 203 is configured to input a low-quality image reconstructed by sparse sampling of the radiotherapy CBCT into a residual error dense block trained and constructed in advance-generate an countermeasure network model, and obtain a radiotherapy CBCT image after image enhancement;
the image display and output module 204 is used for displaying and outputting the image-enhanced radiotherapy CBCT image in a Dicom format for the subsequent procedure of radiotherapy.
The application provides a working principle of a system applying a radiotherapy CBCT sparse sampling image enhancement model, which comprises the following steps: the projection acquisition module 201 acquires multi-angle two-dimensional projection data of a patient, wherein the number of projections is 30-180; the three-dimensional reconstruction module 202 reconstructs the two-dimensional projection data of the patient into sparse sampling CBCT three-dimensional data; the image enhancement module 203 inputs the low-quality image reconstructed by radiotherapy CBCT sparse sampling into a pre-trained and constructed residual dense generation countermeasure (RDN-GAN) network model to obtain a radiotherapy CBCT image after image enhancement; the image display and output module 204 displays and outputs the image-enhanced radiotherapy CBCT image in Dicom format for the subsequent procedure of radiotherapy.
As shown in fig. 10, the present embodiment provides a radiotherapy CBCT imaging apparatus based on deep learning, including: a radiotherapy CBCT imaging component, a three-dimensional reconstruction component, an image enhancement component and the like. The radiotherapy CBCT imaging component comprises an X-ray source, a flat panel detector, a bracket, a motion platform and the like, and is used for carrying out CBCT imaging on a patient; the three-dimensional reconstruction component is used for reconstructing two-dimensional projection data obtained by the radiotherapy CBCT imaging system into a three-dimensional CBCT image; the image enhancement component stores a pre-training and constructed radiotherapy CBCT sparse sampling image enhancement system program based on deep learning, and can enhance the low-quality CBCT image transmitted by the reconstruction and display system into a high-quality radiotherapy CBCT image. And the image display and output assembly is used for displaying the radiotherapy CBCT image after image enhancement and outputting the image in a Dicom format for the subsequent procedure of radiotherapy.
There is provided a computer device, see fig. 10, comprising: the memory 1 and the processor 2 may further comprise a network interface 3, said memory storing a computer program, the memory may comprise non-volatile memory in a computer readable medium, random Access Memory (RAM) and/or non-volatile memory etc. form, such as Read Only Memory (ROM) or flash memory (flash RAM). The computer device stores an operating system 4, the memory being an example of a computer readable medium. The computer program, when executed by the processor, causes the processor to perform the method of deep learning based radiotherapy CBCT sparse sampled image enhancement, the structure shown in fig. 10 is merely a block diagram of a portion of the structure associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements are applied, a particular computer device may include more or fewer components than shown in the drawings, or may combine certain components, or have a different arrangement of components.
In one embodiment, the deep learning-based radiotherapy CBCT imaging image enhancement method provided by the present application may be implemented in the form of a computer program, which may be executed on a computer device as shown in fig. 10.
In some embodiments, the computer program, when executed by the processor, causes the processor to perform the steps of: acquiring multi-angle two-dimensional projection data of a patient, wherein the projection quantity is 30-180 sheets; reconstructing the two-dimensional projection data of the patient into sparse sampling CBCT three-dimensional data; inputting a low-quality image reconstructed by radiotherapy CBCT sparse sampling into a residual error dense block which is trained and constructed in advance-generating an countermeasure network model, and obtaining a radiotherapy CBCT image after image enhancement; and displaying the image-enhanced radiotherapy CBCT image and outputting the image in a Dicom format for the subsequent procedure of radiotherapy.
The present application also provides a computer storage medium, examples of which include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassette storage or other magnetic storage devices, or any other non-transmission medium, that can be used to store information that can be accessed by a computing device.
In some embodiments, the present invention further provides a computer readable storage medium storing a computer program which, when executed by a processor, obtains multi-angle two-dimensional projection data of a patient, the number of projections being between 30 and 180; reconstructing the two-dimensional projection data of the patient into sparse sampling CBCT three-dimensional data; inputting a low-quality image reconstructed by radiotherapy CBCT sparse sampling into a residual error dense block which is trained and constructed in advance-generating an countermeasure network model, and obtaining a radiotherapy CBCT image after image enhancement; and displaying the image-enhanced radiotherapy CBCT image and outputting the image in a Dicom format for the subsequent procedure of radiotherapy.
In summary, the invention provides a method, a system and a device for enhancing sparse sampling image of radiotherapy CBCT, wherein the method comprises the steps of constructing a residual dense generation countermeasure (RDN-GAN) deep learning network model; collecting spiral CT data of a patient, obtaining two-dimensional projection data corresponding to multiple angles through orthographic projection, and splitting the two-dimensional projection data into sparse sampling data and full sampling data; respectively reconstructing the sparse sampling CBCT two-dimensional projection data and the full sampling projection data into three-dimensional tomographic image data by using a CBCT reconstruction algorithm to obtain a sparse sampling-full sampling paired data set; finishing the pre-training of the RDN-GAN network model; collecting radiotherapy CBCT two-dimensional projection data, splitting the radiotherapy CBCT two-dimensional projection data into sparse sampling and full sampling data, reconstructing the sparse sampling two-dimensional projection data and the full sampling projection data into three-dimensional tomographic image data by using a CBCT reconstruction algorithm to obtain a CBCT sparse sampling-full sampling paired data set, and completing optimization training of an RDN-GAN deep learning network model; and (3) carrying out radiotherapy CBCT sparse sampling image enhancement by applying the optimized and trained RDN-GAN deep learning network model to obtain a high-quality radiotherapy CBCT image, and applying the high-quality radiotherapy CBCT image to a follow-up procedure of radiotherapy. The imaging method can obviously reduce the imaging radiation dose and improve the precision and safety of radiotherapy on the premise of ensuring the imaging quality. The invention can provide a new solution for radiotherapy CBCT imaging, has higher practicability and wide clinical application prospect, and can be widely applied to the field of tumor radiotherapy.
The system, the device and the equipment disclosed by the embodiment are relatively simple in description, and the relevant parts refer to the description of the method part because the system, the device and the equipment correspond to the method disclosed by the embodiment.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The foregoing is merely a specific implementation of the present invention, and although the foregoing examples have described advantages and specific implementation of the present invention, the present invention is not limited to the foregoing examples. Modifications and variations of the present invention in its various forms may be suggested to one skilled in the art in view of the teachings of the present principles and teachings to adapt to various application scenarios and specific implementation details. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A method for enhancing a radiation therapy CBCT sparse sampling image is characterized by comprising the following steps:
constructing an RDN-GAN deep learning network model;
collecting spiral CT data, and processing the spiral CT data to obtain a paired CBCT sparse sampling-full sampling pre-training data set; based on the CBCT sparse sampling-full sampling pre-training data set, completing pre-training of an RDN-GAN deep learning network model;
Collecting radiotherapy CBCT two-dimensional projection data, and processing the radiotherapy CBCT two-dimensional projection data to obtain a paired CBCT sparse sampling-full sampling optimization training data set; completing optimization training of an RDN-GAN deep learning network model based on the CBCT sparse sampling-full sampling optimization training data set;
and carrying out radiotherapy CBCT sparse sampling image enhancement through the optimized RDN-GAN deep learning network model, and outputting to obtain a high-quality radiotherapy CBCT image.
2. The method for enhancing a sparse sampled image of a radiotherapy CBCT of claim 1, wherein the collecting spiral CT data and processing the spiral CT data to obtain a paired CBCT sparse sampled-full sampled pre-training dataset, specifically comprises: generating multi-angle two-dimensional CBCT projection data from the spiral CT data by utilizing a front projection algorithm, wherein projection parameters are consistent with radiotherapy positioning CBCT imaging parameters, splitting the two-dimensional CBCT projection data into sparse sampling and full sampling data, reconstructing the sparse sampling and full sampling data into three-dimensional tomographic image data, and obtaining a paired CBCT sparse sampling-full sampling pre-training data set; the sparse sampling in the paired CBCT sparse sampling-full sampling pre-training data set is used for collecting 30-180 pieces of projection data, and the full sampling in the paired CBCT sparse sampling-full sampling pre-training data set is used for collecting 300-900 pieces of projection data.
3. The method for enhancing sparse sampling image of radiotherapy CBCT according to claim 1, wherein the collecting the radiotherapy CBCT two-dimensional projection data and processing the radiotherapy CBCT two-dimensional projection data to obtain a paired CBCT sparse sampling-full sampling optimized training dataset specifically comprises: collecting patient radiotherapy positioning CBCT two-dimensional projection data, splitting the projection data into sparse sampling data and full sampling projection data, and reconstructing the sparse sampling two-dimensional projection data and the full sampling two-dimensional projection data into three-dimensional tomographic image data by using a CBCT three-dimensional reconstruction algorithm to obtain a paired CBCT sparse sampling-full sampling optimization training data set; the sparse sampling in the paired CBCT sparse sampling-full sampling optimization training data set is used for collecting 30-180 pieces of projection data, and the full sampling in the paired CBCT sparse sampling-full sampling optimization training data set is used for collecting 300-900 pieces of projection data.
4. The method for enhancing a sparse sampled image of a radiotherapy CBCT of claim 1, wherein the RDN-GAN deep learning network model comprises a residual dense network and an countermeasure network, the residual dense network comprising a convolutional layer, a residual dense block, a cascade network, upsampling, and deconvolution; the residual error dense block is composed of a convolution layer, an activation function and a cascade layer.
5. The method for sparse sampled image enhancement of radiotherapy CBCT of claim 4, wherein said countermeasure network is comprised of a convolution layer, an activation function, a normalization layer, and density networks, each density network having an input and output relationship of:
wherein,output of layer c convolutional network for layer d density network, +.>And->Let go of->Input and output of layer density network, +.>For RELU activation function, +.>Weights for each convolution layer, +.>Cascading feature map generated for layer-1 density network, ++>Then the output of the 1 st convolutional network for the d-th density network,the output of the c-1 th convolutional network is the d-th density network.
6. The method for enhancing sparse sampled images of radiotherapy CBCT according to claim 1, wherein said constructing an RDN-GAN deep learning network model comprises calculating an RDN-GAN deep learning network model loss function with the specific formula:
wherein,for the perceptual loss function, it is composed of a plurality of loss function weights,Nin order to train the total number of samples entered,nindex for each training sample, +.>Output representing a residual dense network, +.>Sparse sampling of training set is entered with samples +.>Sample for training set total +. >Weights and biases representing the network, +.>The optimal solution of the network model is obtained through training.
7. The method of sparse sampled image enhancement of a radiation therapy CBCT of claim 6, further comprising: constructing features and introducing perceptual loss functions prior to RELU activation functionsThe specific calculation formula is as follows:
wherein,for the content loss function->To combat the loss function;
wherein,representing reconstructed pictures->Probability of being a real picture, < >>Output representing a residual dense network, +.>Input picture representing countermeasure network, < >>Representing the weights and biases of the generated network,Nin order to train the input total sample,nindexing each training sample;
calculation using MSE and PSNRIs calculated by the following formula:
；
；
wherein,at the maximum value of the pixel,m,nthe number of horizontal pixels and the number of vertical pixels respectively representing the resolution of the input image,i,jthe coordinate index number corresponding to each pixel is respectively represented by I and K after the image enhancementAn image and a full sample image.
8. The method for sparse sampling image enhancement of radiotherapy CBCT according to claim 1, wherein the sparse sampling radiotherapy CBCT image quality enhancement by the optimized RDN-GAN deep learning network model comprises: and (3) inputting the sparse sampling reconstructed low-quality radiotherapy CBCT image data into an optimized RDN-GAN deep learning network model, and outputting to obtain a radiotherapy CBCT image with high imaging quality.
9. A radiation therapy CBCT sparse sampled image enhancement system utilizing a radiation therapy CBCT sparse sampled image enhancement method as claimed in any of claims 1-8, comprising:
a projection acquisition module (201) for acquiring multi-angle two-dimensional projection data of a patient;
a three-dimensional reconstruction module (202) for reconstructing two-dimensional projection data of the patient into a CBCT three-dimensional tomographic image;
the image enhancement module (203) is used for inputting CBCT three-dimensional tomographic images reconstructed by sparse sampling of radiotherapy CBCT into a pre-trained and constructed residual error density generation countermeasure network model to obtain radiotherapy CBCT images after image enhancement;
and the image display and output module (204) is used for displaying the radiotherapy CBCT image after image enhancement and outputting the image in a Dicom format for the follow-up procedure of radiotherapy.
10. A radiation therapy CBCT sparse sampled image enhancement device utilizing a radiation therapy CBCT sparse sampled image enhancement method as claimed in any of claims 1-8, comprising: the device comprises a radiotherapy CBCT imaging assembly, a three-dimensional reconstruction assembly, an image enhancement assembly and an image display and output assembly;
the radiotherapy CBCT imaging assembly comprises an X-ray source, a flat panel detector, a bracket and a motion platform, and is used for CBCT imaging; the three-dimensional reconstruction component is used for reconstructing two-dimensional projection data obtained by the radiotherapy CBCT imaging system into a three-dimensional CBCT image; the image enhancement component stores a pre-trained and constructed radiotherapy CBCT sparse sampling image enhancement system program based on deep learning, and enhances the low-quality CBCT image transmitted by the three-dimensional reconstruction component into a high-quality radiotherapy CBCT image; the image display and output component is used for displaying the image after the image enhancement and outputting the image in a Dicom format.
CN202311378973.4A 2023-10-24 2023-10-24 Method, system and device for enhancing sparse sampling image of radiotherapy CBCT Active CN117115046B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311378973.4A CN117115046B (en) 2023-10-24 2023-10-24 Method, system and device for enhancing sparse sampling image of radiotherapy CBCT

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311378973.4A CN117115046B (en) 2023-10-24 2023-10-24 Method, system and device for enhancing sparse sampling image of radiotherapy CBCT

Publications (2)

Publication Number Publication Date
CN117115046A true CN117115046A (en) 2023-11-24
CN117115046B CN117115046B (en) 2024-02-09

Family

ID=88797035

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311378973.4A Active CN117115046B (en) 2023-10-24 2023-10-24 Method, system and device for enhancing sparse sampling image of radiotherapy CBCT

Country Status (1)

Country Link
CN (1) CN117115046B (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160005192A1 (en) * 2014-07-02 2016-01-07 Siemens Medical Solutions Usa, Inc. Prior Image Based Three Dimensional Imaging
CN111899188A (en) * 2020-07-08 2020-11-06 西北工业大学 Neural network learning cone beam CT noise estimation and suppression method
CN111932520A (en) * 2018-08-31 2020-11-13 上海联影智能医疗科技有限公司 Medical image display method, viewing device and computer device
CN111967528A (en) * 2020-08-27 2020-11-20 北京大学 Image identification method for deep learning network structure search based on sparse coding
CN114511497A (en) * 2021-12-24 2022-05-17 北京肿瘤医院(北京大学肿瘤医院) Imaging method and device applied to cone beam CT sparse sampling
KR102477991B1 (en) * 2022-02-11 2022-12-15 서울대학교산학협력단 Medical image processing method and apparatus

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160005192A1 (en) * 2014-07-02 2016-01-07 Siemens Medical Solutions Usa, Inc. Prior Image Based Three Dimensional Imaging
CN111932520A (en) * 2018-08-31 2020-11-13 上海联影智能医疗科技有限公司 Medical image display method, viewing device and computer device
CN111899188A (en) * 2020-07-08 2020-11-06 西北工业大学 Neural network learning cone beam CT noise estimation and suppression method
CN111967528A (en) * 2020-08-27 2020-11-20 北京大学 Image identification method for deep learning network structure search based on sparse coding
CN114511497A (en) * 2021-12-24 2022-05-17 北京肿瘤医院(北京大学肿瘤医院) Imaging method and device applied to cone beam CT sparse sampling
KR102477991B1 (en) * 2022-02-11 2022-12-15 서울대학교산학협력단 Medical image processing method and apparatus

Also Published As

Publication number Publication date
CN117115046B (en) 2024-02-09

Similar Documents

Publication Publication Date Title
Zhang et al. Improving CBCT quality to CT level using deep learning with generative adversarial network
Kida et al. Visual enhancement of cone‐beam CT by use of CycleGAN
Gong et al. PET image denoising using a deep neural network through fine tuning
JP7179757B2 (en) Dose Reduction for Medical Imaging Using Deep Convolutional Neural Networks
JP7150837B2 (en) Image generation using machine learning
CN111325686B (en) Low-dose PET three-dimensional reconstruction method based on deep learning
Tang et al. Unpaired low-dose CT denoising network based on cycle-consistent generative adversarial network with prior image information
JP2020168352A (en) Medical apparatus and program
Fu et al. A deep learning reconstruction framework for differential phase-contrast computed tomography with incomplete data
WO2021041772A1 (en) Dilated convolutional neural network system and method for positron emission tomography (pet) image denoising
CN110298447B (en) Method for processing parameters of machine learning method and reconstruction method
CN113689342A (en) Method and system for optimizing image quality
CN116071401B (en) Virtual CT image generation method and device based on deep learning
CN113516586A (en) Low-dose CT image super-resolution denoising method and device
CN116612174A (en) Three-dimensional reconstruction method and system for soft tissue and computer storage medium
CN114387236A (en) Low-dose Sinogram denoising and PET image reconstruction method based on convolutional neural network
He et al. Downsampled imaging geometric modeling for accurate CT reconstruction via deep learning
CN117813055A (en) Multi-modality and multi-scale feature aggregation for synthesis of SPECT images from fast SPECT scans and CT images
Jiang et al. Fast four‐dimensional cone‐beam computed tomography reconstruction using deformable convolutional networks
WO2021226500A1 (en) Machine learning image reconstruction
CN117115046B (en) Method, system and device for enhancing sparse sampling image of radiotherapy CBCT
Yang et al. Quasi-supervised learning for super-resolution PET
Shang et al. Short-axis pet image quality improvement by attention CycleGAN using total-body pet
CN114241074B (en) CBCT image reconstruction method for deep learning and electronic noise simulation
Kim et al. CNN-based CT denoising with an accurate image domain noise insertion technique

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant