CN115393534B - Deep learning-based cone beam three-dimensional DR (digital radiography) reconstruction method and system - Google Patents
Deep learning-based cone beam three-dimensional DR (digital radiography) reconstruction method and system Download PDFInfo
- Publication number
- CN115393534B CN115393534B CN202211347742.2A CN202211347742A CN115393534B CN 115393534 B CN115393534 B CN 115393534B CN 202211347742 A CN202211347742 A CN 202211347742A CN 115393534 B CN115393534 B CN 115393534B
- Authority
- CN
- China
- Prior art keywords
- prior information
- image
- dimensional
- sequence frame
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/005—General purpose rendering architectures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
- G06T2207/10012—Stereo images
Abstract
The application discloses a cone beam three-dimensional DR reconstruction method and system based on deep learning, wherein the method comprises the following steps: acquiring a DR image sequence frame; performing image geometric correction on the DR image sequence frame, and acquiring pose prior information; performing feature extraction on the DR image sequence frame, and acquiring granularity prior information; modeling the pose prior information and the granularity prior information to acquire fusion prior information; acquiring a perception feature extractor through transfer learning; combining with a preset generation countermeasure network to obtain a DR three-dimensional reconstruction model; and generating a DR three-dimensional image according to the DR image sequence frame, the fusion prior information and the DR three-dimensional reconstruction model. The influence of geometric deviation on three-dimensional reconstruction is reduced by performing geometric correction on the DR image, in addition, the associated information among the features can be modeled by fusing the prior information so as to capture the missing detail features, and the robustness of the reconstruction network under the conditions of background blurring, edge detail missing and the like is improved.
Description
Technical Field
The application relates to the field of computer vision, in particular to a cone beam three-dimensional DR reconstruction method and system based on deep learning.
Background
Digital Radiography (DR) is a radiography technique formed by combining computer digital image processing techniques with radiology.
Because the dynamic DR two-dimensional shooting result is difficult to realize multi-space angle observation of the organ to be detected and easy to cause missed diagnosis, a reconstructed image or an image group of the fault plane of the scanned patient needs to be provided through three-dimensional scanning reconstruction.
When the three-dimensional scanning reconstruction algorithm is applied to the cone beam DR system, the traditional three-dimensional scanning reconstruction algorithm still has the problem of edge detail loss due to the imaging characteristics of ultra-low dose and limited angle of the cone beam DR system.
Disclosure of Invention
The application aims to provide a cone beam three-dimensional DR reconstruction method and system based on deep learning, which not only considers effective information of limited angle projection data, but also can effectively remove noise caused by ultra-low dose, and can effectively and quickly reconstruct a dynamic DR three-dimensional reconstruction image with ultra-high resolution through an end-to-end mode.
In a first aspect, the present application provides a cone beam three-dimensional DR reconstruction method and system based on deep learning, which adopt the following technical scheme:
a cone beam three-dimensional DR reconstruction method based on deep learning comprises the following steps:
obtaining a DR image sequence frame;
performing image correction on the DR image sequence frame through a preset geometric correction algorithm, and acquiring pose prior information representing an image correction angle;
performing feature extraction on the DR image sequence frame through a preset feature extraction network, and acquiring granularity prior information representing features of different network layers;
modeling the pose prior information and the granularity prior information to acquire fusion prior information representing local detail characteristics to global characteristic associated information;
acquiring a perception feature extractor through transfer learning based on a preset feature extraction network;
according to the perception feature extractor and a preset generation countermeasure network, a DR three-dimensional reconstruction model is obtained;
and generating a DR three-dimensional image according to the DR image sequence frame, the fusion prior information and the DR three-dimensional reconstruction model.
By the technical scheme, adaptive geometric correction can be performed on the cone beam DR image, the influence of geometric deviation on three-dimensional reconstruction is reduced, in addition, the feature information of different levels of a depth network is obtained and is fused with pose information after geometric correction to obtain fusion prior information, the description from the whole to local associated information is increased, the robustness of the model under the scene with fuzzy background and missing edge details is improved, meanwhile, the fusion prior information can help the three-dimensional reconstruction network to quickly learn partial key features, and the convergence speed is increased.
Optionally, the image correction is performed on the DR image sequence frame through a preset geometric correction algorithm, and pose prior information representing an image correction angle is obtained, including:
acquiring a corrected image from the DR image sequence frame through a preset geometric correction algorithm;
correcting the image according to the corrected image and a preset standard, and updating a preset geometric correction algorithm through a semi-supervised learning strategy;
and acquiring pose prior information according to the DR image sequence frame and the updated geometric correction algorithm.
Optionally, the performing feature extraction on the DR image sequence frame through a preset feature extraction network, and acquiring granularity prior information representing features of different network layers includes:
performing feature extraction on the DR image sequence frame, and respectively acquiring coarse-grained prior information, fine-grained prior information and fine-grained prior information under different resolutions;
and acquiring the granularity prior information according to the coarse granularity prior information, the fine granularity prior information and the fine prior information.
Optionally, the modeling the pose prior information and the granularity prior information to obtain fusion prior information representing the local detail feature to the global feature association information includes:
modeling the granularity prior information through a preset attention module to obtain an attention diagram representing the correlation information among the pixel characteristics;
and fusing the granularity prior information based on the pose prior information and the attention diagram to obtain the fusion prior information.
Optionally, the generating a countermeasure network according to the perceptual feature extractor and the preset to obtain a DR three-dimensional reconstruction model includes:
according to a preset generation countermeasure network and a perception feature extractor, a three-dimensional reconstruction network is constructed;
and performing a preset round of iterative training on the three-dimensional reconstruction network according to the DR image sequence frame to obtain a DR three-dimensional reconstruction model.
Optionally, the preset generated confrontation network includes a generator and a discriminator, and each round of training process includes:
performing feature extraction on the DR three-dimensional image generated by the generator through a perception feature extractor, and acquiring perception prior information;
adding the perception prior information into a discriminator, calculating the difference between the DR three-dimensional image and a preset real DR three-dimensional image through the discriminator, and feeding back corresponding parameters to a generator;
and updating the generator according to the feedback parameters, and acquiring the DR three-dimensional image from the current DR image through the generator again.
Optionally, after generating the DR three-dimensional reconstructed image according to the DR image sequence frame, the fusion prior information, and the DR three-dimensional reconstructed model, the method includes:
performing feature extraction on the DR three-dimensional image to obtain sequence frame features;
acquiring a super-resolution semantic feature through a preset encoder-decoder according to the sequence frame feature;
and correcting the DR three-dimensional image according to the super-resolution semantic features to obtain the DR three-dimensional image with higher resolution.
Optionally, the obtaining the super-resolution semantic features according to the sequence frame features by a preset encoder-decoder includes:
carrying out position coding on the sequence frame characteristics to obtain sequence coding characteristics;
acquiring a priori information embedded sequence from the sequence frame characteristics through a perception characteristic extractor;
and embedding the sequence according to the sequence coding characteristics and the prior information, and obtaining the super-resolution semantic characteristics through decoding.
In a second aspect, the present application provides a deep learning-based cone beam three-dimensional DR reconstruction system, including:
the image data acquisition module is used for acquiring a DR image sequence frame;
the prior information extraction module is used for carrying out image correction on the DR image sequence frame through a preset geometric correction algorithm, acquiring pose prior information representing an image correction angle, then carrying out feature extraction on the DR image sequence frame through a preset feature extraction network, acquiring granularity prior information representing different network level features, and finally modeling the pose prior information and the granularity prior information to acquire fusion prior information representing local detail features to global feature associated information;
the reconstruction model obtaining module is used for obtaining a perception feature extractor through transfer learning based on a preset feature extraction network, and obtaining a DR three-dimensional reconstruction model according to the perception feature extractor and a preset generation countermeasure network;
and the three-dimensional image generation module is used for generating the DR three-dimensional image according to the DR image sequence frame, the fusion prior information and the DR three-dimensional reconstruction model.
In a third aspect, the present application provides a computer readable storage medium storing a computer program that can be loaded by a processor and execute the above-mentioned cone beam three-dimensional DR reconstruction method based on deep learning.
In summary, according to the method, firstly, a self-adaptive geometric correction algorithm based on the semi-supervised learning technology is constructed, so that the deep network can still realize more accurate geometric correction under the condition of smaller training data volume, and the algorithm can realize self-adaptive real-time geometric correction for images acquired from different angles, so that more accurate three-dimensional reconstruction is realized; in addition, complete prior information such as pose prior, morphological multi-granularity prior and the like is introduced and fused, so that the robustness of the algorithm is improved in the aspects of noise suppression, structure preservation and detail recovery; in addition, the whole network architecture is embedded with light structures which can be operated in parallel, such as deep separable convolution, residual modules, jump connection and the like, so that the network convergence process is accelerated.
Drawings
Fig. 1 is a flowchart of a cone beam three-dimensional DR reconstruction method based on deep learning according to an embodiment of the present application;
fig. 2 is a schematic diagram of a three-dimensional reconstruction network in a deep learning-based cone beam three-dimensional DR reconstruction method according to an embodiment of the present application;
fig. 3 is a flowchart of each round of training of a three-dimensional reconstruction network in a cone-beam three-dimensional DR reconstruction method based on deep learning according to an embodiment of the present application;
FIG. 4 is a flow chart for optimizing the generation of a three-dimensional reconstructed image as provided by an embodiment of the present application;
fig. 5 is a schematic diagram of a deep learning-based cone beam three-dimensional DR reconstruction system according to an embodiment of the present application.
Detailed Description
The present application is described in further detail below with reference to fig. 1-5.
The application provides a cone beam three-dimensional DR reconstruction method based on deep learning, which is shown in figure 1 and comprises the following steps:
and S100, acquiring a DR image sequence frame.
In the embodiment of the application, the DR image sequence frame is a group of ultralow-dose cone-beam two-dimensional truncated images acquired by a DR device at different angles.
After the DR original image is acquired, the DR original image is further subjected to corresponding preprocessing, for example, the image is trimmed and scaled according to the network model parameters to adjust the image size.
S200, image correction is carried out on the DR image sequence frame through a preset geometric correction algorithm, and pose prior information representing an image correction angle is obtained.
The preset geometric correction algorithm is a self-adaptive geometric correction algorithm based on a semi-supervised learning technology, the pose prior information represents corrected image information obtained by geometrically correcting an original image, the original image is an image under different projection angles, corrected images at different projection angles can be obtained by geometrically correcting the original image, and the pose prior information can be formed.
In the process of acquiring a DR image, as movement inertia exists to a certain degree due to movement and rotation of an X-ray source during multi-angle shooting, certain pose information deviation can be caused, and in the calculation of projection positioning in the three-dimensional image reconstruction process, the deviation can finally cause a reconstructed image to generate geometric artifacts, so that the original image is corrected by a self-adaptive geometric correction algorithm according to different errors, and pose prior information is acquired, so as to eliminate the geometric artifacts in the three-dimensional image reconstruction process.
In the embodiment of the application, image correction is performed on a DR image sequence frame through a preset geometric correction algorithm, and pose prior information representing an image correction angle is obtained, which specifically includes the following steps:
s210, obtaining a corrected image for the DR image sequence frame through a preset geometric correction algorithm.
In the embodiment of the application, original images acquired by DR equipment at different angles are used as data input and are transmitted to a self-adaptive geometric correction algorithm based on a semi-supervised learning technology to obtain a corrected image.
And S220, correcting the image according to the corrected image and a preset standard, and updating a preset geometric correction algorithm through a semi-supervised learning strategy.
In the embodiment of the application, the preset standard image is a manually corrected image, after the corrected image of the original image after geometric correction is obtained, the manually corrected image is used as a reference standard to constrain the corrected image, a correction loss value is obtained, then the correction loss value is reversely propagated to the geometric correction network to update the gradient, and the network has more excellent correction performance through continuous training and updating.
Due to the fact that the preset geometric correction algorithm can be updated through the existing original image and the corresponding manual correction image in the training stage, the correction performance of the algorithm is continuously improved, the geometric correction algorithm can be adaptively corrected in real time for images acquired from different angles in the testing stage, the influence of geometric artifacts in the three-dimensional reconstruction process is eliminated, and accurate three-dimensional reconstruction is achieved.
And S230, acquiring pose prior information according to the DR image sequence frame and the updated geometric correction algorithm.
After the last updated geometric correction algorithm is obtained, corresponding corrected images are obtained for the original images at different angles according to the updated geometric correction algorithm, and pose prior information is obtained according to the corrected images so as to provide effective information support for a subsequent three-dimensional reconstruction network.
S300, extracting the characteristics of the DR image sequence frame through a preset characteristic extraction network, and acquiring granularity prior information representing different network level characteristics.
The method comprises the steps that the granularity prior information comprises coarse granularity prior information, fine granularity prior information and fine prior information, and the characteristic information of an original image under different resolutions extracted by a characteristic extraction network is characterized.
The preset feature extraction network adopts a feature extraction network based on CSPDarkNet53, the CSPDarkNet53 takes a CSP module as a framework, the problem of huge calculation amount caused by repeated gradient information in forward reasoning is solved, and in addition, the CSP module divides an input feature diagram of a basic layer into two parts, so that the calculation of partial features bypasses an intensive module and reduces the calculation complexity. In addition, the network uses a Mish activation function after Batch Normalization (BN), has the characteristic of rapid ReLu convergence, is smoother than a ReLu gradient, can better stabilize gradient flow for the permission of a slight negative value, and is helpful for accelerating the convergence speed of the feature extraction network, so that the prior features with different granularities are efficiently fused.
In the embodiment of the application, the method for extracting the characteristics of the DR image sequence frame through the preset characteristic extraction network and obtaining the granularity prior information representing the characteristics of different network layers specifically comprises the following steps:
s310, extracting the characteristics of the DR image sequence frame, and respectively obtaining coarse-grained prior information, fine-grained prior information and fine-grained prior information under different resolutions.
In the embodiment of the application, the DR image is subjected to feature extraction through a feature extraction network based on CSPDarkNet53, features corresponding to different depths of the network are obtained, corresponding feature information is respectively obtained according to a preset resolution ratio and is recorded as coarse-grained prior information, fine-grained prior information and fine-grained prior information.
S320, obtaining the granularity prior information according to the coarse granularity prior information, the fine granularity prior information and the fine granularity prior information.
In the embodiment of the application, after the coarse-grained prior information, the fine-grained prior information and the fine-grained prior information are obtained, the coarse-grained prior information, the fine-grained prior information and the fine-grained prior information are fused, resolution ratios of the coarse-grained prior information, the fine-grained prior information and the fine-grained prior information are converted into consistency in an interpolation or deconvolution mode, and then the feature vectors are added according to corresponding channels to obtain the grained prior information.
By combining the characteristics corresponding to different depths of the network, the description of the network model on the whole to local associated information is increased while the local key characteristics are highlighted, edge artifacts are reduced beneficially, the granularity information can be used as prior information to be sent to the image three-dimensional reconstruction network, the image three-dimensional reconstruction network is helped to learn partial key characteristics in advance, and the network convergence speed is increased.
S400, modeling is carried out on the pose prior information and the granularity prior information so as to obtain fusion prior information representing the local detail features to the global feature associated information.
In the embodiment of the application, the granularity prior information representation is semantic description from local information characteristics to global characteristic information, the pose prior information representation is corrected image angle information, and accordingly context correlation can be given to the characteristics through the pose prior information, namely the spatial position information of the characteristics in an original image, and fusion prior information is formed by modeling the pose prior information and the granularity prior information, so that the network can be helped to learn more key detail characteristics, and the robustness of the network under the scenes such as fuzzy background, missing edge details and the like is prompted.
In the embodiment of the present application, modeling is performed on pose prior information and granularity prior information to obtain fusion prior information representing local detail features to global feature association information, and the method specifically includes the following steps:
and S410, modeling the granularity prior information through a preset attention module to obtain an attention diagram representing the correlation information among the pixel characteristics.
And S420, fusing the granularity prior information based on the pose prior information and the attention map to obtain the fusion prior information.
The preset attention module is mainly a spatial dimension attention module, and focuses on modeling the characteristics of each pixel point in the characteristic diagram. Attention is paid to the fact that the characteristic of each pixel point is associated with other pixel point characteristics.
In the embodiment of the present application, modeling the particle size prior information through the preset attention module is actually to perform global average pooling and global maximum pooling operations on the features characterized by the particle size prior information, and perform operations such as merging, convolution, activating a function, and the like on the generated new feature map to generate an attention map with weight information.
The granularity prior information and the pose prior information are used as context information, and the granularity prior information representing the DR image feature information is overlaid through an attention map, so that feature expression of a key region is improved, and description of local detail feature associated information is highlighted. And finally, fusing the superposed features with pose prior information, and associating the positions of the local pixel point features in the global features by taking the pose prior information as global information to obtain fusion prior information representing the local detail features to the global feature associated information.
S500, acquiring a perception feature extractor through transfer learning based on a preset feature extraction network.
The preset feature extraction network is the feature extraction network based on the CSPDarkNet53, and the perception feature extractor is a trained feature extraction model.
In the embodiment of the application, the feature extraction network based on the CSPDarkNet53 is trained by using the original image as training data, and after the training is completed, the corresponding granularity prior information can be acquired by inputting the original image.
Because the three-dimensional reconstruction network needs to acquire corresponding fusion prior information for the three-dimensional reconstruction image, the input data of the feature extraction network based on the CSPDarkNet53 is a two-dimensional image, and the current input requirement is three-dimensional, a 2D-3D transfer learning mode is adopted, and an initial three-dimensional feature extraction model, namely a perception feature extractor, is acquired through the trained feature extraction network based on the CSPDarkNet53, and the perception prior information acquired through the perception feature extractor is basically the same as the granularity prior information, so that the network can be helped to learn partial key features more quickly, and the network convergence speed is improved.
S600, according to the perception feature extractor and a preset generation countermeasure network, a DR three-dimensional reconstruction model is obtained.
In the embodiment of the application, the DR three-dimensional reconstruction model is a network model obtained by training a constructed DR three-dimensional reconstruction network according to a DR image sequence frame.
In the embodiment of the application, the method for acquiring the DR three-dimensional reconstruction model according to the DR image sequence frame and the perception feature extractor specifically comprises the following steps:
s610, according to the preset generation countermeasure network and the perception feature extractor, a three-dimensional reconstruction network is constructed, and the three-dimensional reconstruction network is shown in the figure 2.
The preset generation countermeasure network comprises a generator and a discriminator.
In the embodiment of the application, the generator is a deep convolutional neural network based on a three-dimensional encoder-decoder and a jump connection mechanism, and the main task of the generator is to generate a DR three-dimensional distribution image according to an original image acquired by a cone beam DR system.
The discriminator adopts a ResNet network structure and is used for comparing the difference between the generated DR three-dimensional image and the real DR three-dimensional image to obtain a discrimination value, and then the discrimination value is fed back to the loss function of the generator for optimization training.
The perception feature extractor is used for extracting features of an output image from the generator and a real DR three-dimensional image to obtain perception prior information, adding the perception prior information to the discriminator, calculating the difference between the DR three-dimensional reconstruction image of the generator and the real DR three-dimensional image through the discriminator, and feeding the difference back to a loss function of the generator through a counterstudy strategy.
And S620, performing a preset round of iterative training on the three-dimensional reconstruction network according to the DR image sequence frame to obtain a DR three-dimensional reconstruction model.
In the embodiment of the application, after the three-dimensional reconstruction network frame is built, training is performed according to a DR image sequence frame, and since the generator and the discriminator perform learning in a mutual confrontation manner, training is performed in an alternate iteration manner, that is, the output of the generator is used as the input of the discriminator, the output of the discriminator is fed back to the generator, iteration is performed according to a specified turn, and a DR three-dimensional reconstruction model is obtained according to training parameters after the iteration is finished.
In the embodiment of the present application, referring to fig. 3, the training process of each round specifically includes the following steps:
and S621, performing feature extraction on the DR three-dimensional image generated by the generator through a perception feature extractor, and acquiring perception prior information.
In the embodiment of the application, the generator generates a corresponding DR three-dimensional reconstruction image according to a single-frame DR image in a DR image sequence frame, the perception feature extractor performs feature extraction on the DR three-dimensional reconstruction image generated by the generator, performs feature extraction on a preset real three-dimensional reconstruction image, and acquires corresponding perception prior information respectively, the perception prior information is the same as the mentioned granularity prior information, the characteristic is the thickness and granularity information of different layers of a depth network, and the perception prior information is transmitted to the discriminator to help the discriminator to learn more recognizable features, so that the discrimination of the discriminator on the DR three-dimensional image is improved.
And S622, adding the perception prior information into a discriminator, calculating the difference between the DR three-dimensional image and a preset real DR three-dimensional image through the discriminator, and feeding back corresponding parameters to a generator.
In the embodiment of the application, after the perception prior information is transmitted to the discriminator, the difference between the DR three-dimensional reconstructed image generated by the generator and the real DR three-dimensional image is calculated by the discriminator, and the difference is fed back to the loss function of the generator in a parameter form, so that the DR three-dimensional image generated by the generator and the real DR three-dimensional image have the same distribution through multiple rounds of counterstudy.
And S623, updating the generator according to the feedback parameters, and obtaining the DR three-dimensional image from the current DR image through the generator again.
In the embodiment of the application, when the generator acquires the difference parameter fed back by the discriminator, the loss function is correspondingly adjusted according to the parameter, and the constraint of the generator on the generated image is increased, so that the DR three-dimensional image generated by the generator is close to the real DR three-dimensional image in a targeted manner.
The updated generator can generate DR three-dimensional images again for the current DR images, through multi-round confrontation iterative learning, the DR three-dimensional images generated by the last generator and the real DR three-dimensional images have the same distribution, then the same iterative training is carried out on different DR image sequence frames in sequence, or the DR image sequence frames are divided according to preset batches, training is carried out in batches, and finally the DR three-dimensional reconstruction model is obtained.
S700, generating a DR three-dimensional image according to the DR image sequence frame, the fusion prior information and the DR three-dimensional reconstruction model.
In the embodiment of the application, after the DR three-dimensional reconstruction network model is obtained, the corresponding DR three-dimensional image can be obtained through the DR three-dimensional reconstruction network model based on the DR image sequence frame, and because the whole image three-dimensional reconstruction process is from the original two-dimensional image to the three-dimensional image, the accuracy of the three-dimensional reconstruction model is improved, and simultaneously the completeness of noise interference information and effective information existing in the original image is also required to be considered.
Therefore, the fusion prior information is added to the three-dimensional reconstruction process of the image as auxiliary information, the fusion prior information comprises pose prior information and granularity prior information, the pose prior information represents that corrected image information is obtained after the original image is geometrically corrected, corresponding corrected position states can be obtained for the original images at different angles, and the geometric artifacts of the three-dimensional reconstructed image caused by pose deviation can be reduced.
The granularity prior information represents the global features and the local features acquired by the image at different levels of the depth network, and the features at different levels are associated, so that the local key features can be highlighted, the description of the network on the overall to local associated information can be increased, and the influence caused by the problems of background blurring or edge detail missing and the like due to the effective information missing at a limited angle can be reduced to a certain extent.
In the embodiment of the present application, after obtaining a DR three-dimensional image according to a DR image sequence frame, fusion prior information, and a DR three-dimensional reconstruction model, the DR three-dimensional image after reconstruction is further optimized, mainly to improve texture details in the three-dimensional reconstruction image, including phenomena such as jagged edges, streak artifacts, and discontinuity, as shown in fig. 4, the method specifically includes the following steps:
and S810, performing feature extraction on the DR three-dimensional image to acquire sequence frame features.
In the embodiment of the application, the DR three-dimensional image subjected to three-dimensional reconstruction is subjected to feature extraction, the adopted feature extraction network is a space-time circulation neural network, the space-time circulation neural network is additionally provided with a space-time memory state on the basis of a gating circulation unit GRU, information transfer among different layers is reserved when memory updating of each layer at different moments in a stacking state is considered, and the network model can be better assisted to model the spatial position relation and semantic relation information of the image by using the related information of adjacent frame images in the process of performing feature extraction on the DR three-dimensional image sequence frame.
And S820, acquiring the super-resolution semantic features through a preset encoder-decoder according to the sequence frame features and the fusion prior information.
In the embodiment of the application, the preset encoder-decoder adopts a transform encoding-decoding structure, and comprises a plurality of stacked self-attention mechanisms self-attentions besides two modules for encoding and decoding, so that different degrees of attention can be given to serialized data, the expression capability of a model is increased, and a parallel training mode is adopted, namely all sequence data are trained simultaneously, and the calculation efficiency is increased.
The super-resolution semantic features are image features generated by a transform encoding-decoding module, and fully utilize the relation between frames in a sequence image, the spatial position relation of local features and the association information of image global features and local features. The features have richer semantic information, natural and real texture details of the image can be better restored, and the image is corrected correspondingly to generate a higher-resolution image.
In the embodiment of the present application, the method for obtaining the super-resolution semantic feature through a preset encoder-decoder according to the sequence frame feature specifically includes the following steps:
and S821, carrying out position coding on the sequence frame characteristics to obtain sequence coding characteristics.
In the embodiment of the present application, since the sequence frame characteristics are used as the input of the transform coding-decoding module, and the sequence frame characteristics are used as a set of serialized data, the order or the position relationship thereof is particularly important, because if the position is deviated, the network may also deviate from the understanding of the whole data, so that the sequence frame characteristics need to be position-coded to specify the position of the sequence of each data in the sequence frame.
After the sequence frame features are subjected to position coding, the sequence coding features are obtained through a coder in a transform coding-decoding module.
And S822, acquiring a priori information embedded sequence from the sequence frame characteristics through a perception characteristic extractor.
In the embodiment of the application, the prior information is obtained by extracting the characteristics of the DR three-dimensional image sequence frame by the above-mentioned perceptual characteristic extractor, local key characteristics can be better highlighted by embedding the prior information, the attention of the model to the local characteristics can be increased when the model passes through a self-attention module in a decoder to highlight more detailed information, in addition, description of global to local associated information is also added to the prior information, more semantic information can be given to the decoded characteristics after the model passes through the self-attention module, and the expression capability of the characteristics is enhanced.
And S823, embedding the sequence according to the sequence coding features and the prior information, and obtaining the super-resolution semantic features through decoding.
In the embodiment of the present application, after the a priori information embedded sequence is obtained, the embedded sequence and the sequence coding features are sent to a decoder, and the super-resolution semantic features can be obtained through decoding.
And S830, optimizing texture details of the DR three-dimensional image according to the super-resolution semantic features.
In the embodiment of the application, after the super-resolution semantic features are obtained, the super-resolution semantic features are sent to a 3D-U-Net network together with an input sequence frame image and the sequence frame features, the 3D-U-Net adopts 3D convolution and is consistent with the dimension of the input image, input image data does not need to be segmented, feature extraction and analysis can be directly carried out on the input image, in addition, the 3D-U-Ne can better help to carry out super-resolution reconstruction on the image by retaining high pixel feature information so as to optimize texture details of the DR three-dimensional image, and therefore the quality of the reconstructed image is improved.
The embodiment of the present application further provides a cone beam three-dimensional DR reconstruction system based on deep learning, referring to fig. 5, the system includes: the image reconstruction method comprises an image data acquisition module 101, a priori information extraction module 102, a reconstruction model acquisition module 103 and a three-dimensional image generation module 104.
The image data acquiring module 101 is configured to acquire DR image sequence frames.
The prior information extraction module 102 is configured to perform image correction on a DR image sequence frame through a preset geometric correction algorithm, acquire pose prior information representing an image correction angle, perform feature extraction on the DR image sequence frame through a preset feature extraction network, acquire granularity prior information representing different network level features, and finally model the pose prior information and the granularity prior information to acquire fusion prior information representing local detail features and global feature association information.
The reconstruction model obtaining module 103 is configured to obtain a perceptual feature extractor through migration learning based on a preset feature extraction network, and obtain a DR three-dimensional reconstruction model according to the perceptual feature extractor and a preset generation countermeasure network;
the three-dimensional image generation module 104 is configured to generate a DR three-dimensional image according to the DR image sequence frame, the fusion prior information, and the DR three-dimensional reconstruction model.
In this embodiment of the application, the image data obtaining module 101 is specifically configured to obtain an original image acquired by a DR device, and perform corresponding preprocessing on the image, where the preprocessing performs corresponding cropping and scaling on the original image, and also generates a real three-dimensional DR image from a part of the original image by using a manual method, so as to use the real three-dimensional DR image as a training data set, and uses another part of the real three-dimensional DR image as a test data set.
The prior information extraction module 102 is specifically configured to perform feature extraction through a preset feature extraction network based on the CSPDarkNet53 according to the acquired DR original image to obtain granularity prior information, and perform modeling and fusion on pose prior information and granularity prior information acquired by a geometric correction algorithm through an attention module to obtain fusion prior information. In both the training stage and the testing stage, fusion prior information is obtained for the input image according to the feature extraction network and the geometric correction algorithm.
The reconstruction model obtaining module 103 is specifically configured to build a reconstruction network framework and train a reconstruction network to obtain a reconstruction network model. The method comprises the steps of building a reconstruction network framework, adding a sensing feature extractor on the basis of a preset generation countermeasure network, wherein the sensing feature extractor is obtained by a feature extraction network model based on CSPDarkNet53 through transfer learning, and training the network according to DR image sequence frames in a training data set after the reconstruction network framework is built, so as to obtain a final three-dimensional reconstruction network model.
The three-dimensional image generation module 104 is specifically configured to generate a DR three-dimensional image by using the DR image sequence frame in the test data set as an input according to the trained three-dimensional reconstruction network model.
Embodiments of the present application further provide a computer-readable storage medium storing a computer program that can be loaded by a processor and execute any one of the above methods for cone beam three-dimensional DR reconstruction based on deep learning.
The embodiments of the present invention are preferred embodiments of the present application, and the scope of protection of the present application is not limited by the embodiments, so: all equivalent changes made according to the principles of the present application should be covered by the protection scope of the present application.
Claims (10)
1. A cone beam three-dimensional DR reconstruction method based on deep learning is characterized by comprising the following steps:
acquiring a DR image sequence frame, wherein the DR image sequence frame is a group of ultra-low dose cone beam two-dimensional truncated images acquired by DR equipment at different angles;
performing image correction on the DR image sequence frame through a preset geometric correction algorithm, and acquiring pose prior information representing an image correction angle;
performing feature extraction on the DR image sequence frame through a preset feature extraction network, and acquiring granularity prior information representing different network level features;
modeling the pose prior information and the granularity prior information to acquire fusion prior information representing local detail features to global feature association information;
acquiring a perception feature extractor through transfer learning based on a preset feature extraction network;
obtaining a DR three-dimensional reconstruction model according to the perception feature extractor and a preset generation countermeasure network;
and generating a DR three-dimensional image according to the DR image sequence frame, the fusion prior information and the DR three-dimensional reconstruction model.
2. The deep learning-based cone beam three-dimensional DR reconstruction method of claim 1, wherein the image correction is performed on the DR image sequence frame by a preset geometric correction algorithm, and pose prior information representing an image correction angle is obtained, including:
acquiring a corrected image from the DR image sequence frame through a preset geometric correction algorithm;
correcting the image according to the corrected image and a preset standard, and updating a preset geometric correction algorithm through a semi-supervised learning strategy;
and acquiring pose prior information according to the DR image sequence frame and the updated geometric correction algorithm.
3. The deep learning-based cone-beam three-dimensional DR reconstruction method of claim 1, wherein the granular prior information comprises: the method comprises the following steps of extracting the characteristics of a DR image sequence frame through a preset characteristic extraction network, and acquiring granularity prior information representing different network level characteristics, wherein the method comprises the following steps:
performing feature extraction on the DR image sequence frame, and respectively acquiring coarse-grained prior information, fine-grained prior information and fine-grained prior information under different resolutions;
and acquiring the granularity prior information according to the coarse granularity prior information, the fine granularity prior information and the fine prior information.
4. The deep learning-based cone beam three-dimensional DR reconstruction method according to claim 1, wherein the modeling pose prior information and granularity prior information to obtain fusion prior information characterizing local detail features to global feature association information comprises:
modeling the granularity prior information through a preset attention module to obtain an attention diagram representing the correlation information among the pixel characteristics;
and fusing the granularity prior information based on the pose prior information and the attention diagram to obtain the fusion prior information.
5. The deep learning-based cone beam three-dimensional DR reconstruction method of claim 1, wherein the obtaining of the DR three-dimensional reconstruction model according to the perceptual feature extractor and the pre-set generation countermeasure network comprises:
according to a preset generation countermeasure network and a perception feature extractor, a three-dimensional reconstruction network is constructed;
and performing a preset round of iterative training on the three-dimensional reconstruction network according to the DR image sequence frame to obtain a DR three-dimensional reconstruction model.
6. The deep learning-based cone beam three-dimensional DR reconstruction method of claim 5, wherein the pre-set generation countermeasure network comprises a generator and a discriminator, and each round of training process comprises:
performing feature extraction on the DR three-dimensional image generated by the generator through a perception feature extractor, and acquiring perception prior information;
adding perception prior information into a discriminator, calculating the difference between the DR three-dimensional image and a preset real DR three-dimensional image through the discriminator, and feeding corresponding parameters back to a generator;
and updating the generator according to the feedback parameters, and acquiring the DR three-dimensional image from the current DR image through the generator again.
7. The method for deep learning-based cone beam three-dimensional DR reconstruction method according to claim 1, wherein after generating DR three-dimensional reconstruction image according to DR image sequence frame, fusion prior information and DR three-dimensional reconstruction model, the method comprises:
performing feature extraction on the DR three-dimensional image to obtain sequence frame features;
acquiring a super-resolution semantic feature through a preset encoder-decoder according to the sequence frame feature;
and correcting the DR three-dimensional image according to the super-resolution semantic features to obtain the DR three-dimensional image with higher resolution.
8. The deep learning-based cone beam three-dimensional DR reconstruction method of claim 7, wherein the obtaining the super-resolution semantic features by a preset encoder-decoder according to the sequence frame features comprises:
carrying out position coding on the sequence frame characteristics to obtain sequence coding characteristics;
acquiring a priori information embedded sequence from the sequence frame characteristics through a perception characteristic extractor;
and embedding the sequence according to the sequence coding characteristics and the prior information, and obtaining the super-resolution semantic characteristics through decoding.
9. A cone beam three-dimensional DR reconstruction system based on deep learning is characterized by comprising:
the system comprises an image data acquisition module (101) for acquiring a DR image sequence frame, wherein the DR image sequence frame is a group of ultra-low dose cone beam two-dimensional truncated images acquired by DR equipment at different angles;
the prior information extraction module (102) is used for carrying out image correction on the DR image sequence frame through a preset geometric correction algorithm, acquiring pose prior information representing an image correction angle, carrying out feature extraction on the DR image sequence frame through a preset feature extraction network, acquiring granularity prior information representing different network level features, and finally modeling the pose prior information and the granularity prior information to acquire fusion prior information representing local detail features to global feature associated information;
the reconstruction model obtaining module (103) is used for obtaining a perception feature extractor through transfer learning based on a preset feature extraction network, and obtaining a DR three-dimensional reconstruction model according to the perception feature extractor and a preset generation countermeasure network;
and the three-dimensional image generation module (104) is used for generating a DR three-dimensional image according to the DR image sequence frame, the fusion prior information and the DR three-dimensional reconstruction model.
10. A computer readable storage medium storing a computer program that can be loaded by a processor and executes a method for deep learning based cone-beam three-dimensional DR reconstruction as claimed in any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211347742.2A CN115393534B (en) | 2022-10-31 | 2022-10-31 | Deep learning-based cone beam three-dimensional DR (digital radiography) reconstruction method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211347742.2A CN115393534B (en) | 2022-10-31 | 2022-10-31 | Deep learning-based cone beam three-dimensional DR (digital radiography) reconstruction method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115393534A CN115393534A (en) | 2022-11-25 |
| CN115393534B true CN115393534B (en) | 2023-01-20 |
Family
ID=84115135
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211347742.2A Active CN115393534B (en) | 2022-10-31 | 2022-10-31 | Deep learning-based cone beam three-dimensional DR (digital radiography) reconstruction method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115393534B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116881639A (en) * | 2023-07-10 | 2023-10-13 | 国网四川省电力公司营销服务中心 | Electricity larceny data synthesis method based on generation countermeasure network |
| CN117064446B (en) * | 2023-10-13 | 2024-01-12 | 山东大学 | Intravascular ultrasound-based vascular dynamic three-dimensional reconstruction system |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112819911A (en) * | 2021-01-23 | 2021-05-18 | 西安交通大学 | Four-dimensional cone beam CT reconstruction image enhancement algorithm based on N-net and CycN-net network structures |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109745062B (en) * | 2019-01-30 | 2020-01-10 | 腾讯科技（深圳）有限公司 | CT image generation method, device, equipment and storage medium |
| CN111179366B (en) * | 2019-12-18 | 2023-04-25 | 深圳先进技术研究院 | Anatomical structure difference priori based low-dose image reconstruction method and system |
| WO2021120069A1 (en) * | 2019-12-18 | 2021-06-24 | 深圳先进技术研究院 | Low-dose image reconstruction method and system on basis of a priori differences between anatomical structures |
| WO2021184195A1 (en) * | 2020-03-17 | 2021-09-23 | 中国科学院深圳先进技术研究院 | Medical image reconstruction method, and medical image reconstruction network training method and apparatus |
| CN112435307B (en) * | 2020-11-26 | 2022-05-10 | 浙江大学 | Deep neural network assisted four-dimensional cone beam CT image reconstruction method |
| EP4080459A1 (en) * | 2021-04-23 | 2022-10-26 | Koninklijke Philips N.V. | Machine learning-based improvement in iterative image reconstruction |
-
2022
- 2022-10-31 CN CN202211347742.2A patent/CN115393534B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112819911A (en) * | 2021-01-23 | 2021-05-18 | 西安交通大学 | Four-dimensional cone beam CT reconstruction image enhancement algorithm based on N-net and CycN-net network structures |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115393534A (en) | 2022-11-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115393534B (en) | Deep learning-based cone beam three-dimensional DR (digital radiography) reconstruction method and system | |
| CN113658051B (en) | Image defogging method and system based on cyclic generation countermeasure network | |
| CN108319932B (en) | Multi-image face alignment method and device based on generative confrontation network | |
| TWI709107B (en) | Image feature extraction method and saliency prediction method including the same | |
| CN111353940B (en) | Image super-resolution reconstruction method based on deep learning iterative up-down sampling | |
| JP2008513882A (en) | Video image processing system and video image processing method | |
| CN112614169B (en) | 2D/3D spine CT (computed tomography) level registration method based on deep learning network | |
| CN113554742B (en) | Three-dimensional image reconstruction method, device, equipment and storage medium | |
| US20220414838A1 (en) | Image dehazing method and system based on cyclegan | |
| CN112102428B (en) | CT cone beam scanning image reconstruction method, scanning system and storage medium | |
| CN117413300A (en) | Method and system for training quantized nerve radiation field | |
| CN113793272B (en) | Image noise reduction method and device, storage medium and terminal | |
| CN116993926B (en) | Single-view human body three-dimensional reconstruction method | |
| CN113421186A (en) | Apparatus and method for unsupervised video super-resolution using a generation countermeasure network | |
| JP2010028807A (en) | Image processing method, image processing system, and computer-readable memory medium | |
| CN111696167A (en) | Single image super-resolution reconstruction method guided by self-example learning | |
| CN113920270B (en) | Layout reconstruction method and system based on multi-view panorama | |
| CN113469882B (en) | Super-resolution reconstruction method and device based on soil CT image | |
| CN115116468A (en) | Video generation method and device, storage medium and electronic equipment | |
| CN114331813A (en) | PossingGAN network-based image cloning method and system | |
| Rafique et al. | Generative Appearance Flow: A Hybrid Approach for Outdoor View Synthesis. | |
| WO2021039211A1 (en) | Machine learning device, machine learning method, and program | |
| CN113743283A (en) | Mesh topology structure acquisition method and device, electronic equipment and storage medium | |
| Seema et al. | Multi-frame image super-resolution by interpolation and iterative backward projection | |
| Song et al. | Super resolution with sparse gradient-guided attention for suppressing structural distortion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |