CN110660123B - Three-dimensional CT image reconstruction method and device based on neural network and storage medium - Google Patents
Three-dimensional CT image reconstruction method and device based on neural network and storage medium Download PDFInfo
- Publication number
- CN110660123B CN110660123B CN201810721039.0A CN201810721039A CN110660123B CN 110660123 B CN110660123 B CN 110660123B CN 201810721039 A CN201810721039 A CN 201810721039A CN 110660123 B CN110660123 B CN 110660123B
- Authority
- CN
- China
- Prior art keywords
- dimensional
- neural network
- layer
- image
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2210/00—Indexing scheme for image generation or computer graphics
- G06T2210/41—Medical
Abstract
Disclosed are a three-dimensional CT image reconstruction method and device based on a neural network and a storage medium. The method comprises the following steps: carrying out three-dimensional CT scanning on an object to be detected to obtain three-dimensional projection data; processing the three-dimensional projection data by using a first convolution neural network to obtain two-dimensional projection data which are independent layer by layer for a three-dimensional body, wherein the first convolution neural network comprises a plurality of convolution layers; and performing projection domain-to-image domain conversion operator operation equivalent to analytic reconstruction on the two-dimensional projection data of each layer, and obtaining a reconstructed image according to layers to form a three-dimensional image. Or, the image is further processed by utilizing a second neural network to inhibit noise and artifacts. By utilizing the scheme of the embodiment of the disclosure, the CT image with higher quality can be obtained through reconstruction.
Description
Technical Field
Embodiments of the present disclosure relate to radiation imaging, and in particular, to a neural network-based three-dimensional CT image reconstruction method and apparatus, and a storage medium.
Background
X-ray CT (computed-Tomography) imaging systems are widely used in medical, security, industrial nondestructive testing, and other fields. The ray source and the detector acquire a series of projection data according to a certain orbit, and the three-dimensional space distribution of the linear attenuation coefficient of the object to be checked can be obtained through the restoration of an image reconstruction algorithm. The CT image reconstruction process is to recover the linear attenuation coefficient distribution from the data acquired by the detector, and is the core step of CT imaging. Currently, in practical applications, Filtered Back-Projection (Filtered Back-Projection), FDK (Feldkmap-Davis-Kress) type analytical Reconstruction algorithms, and art (algebra Reconstruction technique), map (maximum a spatial) and other iterative Reconstruction methods are mainly used.
With the demand of X-ray CT imaging becoming more and more diversified, the requirement for reducing the radiation dose becomes higher and higher, and the image quality that can be achieved by the design of the reconstruction image method from the conventional thinking approaches the limit. New CT reconstruction techniques need to be developed to obtain higher quality reconstructed images.
Disclosure of Invention
In view of one or more problems in the prior art, a method and apparatus for reconstructing a CT image and a storage medium are provided, which can improve the quality of the reconstructed image.
In one aspect of the present disclosure, a three-dimensional CT image reconstruction method based on a neural network is provided, including the steps of: carrying out three-dimensional CT scanning on an object to be detected to obtain three-dimensional projection data; processing the three-dimensional projection data by using a first convolution neural network to obtain independent two-dimensional projection data of each layer of the three-dimensional volume, wherein the first convolution neural network comprises a plurality of convolution layers; and performing projection domain-to-image domain conversion operator operation equivalent to two-dimensional analysis reconstruction on the two-dimensional projection data of each layer, and obtaining a reconstructed image according to layers to form a three-dimensional image.
According to the embodiment of the disclosure, a neural network layer is constructed according to a two-dimensional analytical reconstruction algorithm, the neural network layer is cascaded with a first convolution neural network to form a comprehensive reconstruction network, and a three-dimensional CT reconstruction network is obtained through training in an integral mode.
According to the embodiment of the disclosure, a neural network layer is constructed according to a two-dimensional analytical reconstruction algorithm and is cascaded with a first convolutional neural network, an image processing network, namely a second convolutional neural network, is further cascaded on the basis to form a comprehensive fine reconstruction network, and a three-dimensional CT reconstruction network is obtained through training in an integral mode.
According to an embodiment of the present disclosure, the three-dimensional CT scan is one of the following scan modes: circular cone beam scanning, helical scanning, linear trajectory scanning.
According to an embodiment of the present disclosure, the first convolutional neural network and the second convolutional neural network may be both U-shaped convolutional neural networks, but are not limited thereto.
According to an embodiment of the present disclosure, the method further comprises the steps of: the two-dimensional projection data is filtered using a ramp filter prior to performing a backprojection operation on the two-dimensional projection data.
According to the embodiment of the disclosure, the reconstructed image is denoised and artifact-suppressed and optionally further detail restoration is performed by using a second convolutional neural network, so as to obtain the result image.
According to an embodiment of the present disclosure, one dimension of the convolution kernel of the convolution layer in the first convolution neural network is the detector pixel sequence, the other dimension is the scan angle, and the scale of the convolution kernel of the convolution layer in the first convolution neural network in the detector pixel sequence dimension and the scale in the scan angle dimension are set independently.
According to an embodiment of the present disclosure, a scale of convolution kernels of convolution layers in the first convolutional neural network in a detector pixel sequence dimension is larger than a scale in a scan angle dimension.
According to an embodiment of the present disclosure, the first convolutional neural network includes at least 3 convolutional layers, each convolutional layer having an activation function for performing a nonlinear operation on the convolved projection data.
According to the embodiment of the disclosure, a domain conversion sub-network is used for carrying out conversion operation from a projection domain to an image domain for carrying out two-dimensional analytic reconstruction on two-dimensional projection data, the network of the domain conversion sub-network comprises a two-dimensional back projection or weighted back projection operation layer, and a weight coefficient between nodes of the operation layer is determined by a geometric parameter of a fan beam circumferential scanning system or a parallel beam circumferential scanning system; the network of the domain conversion sub-network comprises a filter layer to realize ramp filtering or Hilbert filtering; the network of the domain conversion sub-network can comprise a differential operation layer to realize the differential operation of the projection domain data; the domain conversion subnetwork may contain a resampling layer to enable data resampling for the image domain.
According to the embodiment of the disclosure, a first convolution neural network and a second neural network are trained step by step in a residual error mode, a layer-by-layer image preliminary estimation of a three-dimensional body is formed on three-dimensional projection data by using an approximate reconstruction method, two-dimensional projection is carried out on the preliminary estimation and is used as a basis for residual error operation of the first convolution neural network, and the layer-by-layer image preliminary estimation of the three-dimensional body is used as a basis for residual error operation of the second convolution neural network.
In another aspect of the present disclosure, a three-dimensional CT image reconstruction apparatus based on a neural network is provided, including: the CT scanning device is used for carrying out three-dimensional CT scanning on the object to be detected to obtain three-dimensional projection data; a processor configured to: processing the three-dimensional projection data by using a first convolution neural network to obtain independent two-dimensional projection data of each layer of the three-dimensional volume, wherein the first convolution neural network comprises a plurality of convolution layers; and performing projection domain-to-image domain conversion operator operation equivalent to two-dimensional analysis reconstruction on the two-dimensional projection data of each layer, and obtaining a reconstructed image according to layers to form a three-dimensional image.
According to the embodiment of the disclosure, the processor constructs a neural network layer according to a two-dimensional analytical reconstruction algorithm, and the neural network layer is cascaded with the first convolution neural network to form a comprehensive reconstruction network, and a three-dimensional CT reconstruction network is obtained through training in an integral mode.
According to the embodiment of the disclosure, the processor constructs a neural network layer according to a two-dimensional analytical reconstruction algorithm, the neural network layer is cascaded with the first convolutional neural network, an image processing network, namely a second convolutional neural network, is further cascaded on the basis to form a comprehensive fine reconstruction network, and the three-dimensional CT reconstruction network is obtained through integral training.
According to an embodiment of the present disclosure, the three-dimensional CT scan is one of the following scan modes: circular cone beam scanning, helical scanning, linear trajectory scanning.
According to an embodiment of the disclosure, the processor is further configured to: and training the first convolution neural network and the second neural network in a residual error mode, forming layer-by-layer image preliminary estimation of a three-dimensional body by using an approximate reconstruction method on three-dimensional projection data, performing two-dimensional projection on the three-dimensional projection data to serve as a basis for residual error operation used by the first convolution neural network, and using the layer-by-layer image preliminary estimation of the three-dimensional body as a basis for residual error operation by the second convolution neural network.
In a further aspect of the disclosure, a computer-readable medium is presented, in which a computer program is stored, which computer program, when being executed by a processor, performs the steps of: processing the three-dimensional projection data by using a first convolution neural network to obtain independent two-dimensional projection data of each layer of the three-dimensional volume, wherein the first convolution neural network comprises a plurality of convolution layers; and performing projection domain-to-image domain conversion operator operation equivalent to two-dimensional analysis reconstruction on the two-dimensional projection data of each layer, and obtaining a reconstructed image according to layers to form a three-dimensional image.
By utilizing the scheme of the embodiment of the disclosure, the CT image with higher quality can be obtained through reconstruction.
Drawings
For a better understanding of the present invention, reference will now be made in detail to the following drawings, in which:
fig. 1 shows a schematic structural diagram of a CT apparatus according to an embodiment of the present disclosure;
FIG. 2 is a schematic diagram of a control and data processing apparatus in the CT apparatus shown in FIG. 1;
FIG. 3 illustrates an example of medium three-dimensional projection data according to an embodiment of the present disclosure;
fig. 4 shows an example of two-dimensional projection data (sinogram) in accordance with an embodiment of the present disclosure;
FIG. 5 illustrates an overall structural schematic of a convolutional neural network according to one embodiment of the present disclosure;
FIG. 6 is a schematic diagram illustrating the specific operation of the modules in the neural network architecture shown in FIG. 5;
7A, 7B, and 7C show schematic dimension diagrams of filter kernels for use in an apparatus according to an embodiment of the disclosure;
FIG. 8 is a schematic flow chart diagram depicting a CT image reconstruction method in accordance with an embodiment of the present disclosure; and
fig. 9 shows a schematic view of a scanning apparatus for performing helical CT scanning according to another embodiment of the present disclosure.
Detailed Description
Specific embodiments of the present invention will be described in detail below, and it should be noted that the embodiments described herein are only for illustration and are not intended to limit the present invention. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one of ordinary skill in the art that: it is not necessary to employ these specific details to practice the present invention. In other instances, well-known structures, materials, or methods have not been described in detail in order to avoid obscuring the present invention.
Throughout the specification, reference to "one embodiment," "an embodiment," "one example," or "an example" means: the particular features, structures, or characteristics described in connection with the embodiment or example are included in at least one embodiment of the invention. Thus, the appearances of the phrases "in one embodiment," "in an embodiment," "one example" or "an example" in various places throughout this specification are not necessarily all referring to the same embodiment or example. Furthermore, the particular features, structures, or characteristics may be combined in any suitable combination and/or sub-combination in one or more embodiments or examples. Further, as used herein, the term "and/or" will be understood by those of ordinary skill in the art to include any and all combinations of one or more of the associated listed items.
In view of the problems in the prior art, embodiments of the present disclosure provide a three-dimensional CT image reconstruction method based on a neural network. First, a three-dimensional CT scan, such as a circular cone beam scan/helical scan/linear trajectory scan, is performed on an object to be inspected to obtain three-dimensional projection data. Then, the three-dimensional projection data is processed by utilizing a first convolution neural network, and independent two-dimensional projection data of each layer is obtained for the three-dimensional volume, wherein the first convolution neural network comprises a plurality of convolution layers. Next, a projection domain-to-image domain conversion operator operation (for example, a two-dimensional analysis reconstruction filtering back-projection operation) equivalent to two-dimensional analysis reconstruction is performed on the two-dimensional projection data of each layer, and a reconstructed image is obtained by layers to form a three-dimensional volume image. By using the scheme of the embodiment of the disclosure, a reconstructed image with higher quality can be obtained, and especially under the condition of cone beam scanning, cone beam artifacts can be eliminated.
Fig. 1 shows a schematic structural diagram of a CT apparatus according to an embodiment of the present disclosure. As shown in fig. 1, the CT apparatus according to the present embodiment includes an X-ray source 10, a mechanical movement device 50, a detector and data acquisition system 20, and a control and data processing device 60, performs three-dimensional CT scanning on an object under examination 40, such as circular cone beam scanning/helical scanning/linear trajectory scanning, and then performs data processing, such as training of a neural network and image reconstruction using the trained network.
The X-ray source 10 is, for example, an X-ray machine, and the appropriate focal spot size of the X-ray machine is selected according to the resolution of the imaging. In other embodiments, instead of using an X-ray machine, a linear accelerator or the like may be used to generate the X-ray beam.
The mechanical movement device 50 includes a stage and a frame, a control system, and the like. The stage is translatable to adjust the position of the center of rotation, and the gantry is translatable to align the X-ray source (X-ray machine) 10, the detector, and the center of rotation. The present embodiment is described in terms of a circumferential cone-beam scan or a helical scan of a rotating stage, a stationary gantry. Since the movement of the object stage and the frame is relative movement, the method of the embodiment can also be realized in a manner that the object stage is static and the frame rotates.
The detector and data acquisition system 20 includes an X-ray detector and data acquisition circuitry, etc. The X-ray detector may use a solid detector, and may also use a gas detector or other detectors, and embodiments of the present disclosure are not limited thereto. The data acquisition circuit comprises a reading circuit, an acquisition trigger circuit, a data transmission circuit and the like.
The control and data processing device 60 includes, for example, a computer device installed with a control program and a data processing program, and is responsible for performing control of the CT system operation process, including mechanical rotation, electrical control, safety interlock control, and the like, training a neural network, and reconstructing a CT image and the like from projection data using the trained neural network.
Fig. 2 shows a schematic structural diagram of the control and data processing device 200 as shown in fig. 1. As shown in FIG. 2, data acquired by the detector and data acquisition system 20 is stored in the memory device 210 via the interface unit 270 and the bus 280. A Read Only Memory (ROM)220 stores configuration information of the computer data processor and programs. Random Access Memory (RAM)230 is used to temporarily store various data during operation of processor 250. In addition, the storage device 210 also stores therein computer programs for performing data processing, such as a program for training a neural network and a program for reconstructing a CT image, and the like. The internal bus 280 connects the above-described storage device 210, the read only memory 220, the random access memory 230, the input device 240, the processor 250, the display device 260, and the interface unit 270.
After an operation command input by the user through the input device 240 such as a keyboard and a mouse, the instruction codes of the computer program instruct the processor 250 to execute an algorithm for training the neural network and/or an algorithm for reconstructing the CT image, and after obtaining the reconstructed result, display it on the display device 260 such as an LCD display or directly output the processed result in the form of a hard copy such as printing.
According to the embodiment of the present disclosure, the above-mentioned apparatus is used to perform three-dimensional CT scanning on an object to be inspected, and three-dimensional projection data is obtained, as shown in fig. 3. The three-dimensional projection data may be presented in a three-dimensional space consisting of an angular direction, a detector line number and a detector line direction. The three-dimensional projection data as described in FIG. 3 includes a plurality of two-dimensional data representations A1, A2, A3, A4, A5 … …, and so on. The processor 250 of the control device in an embodiment of the present invention processes the three-dimensional projection data using a trained convolutional neural network, and may process such three-dimensional projection data into two-dimensional projection data suitable for a backprojection operation, for example, projection data under a circular fan beam scan or a parallel beam scan. Specifically, the three-dimensional projection data a1, a2, A3, a4, and a5 … … are converted into two-dimensional projection data a independent layer by layer, and then two-dimensional filtered back projection operation is performed on each layer of the two-dimensional projection data a, so that a reconstructed image is obtained layer by layer, and a three-dimensional volume image is formed.
Fig. 4 illustrates an example of two-dimensional projection data obtained according to an embodiment of the present disclosure. The direction of the horizontal axis of the sinogram as shown in fig. 4 represents the detector pixel sequence (e.g., from 1 to 256) while the vertical axis represents the angle (e.g., from 1 to 360 degrees). The processor 250 in the control device executes the reconstruction program, performs back projection operation on the two-dimensional projection data to obtain a reconstructed image, and further processes the reconstructed image to obtain a final image. For example, the reconstructed image is processed by using a trained convolutional neural network, for example, the data of the image domain is subjected to denoising and artifact removing processing, and a result image with higher quality is obtained.
In the embodiment of the disclosure, the three-dimensional projection data is processed by using the trained convolutional neural network in the projection domain to obtain two-dimensional projection data independent layer by layer, and then two-dimensional convolutional back projection operation is performed according to the layer to reconstruct a CT image of the corresponding layer. The convolutional neural network may include convolutional layers, pooling, and fully-connected layers. The convolutional layers identify the characterization of the input data set, with each convolutional layer carrying a nonlinear activation function operation. The pooling layer refines the representation of the features and typical operations include averaging pooling and maximizing pooling. One or more layers of full connection layers realize high-order signal nonlinear comprehensive operation, and the full connection layers also carry nonlinear activation functions. Common nonlinear activation functions are Sigmoid, Tanh, ReLU, etc.
Fig. 5 shows an overall structural schematic diagram of a convolutional neural network according to one embodiment of the present disclosure. As shown in fig. 5, a convolutional neural network according to an embodiment of the present disclosure includes a projection domain sub-network 510, a domain transformation sub-network 520, and an image domain sub-network 530. The three sub-networks respectively solve the conversion of a projection domain from three-dimensional space projection to two-dimensional plane projection, the domain conversion of projection to image and the optimization of image domain, and the three networks respectively have the realized functions and are mutually associated through an integral objective function.
As shown in fig. 5, the projection domain subnetwork 510 may process the three-dimensional projection data to obtain two-dimensional projection data, wherein the first convolutional neural network includes a plurality of convolutional layers. The three-dimensional projection data are obtained by performing a three-dimensional CT scan of the object under examination.
For example, the projection domain sub-network 510 outputs two-dimensional fan-beam (or parallel-beam) projection data of an object in layers by the action of several layers of convolutional neural networks with three-dimensional projection data of the object under examination as input. The partial network aims to extract the characteristics of original cone beam CT projection data through convolution kernels so as to estimate fan beam (or parallel beam) projections which are independent from each other according to different sections, and mainly completes the simplification of the high complexity problem of three-dimensional projection into two-dimensional in-plane projection, thereby simplifying the subsequent reconstruction problem. The resources and computational load required for three-dimensional reconstruction are much greater than for reconstruction in a layered two-dimensional plane.
According to an embodiment of the present disclosure, the domain conversion subnetwork 520 is utilized to perform a projection domain to image domain conversion operation of two-dimensional analytical reconstruction of the two-dimensional projection data. The network of domain switching sub-networks 520 comprises a two-dimensional backprojection or weighted backprojection operator layer, the weight coefficients between the nodes of which are determined by the geometric parameters of a fan-beam circular scanning system or a parallel-beam circular scanning system. The network of domain switching sub-network 520 may include a filter layer implementing ramp filtering or hilbert filtering. The network of domain switching sub-network 520 may include a differential operation layer to implement differential operation of the projection domain data. In addition, domain conversion subnetwork 520 can include a resampling layer to enable data resampling for the image domain.
The domain switching subnetwork 520 performs a back projection operation on the two-dimensional projection data in layers (slices) to obtain a reconstructed image. In this embodiment, a domain transformation subnetwork 520, i.e., a neural network layer, is constructed according to a two-dimensional filtered back-projection algorithm, and is cascaded with a projection domain subnetwork 510 (a first convolution neural network) to form an integrated reconstruction network, and the integrated reconstruction network is trained to obtain a three-dimensional CT reconstruction network. For example, the domain conversion sub-network 520 may implement operations from CT two-dimensional fan-beam (or parallel-beam) projection domain data to image domain slices, and the weighting coefficients between the network nodes may be determined by the scan geometry in the two-dimensional fan-beam (or parallel-beam) CT scan relationship. The input to this layer is two-dimensional projection data (e.g., fan beam projection data) output by the projection domain subnetwork, which is output as a preliminary CT reconstructed image. Since the projection domain subnetwork 510 has transformed the reconstruction problem into two dimensions, this part of the domain conversion network 520 can be done directly using operators that resolve the two-dimensional reconstruction.
In addition, a neural network layer can be constructed according to a two-dimensional filtering back projection algorithm and is cascaded with the first convolution neural network, an image processing network, namely a second convolution neural network, is further cascaded on the basis of the neural network layer, a comprehensive fine reconstruction network is formed, and the three-dimensional CT reconstruction network is obtained through training in an integral mode.
For example, the image domain subnetwork 530 can process the reconstructed image to obtain a resultant image. For example, the network of this part takes the preliminary CT reconstructed image output by the domain conversion sub-network 520 as input, and through the action of several convolutional neural networks, acquires the features of the data in the image domain, and takes the target image as the learning target, and performs suppression, amplification and mutual coupling on the image features, thereby achieving the effect of optimizing the image quality in the image domain. Those skilled in the art will appreciate that this portion of the network is not necessary.
According to embodiments of the present disclosure, a cost function of the overall network may be defined, for example, but not limited to, the l-norm | | | f-f commonly used in the art may be used*||lAnd the like. Wherein f ═ { f1，f2，…，fnIs the output image, the target image is f*。
According to the embodiment of the disclosure, a basic mathematical model of the scanned object can be established, and CT simulation data can be generated according to actual system modeling. Then, using CT simulation data of a plurality of scanned objects as network input, using real image values of the scanned objects as labels, training network parameters. The object is scanned on the actual system to obtain CT scan data, which is input to the network to obtain the preliminary reconstruction. These preliminary reconstruction results may be subjected to targeted image processing, such as denoising and artifact suppression and optionally further detail restoration. And using the processed image as a mark to further train the network so as to achieve the fine adjustment of network parameters.
For example, the projection domain sub-network 510 and the image domain sub-network 530 are partially trained using a residual method, a layer-by-layer image preliminary estimation of a three-dimensional body is formed on three-dimensional projection data using an approximate reconstruction method, two-dimensional projection is performed on the three-dimensional projection data, the preliminary estimation is used as a basis for residual operation by the projection domain sub-network 510, and the layer-by-layer image preliminary estimation of the three-dimensional body is used as a basis for residual operation by the image domain sub-network 530.
Taking cone beam circular orbit CT imaging as an example, the projection data obtained by CT three-dimensional scanning is recorded as
P is a matrix of C x V rows and R columns, where C denotes the number of detector columns, R denotes the number of detector rows, and V denotes the number of projections acquired by the detector. I.e. the three-dimensional projection data is organized in a matrix form. For a slice of the image to be reconstructed whose projections on the detector correspond to row R data, the portions of the projection data for low dose (first dose) and normal dose (second dose) are recorded as PL and PN,
the projection data of the line attenuation coefficient distribution of the imaged object under the fan beam projection condition is p,
before training the network, the layer of image corresponding to the input data can be reconstructed by an analytic reconstruction method such as an FDK (fully-drawn reconstruction) method
The system matrix for fan-beam scanning is denoted by H, so
As shown in FIG. 6, a U-net type neural network structure is used as the projection domain sub-network 510, and the partial network is used at a low dosePL is used as input and this partial network functions as a fan beam projection p to estimate the linear attenuation coefficient in a certain two-dimensional cross section. This part of the network consists of a number of convolutional layers, which are configured with 2-dimensional convolution kernels of K scales. For a certain scale, the 2-dimensional convolution kernel has two dimensions, where the first dimension is defined as the detector direction and the second dimension is defined as the scanning angle direction. The convolution kernel lengths in the two dimensions need not be the same, for example, taking convolution kernels of 3 × 1, 3 × 5, 7 × 3, as shown in fig. 7A, 7B and 7C, respectively. Multiple convolution kernels may be set for each scale. The convolution layer carries an activation function. The number of convolution kernels under each convolution scale is set as: { k } is a function of1，k2，…，kKThis layer is formed with K ═ K }1+k2+…+kKA sinogram. All convolution kernels are the network parameters to be determined. In the pooling part of the network, the convolution layers are pooled, the image scale is reduced layer by layer, and in the up-sampling part, the convolution layers are up-sampled, and the image scale is restored layer by layer. In order to keep the image resolution, the images with the same scale of the pooling part and the up-sampling part are spliced. By phiP-net(P) represents the operator corresponding to the projection domain sub-network, and the output result of the last layer of convolution layer uses a residual error mode:
domain conversion subnetwork 520 performs projection domain to image domain conversion operator operations on the two-dimensional image equivalent to two-dimensional analytical reconstruction, for example by performing a pair of calculations in the same manner as for fan-beam CT analytical reconstruction
And performing reverse calculation from the projection domain to the image domain to obtain image domain output. And calculating a projection matrix by using the existing Siddon or other methods in the field, and analyzing and reconstructing the connection weight of the connection layer according to the element correspondence of the system matrix. Taking FBP fan-beam analytical reconstruction as an example,
where w performs the weighting of the projection domain data and F corresponds to a ramp filterThe wave convolution operation is carried out on the wave,
a weighted back projection is completed.
The image domain network 530 employs a U-net type neural network structure similar to the first part of the network
As an input, the role is to achieve image domain optimization. Similar to the first part of the network, in the first half, pooling is performed between the convolutional layers, the image scale is reduced layer by layer, and in the second half, the image scale is restored layer by upsampling between the convolutional layers. The partial network still adopts a residual error training mode, namely the output result of the last convolutional layer is added with the residual error training mode
Equal to the estimation of a two-dimensional reconstructed image
Defining the cost function as a2 norm:
In view of the fact that no truth can be obtained in practical applications, the FDK reconstruction result of the complete data of the normal dose can be used as a label, namely
Other labels may be used if other ways are available to obtain high quality images.
In this embodiment, 3 x 3 convolution kernels may be selected for both the projection domain sub-network 510 and the image domain sub-network 520, 7 convolution layers may be provided, and a sampling interval of 4 x 4 may be used for both pooling and upsampling. ReLu was chosen as the activation function. 100 simulation data may be used for training.
For example, the reconstructed image is processed by using a U-shaped convolutional neural network, so that feature maps of different scales can be obtained, and the feature maps of different scales are combined to obtain a result image. More specifically, feature maps under multiple scales are fused step by utilizing an upsampling operation, and a result image of the inspected object is finally obtained. For example, the image domain network 530 further applies a priori knowledge for artifact removal based on estimating the projection-resolved reconstruction. The image domain network 530 in this example is a U-network design, and the 200 x 200 reconstructed images undergo 4 pooling, gradually reducing the feature map size and thus increasing the global features of the domain-learning image. And then gradually expanding and combining with the feature map with the same size without down sampling, so as to prevent information loss caused by down sampling, and finally restoring the 200 x 200 size again and finally reconstructing an image after network processing. The sizes of convolution kernels of image domains at different levels are all 3 x 3, the number of feature maps is gradually increased along with the reduction of the size of the feature maps in the down-sampling process of the images, and the number of the feature maps is gradually reduced in the up-sampling process.
Although fig. 6 illustrates a U-type network, those skilled in the art will appreciate that other forms of networks may be used to implement the disclosed embodiments.
According to the embodiment of the present disclosure, all convolution kernels of the projection domain network 510 and the image domain network 530 are network parameters to be determined, and may be initialized randomly, or may be updated in the network training process using the pre-training results of other approaches. In addition, the network processes the input data in the projection domain and the projection domain respectively, so that the objective function to be optimized (often called as a loss function in the deep learning field) achieves the optimal result. The projection domain convolution layer and the image domain convolution layer can play a complementary role due to different geometrical relations of adjacent pixels in the projection domain and the image domain.
Fig. 8 is a schematic flow chart diagram depicting a method according to an embodiment of the present disclosure. As shown in fig. 8, in step S810, a three-dimensional CT scan, such as a circular cone beam scan/helical scan/linear trajectory scan, is performed on the object to be inspected to obtain three-dimensional projection data. The CT scan herein may be unipotent or multipotent, and embodiments of the present disclosure are not limited thereto.
In step S820, the three-dimensional projection data is processed in the projection domain by using a first convolutional neural network (e.g., a U-type network shown in fig. 6), and two-dimensional projection data independent for each layer is obtained for the three-dimensional volume. For example, the trained convolutional neural network is used to process the three-dimensional projection data obtained from the cone beam scan, so as to obtain a sinogram under the processed fan beam scan, as shown in fig. 5. The two-dimensional projection data is then optionally filtered using a ramp filter. The projection data is filtered, for example, using an RL ramp filter. Those skilled in the art will appreciate that other filters may be used herein, or no filtering may be performed.
In step S830, a projection domain to image domain conversion operator equivalent to two-dimensional analysis reconstruction is performed on the two-dimensional projection data of each layer, and a reconstructed CT image is obtained by layers to form a three-dimensional volume image.
As described above, as another embodiment, the CT image may be obtained and then post-processed, for example, the reconstructed CT image is processed by using a second convolutional neural network (e.g., a U-type network shown in fig. 6) to obtain a result image. For example, local de-noising and artifact suppression, further detail restoration processing, or other image processing operations such as segmentation, edge enhancement, and equalization are performed on the reconstructed image.
Although the above description has been described primarily in relation to circumferential cone-beam scanning, it will be appreciated by those skilled in the art that the above described scheme may be used in the case of helical scanning, as shown in figure 9.
The method of the embodiment of the invention combines the advantages of deep learning and the particularity of CT imaging problems, designs a specific network architecture, combines simulation and actual data, trains the network, thereby reliably, effectively and comprehensively covering all system information and the set information of the imaged object, accurately reconstructing object images, and inhibiting noise influence caused by low-dose CT and artifacts and numerical deviation caused by cone beams. Although the training process needs a large amount of data and a plurality of iterative operations, the actual reconstruction process does not need iteration, and the calculation amount required by reconstruction is far faster than that of an iterative reconstruction algorithm compared with an analytic reconstruction method. The method can be flexibly applied to different CT scanning modes and system architectures, and can be applied to the fields of medical diagnosis, industrial nondestructive testing and security inspection.
The foregoing detailed description has set forth numerous embodiments of methods and apparatus for reconstructing CT images using schematics, flowcharts, and/or examples. Where such diagrams, flowcharts, and/or examples contain one or more functions and/or operations, it will be understood by those within the art that each function and/or operation within such diagrams, flowcharts, or examples can be implemented, individually and/or collectively, by a wide range of structures, hardware, software, firmware, or virtually any combination thereof. In one embodiment, portions of the subject matter described by embodiments of the invention may be implemented by Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), Digital Signal Processors (DSPs), or other integrated formats. However, those skilled in the art will recognize that some aspects of the embodiments disclosed herein, in whole or in part, can be equivalently implemented in integrated circuits, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as virtually any combination thereof, and that designing the circuitry and/or writing the code for the software and or firmware would be well within the skill of one of skill in the art in light of this disclosure. In addition, those skilled in the art will appreciate that the mechanisms of the subject matter described herein are capable of being distributed as a program product in a variety of forms, and that an illustrative embodiment of the subject matter described herein applies regardless of the particular type of signal bearing media used to actually carry out the distribution. Examples of signal bearing media include, but are not limited to: recordable type media such as floppy disks, hard disk drives, Compact Disks (CDs), Digital Versatile Disks (DVDs), digital tape, computer memory, etc.; and a transmission type medium such as a digital and/or an analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communications link, a wireless communication link, etc.).
While the present invention has been described with reference to several exemplary embodiments, it is understood that the terminology used is intended to be in the nature of words of description and illustration, rather than of limitation. As the present invention may be embodied in several forms without departing from the spirit or essential characteristics thereof, it should also be understood that the above-described embodiments are not limited by any of the details of the foregoing description, but rather should be construed broadly within its spirit and scope as defined in the appended claims, and therefore all changes and modifications that fall within the meets and bounds of the claims, or equivalences of such meets and bounds are therefore intended to be embraced by the appended claims.
Claims (14)
1. A three-dimensional CT image reconstruction method based on a neural network comprises the following steps:
carrying out three-dimensional CT scanning on an object to be detected to obtain three-dimensional projection data;
processing the three-dimensional projection data by using a first convolution neural network to obtain independent two-dimensional projection data of each layer of the three-dimensional volume, wherein the first convolution neural network comprises a plurality of convolution layers; and
performing projection domain-to-image domain conversion operator operation equivalent to two-dimensional analysis reconstruction on the two-dimensional projection data of each layer, and obtaining reconstructed images according to layers to form a three-dimensional image;
a neural network layer is constructed according to a two-dimensional analytical reconstruction algorithm, the neural network layer is cascaded with a first convolution neural network, an image processing network, namely a second convolution neural network, is further cascaded on the basis of the neural network layer, a comprehensive fine reconstruction network is formed, and a three-dimensional CT reconstruction network is obtained through integral training.
2. The method of claim 1, wherein the three-dimensional CT scan is one of the following: circular cone beam scanning, helical scanning, linear trajectory scanning.
3. The method of claim 1, wherein the first convolutional neural network and the second convolutional neural network are both convolutional neural networks.
4. The method of claim 1, further comprising the steps of:
the two-dimensional projection data is filtered using a ramp filter prior to performing a backprojection operation on the two-dimensional projection data.
5. The method of claim 1, wherein the reconstructed image is denoised and artifact suppressed using a second convolutional neural network, and further detail restored, resulting in a resulting image.
6. The method of claim 1, wherein one dimension of the convolution kernel of the convolutional layer in the first convolutional neural network is a detector pixel sequence and the other dimension is a scan angle, and a scale of the convolution kernel of the convolutional layer in the first convolutional neural network in the detector pixel sequence dimension and a scale in the scan angle dimension are set independently.
7. The method of claim 6, wherein a convolution kernel of a convolution layer in the first convolutional neural network has a larger scale in a detector pixel sequence dimension than in a scan angle dimension.
8. The method of claim 1, wherein the first convolutional neural network comprises at least 3 convolutional layers, each convolutional layer having an activation function for performing a non-linear operation on the convolved projection data.
9. The method of claim 1, wherein a domain-to-image domain conversion operation for two-dimensional analytical reconstruction of two-dimensional projection data using a domain conversion subnetwork whose network comprises a two-dimensional backprojection or weighted backprojection operation layer whose weighting coefficients between nodes are determined by geometric parameters of a fan-beam circular scanning system or a parallel-beam circular scanning system; the network of the domain conversion sub-network comprises a filter layer to realize ramp filtering or Hilbert filtering; the network of the domain conversion sub-network can comprise a differential operation layer to realize the differential operation of the projection domain data; the domain conversion subnetwork may contain a resampling layer to enable data resampling for the image domain.
10. The method as claimed in claim 1, wherein the first convolutional neural network and the second neural network are trained partially in a residual manner, the layer-by-layer image preliminary estimation of the three-dimensional volume is formed by using an approximate reconstruction method on the three-dimensional projection data, the two-dimensional projection is performed on the three-dimensional projection data, and the layer-by-layer image preliminary estimation of the three-dimensional volume is used as the basis of the residual operation by the second convolutional neural network.
11. A three-dimensional CT image reconstruction device based on a neural network comprises:
the CT scanning device is used for carrying out three-dimensional CT scanning on the object to be detected to obtain three-dimensional projection data;
a processor configured to:
processing the three-dimensional projection data by using a first convolution neural network to obtain independent two-dimensional projection data of each layer of the three-dimensional volume, wherein the first convolution neural network comprises a plurality of convolution layers; and
performing projection domain-to-image domain conversion operator operation equivalent to two-dimensional analysis reconstruction on the two-dimensional projection data of each layer, and obtaining reconstructed images according to layers to form a three-dimensional image;
the processor constructs a neural network layer according to a two-dimensional analytical reconstruction algorithm, the neural network layer is cascaded with the first convolutional neural network, an image processing network, namely a second convolutional neural network, is further cascaded on the basis of the neural network layer to form a comprehensive fine reconstruction network, and the three-dimensional CT reconstruction network is obtained through training in an integral mode.
12. The apparatus of claim 11, wherein the three-dimensional CT scan is one of: circular cone beam scanning, helical scanning, linear trajectory scanning.
13. The device of claim 11, wherein the processor is further configured to:
and training the first convolution neural network and the second neural network in a residual error mode, forming layer-by-layer image preliminary estimation of a three-dimensional body by using an approximate reconstruction method on three-dimensional projection data, performing two-dimensional projection on the three-dimensional projection data to serve as a basis for residual error operation used by the first convolution neural network, and using the layer-by-layer image preliminary estimation of the three-dimensional body as a basis for residual error operation by the second convolution neural network.
14. A computer-readable medium storing a computer program which, when executed by a processor, performs the steps of:
processing the three-dimensional projection data by using a first convolution neural network to obtain independent two-dimensional projection data of each layer of the three-dimensional volume, wherein the first convolution neural network comprises a plurality of convolution layers; and
performing projection domain-to-image domain conversion operator operation equivalent to two-dimensional analysis reconstruction on the two-dimensional projection data of each layer, and obtaining reconstructed images according to layers to form a three-dimensional image;
a neural network layer is constructed according to a two-dimensional analytical reconstruction algorithm, the neural network layer is cascaded with a first convolution neural network, an image processing network, namely a second convolution neural network, is further cascaded on the basis of the neural network layer, a comprehensive fine reconstruction network is formed, and a three-dimensional CT reconstruction network is obtained through integral training.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810721039.0A CN110660123B (en) | 2018-06-29 | 2018-06-29 | Three-dimensional CT image reconstruction method and device based on neural network and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810721039.0A CN110660123B (en) | 2018-06-29 | 2018-06-29 | Three-dimensional CT image reconstruction method and device based on neural network and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110660123A CN110660123A (en) | 2020-01-07 |
| CN110660123B true CN110660123B (en) | 2021-08-31 |
Family
ID=69028519
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810721039.0A Active CN110660123B (en) | 2018-06-29 | 2018-06-29 | Three-dimensional CT image reconstruction method and device based on neural network and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110660123B (en) |
Families Citing this family (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111583422B (en) * | 2020-04-17 | 2023-03-28 | 清华大学 | Heuristic editing method and device for three-dimensional human body model |
| CN113554742B (en) * | 2020-04-26 | 2024-02-02 | 上海联影医疗科技股份有限公司 | Three-dimensional image reconstruction method, device, equipment and storage medium |
| CN111583389B (en) * | 2020-04-28 | 2023-05-23 | 重庆大学 | Incomplete scanning CT image reconstruction method based on CAD model |
| CN112368738B (en) * | 2020-05-18 | 2024-01-16 | 上海联影医疗科技股份有限公司 | System and method for image optimization |
| WO2022032445A1 (en) * | 2020-08-10 | 2022-02-17 | 深圳高性能医疗器械国家研究院有限公司 | Reconstructed neural network and application thereof |
| CN111968167B (en) * | 2020-09-02 | 2023-12-26 | 广州海兆印丰信息科技有限公司 | Image processing method and device for CT three-dimensional positioning image and computer equipment |
| CN112308764A (en) * | 2020-10-12 | 2021-02-02 | 杭州三坛医疗科技有限公司 | Image registration method and device |
| CN112435341B (en) * | 2020-11-23 | 2022-08-19 | 推想医疗科技股份有限公司 | Training method and device for three-dimensional reconstruction network, and three-dimensional reconstruction method and device |
| CN112381741B (en) * | 2020-11-24 | 2021-07-16 | 佛山读图科技有限公司 | Tomography image reconstruction method based on SPECT data sampling and noise characteristics |
| CN112348936B (en) * | 2020-11-30 | 2023-04-18 | 华中科技大学 | Low-dose cone-beam CT image reconstruction method based on deep learning |
| CN112509091B (en) * | 2020-12-10 | 2023-11-14 | 上海联影医疗科技股份有限公司 | Medical image reconstruction method, device, equipment and medium |
| US20220189011A1 (en) * | 2020-12-16 | 2022-06-16 | Nvidia Corporation | End-to-end training for a three-dimensional tomography reconstruction pipeline |
| US11890124B2 (en) | 2021-02-01 | 2024-02-06 | Medtronic Navigation, Inc. | Systems and methods for low-dose AI-based imaging |
| CN112529821B (en) * | 2021-02-07 | 2021-05-14 | 南京景三医疗科技有限公司 | Method for removing CPR image artifact |
| CN113392955A (en) * | 2021-05-11 | 2021-09-14 | 南方医科大学 | CT reconstruction neural network structure and method based on downsampling imaging geometric modeling |
| CN113223112B (en) * | 2021-06-22 | 2022-07-15 | 清华大学 | Generalized equiangular detector CT image analysis reconstruction method |
| CN115409835B (en) * | 2022-10-31 | 2023-02-17 | 成都浩目科技有限公司 | Three-dimensional imaging method, device, electronic equipment, system and readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101138506A (en) * | 2007-10-11 | 2008-03-12 | 上海交通大学 | Conical bundle CT reestablishment method based on helix saddle line |
| CN102456227A (en) * | 2010-10-28 | 2012-05-16 | 清华大学 | Reconstruction method and device for CT (computerized tomography) image |
| CN103961122A (en) * | 2013-01-31 | 2014-08-06 | 通用电气公司 | Non-equalGamma angle CT system data conversion method and device |
| CN106127686A (en) * | 2016-06-29 | 2016-11-16 | 西安电子科技大学 | The method improving CT reconstructed image resolution based on sinusoidal area image super-resolution |
| CN107192726A (en) * | 2017-05-05 | 2017-09-22 | 北京航空航天大学 | The quick high-resolution 3 D cone-beam computer tomography method of plate shell object and device |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105447866A (en) * | 2015-11-22 | 2016-03-30 | 南方医科大学 | X-ray chest radiograph bone marrow suppression processing method based on convolution neural network |
| CN106056059B (en) * | 2016-05-20 | 2019-02-12 | 合肥工业大学 | The face identification method of multi-direction SLGS feature description and performance cloud Weighted Fusion |
| US10417788B2 (en) * | 2016-09-21 | 2019-09-17 | Realize, Inc. | Anomaly detection in volumetric medical images using sequential convolutional and recurrent neural networks |
-
2018
- 2018-06-29 CN CN201810721039.0A patent/CN110660123B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101138506A (en) * | 2007-10-11 | 2008-03-12 | 上海交通大学 | Conical bundle CT reestablishment method based on helix saddle line |
| CN102456227A (en) * | 2010-10-28 | 2012-05-16 | 清华大学 | Reconstruction method and device for CT (computerized tomography) image |
| CN103961122A (en) * | 2013-01-31 | 2014-08-06 | 通用电气公司 | Non-equalGamma angle CT system data conversion method and device |
| CN106127686A (en) * | 2016-06-29 | 2016-11-16 | 西安电子科技大学 | The method improving CT reconstructed image resolution based on sinusoidal area image super-resolution |
| CN107192726A (en) * | 2017-05-05 | 2017-09-22 | 北京航空航天大学 | The quick high-resolution 3 D cone-beam computer tomography method of plate shell object and device |
Non-Patent Citations (5)
| Title |
|---|
| A Cascaded Convolutional Nerual Network for X-ray Low-dose CT Image Denoising;Dufan Wu et al.;《arXiv》;20170828;1-9 * |
| Few-View CT Reconstruction Method Based on Deep Learning;Ji Zhao et al;《IEEE》;20171019;1-4 * |
| 基于卷积神经网络的低剂量CT图像处理方法;张慧娟;《中国优秀硕士学位论文全文数据库信息科技辑（月刊）》;20180415(第04期);摘要、第1.3、2.1、3.1、3.3、3.4、3.5、4.1节 * |
| 基于平板成像器件的三维CT重建算法研究;蔡文涛;《中国优秀硕士学位论文全文数据库信息科技辑（月刊）》;20131015(第10期);I138-365 * |
| 螺旋锥束CT三维图像重建算法的研究;邹云鹏;《中国优秀硕士学位论文全文数据库信息科技辑（月刊）》;20130415(第04期);第1.3、5.1、5.2、5.3节 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110660123A (en) | 2020-01-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110660123B (en) | Three-dimensional CT image reconstruction method and device based on neural network and storage medium | |
| CN110047113B (en) | Neural network training method and apparatus, image processing method and apparatus, and storage medium | |
| CN109300166B (en) | Method and apparatus for reconstructing CT image and storage medium | |
| CN109300167B (en) | Method and apparatus for reconstructing CT image and storage medium | |
| JP7187476B2 (en) | Tomographic reconstruction based on deep learning | |
| EP3608877B1 (en) | Iterative image reconstruction framework | |
| CN110544282B (en) | Three-dimensional multi-energy spectrum CT reconstruction method and equipment based on neural network and storage medium | |
| Dong et al. | A deep learning reconstruction framework for X-ray computed tomography with incomplete data | |
| CN109509235B (en) | Reconstruction method, device and equipment of CT image and storage medium | |
| CN112085829A (en) | Spiral CT image reconstruction method and equipment based on neural network and storage medium | |
| US7916828B1 (en) | Method for image construction | |
| KR20190135618A (en) | Method for processing interior computed tomography image using artificial neural network and apparatus therefor | |
| CN110751701B (en) | X-ray absorption contrast computed tomography incomplete data reconstruction method based on deep learning | |
| Kazantsev et al. | A novel tomographic reconstruction method based on the robust Student's t function for suppressing data outliers | |
| Wang et al. | Sparse-view cone-beam CT reconstruction by bar-by-bar neural FDK algorithm | |
| Langet et al. | Compressed‐sensing‐based content‐driven hierarchical reconstruction: Theory and application to C‐arm cone‐beam tomography | |
| Sunnegårdh | Iterative filtered backprojection methods for helical cone-beam CT | |
| Dong et al. | Tomographic reconstruction with spatially varying parameter selection | |
| KR102329938B1 (en) | Method for processing conebeam computed tomography image using artificial neural network and apparatus therefor | |
| Abbas et al. | PET image reconstruction and denoising on hexagonal lattices | |
| Hamoud et al. | Beyond Local Processing: Adapting CNNs for CT Reconstruction | |
| Yang et al. | Complex artefact suppression for sparse reconstruction based on compensation approach in X‐ray computed tomography | |
| Freitas et al. | The formation of computed tomography images from compressed sampled one-dimensional reconstructions | |
| Yang et al. | High quality x-ray imaging based on cumulation of forward projection of computed tomography | |
| Ni | Reduce Penumbra for 3D X-Ray Reconstruction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |