CN110211193B - Three-dimensional CT (computed tomography) interlayer image interpolation restoration and super-resolution processing method and device - Google Patents
Three-dimensional CT (computed tomography) interlayer image interpolation restoration and super-resolution processing method and device Download PDFInfo
- Publication number
- CN110211193B CN110211193B CN201910415030.1A CN201910415030A CN110211193B CN 110211193 B CN110211193 B CN 110211193B CN 201910415030 A CN201910415030 A CN 201910415030A CN 110211193 B CN110211193 B CN 110211193B
- Authority
- CN
- China
- Prior art keywords
- image
- interpolation
- optical flow
- interlayer
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 239000011229 interlayer Substances 0.000 title claims abstract description 56
- 238000003672 processing method Methods 0.000 title claims abstract description 21
- 238000002591 computed tomography Methods 0.000 title abstract description 84
- 230000003287 optical effect Effects 0.000 claims abstract description 67
- 238000000034 method Methods 0.000 claims description 52
- 238000004422 calculation algorithm Methods 0.000 claims description 37
- 239000010410 layer Substances 0.000 claims description 25
- 239000011159 matrix material Substances 0.000 claims description 16
- 230000008569 process Effects 0.000 claims description 15
- 230000006870 function Effects 0.000 claims description 12
- 238000009499 grossing Methods 0.000 claims description 12
- 238000006073 displacement reaction Methods 0.000 claims description 9
- 238000013507 mapping Methods 0.000 claims description 8
- 239000013598 vector Substances 0.000 claims description 8
- 230000002146 bilateral effect Effects 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 6
- 238000000354 decomposition reaction Methods 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 6
- 238000012545 processing Methods 0.000 claims description 6
- 230000000007 visual effect Effects 0.000 claims description 6
- 238000009792 diffusion process Methods 0.000 claims description 4
- 239000000654 additive Substances 0.000 claims description 2
- 230000000996 additive effect Effects 0.000 claims description 2
- 125000004432 carbon atom Chemical group C* 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 claims description 2
- 238000005429 filling process Methods 0.000 claims description 2
- 230000004907 flux Effects 0.000 claims description 2
- 238000003780 insertion Methods 0.000 claims description 2
- 230000037431 insertion Effects 0.000 claims description 2
- 238000005259 measurement Methods 0.000 claims description 2
- 230000035945 sensitivity Effects 0.000 claims description 2
- 238000013519 translation Methods 0.000 claims description 2
- 239000004576 sand Substances 0.000 claims 1
- 238000003745 diagnosis Methods 0.000 abstract description 6
- 230000008439 repair process Effects 0.000 abstract description 5
- 230000005855 radiation Effects 0.000 abstract description 4
- 230000000694 effects Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 7
- 230000008859 change Effects 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 5
- 210000004072 lung Anatomy 0.000 description 4
- 238000011158 quantitative evaluation Methods 0.000 description 3
- 239000000523 sample Substances 0.000 description 3
- 206010028980 Neoplasm Diseases 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 201000010099 disease Diseases 0.000 description 2
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 201000011510 cancer Diseases 0.000 description 1
- 208000015114 central nervous system disease Diseases 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 210000000987 immune system Anatomy 0.000 description 1
- 230000001939 inductive effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 210000000056 organ Anatomy 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/003—Reconstruction from projections, e.g. tomography
- G06T11/008—Specific post-processing after tomographic reconstruction, e.g. voxelisation, metal artifact correction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10081—Computed x-ray tomography [CT]
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Apparatus For Radiation Diagnosis (AREA)
- Image Processing (AREA)
Abstract
The invention provides a three-dimensional CT (computed tomography) interlayer image interpolation restoration and super-resolution processing method and device, which are used for performing interlayer interpolation on a CT image by utilizing registration information based on optical flow estimation. Therefore, the invention further repairs the interlayer slice generated by interpolation and improves the quality of the interpolated image. On the basis of interlayer interpolation based on optical flow estimation, the intermediate slices generated by interpolation carry out image restoration by using the non-local self-similarity of the image, the number of CT slices can be increased, and the interlayer resolution of the CT image is improved, so that the quality of the MFSR reconstructed CT image is improved, and accurate diagnosis and treatment of a patient can be realized under the condition that the patient does not receive unnecessary radiation dose.
Description
Technical Field
The invention relates to the technical field of CT image processing, in particular to a three-dimensional CT interlayer image interpolation restoration and super-resolution processing method and device.
Background
Computed Tomography (CT) images are obtained by a probe performing continuous cross-sectional scanning around a certain part of the human body together with X-ray beams, ultrasonic waves, and the like. CT, a commonly used medical imaging method, has a very important role in diagnosis and treatment of central nervous system diseases, chest diseases, and the like in recent years because it can display details of a certain part in a human body, but the resolution of a CT image is closely related to the amount of X-ray dose. If the X-ray dose is reduced, serious artifacts are likely to appear, the focus part is difficult to clearly appear on a CT image, and the reliability of diagnosis is reduced; if the X-ray dose is increased, damage to the immune system may occur, with the potential risk of inducing cancer.
The method for reconstructing the super-resolution CT image is developed rapidly, and the method in recent years is mainly divided into two types, namely 1) single image super-resolution (SISR) which only refers to a current low-resolution image and does not depend on other related images, L edge et al uses a depth residual error network for generating the sense of reality of a resistance network to perform single image super-resolution, L im et al uses an enhanced depth residual error network for single image super-resolution, uses a deconvolution depth neural network to realize the super-resolution reconstruction of a medical image, and also operates on a single image, and 2) multi-image super-resolution (MFSR) replaces the single image with an image sequence adjacent to the same scene, and the series of low-resolution images are fused to generate a high-resolution image by referring to complementary information of the images.
In general, MFSR has more sufficient referenceable information than SISR, enabling higher quality high resolution reconstructed images to be obtained. However, to reduce the X-ray radiation to the patient and the limitations of existing devices, it is often impractical to perform dense sampling in the up and down directions, and usually only a limited number of CT slices can be acquired, resulting in a larger slice spacing of the CT image sequence. The lack of sufficient structural information of the CT images along the up and down directions may cause large differences in local structures between two adjacent CT images, so that in the MFSR reconstruction process, these differences may cause interference to the reconstructed high-resolution image, resulting in inaccurate diagnosis at the pre-tumor stage, thereby affecting the subsequent treatment of the patient.
Disclosure of Invention
The invention applies the frame interpolation idea in the video processing process to the super-resolution research of medical images. A new slice is inserted into the original two adjacent CT slices, so that the problem of large difference of the local structures of the two adjacent layers of CT images is solved.
The method comprises the following steps:
solving the corresponding relation of pixels between adjacent slices of a CT image through a C L G-TV optical flow estimation model;
finding a target position through a mapping function of the corresponding relation between pixels, and interpolating an intermediate slice in an image warping mode;
and thirdly, repairing fine holes and fuzziness generated in the interpolation process by utilizing the inherent interframe non-local self-similarity of the 4D-CT image.
The invention also provides a device for realizing the three-dimensional CT interlayer image interpolation restoration and super-resolution processing method, which comprises the following steps:
the memory is used for storing a computer program and a three-dimensional CT interlayer image interpolation repairing and super-resolution processing method;
and the processor is used for executing the computer program and the three-dimensional CT interlayer image interpolation repairing and super-resolution processing method so as to realize the steps of the three-dimensional CT interlayer image interpolation repairing and super-resolution processing method.
According to the technical scheme, the invention has the following advantages:
according to the method, the CT image is subjected to interlayer interpolation by utilizing the registration information based on the optical flow estimation, but in the interpolation process, due to different motion fields of the optical flow estimation, a plurality of pixels in the original image can be mapped to the same position in the interpolation image, so that an area without the original image pixel mapped exists in the interpolation image, namely, the pixels are lost, and thus, the interpolation image has tiny holes. Therefore, the invention further repairs the interlayer slice generated by interpolation and improves the quality of the interpolated image.
According to the method, on the basis of interlayer interpolation based on optical flow estimation, image restoration is performed on the intermediate slices generated by interpolation by using non-local self-similarity of images, the number of CT slices can be increased, and the interlayer resolution of the CT images is improved, so that the quality of the MFSR reconstructed CT images is improved, and accurate diagnosis and treatment of patients can be realized under the condition that the patients do not receive unnecessary radiation dose.
Drawings
In order to more clearly illustrate the technical solution of the present invention, the drawings used in the description will be briefly introduced, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained based on these drawings without creative efforts.
FIG. 1 is a schematic view of sagittal plane layer interpolation;
FIG. 2 is a schematic diagram of cross-sectional interlayer interpolation;
FIG. 3 is a flow chart of the framework of the present invention;
FIG. 4 is a schematic diagram of motion estimation based on optical flow field
FIG. 5 is a schematic diagram of optical flow estimation of three coronal images using the C L G-TV optical flow estimation algorithm;
FIG. 6 is a schematic diagram of a hole that may be created by interpolating between two successive CT images to create a new intermediate slice;
FIG. 7 is a schematic diagram of block-based non-local self-similarity;
FIG. 8 is a diagram showing the effect of image restoration according to the present invention, (a) is an original image and a partial enlarged view before restoration, and (b) is an original image and a partial enlarged view after restoration;
FIG. 9 is a graph showing interpolation results according to the present invention, wherein inter-layer slices are generated by interpolation in the frame, input continuous CT slices are outside the frame, and the arrow points to the continuously changing salient region;
FIG. 10 is a schematic diagram of comparison of effect and error images achieved by various algorithms;
fig. 11 is a flow chart of a three-dimensional CT interlayer image interpolation restoration and super-resolution processing method.
Detailed Description
The invention applies the frame interpolation idea in the video processing process to the super-resolution research of medical images. A new slice is inserted into the original two adjacent CT slices, so that the problem of large difference of the local structures of the two adjacent layers of CT images is solved. The effect on the sagittal plane of the lung is shown in fig. 1, where the horizontal solid line is the original CT slice obtained by the actual scan and the horizontal dotted line is the intermediate CT slice to be interpolated. The effect on the cross section is shown in fig. 2, (a) represents the input image sequence, and (b) is the intermediate slice sequence that needs to be interpolated.
According to the method, the CT image is subjected to interlayer interpolation by utilizing the registration information based on the optical flow estimation, but in the interpolation process, due to different motion fields of the optical flow estimation, a plurality of pixels in the original image can be mapped to the same position in the interpolation image, so that an area without the original image pixel mapped exists in the interpolation image, namely, the pixels are lost, and thus, the interpolation image has tiny holes. Therefore, the invention further repairs the interlayer slice generated by interpolation and improves the quality of the interpolated image. The method can repair the image while maintaining the image structure information, but in practical application, the method has difficulty in ensuring that the found sample block is optimal, and repair errors are easily caused. And synthesizing a filling block for filling the area to be repaired by using the weighted average of the plurality of sample blocks.
Therefore, the invention carries out image restoration on the intermediate slices generated by interpolation value by using the non-local self-similarity of the image on the basis of carrying out interlayer interpolation based on optical flow estimation, can increase the number of CT slices and improve the interlayer resolution of the CT image, thereby improving the quality of the MFSR reconstructed CT image and helping a patient to obtain accurate diagnosis and treatment under the condition of not receiving unnecessary radiation dose.
The invention provides a three-dimensional CT (computed tomography) interlayer image interpolation and restoration algorithm based on optical flow estimation. Comprises the following three steps: as shown in figure 11 of the drawings,
s1, solving the corresponding relation of the pixels between adjacent slices of the CT image through a C L G-TV optical flow estimation model;
s2, finding a target position through a mapping function of the corresponding relation between pixels, and interpolating an intermediate slice in an image warping mode;
s3, utilizing the inherent interframe non-local self-similarity of the 4D-CT image to carry out image restoration on the tiny holes and the blur generated in the interpolation process.
By the operation, high-quality interpolation CT slices can be quickly obtained, and the interlayer resolution of the CT image is improved. The whole framework of the algorithm is shown in FIG. 3, I1And I2Is two input continuous CT images, F is dense optical flow field, F is mapping function of corresponding relation between pixels, IipIs a synthetic inter-layer image, IsThe method is an original slice sequence set, and a similar block matrix is obtained through non-local self-similarity to fill a hole generated by pixel missing.
In S1, the change of the structure shape of human organs is fast, if the scanning time interval is long, the difference of different slices is large, so it is difficult to solve the problem of solving the large displacement optical flow field of CT image, in the optical flow calculation method, the data item in the global (HS) algorithm is the brightness conservation hypothesis to the single pixel point, the robustness is poor, but the dense optical flow field can be obtained, the data item in the local (L K) algorithm is the brightness conservation hypothesis to the pixel point in the small neighborhood near the single pixel point, the robustness to noise is good, but only the sparse optical flow field can be obtained.
The method for estimating the total-variation light splitting flow field based on the C L G algorithm is an optical flow calculation model which can solve the optical flow field with large displacement and has good robustness to noise, and the C L G-TV optical flow model is applied to the interlayer interpolation process of the CT image to solve the optical flow field with large displacement between adjacent slices of the CT imagex,vy) Is the optical flow at pixel point (x, y).
The C L G algorithm expands the single pixel point in the global algorithm into the neighborhood thereof by using the assumption of brightness conservation, and the smooth term is not changed, so that a more accurate and compact optical flow field is obtained.C L G optical flow estimation model is as follows:
ECLG＝∫Ω(W2(x,y)·(Ixvx+Iyvy+It)2+S)dxdy·(1)
wherein W (x, y) is a weight coefficient of each point in a neighborhood Ω with the point (x, y) as the center, Ix、Iy、ItIs partial derivative of gray level of pixel point along x, y and t directions
Is a smoothing term.
L in C L G optical flow estimation model2Problem of norm sensitivity to image noise, introducing L for optical flow1Norm constraint ensures that the estimation model has more robust anti-noise performance
Compared with the prior optical flow model, the method has the advantages that the influence of neighborhood pixels can be weakened, the image edge can be better kept, and the effect of edge preserving and denoising can be achieved by introducing a bilateral filtering technology and an anisotropic diffusion smoothing strategy, adding bilateral filtering constraint and anisotropic regularization to the data item enhances the specific filling process, and the C L G-TV variational optical flow estimation model with more accurate calculation result is obtained:
in the above formula, the first and second carbon atoms are,
for the data item, ω is a weight coefficient between the data item and the smoothing item, bfw represents smoothing filtering on the data item, and region represents the scope of the bilateral filter. Smoothing term EsmoothUsing image-driven anisotropic diffusion factor
Data item
Is defined as follows:
where X ═ X, y is the pixel point coordinate, and F ═ vx,vy) As the flow of light in two dimensions (x-and y-directions), F0Is an initial estimate of the optical flow field.
In the process of solving the optical flow model, a texture structure decomposition method and a coarse-to-fine Gaussian pyramid algorithm are adopted, so that the optical flow model can be prevented from being influenced by illumination change to the greatest extent, and the solution of the large-displacement optical flow field is realized. Performing texture decomposition on the image, and decomposing the input image I into a part I containing image texture information and a part containing image structure informationg,Ig＝I-IsDue to IgIs not affected by shadow and obstruction basically, will IgAs a new input image, initialize F0；
Secondly, determining the pyramid layer number according to the image size
Each layer of luminous flux has
Is the flow of light passing from the low-precision k layer to the k +1 layer,
is the incremental amount of optical flow passing from the k layer to the k +1 layer. And then rapidly and iteratively solving the optical flow by using an Alternating Direction Multiplier Method (ADMM) until k is equal to n-1. The optical flow model is rapidly solved, and the efficiency of the algorithm is effectively improved.
FIG. 5 shows an example of optical flow estimation for three coronal CT images using C L G-TV optical flow estimation algorithm, where a reference image and images a and b are all selected from the same CT image sequence, optical flow a is a local optical flow field between the reference image and image a, and optical flow b is a local optical flow field between the reference image and image b, image a and reference image are less displaced (position indicated by arrow), and image b and reference image are more displaced, which is consistent with the estimated optical flow field, indicating that C L G-TV can accurately estimate the large displacement optical flow field between adjacent slices of CT images.
In S2, based on the three-dimensional CT image interlayer interpolation process, the warping operation in the conventional method is introduced into the method, so that the problem of large offset can be effectively solved, and a reasonable interpolation result can be obtained. The image warping refers to calculating a mapping function between each pixel in the original image and a motion vector of the pixel, and realizing displacement of the pixel, so as to obtain a target image. The interpolation of the intermediate layer between two images is required to interpolate the generated motion, so that the motion object generates an interpolated image in a natural and appropriate mode along the motion track, and meanwhile, for nonlinear translation motion, warping can compensate some geometric distortion problems.
Calculating a mapping function q of a pixel through a C L G-TV optical flow modeli＝f(pi) Determining the pixel p in the original sliceiTarget position q ofiThen using a bilinear interpolation algorithm at qiA warping operation is performed. The target position q is obtained because the target position can have non-integer conditioniWith its diagonal neighborhood ND (q)i) And selecting the pixel point with the minimum Euclidean distance as the insertion point.
ρ＝(|x1-x2|2+|y1-y2|2)1/2(6)
In S3, depending on the motion field, a plurality of pixels of the original image may be mapped to the same position in the interpolated image. On the other hand, there may be a region in the interpolated image to which no original pixel is mapped, resulting in a fine hole in the interpolated image, as shown in fig. 6. I is1And I2Representing two successive CT images of the input, IipIs the output inter-layer image, and the box is a partial enlargement of the main change area.
In view of the self-similarity of medical images, missing structural details can be found on other existing slices, so that the middle slice is divided into a plurality of overlapped small blocks, for each image block needing repairing, similar blocks matched with the image block can be found on other slices by utilizing the information of the surrounding neighborhood, then an optimization problem is solved to obtain an optimal similar block matrix to fill the missing pixels, and finally an intermediate slice with rich details is reconstructed.
As in FIG. 7, the method slices the original image
N is the number of CT image sequences, N is the sequence IpThe total number of the original slices in the image and an intermediate slice I obtained based on an optical flow registration interpolation methodipDivided into M mutually overlapped image blocks according to step length d
Presentation slice
Of the ith vectorized image block, each block having a size of m × m
Image block of
Using block matching techniques and Euclidean distance based similarity metric criterion
Middle search
Like in (1), for ease of discussion, let x beiSet as the image block of the inter-layer slice obtained by interpolation,
are image blocks on other slices. The judgment basis of the similarity is as follows:
t is a threshold value. Selecting and calculating according to similarity measurement criteria
The most similar L image blocks are used as column vectors to construct a corresponding group matrix HjNamely:
Hjcan also represent the matrix PjAnd noise matrix NjAnd (3) the sum:
Hj＝Pj+Nj·(9)
Njcan be regarded asIs additive noise with standard deviation σ. Replacement of H by H and PjAnd Pj. Singular Value Decomposition (SVD) of H and P:
where U is the left singular vector, V is the right singular vector, Σ is a diagonal matrix of r × r, and the diagonal elements are the singular values of the matrix
If P is known, then assuming that the derivative of the above objective function to M is zero, the following result can be obtained
M＝(H*H)-1H*P. (13)
Then the minimum variance of P is estimated as
Because of the fact that
Diagonal element λ of1,λ2,……,λkAre all non-zero positive integers, therefore
Substituting (15) into (14) can solve the ideal matrix
Then using the determined momentsMatrix of
Processing tiny holes generated by interpolation, and the image block needing to be repaired can be represented by weighting the non-local similar block
xiIs the image block to be repaired,
is with xiThe most similar L similar pieces of data,
is a weighting coefficient
τ is a tuning parameter that controls the decay of the exponential function.
The CT interpolation image (the first row of fig. 8, the original image before restoration, and the partial enlarged image) can be effectively restored by the similar block matching technique, and a CT image with a good visual effect (the second row of fig. 8, the original image after restoration, and the partial enlarged image) is obtained. Since the image blocks overlap each other, it may result in a plurality of pixel estimation values for the area to be processed, where a weighted average method is applied to the estimation values that are different for each pixel.
By using the method related by the invention, through practical application, based on a public data set provided by DIR-lab of Anderson cancer center of Texas university, the interlayer distance of the data is 2.5mm, continuous lung CT images with the interlayer distance of 2.5mm are selected from the CT images, wherein the CT images comprise a coronal plane and a cross section, the image size of the lung cross section is 560 ×, and the image size of the coronal plane is 664 ×.
TABLE 1 Algorithm operating Environment
For quantitative evaluation, an Interpolation Error (IE) is defined, i.e. a real image (Ground Truth) IGTImage and estimated interpolated image IipThe Root Mean Square Error (RMSE),
where M × N is the image size and the root mean square error is defined in terms of Mean Square Error (MSE)
MSE is the distance deviation of the estimated value from the true value. Therefore, the root mean square error can be used to measure the magnitude of the error between the interpolated image and the real image. The smaller the value of RMSE is, the higher the accuracy of a new image generated by the experimental interpolation is, and the new image is closer to a real CT image.
The specific parameters are set to be that in the optical flow solving process, the weight coefficient omega between the data item and the smoothing item is set to be 50, the higher the value of omega is, the higher the confidence coefficient of the data item is, the scope region of the bilateral filter is set to be 7 × 7, the min width is set to be 20, the large-displacement optical flow estimation problem can be well solved, and the average optical flow calculation time is 4.052 s.
The image restoration process is mainly influenced by four key parameters, namely the size m of an image block, the column number L of a group matrix H, a standard deviation sigma and a step length d, wherein the size m of the image block has the deepest influence on the image restoration result, the smaller the size of the image block is, the more the local structure details can be obtained, but the selection of the size of the neighborhood around the missing pixel can be influenced, the estimation of the similar block is inaccurate, the performance and the effect of the algorithm are reduced, the image block with the larger size can obtain more image information for similarity matching, but higher calculation cost can be generated, based on the analysis, in order to balance the performance and the accuracy of the algorithm, the m is 16, the sigma is 40, the L is 60, and the d is 4, and the restored interpolation image with better quality can be obtained.
By the experimental mode and the three-dimensional CT interlayer image interpolation restoration method based on optical flow estimation, high-quality intermediate slices can be inserted between two continuous slices of the CT image. The interpolation result of the invention is shown in fig. 9, three groups of CT slices (a, b and c) of continuous fault examples with different sequences are selected, four input continuous CT slices are arranged outside each group of frames, three middle layer slices generated by interpolation of the method are arranged inside the frames, and the arrow points to the region with obvious change. The change of the outline of the CT image is natural and continuous, and the change of the arrow pointing area is continuous, so that the effect of smooth transition can be obtained.
In terms of visual evaluation, three CT slices (the first row in FIG. 10) with the pixel size of 560 × 420 are randomly selected, the middle slice is taken as the Ground Truth. FIG. 10 shows the results of interlayer interpolation of the upper slice a and the lower slice b in FIG. 10, which are achieved by using the TV-L1 optical flow algorithm, the path-based interpolation algorithm and the shape-based interpolation algorithm respectively.
Compared with the algorithm based on the path, the algorithm time complexity is reduced, and meanwhile, more accurate and clear effects can be obtained in the aspect of processing larger displacement, for better visual comparison, figure 10 also provides error image comparison graphs (the third line of figure 10) of the intermediate slices interpolated by the upper layer slice a and the lower layer slice b in figure 10 by four algorithms and a real interlayer image, so that the algorithm is closest to a real image, and the algorithm can accurately perform CT image interlayer interpolation.
In order to verify that the intermediate slice obtained by the algorithm interpolation is closer to a real image, the method introduces IE evaluation indexes to carry out quantitative evaluation on the quality of the reconstructed image. For the 4 algorithms, three groups of lung CT images are selected, each group has three continuous CT slices, the original middle layer slice is used as a real image, the upper layer slice and the lower layer slice are used as input, and the RMSE values of the middle slice and the real image generated by the four algorithm interpolation are respectively calculated, as shown in table 2.
TABLE 2 RMSE comparison of different Algorithm interlayer slices with the original CT slices
It can be seen from fig. 10 and table 2 that the algorithm of the present invention achieves better results than other algorithms in terms of both visual effect and specific index value, and particularly obtains clearer structural information in a region with larger variation. Therefore, the three-dimensional CT interlayer image generated by the algorithm interpolation is more accurate, the edge and local information is well reserved, and the interlayer interpolation effect is better.
The invention provides a three-dimensional CT (computed tomography) interlayer interpolation and restoration method based on optical flow estimation, which is characterized in that a new interlayer slice is interpolated on the upper layer and the lower layer of a CT image by utilizing registration information obtained by optical flow, and the intermediate slice obtained by interpolation is restored by utilizing non-local self-similarity based on blocks, so that the interlayer resolution of the CT image is improved, the quality of an MFSR (multi frequency synchronous reconstruction) reconstructed CT image is improved, and the method has important significance for doctors to diagnose diseases, patients to receive treatment and the like. The method provided by the invention is superior to shape-based algorithms in both visual and quantitative evaluation aspects.
The invention also provides a device for realizing the three-dimensional CT interlayer image interpolation restoration and super-resolution processing method, which comprises the following steps: the memory is used for storing a computer program and a three-dimensional CT interlayer image interpolation repairing and super-resolution processing method; and the processor is used for executing the computer program and realizing a multi-memory pressure testing system so as to realize the steps of the three-dimensional CT interlayer image interpolation restoration and super-resolution processing method.
The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. Various features are described as modules, units or components that may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices or other hardware devices. In some cases, various features of an electronic circuit may be implemented as one or more integrated circuit devices, such as an integrated circuit chip or chipset.
If implemented in hardware, the invention relates to an apparatus, which may be, for example, a processor or an integrated circuit device, such as an integrated circuit chip or chipset. Alternatively or additionally, if implemented in software or firmware, the techniques may implement a data storage medium readable at least in part by a computer, comprising instructions that when executed cause a processor to perform one or more of the above-described methods. For example, a computer-readable data storage medium may store instructions that are executed, such as by a processor.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (6)
1. The three-dimensional CT interlayer image interpolation restoration and super-resolution processing method is characterized by comprising the following steps:
solving the corresponding relation of pixels between adjacent slices of a CT image through a C L G-TV optical flow estimation model;
finding a target position through a mapping function of the corresponding relation between pixels, and interpolating an intermediate slice in an image warping mode;
in the interlayer interpolation process of the three-dimensional CT image, warping operation is introduced into an algorithm to obtain an interpolation result;
calculating a mapping function between each pixel in the original image and the motion vector of the pixel based on the image warping to realize the displacement of the pixel point and obtain a target image;
the interpolation of the intermediate layer between the two images is to interpolate the generated motion, so that the motion object generates an interpolation image in a natural and proper mode along the motion track, and simultaneously, the geometric distortion is compensated for the nonlinear translation motion;
calculating a mapping function q of a pixel through a C L G-TV optical flow modeli＝f(pi) Determining the pixel p in the original sliceiTarget position q ofi(ii) a Using a bilinear interpolation algorithm at qiImplementing a warping operation;
determining a target position qiWith its diagonal neighborhood ND (q)i) Selecting the pixel point with the minimum Euclidean distance as an insertion point;
ρ＝(|x1-x2|2+|y1-y2|2)1/2(6)；
and thirdly, repairing fine holes and fuzziness generated in the interpolation process by utilizing the inherent interframe non-local self-similarity of the 4D-CT image.
2. The three-dimensional CT interlayer image interpolation restoration and super-resolution processing method according to claim 1, wherein the first step further comprises:
the method comprises the following steps of expanding a single pixel point in a global algorithm into the neighborhood of the single pixel point by using a brightness conservation assumption through a C L G algorithm, and obtaining a more accurate and compact optical flow field without changing a smoothing term, wherein a C L G optical flow estimation model is as follows:
ECLG＝∫Ω(W2(x,y)·(Ixvx+Iyvy+It)2+S)dxdy. (1)
wherein W (x, y) is a weight coefficient of each point in a neighborhood Ω with the point (x, y) as the center, Ix、Iy、ItIs gray of a pixel pointPartial derivatives of degree in x, y, t directions
Is a smoothing term;
l in C L G optical flow estimation model2Problem of norm sensitivity to image noise, introducing L for optical flow1Norm constraint ensures that the estimation model has more robust anti-noise performance
Simultaneously introducing bilateral filtering technology and anisotropic diffusion smoothing strategy;
adding bilateral filtering constraints and anisotropic regularization to the data items enhances a specific filling process to obtain a C L G-TV variational optical flow estimation model:
in the above formula, the first and second carbon atoms are,
the filter is a data item, omega is a weight coefficient between the data item and a smoothing item, bfw represents smoothing filtering on the data item, and region represents a scope of a bilateral filter; smoothing term EsmoothUsing image-driven anisotropic diffusion factor
Data item
Is defined as follows:
where X ═ X, y is the pixel point coordinate, and F ═ vx,vy) Is the optical flow in two dimensions, x denotes the x-direction and y denotes the y-direction, F0Is an initial estimate of the optical flow field.
3. The three-dimensional CT interlayer image interpolation restoration and super-resolution processing method according to claim 2, wherein the first step further comprises:
in the process of solving the optical flow model, a texture structure decomposition method and a coarse-to-fine Gaussian pyramid algorithm are adopted to perform structural texture decomposition on the image, and the input image I is decomposed into a part I containing image structure informationsAnd a part I containing image texture informationg,Ig＝I-IsDue to IgIs not affected by shadow and obstruction basically, will IgAs a new input image, initialize F0；
Pyramid layer number determination from image size
Each layer of luminous flux has
Is the flow of light passing from the low-precision k layer to the k +1 layer,
is the incremental amount of optical flow passing from the k layer to the k +1 layer;
and (4) solving the optical flow by fast iteration by using an alternating direction multiplier method until k is n-1.
4. The three-dimensional CT interlayer image interpolation restoration and super-resolution processing method according to claim 1, wherein the third step further comprises:
slicing the original image
N is the number of CT image sequences, N is the sequence IpThe total number of the original slices in the image and an intermediate slice I obtained based on an optical flow registration interpolation methodipDivided into M mutually overlapped image blocks according to step length d
Presentation slice
The ith vectorized image block, each block being m × m in size;
for the
Image block of
Using block matching techniques and Euclidean distance based similarity metric criterion
Middle search
Similar blocks of (2);
x is to beiSet as the image block of the inter-layer slice obtained by interpolation,
are image blocks on other slices;
the judgment basis of the similarity is as follows:
t is a threshold value; selecting and calculating according to similarity measurement criteria
The most similar L image blocks are used as column vectors to construct a corresponding group matrix HjNamely:
Hjexpressed as a matrix PjAnd noise matrix NjAnd (3) the sum:
Hj＝Pj+Nj. (9)
Njit can be considered as additive noise with standard deviation σ; replacement of H by H and PjAnd Pj(ii) a Singular value decomposition of H and P:
wherein U is a left singular vector, V is a right singular vector, Σ is a diagonal matrix of r × r, and the elements on the diagonal are the singular values of the matrix;
using the minimum variance estimation method, the value of P is derived:
if P is known, then the derivative of the above objective function to M is set to zero, resulting in the following result
M＝(H*H)-1H*P. (13)
Then the minimum variance of P is estimated as
Based on
Diagonal element λ of1,λ2,……,λkAre all non-zero positive integers,
substituting (15) into (14), and solving to obtain an ideal matrix
Using the derived matrix
Processing tiny holes generated by interpolation, and representing image blocks needing to be repaired by weighting non-local similar blocks
xiIs the image block to be repaired,
is with xiThe most similar L similar pieces of data,
is a weighting coefficient
Tau is an adjustment parameter controlling the decay of the exponential function;
and repairing the CT interpolation image through similar block matching to obtain the CT image with good visual effect.
5. The three-dimensional CT interlayer image interpolation restoration and super-resolution processing method according to claim 4, wherein the third step further comprises:
and adopting a weighted average method to image blocks which have a plurality of pixel estimation values and are overlapped with each other to obtain the pixel values of the image blocks.
6. A device for realizing three-dimensional CT interlayer image interpolation restoration and super-resolution processing method is characterized by comprising the following steps:
the memory is used for storing a computer program and a three-dimensional CT interlayer image interpolation repairing and super-resolution processing method;
a processor for executing the computer program and the three-dimensional CT interlayer image interpolation repairing and super-resolution processing method to realize the steps of the three-dimensional CT interlayer image interpolation repairing and super-resolution processing method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910415030.1A CN110211193B (en) | 2019-05-17 | 2019-05-17 | Three-dimensional CT (computed tomography) interlayer image interpolation restoration and super-resolution processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910415030.1A CN110211193B (en) | 2019-05-17 | 2019-05-17 | Three-dimensional CT (computed tomography) interlayer image interpolation restoration and super-resolution processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110211193A CN110211193A (en) | 2019-09-06 |
| CN110211193B true CN110211193B (en) | 2020-08-04 |
Family
ID=67787565
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910415030.1A Active CN110211193B (en) | 2019-05-17 | 2019-05-17 | Three-dimensional CT (computed tomography) interlayer image interpolation restoration and super-resolution processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110211193B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112287973A (en) * | 2020-09-28 | 2021-01-29 | 北京航空航天大学 | Digital image countermeasure sample defense method based on truncated singular value and pixel interpolation |
| WO2022133806A1 (en) * | 2020-12-23 | 2022-06-30 | 深圳迈瑞生物医疗电子股份有限公司 | Fetal face volume image inpainting method and ultrasound imaging system |
| CN115082323B (en) * | 2022-08-19 | 2022-11-04 | 深流微智能科技(深圳)有限公司 | Image processing method, image processing device, electronic equipment and storage medium |
| CN116797457B (en) * | 2023-05-20 | 2024-05-14 | 北京大学 | Method and system for simultaneously realizing super-resolution and artifact removal of magnetic resonance image |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107221013A (en) * | 2017-05-16 | 2017-09-29 | 山东财经大学 | One kind is based on variation light stream estimation lung 4D CT Image Super Resolution Processing methods |
| CN107274347A (en) * | 2017-07-11 | 2017-10-20 | 福建帝视信息科技有限公司 | A kind of video super-resolution method for reconstructing based on depth residual error network |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9247129B1 (en) * | 2013-08-30 | 2016-01-26 | A9.Com, Inc. | Self-portrait enhancement techniques |
| KR101711589B1 (en) * | 2015-12-08 | 2017-03-02 | 연세대학교 산학협력단 | Method and Apparatus of Dictionary Design on Super-Resolution, and Generating Super-Resolution Image based on the Dictionary |
| CN107067367A (en) * | 2016-09-08 | 2017-08-18 | 南京工程学院 | A kind of Image Super-resolution Reconstruction processing method |
| CN107025632B (en) * | 2017-04-13 | 2020-06-30 | 首都师范大学 | Image super-resolution reconstruction method and system |
| CN108154474B (en) * | 2017-12-22 | 2021-08-27 | 浙江大华技术股份有限公司 | Super-resolution image reconstruction method, device, medium and equipment |
| CN109410177B (en) * | 2018-09-28 | 2022-04-01 | 深圳大学 | Image quality analysis method and system for super-resolution image |
| CN109658361B (en) * | 2018-12-27 | 2022-12-06 | 辽宁工程技术大学 | Motion scene super-resolution reconstruction method considering motion estimation errors |
-
2019
- 2019-05-17 CN CN201910415030.1A patent/CN110211193B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107221013A (en) * | 2017-05-16 | 2017-09-29 | 山东财经大学 | One kind is based on variation light stream estimation lung 4D CT Image Super Resolution Processing methods |
| CN107274347A (en) * | 2017-07-11 | 2017-10-20 | 福建帝视信息科技有限公司 | A kind of video super-resolution method for reconstructing based on depth residual error network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110211193A (en) | 2019-09-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110211193B (en) | Three-dimensional CT (computed tomography) interlayer image interpolation restoration and super-resolution processing method and device | |
| Zhou et al. | Handbook of medical image computing and computer assisted intervention | |
| Trinh et al. | Novel example-based method for super-resolution and denoising of medical images | |
| Huang et al. | CaGAN: A cycle-consistent generative adversarial network with attention for low-dose CT imaging | |
| CN107025632B (en) | Image super-resolution reconstruction method and system | |
| He et al. | A nonlinear least square technique for simultaneous image registration and super-resolution | |
| Gao et al. | Zernike-moment-based image super resolution | |
| US7885455B2 (en) | Method of combining images of multiple resolutions to produce an enhanced active appearance model | |
| CN107221013A (en) | One kind is based on variation light stream estimation lung 4D CT Image Super Resolution Processing methods | |
| JP2008511395A (en) | Method and system for motion correction in a sequence of images | |
| Zhi et al. | CycN-Net: A convolutional neural network specialized for 4D CBCT images refinement | |
| CN113793272B (en) | Image noise reduction method and device, storage medium and terminal | |
| CN114241077B (en) | CT image resolution optimization method and device | |
| Pan et al. | Iterative residual optimization network for limited-angle tomographic reconstruction | |
| Liu et al. | Video frame interpolation via optical flow estimation with image inpainting | |
| Zhang et al. | Video super-resolution with 3D adaptive normalized convolution | |
| Chen et al. | DuSFE: Dual-Channel Squeeze-Fusion-Excitation co-attention for cross-modality registration of cardiac SPECT and CT | |
| Barzigar et al. | A video super-resolution framework using SCoBeP | |
| Mukherjee et al. | Complete spatiotemporal quantification of cardiac motion in mice through enhanced acquisition and super-resolution reconstruction | |
| Farhadi et al. | Data augmentation of CT images of liver tumors to reconstruct super-resolution slices based on a multi-frame approach | |
| Khodajou-Chokami et al. | PARS-NET: a novel deep learning framework using parallel residual conventional neural networks for sparse-view CT reconstruction | |
| CN111476888B (en) | Medical image interlayer interpolation method, device and readable storage medium based on three-dimensional space body fitting | |
| Katartzis et al. | Robust Bayesian estimation and normalized convolution for super-resolution image reconstruction | |
| CN101048800A (en) | Method and system for motion correction in a sequence of images | |
| Zhu et al. | Robust MR image super‐resolution reconstruction with cross‐modal edge‐preserving regularization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |