CN116612009A - Multi-scale connection generation countermeasure network medical image super-resolution reconstruction method - Google Patents

Abstract

The invention discloses a multi-scale connection generation countermeasure network medical image super-resolution reconstruction method, and relates to the field of image processing. The multi-scale connection generation countermeasure network medical image super-resolution reconstruction method comprises the following steps: s1, establishing a medical image super-resolution data set; s2, designing a medical image super-resolution algorithm; s3, training a medical image super-resolution model; s4, inputting the low-resolution medical images in the test set into the model to obtain corresponding output results; the invention introduces a multiple combination degradation modeling process to better embody the complexity of real medical image degradation; the invention uses the proposed image degradation module, the generator and the discriminator module to greatly improve the robustness and the medical image detail reconstruction of the proposed model compared with the classical model.

Description

Multi-scale connection generation countermeasure network medical image super-resolution reconstruction method

Technical Field

The invention belongs to the field of image processing, and relates to improvement of an image super-resolution reconstruction algorithm, a residual convolution neural network and realization and application of a generation countermeasure network in the field of high-resolution image reconstruction. In particular to a super-resolution reconstruction method for medical images of a multiscale connection generation countermeasure network.

Background

Image super-resolution is an important image processing technology in computer vision and image processing, and refers to a process of recovering a high-resolution image from a low-resolution image. In practical applications in the field of medical image processing, it is often desirable to obtain a high resolution raw image, because the high resolution image means a higher pixel density, and can provide more abundant high frequency detail information, thereby creating a good basis for accurate extraction and utilization of medical image information. However, in a real situation, due to factors such as image acquisition time, irradiation dose, hardware limitation and the like of an imaging system of the electronic computer tomography equipment, the problems of speckle noise and low resolution of medical images are caused. In order to obtain medical images with high signal-to-noise ratio and high resolution, it is very necessary to achieve super-resolution reconstruction.

Super-resolution reconstruction currently mainly comprises three methods of interpolation-based, reconstruction-based and learning-based. Interpolation-based methods reconstruct HR images using high frequency information of the basis function approximation loss image, such as: bilinear interpolation and bicubic interpolation. Such methods are simple and effective, but suffer from visual artifacts such as blurring, aliasing, and aliasing. The reconstruction-based method is to add constraint conditions in the reconstruction process, such as: convex set projection and iterative back projection. Such methods rely on a priori knowledge and the reconstructed image lacks detailed features. The learning-based method predicts high-frequency information of the LR image by training a mapping relationship between the learning LR image and the HR image. Such methods are currently the dominant direction, with better results than the former two.

The existing super-resolution method based on medical images has the following limitations that firstly, the data set can not completely meet the requirement of real medical image degradation only by adopting a single interpolation method; secondly, with noise, the denoising step loses some of the high frequency content of the image. And thus how to improve the quality of the acquired image. Obtaining high resolution medical images that meet the application needs is a key topic of research in image processing.

Disclosure of Invention

Aiming at the problems in the prior art, the invention provides a multi-scale connection generation countermeasure network medical image super-resolution reconstruction method, which adopts the following technical scheme:

a multi-scale connection generation countermeasure network medical image super-resolution reconstruction method comprises the following steps:

s1, establishing a medical image super-resolution data set: and performing image degradation processing on the original high-resolution medical image to obtain a corresponding low-resolution image, so as to simulate the low-resolution image acquired under the real condition, and taking the pair of images as input.

S2, designing a medical image super-resolution algorithm: the super-resolution generation countermeasure network composed of the enhanced generator and the discriminator is taken as a framework, and mainly comprises two parts, namely a generator model and a discriminator model. The generator model combines a multi-stage residual error network and dense connection, so that the representation capability of the generator network is enhanced, and the performance of the network is improved; the arbiter uses a network model taking the Unet as a basic structure, and can more fully combine low-level semantic information and high-level semantic information to realize the interaction of the image cross-latitude characteristics.

S3, training a medical image super-resolution model: the model is built based on an algorithm designed by S2, a deep learning framework Pytorch platform is utilized to train the model, the training process is divided into two stages, the first stage is trained to obtain a peak signal-to-noise ratio guide model with L1 loss, the training quantity of each batch is 12, the epoch is set to be 100, and the total iteration times are 2.5 multiplied by 10 ⁴ The learning rate is initialized to 2×10 ^-4 Through 6.25X10 ³ The learning rate update decays by half after a number of iterations. Then using the guide model as a pre-training model of the second stage generator, wherein the training number of each batch is 6, the epoch is set to be 100, and the total iteration number is 5 multiplied by 10 ⁴ The learning rate is initialized to 1×10 ^-4 Through 1.25X10 ⁴ And updating the learning rate after the iteration for half attenuation, and continuing the iteration until the loss converges to obtain a final model.

S4, inputting the low-resolution medical images in the test set into the model to obtain a corresponding medical image super-resolution output result.

Preferably, the image degradation processing in S1 includes the steps of:

s101, image blurring: while gaussian blur kernel is widely used to simulate blur degradation, it does not characterize the blur process in medical images well. In order to contain more different blur kernel sizes, generalized Gaussian blur and a plateau-shaped distribution are adopted, and probability density functions are respectively:and->Beta is a shape parameter.

S102, downsampling an image: the classical downsampling operation is modified into a random sampling operation, including medical image downsampling, upsampling and maintaining operations, and the algorithm also performs random selection, including bilinear interpolation, bicubic interpolation and area adjustment. The combination of different scales and different algorithms brings different degradation effects, and plays a certain role in simulating the degradation process of the actual medical image.

S103, image noise: because the long-term working temperature of the CT machine detector is too high, the noise of the equipment and the mutual influence of components can cause Gaussian noise to occur in the generated medical image, and moreover, the detector can not accept all X rays at some time or can receive more X rays at other time, so that the gray value of the image fluctuates and poisson noise occurs.

S104, image JPEG compression: first, an input image is converted into YC by an RGB three-channel model _b C _r And (3) carrying out downsampling operation on each chrominance channel respectively by using the color space model, dividing the image into 8 multiplied by 8 small blocks, processing each small block by using two-dimensional Discrete Cosine Transform (DCT), quantizing three floating point number matrixes representing each small block, and finally obtaining compressed image data.

Preferably, the medical image super-resolution algorithm flow described in S2 mainly includes the following steps:

s201, reconstructing an image by a generator: the low resolution image after image degradation processing and the corresponding high resolution image are input as training pairs into a generator network consisting of 23 residual dense modules for SR reconstruction. Meanwhile, a pyramid attention mechanism is added to the network, so that richer detail information can be extracted from the multi-scale feature pyramid, and finer images can be generated. After reconstruction, two upsampling operations are performed using the nearest interpolated sampling function, and then three channel output images are generated using the convolutional layer.

S202, a discriminator judges true or false: the reconstructed SR image and the corresponding HR image forming image pair are input into a Unet-Plus discriminator which comprehensively fuses multi-scale features, each decoder layer in the discriminator fuses small-scale, same-scale and large-scale feature mapping in an encoder, thereby capturing fine-granularity details and coarse-granularity semantics on the whole scale, outputting real perception for each pixel, accurately identifying whether the image belongs to the SR image or the HR image, and simultaneously providing detailed result pixel-by-pixel difference feedback to a generator for learning.

S203, loss function: in the arbiter, the previous L1 loss is combined, and the content loss based on the VGG network model and the countermeasure loss based on the generated countermeasure network are adopted on the perception characteristic. Experiments prove that when the weight of the L1 loss function and the weight of the content loss function are 1, and the weight of the anti-loss function is 0.1, the model performance achieves a good effect, so that the calculation formula of the loss function is as follows:

wherein: l (L) ^SR Representing the loss function of the algorithm of the present invention;is the L1 penalty function of the generator; />Is the countering loss function.

S204, the generator and the arbiter continue to fight in such a way that the generator can reconstruct the LR image with super resolution as a model until the arbiter cannot determine the difference between the SR image and the HR image.

The invention has the characteristics and beneficial effects that:

the invention provides an improved medical image super-resolution reconstruction network model. The model will generate an antagonism network as a base structure, trained using purely synthetic medical image data. A degradation modeling process is introduced to better embody the complexity of real degradation. A pyramid attention module PANet is added between generator residual modules of the model, so that the model is more focused on capturing image high-frequency information, and the reconstruction capability of the model on image edge textures is improved. In addition, an improved U-Net discriminator with spectral normalization is designed to improve the discrimination capability of the discriminator and stabilize training dynamics. Experiments are carried out on the two medical image data sets of the abdomen, the pelvis and the lung of the model, and the result shows that compared with the prior algorithm, the model has higher improvement on the aspects of robustness and CT image detail reconstruction than the classical model.

Drawings

FIG. 1 is a flow chart of a method for super-resolution reconstruction of medical images of a multiscale connection generation countermeasure network;

FIG. 2 is a high resolution medical image degradation model;

FIG. 3 is a modified medical image super-resolution reconstruction network model;

fig. 4 is a graph showing the comparison of objective quality indicators of images of various algorithms under different conditions.

Figure 5 is a comparison of image results four times the super resolution per algorithm.

Detailed Description

For clarity and completeness of description of the technical scheme and technical effect of the present invention, the following examples are provided for the purpose of detailed description.

Example 1:

referring to fig. 1, a method for generating a super-resolution reconstruction of an antagonistic network medical image by multi-scale connection includes the following steps:

s1, establishing a medical image super-resolution data set: when a single classical degradation model is used to synthesize the training pair dataset, there is a large difference from the real medical image degradation, resulting in a slightly inadequate effect of reconstructing the LR image. Therefore, a plurality of image degradation models are introduced to be combined to carry out image degradation processing on an original high-resolution medical image, so as to obtain a corresponding low-resolution image, thereby simulating the low-resolution image acquired under real conditions, and such a pair of images is taken as input.

S2, designing a medical image super-resolution algorithm: the method adopts a low-resolution medical image and a high-resolution medical image as training pairs, takes a super-resolution generation countermeasure network consisting of a reinforced generator and a discriminator as a framework, and mainly comprises two parts, namely a generator model and a discriminator model. And se:Sup>A pyramid Attention mechanism is added between dense residual blocks in the middle of the generator and is mainly divided into two modules, namely pyramid downsampling and S-A Attention. The pyramid downsampling module mainly performs downsampling processing on an input image and adds pyramid Attention to different feature layers, the S-A Attention module constructs three feature graphs to represent information of different features, and then convolution and deconvolution operations are performed between the feature graphs, and the mechanism fully utilizes feature fusion to enable se:Sup>A generator to effectively aggregate texture detail features and semantic features; the discriminator uses a Unet-Plus network structure, the structure is an improvement on a Unet_3+ network, the low-level semantic information and the high-level semantic information can be combined more fully, the interaction of image latitude-crossing features is realized, meanwhile, the complexity of the discriminator structure and a degradation model can cause the instability of training, the training dynamics are stabilized by using spectrum normalization, and meanwhile, the normalization operation is also beneficial to alleviating the problems of serious image sharpening and artifact caused by the generation of an countermeasure network.

S3, training a medical image super-resolution model: the model is built based on an algorithm designed by S2, a deep learning framework Pytorch platform is utilized to train the model, the training process is divided into two stages, the first stage is trained to obtain a peak signal-to-noise ratio guide model with L1 loss, the training quantity of each batch is 12, the epoch is set to be 100, and the total iteration times are 2.5 multiplied by 10 ⁴ The learning rate is initialized to 2×10 ^-4 Through 6.25X10 ³ The learning rate update decays by half after a number of iterations. Then using the guide model as a pre-training model of the second stage generator, wherein the training number of each batch is 6, the epoch is set to be 100, and the total iteration number is 5 multiplied by 10 ⁴ The learning rate is initialized to 1×10 ^-4 Through 1.25X10 ⁴ And updating the learning rate after the iteration for half attenuation, and continuing the iteration until the loss converges to obtain a final model.

S4, inputting the low-resolution medical images in the test set into the model to obtain a corresponding medical image super-resolution output result.

Referring to fig. 2, the image degradation process in S1 includes the following steps:

s101, image blurring: the blur degradation is typically modeled as a convolution with a linear blur filter, commonly used anisotropic and isotropic gaussian filters. For a gaussian blur kernel k of kernel size 2t+1, its (i, j) e [ -t, t ], the element is sampled from the gaussian distribution in the form:

wherein: a is a normalization constant; x is a covariance matrix, further denoted as:r is a rotation matrix, and the R is a rotation matrix,σ ₁ sum sigma ₂ θ is the rotation angle, which is the standard deviation along the two principal axes. When sigma is ₁ Equal to sigma ₂ K is an isotropic Gaussian blur kernel when it is present, and an anisotropic Gaussian blur kernel otherwise.

While gaussian blur kernel is widely used to simulate blur degradation, it does not characterize the blur process in medical images well. In order to contain more different blur kernel sizes, generalized Gaussian blur and a plateau-shaped distribution are adopted, and probability density functions are respectively:and->Beta is a shape parameter.

S102, downsampling an image: the sampling operation is a basic process of synthesizing a low resolution image in the super resolution of the image. In order to be more similar to a multi-scale realistic medical image, the generalization capability of the generation countermeasure network is improved, classical downsampling operation is modified into random sampling operation, including medical image downsampling, upsampling and maintaining operation, and algorithms also perform random selection, including bilinear interpolation, bicubic interpolation and area adjustment. The combination of different scales and different algorithms brings different degradation effects, and plays a certain role in simulating the degradation process of the actual medical image.

S103, image noise: image noise is the interference of a random signal received during acquisition or transmission, and some random, discrete, isolated pixels appear on the image. Because the long-term working temperature of the CT machine detector is too high, the noise of the equipment and the mutual influence of components can cause Gaussian noise to occur in the generated medical image, and moreover, the detector can not accept all X rays at some time or can receive more X rays at other time, so that the gray value of the image fluctuates and poisson noise occurs. Thus, additive gaussian noise and poisson noise are chosen to be added to the original high resolution image as a degradation process of the image noise type. The probability density function of additive gaussian noise follows a normal distribution as follows:

wherein: a is a constant; μ is mathematical expectation; sigma is standard deviation, and the noise intensity is controlled.

Poisson noise is then subject to poisson distribution as follows:

wherein: delta is the expected value

S104, image JPEG compression: JPEG compression is a common digital image lossy compression technique. Since the degradation of quality and accuracy of medical images is typically caused by JPEG compression, it is an essential loop for the degradation process of input images. First, an input image is converted into YC by an RGB three-channel model _b C _r And (3) carrying out downsampling operation on each chrominance channel respectively by using the color space model, dividing the image into 8 multiplied by 8 small blocks, processing each small block by using two-dimensional Discrete Cosine Transform (DCT), quantizing three floating point number matrixes representing each small block, and finally obtaining compressed image data.

Example 2:

referring to fig. 3, the difference based on embodiment 1 is that:

the medical image super-resolution algorithm flow described in S2 mainly comprises the following steps:

s201, reconstructing an image by a generator: the low resolution image after image degradation processing and the corresponding high resolution image are input as training pairs into a generator network consisting of 23 residual dense modules for SR reconstruction. Meanwhile, a pyramid attention mechanism is added to the network, the pyramid attention mechanism comprises two parts of self-adaptive feature pooling and full-connection fusion, the self-adaptive feature pooling extracts information on all feature levels and fuses, richer background information is obtained from high-level features, and finer features are obtained from low-level features. The full connection fusion ensures that the network is more efficient and has stronger generalization capability. After reconstruction, two upsampling operations are performed using the nearest interpolated sampling function, and then three channel output images are generated using the convolutional layer.

S202, a discriminator judges true or false: the reconstructed SR image and the corresponding HR image forming image pair are input into a Unet-Plus discriminator which comprehensively fuses multi-scale features, each decoder layer in the discriminator fuses small-scale, same-scale and large-scale feature mapping in an encoder, thereby capturing fine-granularity details and coarse-granularity semantics on the whole scale, outputting real perception for each pixel, accurately identifying whether the image belongs to the SR image or the HR image, and simultaneously providing detailed result pixel-by-pixel difference feedback to a generator for learning.

S203, loss function: for the generator, a loss function (L1 loss) of all absolute errors is adopted, and the loss function is more stable and insensitive to abnormal values relative to an L2 loss function, and is more suitable for the super-resolution task of medical images. In the arbiter, the previous L1 loss is combined, and the content loss based on the VGG network model and the countermeasure loss based on the generated countermeasure network are adopted on the perception characteristic. Experiments prove that when the weight of the L1 loss function and the weight of the content loss function are 1, and the weight of the anti-loss function is 0.1, the model performance achieves a good effect, so that the calculation formula of the loss function is as follows:

wherein: l (L) ^SR Representing the loss function of the algorithm of the present invention;is the L1 penalty function of the generator; />Is the countering loss function.

S204, the generator and the arbiter continue to fight in such a way that the generator can reconstruct the LR image with super resolution as a model until the arbiter cannot determine the difference between the SR image and the HR image.

In order to obtain CT images with high peak signal-to-noise ratio and high resolution, the invention provides an improved medical image super-resolution reconstruction network model. The model will generate an antagonism network as a base structure, trained using purely synthetic medical image data. A degradation modeling process is introduced to better embody the complexity of real degradation. A pyramid attention module PANet is added between generator residual modules of the model, so that the model is more focused on capturing image high-frequency information, and the reconstruction capability of the model on image edge textures is improved. In addition, an improved U-Net discriminator with spectral normalization is designed to improve the discrimination capability of the discriminator and stabilize training dynamics. Experiments are carried out on the PAUP-ESRGAN model on two medical image data sets of abdomen pelvis and lung, and the result shows that compared with the prior algorithm, the model has higher robustness and CT image detail reconstruction than the classical model. What has been described above is merely some embodiments of the present invention. It will be apparent to those skilled in the art that various modifications and improvements can be made without departing from the inventive concept, and any changes, equivalent substitutions and improvements made within the inventive concept will be within the scope of the invention.

Claims (3)

1. A multi-scale connection generation countermeasure network medical image super-resolution reconstruction method is characterized by comprising the following steps of: the method comprises the following steps:

s1, establishing a medical image super-resolution data set: performing image degradation processing on an original high-resolution medical image to obtain a corresponding low-resolution image, so as to simulate the low-resolution image acquired under the real condition, and taking the pair of images as input;

s2, designing a medical image super-resolution algorithm: the super-resolution generation countermeasure network composed of the reinforced generator and the discriminator is taken as a framework, and mainly comprises two parts, namely a generator model and a discriminator model, wherein the generator model combines a multi-stage residual error network and dense connection, the representation capability of the generator network is enhanced, and the performance of the network is improved; the discriminator uses a network model taking the Unet as a basic structure, and can more fully combine low-level semantic information and high-level semantic information to realize the interaction of the image cross-latitude characteristics;

s3, training a medical image super-resolution model: the model is built based on an algorithm designed by S2, a deep learning framework Pytorch platform is utilized to train the model, the training process is divided into two stages, the first stage is trained to obtain a peak signal-to-noise ratio guide model with L1 loss, the training quantity of each batch is 12, the epoch is set to be 100, and the total iteration times are 2.5 multiplied by 10 ⁴ The learning rate is initialized to 2×10 ^-4 Through 6.25X10 ³ After iteration, the learning rate is updated and attenuated by half, then a guide model is used as a pre-training model of a second stage generator, the training quantity of each batch is 6, epoch is set to be 100, and the total iteration times are 5 multiplied by 10 ⁴ The learning rate is initialized to 1×10 ^-4 Through 1.25X10 ⁴ Updating the learning rate after the iteration for half attenuation, continuing the iteration until the loss converges to obtain a final model;

s4, inputting the low-resolution medical images in the test set into the model to obtain a corresponding medical image super-resolution output result.

2. The multi-scale connected generation countermeasure network medical image super-resolution reconstruction method according to claim 1, wherein: the image degradation process includes the steps of:

s101, image blurring: while gaussian blur kernels are widely used to model blur degradation, they do not characterize the blur process well in medical images, and in order to contain more diverse blur kernel sizes, a generalized gaussian blur and a plateau-shaped distribution are used with probability density functions of:and->Beta is a shape parameter;

s102, downsampling an image: the classical downsampling operation is modified into random sampling operation, including downsampling, upsampling and maintaining operation of medical images, and algorithms are randomly selected, including bilinear interpolation, bicubic interpolation and area adjustment, different degradation effects are brought by the mutual combination of different scales and different algorithms, and a certain effect is played for simulating the degradation process of real medical images;

s103, image noise: because the long-term working temperature of the CT machine detector is too high, the noise of the equipment and the mutual influence of components can cause Gaussian noise to the generated medical image, and the detector can not accept all X rays at some time or can receive more X rays at other time, so that the gray value of the image fluctuates and poisson noise appears;

s104, image JPEG compression: first, an input image is converted into YC by an RGB three-channel model _b C _r And (3) carrying out downsampling operation on each chrominance channel respectively by using the color space model, dividing the image into 8 multiplied by 8 small blocks, processing each small block by using two-dimensional Discrete Cosine Transform (DCT), quantizing three floating point number matrixes representing each small block, and finally obtaining compressed image data.

3. The multi-scale connected generation countermeasure network medical image super-resolution reconstruction method according to claim 1, wherein: the medical image super-resolution algorithm flow described in S2 mainly comprises the following steps:

s201, reconstructing an image by a generator: the low-resolution image subjected to image degradation processing and the corresponding high-resolution image are used as training pairs to be input into a generator network consisting of 23 residual intensive modules for SR reconstruction; meanwhile, a pyramid attention mechanism is added to the network, so that richer detail information can be extracted from the multi-scale feature pyramid, a finer image is generated, after reconstruction, two upsampling operations are performed by using a sampling function nearest to interpolation, and then three-channel output images are generated by using a convolution layer;

s202, a discriminator judges true or false: inputting the reconstructed SR image and the corresponding HR image forming image pair into a Unet-Plus discriminator which comprehensively fuses multi-scale features, wherein each decoder layer in the discriminator fuses small-scale, same-scale and large-scale feature mapping in an encoder, thereby capturing fine-granularity details and coarse-granularity semantics on the whole scale, outputting real perception for each pixel, accurately identifying whether the image belongs to the SR image or the HR image, and simultaneously providing detailed result pixel-by-pixel difference feedback to a generator for learning;

s203, loss function: in the discriminator, the previous L1 loss is combined, and the content loss based on the VGG network model and the countermeasure loss based on the generated countermeasure network are adopted on the perception characteristic, through experimental verification, when the weights of the L1 loss function and the content loss function are taken as 1, and the weight of the countermeasure loss function is taken as 0.1, the model performance achieves a good effect, so that the calculation formula of the loss function is as follows:

wherein: l (L) ^SR Representing a loss function of the algorithm;is the L1 penalty function of the generator; />Is the countering loss function;

s204, the generator and the arbiter continue to fight in such a way that the generator can reconstruct the LR image with super resolution as a model until the arbiter cannot determine the difference between the SR image and the HR image.