CN111724307A - An image super-resolution reconstruction method based on maximum posterior probability and non-local low-rank prior, terminal and readable storage medium - Google Patents

Abstract

The present invention provides an image super-resolution reconstruction method based on maximum a posteriori probability and non-local low-rank prior, a terminal and a readable storage medium, using a continuous image sequence as data input, and using the similarity within a single image and between continuous images Using the property as prior knowledge, combining the local grouping method of image blocks to match similar blocks, mining the spatial structure relationship at the pixel level of the image; modeling with the maximum posterior probability framework, using Gaussian distribution and Gibbs distribution to fit model parameters, Improve the generalization ability of the model; use low-rank truncation to suppress noise interference; use non-local low-rank constraints to regularize the image reconstruction process, and then use local information in a single image and local information between consecutive images to improve the quality of the target image. Alternately optimize the parameters in the model in each iteration to improve the robustness of the model and avoid local convergence. Finally, the reconstructed image blocks are weighted and averaged to obtain the target high-resolution image.

Description

An Image Super-Resolution Reconstruction Based on Maximum Posterior Probability and Nonlocal Low-Rank Priors Method, terminal and readable storage medium

technical field

The present invention relates to the technical field of image processing, and in particular, to an image super-resolution reconstruction method based on maximum a posteriori probability and non-local low-rank prior, a terminal and a readable storage medium.

Background technique

Vision is one of the main ways for humans to obtain information from the outside world, and the performance of most vision-based applications depends on the quality of images. High Resolution (HR) images have high resolution and contain rich image details and more image information, so they are of great value in practical applications such as medicine, video surveillance, and remote sensing. Although image imaging technology has matured in these practical application fields, the resolution of most images is very low due to the mutual constraints of imaging equipment, imaging environment, human interference and other factors. For example, in the field of medicine, medical image imaging technology is constrained by factors such as medical imaging equipment, radioactive element hazards, and human physiological health. Many medical image imaging resolutions are very low, and low resolution (LR) images cannot effectively assist The task of classification and detection of lesion tissue, therefore, image super-resolution technology came into being. Super Resolution (SR) technology is an image analysis and processing technology that takes the observed low-resolution single image or image sequence as input to generate a high-resolution single image or image sequence. Interpolation-based methods, learning-based methods, and reconstruction-based methods are currently three types of popular image super-resolution methods.

The interpolation-based method is an earlier and relatively simple algorithm. Firstly, the registration relationship between the low-resolution image and the target high-resolution image is calculated, and then according to the interpolation formula, the pixel value of the point to be interpolated is obtained by using the known pixel value in the neighborhood, so as to obtain the target high-resolution image. Common ones are Bilinear Interpolation, Bicubic Interpolation and Nearest Neighbor Interpolation. The method based on interpolation is simple in implementation and low in computational complexity, so it has good real-time performance. However, the anisotropy of the image is not considered in the interpolation process, and the high-frequency information of the image cannot be effectively preserved, resulting in blurred contours and textures of the enlarged image, prone to blocky appearance, false edges, and poor image quality. The interpolation algorithm is difficult to deal with the blur phenomenon in the image and the noise introduced by the image, and cannot add prior information, so the adaptability of the method is also poor.

In the prior art, learning-based methods have gradually become a popular type of technology in recent years. The basic idea of the algorithm is to learn the mapping relationship between low-resolution images and high-resolution images from the training sample set, so as to predict the unknown low-resolution images and improve the resolution. For example, using the idea of local embedding for popular learning, learning strategies based on Neighbor Embedding, algorithms based on Example Learning, neural network algorithms combined with deconvolution, etc., but such methods are not suitable for external training. The data set has a large dependency, and the model increment is poor. Recent research on image statistics shows that image patches can be sparsely linearly represented by the elements of their overcomplete dictionary, so the learning method based on sparse representation (Sparse Representation) has been applied, but this algorithm requires a large number of training data sets, and some Datasets require manual annotation, and internal dictionaries are often insufficient to contain complex texture information for good reconstruction. To sum up, it can be seen that the learning model is very important for the effect of image super-resolution, but the current model cannot effectively combine all the prior knowledge required for image reconstruction, and the learning-based method runs for a long time, which is difficult to meet the real-time performance. requirements, so it is more suitable for offline image preprocessing.

Reconstruction-based methods aim to reconstruct high-frequency signals lost during degradation. Assuming that the input low-resolution image has a total of n frames, the mathematical model of the reconstruction-based super-resolution problem can be expressed as:

是大气模糊算子,M_k是运动变换算子,

指成像模糊算子。D为降采样算子,N_k代表在成像过程中引入的加性噪声。已知输入l_k,则基于重建的方法目标是寻找真实高分辨率图像H的最优估计

Here _lk is a low-resolution image obtained from the original high-resolution image H to be reconstructed through a series of image transformation processes.

is the atmospheric blur operator, M _k is the motion transformation operator,

Refers to the imaging blur operator. D is the down-sampling operator, and N _k represents the additive noise introduced in the imaging process. Given the input l _k , the goal of the reconstruction-based method is to find the optimal estimate of the real high-resolution image H

The frequency domain method is one of the earliest reconstruction-based super-resolution methods proposed by Tsai and Huang in 1984. The discrete Fourier transform and the continuous Fourier transform are performed on the low-resolution image and the target high-resolution image respectively, and the linear relationship between the two is established in the frequency domain according to the properties of the Fourier transform. Rhee and Kang et al. replaced the discrete Fourier transform in the frequency domain with discrete cosine transform, which reduced storage requirements and costs. Although the theory of frequency domain method is simple and has certain advantages in derivation and calculation, it is difficult to deal with noise and add prior information. In addition, due to the complex transformation relationship between the frequency domain and the space domain, it can only deal with the situation of global overall motion, and it is difficult to deal with the situation with local motion.

There is also an iterative backprojection method in the prior art, and the iterative backprojection (IBP) method was proposed by Irani and Peleg. The method is to back-project the difference between the low-resolution image generated by the degradation model and the input low-resolution image onto the high-resolution image, and iterate continuously to make the error converge to obtain the target high-resolution image. The IBP method is intuitive and easy to understand, but IBP is an inverse problem, and its ill-posedness will lead to non-unique solutions. Projection onto convex sets (POCS) is an iterative super-resolution reconstruction method. In the POCS method, the influence of prior information on the results can be added, such as constraints on the peak pixels of the target image. The POCS method is more flexible in form and can easily add prior information. However, the computational complexity of the method is high, requiring multiple iterations and projections, the convergence speed is relatively slow, and the algorithm stability is not high.

SUMMARY OF THE INVENTION

In order to overcome the above-mentioned deficiencies in the prior art, the present invention provides an image super-resolution reconstruction method based on maximum a posteriori probability and non-local low-rank prior, the method comprising:

Step 1: Using a continuous image sequence as data input, using the similarity between a single image and the continuous image as prior knowledge, and combining the local grouping of image blocks to perform block matching on similar blocks, and mining the spatial structure relationship at the pixel level of the image;

Step 2. Use the maximum posterior probability framework to model, and use Gaussian distribution and Gibbs distribution to fit model parameters to improve the generalization ability of the model;

Step 3, estimating the singular value of the block to be determined by the maximum a posteriori of the singular value of the similar blocks, and suppressing the noise interference by means of low-rank truncation;

Step 4: Use the non-local low-rank constraint to regularize the image reconstruction process, and then use the local information in a single image and the local information between consecutive images to improve the quality of the target image.

It should be further noted that, after step 4, it also includes:

The reconstructed image quality was evaluated by peak signal-to-noise ratio, structural similarity and feature similarity.

It should be further noted that step 1 also includes:

The continuous image sequence is Y{y _k }, k=1,...,10; the continuous image sequence consists of a series of low-resolution images, and the image y in the middle of the sequence is used as the reference image for super-resolution reconstruction, and other images are cooperatively assisted ;

The known image degradation model is:

y=DB _k x+n _k (1)

In the formula, x is the high-resolution image to be reconstructed, and y is the input low-resolution image, which is obtained by bicubic interpolation of the original image;

的加性高斯白噪声；D is the down-sampling operator, and B _k is the fuzzy operator. Suppose n _k is zero mean and variance is

The additive white Gaussian noise;

In step 2, the purpose of super-resolution is to reconstruct a high-resolution image x from a low-resolution reference image y; expressed by the MAP model as the following objective function:

In formula (2), a Gaussian function is used to fit,

Assuming that the pixel value is related to the adjacent pixels that satisfy the Gibbs probability density function, the Gibbs function is used to fit,

Substituting equations (3) and (4) into equation (2), we get:

Equation (5) is the objective function to be sought.

It should be further noted that step 4 also includes:

Use a non-local low-rank prior, and express the non-local low-rank prior as a maximum posterior probability estimate;

表示图像块x_j的一系列相似块,其中j为图像块索引,相似块是以j为中心形成的大小为

的图像块；Assumption

Represents a series of similar blocks of the image block x _j , where j is the image block index, and the size of the similar block is formed with j as the center

the image block;

When similar blocks are matched, select the most similar p-block similar blocks in the whole sequence image, and then perform low-rank truncation processing;

Assuming that each group of low-rank blocks is independent of each other, the low-rank prior is expressed as:

Substitute equation (6) into equation (5) to get:

Usually low-rank matrices are usually solved in nuclear normal form,

的低秩块,||L_j||_*为L_j的核范数,用以表示奇异值的和；使用迭代方向乘子,通过构造增强拉格朗日方程,求解式(8)；Among them, L _j is the desired

The low-rank block of , ||L _j || _* is the nuclear norm of L _j , which is used to represent the sum of singular values; using the iterative direction multiplier, by constructing the enhanced Lagrangian equation, solve Equation (8);

where U _j is the Lagrange multiplier and μ is a constant parameter. Solving equation (9) can be decomposed into two sub-problems:

Among them, the solution of x can be directly obtained by formula (10):

的奇异值估计出L_j的奇异值，得出L_j；Using the MAP method to solve the low-rank block L _j , use

Estimate the singular value of L _j from the singular value of , and obtain L _j ;

Equation (12) can be obtained from the Bayesian criterion,

Suppose the distortion degree f represents the distortion degree of the singular value of the high-resolution image block and the singular value of the low-resolution image block; in formula (13), the first part is fitted with a Gaussian function with a mean value of 0 and a standard deviation of f,

P(σ _i (L _j )) is calculated by kernel density estimation, and its probability density function is regarded as the sum of a series of kernel functions; the sum of singular values is the sum of a series of kernel functions, and the kernel function is determined by σ _i ( L _j ) is determined by the 1×3 neighborhood Ω _i of the center;

标准差为h_i的高斯分布,则奇异值的概率密度函数被定义为:Assuming that the kernel function mean fits the mean

If the standard deviation is a Gaussian distribution of _hi , the probability density function of singular values is defined as:

Substituting the i-th index in equations (14) and (15) into equation (13), we get,

Let the derivative of Eq. (16) be 0, the solution is:

Averaging all the resulting MAP estimates:

Finally, the estimation of the low-rank image similarity block L _j is obtained,

The enhanced Lagrange multiplier U _j can be updated by formula (20),

It should be further noted that the evaluation methods of peak signal-to-noise ratio include:

The formula for calculating the peak signal-to-noise ratio is as follows:

Among them, MSE is the mean square error between the image to be evaluated and the reference image; G is the gray level of the image; the larger the PSNR value, the smaller the difference between the image to be evaluated and the reference image, and the higher the image quality.

It should be further noted that the evaluation methods of structural similarity include:

The formula for calculating structural similarity is as follows:

Among them, μ _x and μ _y are the grayscale average values of the image to be evaluated and the reference image, respectively, σ _x and σ _y represent the standard deviation, C ₁ =(k ₁ G) ² , C ₂ =(k ₂ G) ² as Constant to maintain numerical stability, in the formula k ₁ =0.01, k ₂ =0.03, G is the image gray scale.

It should be further noted that the evaluation methods of feature similarity include:

The formula for calculating feature similarity is as follows:

Among them, S _L (x)=S _PC (x) · S _G (x), S _PC (x) and S _G (x) are the values of PC and GM between images, respectively; PC _m (x) is the largest The PC value is used to weight the contribution of each point to the overall similarity of the two images; x is the position of a given pixel, and Ω is the full airspace of the image.

The present invention also provides a device for realizing an image super-resolution reconstruction method based on maximum a posteriori probability and non-local low-rank prior, including:

a memory for storing a computer program and an image super-resolution reconstruction method based on maximum a posteriori probability and non-local low-rank prior;

a processor for executing the computer program and the image super-resolution reconstruction method based on the maximum a posteriori probability and the non-local low-rank prior, so as to realize the image super-resolution reconstruction method based on the maximum a posteriori probability and the non-local low-rank prior A step of.

The present invention also provides a readable storage medium with an image super-resolution reconstruction method based on maximum a posteriori probability and non-local low-rank prior, where a computer program is stored on the readable storage medium, and the computer program is executed by a processor to realize Steps of an image super-resolution reconstruction method with maximum posterior probability and non-local low-rank priors.

As can be seen from the above technical solutions, the present invention has the following advantages:

Based on the maximum a posteriori probability reconstruction method, the present invention applies the MAP framework to the image super-resolution reconstruction technology, and combines the self-similarity of the image and the non-local low-rank prior to establish a super-resolution reconstruction model that fully utilizes the information contained in the image; The method of the invention takes a continuous image sequence as an input, and adopts a similar block grouping technology to preprocess a single reference image and the images before and after it into a high-dimensional tensor form, which facilitates the similarity comparison of image blocks, achieves the purpose of fast block matching, and improves the computational efficiency. speed. A non-local low-rank prior based on image patches is added to the constructed MAP model, which makes full use of image details and avoids the loss of narrow features. The singular value of the block to be determined is estimated by the singular value of the similar block, and the most similar image block is selected and processed by low-rank truncation, so as to suppress the interference of small factors such as noise and improve the quality of the reconstructed image. At the same time, the parameters in the model are optimized alternately in each iteration, which can improve the robustness of the model and avoid local convergence. Finally, the reconstructed image blocks are weighted and averaged to obtain the target high-resolution image.

Description of drawings

In order to illustrate the technical solutions of the present invention more clearly, the accompanying drawings required in the description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention, which are not relevant to ordinary skills in the art. As far as personnel are concerned, other drawings can also be obtained from these drawings on the premise of no creative work.

Fig. 1 is the flow chart of the image super-resolution reconstruction method;

Fig. 2 is a schematic diagram of image gradient distribution;

Figure 3 is a schematic diagram of the influence of the number of iterations on PSNR, SSIM, and FSIM;

Figure 4 is an experimental comparison diagram of different regular terms;

Fig. 5 is the comparative medical image example diagram of four kinds of methods and the method of the present invention;

Fig. 6 is the contrast natural image example diagram of four kinds of methods and the method of the present invention;

FIG. 7 is an example diagram of a natural image comparing the four methods and the method of the present invention.

Detailed ways

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented by electronic hardware, computer software or a combination of the two. In order to clearly illustrate the hardware and software In the above description, the components and steps of each example have been generally described according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of the present invention.

The block diagrams shown in the figures are merely functional entities and do not necessarily necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices entity.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may also be electrical, mechanical or other forms of connection.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided in order to give a thorough understanding of embodiments of the present invention. However, those skilled in the art will appreciate that the technical solutions of the present invention may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be employed. In other instances, well-known methods, devices, implementations, or operations have not been shown or described in detail to avoid obscuring aspects of the present invention.

The present invention provides an image super-resolution reconstruction method based on maximum a posteriori probability and non-local low-rank prior, and the process is shown in FIG. 1 . The continuous image sequence is used as the data input, and the similarity between the single image and the continuous image is used as the prior knowledge to improve the matching degree of similar image blocks and eliminate the loss of image details. Then, the model is modeled with the maximum posterior probability framework, and the model parameters are fitted with Gaussian distribution and Gibbs distribution to improve the generalization ability of the model. The singular value of the block to be determined is estimated by the singular value of the similar block, and the low-rank truncation is used to suppress the noise introduced in the reconstruction process. Finally, using the non-local self-similarity and low-rank properties of the image, the image reconstruction process is regularized with non-local low-rank constraints, and the local and global information of the image is added to improve the reconstruction effect. The method of the present invention effectively improves the quality of the reconstructed image, and achieves better results in performance indicators such as peak signal-to-noise ratio and structural similarity compared with the existing algorithms.

Among them, the maximum a posteriori (Maximum a posterior, MAP) method is an algorithm framework based on statistical probability, which is the most widely used method in practical application and scientific research. The basic idea of the algorithm is derived from conditional probability, which takes the known LR image sequence as the observation result and estimates the unknown HR image. In the MAP framework, the regularization term plays a key role in controlling the quality of the reconstructed image. The MAP method is more flexible, especially in the regular term part of the MAP framework, which can add specific constraints to specific problems. Therefore, an effective regular term is the key to ensure the performance of the MAP framework. For example, commonly used regularization terms include Tikhonov regularization terms in two-norm form, Total Variation (TV) regularization terms in one-norm form, and Bilateral TV (BTV) regularization terms in one-norm form, and there are more complex regularization terms. The Student-t regular term. MAP has a complete theoretical framework, a flexible spatial domain model and strong prior knowledge inclusion. The maximum a posteriori probability method involved in the present invention has better adaptability, flexibility and robustness, and can produce excellent reconstruction results. , is an effective super-resolution reconstruction method.

进而重建出目标高分辨图像。加入图像相似性和低秩性能有效地提升重建图像质量,针对低分辨率图像序列Y{y_k},k＝1,...,10的最大后验概率模型,以及利用自然图像的非局部自相似性,采用非局部低秩正则项(NLR)对图像重建过程正则化。The MAP method involved in the present invention is an algorithm framework based on statistical probability.

Then the high-resolution image of the target is reconstructed. The addition of image similarity and low-rank performance can effectively improve the quality of reconstructed images, the maximum posterior probability model for low-resolution image sequences Y{y _k }, k=1,...,10, and the non-local use of natural images. For self-similarity, a non-local low-rank regularizer (NLR) is used to regularize the image reconstruction process.

In the present invention, Y{y _k } is composed of a series of low-resolution images, and the image y in the middle of the sequence is used as a reference image for super-resolution reconstruction, and other images are cooperatively assisted. The known image degradation model is:

y=DB _k x+n _k (1)

In the formula, x is the high-resolution image to be reconstructed, and y is the input low-resolution image, which is obtained by bicubic interpolation of the original image.

的加性高斯白噪声。超分辨的目的是从低分辨率基准图像y重建出高分辨率图像x。该问题可由MAP模型表述为下面的目标函数:D is the down-sampling operator, and B _k is the fuzzy operator. Suppose n _k is zero mean and variance is

additive white Gaussian noise. The purpose of super-resolution is to reconstruct a high-resolution image x from a low-resolution reference image y. The problem can be formulated by the MAP model as the following objective function:

Among them, the gradient distribution of natural images has a heavy-tailed phenomenon. As shown in Figure 2, the gradient distribution of the urban natural image (City) conforms to the heavy-tailed distribution. Through the study of the gradient of medical tomography (CT) images, it also exists in medical images. Heavy tail phenomenon. Therefore, the first term in the above formula can be fitted by a Gaussian function,

Assuming that pixel values are only related to adjacent pixels that satisfy the Gibbs probability density function, the second term can be fitted with the Gibbs function,

Substituting equations (3) and (4) into equation (2), we get:

Equation (5) is the objective function to be sought.

表示图像块x_j的一系列相似块,其中j为图像块索引,相似块是以j为中心形成的大小为

的图像块。在相似块匹配时,选取整个序列图像中最相似的p块相似块,再进行低秩截断处理。假设每组低秩块是相互独立的,因此低秩先验表示为:The non-local low-rank regularization and process involved in the present invention are as follows: the image block-based non-local low-rank prior has been well applied in the fields of image denoising and image super-resolution. The present invention adds a non-local low-rank prior on the basis of the above invention, and expresses it as a maximum a posteriori probability estimate. Assumption

Represents a series of similar blocks of the image block x _j , where j is the image block index, and the size of the similar block is formed with j as the center

image block. When the similar blocks are matched, the most similar p-block similar blocks in the whole sequence image are selected, and then low-rank truncation processing is performed. It is assumed that each group of low-rank blocks is independent of each other, so the low-rank prior is expressed as:

Substitute equation (6) into equation (5) to get:

Usually low-rank matrices are usually solved in nuclear normal form,

的低秩块,||L_j||_*为L_j的核范数,用以表示奇异值的和。Among them, L _j is the desired

The low-rank block of , ||L _j || _* is the nuclear norm of L _j , which is used to represent the sum of singular values.

The invention uses the iterative direction multiplier to solve the formula (8) by constructing the enhanced Lagrangian equation.

where U _j is the Lagrange multiplier and μ is a constant parameter. Solving equation (9) can be decomposed into two sub-problems:

Among them, the solution of x can be directly obtained by formula (10):

的奇异值估计出L_j的奇异值,从而得出L_j。The present invention adopts the MAP method to solve the low-rank block L _j , using

The singular values of L _j are estimated to obtain L _j .

Equation (12) can be obtained from the Bayesian criterion,

It is assumed that the distortion degree f represents the distortion degree of the singular value of the high-resolution image block and the singular value of the low-resolution image block. In the above formula, the first part can be regarded as fitting with a Gaussian function with a mean of 0 and a standard deviation of f,

标准差为h_i的高斯分布,则奇异值的概率密度函数被定义为:In addition, P(σ _i (L _j )) can be calculated by kernel density estimation, and its probability density function is identified as the sum of a series of kernel functions, so the sum of singular values is the sum of a series of kernel functions, and the kernel function is given by The 1×3 neighborhood Ω _i centered on σ _i (L _j ) is determined. In the present invention, it is assumed that the mean value of the kernel function conforms to the mean value

If the standard deviation is a Gaussian distribution of _hi , the probability density function of singular values is defined as:

Substituting the i-th index in equations (14) and (15) into equation (13), we get,

Let the derivative of Eq. (16) be 0, the solution is:

Averaging all the resulting MAP estimates:

Finally, the estimation of the low-rank image similarity block L _j is obtained,

Here, the enhanced Lagrangian multiplier U _j can be updated by Equation (20),

In order to evaluate the above-mentioned image super-resolution reconstruction method based on maximum a posteriori probability and non-local low-rank prior, the present invention sets three evaluation indexes, wherein the first index is peak signal-to-noise ratio;

Peak Signal to Noise Ratio (PSNR) is an objective standard for evaluating the similarity between an image and a reference image. It measures the image quality after processing and is widely used in image quality evaluation. Its calculation formula is as follows:

Among them, MSE is the mean square error between the image to be evaluated and the reference image. G is the gray scale of the image. The larger the PSNR value, the smaller the difference between the image to be evaluated and the reference image, and the higher the image quality. However, the PSNR value is only an objective standard for image quality evaluation, and does not take human visual factors into account. Even if the PSNR value is high, there may be a large error between the actual image quality and the expected image quality, so the evaluation of image quality needs to be comprehensive. Analyze various evaluation indicators.

Another evaluation index is: structural similarity. Structural Similarity (SSIM) is a degradation-based image quality evaluation method, which judges the image quality by comparing the structural similarity between the image to be evaluated and the reference image. Each pixel in an image has strong dependencies on surrounding pixels, and these dependencies carry important information about the target image structure in visual perception. Its calculation formula is as follows:

Among them, μ _x and μ _y are the grayscale average values of the image to be evaluated and the reference image, respectively, σ _x and σ _y represent the standard deviation, C ₁ =(k ₁ G) ² , C ₂ =(k ₂ G) ² as Constant to maintain numerical stability, in the formula k ₁ =0.01, k ₂ =0.03, G is the image gray scale. SSIM is defined by the brightness and contrast associated with structural information, with the mean gray value as an estimate of the brightness measure and the standard deviation as an estimate of the contrast measure. In addition, because SSIM is a symmetric metric, it can be regarded as a similarity metric for comparing any two signals. Signals can be discrete or continuous, and can exist in any dimension of space.

Another evaluation indicator is: feature similarity.

Feature Similarity (FSIM) is a relatively successful variant of SSIM. Among them, Phase Congruency (PC) in FSIM is used to measure the importance of local structure. Considering that PC has contrast invariance, and contrast affects the perception of image quality by the human visual system, the image quality is used in FSIM. Gradient Magnitude (GM) is used as a secondary feature. The FSIM is divided into two parts, PC and GM, and the calculation formula is as follows:

Among them, S _L (x)=S _PC (x) · S _G (x), S _PC (x) and S _G (x) are the values of PC and GM between images, respectively. PC _m (x) is the maximum PC value used to weight the contribution of each point to the overall similarity of the two images. x is the position of a given pixel, and Ω is the entire airspace of the image. FSIM mainly uses phase similarity and image gradient similarity to measure the importance of local structure. In the evaluation quality score stage, the phase similarity is used as a weight, which increases the correlation with human visual perception and achieves good quality. Evaluate the effect.

The present invention also verifies and analyzes the above method. The experimental images of the present invention are selected from 5 groups of various types of images such as medical image sets, natural image sets and video framed image sets, each group of 10 image sequences, and any one image is selected as the reconstructed image. Among them, the size of each image is 128×128, and the size of the reconstructed HR image is 256×256. The experimental medical image data were provided by the relevant hospitals. In the natural image dataset tested, the urban dataset comes from the video database developed by New York University Institute of Technology, etc. The database contains 10 videos encoded with H.264, the resolution ranges from QCIF to 4CIF, and the quantization parameters range from 28 to 44. The frame rate is between 3.75 and 30 frames per second, and only 10 frames are selected as experimental data in the experiment of the present invention. The small orchard data set is derived from the optical flow standard experimental data set, which is selected as a natural image to test the reconstruction performance of the method of the present invention. The experiment uses MATLAB 2014(b) to realize the image super-resolution reconstruction algorithm in the present invention. The hardware facilities of the experiment are Intel(R) Xeon(R) E5-2643 v4@3.40GHz CPU, NVIDIA GeForce GTX 1080M GPU, 256GB memory, and the operating system is ubuntu 14.04.

The images used in the experiments are standard continuous image sequences, and the number of iterations is set to 12. The experimental results were evaluated by peak signal-to-noise ratio, structural similarity and feature similarity.

In the experimental model based on MAP and non-local low-rank prior proposed in the present invention, the experimental parameter settings are optimized through different types of comparative experiments. The research shows that the size of the image patch not only affects the running speed of the experiment, but also affects the accuracy of image registration and the quality of the reconstructed image. In the experiment, the block size is set to 3×3, 5×5, 7×7, 9×9, and 11×11 respectively, and the number of iterations is set to 12 times. The PSNR, SSIM, and FSIM of the experiment are shown in Table 1.

Table 1 Influence of image block size on FSIM and SSIM

The numerical statistics of image blocks of 3×3 and 5×5 are significantly higher than the experimental results of other groups. When the image blocks are set to 7×7, 9×9, 11×11, compared with 5×5, the difference in the numerical statistical indicators of PSNR value is smaller, but the operating efficiency of the algorithm will increase with the increase of image block size. According to the experimental results in Table 1, the 11×11 image block has better performance only in SSIM; among them, the difference between the 3×3 and 5×5 image blocks is only 0.0020 in the three comparisons of FSIM, but in the comparison of SSIM Among them, 5×5 was significantly higher than 3×3, and the three-term differences were 0.0330, 0.0214, and 0.0160. To sum up, in this experiment, the image block size of 5×5 is selected as the optimal parameter setting, and the running time is also very small compared with 3×3, which is greatly reduced compared with 7×7. From the objective evaluation index and the visual effect of the reconstructed image of the invention, it can be seen that the block processing of the image can effectively improve the quality of the reconstructed image.

Among them, the number of iterations of the regular term in the experiment is set to 12 times, and the different number of iterations will have different effects on the experimental results. As shown in Figure 3, the medical lung image (CT) and natural image (City) are used as experimental data to compare the effects of different iteration times on PSNR, SSIM, and FSIM, where the arrow points to the data point where the iteration number is set to 12 times. . With the increase of the number of iterations, the values of PSNR, FSIM and SSIM tend to be stable, but it can be seen from the PSNR curve that after 12 experiments, the two groups of experiments will have small fluctuations in various indicators. Compared with FSIM and SSIM, the change curve is relatively flat, and the PSNR curve has a relatively obvious downward trend. This is because in the reconstruction-based super-resolution method, the image solution is a typical ill-posed inverse problem, which will be introduced in the iterative solution process. Interference from other minor factors like noise. Therefore, the number of experimental iterations in the present invention is set to 12 to avoid interference when solving ill-conditioned problems, and at the same time, the use of low-rank truncation in the process of solving similar blocks can effectively suppress the disturbance of small factors such as noise to the experiment.

The present invention is a restrictive study of the maximum a posteriori prior probability term. Different regular terms have different functions in the reconstruction-based method. The present invention is concerned with the Tikhonov regular term in the form of two norm, the regular term of total variation in the form of one norm (L1TV), and the regular term of total variation in the form of two norm ( In the research of L2TV), the regularization term based on low rank and total variation (LRTV) and the non-local low-rank regularization term (MAP_NLR) of the present invention, the tests were carried out with the number of iterations being 6, 8, 10, 12, and 14, respectively. (For simplicity of expression, the number of iterations is equal to 12 as an example).

By studying the comparison experiments of Tikhonov regular term, one-norm TV regular term, two-norm TV regular term, LRTV regular term and MAP_NLR regular term, the benchmark images in the process of image super-resolution reconstruction are used for testing. After experiments, the present invention selects the most representative example: if it is a Tikhonov regular term, the regular term coefficient takes a value of 0.01; if it is an L1TV regular term, it takes a value of 0.005; if it is an L2TV, it takes a value of 0.05; if it is an LRTV regular term term, the average value is 0.01; MAP_NLR regular term, with 0.1 as the test value. Select the respective best experimental results, carry out comparative experiments, and observe the peak signal-to-noise ratio with the regular term of the present invention, as shown in Table 2:

Table 2 PSNR comparison (the number of iterations is equal to 12)

From the experimental results in Table 2, it can be seen that the PSNR results of MAP_NLR and LRTV on lung images are better than the PSNR results of L1TV, L2TV, and Tikhonov. And the PSNR value of MAP_NLR on the urban natural image is better than the PSNR value of the other four regular terms, which shows that the existence of low-rank prior is the premise for the better effect of the model of the present invention; among them, the PSNR value of the two test images is compared. Among them, MAP_NLR is better than LRTV, it can be seen that the existence of non-local low-rank prior is the core element of the model of the present invention.

Figure 4 shows the actual results of selecting lung medical images and urban natural images as test images. Although the overall images are not very different, through local display, it can be seen that the differences are more reflected in the image details. According to the image regions selected jointly, the detail description ability of MAP_NLR, LRTV and Tikhonov is obviously higher than that of L1TV and L2TV. Compared with LRTV and Tikhonov, the texture of MAP_NLR is clear, and there is no heavy blurred edge phenomenon, which proves the necessity of grouping similar blocks and the effectiveness of the low-rank truncation method to suppress noise from the image visual aspect. In addition, although the PSNR of the image processed by Tikhonov is relatively low, its detail restoration ability is not bad, which is better than L1TV and L2TV.

The present invention combines the model with the nearest neighbor interpolation (Nearest Neighbor Interpolation), bicubic interpolation (Bicubic), method based on low rank and total variation regularization (LRTV) and based on two norm quadratic term parsing algorithm (L2-L2) The results of image super-resolution processing are compared with each other, in which, in the comparison experiment, the parameters are unified, the images used by the model for the experiment are the same images in the continuous image sequence, and the number of iterations is set to 12. The hardware facilities of the experiments are all the same and are Intel(R) Xeon(R) E5-2643 v4@3.40GHz CPU, NVIDIA GeForce GTX 1080M GPU, 256GB memory, and the operating system is ubuntu 14.04. The experimental results are visually compared by calculating the corresponding PSNR, SSIM and FSIM respectively.

Under the complete data set of the present invention, the results of calculating PSNR, SSIM and FSIM between the method of the present invention and the four methods are shown in Table 2.

PSNR is one of the important indicators to objectively evaluate the image quality, and its value can determine the effectiveness of the super-resolution reconstruction effect of the algorithm to a large extent. SSIM and FSIM can form a supplementary reference to PSNR. As can be seen from Table 3, under the image data set used in the present invention, among the four methods used in the comparative experiments, the calculated value of PSNR of L2-L2 is relatively high, and the calculated value of SSIM and FSIM of LRTV is relatively high. It is higher, which proves that the low-rank nature of the image can reconstruct the image details well.

In addition to comparing the experimental results of all data sets, the present invention also separately lists PSNR, SSIM, and FSIM of lung medical images, urban natural images, and small orchard natural images. Table 3 shows the comparison results of PSNR, SSIM and FSIM in the liver image, lung image medical dataset, city image, orchard image and flower image natural dataset with image size of 128×128. Among them, the experimental results index values of the image datasets corresponding to different experimental methods are PSNR, SSIM, FSIM from top to bottom.

As shown in Fig. 3, the images obtained by using different algorithms for image super-resolution are shown in Fig. 5, Fig. 6 and Fig. 7. It can be seen that the average PSNR of the method of the present invention is 0.17dB worse than the previous one. Compared with the liver image data, the differences of the three indicators compared with LRTV were 1.91dB, 0.0270, and 0.0227, respectively, but the average PSNR, SSIM, FSIM of the lung image, city image, small orchard image, and SSIM and FSIM of all images were averaged. Including the value, the method of the present invention has certain advantages. As shown in Fig. 5, Fig. 6 and Fig. 7, when comparing images locally, the method of the present invention has great advantages in both image reconstruction of lung images and super-resolution reconstruction of natural images, and can better Image details are preserved without blurring artifacts and aliasing effects. By comparing the indicators of the visual effect and image quality of the reconstructed image, it can be seen that the method of the present invention is superior to the other four methods.

Table 3 Comparison of PSNR, SSIM and FSIM for image super-resolution with different algorithms

Fig. 5 is a lung image selected from the experimental data set, and the results obtained in the five reconstruction models including the present invention, the original image is located at the far left, the Ground Truth is located at the far right of the image, and the rest are among The arrangement of super-resolution methods, the last method in the verification method (located on the left side of the Ground Truth image) is the method of the present invention. Among them, the topmost reconstructed image is displayed locally, and it can be seen from the visual effect that the method of the present invention is the closest to Ground Truth. According to the above images, it can be seen that LRTV and the method of the present invention are the most stable, and the image fluctuation is not large, but the nearest neighbor method and the bicubic algorithm are in an unstable state, especially the results obtained by the nearest neighbor method have obvious sawtooth effect. The local display of the result image of the L2L2 method shows obvious blurring effect, which is almost absent in the method of the present invention and LRTV, indicating that the L2L2 cannot effectively preserve the image details. LRTV is the closest to the method of the present invention, but it is easy to blur the image texture, resulting in blurred image edges. It can also be seen from the error result graph at the bottom that the difference between the method of the present invention and the original image is small, and the reconstructed image is of high quality and extremely stable.

Natural images contain a lot of detailed information, and it is very important to reconstruct the high-frequency information of the image effectively for the image quality. It can be seen from Figure 6 and Figure 7 that the nearest neighbor method and the bicubic method cannot effectively restore the high-frequency information of the image, which in turn affects the quality of the reconstructed image, resulting in blurred edge texture and block artifacts. Observing the local display of natural images, it can be seen that LRTV will coarsen the edge texture, indicating that low-rank regularization and total variational regularization have a superposition effect in the realization of detail preservation, resulting in the inability to effectively restore image details. The L2L2 method produces granular artifacts in the high-frequency details of the image, which cannot effectively preserve the image information in smooth areas, resulting in blurring effects at the edges of the texture, unable to suppress noise interference, and poor definition of the reconstructed image. Compared with the above four methods, the method of the present invention can better preserve the image details, and can effectively suppress the interference of small factors such as noise during the reconstruction process of the smooth area of the image, thereby improving the image quality of the reconstruction result.

Based on the scheme of the present invention, the maximum a posteriori framework is improved, and a non-local low-rank prior is used as a regularization term to perform image super-resolution reconstruction. The continuous image sequence is used as the data input, the non-local self-similarity of the image is used as the prior knowledge, and the similar blocks are matched by the local grouping technology of image blocks, and the spatial structure relationship at the pixel level of the image is fully exploited. Secondly, in the MAP framework of the present invention, the singular value of the block to be sought is estimated by the maximum a posteriori of the singular value of the similar blocks, and the low-rank truncation method is used to suppress the interference of small factors such as noise. Finally, the non-local low-rank constraint regularization image reconstruction process is adopted to make full use of the local and global information within a single image and between consecutive images to improve the quality of the target high-resolution image. The model of the present invention adopts multiple evaluation indicators: peak signal-to-noise ratio, structural similarity, etc., and shows advantages. The experimental results show that the super-resolution model of the present invention has good robustness in image super-resolution reconstruction. However, in the low-rank truncation process, some small details are lost. The research found that the residual has a good effect in preserving the details. How to add the residual to the model to further preserve the image details is the focus of the next step.

Based on the above method, the present invention also provides a device for realizing an image super-resolution reconstruction method based on maximum a posteriori probability and non-local low-rank prior, including:

a memory for storing a computer program and an image super-resolution reconstruction method based on maximum a posteriori probability and non-local low-rank prior;

a processor for executing the computer program and the image super-resolution reconstruction method based on the maximum a posteriori probability and the non-local low-rank prior, so as to realize the image super-resolution reconstruction method based on the maximum a posteriori probability and the non-local low-rank prior A step of.

Based on the above method, the present invention also provides a readable storage medium with an image super-resolution reconstruction method based on maximum a posteriori probability and non-local low-rank prior, where a computer program is stored on the readable storage medium, and the computer program is executed by a processor To implement the steps of image super-resolution reconstruction method based on maximum posterior probability and non-local low-rank prior.

The device for realizing the image super-resolution reconstruction method based on the maximum a posteriori probability and the non-local low-rank prior is the unit and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention, and can use electronic hardware, computer software or two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of the present invention.

From the description of the above embodiments, those skilled in the art can easily understand that the device for realizing the image super-resolution reconstruction method based on the maximum a posteriori probability and the non-local low-rank prior described here can be realized by software, or can be combined by software necessary hardware way to achieve. Therefore, the technical solutions according to the disclosed embodiments for realizing the image super-resolution reconstruction method based on the maximum a posteriori probability and the non-local low-rank prior can be embodied in the form of a software product, and the software product can be stored in a non-volatile storage medium (which may be CD-ROM, U disk, mobile hard disk, etc.) or on the network, including several instructions to cause a computing device (which may be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the implementation according to the present disclosure index method.

Those skilled in the art can understand that various aspects of implementing an image super-resolution reconstruction method based on a maximum a posteriori probability and a non-local low-rank prior can be implemented as a system, method or program product. Therefore, various aspects of the present disclosure can be embodied in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, which may be collectively referred to herein as implementations "circuit", "module" or "system".

Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages—such as Java, C++, etc., as well as conventional procedural Programming Language - such as the "C" language or similar programming language. The program code may execute entirely on the user computing device, partly on the user device, as a stand-alone software package, partly on the user computing device and partly on a remote computing device, or entirely on the remote computing device or server execute on. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (eg, using an Internet service provider business via an Internet connection).

The above description of the disclosed embodiments enables any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined in this invention may be implemented in other embodiments without departing from the spirit or scope of this invention. Thus, the present invention is not intended to be limited to the embodiments of the present invention shown, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (9)

1. an image super-resolution reconstruction method based on maximum posterior probability and non-local low-rank prior, is characterized in that, method comprises: Step 1: Using a continuous image sequence as data input, using the similarity between a single image and the continuous image as prior knowledge, and combining the local grouping of image blocks to perform block matching on similar blocks, and mining the spatial structure relationship at the pixel level of the image; Step 2. Use the maximum posterior probability framework to model, and use Gaussian distribution and Gibbs distribution to fit model parameters to improve the generalization ability of the model; Step 3, estimating the singular value of the block to be determined by the maximum a posteriori of the singular value of the similar blocks, and suppressing the noise interference by means of low-rank truncation; Step 4: Use the non-local low-rank constraint to regularize the image reconstruction process, and then use the local information in a single image and the local information between consecutive images to improve the quality of the target image. 2. image super-resolution reconstruction method according to claim 1, is characterized in that, After step 4, it also includes: The reconstructed image quality was evaluated by peak signal-to-noise ratio, structural similarity and feature similarity. 3. image super-resolution reconstruction method according to claim 1, is characterized in that, Step 1 also includes: The continuous image sequence is Y{y _k }, k=1,...,10; the continuous image sequence consists of a series of low-resolution images, and the image y in the middle of the sequence is used as the reference image for super-resolution reconstruction, and other images are cooperatively assisted ; The known image degradation model is: y=DB _k x+n _k (1) In the formula, x is the high-resolution image to be reconstructed, and y is the input low-resolution image, which is obtained by bicubic interpolation of the original image; D is the downsampling operator, and B _k is the fuzzy operator; suppose n _k is the mean value of 0 and the variance of

The additive white Gaussian noise; In step two, The purpose of super-resolution is to reconstruct a high-resolution image x from a low-resolution reference image y; expressed by the MAP model as the following objective function:

In formula (2), a Gaussian function is used to fit,

Assuming that the pixel value is related to the adjacent pixels that satisfy the Gibbs probability density function, the Gibbs function is used to fit,

Substituting equations (3) and (4) into equation (2), we get:

Equation (5) is the objective function to be sought. 4. image super-resolution reconstruction method according to claim 1, is characterized in that, Step 4 also includes: Use a non-local low-rank prior, and express the non-local low-rank prior as a maximum posterior probability estimate; Assumption

Represents a series of similar blocks of the image block x _j , where j is the image block index, and the size of the similar block is formed with j as the center

the image block; When similar blocks are matched, select the most similar p-block similar blocks in the whole sequence image, and then perform low-rank truncation processing; Assuming that each group of low-rank blocks is independent of each other, the low-rank prior is expressed as:

Substitute equation (6) into equation (5) to get:

Usually low-rank matrices are usually solved in nuclear normal form,

Among them, L _j is the desired

The low-rank block of , ||L _j || _* is the nuclear norm of L _j , which is used to represent the sum of singular values; using the iterative direction multiplier, by constructing the enhanced Lagrangian equation, solve Equation (8);

where U _j is the Lagrange multiplier, and μ is a constant parameter; the solution of equation (9) can be decomposed into two sub-problems:

Among them, the solution of x can be directly obtained by formula (10):

Using the MAP method to solve the low-rank block L _j , use

Estimate the singular value of L _j from the singular value of , and obtain L _j ;

Equation (12) can be obtained from the Bayesian criterion,

Suppose the distortion degree f represents the distortion degree of the singular value of the high-resolution image block and the singular value of the low-resolution image block; In formula (13), the first part is fitted with a Gaussian function with a mean of 0 and a standard deviation of f,

P(σ _i (L _j )) is calculated by kernel density estimation, and its probability density function is identified as the sum of a series of kernel functions; The sum of singular values is the sum of a series of kernel functions, which are determined by the 1×3 neighborhood Ω _i centered on σ _i (L _j ); Assuming that the kernel function mean fits the mean

If the standard deviation is a Gaussian distribution of _hi , the probability density function of singular values is defined as:

Substituting the i-th index in equations (14) and (15) into equation (13), we get,

Let the derivative of Eq. (16) be 0, the solution is:

Averaging all the resulting MAP estimates:

Finally, the estimation of the low-rank image similarity block L _j is obtained,

The enhanced Lagrange multiplier U _j can be updated by formula (20),

5. image super-resolution reconstruction method according to claim 2, is characterized in that, The evaluation methods of peak signal-to-noise ratio include: The formula for calculating the peak signal-to-noise ratio is as follows:

Among them, MSE is the mean square error between the image to be evaluated and the reference image; G is the gray level of the image; The larger the PSNR value, the smaller the difference between the image to be evaluated and the reference image, and the higher the image quality. 6. The image super-resolution reconstruction method according to claim 2, wherein, Evaluation methods for structural similarity include: The formula for calculating structural similarity is as follows:

Among them, μ _x and μ _y are the grayscale average values of the image to be evaluated and the reference image, respectively, σ _x and σ _y represent the standard deviation, C ₁ =(k ₁ G) ² , C ₂ =(k ₂ G) ² as Constant to maintain numerical stability, in the formula k ₁ =0.01, k ₂ =0.03, G is the image gray scale. 7. The image super-resolution reconstruction method according to claim 2, wherein, The evaluation methods of feature similarity include: The formula for calculating feature similarity is as follows:

Among them, S _L (x)=S _PC (x) · S _G (x), S _PC (x) and S _G (x) are the values of PC and GM between images, respectively; PC _m (x) is the largest The PC value is used to weight the contribution of each point to the overall similarity of the two images; x is the position of a given pixel, and Ω is the full airspace of the image. 8. A device for realizing an image super-resolution reconstruction method based on maximum a posteriori probability and non-local low-rank prior, characterized in that, comprising: a memory for storing a computer program and an image super-resolution reconstruction method based on maximum posterior probability and non-local low-rank prior; A processor for executing the computer program and the image super-resolution reconstruction method based on the maximum a posteriori probability and the non-local low-rank prior, so as to realize the method based on the maximum a posteriori probability and the non-local low-rank prior according to any one of claims 1 to 7. Steps of an image super-resolution reconstruction method with local low-rank priors. 9. A readable storage medium having an image super-resolution reconstruction method based on maximum a posteriori probability and non-local low-rank prior, characterized in that, a computer program is stored on the readable storage medium, and the computer program is The processor performs the steps of implementing the image super-resolution reconstruction method based on a maximum a posteriori probability and a non-local low-rank prior according to any one of claims 1 to 7.

Images