CN113920394B - No-reference image quality evaluation method and system - Google Patents
No-reference image quality evaluation method and system Download PDFInfo
- Publication number
- CN113920394B CN113920394B CN202111121862.6A CN202111121862A CN113920394B CN 113920394 B CN113920394 B CN 113920394B CN 202111121862 A CN202111121862 A CN 202111121862A CN 113920394 B CN113920394 B CN 113920394B
- Authority
- CN
- China
- Prior art keywords
- feature
- matrix
- image
- neural network
- quality evaluation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013441 quality evaluation Methods 0.000 title claims abstract description 78
- 238000000034 method Methods 0.000 title claims abstract description 74
- 230000007246 mechanism Effects 0.000 claims abstract description 50
- 238000000605 extraction Methods 0.000 claims abstract description 22
- 239000000284 extract Substances 0.000 claims abstract description 6
- 239000011159 matrix material Substances 0.000 claims description 116
- 239000013598 vector Substances 0.000 claims description 109
- 238000013527 convolutional neural network Methods 0.000 claims description 66
- 238000013528 artificial neural network Methods 0.000 claims description 43
- 238000007500 overflow downdraw method Methods 0.000 claims description 8
- 230000001629 suppression Effects 0.000 claims description 7
- 238000012216 screening Methods 0.000 claims description 6
- 238000003384 imaging method Methods 0.000 abstract description 10
- 238000005516 engineering process Methods 0.000 abstract description 3
- 238000011156 evaluation Methods 0.000 description 18
- 230000000694 effects Effects 0.000 description 14
- 238000011176 pooling Methods 0.000 description 8
- 238000001303 quality assessment method Methods 0.000 description 8
- 230000004927 fusion Effects 0.000 description 6
- 238000012549 training Methods 0.000 description 5
- 238000012795 verification Methods 0.000 description 5
- 238000013461 design Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000003646 Spearman's rank correlation coefficient Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000005562 fading Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000013178 mathematical model Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Software Systems (AREA)
- Evolutionary Computation (AREA)
- Computational Linguistics (AREA)
- Pure & Applied Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Mathematical Optimization (AREA)
- Mathematical Analysis (AREA)
- Computational Mathematics (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Algebra (AREA)
- Databases & Information Systems (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The application discloses a reference-free image quality evaluation method and a reference-free image quality evaluation system; the key points are that: comprising the following steps: an image input system, a feature extraction system and a quality evaluation network system; the output end of the image input system is connected with the input end of the characteristic extraction system, and the output end of the characteristic extraction system is connected with the input end of the quality evaluation network system. The application aims to provide a non-reference image quality evaluation method and a system thereof, and an attention mechanism module can effectively extract attention characteristics of a blurred image, has better descriptive property on the blurred condition of the image, solves the problem that the natural blurred image characteristics are complex and difficult to learn, provides effective help for application of an imaging technology in multiple fields, and has wide application prospect.
Description
Technical Field
The application relates to the technical field of performance evaluation of image processing systems, in particular to a non-reference image quality evaluation method and a non-reference image quality evaluation system.
Background
With the development of science and technology, imaging technology has been widely used in social media, medicine, agriculture, industry, traffic, military, etc. However, some interference factors, such as noise, blurring, data loss, etc., are inevitably introduced in the process of acquiring, storing, transmitting, displaying, etc., and these may cause degradation of image quality. The quality of the image determines the credibility of the image information, and influences the subjective feeling of people on the image and the acquisition of information quantity. Therefore, studies on image quality evaluation (Image quality assessment, IQA) have been widely paid attention to.
The image quality evaluation is classified into a subjective evaluation method and an objective evaluation method. Subjective evaluation is carried out on image quality by an observer, and is generally represented by average subjective scores (Mean opinion score, MOS) or average subjective score differences (DIFFERENTIALMEAN OPINION SCORE, DMOS) (namely, the difference of evaluation scores of undistorted images and distorted images by human eyes), but the subjective evaluation is large in workload, long in time consumption and inconvenient to use; the objective evaluation method is that the computer calculates the quality index of the image according to a certain method, and the quality index can be divided into three types of Full-reference image quality evaluation (Full-REFERENCE IMAGE Quality Assessment, FR-IQA), half-reference image quality evaluation (Reduced-REFERENCE IMAGE Quality Assessment, RR-IQA) and No-reference image quality evaluation (No-REFERENCE IMAGE Quality Assessment, NR-IQA) according to whether the corresponding clear image is needed as a reference image during evaluation.
When evaluating distorted images, the FR-IQA method needs to provide an undistorted original image, and an evaluation result of the distorted image is obtained by comparing the original image and the distorted image, but the method has the defect that an undistorted reference image needs to be provided, which is often difficult to obtain in practical application.
The RR-IQA method does not need to compare a distorted image with an original image, but only needs to compare certain characteristics of the distorted image with the same characteristics of the original image, and is widely applied to the fields of digital watermark verification, video quality monitoring, code flow rate control and the like.
The NR-IQA method estimates the quality of an image from the characteristics of the distorted image itself, completely without reference to the image. The reference-free method has the most practical value and very wide application range. But it is the absence of reference images and the ever-changing content of images that makes the quality assessment of reference-free images relatively more difficult. The NR-IQA method is classified into a conventional no-reference image quality evaluation method and a neural network-based no-reference image quality evaluation method. The traditional reference-free image quality method is based on spatial domain/frequency domain extraction of image characteristic information, and image characteristic is processed through simple weighting or filtering until the image quality evaluation result is obtained. The non-reference image quality evaluation method based on the neural network realizes the mapping from the image characteristics to the image quality through the neural network, thereby evaluating the image quality.
Blur is the most common cause of image quality loss, and the cause of blur is determined by two types: artificial and natural blur. Artificial blurring is the artificial addition of different types of distortion types in the reference image through different types of filters, while natural blurring is taken from the real image due to motion of objects or camera shake, etc.
The artificial blurred image is usually degraded by Gaussian blur, and a reference image is convolved with a Gaussian filter to obtain a distorted image, so that the common defocus blur in life is simulated. The natural blurred image is taken from an image in which blurring actually occurs in life, and comprises motion blurring and mixed blurring in addition to defocus blurring, if a camera is only slightly shifted when a long-distance scene is shot, the obtained blurring approximation space is unchanged, linear motion blurring can be generated, and when long-distance imaging encounters turbulence, atmospheric turbulence blurring and the like can be generated. The presence of multiple blurring conditions results in a natural blurred image containing a random mixture of multiple distortion types that makes fitting difficult by mathematical models.
In the figure 1, the upper four images are artificial blurred images, the lower four images are natural blurred images, and the comparison can show that the artificial blurred images have better blurring effect but obvious regularity in distortion, and the natural blurred images have poor regularity in distortion, are more disordered and contain various changes such as color, illumination and the like.
With the deep research, the traditional non-reference image quality evaluation method and the neural network-based non-reference image quality evaluation method have better evaluation effects on artificial blurred pictures, but have poorer evaluation effects on natural blurred images.
Disclosure of Invention
The present application aims to overcome the above-mentioned drawbacks of the prior art, and to provide a reference-free image quality evaluation system and an evaluation method thereof.
A no-reference image quality assessment system, comprising: an image input system, a feature extraction system and a quality evaluation network system; the output end of the image input system is connected with the input end of the characteristic extraction system, and the output end of the characteristic extraction system is connected with the input end of the quality evaluation network system;
The image input system is used for selecting pictures;
The feature extraction system is used for solving an NSS feature matrix, a simple feature matrix and an attention mechanism feature and quality evaluation feature matrix of the image input by the image input system;
The quality evaluation network system is used for solving the image quality estimated value.
Further, the feature extraction system includes: a convolutional neural network subsystem, an attention mechanism feature enhancement network subsystem;
The convolutional neural network subsystem comprises a deep convolutional neural network and a shallow convolutional neural network, wherein an input image extracts an NSS feature matrix f nss of the image through the deep convolutional neural network, extracts a simple feature matrix f ba of the image through the shallow convolutional neural network and scores vectors
The attention mechanism feature enhances a network subsystem through a score vector of the convolutional neural network subsystemSimple feature matrix f ba, solving quality evaluation feature matrix f q:
The output of the shallow convolutional neural network includes: simple feature matrix f ba and score vector of picture corresponding to each type of object Wherein L represents the number of feature vectors;
The simple feature matrix and the score vector are used as inputs of the attention mechanism feature enhancement network subsystem, and the outputs of the attention mechanism feature enhancement network subsystem are an attention feature f sign and a quality evaluation feature matrix f q:
the solving process of the attention mechanism characteristic enhancement network subsystem comprises the following steps:
A. Computing a suppression vector for the score vector
B. Calculating parameters
Will beInputting into a full-connection layer neural network, and solving/>I.e./>As the input value of the full-connection layer neural network, the output result is/> As the input value of the full-connection layer neural network, the output result is/>
C. Calculating the characteristic vector differenceAnd calculating a weight θ i for each of the feature differences e i in the feature vector differences described above:
Inputting the data into an MLP multi-layer perceptron (which is also a neural network in essence), wherein the output result of the MLP neural network is theta i; the above process can be expressed by the following mathematical formula:
Wherein MLP stands for multi-layer perceptron, omega 4 is the weight mark of MLP multi-layer perceptron;
D. Input characteristic vector according to weight value theta i Screening to obtain a feature set f 1 with strong significance:
f ba is a feature matrix made up of L feature vectors, Wherein each feature vector/>The score difference e i corresponds to the score a i in the score vector one by one with the weight value theta i; sorting theta i according to the size, selecting the feature vector corresponding to the first 30% weight to be reserved as the feature with strong significance, changing the rest feature vectors into 0 vector, and obtaining a feature set f 1 with strong significance;
The above examples are as follows:
If the size of theta 5 is the first 30%, Then remain; if the magnitude of theta 5 is not the first 30%,
E. And (3) adopting a 1 multiplied by 1 convolutional neural network, and adjusting a simple feature matrix f ba to obtain a feature f 2:
The matrix f ba is input into a 1×1 convolutional neural network to obtain a matrix f 2, that is, the data of the matrix f ba is used as the input data of the 1×1 convolutional neural network, and the output data of the 1×1 convolutional neural network is used as a matrix f 2;
F. The final attention feature f sign is calculated:
fsign＝f1·f2
G. The simple feature matrix f ba, the NSS feature matrix f nss and the attention feature f sign are combined by adopting a vector fusion method to obtain a quality evaluation feature matrix f q:
where α (i.e., the attention mechanism weight) is a non-negative parameter that controls the significance of the saliency map.
Further, the quality evaluation feature matrix f q is used as an input value, and an image quality estimated value is obtained through the quality evaluation network.
Further, α is 0.3.
Further, before step S100, the method further includes: all neural networks need to be trained through a data set.
After the image quality evaluation score is obtained, the image imaging quality is judged according to the image quality score, the image with low imaging quality is selectively removed, and tracking drift caused by blurring of a certain frame is prevented.
A no-reference image quality evaluation method, characterized by comprising the steps of:
s100, reading an image;
s200, extracting natural image statistical characteristics from the image acquired in the step S100:
the image obtained in the step S100 is used as input quantity, and NSS feature matrix is output through a deep convolutional neural network;
s300, solving a simple feature matrix and a score vector for the image acquired in the step S100:
The image acquired in step S100 is used as an input, and a shallow convolutional neural network SCNN is used, and output is as follows: simple feature matrix f ba and score vector of picture corresponding to each type of object Wherein L represents the number of feature vectors;
s400, solving a quality evaluation feature matrix;
s500, obtaining an image quality estimated value.
Further, the step S400 includes:
S401, calculating a suppression vector of the score vector:
S402, calculating parameters
Wherein ω 3 represents the weight of the full connection layer, mean represents taking the mean of each term of the vector;
S403, calculating the characteristic vector difference And calculating a weight θ i for each of the feature differences e i in the feature vector differences described above:
Inputting the result into an MLP multi-layer perceptron, wherein the output result of the MLP neural network is theta i;
S404, screening f ba according to the weight value theta i to obtain a feature matrix f 1 with strong significance:
f ba is a feature matrix made up of L feature vectors, Wherein each feature vector/>One-to-one correspondence with the score a i and the weight theta i in the score vector;
Sorting theta i according to the size, selecting the feature vector corresponding to the first 30% weight to be reserved as the feature with strong significance, changing the rest feature vectors into 0 vector, and obtaining a feature matrix f 1 with strong significance;
S405, a1 multiplied by 1 convolutional neural network is adopted to adjust a simple feature matrix f ba, and a feature f 2 is obtained:
The matrix f ba is input into a 1×1 convolutional neural network to obtain a matrix f 2, that is, the data of the matrix f ba is used as the input data of the 1×1 convolutional neural network, and the output data of the 1×1 convolutional neural network is used as a matrix f 2;
s406, calculating a final attention feature f sign:
fsign＝f1·f2
S407, combining the simple feature matrix f ba, the NSS feature matrix f nss and the attention feature f sign by adopting a vector fusion method to obtain a quality evaluation feature matrix f q:
Wherein α refers to the attention mechanism weight, a non-negative parameter;
Further, the step S400 includes:
0.2≤α≤0.5。
Further, the step S400 includes:
α＝0.3。
further, before step S100, the method further includes: all neural networks need to be trained through a data set.
The application has the beneficial effects that:
Firstly, the application provides a neural network-based reference-free image quality evaluation method and a neural network-based reference-free image quality evaluation system guided by an attention mechanism, wherein the neural network-based reference-free image quality evaluation method and the neural network-based reference-free image quality evaluation system have the following core concept: attention mechanisms are introduced. Compared with other non-reference image quality evaluation methods based on convolutional neural networks, the evaluation precision of the natural fuzzy image has a very remarkable effect (can be improved by 14% -40%), and can meet the effect requirements of most fields on image quality evaluation.
The specific embodiments are as follows:
the attention mechanism feature enhances the solution process of the network subsystem:
A. Computing a suppression vector for the score vector
B. Calculating parameters
Will beInputting into a full-connection layer neural network, and solving/>I.e./>As the input value of the full-connection layer neural network, the output result is/>As the input value of the full-connection layer neural network, the output result is/>
C. Calculating the characteristic vector differenceAnd calculating a weight θ i for each of the feature differences e i in the feature vector differences described above:
D. Input characteristic vector according to weight value theta i Screening to obtain a feature set f 1 with strong significance:
f ba is a feature matrix made up of L feature vectors, Wherein each feature vector/>The score difference e i corresponds to the score a i in the score vector one by one with the weight value theta i; sorting theta i according to the size, selecting the feature vector corresponding to the first 30% weight to be reserved as the feature with strong significance, changing the rest feature vectors into 0 vector, and obtaining a feature set f 1 with strong significance;
The above examples are as follows:
If the size of theta 5 is the first 30%, Then remain; if the magnitude of theta 5 is not the first 30%,
E. And (3) adopting a 1 multiplied by 1 convolutional neural network, and adjusting a simple feature matrix f ba to obtain a feature f 2:
The matrix f ba is input into a 1×1 convolutional neural network to obtain a matrix f 2, that is, the data of the matrix f ba is used as the input data of the 1×1 convolutional neural network, and the output data of the 1×1 convolutional neural network is used as a matrix f 2;
F. The final attention feature f sign is calculated:
fsign＝f1·f2
G. The simple feature matrix f ba, the NSS feature matrix f nss and the attention feature f sign are combined by adopting a vector fusion method to obtain a quality evaluation feature matrix f q:
where α (i.e., the attention mechanism weight) is a non-negative parameter that controls the significance of the saliency map.
Secondly, according to the application, through an actual case, namely two artificial fuzzy data sets and one natural fuzzy data set are selected, eight methods including the method of the application are subjected to performance verification and comparison; the validity of the method of the application was verified. Meanwhile, it was found that: alpha is 0.3 with the best effect.
Thirdly, the meaning of the application is that: the image quality score obtained by the image quality evaluation algorithm directly reflects the blurring degree of the image, and the lower the quality score is, the more blurred the image is, namely the worse the imaging quality of the image is. The image quality score is a quantitative expression mode of the credibility of the acquired information in the image, the information obtained in the image with high quality score is true and reliable, otherwise, the credibility is lower, and the method can be used in various fields. In the traditional image fusion, the fusion of an infrared image, a radar image and a visible light image is generally realized by adopting an equal proportion fusion or weighting fusion method according to a fixed weight, and the influence of factors such as weather, time, illumination, damage of imaging equipment and the like on the imaging quality of one or two images is not considered, so that the interference resistance of most image fusion algorithms is poor. After the image quality evaluation algorithm is applied, the image imaging quality can be judged according to the image quality score, the image with low imaging quality can be selectively removed, and tracking drift caused by blurring of a certain frame can be prevented.
Fourth, the application effect of the application is: for a blurred image data set, the PLCC coefficient of the algorithm can reach 0.870, the SROCC coefficient can reach 0.849, and the method has the accuracy of more than 80% for more than 85% of images, and meets the effect requirements of most fields on image quality evaluation. Compared with the traditional neural network-based non-reference image quality evaluation algorithm, the method has the advantages that the method is improved by 40% at most, and compared with the neural network-based non-reference image quality evaluation algorithm of the improved model, the method has the advantage that the method is improved by more than 14%.
Drawings
The application is described in further detail below in connection with the embodiments in the drawings, but is not to be construed as limiting the application in any way.
Fig. 1 is a schematic diagram of an artificial blurred image and a natural blurred image.
Fig. 2 is a diagram showing a design of a no-reference image quality evaluation system according to the present application.
Detailed Description
For a clearer, more complete description of the present application, from the standpoint of specific implementations, reference will be made to the accompanying drawings, which form a part, but not all, of the embodiments of the application that provide a single way of carrying out the innovative concepts presented, without excluding other ways of implementing the application already mentioned.
Example 1: the application provides a neural network-based non-reference image quality evaluation method (Combined attentional MECHANISM DEEP LEARING network, CADnet) guided by an attention mechanism, which introduces the attention mechanism based on the existing neural network-based method and integrates various characteristics. Under the condition of ensuring that the evaluation effect on the artificial fuzzy data set is unchanged, the evaluation accuracy of the natural fuzzy image is improved by 14% at least and 40% at most compared with other non-reference image quality evaluation methods based on convolutional neural networks, and the effect requirement of most fields on image quality evaluation can be met.
[ Method design ]
A neural network-based reference-free image quality evaluation method guided by an attention mechanism comprises the following steps:
s100, reading an image;
S200, extracting natural image statistical characteristics from the image obtained in the step S100 (an executing mechanism corresponding to the step S200 is a convolutional neural network subsystem of a characteristic extraction system): the image obtained in step S100 is used as input quantity, NSS feature matrix is output through Deep Convolutional Neural Network (DCNN) (the mode of processing the image by using the deep convolutional neural network belongs to the prior art, namely https:// blog. Csdn. Net/wutongyutt/arc/details/80496860):
s300, solving a simple feature matrix and a score vector for the image obtained in the step S100 (an actuating mechanism corresponding to the step S300 is a convolutional neural network subsystem of a feature extraction system, wherein a VGGnet network can be adopted as a shallow convolutional neural network SCNN of the application):
The image acquired in step S100 is used as an input, and a shallow convolutional neural network SCNN is used, and output is as follows: simple feature matrix f ba and score vector of picture corresponding to each type of object Wherein L represents the number of feature vectors;
s400, solving a quality evaluation feature matrix (an actuating mechanism corresponding to the step S400 is an attention mechanism feature enhancement network subsystem of a feature extraction system);
The attention mechanism features can be divided into location-based attention mechanism features and item-based attention mechanism features; the input of the attention mechanism characteristic based on the position is a characteristic with space dimension, the input of the attention based on the item is a characteristic vector with sequence, and the attention mechanism characteristic of combining the position and the item is adopted;
s400 specifically includes the following sub-steps:
S401, calculating a suppression vector of the score vector:
S402, calculating parameters
Wherein ω 3 represents the weight of the full connection layer, mean represents taking the mean of each term of the vector;
S403, calculating the characteristic vector difference And calculating a weight θ i for each of the feature differences e i in the feature vector differences described above:
Inputting the result into an MLP multi-layer perceptron, wherein the output result of the MLP neural network is theta i;
S404, screening f ba according to the weight value theta i to obtain a feature matrix f 1 with strong significance:
f ba is a feature matrix made up of L feature vectors, Wherein each feature vector/>One-to-one correspondence with the score a i and the weight theta i in the score vector;
Sorting theta i according to the size, selecting the feature vector corresponding to the first 30% weight to be reserved as the feature with strong significance, changing the rest feature vectors into 0 vector, and obtaining a feature matrix f 1 with strong significance;
S405, a1 multiplied by 1 convolutional neural network is adopted to adjust a simple feature matrix f ba, and a feature f 2 is obtained:
The matrix f ba is input into a 1×1 convolutional neural network to obtain a matrix f 2, that is, the data of the matrix f ba is used as the input data of the 1×1 convolutional neural network, and the output data of the 1×1 convolutional neural network is used as a matrix f 2;
s406, calculating a final attention feature f sign:
fsign＝f1·f2
S407, combining the simple feature matrix f ba, the NSS feature matrix f nss and the attention feature f sign by adopting a vector fusion method to obtain a quality evaluation feature matrix f q:
where α refers to the attention mechanism weight and is a non-negative parameter.
S500, an image quality estimation value is obtained (the actuator corresponding to step S500 is a quality evaluation network):
The quality evaluation feature matrix f q is taken as an input value, and then the image quality estimation value is solved (this step belongs to the prior art).
Structural design
An attention mechanism directed neural network based referenceless image quality assessment system comprising: an image input system, a feature extraction system and a quality evaluation network system;
The output end of the image input system is connected with the input end of the characteristic extraction system, and the output end of the characteristic extraction system is connected with the input end of the quality evaluation network system;
the feature extraction system is used for solving NSS feature matrix, simple feature matrix and attention mechanism features of the image input by the image input system;
the quality evaluation network system is used for solving the image quality estimated value.
Wherein, the feature extraction system includes: a convolutional neural network subsystem that extracts Natural image statistics (Natural SCENE STATISTICS, NSS) features, an attention mechanism feature enhancement network subsystem;
The convolutional neural network subsystem uses a deep convolutional neural network (Deep convolutional Neural Networks, DCNN) to extract the NSS feature matrix of the image (i.e., natural image statistics Natural SCENE STATISTICS), and uses a shallow convolutional neural network (Shallow convolutional Neural Networks, SCNN) to extract the simple feature matrix of the image.
The attention mechanism feature enhances a network subsystem through a score vector of the convolutional neural network subsystemThe simple feature matrix f ba is used for solving the quality evaluation feature matrix f q;
Wherein the quality assessment network (using prior art) comprises: global average pooling layer subsystem (Global Average Pooling, GAP) and deep neural network subsystem (Deep Neural Networks, DNN);
The global average pooling layer subsystem is characterized in that the input of the global average pooling layer subsystem is a fusion feature, and the output of the global average pooling layer subsystem is a feature vector (the fusion feature is reduced to be a feature vector which is convenient to process; the global average pooling layer simplifies a convolution structure through the corresponding relation between a feature map and a channel, and in addition, compared with a full-connection layer, the global average pooling layer subsystem has fewer parameters, does not need parameter optimization and can effectively prevent overfitting);
The deep neural network subsystem takes the characteristic vector output by the global average pooling layer subsystem as an input value, and outputs the characteristic vector as an image quality estimated value.
It should be noted that: the attention mechanism feature enhanced network subsystem includes: full-connected layer neural networks, MLP multi-layer perceptrons, 1 x 1 convolutional neural networks (1 x 1 convolutional layers) (these three neural networks require prior training).
F ba is a feature matrix made up of L feature vectors,
The design key points are as follows:
Firstly, the purpose of the full-connection layer neural network and the MLP multi-layer perceptron is to solve theta 1～θL (L numerical values);
Second, based on θ 1～θL to operate on f ba, solving to obtain f 1;
thirdly, solving f 2 to increase the nonlinearity of the f ba characteristic and improving the characteristic expression capability of fba;
fourth, how α is calculated, i.e., how much its size is advantageous.
The solving process of the attention mechanism characteristic enhancement network subsystem comprises the following steps:
A. Computing a suppression vector for the score vector
B. Calculating parameters
Will beInputting into a full-connection layer neural network, and solving/>I.e./>As the input value of the full-connection layer neural network, the output result is/> As the input value of the full-connection layer neural network, the output result is/>
C. Calculating the characteristic vector differenceAnd calculating a weight θ i for each of the feature differences e i in the feature vector differences described above:
Inputting the data into an MLP multi-layer perceptron (which is also a neural network in essence), wherein the output result of the MLP neural network is theta i; the above process can be expressed by the following mathematical formula:
Wherein MLP stands for multi-layer perceptron, omega 4 is the weight mark of MLP multi-layer perceptron;
D. Input characteristic vector according to weight value theta i Screening to obtain a feature set f 1 with strong significance:
f ba is a feature matrix made up of L feature vectors, Wherein each feature vector/>The score difference e i corresponds to the score a i in the score vector one by one with the weight value theta i; sorting theta i according to the size, selecting the feature vector corresponding to the first 30% weight to be reserved as the feature with strong significance, changing the rest feature vectors into 0 vector, and obtaining a feature set f 1 with strong significance;
The above examples are as follows:
If the size of theta 5 is the first 30%, Then remain; if the magnitude of theta 5 is not the first 30%,
E. And (3) adopting a 1 multiplied by 1 convolutional neural network, and adjusting a simple feature matrix f ba to obtain a matrix f 2:
The matrix f ba is input into a 1×1 convolutional neural network to obtain a matrix f 2, that is, the data of the matrix f ba is used as the input data of the 1×1 convolutional neural network, and the output data of the 1×1 convolutional neural network is used as a matrix f 2;
F. The final attention feature f sign is calculated:
fsign＝f1·f2
G. The simple feature matrix f ba, the NSS feature matrix f nss and the attention feature f sign are combined by adopting a vector fusion method to obtain a quality evaluation feature matrix f q:
where α (i.e., the attention mechanism weight) is a non-negative parameter that controls the significance of the saliency map.
[ Effect verification ]
To verify the effect of the method, two artificial fuzzy data sets and one natural fuzzy data set are selected, and performance verification and comparison are performed on eight methods including the method of the application.
The adopted artificial data set is as follows: image and video factory laboratory dataset (Laboratory for Image and Video Engineering, LIVE) at university of texas and the updated tank-mine image dataset (TAMPERE IMAGE Database 2013, tid 2013) at 2013. The LIVE dataset contains 29 original images and 779 Zhang Shizhen images, 5 distortion types, wherein 175 JPEG compressed images, 169 JPEG2000 compressed images, 145 white noise images, 145 gaussian blur images, 145 fast fading images, each distortion type contains 4 or 5 distortion levels, and the image size is 768×512. The TID2013 dataset contains 25 original images and 3000 distorted images, 17 distortion types, each distortion type containing 4 levels, with an image size of 512 x 384. The adopted natural fuzzy data set is as follows: a natural distortion dataset of university of texas (LIVE IN THE WILD IMAGE quality challenge database, CLIVE). CLIVE contains 1162 blurred pictures taken in reality with a resolution of 500 x 500.
Two NR-IQA evaluation index evaluation method performances are selected: spearman rank correlation coefficients (Spearman's Rank Ordered Correlation Coefficient, SROCC) for measuring monotonicity of the method estimates and pearson linear correlation coefficients (Pearson Linear Correlation Coefficient, PLCC) for describing correlation between the method estimates and the true values, the closer these two indices are to 1, the better the performance of the method.
The training method adopts SGD with momentum, all weights of the network are initialized by a pre-training model, the learning rate is set to be 0.01, the learning rate is set to be 0.94, the weight attenuation coefficient is 0.0005, and the training times are 100 generations. Each dataset was randomly divided into two subsets, 80% of the images were used for training and 20% for testing, the result of each evaluation parameter was the average after repeating the five data random assignment process. The attention mechanism weights α are set to 0.3, respectively, to highlight the importance of the original features of the image, as shown in table 1.
Table 1 experimental results
The performance of CADnet methods combined with an attention mechanism module is not outstanding on the artificial fuzzy data set, and on the LIVE data set with less reference pictures and fuzzy pictures, SROCC is reduced by 3.11 percent, PLCC is reduced by 2.78 percent compared with the SFA method with the best performance, and the performance difference between the method and other NR-IQA methods based on a depth network is within 3 percent. The reason for this is that the CADnet backbone network is simpler and has shallower network depth, and although the correlation feature and the attention feature are introduced, the problem of poor descriptive capacity of the image simple feature matrix is made up to a certain extent, the performance of the image simple feature matrix for the simple artificial blurred image dataset still has some differences with the NR-IQA method using ResNet50 as the backbone network. On the natural fuzzy dataset cLIVE, CADnet achieves quite good results, and the performance of the method is basically the same as that of SGDnet and DB-CNN methods, but the former needs to provide a significant picture dataset in advance and cannot evaluate unknown images, and the latter uses a bilinear model to evaluate the image distortion type, so that the speed is low. Compared with other NR-IQA methods based on convolutional neural networks, CADnet has the advantages that the performance is improved by 14% at least and by more than 40% at most.
To verify the effect of the attention mechanism on the performance of the method, five sets of control experiments were performed with the attention mechanism weights of CADnet set to 0, 0.1, 0.2, 0.4, 0.5, respectively, and the results are shown in table 2.
From table 2 it can be seen that:
1) With the introduction of attention mechanisms, the performance of the method is remarkably enhanced.
2) On a LIVE artificial fuzzy data set with fewer images and simpler distortion condition, the method performance is slowly improved along with the increase of attention characteristic weight. When alpha is 0.5, the performance is improved by more than 2%.
3) On the more complex TID2013 dataset, with increasing attention feature weights, the method performance improved significantly, by about 15% when α was 0.5. On the natural blurring dataset cLIVE, the performance of the method is obviously improved along with the introduction of a attention mechanism, and when alpha is 0.3, the performance is improved by more than 55 percent. When the attention feature weight is too large, the importance of the image itself feature and quality prediction task is reduced, resulting in reduced performance of the method on the natural blurring dataset. It can be seen that the introduction of the attention mechanism can significantly improve the performance of the NR-IQA method.
TABLE 2 attention mechanism Effect verification
The above examples are provided for convenience of description of the present application and are not to be construed as limiting the application in any way, and any person skilled in the art will make partial changes or modifications to the application by using the disclosed technical content without departing from the technical features of the application.
Claims (7)
1. A no-reference image quality evaluation system, comprising: an image input system, a feature extraction system and a quality evaluation network system; the output end of the image input system is connected with the input end of the characteristic extraction system, and the output end of the characteristic extraction system is connected with the input end of the quality evaluation network system;
The image input system is used for selecting pictures;
The feature extraction system is used for solving an NSS feature matrix, a simple feature matrix and an attention mechanism feature and quality evaluation feature matrix of the image input by the image input system;
The quality evaluation network system is used for solving the image quality estimated value;
The feature extraction system includes: a convolutional neural network subsystem, an attention mechanism feature enhancement network subsystem;
The convolutional neural network subsystem comprises a deep convolutional neural network and a shallow convolutional neural network, wherein an input image extracts an NSS feature matrix f nss of the image through the deep convolutional neural network, extracts a simple feature matrix f ba of the image through the shallow convolutional neural network and scores vectors
The attention mechanism characteristic enhancement network subsystem solves a quality evaluation characteristic matrix f q through a score vector a and a simple characteristic matrix f ba of the convolutional neural network subsystem:
The output of the shallow convolutional neural network includes: simple feature matrix f ba and score vector of picture corresponding to each type of object Wherein L represents the number of feature vectors;
The simple feature matrix and the score vector are used as inputs of the attention mechanism feature enhancement network subsystem, and the outputs of the attention mechanism feature enhancement network subsystem are an attention feature f sign and a quality evaluation feature matrix f q:
the solving process of the attention mechanism characteristic enhancement network subsystem comprises the following steps:
A. Computing a suppression vector for the score vector
B. Calculating parameters
Will be
Inputting into a full-connection layer neural network, and solvingI.e./>As the input value of the full-connection layer neural network, the output result is/>
As the input value of the full-connection layer neural network, the output result is
C. Calculating the characteristic vector differenceTo be used for
And calculating a weight θ i for each of the feature differences e i in the feature vector differences:
e, inputting the data into an MLP multi-layer perceptron, wherein the output result of the MLP neural network is theta i;
D. F ba is screened according to the weight value theta i, and a characteristic matrix f 1 with strong significance is obtained:
f ba is a feature matrix made up of L feature vectors, Wherein each feature vector/>One-to-one correspondence with the score a i and the weight theta i in the score vector;
Sorting theta i according to the size, selecting the feature vector corresponding to the first 30% weight to be reserved as the feature with strong significance, changing the rest feature vectors into 0 vector, and obtaining a feature matrix f 1 with strong significance;
E. And (3) adopting a 1 multiplied by 1 convolutional neural network, and adjusting a simple feature matrix f ba to obtain a feature f 2:
The matrix f ba is input into a 1×1 convolutional neural network to obtain a matrix f 2, that is, the data of the matrix f ba is used as the input data of the 1×1 convolutional neural network, and the output data of the 1×1 convolutional neural network is used as a matrix f 2;
F. The final attention feature f sign is calculated:
fsign＝f1·f2
G. The simple feature matrix f ba, the NSS feature matrix f nss and the attention feature f sign are combined by adopting a vector fusion method to obtain a quality evaluation feature matrix f q:
where α refers to the attention mechanism weight and is a non-negative parameter.
2. The reference-free image quality evaluation system according to claim 1, wherein the image quality estimation value is obtained by the quality evaluation network using the quality evaluation feature matrix f q as an input value.
3. The reference-less image quality evaluation system according to claim 1, wherein α=0.3.
4. A no-reference image quality evaluation method, characterized by comprising the steps of:
s100, reading an image;
s200, extracting natural image statistical characteristics from the image acquired in the step S100:
the image obtained in the step S100 is used as input quantity, and NSS feature matrix is output through a deep convolutional neural network;
s300, solving a simple feature matrix and a score vector for the image acquired in the step S100:
The image acquired in step S100 is used as an input, and a shallow convolutional neural network SCNN is used, and output is as follows: simple feature matrix f ba and score vector of picture corresponding to each type of object Wherein L represents the number of feature vectors;
s400, solving a quality evaluation feature matrix;
s500, obtaining an image quality estimated value;
Wherein, step S400 includes:
S401, calculating a suppression vector of the score vector:
S402, calculating parameters
Wherein ω 3 represents the weight of the full connection layer, mean represents taking the mean of each term of the vector;
S403, calculating the characteristic vector difference To be used for
And calculating a weight θ i for each of the feature differences e i in the feature vector differences:
Conveying device
Inputting the data into an MLP multi-layer perceptron, wherein the output result of the MLP neural network is theta i;
S404, screening f ba according to the weight value theta i to obtain a feature matrix f 1 with strong significance:
f ba is a feature matrix made up of L feature vectors, Wherein each feature vector/>One-to-one correspondence with the score a i and the weight theta i in the score vector;
Sorting theta i according to the size, selecting the feature vector corresponding to the first 30% weight to be reserved as the feature with strong significance, changing the rest feature vectors into 0 vector, and obtaining a feature matrix f 1 with strong significance;
S405, a1 multiplied by 1 convolutional neural network is adopted to adjust a simple feature matrix f ba, and a feature f 2 is obtained:
The matrix f ba is input into a 1×1 convolutional neural network to obtain a matrix f 2, that is, the data of the matrix f ba is used as the input data of the 1×1 convolutional neural network, and the output data of the 1×1 convolutional neural network is used as a matrix f 2;
s406, calculating a final attention feature f sign:
fsign＝f1·f2
S407, combining the simple feature matrix f ba, the NSS feature matrix f nss and the attention feature f sign by adopting a vector fusion method to obtain a quality evaluation feature matrix f q:
where α refers to the attention mechanism weight and is a non-negative parameter.
5. The reference-free image quality evaluation method according to claim 4, wherein said step S400 comprises:
0.2≤α≤0.5。
6. The reference-free image quality evaluation method according to claim 5, wherein said step S400 comprises:
α＝0.3。
7. the reference-free image quality evaluation method according to claim 4, comprising the steps of: before step S100, the method further includes: all neural networks need to be trained through a data set.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111121862.6A CN113920394B (en) | 2021-09-24 | 2021-09-24 | No-reference image quality evaluation method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111121862.6A CN113920394B (en) | 2021-09-24 | 2021-09-24 | No-reference image quality evaluation method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113920394A CN113920394A (en) | 2022-01-11 |
| CN113920394B true CN113920394B (en) | 2024-06-21 |
Family
ID=79236231
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111121862.6A Active CN113920394B (en) | 2021-09-24 | 2021-09-24 | No-reference image quality evaluation method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113920394B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109919920A (en) * | 2019-02-25 | 2019-06-21 | 厦门大学 | The full reference of unified structure and non-reference picture quality appraisement method |
| CN112419242A (en) * | 2020-11-10 | 2021-02-26 | 西北大学 | No-reference image quality evaluation method based on self-attention mechanism GAN network |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110853032B (en) * | 2019-11-21 | 2022-11-01 | 北京航空航天大学 | Unmanned aerial vehicle video tag acquisition method based on multi-mode deep learning |
| CN113421237B (en) * | 2021-06-18 | 2023-04-18 | 西安电子科技大学 | No-reference image quality evaluation method based on depth feature transfer learning |
-
2021
- 2021-09-24 CN CN202111121862.6A patent/CN113920394B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109919920A (en) * | 2019-02-25 | 2019-06-21 | 厦门大学 | The full reference of unified structure and non-reference picture quality appraisement method |
| CN112419242A (en) * | 2020-11-10 | 2021-02-26 | 西北大学 | No-reference image quality evaluation method based on self-attention mechanism GAN network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113920394A (en) | 2022-01-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Gu et al. | No-reference quality assessment of screen content pictures | |
| CN108428227B (en) | No-reference image quality evaluation method based on full convolution neural network | |
| CN105208374B (en) | A kind of non-reference picture assessment method for encoding quality based on deep learning | |
| CN109325550B (en) | No-reference image quality evaluation method based on image entropy | |
| CN105787889B (en) | A kind of Fast Image Denoising based on non-local mean | |
| CN111126278B (en) | Method for optimizing and accelerating target detection model for few-class scene | |
| CN114612714B (en) | Curriculum learning-based reference-free image quality evaluation method | |
| CN111047543A (en) | Image enhancement method, device and storage medium | |
| CN109685772B (en) | No-reference stereo image quality evaluation method based on registration distortion representation | |
| CN110070539A (en) | Image quality evaluating method based on comentropy | |
| CN109978854A (en) | A kind of screen content image quality measure method based on edge and structure feature | |
| CN112767385A (en) | No-reference image quality evaluation method based on significance strategy and feature fusion | |
| CN109754390B (en) | No-reference image quality evaluation method based on mixed visual features | |
| CN105894507B (en) | Image quality evaluating method based on amount of image information natural scene statistical nature | |
| CN108830829B (en) | Non-reference quality evaluation algorithm combining multiple edge detection operators | |
| CN111079864A (en) | Short video classification method and system based on optimized video key frame extraction | |
| CN116309178A (en) | Visible light image denoising method based on self-adaptive attention mechanism network | |
| CN112348762A (en) | Single image rain removing method for generating confrontation network based on multi-scale fusion | |
| CN111047618A (en) | Multi-scale-based non-reference screen content image quality evaluation method | |
| CN107590804A (en) | Screen picture quality evaluating method based on channel characteristics and convolutional neural networks | |
| Morzelona | Human visual system quality assessment in the images using the IQA model integrated with automated machine learning model | |
| CN111723864A (en) | Method and device for performing countermeasure training by using internet pictures based on active learning | |
| CN113920394B (en) | No-reference image quality evaluation method and system | |
| CN114677670B (en) | Method for automatically identifying and positioning identity card tampering | |
| CN116542865A (en) | Multi-scale real-time defogging method and device based on structural re-parameterization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |