CN115222635A - No-reference image quality evaluation method and device based on image feature fusion and computer readable storage medium - Google Patents

Abstract

The invention belongs to the field of computer vision, and particularly relates to a no-reference image quality evaluation method and a no-reference image quality evaluation device based on image feature fusion and a computer readable storage medium, wherein the no-reference image quality evaluation method comprises the following steps of: acquiring a natural distortion image to be evaluated, generating a gradient image according to the natural distortion image, and inputting the natural distortion image and the gradient image into a trained non-reference image quality evaluation model based on image feature fusion to obtain a quality evaluation score; the no-reference image quality evaluation model based on image feature fusion comprises the following steps: the invention not only evaluates the image, but also fully considers the global semantic information and the local semantic information of the natural distortion image domain, thereby being capable of evaluating the quality of the natural distortion image more accurately.

Description

A non-reference image quality evaluation method, device and device based on image feature fusion computer readable storage medium

technical field

The invention belongs to the field of computer vision, and in particular relates to a reference-free image quality evaluation method, device and computer-readable storage medium based on image feature fusion.

Background technique:

No-reference image quality assessment (No-Reference Image Quality Assessment) refers to the quality assessment of itself only through the distortion map. Different from the full reference image quality assessment method, no reference image quality assessment does not require a reference image.

With the in-depth research of backbone network, the extraction methods of image features have been very rich, but how to apply the extracted features to image quality evaluation is still a challenging task, which is also one of the factors hindering the development of image quality evaluation.

The existing image quality evaluation technologies only use the feature information of the image itself to evaluate the quality of the image. For example, the patent CN201810759247.X mainly uses the generative adversarial network to construct the image training model, and then uses the high-definition lossless image as the training model. The data set is sent to the image training model for training and learning, and a non-reference image quality evaluation model with a fully trained identification network is obtained. Finally, the image to be evaluated is sent to the fully trained identification network, and the final evaluation result is obtained by scoring and weighting. This method Mainly based on the characteristics of the image itself, the quality of the image is evaluated without reference images, and the effect of other images on the quality evaluation of the original image is not paid attention to, and the global semantic information and local information of the image cannot be taken into account, resulting in poor image quality. evaluation is inaccurate.

SUMMARY OF THE INVENTION

In order to solve the problem that the prior art does not pay attention to the effect of other images on the quality evaluation of the original image, and cannot take into account the global semantic information and local information of the image, resulting in inaccurate evaluation of the image quality, the present invention proposes a method based on image feature fusion. A reference-free image quality evaluation method, device, and computer-readable storage medium are used for fusion according to the characteristics of the original image quality and the characteristics of the original image gradient map, so as to evaluate the quality of the original image, fully considering the global semantic information of the original image and Local information improves the accuracy of image quality assessment.

The present invention adopts the following technical solutions:

A reference-free image quality evaluation method based on image feature fusion, comprising the following steps:

Obtain the natural distortion image to be evaluated, generate a gradient image according to the natural distortion image, input the natural distortion image and the gradient image into the trained non-reference image quality evaluation model based on image feature fusion, and obtain the quality evaluation score; The no-reference image quality evaluation model of feature fusion includes: backbone network, cross-domain feature fusion model and cross-scale feature fusion model, and two linear regression layers;

The process of training a no-reference image quality assessment model based on image feature fusion includes:

S1: Obtain the naturally distorted image domain with ground truth labels, where the ground truth label represents the true score of the naturally distorted image in the naturally distorted image domain;

S2: Generate gradient image domain according to natural distortion image domain;

S3: Input the natural distortion image domain and its corresponding gradient image domain into the backbone network, and extract the hierarchical feature of the natural distortion image domain and the hierarchical feature of the gradient image domain;

S4: Integrate the hierarchical features of the natural distortion image domain and the hierarchical features of the gradient image domain corresponding to the input cross-domain feature fusion model, and calculate the cross-domain fusion hierarchical features of the natural distortion image domain;

S5: Input the cross-domain fusion hierarchical feature into the cross-scale feature fusion model for fusion, and calculate the cross-scale fusion feature of the natural distortion image domain;

S6: Input the cross-scale fusion features into two linear regression layers for regression processing, and calculate the quality evaluation score in the natural distortion image domain;

S7: Calculate the loss function of the no-reference image quality evaluation model based on image feature fusion according to the real label and the quality evaluation score in the natural distortion image domain;

S8: Continuously adjust the parameters of the model, and complete the training of the model when the loss function is less than the set threshold.

To achieve the above object, the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the above-mentioned image feature fusion-based image quality evaluation method without reference.

In order to achieve the above object, the present invention also provides a reference-free image quality evaluation device based on image feature fusion, comprising a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory for The computer program stored in the memory is executed, so that the apparatus for evaluating the quality of an image without reference based on image feature fusion executes the above-mentioned method for evaluating image quality without reference based on image feature fusion.

The present invention has at least the following beneficial effects:

The invention extracts features from the natural distorted image domain and the gradient image domain through the backbone network, and fuses the features between different levels of the gradient image domain and the natural distorted image domain through the cross-domain feature fusion model to obtain a feature with local semantic information. Cross-domain fusion of hierarchical features, through the cross-scale feature fusion module, the cross-domain fusion of hierarchical features can be used for cross-scale feature fusion, so as to obtain natural distorted image domain features with both global semantic information and local semantic information. The natural distorted image domain feature of local semantic information is subjected to regression processing, and the quality evaluation score of the natural distorted image domain is calculated, so that the obtained quality evaluation score can better reflect the quality information of the natural distorted image domain. Accurate image quality information is obtained under the premise of the image. Compared with the traditional non-reference image quality evaluation method, the present invention fully considers the gradient image domain as the auxiliary image domain of the natural distortion image domain. Obtaining more abundant prior information and local information is conducive to improving the accuracy of quality evaluation. By merging the features of different levels in the cross-domain fusion hierarchical features, the image features with global and local information can be obtained by cross-scale fusion. The image features with global and local information are linearly regressed to obtain the evaluation score of the image quality, which improves the accuracy of the image quality evaluation. The non-reference image quality evaluation method based on image feature fusion provided by the present invention can be widely used In the fields of artificial intelligence, image recognition, photography, target detection, etc., improve the quality evaluation ability of natural distorted images.

Description of drawings

Fig. 1: It is the block diagram of the implementation system of the present invention;

Fig. 2 is a flow chart of the model training of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and are based on the embodiments of the present invention. , other embodiments obtained by those of ordinary skill in the art without creative work, all belong to the protection scope of the present invention.

Referring to FIG. 1, the present invention proposes a reference-free image quality evaluation method based on image feature fusion, which specifically includes the following steps:

Obtain the natural distortion image to be evaluated, generate a gradient image according to the natural distortion image, input the natural distortion image and the gradient image into the trained non-reference image quality evaluation model based on image feature fusion, and obtain the quality evaluation score; The no-reference image quality evaluation model for feature fusion includes: a backbone network, a cross-domain feature fusion model and a cross-scale feature fusion model, and two linear regression layers, where the natural distortion image to be evaluated can be an image captured by any camera, or It is an image downloaded from the Internet, or an image obtained from other storage devices, or a static image or an image in a moving state, etc. The natural distortion image to be evaluated may include one or more objects, such as When taking a picture of a person, the environment around the person, such as cars, trees, etc., may be captured, and people, cars, electronics, trees, etc. are all objects included in the image to be processed.

Referring to Figure 2, the process of training a reference-free image quality evaluation model based on image feature fusion includes:

S1: Obtain the naturally distorted image domain with ground-truth labels, where the ground-truth labels represent the ground-truth scores of the naturally distorted images in the naturally distorted image domain.

Given that most traditional NR-IQA methods only use RGB images as input to the model, there is the question of whether it is beneficial to use multi-feature fusion or whether one image is sufficient, and the gradient of an image also plays a crucial role in many vision tasks , the image gradient sharply reflects the structural components of the image, such as image edges, and the gradient image can robustly reflect the details of the image structure under the changes of image intensity and color. Therefore, using the gradient map as data input makes it as a natural distortion The supplement of images and the assistance of natural distortion image feature extraction, this design assists the texture quality feature extraction of natural distortion images, and eases the difficulty of extracting features from a single natural distortion image. In the present invention, the natural distortion image domain mainly adopts LIVE -Challenge dataset, the LIVE-Challenge dataset contains 1169 naturally distorted images, no reference images; the image size is 500x 500, and the true label is the subjective evaluation score (MOS value) of the naturally distorted image in the natural distorted image domain, MOS's The size range is [0, 100], and it is tested by 8100 testers. The higher the score, the better the picture quality, where the natural distortion image domain is represented as an image set containing multiple natural distortion images.

S2: The gradient image domain is generated from the natural distortion image domain.

A specific embodiment of a reference-free image quality evaluation method based on image feature fusion, the specific implementation method for processing a natural distortion image into a gradient image includes:

表示自然失真图像I_Z在X方向的偏导；

表示自然失真图像I_Z在Y方向的偏导；

表示自然失真图像I_Z的总偏导；G(I_Z)表示自然失真图像I_Z的梯度幅度。where (x, y) are the pixel coordinates of the natural distortion image I _Z ,

Represents the partial derivative of the natural distortion image I _Z in the X direction;

Represents the partial derivative of the natural distorted image I _Z in the Y direction;

represents the total partial derivative of the naturally distorted image I _Z ; G(I _Z ) represents the gradient magnitude of the naturally distorted image I _Z.

S3: Input the natural distortion image domain and its corresponding gradient image domain into the backbone network, and extract the hierarchical features of the natural distortion image domain and the gradient image domain.

A specific embodiment of a non-reference image quality evaluation method based on image feature fusion, as shown in Figure 2:

Input the processed natural distortion image domain and gradient image domain into the Resnet50 backbone network, and extract L level features Stage1-L of the natural distortion image domain and the gradient image domain respectively. The square containing D in Figure 2 represents the natural distortion image domain. The L level features of (actually there are L level features, only one level is drawn), the square containing G represents the L level features of the gradient image domain (there are actually L level features, only one level is drawn), It should be noted that, in this embodiment, in order to reduce the amount of calculation, only four hierarchical features (L=4) extracted from the backbone network are selected, which does not mean that only four hierarchical features can be processed in the present invention. Therefore, the selection of 4-level features is because in the existing technology and this field, it is found through a large number of experimental studies that extracting 4-level features has the best cost performance and the best processing effect.

S4: The hierarchical features of the natural distortion image domain and the hierarchical features of the gradient image domain are correspondingly input into the cross-domain feature fusion model for fusion, and the cross-domain fusion hierarchical features of the natural distortion image domain are calculated.

A specific embodiment of a non-reference image quality evaluation method based on image feature fusion, as shown in Figure 2, the cross-domain feature fusion model (CDFM) includes: a fully connected layer, a self-attention mechanism layer, and a multi-layer perceptron MLP.

A specific embodiment of a non-reference image quality evaluation method based on image feature fusion, as shown in Figure 2, the calculation method of the cross-domain fusion hierarchical feature includes:

S41: Generate the query Q _1n , the key K _1n and the value V _1n from the features of the nth level of the natural distortion image domain through the fully connected layer, and process Q _1n , K _1n and V _1n according to the self-attention mechanism, and calculate the natural distortion the first global semantic level feature X _n of the nth level in the image domain;

S42: Generate the query Q _2n , the key K _2n and the value V _2n from the features of the nth level of the gradient image domain through the fully connected layer, and process Q _2n , K _2n and V _2n according to the self-attention mechanism, and calculate the gradient image domain The second global semantic level feature Y _n of the nth level;

S43: Process Q _1n , K _2n and V _2n according to the self-attention mechanism, and calculate the fusion feature Z _n of the nth level in the natural distortion image domain and the gradient image domain;

S44: Input X _n , Y _n and Z _n into the multi-layer perceptron MLP, and calculate the cross-domain fusion hierarchical feature F _n of the nth level in the natural distortion image domain, wherein, by combining the natural distortion image domain and the gradient image domain The features between different levels are fused, so that the obtained cross-domain fusion level features have rich prior information and local information, which is beneficial to improve the accuracy of quality evaluation.

S5: Input the cross-domain fusion hierarchical feature into the cross-scale feature fusion model for fusion, and calculate the cross-scale fusion feature of the natural distortion image domain.

A specific embodiment of a non-reference image quality evaluation method based on image feature fusion, as shown in Figure 2, the cross-scale feature fusion model includes: STB model (Swin-Transformer Block) and global average pooling layer (GAP) ;

A specific embodiment of a non-reference image quality evaluation method based on image feature fusion, as shown in Figure 2, the calculation method of the cross-scale fusion feature includes:

S51 : Input the cross-domain fusion hierarchical feature F _n of the nth level in the natural distortion image domain into the STB model to calculate the feature map A _n :

S52: splicing the cross-domain fusion hierarchical feature F _n+1 of the n+1 th level in the natural distortion image domain and the feature map A _n together with the Cat function and inputting the STB model to calculate the feature map A _n+1 ;

S53: Repeat step S52 to obtain feature map _AL ;

S54: Input the feature map _AL into the GAP layer to output cross-scale fusion features;

Among them, A _L represents the feature map of the L-th level in the natural distortion image domain, n=1, 2, 3...L-1, L represents the number of hierarchical features extracted by the backbone network. The features will be fused across scales to obtain rich global and local image information, which is conducive to improving the accuracy of quality evaluation. In this embodiment, only the four hierarchical features extracted from the backbone network are selected above, so L is 4 .

A specific embodiment of a reference-free image quality evaluation method based on image feature fusion, the specific calculation method of cross-scale fusion features includes:

The cross-domain fusion hierarchical features output by the cross-domain feature fusion model CDFM are sequentially sent to the Swin-TransformerBlock (STB). Window-based self-attention, and then fused through a LayerNorm layer and MLP layer; then sent to the ShiftWindowMulti-headSelfAttention (SW-MSA) module through the LayerNorm layer, followed by a LayerNorm layer and MLP layer, and finally a Conv The (convolution) operation performs feature scale reduction.

S6: Input the cross-scale fusion features into two linear regression layers (FC1 and FC2) for regression processing, and calculate the quality evaluation score in the natural distorted image domain.

S7: Calculate the loss function of the no-reference image quality evaluation model based on image feature fusion according to the real label and the quality evaluation score in the natural distortion image domain.

The loss function of the no-reference image quality evaluation model based on image feature fusion includes:

Among them, _N represents the number of naturally distorted images in the natural distortion image domain, _F ⁱ (ID , _IG ) represents the quality evaluation score of the ith natural distortion image domain, ID represents the natural distortion image domain, and _IG represents the gradient In the image domain, Q _i represents the ground-truth label of the i-th naturally distorted image, and loss represents the loss function.

S8: Constantly adjust the parameters of the model, and complete the training of the model when the loss function is less than a set threshold, wherein the threshold is set by those skilled in the art according to actual conditions.

In this design, ViT-B/8 is selected as the pre-training model. The model is trained on ImageNet-21k and fine-tuned on ImageNet 1k. The patch size P is set to 8. Following the standard training strategy of the existing IQA algorithm, the pre-trained model is The learning rate is set to 1×10-5, the batch Batch-size (B) is set to 32, and the model is optimized using the Adaptive Moment Estimation (ADAM) optimizer with a weight decay of 1×10-5 function and cosine annealing learning rate ( The CosineannealingLR) strategy optimizes the learning rate of the model.

In an embodiment of the present invention, the present invention further includes a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements any of the above-mentioned non-reference image quality evaluation methods based on image feature fusion. .

Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments may be completed by hardware related to computer programs. The aforementioned computer program may be stored in a computer-readable storage medium. When the program is executed, the steps including the above method embodiments are executed; and the foregoing storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

A reference-free image quality evaluation device based on image feature fusion, comprising a processor and a memory; the memory is used for storing a computer program; the processor is connected with the memory, and is used for executing the computer program stored in the memory, Any one of the above-mentioned non-reference image quality evaluation based on image feature fusion is performed by the apparatus for evaluating the quality of an image without reference based on image feature fusion.

Specifically, the memory includes various media that can store program codes, such as ROM, RAM, magnetic disk, U disk, memory card, or optical disk.

Preferably, the processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; it may also be a digital signal processor (Digital Signal Processor, for short) DSP), Application Specific Integrated Circuit (ASIC for short), Field Programmable Gate Array (FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

The method designed by the present invention can improve the accuracy and generalization ability of natural image quality evaluation. Specifically, the designed model can further utilize the local information and global semantic information of the image, so as to provide an image quality that is closer to the human visual system. Evaluation results to achieve quality evaluation tasks for various types of images without reference images, such as assisting natural distorted images for image restoration based on the quality evaluation scores of naturally distorted images; when applied to artificial intelligence, by quality Evaluation score, the robot judges the target; when it is used for taking pictures, it automatically adjusts parameters such as the resolution of the lens according to the quality evaluation score.

Furthermore, the above-mentioned figures are merely schematic illustrations of the processes included in the methods according to the exemplary embodiments of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the chronological order of these processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, for example, in multiple modules.

Other embodiments of the present disclosure will readily suggest themselves to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or techniques in the technical field not disclosed by the present disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the claims.

It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (8)

1. a non-referenced image quality evaluation method based on image feature fusion, is characterized in that, comprising: Obtain the natural distortion image to be evaluated, generate a gradient image according to the natural distortion image, input the natural distortion image and the gradient image into the trained non-reference image quality evaluation model based on image feature fusion, and obtain the quality evaluation score; The no-reference image quality evaluation model of feature fusion includes: backbone network, cross-domain feature fusion model and cross-scale feature fusion model, and two linear regression layers; The process of training a no-reference image quality assessment model based on image feature fusion includes: S1: Obtain the naturally distorted image domain with ground truth labels, where the ground truth label represents the true score of the naturally distorted image in the naturally distorted image domain; S2: Generate gradient image domain according to natural distortion image domain; S3: Input the natural distortion image domain and its corresponding gradient image domain into the backbone network, and extract the hierarchical feature of the natural distortion image domain and the hierarchical feature of the gradient image domain; S4: Integrate the hierarchical features of the natural distortion image domain and the hierarchical features of the gradient image domain corresponding to the input cross-domain feature fusion model, and calculate the cross-domain fusion hierarchical features of the natural distortion image domain; S5: Input the cross-domain fusion hierarchical feature into the cross-scale feature fusion model for fusion, and calculate the cross-scale fusion feature of the natural distortion image domain; S6: Input the cross-scale fusion features into two linear regression layers for regression processing, and calculate the quality evaluation score in the natural distortion image domain; S7: Calculate the loss function of the no-reference image quality evaluation model based on image feature fusion according to the real label and the quality evaluation score in the natural distortion image domain; S8: Continuously adjust the parameters of the model, and complete the training of the model when the loss function is less than the set threshold. 2. A reference-free image quality evaluation method based on image feature fusion according to claim 1, wherein the cross-domain feature fusion model comprises: a fully connected layer, a self-attention mechanism layer and a multi-layer perceptron MLP. 3. The non-reference image quality evaluation method based on image feature fusion according to claim 2, wherein the calculation method of the cross-domain fusion hierarchical feature comprises: S41: Generate the query Q _1n , the key K _1n and the value V _1n from the features of the nth level of the natural distortion image domain through the fully connected layer, and process Q _1n , K _1n and V _1n according to the self-attention mechanism, and calculate the natural distortion the first global semantic level feature X _n of the nth level in the image domain; S42: Generate the query Q _2n , the key K _2n and the value V _2n from the features of the nth level of the gradient image domain through the fully connected layer, and process Q _2n , K _2n and V _2n according to the self-attention mechanism, and calculate the gradient image domain The second global semantic level feature Y _n of the nth level; S43: Process Q _1n , K _2n and V _2n according to the self-attention mechanism, and calculate the fusion feature Z _n of the nth level in the natural distortion image domain and the gradient image domain; S44: Input X _n , Y _n and Z _n into the multi-layer perceptron MLP, and calculate the cross-domain fusion hierarchical feature F _n of the nth level in the natural distortion image domain. 4 . The reference-free image quality evaluation method based on image feature fusion according to claim 1 , wherein the cross-scale feature fusion model comprises: an STB model and a GAP layer. 5 . 5. a kind of non-reference image quality evaluation method based on image feature fusion according to claim 4, is characterized in that, the calculation method of described cross-scale fusion feature comprises: S51: Input the cross-domain fusion hierarchical feature F _n of the nth level in the natural distortion image domain into the STB model to calculate the feature map A _n ; S52: The cross-domain fusion hierarchical feature F _n+1 of the n+1th level in the natural distortion image domain and the feature map A _n are spliced together with the Cat function and input in the STB model to calculate the feature map A _n+1 ; S53: Repeat step S52 to obtain feature map _AL ; S54: Input the feature map _AL into the GAP layer to output cross-scale fusion features; Among them, A _L represents the feature map of the L-th layer in the natural distortion image domain, n=1, 2, 3...L-1, L represents the number of hierarchical features extracted by the backbone network. 6. a kind of non-reference image quality evaluation method based on image feature fusion according to claim 1, is characterized in that, the loss function of the non-reference image quality evaluation model based on image feature fusion comprises:

Among them, N represents the number of naturally distorted images, _F ⁱ (ID , _IG ) represents the quality evaluation score of the _i -th naturally distorted image, ID represents the natural distortion image domain, _{IG represents} the gradient image domain, and Qi represents the The ground-truth label of the i-th naturally distorted image, loss represents the loss function. 7. A computer-readable storage medium on which a computer program is stored, wherein the computer program is executed by a processor to realize a kind of image feature fusion-based image feature fusion according to any one of claims 1 to 7. No reference image quality assessment method. 8. A reference-free image quality evaluation device based on image feature fusion, comprising a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory, and is used to execute the A computer program stored in the memory, so that the apparatus for evaluating the quality of an image without reference based on image feature fusion executes the method for evaluating the quality of an image without reference based on image feature fusion according to any one of claims 1 to 7.

Images