CN110728656A - Meta-learning-based no-reference image quality data processing method and intelligent terminal - Google Patents
Meta-learning-based no-reference image quality data processing method and intelligent terminal Download PDFInfo
- Publication number
- CN110728656A CN110728656A CN201910844666.8A CN201910844666A CN110728656A CN 110728656 A CN110728656 A CN 110728656A CN 201910844666 A CN201910844666 A CN 201910844666A CN 110728656 A CN110728656 A CN 110728656A
- Authority
- CN
- China
- Prior art keywords
- image
- quality evaluation
- training
- model
- distortion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30168—Image quality inspection
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Quality & Reliability (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention belongs to the technical field of image processing and computer vision, and discloses a no-reference image quality data processing method based on meta-learning and an intelligent terminal, wherein the image quality in an image database simulating distortion is divided into different tasks according to distortion types, and training data of each task is divided into a support set and a query set; performing a two-stage gradient optimization mode from support set data to query set data on the constructed deep convolutional neural network by using each task to obtain a prior model of image quality evaluation; dividing an image database of real distortion into a training set and a test set, and carrying out fine tuning training on the established prior model by using training set data to obtain a quality evaluation model of the real distortion image; and for the images to be tested in the test set, automatically realizing the non-reference quality evaluation of the images by utilizing the established quality evaluation model of the real distorted images. The method has better performance than the existing mainstream non-reference image quality evaluation method.
Description
Technical Field
The invention belongs to the technical field of image processing and computer vision, and particularly relates to a no-reference image quality data processing method based on meta-learning and an intelligent terminal.
Background
Currently, the closest prior art: with the development of mobile internet and information technology, image processing technology is widely applied to various fields. However, since images are acquired, compressed, transmitted and stored with varying degrees of distortion. There is therefore an urgent need for an efficient and reliable method for assessing the quality of these images. Currently, Image Quality Assessment (IQA) has become an important aspect of various computer vision and image processing applications, such as image acquisition, transmission, restoration and enhancement, image search and retrieval, and image recognition. The image quality evaluation includes subjective and objective methods, and the subjective evaluation method is that an observer scores an observed image according to certain standards and experiences to obtain a subjective quality evaluation score, and a mean opinion value (MOS) and a differential mean opinion value (DMOS) are often used. Since the human eye is the ultimate recipient of the image, the subjective evaluation method best fits the true quality of the image. However, the subjective evaluation method is time-consuming and labor-consuming, and is easily affected by subjective factors and experimental environments, and in comparison, the objective image quality evaluation method has the characteristics of simplicity, real-time performance, repeatability, easiness in integration and the like, so that the research of the objective evaluation method conforming to the subjective visual system (HVS) is the key point of image quality evaluation.
Objective image quality evaluation is generally classified into three types, i.e., full reference type, partial reference type, and no-reference type, according to the degree of dependence on an original image. The full-reference evaluation method is to calculate the perception errors between the original image and the distorted image by using all information of the original image, and synthesize the errors to obtain the quality evaluation score of the distorted image. Although the evaluation accuracy of the full-reference evaluation system is relatively high, the full-reference evaluation system requires all information of the reference image, which is difficult in many cases. However, if the partial feature information of the reference image is obtained, in this case, a partial reference quality evaluation method may be used. The no-reference quality evaluation method is also called blind evaluation, namely, the quality evaluation is carried out completely depending on the information of the image to be evaluated. It is relatively difficult to implement since it does not require any information of the original image, but it has attracted much attention from researchers due to its utility in the field of application.
Currently, the no-reference image quality evaluation methods can be further divided into two categories: the distortion-specific quality evaluation method and the distortion-unspecific general-purpose quality evaluation method depend on whether or not the type of distortion is known. In real life, if the quality of an image is to be evaluated, its distortion type is often unknown. Therefore, general-purpose image quality evaluation is becoming a hotspot of objective quality evaluation of digital images, and has been developed to some extent. The method generally comprises the steps of training by a machine learning method or a natural scene statistical method by utilizing relevant characteristics of images to obtain a quality evaluation model, then using the quality evaluation model as prior knowledge, and finally performing quality scoring on the images to be evaluated by utilizing the prior knowledge model. The main idea of the natural scene-based statistical method is that natural images show certain statistical rules, and the rules change due to image distortion, so that the quality scores of the images can be obtained by extracting natural statistical features of the images and calculating the deviation degree of statistical data. Mittal et al, in the paper "No-reference image quality assessment in the spatial domain", proposed a rapid No-reference image quality assessment method BRISQUE based on spatial statistical properties, which extracts features in two scale spaces, 18 statistical features (3 scales with 3 orientations with 2 parameters) in each scale space, and then performs quality assessment on the image using a two-level classification/regression framework. The method based on machine learning mainly extracts the low-level features of the image related to distortion and carries out modeling in a training mode, and the image quality is automatically evaluated by utilizing the established model. However, the features extracted by the above method are difficult to comprehensively describe various distortion characteristics of the image.
In recent years, the rapid development of no-reference image quality evaluation is promoted by the wide application of the deep convolutional neural network, and a multitask End-to-End optimization deep neural network (MEON) is proposed in a paper "End-to-End thin image quality assessment using deep neural networks" by Ma et al to realize the image quality evaluation. The MEON consists of two sub-networks, a distortion recognition network and a quality prediction network, both sharing part of the convolutional layer. Unlike conventional methods for training multitask networks, the training process herein is performed in two steps, the first step training a sub-network that identifies the type of distortion; in a second step, the quality prediction sub-network is trained using a variant of the stochastic gradient descent method, from the output of the pre-trained shared layer and the first sub-network as initialization. Then, the method has the following defects: (1) for real-world images, it is generally difficult to limit to a particular type of distortion; (2) the deep neural network needs a large amount of data to train so as to achieve a good effect, but for most quality evaluation image databases, the number of sample images is far from enough, so that an image quality evaluation depth model obtained based on small sample training often generates an overfitting phenomenon, and the generalization ability is not strong.
In summary, the problems of the prior art are as follows: at present, in image quality evaluation, training samples are few, and the generalization capability of a model to images with different distortion types is poor.
The difficulty of solving the technical problems is as follows: the difficulty of the current non-reference image quality evaluation is that the number of available training samples is small and the generalization capability of a training model to images with different distortion types is poor. The method utilizes the superior performance of a meta-learning method in processing small sample problems, utilizes the quality evaluation tasks of different distortion type images to train a universal quality evaluation prior model, and learns that the generalization capability of the prior model to the unknown distortion type images is strong.
The significance of solving the technical problems is as follows: the quality evaluation method for a single distorted image is researched more, but in practical situations, the distortion type of the image is difficult to obtain, so that the quality evaluation of the image is difficult to effectively carry out by using the existing quality evaluation method, and therefore a quality evaluation model with strong generalization capability on any distorted image needs to be constructed, and the quality evaluation method can be quickly applied to the quality evaluation of unknown distorted images in real life.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a no-reference image quality data processing method based on meta-learning and an intelligent terminal.
The invention is realized in such a way that a no-reference image quality data processing method based on meta-learning comprises the following steps:
the method comprises the following steps that firstly, an image quality evaluation data set containing a large amount of simulation distortion is divided into different tasks according to distortion types;
secondly, dividing a data set of the quality evaluation task of each distortion type into a support set and a query set, and preprocessing image data;
thirdly, constructing a deep convolutional neural network as an image quality evaluation model to be trained;
inputting the preprocessed image data and the quality scores into a constructed network model for meta-learning mode training based on gradient optimization to obtain a quality evaluation prior model;
fifthly, dividing a real distorted image quality evaluation database into a training set and a testing set, and performing fine tuning training on the obtained quality evaluation prior model by utilizing the preprocessed training image and the corresponding quality score to obtain a quality evaluation model conforming to the real distorted image;
and sixthly, automatically evaluating the quality of the real distorted image by using the established image quality evaluation model for the image to be tested in the test set.
Further, the dividing of the image quality evaluation dataset, which further includes a large number of simulated distortions, into different task method slabs according to the distortion type specifically includes:
(1) collecting an image data set containing a large amount of simulation distortion, acquiring image data and corresponding quality scores, and normalizing all the quality scores to [0,1 ];
(2) with the type of distortion as a research goal, independent tasks are seen for image quality evaluation of each simulated distortion.
Further, the data set of the quality evaluation task of each distortion type in the second step is divided into a support set and a query set, and the image data is preprocessed by the following method:
(1) dividing the image quality evaluation data set of each distortion type into a support set and a query set according to the proportion of 8: 2;
(2) scaling the size of the image to a predetermined size, the predetermined size being consistent with the required input size of the constructed deep convolutional neural network;
(3) and carrying out normalization operation on the image data, counting the mean value of the sample images in the training data, and carrying out mean value removing operation on all the sample images to obtain the preprocessed images.
Further, the third step constructs a deep convolutional neural network as a block of an image quality evaluation model to be trained:
(1) the constructed deep convolutional neural network consists of a basic network model, two full-connection layers and an output layer;
(2) wherein the foundation network model removes the inclusion-V3 convolution network part of the full connection layer;
(3) the two full connection layers respectively consist of 1024 nodes and 512 nodes;
(4) the output layer is a prediction result of image quality evaluation, and a Sigmoid activation function is used as an activation function of the output layer.
Further, the fourth step inputs the preprocessed image data and the quality scores into the constructed network model for meta-learning mode training based on gradient optimization, so as to obtain a quality evaluation prior model method plate:
(1) taking the image quality evaluation task of each distortion type as a training target, and respectively inputting the image data of the support set and the image data of the query set into the constructed network model for prediction, wherein the parameters of the network model come from a pre-training network;
(2) the meta-learning training mode based on gradient optimization is a two-stage gradient optimization method, firstly, image data of a support set in each distortion type is used for updating network model parameters for one time, and then, the updated network model is used for carrying out secondary gradient updating on the image data of a query set;
(3) continuously training parameters of the network model by using a large number of image quality evaluation tasks of different distortion types to obtain a quality evaluation prior model;
(4) the network model training adopts a random gradient descent method SGD to carry out parameter optimization, cross entropy is used as a loss function, and a calculation formula is as follows:
wherein, ynAnd
respectively representing the real result and the prediction result of the image quality fraction, wherein N is the number of the trained images; training network model parameters by a gradient optimization method until a calculated loss function result is smaller than a threshold value, and finally obtaining a quality evaluation prior model of the image;
further, the image quality evaluation database of the fifth true distortion is divided into a training set and a test set, and the obtained quality evaluation prior model is subjected to fine tuning training by using the preprocessed training image and the corresponding quality score to obtain a quality evaluation model method block which accords with the true distortion image:
(1) randomly selecting 80% of images in a real distorted image database as a training set, using the rest 20% of images as a test set, normalizing the quality scores of the images to [0,1], and scaling the size of the images to a preset size, wherein the preset size is consistent with the input size required by the constructed deep convolutional neural network; carrying out normalization operation on image data, firstly carrying out statistics on the average value of sample images in training data, and then carrying out mean value removing operation on all sample images to obtain a preprocessed image;
(2) performing fine tuning training on the obtained quality evaluation prior model by using the image data and the corresponding quality fraction to obtain a quality evaluation model which accords with a real distorted image;
(3) the process of fine tuning training adopts a random gradient descent method SGD to carry out parameter optimization, cross entropy is used as a loss function, and a calculation formula is as follows:
wherein, ymAnd
respectively representing the real result and the prediction result of the real distorted image quality score, and M represents the number of images for fine tuning training. And carrying out fine tuning training on the network model parameters by a gradient optimization method until the calculated loss function result is smaller than a threshold value, and finally obtaining a quality evaluation model which accords with a real distorted image.
Another object of the present invention is to provide an intelligent terminal applying the meta-learning based no-reference image quality data processing method.
In summary, the advantages and positive effects of the invention are: the invention utilizes the prior model to learn in the image quality evaluation tasks of different distortion types of the meta-learning idea, and can effectively learn the prior knowledge of the image distortion; in the meta-learning training process, a two-stage gradient optimization method from a support set to a query set is adopted, so that the network model can effectively learn the adaptive capacity from a training sample to a test sample, and the quality evaluation can be quickly and accurately realized by finely adjusting a prior model through a small number of samples in the face of unknown distorted images.
Compared with the prior art, the invention has the following advantages:
1. different from the existing non-reference quality evaluation method, the method utilizes a meta-learning method to learn the common prior knowledge of different distortion type images on quality evaluation, the quality evaluation of each distortion type image is regarded as an independent task, and each task is divided into a support set query set; and training network parameters by using a two-stage gradient optimization method based on meta learning to obtain the fast learning capability of the model, and the learned prior knowledge model can be quickly adapted to the quality evaluation of an unknown distorted image.
2. The method provided by the invention is a non-reference image quality evaluation method with strong expansibility, can be suitable for any depth regression network, and can obtain good generalization performance of non-reference image quality evaluation only by explicitly training model parameters through two-stage gradient optimization strategies.
3. Aiming at the learning characteristics of small samples of image quality evaluation, the method solves the problem of few training samples of image quality evaluation by using a few-sample element learning strategy; the image quality evaluation prior model is learned through quality evaluation tasks of a large number of different distorted images, and experimental results on a real distorted image database prove that the performance of the method is better than that of the current mainstream non-reference image quality evaluation method.
Drawings
Fig. 1 is a flowchart of a method for processing reference-free image quality data based on meta learning according to an embodiment of the present invention.
Fig. 2 is a flowchart of an implementation of a meta-learning based no-reference image quality data processing method according to an embodiment of the present invention.
Fig. 3 is a network structure diagram of a meta-learning based non-reference image quality evaluation method according to an embodiment of the present invention.
FIG. 4 is a schematic diagram of a set of true distorted images in a LIVE-CH image library according to an embodiment of the present invention; the subjective average scores of the images were: (a) MOS 66.36; (b) MOS 44.69; (c) 39.23; (d) MOS is 9.23; MOS (mean Opinion score) is a subjective quality score and is used for subjective quality evaluation of the image, and the image quality is better when the MOS value of the image is larger; the smaller the MOS value, the worse the image quality.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In view of the problems in the prior art, the present invention provides a method for processing non-reference image quality data based on meta-learning and an intelligent terminal, and the present invention is described in detail below with reference to the accompanying drawings.
As shown in fig. 1, the method for processing reference-free image quality data based on meta learning according to the embodiment of the present invention includes the following steps:
s101: dividing image quality in an image database simulating distortion into different tasks according to distortion types, and dividing training data of each task into a support set and a query set;
s102: performing a two-stage gradient optimization mode from support set data to query set data on the constructed deep convolutional neural network by using each task to obtain a prior model of image quality evaluation;
s103: dividing an image database of real distortion into a training set and a test set, and carrying out fine tuning training on the established prior model by using training set data to obtain a quality evaluation model of the real distortion image;
s104: and for the images to be tested in the test set, automatically realizing the non-reference quality evaluation of the images by utilizing the established quality evaluation model of the real distorted images.
The technical solution of the present invention is further described below with reference to the accompanying drawings.
Aiming at the small sample characteristics of the image quality evaluation method, the invention designs a no-reference image quality evaluation method based on meta-learning through a meta-learning thought. The invention aims to solve the problem of weak generalization capability between image quality evaluations of different distortion types, and in order to solve the problem, firstly, the image quality evaluations of different distortion types are regarded as an independent task, and training data of each task is divided into a support set and a query set; then, performing two-stage gradient optimization on the deep convolutional neural network from support set data to query set data by using a large number of quality evaluation tasks of different distorted images to obtain a quality evaluation prior model; and finally, fine tuning the obtained quality evaluation model by using the real image with unknown distortion to obtain the quality evaluation model which accords with the real distortion image. The specific implementation method is shown in fig. 2, and the embodiment includes a meta-training data set and preprocessing module, a quality evaluation prior model training module, a meta-testing data set and preprocessing module, and a true distorted image quality evaluation module. The meta-training data set and preprocessing module comprises parts of constructing a large number of image quality evaluation tasks of different distortion types from a simulation distortion database, dividing an image data set into a support set and a query set, preprocessing an image and the like; the quality evaluation prior model training module comprises a two-stage gradient optimization part for constructing a deep convolutional neural network and a deep network model based on meta learning; the meta-test data set and preprocessing module comprises a training set and a test set which are used for dividing a real distortion database and an image preprocessing component; the real distorted image quality evaluation module is used for performing fine tuning training on the quality evaluation prior model by utilizing a small amount of real distorted training images to obtain a quality evaluation model of the real distorted image.
1. Meta-training data set and pre-processing module
Constructing a large number of image quality evaluation tasks of different distortion types from a simulation distortion database: the simulation distortion database used by the method is an existing image quality evaluation data set TID2013, wherein the simulation distortion database comprises 24 single-distortion images, the number of the images of each distortion type is 125, and the total number of the images is 3000. The quality score of each image is Mean Opinion Score (MOS), the score range is 0 to 9, and for calculation convenience, the image quality score is normalized to be 0-1. In order to learn the common characteristics of images with different distortion types in the aspect of quality evaluation, the learned model has better generalization performance. The invention considers the image quality evaluation task of each distortion type as an independent task, namely considers the quality evaluation of 24 distortion images as 24 tasks to construct a meta-training data set.
The image dataset is divided into a support set and a query set: because a meta-training data set containing a plurality of image quality evaluation tasks is constructed, image data of each distortion type is divided into a support set and a query set according to the proportion of 8:2 by using the thought of meta-learning, the support set mainly has the function of network model optimization in the training process, and the query set mainly has the function of verifying whether a network model optimized by using the support set can be better applied to untrained data to perform secondary correction on model parameters trained by using the support set.
Image preprocessing: the image preprocessing mainly comprises size normalization of the image and image de-averaging operation. Since the input size of the deep convolutional neural network is fixed, scaling operations must be performed on sample images of different sizes, the present invention first scales all sample images to 448 × 448 × 3 size, and then randomly crops the scaled image according to 299 × 299 × 3 size to adapt to the network input size, where 3 represents 3 color channels of a color image, i.e., three color channels of RGB; the convolutional neural network training model is utilized, mean value removing operation needs to be carried out on training data, so that the image data of training can be guaranteed to be distributed near the mean value, and the specific process is as follows: firstly, the mean value of sample images in training data is counted, and then the mean value removing operation is carried out on all the sample images to obtain preprocessed images.
2. Quality evaluation prior model training module
Constructing a deep convolutional neural network: the deep convolutional neural network in fig. 3 is a network structure model used in the present invention, in which the convolutional network part is an inclusion-V3 network with a full link layer removed, and a full link layer containing 1024 nodes is generated by using a Global Average Pooling (GAP) operation, and then two full link layers and an output layer are constructed after the full link layer; the two fully-connected layers are respectively composed of 1024 nodes and 512 nodes, and the output layer is the predicted image quality score. In order to achieve the effect of more rapid and stable training, a BN layer and a Dropout layer are added after each full connection layer, and finally, a Sigmoid activation function is used as an activation function of an output layer in order to enable the prediction score to be between [0,1 ].
The deep network model is based on two-stage gradient optimization of meta-learning: because the constructed meta-training data set comprises a plurality of image quality evaluation tasks which basically belong to the problem of small sample learning, the invention effectively processes the idea of small sample learning by utilizing a large number of learning tasks by means of meta-learning and obtains the quality evaluation prior model by performing network model training based on two-stage gradient optimization on 24 image quality evaluation tasks in the meta-training set. The model training process comprises the following steps:
(1) randomly selecting one of distortion type image training data, inputting the preprocessed support set image data into a deep convolutional neural network to obtain a predicted image quality score, and performing back propagation calculation and gradient updating on network parameters by using a Euclidean distance between the two as a loss function in order to keep the predicted quality score of a network model consistent with a real quality score;
(2) in order to verify whether the network model trained by using the image data of the support set can effectively evaluate the quality of the unknown image data, the invention uses the network model updated in the query set image data input step (1) to perform secondary gradient calculation updating to correct network parameters, and the two-stage gradient optimization method can effectively use a small amount of existing training samples (support sets) to learn the model and has the capability of rapidly adapting to the unknown samples (query sets).
(3) And then randomly selecting image data of another distortion type, and repeating the steps (1) and (2) to train the network model until the image data of each distortion type in the meta-training data set trains the network model 50 times.
The network model training adopts a random gradient descent method SGD to carry out parameter optimization, cross entropy is used as a loss function, and a calculation formula is as follows:
wherein, ynAnd
respectively representing the real result and the prediction result of the quality fraction of the image, and N is the number of training images. Training the network model parameters by a gradient optimization method,and finally obtaining the image quality evaluation prior model of the image until the calculated loss function result is less than 0.0001. The acquired prior model is optimized by using a large number of image quality evaluation tasks of different distortion types, and prior knowledge of different distortion images on quality evaluation is acquired, so that the method can be quickly adapted to the quality evaluation of unknown distortion images.
3. Meta-test data set and pre-processing module
Dividing a real distortion database into a training set and a testing set: the real distortion database used by the method is an existing image quality evaluation data set LIVE-CH, which comprises 1162 real distortion images. The quality score of each image is Mean Opinion Score (MOS), the score range is between 0 and 100, and for calculation convenience, the image quality score is normalized to be between 0 and 1. In order to enable the quality evaluation prior model to well evaluate the quality of a real distorted image, the LIVE-CH database is divided into a training set and a test set according to the proportion of 8:2, the image data of the training set is used for carrying out fine tuning training on the prior model, and the test set is used for verifying the performance of the model.
Image preprocessing: the image preprocessing mainly comprises size normalization of the image and image de-averaging operation. Since the input size of the deep convolutional neural network is fixed, scaling operations must be performed on sample images of different sizes, the present invention first scales all sample images to 448 × 448 × 3 size, and then randomly crops the scaled image according to 299 × 299 × 3 size to adapt to the network input size, where 3 represents 3 color channels of a color image, i.e., three color channels of RGB; the convolutional neural network training model is utilized, mean value removing operation needs to be carried out on training data, so that the image data of training can be guaranteed to be distributed near the mean value, and the specific process is as follows: firstly, the mean value of sample images in training data is counted, and then the mean value removing operation is carried out on all the sample images to obtain preprocessed images.
4. True distortion image quality evaluation module
According to the quality evaluation prior model obtained by training in the step 2, the quality evaluation model conforming to the real distorted image can be obtained by performing fine tuning training on the model parameters by using the training data of the real distorted image. The fine tuning training process adopts a random gradient descent method SGD to carry out parameter optimization, cross entropy is used as a loss function, and a calculation formula is as follows:
wherein, ymAnd
respectively, a real result and a prediction result of the image quality score, and M is the number of training samples of the real distorted image. And carrying out fine tuning training on the network model parameters by a gradient optimization method until the calculated loss function result is less than 0.0001, and finally obtaining a no-reference quality evaluation model which accords with a real distorted image.
And finally, for the test sample image of the test set, automatically evaluating the quality of the test image by calling the image quality evaluation model, and outputting a quality score.
The technical effects of the present invention will be described in detail below in conjunction with performance tests and experimental analysis.
In order to prove the effect of the invention, the quality evaluation is carried out on the images with different distortion degrees, and the method is compared with other image quality evaluation methods without reference.
In order to verify the correctness of the invention, four distorted images are selected from a LIVE-CH image database for verification. Fig. 4 shows a distorted image used in an experiment, and the subjective quality score DMOS and the quality score S (range 0 to 100) calculated by the method of the present invention are respectively: (a) MOS 66.36 and S73.51. (b) MOS 44.69 and S55.21. (c) MOS 39.23, S29.87. (d) MOS 9.23 and S16.43. The experimental test results show that the consistency of the result obtained in the image quality evaluation and the subjective quality evaluation result is better, and the image quality can be evaluated more accurately.
In order to verify the overall performance of the method provided by the invention, the method of the embodiment and ten methods such as BLIINDS-II, BRISQE, ILNIQE, CORNIA, HOSA, BIECON, WaDIQaM-NR, NEON, DIQA and NSSADNN are compared on the LIVE-CH database for non-reference image quality evaluation performance. The invention uses Pearson Correlation Coefficient (PLCC) and Spearman Correlation Coefficient (SROCC) to measure the performance of image quality evaluation, the PLCC is used for quantitatively measuring the consistency of the quality score prediction result and the real result, the SROCC is used for quantitatively measuring the sequencing Correlation of the quality score prediction result and the real result, and the prediction performance of the method is better when the PLCC/SROCC value is larger.
Table 1 shows the comparison of the image quality evaluation performance of the method of the present invention with other 10 methods. As can be seen from the table, the method of the present invention has obvious advantages compared with the existing no-reference image quality evaluation method, i.e. the PLCC/SRCC value is obviously higher than other methods, which shows that the present invention has good prediction performance on no-reference image quality evaluation.
Table 1: no reference image quality evaluation performance comparison
| Method of producing a composite material | PLCC | SROCC |
| BLIINDS-II | 0.507 | 0.463 |
| BRISQUE | 0.629 | 0.607 |
| ILNIQE | 0.589 | 0.594 |
| CORNIA | 0.671 | 0.618 |
| HOSA | 0.678 | 0.659 |
| BIECON | 0.613 | 0.595 |
| WaDIQaM-NR | 0.680 | 0.671 |
| NEON | 0.693 | 0.688 |
| DIQA | 0.704 | 0.703 |
| NSSADNN | 0.813 | 0.745 |
| The method of the invention | 0.839 | 0.802 |
In conclusion, the quality evaluation prior model based on the two-stage gradient optimization has good generalization performance, can effectively capture prior knowledge of images of different distortion types on quality evaluation, and has better quality evaluation performance compared with the prior method.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.
Claims (7)
1. A method for processing non-reference image quality data based on meta-learning is characterized in that the method for processing non-reference image quality data based on meta-learning comprises the following steps:
the method comprises the following steps that firstly, an image quality evaluation data set containing a large amount of simulation distortion is divided into different tasks according to distortion types;
secondly, dividing a data set of the quality evaluation task of each distortion type into a support set and a query set, and preprocessing image data;
thirdly, constructing a deep convolutional neural network as an image quality evaluation model to be trained;
inputting the preprocessed image data and the quality scores into a constructed network model for meta-learning mode training based on gradient optimization to obtain a quality evaluation prior model;
fifthly, dividing a real distorted image quality evaluation database into a training set and a testing set, and performing fine tuning training on the obtained quality evaluation prior model by utilizing the preprocessed training image and the corresponding quality score to obtain a quality evaluation model conforming to the real distorted image;
and sixthly, automatically evaluating the quality of the real distorted image by using the established image quality evaluation model for the image to be tested in the test set.
2. The meta-learning based no-reference image quality data processing method according to claim 1, wherein the first step of partitioning the image quality assessment data set containing a large number of simulated distortions into different task method slabs according to distortion types specifically comprises:
(1) collecting an image data set containing a large amount of simulation distortion, acquiring image data and corresponding quality scores, and normalizing all the quality scores to [0,1 ];
(2) with the type of distortion as a research goal, independent tasks are seen for image quality evaluation of each simulated distortion.
3. The meta-learning based no-reference image quality data processing method according to claim 1, wherein the data set of the quality evaluation task of each distortion type of the second step is divided into a support set and a query set, and the image data is preprocessed as follows:
(1) dividing the image quality evaluation data set of each distortion type into a support set and a query set according to the proportion of 8: 2;
(2) scaling the size of the image to a predetermined size, the predetermined size being consistent with the required input size of the constructed deep convolutional neural network;
(3) and carrying out normalization operation on the image data, counting the mean value of the sample images in the training data, and carrying out mean value removing operation on all the sample images to obtain the preprocessed images.
4. The meta-learning based no-reference image quality data processing method according to claim 1, wherein the third step constructs a deep convolutional neural network as an image quality evaluation model method block to be trained:
(1) the constructed deep convolutional neural network consists of a basic network model, two full-connection layers and an output layer;
(2) wherein the foundation network model removes the inclusion-V3 convolution network part of the full connection layer;
(3) the two full connection layers respectively consist of 1024 nodes and 512 nodes;
(4) the output layer is a prediction result of image quality evaluation, and a Sigmoid activation function is used as an activation function of the output layer.
5. The method for processing quality data of non-reference images based on meta-learning according to claim 1, wherein the fourth step inputs the preprocessed image data and the quality scores into the constructed network model for training in a meta-learning manner based on gradient optimization, so as to obtain a quality evaluation prior model method plate:
(1) taking the image quality evaluation task of each distortion type as a training target, and respectively inputting the image data of the support set and the image data of the query set into the constructed network model for prediction, wherein the parameters of the network model come from a pre-training network;
(2) the meta-learning training mode based on gradient optimization is a two-stage gradient optimization method, firstly, image data of a support set in each distortion type is used for updating network model parameters for one time, and then, the updated network model is used for carrying out secondary gradient updating on the image data of a query set;
(3) continuously training parameters of the network model by using a large number of image quality evaluation tasks of different distortion types to obtain a quality evaluation prior model;
(4) the network model training adopts a random gradient descent method SGD to carry out parameter optimization, cross entropy is used as a loss function, and a calculation formula is as follows:
wherein, ynAnd
respectively representing the real result and the prediction result of the image quality fraction, wherein N is the number of the trained images; network pair through gradient optimization methodAnd training the model parameters until the calculated loss function result is smaller than a threshold value, and finally obtaining the quality evaluation prior model of the image.
6. The meta-learning based no-reference image quality data processing method of claim 1, wherein the image quality evaluation database of the fifth true distortion is divided into a training set and a test set, and the obtained quality evaluation prior model is subjected to fine tuning training by using a preprocessed training image and a corresponding quality score to obtain a quality evaluation model method block conforming to a true distortion image:
(1) randomly selecting 80% of images in a real distorted image database as a training set, using the rest 20% of images as a test set, normalizing the quality scores of the images to [0,1], and scaling the size of the images to a preset size, wherein the preset size is consistent with the input size required by the constructed deep convolutional neural network; carrying out normalization operation on image data, firstly carrying out statistics on the average value of sample images in training data, and then carrying out mean value removing operation on all sample images to obtain a preprocessed image;
(2) performing fine tuning training on the obtained quality evaluation prior model by using the image data and the corresponding quality fraction to obtain a quality evaluation model which accords with a real distorted image;
(3) the process of fine tuning training adopts a random gradient descent method SGD to carry out parameter optimization, cross entropy is used as a loss function, and a calculation formula is as follows:
wherein, ymAnd
respectively representing the real result and the prediction result of the real distorted image quality score, and M represents the number of images for fine tuning training. The network model parameters are subjected to fine tuning training through a gradient optimization method,and finally obtaining a quality evaluation model which accords with the real distorted image until the calculated loss function result is smaller than a threshold value.
7. An intelligent terminal applying the meta-learning based no-reference image quality data processing method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910844666.8A CN110728656A (en) | 2019-09-06 | 2019-09-06 | Meta-learning-based no-reference image quality data processing method and intelligent terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910844666.8A CN110728656A (en) | 2019-09-06 | 2019-09-06 | Meta-learning-based no-reference image quality data processing method and intelligent terminal |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110728656A true CN110728656A (en) | 2020-01-24 |
Family
ID=69217951
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910844666.8A Pending CN110728656A (en) | 2019-09-06 | 2019-09-06 | Meta-learning-based no-reference image quality data processing method and intelligent terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110728656A (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111507426A (en) * | 2020-04-30 | 2020-08-07 | 中国电子科技集团公司第三十八研究所 | No-reference image quality grading evaluation method and device based on visual fusion characteristics |
| CN111583259A (en) * | 2020-06-04 | 2020-08-25 | 南昌航空大学 | Document image quality evaluation method |
| CN112149805A (en) * | 2020-09-24 | 2020-12-29 | 济南大学 | Deep neural network acceleration and compression method and system based on frame search |
| CN112419270A (en) * | 2020-11-23 | 2021-02-26 | 深圳大学 | No-reference image quality evaluation method and device under meta learning and computer equipment |
| CN113313683A (en) * | 2021-05-28 | 2021-08-27 | 西安电子科技大学 | Non-reference video quality evaluation method based on meta-migration learning |
| CN113537407A (en) * | 2021-08-31 | 2021-10-22 | 平安医疗健康管理股份有限公司 | Image data evaluation processing method and device based on machine learning |
| CN113554045A (en) * | 2020-04-23 | 2021-10-26 | 国家广播电视总局广播电视科学研究院 | Data set manufacturing method, device, equipment and storage medium |
| CN113724197A (en) * | 2021-07-26 | 2021-11-30 | 南京邮电大学 | Screw thread screwing judgment method based on meta-learning |
| CN113722727A (en) * | 2021-07-21 | 2021-11-30 | 山东师范大学 | No-reference visual security analysis method and system for selectively encrypted image |
| CN114066857A (en) * | 2021-11-18 | 2022-02-18 | 烟台艾睿光电科技有限公司 | Infrared image quality evaluation method and device, electronic equipment and readable storage medium |
| CN115760822A (en) * | 2022-11-28 | 2023-03-07 | 深圳市捷易科技有限公司 | Image quality detection model establishing method and system |
| CN116502959A (en) * | 2023-06-21 | 2023-07-28 | 南京航空航天大学 | Product manufacturing quality prediction method based on meta learning |
| CN117152067A (en) * | 2023-08-14 | 2023-12-01 | 安徽大学 | Non-reference light field image quality evaluation method and system based on deep element learning |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107633520A (en) * | 2017-09-28 | 2018-01-26 | 福建帝视信息科技有限公司 | A kind of super-resolution image method for evaluating quality based on depth residual error network |
| CN107680077A (en) * | 2017-08-29 | 2018-02-09 | 南京航空航天大学 | A kind of non-reference picture quality appraisement method based on multistage Gradient Features |
| CN108401154A (en) * | 2018-05-25 | 2018-08-14 | 同济大学 | A kind of image exposure degree reference-free quality evaluation method |
| CN109272499A (en) * | 2018-09-25 | 2019-01-25 | 西安电子科技大学 | Non-reference picture quality appraisement method based on convolution autoencoder network |
| CN109978836A (en) * | 2019-03-06 | 2019-07-05 | 华南理工大学 | User individual image esthetic evaluation method, system, medium and equipment based on meta learning |
-
2019
- 2019-09-06 CN CN201910844666.8A patent/CN110728656A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107680077A (en) * | 2017-08-29 | 2018-02-09 | 南京航空航天大学 | A kind of non-reference picture quality appraisement method based on multistage Gradient Features |
| CN107633520A (en) * | 2017-09-28 | 2018-01-26 | 福建帝视信息科技有限公司 | A kind of super-resolution image method for evaluating quality based on depth residual error network |
| CN108401154A (en) * | 2018-05-25 | 2018-08-14 | 同济大学 | A kind of image exposure degree reference-free quality evaluation method |
| CN109272499A (en) * | 2018-09-25 | 2019-01-25 | 西安电子科技大学 | Non-reference picture quality appraisement method based on convolution autoencoder network |
| CN109978836A (en) * | 2019-03-06 | 2019-07-05 | 华南理工大学 | User individual image esthetic evaluation method, system, medium and equipment based on meta learning |
Non-Patent Citations (2)
| Title |
|---|
| AI研习社-译站: "图像样本不够用？元学习帮你解决", 《雷锋网：HTTPS://WWW.LEIPHONE.COM/CATEGORY/AI/CCXZRHOPMEJN4CUD.HTML》 * |
| CHELSEA FINN，ET AL.: "Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks", 《ARXIV》 * |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113554045B (en) * | 2020-04-23 | 2024-04-09 | 国家广播电视总局广播电视科学研究院 | Data set manufacturing method, device, equipment and storage medium |
| CN113554045A (en) * | 2020-04-23 | 2021-10-26 | 国家广播电视总局广播电视科学研究院 | Data set manufacturing method, device, equipment and storage medium |
| CN111507426B (en) * | 2020-04-30 | 2023-06-02 | 中国电子科技集团公司第三十八研究所 | Non-reference image quality grading evaluation method and device based on visual fusion characteristics |
| CN111507426A (en) * | 2020-04-30 | 2020-08-07 | 中国电子科技集团公司第三十八研究所 | No-reference image quality grading evaluation method and device based on visual fusion characteristics |
| CN111583259B (en) * | 2020-06-04 | 2022-07-22 | 南昌航空大学 | Document image quality evaluation method |
| CN111583259A (en) * | 2020-06-04 | 2020-08-25 | 南昌航空大学 | Document image quality evaluation method |
| CN112149805A (en) * | 2020-09-24 | 2020-12-29 | 济南大学 | Deep neural network acceleration and compression method and system based on frame search |
| CN112149805B (en) * | 2020-09-24 | 2023-08-22 | 法正互联(北京)科技有限公司 | Acceleration and compression method and system of deep neural network based on frame search |
| CN112419270A (en) * | 2020-11-23 | 2021-02-26 | 深圳大学 | No-reference image quality evaluation method and device under meta learning and computer equipment |
| CN112419270B (en) * | 2020-11-23 | 2023-09-26 | 深圳大学 | No-reference image quality evaluation method and device under meta-learning and computer equipment |
| CN113313683A (en) * | 2021-05-28 | 2021-08-27 | 西安电子科技大学 | Non-reference video quality evaluation method based on meta-migration learning |
| CN113722727A (en) * | 2021-07-21 | 2021-11-30 | 山东师范大学 | No-reference visual security analysis method and system for selectively encrypted image |
| CN113724197B (en) * | 2021-07-26 | 2023-09-15 | 南京邮电大学 | Thread screwing property judging method based on meta-learning |
| CN113724197A (en) * | 2021-07-26 | 2021-11-30 | 南京邮电大学 | Screw thread screwing judgment method based on meta-learning |
| CN113537407A (en) * | 2021-08-31 | 2021-10-22 | 平安医疗健康管理股份有限公司 | Image data evaluation processing method and device based on machine learning |
| CN113537407B (en) * | 2021-08-31 | 2022-05-17 | 平安医疗健康管理股份有限公司 | Image data evaluation processing method and device based on machine learning |
| CN114066857A (en) * | 2021-11-18 | 2022-02-18 | 烟台艾睿光电科技有限公司 | Infrared image quality evaluation method and device, electronic equipment and readable storage medium |
| CN115760822A (en) * | 2022-11-28 | 2023-03-07 | 深圳市捷易科技有限公司 | Image quality detection model establishing method and system |
| CN115760822B (en) * | 2022-11-28 | 2024-03-19 | 深圳市捷易科技有限公司 | Image quality detection model building method and system |
| CN116502959A (en) * | 2023-06-21 | 2023-07-28 | 南京航空航天大学 | Product manufacturing quality prediction method based on meta learning |
| CN116502959B (en) * | 2023-06-21 | 2023-09-08 | 南京航空航天大学 | Product manufacturing quality prediction method based on meta learning |
| CN117152067A (en) * | 2023-08-14 | 2023-12-01 | 安徽大学 | Non-reference light field image quality evaluation method and system based on deep element learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110728656A (en) | Meta-learning-based no-reference image quality data processing method and intelligent terminal | |
| CN108428227B (en) | No-reference image quality evaluation method based on full convolution neural network | |
| CN108665460B (en) | Image quality evaluation method based on combined neural network and classified neural network | |
| CN110533631B (en) | SAR image change detection method based on pyramid pooling twin network | |
| CN108090902B (en) | Non-reference image quality objective evaluation method based on multi-scale generation countermeasure network | |
| CN111182292B (en) | No-reference video quality evaluation method and system, video receiver and intelligent terminal | |
| CN108596902B (en) | Multi-task full-reference image quality evaluation method based on gating convolutional neural network | |
| CN109325550B (en) | No-reference image quality evaluation method based on image entropy | |
| CN109308696B (en) | No-reference image quality evaluation method based on hierarchical feature fusion network | |
| CN107633255A (en) | A kind of rock lithology automatic recognition classification method under deep learning pattern | |
| Deng et al. | Blind noisy image quality assessment using sub-band kurtosis | |
| CN110689523A (en) | Personalized image information evaluation method based on meta-learning and information data processing terminal | |
| CN110400293B (en) | No-reference image quality evaluation method based on deep forest classification | |
| CN109816646B (en) | Non-reference image quality evaluation method based on degradation decision logic | |
| CN111429402A (en) | Image quality evaluation method for fusing advanced visual perception features and depth features | |
| CN112767385B (en) | No-reference image quality evaluation method based on significance strategy and feature fusion | |
| CN109919901B (en) | Image quality evaluation method based on ensemble learning and random forest | |
| CN115205196A (en) | No-reference image quality evaluation method based on twin network and feature fusion | |
| CN109754390A (en) | A kind of non-reference picture quality appraisement method based on mixing visual signature | |
| CN114187261A (en) | Non-reference stereo image quality evaluation method based on multi-dimensional attention mechanism | |
| CN111242131B (en) | Method, storage medium and device for identifying images in intelligent paper reading | |
| CN112766419A (en) | Image quality evaluation method and device based on multitask learning | |
| CN114265954B (en) | Graph representation learning method based on position and structure information | |
| CN114818945A (en) | Small sample image classification method and device integrating category adaptive metric learning | |
| CN114529096A (en) | Social network link prediction method and system based on ternary closure graph embedding |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200124 |