CN117372387A - Lightweight image quality evaluation method and system based on convolutional neural network pruning - Google Patents
Lightweight image quality evaluation method and system based on convolutional neural network pruning Download PDFInfo
- Publication number
- CN117372387A CN117372387A CN202311389543.2A CN202311389543A CN117372387A CN 117372387 A CN117372387 A CN 117372387A CN 202311389543 A CN202311389543 A CN 202311389543A CN 117372387 A CN117372387 A CN 117372387A
- Authority
- CN
- China
- Prior art keywords
- neural network
- image quality
- pruning
- convolutional neural
- quality evaluation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013441 quality evaluation Methods 0.000 title claims abstract description 127
- 238000013138 pruning Methods 0.000 title claims abstract description 126
- 238000013527 convolutional neural network Methods 0.000 title claims abstract description 94
- 238000000034 method Methods 0.000 title claims abstract description 77
- 238000012549 training Methods 0.000 claims abstract description 58
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 12
- 238000013528 artificial neural network Methods 0.000 claims abstract description 9
- 230000015654 memory Effects 0.000 claims description 25
- 238000001303 quality assessment method Methods 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 16
- 230000008569 process Effects 0.000 claims description 15
- 230000004913 activation Effects 0.000 claims description 12
- 238000001994 activation Methods 0.000 claims description 12
- 230000006835 compression Effects 0.000 claims description 8
- 238000007906 compression Methods 0.000 claims description 8
- 238000011156 evaluation Methods 0.000 claims description 8
- 238000000605 extraction Methods 0.000 claims description 8
- 230000011218 segmentation Effects 0.000 claims description 6
- 238000013526 transfer learning Methods 0.000 claims description 5
- 230000005540 biological transmission Effects 0.000 claims description 4
- 238000013508 migration Methods 0.000 claims description 4
- 230000005012 migration Effects 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 210000002569 neuron Anatomy 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 238000012360 testing method Methods 0.000 description 6
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 238000002474 experimental method Methods 0.000 description 4
- 238000010606 normalization Methods 0.000 description 4
- 230000004075 alteration Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 230000007423 decrease Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000009966 trimming Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30168—Image quality inspection
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a lightweight image quality evaluation method and system based on convolutional neural network pruning, wherein the method comprises the following steps: acquiring a data set to be evaluated; selecting a convolutional neural network structure according to a data set to be evaluated, and training the convolutional neural network; pruning operation is carried out on the trained convolutional neural network, and an image quality evaluation model is obtained; inputting the data set to be evaluated into the image quality evaluation model to obtain an image quality evaluation result; the method comprises the steps of splitting a convolution layer in a convolution neural network into a sub-convolution kernel and a decorator, and initializing the decorator by using a hierarchical correlation propagation algorithm (LRP); training the decorator by using the data set to be evaluated; setting a mask according to the decorator and completing pruning; fine tuning the model after pruning using the data set to be evaluated. The invention can complete the image quality evaluation task with faster operation speed and smaller existing occupation.
Description
Technical Field
The invention relates to the technical field of multi-mode media quality evaluation, in particular to a lightweight image quality evaluation method, system and terminal based on convolutional neural network pruning.
Background
The goal of blind image quality assessment is to assess the image quality of the absence of reference images at various stages of image processing, such as image acquisition, compression and transmission. In the past few years, deep neural networks have achieved significant performance in the field of image quality evaluation due to their strong feature representation capabilities.
While these DNN-based models have achieved great success in performance, model reasoning is expensive due to the time, power, and memory required for heavyweight architectures, which makes them difficult to deploy on resource-constrained devices and requires a significant amount of computing resources in the reasoning process.
Filter pruning is one of the most popular methods in the field of model compression, which can reduce both parameter and floating point operations (FLOP) by removing unimportant convolution channels. For example, some researchers compute the L1 norm of each filter weight as a measure of the importance of each channel; some researchers use the scale factor of the Bulk Normalized (BN) layer after the convolutional layer to measure channel importance. Despite the significant differences between these pruning methods, most methods include the following steps: the importance of each filter is evaluated, the unimportant filters are trimmed, and the entire model is retrained to recover the performance of the model. However, image quality assessment is typically based on transfer learning, and model training processes are typically complex. For example, two modules of the DBCNN perform a distortion classification task on a database containing 80000 multiple distorted images and pre-training of the image classification task on ImageNet, respectively, and then fine-tune the entire model after training the regressor. For another example, the Star image quality assessment utilizes ResNet-50 as the backbone network that is initialized by pre-training on ImageNet, and then training the model simultaneously on multiple databases. The complexity of the non-reference image quality assessment model training process and the large database used in the pre-training stage make retraining after pruning resource-consuming and slow, especially using a conventional iterative pruning process.
Similar to image quality assessment, other models based on transfer learning face excessive computational resource consumption problems when pruning. Some research on shift learning pruning has emerged, but the difficulty of pruning without the large database used in pre-training. There have been studies to train the scale factors in Batch Normalization layers after the convolutional layers to measure the importance of each convolutional channel, however, this approach can cause varying degrees of damage to different parts of the network, which can make retraining after pruning more difficult. In addition, transTailor uses the scale factors in Batch Normalization to identify the most important channels that some parts of the image quality assessment model would not be equipped with. After pruning the image quality evaluation method by using the methods, the image quality evaluation performance is seriously degraded, or a large amount of computing resources are required to restore the performance of the image quality evaluation method.
Therefore, it is necessary to develop a lightweight image quality evaluation technique to solve the above-described problems.
Disclosure of Invention
In order to overcome the defects in the prior art, the invention aims to provide a lightweight image quality evaluation method, a lightweight image quality evaluation system and a lightweight image quality evaluation terminal based on convolutional neural network pruning, which can complete an image quality evaluation task with higher operation speed and smaller existing occupation, and ensure the image quality evaluation performance.
The first object of the present invention is to provide a lightweight image quality evaluation method based on convolutional neural network pruning, comprising:
acquiring a data set to be evaluated;
selecting a convolutional neural network structure according to a data set to be evaluated, and training the convolutional neural network;
pruning operation is carried out on the trained convolutional neural network, and an image quality evaluation model is obtained;
inputting the data set to be evaluated into the image quality evaluation model to obtain an image quality evaluation result;
wherein: the pruning operation is carried out on the trained convolutional neural network to obtain an image quality evaluation model, which comprises the following steps:
splitting a convolutional layer in the convolutional neural network into two trainable portions: the method comprises the steps of deconvolution of a kernel and a decorator, and initializing the decorator by using a hierarchical correlation propagation algorithm; each decorator represents the importance of the current convolutional layer;
training a decorator by using the data set to be evaluated;
setting a mask according to the decorator and completing pruning;
and fine tuning the convolutional neural network after pruning by using the data set to be evaluated to obtain an image quality evaluation model.
Optionally, the selecting a convolutional neural network structure according to the data set to be evaluated, and training the convolutional neural network includes:
selecting a convolutional neural network structure according to a data set to be evaluated, and taking image classification as a source task to finish pre-training on a public data set;
and further training the convolutional neural network obtained by pre-training on a data set to be evaluated to finish migration learning. The feature extraction part of the pre-trained convolutional neural network is connected with a new regressor, transfer learning is carried out on image quality evaluation, and the performance of the image quality evaluation is improved by utilizing semantic knowledge in the feature extraction part;
optionally, the training decorator using the data set to be evaluated includes:
in a pruning stage, convolutional layers in the convolutional neural network, each portion of the convolutional neural network being frozen except for a decorator;
training the decorator by using a target image quality evaluation database, namely a data set to be evaluated, so as to better adapt to an image quality evaluation task;
the trained decorators are converted into global importance factors of each channel by using a Taylor expansion method, and pruning masks are set under the guidance of the importance factors.
The target image quality evaluation database, namely the data set to be evaluated, can be a small-scale data set, so that the calculation speed is higher, and the occupied storage space is smaller.
Optionally, the setting the mask according to the decorator and completing the clipping includes:
measuring the importance of each channel by using the global importance factor, setting the mask of the least important channel to 0 and setting the masks of the rest channels to 1; during the pruning phase, the output of each channel will be multiplied by the mask to simulate pruning.
Optionally, the convolutional neural network after pruning is finely tuned by using the data set to be evaluated to obtain an image quality evaluation model
Merging the convolution layers in the convolution neural network, and retraining the whole convolution neural network except for the decorator by using the data set to be evaluated so as to reduce performance loss caused by pruning; after retraining, splitting the convolutional layer again using the unmodified decorator;
and iterating the pruning operation until the compression ratio reaches a target or the performance is lower than a predefined threshold value, and obtaining an image quality evaluation model.
Optionally, the merging of the convolutional layers in the convolutional neural network is to restore the convolutional layers to their normal state, wherein a "merge" operation is defined as the product between the sub-convolutional kernel and its corresponding decorator.
Optionally, splitting the convolutional layer in the convolutional neural network into a sub-convolutional kernel and a decorator, wherein: the split convolution layer is realized through convolution kernel factorization;
the original convolution kernel W for each channel ij Divided into sub-convolution kernelsAnd a decorator alpha ij And they are all trainable:
wherein i and j respectively indicate the ith layer and the jth channel of the current layer of the image quality evaluation model; decorators are used to measure the importance of each lane and trim the least important lane under the direction of these decorators; in the split state, the input feature map F from the previous layer i-1 With a sub-convolution kernel W i * The process is expressed asIs then convolved with the decorator alpha in the ith layer i Multiplying:
thus, the feature map F is output i The results of the operations are the same before and after the segmentation and the segmentation operation does not lead to additional errors in the reasoning results.
Optionally, the initializing the decorator using a hierarchical relevance propagation algorithm, wherein:
taking the loss image x as input to the convolutional neural network for reasoning, and collecting an activation map of each convolutional layer;
the output f (x) result of the convolutional neural network is transmitted backwards through the network according to an LRP transmission rule, and all the activation graphs acquire the correlation between the result and the output of the image quality evaluation model;
the correlations of activations belonging to the same convolution channel are summed to the correlation that the channel initialized its decorator.
A second object of the present invention is to provide a lightweight image quality evaluation system based on convolutional neural network pruning, including:
an image acquisition module: acquiring a data set to be evaluated;
model pre-training module: selecting a convolutional neural network structure according to a data set to be evaluated, and training the convolutional neural network;
model pruning module: pruning operation is carried out on the trained convolutional neural network, and an image quality evaluation model is obtained;
and an evaluation module: inputting the data set to be evaluated into the image quality evaluation model to obtain an image quality evaluation result;
wherein: the model pruning module comprises:
splitting a convolution layer in the convolution neural network into two trainable parts, namely a sub-convolution kernel and a decorator, and initializing the decorator by using a hierarchical correlation propagation algorithm; each decorator represents the importance of the current convolutional layer;
training a decorator by using the data set to be evaluated;
setting a mask according to the decorator and completing pruning;
and fine tuning the convolutional neural network after pruning by using the data set to be evaluated to obtain an image quality evaluation model.
The third object of the invention is to provide a lightweight image quality evaluation terminal based on convolutional neural network pruning, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor is used for executing the lightweight image quality evaluation method based on convolutional neural network pruning when executing the program.
Compared with the prior art, the embodiment of the invention has at least one of the following beneficial effects:
according to the lightweight image quality evaluation method and system based on convolutional neural network pruning, pretraining is completed on the image quality evaluation data set, then pruning is carried out, the model after pruning only needs to be retrained on the small image quality evaluation data set, the calculation speed is faster, the lightweight image quality evaluation model occupying smaller storage space is used for image evaluation, wherein the convolutional layer is divided into the sub-convolutional kernel and the decorator, the importance of a convolutional channel can be measured more accurately, and therefore the requirement on retraining conditions is reduced. Experiments on several image quality evaluation models prove the effectiveness of the image quality evaluation method in the invention.
In addition, since the present invention can identify important, useless and harmful filters, the performance of some image quality evaluation models can be improved even after model compression at a proper pruning rate, such as: the spearman correlation coefficient and the pearson correlation coefficient between the reasoning result and the real MOS score of the image quality evaluation method are higher, and the more accurate image quality evaluation result can be obtained by using less storage space and shorter running time in the actual image evaluation application.
Drawings
Other features, objects and advantages of the present invention will become more apparent upon reading of the detailed description of non-limiting embodiments, given with reference to the accompanying drawings in which:
FIG. 1 is a flowchart of an image quality evaluation method according to an embodiment of the present invention;
FIG. 2 is a flow chart of an image quality evaluation method according to a preferred embodiment of the present invention;
fig. 3 is a schematic diagram illustrating data reasoning in pruning operation according to an embodiment of the present invention.
Detailed Description
The following describes embodiments of the present invention in detail: the embodiment is implemented on the premise of the technical scheme of the invention, and detailed implementation modes and specific operation processes are given. It should be noted that variations and modifications can be made by those skilled in the art without departing from the spirit of the invention, which falls within the scope of the invention.
In the prior art, because model training is complex, a huge database is needed in the traditional iterative pruning process, so that resource consumption and slow speed of retraining after pruning are realized, and the realization speed of the image quality evaluation method is slow and the required training data is large. In order to avoid complex retraining processes by using huge multiple databases and ensure the generalizability of most image quality evaluation models, the embodiment of the invention provides a lightweight image quality evaluation method which is efficient in resources and easy to use, and the method only uses one small image quality evaluation data set to complete the whole pruning process, and simultaneously maintains the performance of the models on the used data set. In addition, as important, useless and harmful filters can be identified by pruning, the performance of the image quality evaluation model can be even improved after the image quality evaluation model is compressed by proper pruning rate, so that a lightweight image quality evaluation model with higher calculation speed and smaller occupied storage space is obtained on the premise of ensuring the performance. The model is used for evaluating the image quality, so that the consumption of computing resources can be greatly reduced, and the processing speed can be improved.
Fig. 1 is a flowchart of an image quality evaluation method according to an embodiment of the invention. Referring to fig. 1, the lightweight image quality evaluation method based on convolutional neural network pruning in the present embodiment includes the following steps:
s100: acquiring a data set to be evaluated;
in this step, the data set to be evaluated may be a quality evaluation database containing the target image. Specifically, the data set to be evaluated consists of images with perceived quality and corresponding average subjective scores, and the images can be obtained from social networking sites, network media and other approaches.
S200: selecting a convolutional neural network structure according to a data set to be evaluated, and training the convolutional neural network;
in the step, a convolutional neural network structure is selected according to a data set to be evaluated, wherein the convolutional neural network structure can be selected from a convolutional neural network-based image quality evaluation model which is disclosed in the existing image quality evaluation field and has good performance. Specifically, the data set to be evaluated can be uniformly divided into a training set and a testing set, the training set is used for training the existing image quality evaluation model, the testing set is used for testing, and a convolutional neural network model with higher testing performance is selected.
S300: pruning operation is carried out on the trained convolutional neural network, and an image quality evaluation model is obtained;
the step is executed, and specifically can comprise the following steps:
s301: splitting the convolutional layer in the convolutional neural network into two trainable portions: the method comprises the steps of deconvolution of a kernel and a decorator, and initializing the decorator by using a hierarchical correlation propagation algorithm; each decorator represents the importance of the current convolutional layer;
s302: training the decorator by using the data set to be evaluated; the data set to be evaluated may be a data set to be evaluated of a small-scale image quality, for example, a data set to be evaluated of about 1000 pictures.
S303: setting a mask according to the decorator and completing pruning;
s304: and fine tuning the convolutional neural network after pruning by using the image quality evaluation data set to obtain an image quality evaluation model.
S400: inputting the data set to be evaluated into an image quality evaluation model to obtain an image quality evaluation result;
in this embodiment, the convolutional neural network model is trained on the image quality evaluation data set first, and then pruning is performed, and the model after pruning can obtain a lightweight image quality evaluation model with faster calculation speed and smaller occupied storage space only by retraining on the small image quality evaluation data set. In particular, in this embodiment, the convolutional layer is divided into a sub-convolutional kernel and a decorator, so that the importance of the convolutional channel can be measured more accurately, thereby reducing the requirement on retraining conditions. Therefore, the calculation speed of the whole image evaluation is faster, and the occupied storage space is smaller.
As a preferred embodiment, referring to fig. 2, the lightweight image quality evaluation method based on convolutional neural network pruning provided in this embodiment includes the following steps:
s1: selecting a proper convolutional neural network structure according to the data set to be evaluated, and taking image classification as a source task to finish pre-training on the ImageNet data set; of course, in other embodiments, other pre-training may be performed on other disclosed data sets;
s2: further training the neural network obtained in the step S1 on an image quality evaluation data set to finish migration learning;
s3: the original convolutional layer in the convolutional neural network is divided into two trainable parts: a deconvolution kernel and a decorator, which represent the importance of this channel. In particular, the decorator is initialized using a hierarchical relevance propagation (LRP) algorithm that can measure the contribution of each element in the model to the network information flow;
s4: in the pruning stage, the convolution layer is split according to S2, with each part of the model being frozen except for the decorator. Only the decorator is trained using the target image quality assessment database (data set to be assessed) to better accommodate the image quality assessment task. The trained decorator is then converted to global importance factors for each channel using taylor expansion and pruning masks are set under the direction of the importance factors.
S5: the global importance factor obtained in S4 is used to measure the importance of each channel, the mask of the least important channel is set to 0, and the masks of the remaining channels are set to 1. During the pruning phase, the output of each channel will be multiplied by the mask to simulate pruning.
S6: the convolutional layers in the S4 network are merged and the entire model, except the decorator, is retrained using only the image quality assessment database (data set to be assessed) to mitigate performance loss due to pruning. After retraining, splitting the convolutional layer again using the unmodified decorator;
s7: S4-S6 may iterate until the compression ratio reaches a target or the performance is below a predefined threshold. Because masks are used to simulate pruning during the pruning phase, it is necessary to extract the actual compact model based on the masks to facilitate post-pruning deployment. Retraining after model extraction is optional because the extraction operation does not cause too much performance loss.
Corresponding to the same concept of the lightweight image quality evaluation method based on convolutional neural network pruning in the above embodiment, another embodiment of the present invention further provides an image quality evaluation system based on convolutional neural network, as shown in fig. 2, after acquiring a data set to be evaluated, the system includes three major modules:
model pre-training module: selecting a convolutional neural network structure according to a data set to be evaluated, and training the convolutional neural network; specifically, selecting a proper convolutional neural network structure according to a data set to be evaluated, performing pre-training on an ImageNet data set by taking image classification as a source task, and performing further training on the obtained neural network on the image quality evaluation data set to complete migration learning;
model pruning module: pruning operation is carried out on the trained convolutional neural network, and an image quality evaluation model is obtained; this is also an important link of this embodiment, where the convolutional layer in the convolutional neural network is split into a sub-convolutional kernel and a decorator, and the decorator is initialized using a hierarchical correlation propagation algorithm; training a decorator using the small-scale image quality assessment dataset; setting a mask according to the decorator and completing pruning; fine tuning the convolutional neural network after pruning by using an image quality evaluation data set to obtain an image quality evaluation model;
and an evaluation module: and inputting the data set to be evaluated into an image quality evaluation model to obtain an image quality evaluation result.
The model pruning module of the embodiment includes a channel importance sensing portion that divides the original convolution layer into two trainable portions: a deconvolution kernel and a decorator, which represent the importance of this channel. In particular, the decorator is initialized using a hierarchical relevance propagation (LRP) algorithm that can measure the contribution of each element in the model to the network information flow; in the pruning stage, the convolution layer is split into a sub-convolution kernel and a decorator, with each part of the model being frozen except for the decorator. Only the decorator is trained using the target image quality assessment database (data set to be assessed) to better accommodate the image quality assessment task. The trained decorator is then converted to global importance factors for each channel using taylor expansion and pruning masks are set under the direction of the importance factors.
Still further, the importance of each channel may be measured using a global importance factor, with the mask of the least important channel set to 0 and the masks of the remaining channels set to 1. During the pruning phase, the output of each channel will be multiplied by the mask to simulate pruning; the convolutional layers in the network are combined and the entire model except the decorator is retrained using only the image quality assessment database (data set to be assessed) to mitigate performance loss due to pruning. After retraining, splitting the convolutional layer again using the unmodified decorator; the above steps are iterated until the compression ratio reaches a target or the performance is below a predefined threshold. Because masks are used to simulate pruning during the pruning phase, it is necessary to extract the actual compact model based on the masks to facilitate post-pruning deployment; and training the extracted model on the image quality evaluation data set.
The channel importance sensing part corresponds to S3-S4 in the embodiment of the lightweight image quality evaluation method based on convolutional neural network pruning, and the specific implementation technology is the same.
The image quality evaluation method, the system and the terminal based on the convolutional neural network pruning provided by the embodiment of the invention can be used for rapidly and effectively acquiring the lightweight image quality evaluation model with higher calculation speed and smaller occupied storage space.
In a preferred embodiment of the present invention, the model pre-training module for image quality evaluation may comprise two parts,
(1) Taking image classification as a source task, and training on an ImageNet large-scale data set, so that the network has the capability of understanding image semantics;
(2) Connecting the feature extraction part of the pre-training network in the S1 with a new regressor, performing transfer learning on image quality evaluation, and improving the performance of the image quality evaluation by utilizing semantic knowledge in the feature extractor;
as shown in fig. 2, in a preferred embodiment, in the model pruning module, the channel importance sensing (corresponding to the channel importance sensing module in fig. 2) includes the following two parts, namely splitting the convolution layer and decorator initialization:
(1) Splitting the convolution layer is achieved by convolution kernel factorization. The original convolution kernel W for each channel ij Divided into sub-convolution kernelsAnd a decorator alpha ij And they are all trainable:
where i and j indicate the ith layer and the jth channel of the current layer of the image quality assessment model, respectively. Decorators are used to measure the importance of each lane and trim the least important lane under the direction of these decorators. In the split state, the input feature map F from the previous layer i-1 With a sub-convolution kernel W i * The process is expressed asIs then convolved with the decorator alpha in the ith layer i Multiplying:
thus, the feature map F is output i The results of the operations are the same before and after the segmentation and the segmentation operation does not lead to additional errors in the reasoning results.
(2) Decorator initialization
The decorator is initialized using a hierarchical relevance propagation (LRP) algorithm. Sending the loss image x as input into a network for reasoning, and collecting an activation graph of each convolution layer; the network output f (x) result is transmitted backwards through the network according to the LRP transmission rule, and all the activation graphs acquire the correlation between the activation graphs and the model output; the correlations of activations belonging to the same convolution channel are summed to the correlation that the channel initialized its decorator.
Specific implementations may be made with reference to the following:
is provided withIs a correlation of neurons at layer l,/->For the association of neuron b at the next layer l+1, then the LRP propagation rule can be given by:
wherein the method comprises the steps ofRepresentation->To->Is the correlation propagation process, z ab And z b The mapping used to describe one layer to the next can be described by the following equation:
z ab ＝x a ·w ab ,
x b ＝g(z b )
wherein w is ab Is the weight connecting neuron a and neuron b, bias b Is the bias of layer b, z b Is the pre-activation of the b-th neuron of the layer 1+1 of neurons, g (·) is a nonlinear activation function.
The embodiment of the invention selects an alpha-beta criterion to prevent z ab When small, the correlationPossibly taking an unbounded value, the criteria may be described as
Wherein α+β=1, "+" and "-" respectively represent z ab And z b In the present invention, α=1, β=0;
the trained decorator may reflect the relative importance of the different channels within the convolution layer. To achieve more efficient, automated global pruning, the decorators need to be converted into global importance factors. The present embodiment adopts a method based on taylor expansion, expressed as:
where θ may be any factor associated with the convolved channel. Thus, decorator globalization may be defined as
Where L (·) is the loss function for training the model, D IQA Is the data set to be evaluated used in the pruning process, W' represents the current model parameters, phi ij Is the global importance factor for the j-th channel of the i-th layer.
In another preferred embodiment, the pruning section (corresponding to the pruning and retraining module of fig. 2) may specifically operate as follows: the global importance of each channel is measured using a global importance factor, the mask of the least important channel is set to 0, and the masks of the remaining channels are set to 1. During the pruning phase, the output of each channel will be multiplied by the mask to simulate pruning.
As shown in fig. 3, if the convolved layer is followed by Batch Normalization layers, the output of Batch Normalization layers should be multiplied again by the mask. This arrangement ensures that the inference process of the pruned model can be properly simulated using the mask.
Further, in some embodiments, merging convolutional layers in a network, operations are defined as the product between a sub-convolutional kernel and its corresponding decorator:
the decorator will not participate in subsequent reasoning.
The entire image quality evaluation model except the decorator is retrained using only the data set to be evaluated, and the performance loss due to trimming can be reduced. After retraining, splitting the convolutional layer again using the unmodified decorator; the above steps are iterated until the compression ratio reaches a target or the performance is below a predefined threshold. Because masks are used to simulate pruning during the pruning phase, it is necessary to extract the actual compact model (pruned model) based on the masks to facilitate post-pruning deployment. And training the extracted model on the image quality evaluation data set, and recovering the performance degradation of the model possibly caused by the extraction operation through retraining.
The implementation effect is as follows:
in order to verify the effectiveness and the universality of the quality evaluation method, three most advanced IQA models based on complex DNN structures are selected for verification: DBCNN, stariqa with a res net50 backbone, and HyperIQA. All of these IQA models are independently trained tests on three true distortion IQA databases.
In order to show the effectiveness and convenience of acquisition of the lightweight image quality evaluation method of the present invention, two smaller-scale databases were selected: clear, BID and a larger database: koniQ-10k. The clear consists of 1162 images with different true distortions taken by the mobile device. BID is a blurred image database containing 586 images with true blur distortion. KoniQ-10k contains 10073 images selected from the large public multimedia database YFCC100m, covering a wide range of distortions in brightness, color, contrast, noise, sharpness and other quality dimensions. In the experiment, the following two evaluation criteria are selected to measure the performance of the audio and video objective quality evaluation method: pearson Linear Correlation Coefficients (PLCC) and Spearman rank order correlation coefficients (SRCC).
The image quality evaluation method provided by the embodiment of the invention starts from a convolutional neural network and uses a training strategy and pre-training parameters which are suitable for each network. All experiments were implemented on a server of NVIDIA RTX 3080 using the PyTorch framework.
For the database, the resolution of the minimum size of the images was set to 380 while maintaining their aspect ratio, and the images with the resolution 320×320 were randomly cropped during the training and testing phases. The hyper-parameters of the decorator training and retraining phases need to be different depending on the sparsity of the network itself.
The application performance of the lightweight image quality assessment model obtained by performing appropriate super-parameter settings for decorator training and whole model retraining using three models, DBCNN, stairIQA and HyperIQA, is shown in table 1.
TABLE 1
On a mini database: pruning using a small IQA database is most challenging where only the IQA database is available for retraining. To verify the effectiveness of the method proposed in this example under the above conditions, three models were trimmed on the mini-databases, CLIVE and BID, the results of which are shown in Table 1. From the results, it can be seen that as the pruning rate increases, the model performance increases and then decreases, and splitting the convolutional layer pruning can introduce performance gains for the pruned model on a small IQA database. In most cases, the same performance as the original pruning model can be obtained by splitting the model of the convolutional layer pruning on a small database with only 50% of the parameters remaining.
On a large database: the method of the present invention was further tested on a larger database, koniQ-10k. The results show that, unlike the small database, the performance of the model decreases with increasing pruning rate and no improvement in performance can be observed during pruning. This is reasonable because a larger database provides significantly better conditions for retraining and the performance of the model drops much slower as the pruning rate increases and the performance of the lightweight image quality assessment model obtained based on StairIQA and HyperIQA on the KonIQ-10k database is still acceptable after 80% of the parameters are pruned.
In conclusion, experiments on several image quality evaluation models prove the effectiveness of the image quality evaluation model pruning method. For example, the performance of DBCNN, stairIQA and HyperIQA may remain at baseline levels when 50% of the parameters are trimmed, and the model is retrained on a true distorted image quality assessment database containing only 586 images, and the performance of stariqa and HyperIQA on the KonIQ-10k database is still acceptable after 80% of the parameters are trimmed.
According to the lightweight image quality evaluation method based on the convolutional neural network pruning, the convolutional layer in the image quality evaluation network is split into the sub-convolutional kernel and the decorator, so that the importance of a convolutional channel is more accurately measured, the requirement on the condition of retraining after pruning is reduced, and the image quality evaluation method with higher calculation speed and smaller occupied storage space is obtained. The method can complete the image quality evaluation task with faster operation speed and smaller existing occupation.
Based on the same technical concept, in another embodiment, the invention further provides an image quality evaluation terminal based on convolutional neural network pruning, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor is used for executing the image quality evaluation method based on convolutional neural network pruning when executing the program.
Optionally, a memory for storing a program; memory, which may include volatile memory (english) such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (Double Data Rate Synchronous Dynamic Random Access Memory, DDR SDRAM), and the like; the memory may also include a non-volatile memory (English) such as a flash memory (English). The memory is used to store computer programs (e.g., application programs, functional modules, etc. that implement the methods described above), computer instructions, etc., which may be stored in one or more memories in a partitioned manner. And the above-described computer programs, computer instructions, data, etc. may be invoked by a processor.
The computer programs, computer instructions, etc. described above may be stored in one or more memories in partitions. And the above-described computer programs, computer instructions, data, etc. may be invoked by a processor.
A processor for executing the computer program stored in the memory to implement the steps in the method according to the above embodiment. Reference may be made in particular to the description of the embodiments of the method described above.
The processor and the memory may be separate structures or may be integrated structures that are integrated together. When the processor and the memory are separate structures, the memory and the processor may be connected by a bus coupling.
In an embodiment of the present invention, there is also provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the reference-free video evaluation method in any of the method embodiments described above.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the invention.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (10)
1. A lightweight image quality evaluation method based on convolutional neural network pruning is characterized by comprising the following steps:
acquiring a data set to be evaluated;
selecting a convolutional neural network structure according to a data set to be evaluated, and training the convolutional neural network;
pruning operation is carried out on the trained convolutional neural network, and an image quality evaluation model is obtained;
inputting the data set to be evaluated into the image quality evaluation model to obtain an image quality evaluation result;
wherein: the pruning operation is carried out on the trained convolutional neural network to obtain an image quality evaluation model, which comprises the following steps:
splitting a convolutional layer in the convolutional neural network into two trainable portions: the method comprises the steps of deconvolution of a kernel and a decorator, and initializing the decorator by using a hierarchical correlation propagation algorithm; each decorator represents the importance of the current convolutional layer;
training a decorator by using the data set to be evaluated;
setting a mask according to the decorator and completing pruning;
and fine tuning the convolutional neural network after pruning by using the data set to be evaluated to obtain an image quality evaluation model.
2. The lightweight image quality evaluation method based on convolutional neural network pruning according to claim 1, wherein the selecting a convolutional neural network structure according to a data set to be evaluated, training the convolutional neural network, comprises:
selecting a convolutional neural network structure according to a data set to be evaluated, and taking image classification as a source task to finish pre-training on a public data set;
further training the convolutional neural network obtained by pre-training on a data set to be evaluated to complete migration learning; the feature extraction part of the pretrained convolutional neural network is connected with a new regressor, transfer learning is carried out on image quality evaluation, and the performance of the image quality evaluation is improved by utilizing semantic knowledge in the feature extraction part.
3. The lightweight image quality evaluation method based on convolutional neural network pruning according to claim 1, wherein the training decorator using the data set to be evaluated comprises:
in a pruning stage, convolutional layers in the convolutional neural network, each portion of the convolutional neural network being frozen except for a decorator;
training the decorator by using a data set to be evaluated so as to better adapt to an image quality evaluation task;
the trained decorators are converted into global importance factors of each channel by using a Taylor expansion method, and pruning masks are set under the guidance of the importance factors.
4. The lightweight image quality evaluation method based on convolutional neural network pruning according to claim 3, wherein the setting of a mask according to a decorator and the completion of pruning comprises:
measuring the importance of each channel by using the global importance factor, setting the mask of the least important channel to 0 and setting the masks of the rest channels to 1; during the pruning phase, the output of each channel will be multiplied by the mask to simulate pruning.
5. The method for evaluating the quality of a lightweight image based on pruning of a convolutional neural network according to claim 4, wherein the fine tuning of the convolutional neural network after pruning using the image quality evaluation dataset to obtain the image quality evaluation model comprises:
merging the convolution layers in the convolution neural network, and retraining the whole convolution neural network except for the decorator by using the data set to be evaluated so as to reduce performance loss caused by pruning; after retraining, splitting the convolutional layer again using the unmodified decorator;
and iterating the pruning operation until the compression ratio reaches a target or the performance is lower than a predefined threshold value, and obtaining an image quality evaluation model.
6. The method of claim 5, wherein the merging of the convolutional layers in the convolutional neural network is to restore the convolutional layers to their normal state, wherein the "merge" operation is defined as the product between the sub-convolutional kernels and their corresponding decorators.
7. The lightweight image quality assessment method based on convolutional neural network pruning according to claim 1, wherein the convolutional layer in the convolutional neural network is split into two trainable parts: a deconvolution kernel and a decorator, wherein:
the split convolution layer is realized through convolution kernel factorization;
the original convolution kernel W for each channel ij Divided into sub-convolution kernelsAnd a decorator alpha ij And they are all trainable:
wherein i and j respectively indicate the ith layer and the jth channel of the current layer of the image quality evaluation model; decorators are used to measure the importance of each lane and trim the least important lane under the direction of these decorators; in the split state, the input feature map F from the previous layer i-1 With a sub-convolution kernel W i * The process is expressed asIs then convolved with the decorator alpha in the ith layer i Multiplying:
thus, the feature map F is output i The results of the operations are the same before and after the segmentation and the segmentation operation does not lead to additional errors in the reasoning results.
8. The method for lightweight image quality assessment based on convolutional neural network pruning according to claim 1, wherein the decorator is initialized using a hierarchical relevance propagation algorithm, wherein:
taking the loss image x as input to the convolutional neural network for reasoning, and collecting an activation map of each convolutional layer;
the output f (x) result of the convolutional neural network is transmitted backwards through the network according to an LRP transmission rule, and all the activation graphs acquire the correlation between the result and the output of the image quality evaluation model;
the correlations of activations belonging to the same convolution channel are summed to the correlation that the channel initialized its decorator.
9. A lightweight image quality evaluation system based on convolutional neural network pruning, comprising:
an image acquisition module: acquiring a data set to be evaluated;
model pre-training module: selecting a convolutional neural network structure according to a data set to be evaluated, and training the convolutional neural network;
model pruning module: pruning operation is carried out on the trained convolutional neural network, and an image quality evaluation model is obtained;
and an evaluation module: inputting the data set to be evaluated into the image quality evaluation model to obtain an image quality evaluation result;
wherein: the model pruning module comprises:
splitting a convolutional layer in the convolutional neural network into two trainable parts, namely a sub-convolutional kernel and a decorator, and initializing the decorator by using a hierarchical correlation propagation algorithm;
training a decorator by using the data set to be evaluated;
setting a mask according to the decorator and completing pruning;
and fine tuning the convolutional neural network after pruning by using the data set to be evaluated to obtain an image quality evaluation model.
10. A lightweight image quality evaluation terminal comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor is adapted to perform the method of any of claims 1-8 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311389543.2A CN117372387A (en) | 2023-10-24 | 2023-10-24 | Lightweight image quality evaluation method and system based on convolutional neural network pruning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311389543.2A CN117372387A (en) | 2023-10-24 | 2023-10-24 | Lightweight image quality evaluation method and system based on convolutional neural network pruning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117372387A true CN117372387A (en) | 2024-01-09 |
Family
ID=89407448
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311389543.2A Pending CN117372387A (en) | 2023-10-24 | 2023-10-24 | Lightweight image quality evaluation method and system based on convolutional neural network pruning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117372387A (en) |
-
2023
- 2023-10-24 CN CN202311389543.2A patent/CN117372387A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108509978B (en) | Multi-class target detection method and model based on CNN (CNN) multi-level feature fusion | |
| Wang et al. | Detect globally, refine locally: A novel approach to saliency detection | |
| CN109754391B (en) | Image quality evaluation method and device and electronic equipment | |
| CN112927209B (en) | CNN-based significance detection system and method | |
| CN107945210B (en) | Target tracking method based on deep learning and environment self-adaption | |
| CN111967464B (en) | Weak supervision target positioning method based on deep learning | |
| CN113420651A (en) | Lightweight method and system of deep convolutional neural network and target detection method | |
| CN106339753A (en) | Method for effectively enhancing robustness of convolutional neural network | |
| US20220415007A1 (en) | Image normalization processing | |
| CN111127354A (en) | Single-image rain removing method based on multi-scale dictionary learning | |
| CN113066089B (en) | Real-time image semantic segmentation method based on attention guide mechanism | |
| CN115035418A (en) | Remote sensing image semantic segmentation method and system based on improved deep LabV3+ network | |
| CN114266894A (en) | Image segmentation method and device, electronic equipment and storage medium | |
| CN113159236A (en) | Multi-focus image fusion method and device based on multi-scale transformation | |
| CN110264407A (en) | Image Super-resolution model training and method for reconstructing, device, equipment and storage medium | |
| CN111461211B (en) | Feature extraction method for lightweight target detection and corresponding detection method | |
| Young et al. | Feature-align network with knowledge distillation for efficient denoising | |
| CN113962905A (en) | Single image rain removing method based on multi-stage feature complementary network | |
| CN112465847A (en) | Edge detection method, device and equipment based on clear boundary prediction | |
| CN117455757A (en) | Image processing method, device, equipment and storage medium | |
| CN116758092A (en) | Image segmentation method, device, electronic equipment and storage medium | |
| US20230073175A1 (en) | Method and system for processing image based on weighted multiple kernels | |
| CN115937121A (en) | Non-reference image quality evaluation method and system based on multi-dimensional feature fusion | |
| CN117372387A (en) | Lightweight image quality evaluation method and system based on convolutional neural network pruning | |
| CN115661618A (en) | Training method of image quality evaluation model, image quality evaluation method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |