CN116992944A - Image processing method and device based on leavable importance judging standard pruning - Google Patents
Image processing method and device based on leavable importance judging standard pruning Download PDFInfo
- Publication number
- CN116992944A CN116992944A CN202311257199.1A CN202311257199A CN116992944A CN 116992944 A CN116992944 A CN 116992944A CN 202311257199 A CN202311257199 A CN 202311257199A CN 116992944 A CN116992944 A CN 116992944A
- Authority
- CN
- China
- Prior art keywords
- importance
- image processing
- search space
- pruning
- processing model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000013138 pruning Methods 0.000 title claims abstract description 74
- 238000003672 processing method Methods 0.000 title claims abstract description 14
- 238000012545 processing Methods 0.000 claims abstract description 76
- 238000000034 method Methods 0.000 claims abstract description 34
- 238000012549 training Methods 0.000 claims description 20
- 238000009826 distribution Methods 0.000 claims description 14
- 238000003860 storage Methods 0.000 claims description 14
- 238000011156 evaluation Methods 0.000 claims description 12
- 230000008569 process Effects 0.000 claims description 11
- 230000006870 function Effects 0.000 claims description 7
- 238000005070 sampling Methods 0.000 claims description 7
- 230000004913 activation Effects 0.000 claims description 6
- 230000008859 change Effects 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 4
- 238000009966 trimming Methods 0.000 claims description 4
- 238000010200 validation analysis Methods 0.000 claims description 4
- 238000009827 uniform distribution Methods 0.000 claims description 3
- 230000006835 compression Effects 0.000 abstract description 5
- 238000007906 compression Methods 0.000 abstract description 5
- 230000001133 acceleration Effects 0.000 abstract description 3
- 239000010410 layer Substances 0.000 description 52
- 238000001514 detection method Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 238000013527 convolutional neural network Methods 0.000 description 5
- 238000013135 deep learning Methods 0.000 description 5
- 210000002569 neuron Anatomy 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 238000005520 cutting process Methods 0.000 description 4
- 230000011218 segmentation Effects 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 3
- 238000005457 optimization Methods 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 239000011229 interlayer Substances 0.000 description 2
- 238000013140 knowledge distillation Methods 0.000 description 2
- 239000012633 leachable Substances 0.000 description 2
- 238000013139 quantization Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Biophysics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The application discloses an image processing method and device based on a leavable importance judging standard pruning, belonging to the field of artificial intelligent model compression and reasoning acceleration, wherein the method comprises the following steps: acquiring an image processing model to be pruned; adding an importance judgment standard search space in forward propagation of the image processing model to be pruned, wherein the importance judgment standard search space is used for calculating the part of each layer of convolution layer in the image processing model to be pruned, which needs pruning; back-propagating based on the importance judgment standard search space and updating parameters of the importance judgment standard search space; and fine tuning is carried out on the model pruned based on the final judgment standard.
Description
Technical Field
The application belongs to the technical field of artificial intelligence, and particularly relates to an image processing method and device based on a leavable importance judging standard pruning, aiming at model compression and reasoning acceleration direction research.
Background
In recent years, with the development of deep learning, convolutional neural networks (Convolutional Neural Networks, CNN) perform surprisingly well in visual target detection, segmentation, recognition, and classification tasks. The convolutional neural network comprises a feature extractor consisting of a convolutional layer and a sub-sampling layer, the input image is subjected to convolutional extraction of local features by a filter, and then a feature mapping layer is obtained by the sampling layer. Compared with the traditional neural network, the convolutional neural network simplifies network parameters through local receptive fields, weight sharing and pooling layers, but as the network layer number is deepened, the parameters of the network are continuously enlarged, for example, googleNet has 6.7M parameters, alexNet has 60M parameters and VGG16 has 138M parameters. Various acceleration modes are mainly divided into: first, pruning, deleting unimportant parameters in the network, such as weight pruning, channel pruning, convolution kernel pruning, neuron pruning. Pruning can ensure precision and reduce the parameter quantity at the same time, but the consumed time and the calculated amount are large; second, quantization, converting floating point operation into fixed point operation. The quantization can reduce the volume of the model with less loss of precision, but the accuracy is unstable and the universality is poor; knowledge distillation, large network guidance training generates a small network. The knowledge distillation training is simple, but the design migration mode is difficult; in order to reduce the size of a network model in image processing and to reduce the computational complexity in image processing, network compression represented by model pruning is now widely used in image processing tasks.
Among the convolution kernel pruning, it can be classified into hard pruning, where the last round of deleted convolution kernel does not participate in iteration in the next round, soft pruning, and where the last round of deleted convolution kernel still participates in iteration in the next round. However, they all have a common disadvantage that different layers of the network adopt the same importance criterion, and cannot fully consider different distributions of features of different layers. How to select the appropriate importance criteria should be paid attention.
On the one hand, the importance of the measured neuron can be calculated based on the weight standard, the output of the characteristic is the weight of the multiplication of the input and the weight, and the smaller the weight is, the smaller the contribution to the output is. By carrying out the weight amplitude valueSorting, removing the connection lower than the preset value, and obtaining the pruned network. Based onJudging the importance of the convolution kernel; geometric Median (GM) is not considered important by pruning redundant convolution kernels instead of those with relatively low weights, which are considered too similar to other convolution kernels; on the other hand, neurons with minimal loss effects can be found based on the loss function. Performing taylor expansion on the activation value, calculating a gradient product of a convolution kernel and each weight, and summing to obtain an average value; squaring the spread of the weights; the activation value may also be used as a metric. Most neurons with activation values approaching 0 are redundant, and the removal of the part of neurons can greatly reduce the size and the operand of the model without affecting the model precision. APoZ calculates redundancy and clipping space of the proportional estimate model of each layer 0; clipping is also performed based on the information entropy. However, there is still relatively little research on whether the model as a whole uses a criterion, requiring further research.
Disclosure of Invention
Aiming at the defects of the prior art, the embodiment of the application aims to provide an image processing method and device based on a leavable importance judgment standard pruning, and a selectable importance judgment search space is added to the traditional soft pruning, so that each layer can select a proper judgment standard. In the leachable pruning, each layer "learns" to select the most appropriate criterion, so that the network obtains better pruning, and the running memory is reduced when the image processing task is performed.
According to a first aspect of an embodiment of the present application, there is provided an image processing method based on a leavable importance assessment standard pruning, including:
acquiring an image processing model to be pruned;
adding an importance judgment standard search space in forward propagation of the image processing model to be pruned, wherein the importance judgment standard search space is used for calculating the part of each layer of convolution layer in the image processing model to be pruned, which needs pruning;
back-propagating based on the importance judgment standard search space and updating parameters of the importance judgment standard search space;
trimming the image processing model after pruning based on the final judgment standard;
and deploying the trimmed image processing model on a terminal so as to process the image through the trimmed image processing model.
Further, the image processing model to be pruned comprises L layers of convolution layers and marks the convolution kernel information of the L layers.
Further, in the importance criterion search space:
by calculating the probability of selecting the ith importance criterionTo select criteria for the layer l convolution layer:
，
wherein ,importance criterion search space representing a layer I convolution layer>An importance criterion superparameter +_>，/>Representing the number of importance criteria;
and processing the probability p by adopting a gumme-Softmax discrete distribution sampling method:
，
，
wherein Is a uniform distribution of 0 to 1, +.>For one of the samples->Random variable noise representing independent distribution, +.>Is the temperature coefficient of softmax, when +.>When changing from big to small->One element will be chosen from evenly distributed to fixed.
Further, the number s=5 of the importance evaluation criteria, and the 5 importance evaluation criteria are respectively: (1)Criteria for evaluation, (2)/(2)>The method comprises the following steps of (1) judging the standard, (3) the geometric median judging standard, (4) the judging standard capable of removing loss change caused by partial feature mapping by calculating the change of the loss function before and after the pruning is expanded by using the Taylor approximation, and (5) the APoZ activation value is selected as the judging standard.
Further, after adding an importance judgment standard search space in the forward propagation of the image processing model to be pruned, the output feature graphs of different importance judgment standards are selected from the importance judgment standard search space to be aligned.
Further, the output feature graphs of different importance judgment standards are selected from the importance judgment standard search space to be aligned, specifically:
，
wherein For the aligned feature map, +.>For selecting the probability of the ith importance criterion +.>To select the ith evaluation standard, the feature map is output after pruning, and the item is added with->The operation sums the feature maps that remain the same location.
Further, back-propagating and updating parameters of the importance criterion search space based on the importance criterion search space includes:
for the followingImage processing model of layer, loss by minimizing criterion +.>Training the importance criterion search space +.>；
After the importance criteria search space training is completed, by minimizing the loss on the validation set in the datasetTo calculate the structural parameters +.>：
，
，
，
wherein Is the adjusted criterion search space, < ->Is a set of parameters of the criteria of the judgment,is a loss of criterion search space, +.>Is the computational penalty in pruning the network, +.>Is balance-> and />Is a weight of (2).
According to a second aspect of an embodiment of the present application, there is provided an image processing apparatus based on a leavable importance assessment criterion pruning, including:
the acquisition module is used for acquiring an image processing model to be pruned;
the adding module is used for adding an importance judgment standard search space in forward propagation of the image processing model to be pruned, wherein the importance judgment standard search space is used for calculating the part of each layer of convolution layer in the image processing model to be pruned, which needs pruning;
the updating module is used for carrying out back propagation based on the importance judgment standard search space and updating parameters of the importance judgment standard search space;
the fine adjustment module is used for fine adjustment of the image processing model based on the final judgment standard pruning;
the deployment module is used for deploying the trimmed image processing model to the terminal so as to process the image through the trimmed image processing model.
According to a third aspect of an embodiment of the present application, there is provided an electronic apparatus including:
one or more processors;
a memory for storing one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of the first aspect.
According to a fourth aspect of embodiments of the present application there is provided a computer readable storage medium having stored thereon computer instructions which when executed by a processor perform the steps of the method according to the first aspect.
The technical scheme provided by the embodiment of the application can comprise the following beneficial effects:
according to the embodiment, the distribution difference problem in the deep learning network layer is considered, different pruning judgment standards are selected for different layers, and optimization of model pruning is realized by combining the advantages of the different judgment standards; according to the application, the overall interlayer cutting result is considered, and the synergy of each layer of the network to the final result is considered not only in a greedy independent cutting mode; according to the application, the accuracy of the network can be maximally reserved in the process of pruning the network by selecting the optimal pruning judgment standard. The method provided by the application can better eliminate unnecessary values in the weight tensor in various deep learning networks, reduce the number of connections between the neural network layers and reduce the parameters involved in calculation, thereby reducing the operation times. In particular, tasks involving large images, such as semantic segmentation or object detection, during image processing, intermediate representations may consume a significant amount of memory, far beyond the network itself. Pruning can reduce the storage space of the model, reduce the intermediate calculation generation diagram and reduce the memory required during operation.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.
Fig. 1 is a structural overview diagram showing an image processing method based on a leavable importance judging standard pruning according to an exemplary embodiment.
Fig. 2 is a flowchart illustrating an image processing method based on a leavable importance criterion pruning according to an exemplary embodiment.
Fig. 3 is an importance criterion spatial flowchart illustrating an image processing method based on a leachable importance criterion pruning according to an exemplary embodiment.
Fig. 4 is a block diagram showing an image processing apparatus based on a leavable importance assessment criterion pruning according to an exemplary embodiment.
Fig. 5 is a schematic diagram of an electronic device, according to an example embodiment.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the application.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any or all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used herein to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the application. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "responsive to a determination", depending on the context.
As shown in fig. 1 and fig. 2, the image processing method based on the leavable importance judging standard pruning provided by the application may include the following steps:
step S11: acquiring an image processing model to be pruned;
step S12: adding an importance judgment standard search space in forward propagation of the image processing model to be pruned, wherein the importance judgment standard search space is used for calculating the part of each layer of convolution layer in the image processing model to be pruned, which needs pruning;
step S13: back-propagating based on the importance judgment standard search space and updating parameters of the importance judgment standard search space;
step S14: trimming the image processing model after pruning based on the final judgment standard;
step S15: and deploying the trimmed image processing model on a terminal so as to process the image through the trained model.
According to the embodiment, the distribution difference problem in the deep learning network layer is considered, different pruning judgment standards are selected for different layers, and optimization of model pruning is realized by combining the advantages of the different judgment standards; according to the application, the overall interlayer cutting result is considered, and the synergy of each layer of the network to the final result is considered not only in a greedy independent cutting mode; according to the application, the accuracy of the network can be maximally reserved in the process of pruning the network by selecting the optimal pruning judgment standard. The method provided by the application can better eliminate unnecessary values in the weight tensor in various deep learning networks, reduce the number of connections between the neural network layers and reduce the parameters involved in calculation, thereby reducing the operation times. In particular, tasks involving large images, such as semantic segmentation or object detection, during image processing, intermediate representations may consume a significant amount of memory, far beyond the network itself. Pruning can reduce the storage space of the model, reduce the intermediate calculation generation diagram and reduce the memory required during operation.
In the implementation of step S11, an image processing model to be pruned is obtained;
in particular, a pre-trained model with parameters is obtained that achieves advanced performance in image processing.
In specific implementation, pretrained network models such as VGG16, resnet32, resnet56, resNet50 and the like can be adopted in the classification task, and a YOLO series pretrained model can be adopted in the target detection task. The pre-training model with parameters comprisesLayer convolution layer and label->The convolution kernel information of the layer. Acquiring all filter sets in an image processing model to be pruned>Scoring according to each layer of judgment standard in pruning process, and collecting filter>Is divided into reserved sets->And filter set pruned +.>。
In particular implementations, training data sets employed in verifying pruning effects and fine-tuning model loss in image processing task adoptionMay include:
classifying the data set: CIFAR-10, CIFAR-100 and ILSVRC-2012 data sets, wherein the CIFAR-10 data sets comprise 5 ten thousand training images and 1 ten thousand test images, and 6 ten thousand 32X 32 color images are divided into 10 different categories. CIFAR-100 has 100 classes, and the number of images is the same as CIFAR-10. ILSVRC2012 contains 128 tens of thousands of training images and 50 tens of thousands of 1000 classes of verification images;
semantic segmentation and object detection dataset: VOC 2012 and COCO dataset, VOC 2012 contains 17125 pictures with 11540 for the detection task. The COCO dataset is another common large-scale dataset for target detection, comprising 80 categories of large-scale datasets, the data of which are divided into three parts: training, validation and testing, each section contained 118287, 5000 and 40670 pictures, respectively.
In the implementation of step S12, adding an importance criterion search space in the forward propagation of the image processing model to be pruned, where the importance criterion search space is used to calculate a portion of each layer of convolution layers in the image processing model to be pruned, where pruning is required;
specifically, in the importance criterion search space:
by calculating the probability of selecting the ith importance criterionTo select criteria for the layer l convolution layer:
，
wherein ,importance criterion search space representing a layer I convolution layer>An importance criterion superparameter +_>，/>Representing the number of importance criteria;
and processing the probability p by adopting a gumme-Softmax discrete distribution sampling method:
，
，
wherein Is a uniform distribution of 0 to 1, +.>For one of the samples->The random variable noise representing independent distribution can obtain distribution approaching one-hot, and meanwhile, the noise also keeps the sampling probability consistent with the original distribution. />Is the temperature coefficient of softmax, when +.>When changing from big to small->One element will be chosen from evenly distributed to fixed. In a specific implementation, the temperature coefficient +.>Is set to linearly decrease from 5 to 0.1 to gradually approach the true discrete distribution.
Is sampling in distributed discrete data, the process is not conductive, guble-Softmax is a continuous distribution of estimated samples from class distribution, after Guble-Softmax re-referencing, the->The gradient can be propagated by direction to +.>Achieving the purpose of learning.
Wherein the number s=5 of the importance evaluation criteria, and the 5 importance evaluation criteria are respectively: (1)Criteria for evaluation, (2)/(2)>The method comprises the following steps of (1) judging the standard, (3) the geometric median judging standard, (4) the judging standard capable of removing loss change caused by partial feature mapping by calculating the change of the loss function before and after the pruning is expanded by using the Taylor approximation, and (5) the APoZ activation value is selected as the judging standard.
The forward propagation of the importance criteria search space can be expressed as:
wherein Function representing pruning given +.>Return->, wherein />Represents +.o under the S importance criterion>Layer-retained filter set, +.>Representation->Layer filter set, filter set removed->，Representing the judgment standard space, & lt & gt>Representing the compression ratio.
As shown in fig. 3, according to importance judgment standards of different probabilities, pruning is performed on the filter with smaller importance score according to the importance score of each importance judgment standard to obtain pruned layers with different results, and in order to comprehensively consider the sharing of the results of different judgment standards in the training process, the output feature map is defined as an alignment weighted sum of the feature maps of the pruned layers:
，
wherein For the aligned feature map, +.>For selecting the probability of the ith importance criterion +.>To select the ith evaluation standard, the feature map is output after pruning, and the item is added with->The operation sums the feature maps that remain the same location.
In the implementation of step S13, the parameters of the importance criterion search space are back-propagated and updated based on the importance criterion search space;
in particular toThe image processing model of the layer evaluates the standard parameter set and finds out the appropriate structural parameter +.>And guiding to select the optimal judgment standard. Loss by minimizing criteria for evaluation>Training the importance criterion search space +.>Loss on minimum validation set +.>Calculating structural parameters +.>：
，
，
，
wherein Is the adjusted criterion search space, < ->Is a set of parameters of the criteria of the judgment,is a loss of criterion search space, +.>Is the computational penalty in pruning the network, +.>Is balance-> and />Weight of (2) will +.>Set to 2
By minimizing network loss over training setsObtain the optimal structural parameters->Adam's optimization function is specifically employed in one embodiment and uses a constant learning rate of 0.001 and a weight decay of 0.001. On CIFAR, training was performed for 600 periods, with a batch size of 256. On ILSVRC-2012, training was performed for 35 periods, with a batch size of 256, and the weights of the pre-training model were not updated during training to reduce overfitting.
Finally, according to the S parameter of the final important judgment space after trainingSelecting the judgment standard of the maximum probability, wherein the finally selected judgment standard of each layer forms a judgment standard set suitable for each layer>According to the criterion set ∈ ->And ranking the filters of the network layer, removing the filters with lower ranks according to the compression rate n, and finishing final pruning.
In the specific implementation of step S14, fine tuning is performed on the image processing model pruned based on the final criterion; specifically, the network after final pruning has precision loss, needs fine adjustment, reduces learning rate in the specific implementation process, and is in the data setTraining the network after trimming pruning again; the reduced learning rate is set to be 1/10 of the initial in the specific embodiment.
In the implementation of step S15, the trimmed image processing model is deployed at a terminal, so as to perform image processing through the trained model;
in this embodiment, the image processing model after pruning and fine tuning is deployed to a terminal to perform image classification.
Corresponding to the embodiment of the image processing method based on the learning importance assessment standard pruning, the application also provides an embodiment of the image processing device based on the learning importance assessment standard pruning.
Fig. 4 is a block diagram of an image processing apparatus based on a leavable importance assessment criterion pruning, which is shown according to an exemplary embodiment. Referring to fig. 4, the apparatus may include:
an acquisition module 21, configured to acquire an image processing model to be pruned;
the adding module 22 is configured to add an importance criterion search space in forward propagation of the image processing model to be pruned, where the importance criterion search space is used to calculate a portion of each layer of convolution layer in the image processing model to be pruned, where pruning is required;
an updating module 23, configured to back-propagate based on the importance criterion search space and update parameters of the importance criterion search space;
a fine tuning module 24, configured to fine tune the image processing model pruned based on the final criterion;
the deployment module 25 is configured to deploy the trimmed image processing model to a terminal, so as to perform image processing through the trained model.
The specific manner in which the various modules perform the operations in the apparatus of the above embodiments have been described in detail in connection with the embodiments of the method, and will not be described in detail herein.
For the device embodiments, reference is made to the description of the method embodiments for the relevant points, since they essentially correspond to the method embodiments. The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purposes of the present application. Those of ordinary skill in the art will understand and implement the present application without undue burden.
Correspondingly, the application also provides electronic equipment, which comprises: one or more processors; a memory for storing one or more programs; the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method for processing images based on the criterion pruning for learning importance as described above. As shown in fig. 5, a hardware structure diagram of an arbitrary device with data processing capability according to the image processing method based on the learning importance evaluation standard pruning according to the embodiment of the present application is shown in fig. 5, and in addition to the processor, the memory, the DMA controller, the magnetic disk, and the nonvolatile memory shown in fig. 5, the arbitrary device with data processing capability according to the embodiment of the present application generally includes other hardware according to the actual function of the arbitrary device with data processing capability, which is not described herein again.
Correspondingly, the application also provides a computer readable storage medium, wherein computer instructions are stored on the computer readable storage medium, and the instructions are executed by a processor to realize the image processing method based on the learning importance judging standard pruning. The computer readable storage medium may be an internal storage unit, such as a hard disk or a memory, of any of the data processing enabled devices described in any of the previous embodiments. The computer readable storage medium may also be an external storage device of the wind turbine generator, such as a plug-in hard disk, a Smart Media Card (SMC), an SD Card, a Flash memory Card (Flash Card), etc. provided on the device. Further, the computer readable storage medium may include both internal storage units and external storage devices of any device having data processing capabilities. The computer readable storage medium is used for storing the computer program and other programs and data required by the arbitrary data processing apparatus, and may also be used for temporarily storing data that has been output or is to be output.
Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains.
It is to be understood that the application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof.
Claims (10)
1. An image processing method based on a leavable importance criterion pruning is characterized by comprising the following steps:
acquiring an image processing model to be pruned;
adding an importance judgment standard search space in forward propagation of the image processing model to be pruned, wherein the importance judgment standard search space is used for calculating the part of each layer of convolution layer in the image processing model to be pruned, which needs pruning;
back-propagating based on the importance judgment standard search space and updating parameters of the importance judgment standard search space;
trimming the image processing model after pruning based on the final judgment standard;
and deploying the trimmed image processing model on a terminal so as to process the image through the trimmed image processing model.
2. The method according to claim 1, wherein the image processing model to be pruned comprises L layers of convolution layers and labels the L layers of convolution kernel information.
3. The method of claim 1, wherein in the importance criteria search space:
by calculating the probability of selecting the ith importance criterionTo select criteria for the layer l convolution layer:
，
wherein ,importance criterion search space representing a layer I convolution layer>The importance judgment standard exceeds the parameters,，/>representing the number of importance criteria;
and processing the probability p by adopting a gumme-Softmax discrete distribution sampling method:
，
，
wherein Is a uniform distribution of 0 to 1, +.>For one of the samples->Random variable noise representing independent distribution, +.>Is the temperature coefficient of softmax, when +.>When changing from big to small->One element will be chosen from evenly distributed to fixed.
4. A method according to claim 3, wherein the number of importance criteria S = 5,5 of saidThe importance judgment standards are respectively as follows: (1)Criteria for evaluation, (2)/(2)>The method comprises the following steps of (1) judging the standard, (3) the geometric median judging standard, (4) the judging standard capable of removing loss change caused by partial feature mapping by calculating the change of the loss function before and after the pruning is expanded by using the Taylor approximation, and (5) the APoZ activation value is selected as the judging standard.
5. The method according to claim 1, wherein after adding an importance criterion search space in the forward propagation of the image processing model to be pruned, the output feature maps of different importance criteria are selected in the importance criterion search space for alignment.
6. The method according to claim 5, wherein the output feature graphs of the different importance criteria are selected in the importance criteria search space for alignment, specifically:
，
wherein For the aligned feature map, +.>For selecting the probability of the ith importance criterion +.>To select the ith evaluation standard, the feature map is output after pruning, and the item is added with->The operation sums the feature maps that remain the same location.
7. The method of claim 1, wherein back-propagating and updating parameters of the importance criteria search space based on the importance criteria search space comprises:
for the followingImage processing model of layer, loss by minimizing criterion +.>Training the importance criterion search space +.>；
After the importance criteria search space training is completed, by minimizing the loss on the validation set in the datasetTo calculate the structural parameters +.>：
，
，
，
wherein Is the adjusted judgment standardSearch space->Is a set of criterion parameters, < >>Is a loss of criterion search space, +.>Is the computational penalty in pruning the network, +.>Is balance-> and />Is a weight of (2).
8. An image processing apparatus based on a leavable importance criterion pruning, comprising:
the acquisition module is used for acquiring an image processing model to be pruned;
the adding module is used for adding an importance judgment standard search space in forward propagation of the image processing model to be pruned, wherein the importance judgment standard search space is used for calculating the part of each layer of convolution layer in the image processing model to be pruned, which needs pruning;
the updating module is used for carrying out back propagation based on the importance judgment standard search space and updating parameters of the importance judgment standard search space;
the fine adjustment module is used for fine adjustment of the image processing model based on the final judgment standard pruning;
the deployment module is used for deploying the trimmed image processing model to the terminal so as to process the image through the trimmed image processing model.
9. An electronic device, comprising:
one or more processors;
a memory for storing one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-7.
10. A computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the steps of the method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311257199.1A CN116992944B (en) | 2023-09-27 | 2023-09-27 | Image processing method and device based on leavable importance judging standard pruning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311257199.1A CN116992944B (en) | 2023-09-27 | 2023-09-27 | Image processing method and device based on leavable importance judging standard pruning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116992944A true CN116992944A (en) | 2023-11-03 |
| CN116992944B CN116992944B (en) | 2023-12-19 |
Family
ID=88525208
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311257199.1A Active CN116992944B (en) | 2023-09-27 | 2023-09-27 | Image processing method and device based on leavable importance judging standard pruning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116992944B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3876166A2 (en) * | 2020-10-23 | 2021-09-08 | Beijing Baidu Netcom Science And Technology Co. Ltd. | Method and apparatus for determining network model pruning strategy, device and storage medium |
| CN114170512A (en) * | 2021-12-08 | 2022-03-11 | 西安电子科技大学 | Remote sensing SAR target detection method based on combination of network pruning and parameter quantification |
| CN114330713A (en) * | 2022-01-11 | 2022-04-12 | 平安科技（深圳）有限公司 | Convolutional neural network model pruning method and device, electronic equipment and storage medium |
| CN114358257A (en) * | 2022-02-21 | 2022-04-15 | Oppo广东移动通信有限公司 | Neural network pruning method and device, readable medium and electronic equipment |
| CN114581868A (en) * | 2022-03-04 | 2022-06-03 | 京东鲲鹏(江苏)科技有限公司 | Image analysis method and device based on model channel pruning |
| WO2023024407A1 (en) * | 2021-08-24 | 2023-03-02 | 平安科技（深圳）有限公司 | Model pruning method and apparatus based on adjacent convolutions, and storage medium |
| WO2023159760A1 (en) * | 2022-02-22 | 2023-08-31 | 平安科技（深圳）有限公司 | Convolutional neural network model pruning method and apparatus, electronic device, and storage medium |
-
2023
- 2023-09-27 CN CN202311257199.1A patent/CN116992944B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3876166A2 (en) * | 2020-10-23 | 2021-09-08 | Beijing Baidu Netcom Science And Technology Co. Ltd. | Method and apparatus for determining network model pruning strategy, device and storage medium |
| WO2023024407A1 (en) * | 2021-08-24 | 2023-03-02 | 平安科技（深圳）有限公司 | Model pruning method and apparatus based on adjacent convolutions, and storage medium |
| CN114170512A (en) * | 2021-12-08 | 2022-03-11 | 西安电子科技大学 | Remote sensing SAR target detection method based on combination of network pruning and parameter quantification |
| CN114330713A (en) * | 2022-01-11 | 2022-04-12 | 平安科技（深圳）有限公司 | Convolutional neural network model pruning method and device, electronic equipment and storage medium |
| WO2023134086A1 (en) * | 2022-01-11 | 2023-07-20 | 平安科技（深圳）有限公司 | Convolutional neural network model pruning method and apparatus, and electronic device and storage medium |
| CN114358257A (en) * | 2022-02-21 | 2022-04-15 | Oppo广东移动通信有限公司 | Neural network pruning method and device, readable medium and electronic equipment |
| WO2023159760A1 (en) * | 2022-02-22 | 2023-08-31 | 平安科技（深圳）有限公司 | Convolutional neural network model pruning method and apparatus, electronic device, and storage medium |
| CN114581868A (en) * | 2022-03-04 | 2022-06-03 | 京东鲲鹏(江苏)科技有限公司 | Image analysis method and device based on model channel pruning |
Non-Patent Citations (4)
| Title |
|---|
| 卢海伟;袁晓彤;: "基于层融合特征系数的动态网络结构化剪枝", 模式识别与人工智能, no. 11 * |
| 尹文枫;梁玲燕;彭慧民;曹其春;赵健;董刚;赵雅倩;赵坤;: "卷积神经网络压缩与加速技术研究进展", 计算机系统应用, no. 09 * |
| 徐嘉荟;: "基于模型剪枝的神经网络压缩技术研究", 信息通信, no. 12 * |
| 马治楠;韩云杰;彭琳钰;周进凡;林付春;刘宇红;: "基于深层卷积神经网络的剪枝优化", 电子技术应用, no. 12 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116992944B (en) | 2023-12-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113378632B (en) | Pseudo-label optimization-based unsupervised domain adaptive pedestrian re-identification method | |
| CN107480261B (en) | Fine-grained face image fast retrieval method based on deep learning | |
| CN106228185B (en) | A kind of general image classifying and identifying system neural network based and method | |
| CN111191583B (en) | Space target recognition system and method based on convolutional neural network | |
| CN110929610B (en) | Plant disease identification method and system based on CNN model and transfer learning | |
| CN110728224A (en) | Remote sensing image classification method based on attention mechanism depth Contourlet network | |
| CN111723915B (en) | Target detection method based on deep convolutional neural network | |
| CN114037844A (en) | Global rank perception neural network model compression method based on filter characteristic diagram | |
| CN111696101A (en) | Light-weight solanaceae disease identification method based on SE-Inception | |
| CN113657421B (en) | Convolutional neural network compression method and device, and image classification method and device | |
| CN109740734B (en) | Image classification method of convolutional neural network by optimizing spatial arrangement of neurons | |
| CN111178525A (en) | Pruning-based convolutional neural network compression method, system and medium | |
| CN113837376B (en) | Neural network pruning method based on dynamic coding convolution kernel fusion | |
| CN111694977A (en) | Vehicle image retrieval method based on data enhancement | |
| CN115035418A (en) | Remote sensing image semantic segmentation method and system based on improved deep LabV3+ network | |
| CN114819061A (en) | Sparse SAR target classification method and device based on transfer learning | |
| CN110096976A (en) | Human behavior micro-Doppler classification method based on sparse migration network | |
| CN114819143A (en) | Model compression method suitable for communication network field maintenance | |
| CN112132062B (en) | Remote sensing image classification method based on pruning compression neural network | |
| CN114065831A (en) | Hyperspectral image classification method based on multi-scale random depth residual error network | |
| CN117217282A (en) | Structured pruning method for deep pedestrian search model | |
| CN112651499A (en) | Structural model pruning method based on ant colony optimization algorithm and interlayer information | |
| CN117034060A (en) | AE-RCNN-based flood classification intelligent forecasting method | |
| CN116992944B (en) | Image processing method and device based on leavable importance judging standard pruning | |
| CN111783688A (en) | Remote sensing image scene classification method based on convolutional neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |