CN116882461A - Neural network evaluation optimization method and system based on neuron plasticity - Google Patents
Neural network evaluation optimization method and system based on neuron plasticity Download PDFInfo
- Publication number
- CN116882461A CN116882461A CN202311118140.4A CN202311118140A CN116882461A CN 116882461 A CN116882461 A CN 116882461A CN 202311118140 A CN202311118140 A CN 202311118140A CN 116882461 A CN116882461 A CN 116882461A
- Authority
- CN
- China
- Prior art keywords
- neural network
- neuron
- deep neural
- compression
- plasticity
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 210000002569 neuron Anatomy 0.000 title claims abstract description 258
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 218
- 238000005457 optimization Methods 0.000 title claims abstract description 59
- 238000000034 method Methods 0.000 title claims abstract description 53
- 238000011156 evaluation Methods 0.000 title claims abstract description 38
- 230000006835 compression Effects 0.000 claims abstract description 79
- 238000007906 compression Methods 0.000 claims abstract description 79
- 238000012549 training Methods 0.000 claims abstract description 30
- 238000012545 processing Methods 0.000 claims abstract description 20
- 230000001537 neural effect Effects 0.000 claims abstract description 17
- 238000012795 verification Methods 0.000 claims abstract description 14
- 210000005036 nerve Anatomy 0.000 claims abstract description 4
- 230000003993 interaction Effects 0.000 claims description 26
- 238000004364 calculation method Methods 0.000 claims description 24
- 238000005070 sampling Methods 0.000 claims description 14
- 230000000007 visual effect Effects 0.000 claims description 12
- 238000013527 convolutional neural network Methods 0.000 claims description 11
- 238000004422 calculation algorithm Methods 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 3
- 238000010276 construction Methods 0.000 claims description 3
- 238000013461 design Methods 0.000 abstract description 3
- 230000000694 effects Effects 0.000 description 10
- 238000013135 deep learning Methods 0.000 description 7
- 210000004556 brain Anatomy 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000009191 jumping Effects 0.000 description 4
- 230000003068 static effect Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000010365 information processing Effects 0.000 description 3
- 239000004433 Thermoplastic polyurethane Substances 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000005265 energy consumption Methods 0.000 description 2
- 238000013140 knowledge distillation Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000013138 pruning Methods 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 229920002803 thermoplastic polyurethane Polymers 0.000 description 2
- 102100024607 DNA topoisomerase 1 Human genes 0.000 description 1
- 101000830681 Homo sapiens DNA topoisomerase 1 Proteins 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000012804 iterative process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0495—Quantised networks; Sparse networks; Compressed networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a neural network evaluation optimization method and system based on neuron plasticity, comprising deep neural network structure compression and deep neural network parameter optimization: acquiring a verification data set, a training data set, a trained deep neural network to be compressed and parameters; calculating the plasticity and the neuron importance of a neuron cluster in the deep neural network to be compressed; calculating the number of necessary neurons in the neuron clusters; acquiring an unoptimized compressed deep neural network according to the number of necessary neurons and the importance of each neuron; calculating the neural micro-loop modeling difference of the depth neural network before and after compression; optimizing the network weight according to the modeling difference of the nerve micro-loops before and after compression to obtain an optimized compressed deep neural network; the invention designs a lightweight deep neural network and applies the lightweight deep neural network to natural image classification, accords with biology and has the characteristics of stronger interpretation, low computing resource cost and high image processing precision.
Description
Technical Field
The invention relates to the technical field of deep learning and image processing, in particular to a neural network evaluation optimization method and system based on neuron plasticity.
Background
The aim of applying artificial intelligence technology in the technical field of image processing is to realize intelligent image processing, however, a deep learning algorithm based on data driving can only simply simulate a human brain thinking process by increasing the quantity and complexity of modeling parameters, and cannot realize low energy consumption equivalent to human brain when similar tasks are processed.
The theory of Hubby in computational neuroscience indicates the plasticity of biological neurons, and a large number of deep neural network explanatory researches indicate that neurons in the existing deep neural network also have the plasticity by taking a brain information processing mechanism as a theoretical basis of network structure design: while deep neural networks learn information processing capabilities on specific tasks, multiple neurons in the network also commonly undergo modeling, and in order to achieve low energy consumption, the human brain tends to use fewer neurons for information processing after the neuron modeling is completed (task processing can be performed "proficiently").
However, the structure of the conventional general convolutional neural network is fixed, and the neuron activity cannot be actively restrained, so that the same operation amount as that when the 'sparse' image processing task is finished still needs to be consumed.
In addition, in order to deploy the general convolutional neural network with high precision and high operation amount in the embedded system, the structured pruning technology can greatly reduce the operation cost and ensure the precision of the compressed deep neural network through the width of the compressed network architecture, so that the method has wide application in the embedded environment, and the evaluation of the redundant structure is the core of the structured pruning technology.
However, the existing evaluation method has the problems of inaccurate static evaluation and large cost in dynamic evaluation, and is difficult to realize efficient and accurate redundant filter positioning: on one hand, the reason for inaccurate static evaluation is that the existing static evaluation method of the redundant filter only considers the inherent attribute (such as weight or characteristic) of a single filter, but ignores that the inherent attribute of the filter is the result of common modeling of neuron clusters, and as the compression ratio increases, the static evaluation method inevitably deletes neurons with forward contribution to other neuron function modeling and lower evaluation importance, thereby reducing the modeling effect of a neural micro-loop, so that the accuracy of a compressed deep neural network cannot be ensured; on the other hand, the reason that the dynamic evaluation accuracy is high but the operation cost is high is that the existing redundancy filter dynamic evaluation method carries out redundancy neuron prediction by embedding an additional operation structure in a network structure or designing an additional network, so that neurons with low importance but capable of forming effective neuron micro-loops together with other neurons are reserved by larger training cost or operation cost; in addition, the existing parameter optimization method cannot ensure that the compressed deep learning network with low parameter number is effectively guided to perform effective learning, so that the accuracy of the compressed deep learning network is difficult to ensure while the operation cost is reduced.
Therefore, how to make the existing deep learning algorithm biological, effectively guide the low parameter compression of the deep neural network, and how to make the compressed deep learning network perform effective learning so as to ensure that the precision of the compressed deep learning network is stable while reducing the operation cost is a problem to be solved by those skilled in the art.
Disclosure of Invention
In view of the above, the present invention provides a neural network evaluation optimization method and system based on neuron plasticity to solve some of the technical problems mentioned in the background art.
In order to achieve the above purpose, the present invention adopts the following technical scheme:
a neural network evaluation optimization method based on neuron plasticity comprises deep neural network structure compression and deep neural network parameter optimization;
the method specifically comprises the following steps:
s1, acquiring a visual image training data set and a verification set, training a computer visual image processing convolutional neural network to obtain a trained depth neural network to be compressed and parameters, and counting a set of compressible depth neural network structures in the trained depth neural network to be compressed;
s2, calculating the plasticity of the neuron clusters and the importance of each neuron according to the depth neural network to be compressed;
s3, calculating the number of necessary neurons in the neuron clusters according to the plasticity of the neuron clusters and the importance of each neuron;
s4, acquiring an unoptimized compressed deep neural network according to the number of necessary neurons in the neuron clusters and the importance of each neuron in the neuron clusters;
s5, calculating the modeling difference of the neural micro-loops of the depth neural network before and after compression according to the depth neural network to be compressed and the depth neural network after non-optimized compression;
s6, training iteration calculation optimization loss is carried out according to the nerve micro-loop modeling difference of the depth neural network before and after compression, the weight of the depth neural network after compression is optimized, and the optimized depth neural network after compression and parameters are obtained.
Preferably, the neuron cluster is a convolution layer of a deep neural network structure to be compressed, a full-connection layer or a construction module of the deep neural network, and the neuron is a filter core, a filter or a sensor in the full-connection layer in the convolution layer.
Preferably, the specific content of the plasticity of the neuron colony in the step S2 is as follows:
s21, sampling a neuron number set FN;
s22, sampling arbitrarilyThe neuron pairs form a neuron pair set FP;
s23 for each pair of neuronal combinationsSampling num kinds of neuron combination sets；
S24, for each current neuron number setCalculating the average output contribution of each neuron on the check data set to the deep neural network to be compressed>；
S25, for each sampling neuron numberThe calculation includes->The neuron combinations of the individual neurons contribute +.>；
S26, forCalculating neuron utilization ∈>I.e. the plasticity that can be achieved when the first neuron population comprises m neurons;
the specific content for calculating the importance of each neuron is as follows:
acquisition of each neuronSampling the neuron combinations FCC, for each neuron combination, calculating the output contribution of the c-th neuron in the first neuron group to the deep neural network to be compressed by applying SHAP algorithm>Calculating the importance of each neuron in the first neuron group +.>。
Preferably, the plasticity achievable when the first neuron population comprises m neurons is calculatedThe method comprises the following steps:
wherein E is the expected operation, I is any image in the verification set omega, N l For the set of neurons in the ith neuron group, i and j are N l Any two of the different elements of the set,the number of neurons for the first neuron group,/-for the first neuron group>For forming a set M with other q neurons when i and j l At this time, the contribution of q+2 neurons to compressible deep neural network output calculated from Xia Puli interaction index, +.>For forming a set M with other p neurons when i and j l ´ At that time, the contribution of p+2 neurons to the compressible deep neural network output calculated from the Xia Puli interaction index;
wherein ,output of deep neural network for neuron set A on image I>Contribution of->Is according to->Results of the softmax operation, +.>，/>；
Importance of each neuron in the first neuron groupThe method comprises the following steps:
wherein ,importance for the c-th neuron in the i-th neuron cluster, +.>For set M l The number of the medium elements is calculated, the following is carried out! Is a factorial operation.
Preferably, the specific content of calculating the number of necessary neurons in the neuron population in S3 is:
the conditions for the satisfaction of the necessary neurons of the first neuron group are:
searching to obtain the number of necessary neurons meeting the condition by a binary search methodWill->Added to necessary nervesThe number of elements is NF.
Preferably, the specific content of obtaining the deep neural network after the non-optimized compression is as follows:
s41, initializing a compressed deep neural network according to NF;
s42, obtaining the corresponding weight of the important neurons according to the importance of each neuron of the NF and the neuron clusters;
s43, initializing the weight of the compressed deep neural network according to the weight of the important neuron, and obtaining the non-optimized compressed deep neural network.
Preferably, the specific contents of the neural micro-loop modeling difference of the depth neural network before and after compression in S5 are as follows:
wherein ,Nl S is the neuron set in the first neuron group l Is thatA combination of neurons consisting of the individual necessary neurons.
Preferably, the optimization loss is specifically:
wherein ,for classifying task loss calculated according to the compressed deep neural network weight and the real label, the EID is a deep neural network structure set with interaction difference, alpha is the interaction difference weight, and the information gain is brought by the interaction difference of the first neuron cluster;
wherein ,for cross entropy function>The CA is a compressible deep neural network structure set, which is a true class label of the image;
。
a neural network evaluation optimization system based on neuron plasticity comprises a deep neural network structure compression subsystem and a deep neural network parameter optimization subsystem;
the deep neural network structure compression subsystem comprises a data set acquisition module, a plasticity and importance calculation module, a necessary neuron calculation module and a neural network compression module;
the deep neural network parameter optimization subsystem comprises an interaction difference calculation module and a neural network weight optimization module;
the data set acquisition module is used for acquiring a visual image training data set and a verification set, training a computer visual image processing convolutional neural network to obtain a trained depth neural network to be compressed and parameters, and counting a set of compressible depth neural network structures in the trained depth neural network to be compressed;
the plasticity and importance calculating module is used for calculating the plasticity of the neuron clusters and the importance of each neuron according to the depth neural network to be compressed;
the necessary neuron calculation module is used for calculating the number of necessary neurons in the neuron clusters according to the plasticity of the neuron clusters and the importance of each neuron;
the neural network compression module is used for acquiring the deep neural network after non-optimized compression according to the necessary neuron number of the neuron clusters and the importance of each neuron in the neuron clusters;
the interaction difference calculation module is used for calculating the neural micro-loop modeling difference of the depth neural network before and after compression according to the depth neural network to be compressed and the depth neural network after non-optimized compression;
the neural network weight optimization module is used for training iteration calculation optimization loss according to the neural micro-loop modeling difference of the depth neural network before and after compression, optimizing the weight of the depth neural network after compression, and obtaining the optimized depth neural network after compression and parameters.
A computer-readable storage medium having stored thereon a computer program which, when run on a computer, causes the computer to perform the described neural network evaluation optimization method based on neuron plasticity for image processing.
Compared with the prior art, the neural network evaluation optimization method and system based on neuron plasticity are disclosed, and are applied to natural image classification, so that the neural network evaluation optimization method and system are in accordance with biology and have the characteristics of higher interpretation, low operation resource cost, high image processing precision and the like;
firstly, estimating the number of redundant neurons in a neuron cluster through the plasticity of the neuron cluster, then accurately positioning the redundant neurons in the neuron cluster through the importance of the neurons, compressing a network structure without modifying the deep neural network structure to be compressed, and eliminating the deep neural network architecture redundancy caused by network design accurately and rapidly without special software or hardware dependence;
the maximization of the accuracy of the depth neural network after compression is realized by minimizing the difference information entropy of the neural micro-loops of the depth neural network before and after compression, the gradient disappearance or gradient explosion phenomenon is not easy to occur in the optimization method, and the optimization effect of the depth neural network after compression with low parameter number can be effectively ensured.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a neural network evaluation optimization method based on neuron plasticity provided by the invention;
FIG. 2 is a diagram showing a comparison result of evaluating the number of necessary filters with the deep neural network sparseness;
FIG. 3 is a schematic diagram of a deep neural network before and after compression according to the present invention;
FIG. 4 is a graph showing the result of comparing the effect of the knowledge distillation method provided by the invention on the optimization of the parameters of the compressed deep neural network;
FIG. 5 is a diagram showing the comparison of compression effects with other deep neural network compression methods according to the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Example 1
The embodiment of the invention discloses a neural network evaluation optimization method based on neuron plasticity, which comprises deep neural network structure compression and deep neural network parameter optimization;
the method specifically comprises the following steps:
s1, acquiring a visual image training data setTraining a computer vision image processing convolutional neural network to obtain a trained deep neural network to be compressed and parameters, counting a set CA of compressible deep neural network structures in the trained deep neural network to be compressed, and initializing a current neuron cluster index l=0;
s2, calculating the plasticity of the neuron clusters and the importance of each neuron according to the depth neural network to be compressed;
s3, calculating the number of necessary neurons in the neuron clusters according to the plasticity of the neuron clusters and the importance of each neuron;
s4, acquiring an unoptimized compressed deep neural network according to the number of necessary neurons in the neuron clusters and the importance of each neuron in the neuron clusters;
s5, calculating the modeling difference of the neural micro-loops of the depth neural network before and after compression according to the depth neural network to be compressed and the depth neural network after non-optimized compression;
s6, training iteration calculation optimization loss is carried out according to the nerve micro-loop modeling difference of the depth neural network before and after compression, the weight of the depth neural network after compression is optimized, and the optimized depth neural network after compression and parameters are obtained.
In this embodiment, the verification dataset is a subset of the training dataset, and may be composed of a single-band image or a multi-band image; the training data set is used for training a deep neural network to be compressed, the rule of sampling the verification data set from the training data set is that 10 random images are sampled in each class of classified targets, the trained deep neural network to be compressed is a convolutional neural network trained on the training data set, and the convolutional neural network can be used for target classification, target detection or target segmentation and other computer vision image processing tasks; indexing layers in trained deep neural networks to be compressedMapping rules F mapping to a compressible deep neural network Structure set CA ca The method comprises the following steps:
wherein ,nQ The number of layers is calculated for all trained deep neural networks to be compressed.
In order to further implement the above technical solution, the neuron clusters are convolution layers of the deep neural network structure to be compressed, the full connection layers or the construction modules of the deep neural network, and the neurons are filter kernels in the convolution layers, filters or perceptors in the full connection layers.
The present embodiment takes neurons as a filter as an example.
The method also comprises initializing l=0 before S2, and the number of the necessary filters of the neuron colony is setIf the current neuron population index +.>Step S3, jumping to the step; if->Then jump to S2.
In order to further implement the above technical solution, for the first neuron cluster, the filter utilization rate of the compressible deep neural network structure and the importance of each filter are calculated, where the filter refers to a feature operation structure composed of a plurality of filter kernels, specifically, the weight of the first compressible deep neural network structure isIt comprises->Filters, each filter comprising +.>Filter kernel of size +.>；
Wherein, the filter utilization rate is calculatedThe specific contents are as follows:
s21, sampling filter number set FN:
wherein if itThen->For>Then->For>Then->For>Then->；
S22, sampling arbitrarilyThe filter pairs form a filter pair set FP;
s23 for each pair of filter combinationsSample num filter combination set；
；
S24, for each current filter number setCalculating the average output contribution of each filter to the depth neural network to be compressed on the check data set>：
Wherein E is the expected operation, I is any image in the verification set omega, N l For filter sets i and j in the ith neuron cluster N l Any two of the different elements of the set,output of the depth neural network on the image I for the filter set A +.>Contribution of->Is according to->Results of the softmax operation;
s25, for each sampling filter numberCalculating the average output contribution of a filter combination comprising m+2 filters to the deep neural network to be compressed>：
S26, forCalculating the filter utilization ∈ ->I.e. the achievable plasticity when the first neuron population contains m filters:
wherein ,the number of neurons for the first neuron group,/-for the first neuron group>For forming a set M when i and j are combined with other q filters l At this time, the q+2 filters calculated from Xia Puli interaction index contribute +.>For forming a set M with other p neurons when i and j l ´ At this time, the contribution of p+2 neurons to compressible deep neural network output calculated from Xia Puli interaction index, +.>，/>：
The specific content for calculating the importance of each neuron is as follows:
acquiring each filterFor each filter combination, calculating the output contribution +.f. of the c-th filter in the i-th neuron cluster to the deep neural network to be compressed using the SHAP algorithm>Calculating the importance of each filter in the first neuron group +.>：
。
Importance of each filter in the first neuron groupThe method comprises the following steps:
wherein ,importance for the c-th neuron in the i-th neuron cluster, +.>For set M l The number of the medium elements is calculated, the following is carried out! Is a factorial operation.
In order to further implement the above technical solution, the specific content of calculating the number of necessary neurons in the neuron clusters in S3 is:
initializing the number of currently necessary filters；
The conditions for the satisfaction of the necessary neurons of the first neuron group are:
searching to obtain the number of necessary filters meeting the condition by a binary search method；
Will beAdded to the set of necessary neuron numbers NF.
In this embodiment, as shown in fig. 2, the filter utilization rate more accurately reflects the relationship between the number of the filters of the neuron population deep neural network and the precision of the compressed deep neural network than the sparseness of the deep neural network; on the one hand, the filter utilization rate can reflect how to realize higher TOP1 verification accuracy on the CIFAR-10 data set more quickly; on the other hand, the filter utilization rate can more accurately reflect the relation between the number of the filters and the precision; specifically, on the ResNet-50 shallow layer, when the reserved channel proportion and the filter utilization are both 0.4, the filter utilization is recommended to use fewer filters to obtain approximate precision; on the ResNet-50 deep layer, when the reserved channel proportion and the filter utilization rate are both 0.5, the filter utilization rate suggests using a little more channels to realize rapid precision rise, so that the precision loss risk caused by the deep neural network structure compression is reduced; when the filter utilization rate is about 0.5, the accuracy of the depth neural network after compression begins to converge, which proves that the effectiveness of the filter utilization rate in the depth neural network compression is expected in the step S3, so that the relation between the number of estimated filters and the accuracy of the depth neural network after compression is more accurate by adopting the filter utilization rate than the existing redundancy evaluation method.
Accumulating current neuron cluster indexAnd jumping to the step before S2;
in order to further implement the above technical solution, according to the number of necessary filters of the compressible deep neural network structure and the importance of each filter of the neuron clusters, the specific content of obtaining the deep neural network after the non-optimized compression is as follows:
s41, initializing a compressed deep neural network according to NF;
s42, obtaining the corresponding weight of the important filter according to the importance of each filter of the NF and the neuron clusters;
s43, initializing the weight of the compressed deep neural network according to the weight of the important filter to obtain an unoptimized compressed deep neural network;
as shown in fig. 3, the parameter and the operand of the depth neural network after compression are significantly reduced, wherein the operand is reduced by about 54% and the parameter is reduced by about 60%.
Before S5, the method further comprises the following steps:
initializing l=0, and enabling an interaction difference deep neural network structure set EID to exist.
If the current neuron population indexStep S5, jumping to the step; if->Then jump to S6.
In order to further implement the above technical solution, in S5, a neural micro-loop modeling difference of the depth neural network before and after compression, that is, an interaction difference of the first neuron population, is calculated, and the specific contents are as follows:
wherein ,Nl For the filter set in the first neuron group, S l Is thatA filter combination of the necessary filters.
If it isThen add l to EID; l=l+1, and judging the index step of the current neuron colony before jumping to S5;
if it isInteraction difference weight ∈>The method comprises the steps of carrying out a first treatment on the surface of the On the contrary, let(s)>。
According to the compressed deep neural network weight and the real label, calculating the classification task loss；
Initialization of,/>；
If it isCalculating information gain caused by interaction difference of the first neuron colony; on the contrary, ifThen l=l+1 continues iteration, otherwise the final optimization loss L is calculated;
in order to further implement the above technical solution, the optimization loss is specifically:
wherein ,for classifying task loss calculated according to the compressed deep neural network weight and the real label, the EID is a deep neural network structure set with interaction difference, alpha is the interaction difference weight, and the information gain is brought by the interaction difference of the first neuron cluster;
accumulating information gain caused by interaction difference of the first neuron colony:
wherein ,for cross entropy function>The CA is a compressible deep neural network structure set, which is a true class label of the image;
。
in this embodiment, the iterative process is a process of updating the convolutional neural network weight using the SGD and its variant optimization algorithm; as shown in fig. 4, and using only ordinary trainingCompared with a knowledge distillation method, the interactive difference training is applied to ensure that the accuracy of the training data set and the accuracy of the verification data set rise more quickly, and the loss of the training data set and the loss of the verification data set drop more quickly; therefore, the parameter optimization of the interaction difference can obtain better training effect and promote generalization of the compressed deep neural network.
In practical application, the compression effect of the method is compared with the compression effect of other deep neural network compression methods, and the compression effect is compared with the compression effect of other deep neural network compression methods from the aspects of precision loss and calculation amount reduction, as shown in fig. 5, when the method is applied to compressing the deep neural network on different image classification tasks of different convolution neural networks such as light weight, dense connection, sparse connection and the like, the extremely high compression ratio can be realized, and the precision of the compressed deep neural network is ensured. In addition, the method can obtain the convolution neural network compression result which is similar to or even better than the existing advanced deep neural network compression algorithm on the premise of not adding a new network structure and not having extra hardware dependence.
A neural network evaluation optimization system based on neuron plasticity comprises a deep neural network structure compression subsystem and a deep neural network parameter optimization subsystem;
the deep neural network structure compression subsystem comprises a data set acquisition module, a plasticity and importance calculation module, a necessary neuron calculation module and a neural network compression module;
the deep neural network parameter optimization subsystem comprises an interaction difference calculation module and a neural network weight optimization module;
the data set acquisition module is used for acquiring a visual image training data set and a verification set, training a computer visual image processing convolutional neural network to obtain a trained depth neural network to be compressed and parameters, and counting a set of compressible depth neural network structures in the trained depth neural network to be compressed;
the plasticity and importance calculating module is used for calculating the plasticity of the neuron clusters and the importance of each neuron according to the depth neural network to be compressed;
the necessary neuron calculation module is used for calculating the number of necessary neurons in the neuron clusters according to the plasticity of the neuron clusters and the importance of each neuron;
the neural network compression module is used for acquiring the deep neural network after non-optimized compression according to the necessary neuron number of the neuron clusters and the importance of each neuron in the neuron clusters;
the interaction difference calculation module is used for calculating the neural micro-loop modeling difference of the depth neural network before and after compression according to the depth neural network to be compressed and the depth neural network after non-optimized compression;
the neural network weight optimization module is used for training iteration calculation optimization loss according to the neural micro-loop modeling difference of the depth neural network before and after compression, optimizing the weight of the depth neural network after compression, and obtaining the optimized depth neural network after compression and parameters.
The neural network evaluation optimization system based on the neuron plasticity is an embedded system, and the system can comprise a terminal, a server and other equipment, wherein the terminal can be an embedded system carrying an artificial intelligent acceleration chip, or an embedded system carrying a DSP (digital signal processor) chip and an FPGA (field programmable gate array) chip, or a system carrying an artificial intelligent acceleration board card such as a GPU (graphics processing unit) and a TPU (thermoplastic polyurethane).
A computer-readable storage medium having stored thereon a computer program which, when run on a computer, causes the computer to perform an optimization method for neural network evaluation based on neuron plasticity for image processing.
The specific content of performing image processing by a neural network evaluation optimization method based on neuron plasticity is as follows: and acquiring a visual image, preprocessing, inputting the preprocessed image into the optimized compressed deep neural network model, and outputting the processing results of target classification, target detection or target segmentation.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. The neural network evaluation optimization method based on neuron plasticity is characterized by comprising deep neural network structure compression and deep neural network parameter optimization;
the method specifically comprises the following steps:
s1, acquiring a visual image training data set and a verification set, training a computer visual image processing convolutional neural network to obtain a trained depth neural network to be compressed and parameters, and counting a set of compressible depth neural network structures in the trained depth neural network to be compressed;
s2, calculating the plasticity of the neuron clusters and the importance of each neuron according to the depth neural network to be compressed;
s3, calculating the number of necessary neurons in the neuron clusters according to the plasticity of the neuron clusters and the importance of each neuron;
s4, acquiring an unoptimized compressed deep neural network according to the number of necessary neurons in the neuron clusters and the importance of each neuron in the neuron clusters;
s5, calculating the modeling difference of the neural micro-loops of the depth neural network before and after compression according to the depth neural network to be compressed and the depth neural network after non-optimized compression;
s6, training iteration calculation optimization loss is carried out according to the nerve micro-loop modeling difference of the depth neural network before and after compression, the weight of the depth neural network after compression is optimized, and the optimized depth neural network after compression and parameters are obtained.
2. The neural network evaluation optimization method based on neuron plasticity according to claim 1, wherein the neuron clusters are convolution layers of a deep neural network structure to be compressed, full-connection layers or construction modules of the deep neural network, and the neurons are filter kernels in the convolution layers, filters or perceptrons in the full-connection layers.
3. The neural network evaluation optimization method based on neuron plasticity according to claim 1, wherein the specific content of the calculated neuron colony plasticity in S2 is:
s21, sampling a neuron number set FN;
s22, sampling arbitrarilyThe neuron pairs form a neuron pair set FP;
s23 for each pair of neuronal combinationsSampling num kinds of neuron combination sets；
S24, for each current neuron number setCalculating the average output contribution of each neuron on the check data set to the deep neural network to be compressed>；
S25, for each sampling neuron numberThe calculation includes->The neuron combinations of the individual neurons contribute +.>；
S26, forCalculating neuron utilization ∈>I.e. the plasticity that can be achieved when the first neuron population comprises m neurons;
the specific content for calculating the importance of each neuron is as follows:
acquisition of each neuronSampling the neuron combinations FCC, for each neuron combination, calculating the output contribution of the c-th neuron in the first neuron group to the deep neural network to be compressed by applying SHAP algorithm>Calculating the importance of each neuron in the first neuron group +.>。
4. A neural network evaluation optimization method based on neuron plasticity as claimed in claim 3, wherein the plasticity achievable when the first neuron group contains m neurons is calculatedThe method comprises the following steps:
；
wherein E is the expected operation, I is any image in the verification set omega, N l For the set of neurons in the ith neuron group, i and j are N l Any two of the different elements of the set,the number of neurons for the first neuron group,/-for the first neuron group>For forming a set M with other q neurons when i and j l At this time, the contribution of q+2 neurons to compressible deep neural network output calculated from Xia Puli interaction index, +.>For forming a set M with other p neurons when i and j l ´ At that time, the contribution of p+2 neurons to the compressible deep neural network output calculated from the Xia Puli interaction index;
；
；
；
；
wherein ,output of deep neural network for neuron set A on image I>Contribution of->Is based onResults of the softmax operation, +.>，/>；
Importance of each neuron in the first neuron groupThe method comprises the following steps:
；
wherein ,importance for the c-th neuron in the i-th neuron cluster, +.>For set M l The number of the medium elements is calculated, the following is carried out! Is a factorial operation.
5. The neural network evaluation optimization method based on neuron plasticity according to claim 4, wherein the specific content for calculating the number of necessary neurons in the neuron clusters in S3 is as follows:
the conditions for the satisfaction of the necessary neurons of the first neuron group are:
；
searching to obtain the number of necessary neurons meeting the condition by a binary search methodWill->Added to the set of necessary neuron numbers NF.
6. The neural network evaluation optimization method based on neuron plasticity according to claim 5, wherein the specific content of obtaining the deep neural network after non-optimized compression is:
s41, initializing a compressed deep neural network according to NF;
s42, obtaining the corresponding weight of the important neurons according to the importance of each neuron of the NF and the neuron clusters;
s43, initializing the weight of the compressed deep neural network according to the weight of the important neuron, and obtaining the non-optimized compressed deep neural network.
7. The neural network evaluation optimization method based on neuron plasticity according to claim 6, wherein the specific contents of the neural micro-loop modeling difference of the depth neural network before and after compression in S5 are:
；
wherein ,Nl S is the neuron set in the first neuron group l Is thatA combination of neurons consisting of the individual necessary neurons.
8. The neural network evaluation optimization method based on neuron plasticity according to claim 7, wherein the optimization loss is specifically:
；
wherein ,for classifying task loss calculated according to the compressed deep neural network weight and the real label, the EID is a deep neural network structure set with interaction difference, alpha is the interaction difference weight, and the information gain is brought by the interaction difference of the first neuron cluster;
；
；
wherein ,for cross entropy function>The CA is a compressible deep neural network structure set, which is a true class label of the image;
；
。
9. a neural network evaluation optimization system based on neuron plasticity, which is characterized by comprising a deep neural network structure compression subsystem and a deep neural network parameter optimization subsystem based on the neural network evaluation optimization method based on neuron plasticity as claimed in any one of claims 1-8;
the deep neural network structure compression subsystem comprises a data set acquisition module, a plasticity and importance calculation module, a necessary neuron calculation module and a neural network compression module;
the deep neural network parameter optimization subsystem comprises an interaction difference calculation module and a neural network weight optimization module;
the data set acquisition module is used for acquiring a visual image training data set and a verification set, training a computer visual image processing convolutional neural network to obtain a trained depth neural network to be compressed and parameters, and counting a set of compressible depth neural network structures in the trained depth neural network to be compressed;
the plasticity and importance calculating module is used for calculating the plasticity of the neuron clusters and the importance of each neuron according to the depth neural network to be compressed;
the necessary neuron calculation module is used for calculating the number of necessary neurons in the neuron clusters according to the plasticity of the neuron clusters and the importance of each neuron;
the neural network compression module is used for acquiring the deep neural network after non-optimized compression according to the necessary neuron number of the neuron clusters and the importance of each neuron in the neuron clusters;
the interaction difference calculation module is used for calculating the neural micro-loop modeling difference of the depth neural network before and after compression according to the depth neural network to be compressed and the depth neural network after non-optimized compression;
the neural network weight optimization module is used for training iteration calculation optimization loss according to the neural micro-loop modeling difference of the depth neural network before and after compression, optimizing the weight of the depth neural network after compression, and obtaining the optimized depth neural network after compression and parameters.
10. A computer-readable storage medium, having stored thereon a computer program which, when run on a computer, causes the computer to perform an image processing by a neural network evaluation optimization method based on neuron plasticity as claimed in any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311118140.4A CN116882461B (en) | 2023-09-01 | 2023-09-01 | Neural network evaluation optimization method and system based on neuron plasticity |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311118140.4A CN116882461B (en) | 2023-09-01 | 2023-09-01 | Neural network evaluation optimization method and system based on neuron plasticity |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116882461A true CN116882461A (en) | 2023-10-13 |
| CN116882461B CN116882461B (en) | 2023-11-21 |
Family
ID=88257161
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311118140.4A Active CN116882461B (en) | 2023-09-01 | 2023-09-01 | Neural network evaluation optimization method and system based on neuron plasticity |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116882461B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109523017A (en) * | 2018-11-27 | 2019-03-26 | 广州市百果园信息技术有限公司 | Compression method, device, equipment and the storage medium of deep neural network |
| US20190392323A1 (en) * | 2018-06-22 | 2019-12-26 | Moffett AI, Inc. | Neural network acceleration and embedding compression systems and methods with activation sparsification |
| CN111814662A (en) * | 2020-07-07 | 2020-10-23 | 北京航空航天大学 | Visible light image airplane rapid detection method based on miniature convolutional neural network |
| WO2021143070A1 (en) * | 2020-01-16 | 2021-07-22 | 北京智芯微电子科技有限公司 | Compression method and apparatus for deep neural network model, and storage medium |
| CN114139680A (en) * | 2021-11-09 | 2022-03-04 | 天津大学 | Automatic negotiation intelligent agent design method based on deep reinforcement learning |
| CN116502697A (en) * | 2023-04-26 | 2023-07-28 | 宁夏青铜峡市华能雷避窑光伏发电有限公司 | Deep neural network model compression method and compression device based on CIFAR data set |
-
2023
- 2023-09-01 CN CN202311118140.4A patent/CN116882461B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190392323A1 (en) * | 2018-06-22 | 2019-12-26 | Moffett AI, Inc. | Neural network acceleration and embedding compression systems and methods with activation sparsification |
| CN109523017A (en) * | 2018-11-27 | 2019-03-26 | 广州市百果园信息技术有限公司 | Compression method, device, equipment and the storage medium of deep neural network |
| WO2021143070A1 (en) * | 2020-01-16 | 2021-07-22 | 北京智芯微电子科技有限公司 | Compression method and apparatus for deep neural network model, and storage medium |
| CN111814662A (en) * | 2020-07-07 | 2020-10-23 | 北京航空航天大学 | Visible light image airplane rapid detection method based on miniature convolutional neural network |
| CN114139680A (en) * | 2021-11-09 | 2022-03-04 | 天津大学 | Automatic negotiation intelligent agent design method based on deep reinforcement learning |
| CN116502697A (en) * | 2023-04-26 | 2023-07-28 | 宁夏青铜峡市华能雷避窑光伏发电有限公司 | Deep neural network model compression method and compression device based on CIFAR data set |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116882461B (en) | 2023-11-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2020238293A1 (en) | Image classification method, and neural network training method and apparatus | |
| CN110378383B (en) | Picture classification method based on Keras framework and deep neural network | |
| Blundell et al. | Model-free episodic control | |
| Alvarez et al. | Learning the number of neurons in deep networks | |
| CN111858989B (en) | Pulse convolution neural network image classification method based on attention mechanism | |
| CN111898703B (en) | Multi-label video classification method, model training method, device and medium | |
| CN109063719B (en) | Image classification method combining structure similarity and class information | |
| CN109389171B (en) | Medical image classification method based on multi-granularity convolution noise reduction automatic encoder technology | |
| CN114332545B (en) | Image data classification method and device based on low-bit pulse neural network | |
| CN110991554B (en) | Improved PCA (principal component analysis) -based deep network image classification method | |
| Chen et al. | Image classification based on convolutional denoising sparse autoencoder | |
| Peng et al. | A survey: Image classification models based on convolutional neural networks | |
| CN108388918B (en) | Data feature selection method with structure retention characteristics | |
| Tmamna et al. | Pruning deep neural networks for green energy-efficient models: A survey | |
| CN115909336A (en) | Text recognition method and device, computer equipment and computer-readable storage medium | |
| Jeczmionek et al. | Input reduction of convolutional neural networks with global sensitivity analysis as a data-centric approach | |
| Sun et al. | A Review of AIoT-based Edge Devices and Lightweight Deployment | |
| CN116882461B (en) | Neural network evaluation optimization method and system based on neuron plasticity | |
| JP7546630B2 (en) | Neural network optimization method, computer system, and computer-readable storage medium | |
| Liang et al. | Deep hashing with multi-task learning for large-scale instance-level vehicle search | |
| Janssen | Attention based Temporal Convolutional Network for stock price prediction | |
| CN112801153B (en) | Semi-supervised image classification method and system of image embedded with LBP (local binary pattern) features | |
| US20220202348A1 (en) | Implementing brain emulation neural networks on user devices | |
| Xie et al. | ResNet with attention mechanism and deformable convolution for facial expression recognition | |
| Liu et al. | Learning compact ConvNets through filter pruning based on the saliency of a feature map |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |