CN117114053A - Convolutional neural network model compression method and device based on structure search and knowledge distillation - Google Patents
Convolutional neural network model compression method and device based on structure search and knowledge distillation Download PDFInfo
- Publication number
- CN117114053A CN117114053A CN202311073117.8A CN202311073117A CN117114053A CN 117114053 A CN117114053 A CN 117114053A CN 202311073117 A CN202311073117 A CN 202311073117A CN 117114053 A CN117114053 A CN 117114053A
- Authority
- CN
- China
- Prior art keywords
- network model
- network
- convolutional neural
- student
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013527 convolutional neural network Methods 0.000 title claims abstract description 66
- 238000013140 knowledge distillation Methods 0.000 title claims abstract description 52
- 238000000034 method Methods 0.000 title claims abstract description 48
- 230000006835 compression Effects 0.000 title claims abstract description 46
- 238000007906 compression Methods 0.000 title claims abstract description 46
- 238000012549 training Methods 0.000 claims abstract description 77
- 230000001537 neural effect Effects 0.000 claims abstract description 6
- 238000005516 engineering process Methods 0.000 claims abstract description 5
- 230000006870 function Effects 0.000 claims description 30
- 238000009826 distribution Methods 0.000 claims description 28
- 238000012795 verification Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 13
- 230000004913 activation Effects 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 8
- 238000013507 mapping Methods 0.000 claims description 7
- 238000012360 testing method Methods 0.000 claims description 6
- 230000003044 adaptive effect Effects 0.000 claims description 3
- 238000000605 extraction Methods 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 3
- 238000013461 design Methods 0.000 abstract description 10
- 238000013138 pruning Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 5
- 239000002699 waste material Substances 0.000 description 3
- 241000282414 Homo sapiens Species 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 239000000969 carrier Substances 0.000 description 1
- 230000019771 cognition Effects 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000004821 distillation Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000011002 quantification Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- General Engineering & Computer Science (AREA)
- Molecular Biology (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a convolutional neural network model compression method and device based on structure search and knowledge distillation, comprising the following steps: acquiring a target task data set; dividing a training set from a target task data set, and training a convolutional neural network model by using the training set; the convolutional neural network model is used as a teacher network after training is completed; searching a convolutional neural network model by utilizing a neural structure searching technology to obtain a lightweight network structure; taking the light-weight network structure obtained by searching as a student network; performing knowledge distillation, calculating the difference of output of a teacher network and a student network Softmax layer, taking the difference as a part of the loss of the student network, and performing iterative training on the student network until convergence; and outputting the student network model after knowledge distillation, namely the model after compression. The method can effectively reduce the redundancy parameters of the model, realizes the automatic design of the student network model, and is suitable for efficient and self-adaptive model compression application scenes.
Description
Technical Field
The invention belongs to the technical field of artificial intelligence and deep learning, and particularly relates to a convolutional neural network model compression method and device based on structural search and knowledge distillation.
Background
Artificial intelligence techniques represented by deep learning in recent years have exhibited performance beyond human cognition in the scenes of word recognition, voice recognition, target detection, natural language processing, automatic driving, biological information processing, computer games, recommendation systems, medical automatic diagnosis, financial analysis, and the like, and the performance is still being continuously improved. But with the accompanying increase of parameter quantity, the network structure with more and more complex structure is adopted.
In the practical application scene, the air-based platform type informationized equipment is a key for capturing empty rights to obtain initiative rights in modern development according to the advantage of wide air monitoring range. The air-based platform equipment generally takes an aircraft and a floating platform as carriers, carries a task system for bearing various combat tasks, and has various functions of information detection, transmission, control, guidance, countermeasure and the like. Due to the development of embedded platforms and the increase of unmanned combat demands, unmanned aerial vehicle-based edge space-based platforms have gradually become the main angle. However, computing resources on edge space-based platforms are very limited. Thus, compressing the model is an indispensable step in deploying the model onto an empty base platform.
Currently, the methods for model compression are mainly: pruning, quantification, lightweight network structure design and knowledge distillation. Among them, knowledge distillation is widely used because of its low loss of precision and ease of use. Knowledge distillation includes two network models, of which a larger number of parameters is called a teacher network model and a smaller number of parameters is called a student network model. Knowledge distillation achieves compression of the model by aligning the outputs of the teacher network model and the student network model logits layers so that the student network model reaches a level close to that of the teacher network model. However, in the current knowledge distillation method, the network model of the student is designed mainly by engineers, and a great deal of parameter redundancy often exists. Moreover, designing a high-performance neural network requires a great deal of trial and error and engineering experience, and is extremely high in cost.
The Chinese patent application with publication number of CN113159173A discloses a convolutional neural network model compression method combining pruning and knowledge distillation, which takes a trained convolutional neural network model as a teacher network, performs pruning on the teacher network to obtain a student network, and further performs knowledge distillation on the teacher network and the student network. However, the pruning method for setting the threshold is adopted to sparsify the teacher model structure, the threshold setting needs to be tried continuously, the experience requirement is high, and universality is not achieved.
The Chinese patent application with publication number of CN113011570A discloses a self-adaptive high-precision compression method and system of a convolutional neural network model, which searches pruning through a two-stage structure with thickness and fineness to obtain an optimal model structure, and finally restores the precision of the network model after pruning through a knowledge distillation method. The method is still essentially to perform model pruning, and network redundancy parameters are still to be reduced.
Therefore, there is a need for a method that automatically designs a student network model structure to efficiently implement model compression through knowledge distillation.
Disclosure of Invention
In view of the above, the present invention aims to provide a convolutional neural network model compression method and apparatus based on structure search and knowledge distillation, which uses a trained convolutional neural network model as a teacher network, and adopts a neural structure search (NAS) technology to obtain a lightweight network structure as a student network, so as to implement model compression by performing knowledge distillation on the teacher network and the student network, thereby effectively reducing model redundancy parameters, implementing automatic design on the student network model, and being suitable for efficient and self-adaptive model compression application scenarios of resource-limited airborne systems such as inspection unmanned aerial vehicles, reconnaissance unmanned aerial vehicles, etc. in space-based platform equipment task systems.
In order to achieve the above purpose, the technical scheme provided by the invention is as follows:
the embodiment of the invention provides a convolutional neural network model compression method based on structure search and knowledge distillation, which comprises the following steps:
step S1, acquiring a target task data set T;
step S2, dividing a training set T from the target task data set T train， Using training set T train Training a convolutional neural network model;
step S3, the convolutional neural network model is used as a teacher network after training is completed;
s4, searching a lightweight network structure for the convolutional neural network model by utilizing a neural structure searching technology;
s5, taking the light-weight network structure obtained by searching as a student network;
step S6, knowledge distillation is carried out, and the training set T is obtained train Respectively inputting the difference into a teacher network and a student network, calculating the difference of output of a Softmax layer of the teacher network and the student network, taking the difference as a part of the loss of the student network, and performing iterative training on the student network until convergence;
and S7, outputting a student network model after knowledge distillation, namely a model after compression.
Further, the step S2 includes:
s21, dividing the target task data set T to obtain a training set T train Verification set T val And test set T test ；
S22, training set T train Inputting into a convolutional neural network model;
s23, setting a loss function L of a training convolutional neural network model:
where x represents the training input, y represents the resulting label, W represents the target network weight, f (x, W) represents the output of the target network logits layer, softmax (·) represents converting the output of the logits layer using the softmax functionIs the probability distribution of the category, l cross (. Cndot.) represents the cross entropy of the convolutional neural network as a loss function of training;
s24, training target network parameters according to the set loss function;
s25, outputting the convolutional neural network model with the trained parameters.
Further, the step S4 includes:
s41, defining a student network search space, constructing a mixed model by the convolutional neural network model through the search space, wherein the mixed model comprises a plurality of independent models, and respectively endowing different structural weights v to the operation among nodes of the search space;
s42, training set T train Divided into a first training set T train1 And a second training set T train2 The method comprises the steps of respectively training structural weights v of the hybrid model and network model weights w of the hybrid model;
s43, fixing the network model weight w in the first training set T train1 Optimizing structural weight v by the upper training mixed model;
s44, fixing the structural weight v, in the second training set T train2 The upper training mixed model optimizes the network model weight w;
s45, repeating the steps S43-S44 for a plurality of rounds, determining an independent model structure according to the maximum value of the structural weight of each operation after each round is finished, and inputting a verification set T val And evaluating the independent model, namely obtaining the lightweight network structure by searching the independent model with highest precision.
Preferably, the search space is a chain structure, and includes M paths, each path has N nodes, and K operations can be adopted between every two nodes.
Preferably, the search space is designed based on VGGNet, the operation between every two nodes consists of different VGGConv, VGGConv is a building block of VGGNet, when a feature map in a target task data set T is input into the building block VGGConv, the input feature map is firstly subjected to convolution with a convolution kernel size of k multiplied by k and a convolution step size of s, then activated by a ReLU function, and finally subjected to batch normalization to obtain an output feature map.
Preferably, the convolution step length is obtained by calculating the input feature map size fm_size1 and the output feature map size fm_size2, so that the number of operation stacks of the adaptive search building block VGGConv is realized in the search space, the alignment of the output feature map sizes of different operations is ensured, and the calculation formula of the convolution step length s is as follows:
preferably, the search space is designed based on mobilet, the operation between every two nodes consists of different MBConv, which is a building block of mobilet, comprising an extended convolution, a depth separable convolution, and a mapped convolution; the extended convolution comprises a convolution layer with the size of 1 multiplied by 1, a BatchNorm layer and a ReLU activation layer, and expands the characteristic map channel according to the expansion coefficient t; the depth separable convolution comprises a k multiplied by k convolution layer, a BatchNorm layer and a ReLU activation layer, wherein each channel in the convolution layer only uses a single convolution kernel to reduce the number of parameters, and the feature extraction is carried out through the depth separable convolution; the mapping convolution comprises a 1×1 convolution layer, a BatchNorm layer and a ReLU activation layer, and the feature map is mapped back to the input channel number through the mapping convolution.
Further, the step S6 includes:
s61, training set T train Inputting the output data to a teacher network and a student network, and obtaining the output of the teacher network and the student network in an Iogits layer;
s62, converting the output of the logits layer into probability distribution of category by using softmax function;
s63, calculating differences of the class probability distribution of the teacher network and the student network by using the KL divergence, and taking the differences as soft target loss L of the student network soft-Target (y student ，y teacher ) The calculation formula is as follows:
wherein y is student Representing a class probability distribution of student network predictions, y teacher Representing class probability distribution predicted by a teacher network;
s64, designing a knowledge validity verification weight H (y teacher Y), avoiding the student network model learning incorrect knowledge from the teacher network model:
wherein argmax (·) represents a parametrization function, and when the teacher network model predicts the result y teacher Outputting a validity verification weight value of 1 when the validity verification weight value is consistent with the label y, and predicting a result y when a teacher network model teacher When the tag y is inconsistent with the tag y, outputting a validity verification weight value of 0;
s65, defining training loss function L of student network stuedent ：
Wherein α represents hard target loss L hard-Target (y student Weight of y), beta represents soft target loss L soft-Target (y student ，y teacher ) Weights of (2);
and S66, performing iterative training on the student network according to the defined student network training loss function until the student network model converges.
Based on the above object, the embodiment of the present invention further provides a convolutional neural network model compression device based on structure search and knowledge distillation, which includes a memory and a processor, wherein the memory is used for storing a computer program, and the processor is used for implementing the convolutional neural network model compression method based on structure search and knowledge distillation when executing the computer program.
Based on the above object, the embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored, and when the computer program is used in a computer, the above convolutional neural network model compression method based on structure search and knowledge distillation is implemented.
Compared with the prior art, the invention has the beneficial effects that at least the following steps are included:
(1) The invention can effectively reduce the redundant parameters of the convolutional neural network model and reduce the waste of computing resources such as computing power, memory, storage and the like of an operation platform.
(2) According to the invention, the neural structure search (NAS) algorithm is adopted to automatically design the student network model, so that the self-adaption degree is higher, the dependence on the experience of engineers is reduced, and the time cost of student network design is reduced.
(3) The invention has the obvious advantages of high compression rate of the model, less precision loss and the like, and has strong practicability.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method for convolutional neural network model compression based on structure search and knowledge distillation provided by an embodiment of the invention;
FIG. 2 is a schematic diagram of Stage structure in a search space designed based on VGGNet according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of VGGConv building blocks in VGGNet according to an embodiment of the present invention;
fig. 4 is a schematic diagram of Stage structure in a search space based on a MobileNet design according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of MBConv building blocks in MobileNet provided by an embodiment of the present invention;
fig. 6 is a schematic diagram of knowledge distillation provided by an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the detailed description is presented by way of example only and is not intended to limit the scope of the invention.
The invention is characterized in that: aiming at the problems that in the prior art, a great amount of parameter redundancy and automation of student network model structural design and model compression rate are still required to be improved, the embodiment of the invention provides a convolutional neural network model compression method and device based on structure search and knowledge distillation.
FIG. 1 is a flow chart of a method for convolutional neural network model compression based on structural search and knowledge distillation, provided by an embodiment of the invention. As shown in fig. 1, an embodiment provides a method for compressing a convolutional neural network model based on structural search and knowledge distillation, which is applied to compressing the convolutional neural network model of an airborne system with limited resources such as a patrol unmanned aerial vehicle and a reconnaissance unmanned aerial vehicle in a space-based platform equipment task system, and is used for solving the problem of redundancy of parameters of the convolutional neural network model and reducing the waste of computing resources such as computing power, memory and storage of an operation platform, and the method comprises the following steps:
s1, acquiring a target task data set T.
In an embodiment, the Cifar-10 dataset is taken as the target dataset.
S2, dividing a training set T from the target task data set T train And training the convolutional neural network model.
In the embodiment, the target task data set T is divided into training sets T according to the proportion of 60%, 20% and 20% train Verification set T val And test set T test The method comprises the steps of carrying out a first treatment on the surface of the Then the training setT train Inputting into a convolutional neural network model; meanwhile, a loss function L of a training convolutional neural network model is set:
where x represents the training input, y represents the resulting label, W represents the weight of the target network, f (x, W) represents the output of the target network logits layer, softmax (·) represents the probability distribution of converting the output of the logits layer into categories using the softmax function, l cross (. Cndot.) represents the cross entropy of the convolutional neural network as a loss function of training;
after the loss function is set, training the target network parameters according to the set loss function; and finally, outputting the convolutional neural network model with the trained parameters.
S3, the convolutional neural network model is used as a teacher network after training is completed;
s4, searching a convolutional neural network model by utilizing a neural structure searching technology to obtain a lightweight network structure;
specifically, a student network search space is defined firstly, a convolutional neural network model is used for constructing a hybrid model through the search space, and the hybrid model comprises a plurality of independent models; the search space is of a chain structure and comprises M paths, N nodes are arranged on each path, K operations can be adopted between every two nodes, different structural weights v are respectively assigned to the operations between the nodes of the search space, parameters M, N and K are set according to the space to be searched, and the structural weights v are randomly initialized.
In an embodiment, a search space is designed based on VGGNet, the search space comprises 1 path, 6 nodes are arranged on the path, one Stage (Stage) is called between every two nodes, 6 optional operations in each Stage are arranged, and the structure of the Stage is shown in fig. 2. Each optional operation in Stage is composed of VGGConv, VGGConv is a building block of VGGNet, the structure of VGGConv is shown in fig. 3, when a feature map in a target task data set T is input into a building block vggconv_ka_sd, the input feature map is firstly subjected to convolution of a set convolution kernel size k×k and a convolution step s, then activated by a ReLU function, and finally subjected to batch normalization to obtain an output feature map, wherein k=a in the convolution kernel size is set by human beings, and the convolution step s=d. In order to be able to adaptively search the number of stacks of VGGConv operations of a building block, the alignment of the sizes of different operation output feature maps is ensured, the value of the convolution step s is obtained by calculating the size fm_size1 of the Stage input feature map and the size fm_size2 of the output feature map, so that the search space is adaptive, and the calculation formula of the convolution step s is as follows:
as shown in fig. 2, the 6 operations can be divided into a convolution kernel size, a convolution step size, and different stacking manners: [ VGGConv_k3_sd ], [ VGGConv_k5_sd ], [ VGGConv_k3_sd, VGGConv_k3_s1], [ VGGConv_k3_sd, VGGConv_k5_s1], [ VGGConv_k5_sd, VGGConv_k3_s1], [ VGGConv_k5_sd, VGGConv_k5_s1]. The structural weights v are randomly initialized.
The embodiment also provides a search space design, which is based on the MobileNet, wherein the search space comprises 1 path, 8 nodes are arranged on the path, each two nodes are called one Stage, 7 optional operations in each Stage are arranged, and the structure of the Stage is shown in figure 4. Each optional operation in Stage consists of MBConv, which is a building block of MobileNet, whose structure is shown in fig. 5, where building block mbconv_ka_tb contains three parts of extended convolution, depth separable convolution and mapped convolution, convolution kernel size k=a, expansion coefficient t=b, where convolution kernel size k controls the convolution kernel size of the depth separable convolution, and expansion coefficient t controls the number of output channels of the extended convolution.
The extended convolution comprises a 1 multiplied by 1 convolution layer, a BatchNorm layer and a ReLU activation layer, and the extended convolution has the function of expanding the characteristic map channel to b times of the number of input channels according to the expansion coefficient t. The depth separable convolution comprises an a×a convolution layer, a BatchNorm layer and a ReLU activation layer, and is used for feature extraction, so that only a single convolution kernel is applied to each channel in the convolution layer of the depth separable convolution to reduce the number of parameters. The mapping convolution comprises a 1×1 convolution layer, a BatchNorm layer and a ReLU activation layer, and the mapping convolution is used for mapping the feature map back to the input channel number.
As shown in fig. 4, the convolution kernel size and expansion coefficient of 7 operations are different, wherein the convolution kernel size k is 3 and MBConv with expansion coefficient t of 1 is denoted as mbconv_k3_t1. In the above nomenclature, the 7 operations are mbconv_k3_t1, mbconv_k3_t3, mbconv_k3_t6, mbconv_k5_t3, mbconv_k5_t6, mbconv_k7_t3, and mbconv_k7_t6, respectively. The structural weights v are randomly initialized.
Then, training set T train Divided into a first training set T train1 And a second training set T train2 The structure weight v and the network model weight w of the hybrid model are respectively used for training the search space. The division ratio can be adjusted according to the size of the search space and the complexity of the network model through experience of engineers: when the search space is small and the independent model structure is complex, T can be calculated train1 Is adjusted to 40 percent, T train2 Is adjusted to 60%; when the search space is large and the independent model has simple structure, T can be calculated train1 Is adjusted to 60 percent, T train2 Is adjusted to 40%; in the present embodiment, T train1 And T train2 The division is made in a ratio of 50% and 50%.
In the training process, firstly, the network model weight w is fixed, and in a first training set T train1 Optimizing structural weight v by the upper training mixed model; the structural weight v is fixed again, and in the second training set T train2 The upper training hybrid model optimizes the network model weights w.
The network model weight and the structure weight are optimized to count one round (epoch) at a time, and the number of epochs can be set according to the complexity degree of the search space. In the embodiment, the iterative training of 150 epochs is performed, and after each epoch is finished, an independent model structure is determined according to the maximum value of the structural weight of each operation. Inputting a validation set T val The evaluation of the independent model was carried out,the independent model with highest precision is the lightweight network structure obtained by searching the structure.
S5, taking the light-weight network structure obtained by searching as a student network;
s6, performing knowledge distillation on the teacher network model and the student network model.
Specifically, as shown in fig. 6, the training set T is first set train Input to the teacher network and the student network, and output of the teacher network and the student network in the logits layer is obtained. The output of the logits layer is then converted into a probability distribution of categories using a softmax function. Then, the difference of the teacher network and the student network class probability distribution is calculated by using the KL divergence, and the difference is used as a soft target loss L of the student network soft-Target (y student ，y teacher ) The calculation formula of the KL divergence is as follows:
wherein y is student Representing a class probability distribution of student network predictions, y teacher Representing the predicted class probability distribution of the teacher network. KL divergence can be used to measure the degree of difference between two distributions. The smaller the difference in the two distributions, the smaller the KL divergence and vice versa. When the two distributions are identical, their KL divergence is 0.
In an embodiment, sample A obtains a set of class probability distributions y teacher ＝[0.1，0，0，0，0.7，0，0，0，0，0.2]And y student ＝[0.2，0.1，0，0，0.5，0，0，0，0，0.2]The KL divergence calculation formula can be obtained according to the following formula:
the smaller the soft target loss, the closer the class distribution probability predicted by the student network model is to the class distribution probability predicted by the teacher network model. When the class distribution probability predicted by the student network model is completely consistent with the class distribution probability predicted by the teacher network model, the soft target loss is 0. In the knowledge distillation process, the student network model can quickly learn the predicted class probability distribution of the teacher network model by means of the loss of the soft target based on the KL divergence, so that the performance level close to the teacher network model is obtained.
Specifically, in order to avoid the student network model learning incorrect knowledge from the teacher network model, a knowledge validity verification weight H (y teacher Y), the calculation formula is as follows:
wherein argmax (·) represents the parametrization function as the result y predicted by the teacher network model teacher When the model is consistent with the label y, the knowledge of the teacher network model distillation is considered to be effective, so that a weight of 1 is output, namely the soft target loss of the sample is reserved; when teacher network model predicts result y teacher When the output weight is inconsistent with the label y, namely when the teacher network model is misclassified, the teacher network model is considered invalid in distilling knowledge, and the output weight is 0 at the moment, namely, the soft target loss is not reserved, and the student network model only optimizes the hard target loss.
Defining training loss function L of student network by combining soft target loss and knowledge effectiveness weight stuedent ：
Wherein α represents hard target loss L hard-Target (y student Weight of y), beta represents soft target loss L soft-Target (y student ，y teacher ) Is a weight of (2).
In an embodiment, a hard target loss L is set hard-Target (y student Weight α=1 of y), soft target loss L soft-Target (y student ，y teacher ) The purpose of the weight β=100 is to make the predicted class probability distribution of the student network model as close as possible to the teacher network model, thereby reducing the loss of accuracy. In practical application, the weights alpha and beta can be adjusted with the learning condition of the student network model. In an embodiment, sample A is labeled [0,0,0,0,1,0,0,0,0,0 ]]The loss of the last sample a can be calculated as:
L stuedent the larger the difference between the class distribution probability predicted by the student network model and the class distribution probability predicted by the teacher network model and the actual class distribution probability is larger.
Specifically, the student network is iteratively trained according to the defined student network training loss function until the student network model converges.
In the embodiment, according to the defined student network training loss function, iterative training is carried out on the student network through an Adam optimization algorithm until the loss of the student network model converges, so that the precision level is close to the teacher network level.
And S7, outputting a student network model after knowledge distillation, namely a model after compression.
In summary, the embodiment of the invention provides a convolutional neural network model compression method based on structure search and knowledge distillation, which solves the problem of convolutional neural network model parameter redundancy and reduces the waste of computing resources such as computing power, memory, storage and the like of an operation platform. Meanwhile, in the traditional knowledge distillation method, the student network model is designed manually, a great deal of time and engineer experience are needed to design the efficient lightweight chemical network model and other problems. In practical application, compared with the existing model compression method, the method has the advantages of high model compression rate, less precision loss and the like, and the model compression rate is more than 70%, and the precision loss is less than 1%, so that the method has practicability.
Based on the same inventive concept, the embodiment also provides a convolutional neural network model compression device based on structure search and knowledge distillation, which comprises a memory and a processor, wherein the memory is used for storing a computer program, and the processor is used for realizing the convolutional neural network model compression method based on structure search and knowledge distillation when executing the computer program.
Based on the same inventive concept, the embodiment also provides a computer readable storage medium, wherein the storage medium stores a computer program, and the computer program realizes the convolutional neural network model compression method based on the structure search and the knowledge distillation when the computer program uses a computer.
It should be noted that, the convolutional neural network model compression device and the computer-readable storage medium based on the structure search and the knowledge distillation provided in the foregoing embodiments belong to the same concept as the convolutional neural network model compression method based on the structure search and the knowledge distillation, and detailed implementation processes of the convolutional neural network model compression method based on the structure search and the knowledge distillation are described in the embodiment, which is not repeated here.
The foregoing detailed description of the preferred embodiments and advantages of the invention will be appreciated that the foregoing description is merely illustrative of the presently preferred embodiments of the invention, and that no changes, additions, substitutions and equivalents of those embodiments are intended to be included within the scope of the invention.
Claims (10)
1. The convolutional neural network model compression method based on structure search and knowledge distillation is characterized by comprising the following steps of:
step S1, acquiring a target task data set T;
step S2, dividing a training set T from the target task data set T train By means of training sets T train Training a convolutional neural network model;
step S3, the convolutional neural network model is used as a teacher network after training is completed;
s4, searching a lightweight network structure for the convolutional neural network model by utilizing a neural structure searching technology;
s5, taking the light-weight network structure obtained by searching as a student network;
step S6, knowledge distillation is carried out, and the training set T is obtained train Respectively inputting the difference into a teacher network and a student network, calculating the difference of output of a Softmax layer of the teacher network and the student network, taking the difference as a part of the loss of the student network, and performing iterative training on the student network until convergence;
and S7, outputting a student network model after knowledge distillation, namely a model after compression.
2. The convolutional neural network model compression method based on structural search and knowledge distillation of claim 1, wherein step S2 comprises:
s21, dividing the target task data set T to obtain a training set T train Verification set T val And test set T test ；
S22, training set T train Inputting into a convolutional neural network model;
s23, setting a loss function L of a training convolutional neural network model:
where x represents the training input, y represents the resulting label, W represents the weight of the target network, f (x, W) represents the output of the target network logits layer, softmax (·) represents the probability distribution of converting the output of the logits layer into categories using the softmax function, l cross (. Cndot.) represents the cross entropy of the convolutional neural network as a loss function of training;
s24, training target network parameters according to the set loss function;
s25, outputting the convolutional neural network model with the trained parameters.
3. The convolutional neural network model compression method based on structural search and knowledge distillation of claim 2, wherein step S4 comprises:
s41, defining a student network search space, constructing a mixed model by the convolutional neural network model through the search space, wherein the mixed model comprises a plurality of independent models, and respectively endowing different structural weights v to the operation among nodes of the search space;
s42, training set T train Divided into a first training set T train1 And a second training set T train2 The method comprises the steps of respectively training structural weights v of the hybrid model and network model weights w of the hybrid model;
s43, fixing the network model weight w in the first training set T train1 Optimizing structural weight v by the upper training mixed model;
s44, fixing the structural weight v, in the second training set T train2 The upper training mixed model optimizes the network model weight w;
s45, repeating the steps S43-S44 for a plurality of rounds, determining an independent model structure according to the maximum value of the structural weight of each operation after each round is finished, and inputting a verification set T val And evaluating the independent model, namely obtaining the lightweight network structure by searching the independent model with highest precision.
4. The convolutional neural network model compression method based on structure search and knowledge distillation of claim 3, wherein the search space is a chain structure and comprises M paths, each path has N nodes, and K operations can be adopted between every two nodes.
5. The method for compressing convolutional neural network model based on structural search and knowledge distillation as recited in claim 4, wherein the search space is designed based on VGGNet, the operations between every two nodes are composed of different VGGConv, VGGConv is a building block of VGGNet, when the feature map in the target task data set T is input into the building block VGGConv, the input feature map is first subjected to convolution with a convolution kernel size of k×k and a convolution step size of s, then activated by a ReLU function, and finally subjected to batch normalization to obtain the output feature map.
6. The convolutional neural network model compression method based on structural search and knowledge distillation according to claim 5, wherein the convolution step size is obtained by calculating an input feature map size fm_size1 and an output feature map size fm_size2, so that the number of operation stacks of the adaptive search building block VGGConv is realized in a search space, the alignment of different operation output feature map sizes is ensured, and a calculation formula of the convolution step size s is as follows:
7. the convolutional neural network model compression method based on structural search and knowledge distillation of claim 4, wherein the search space is designed based on MobileNet, the operation between every two nodes consists of different MBConv, which is a building block of MobileNet, the building block MBConv comprising extended convolution, depth separable convolution and mapped convolution; the extended convolution comprises a convolution layer with the size of 1 multiplied by 1, a BatchNorm layer and a ReLU activation layer, and expands the characteristic map channel according to the expansion coefficient t; the depth separable convolution comprises a k multiplied by k convolution layer, a BatchNorm layer and a ReLU activation layer, wherein each channel in the convolution layer only uses a single convolution kernel to reduce the number of parameters, and the feature extraction is carried out through the depth separable convolution; the mapping convolution comprises a 1×1 convolution layer, a BatchNorm layer and a ReLU activation layer, and the feature map is mapped back to the input channel number through the mapping convolution.
8. The convolutional neural network model compression method based on structure search and knowledge distillation of claim 2, wherein step S6 comprises:
s61, training set T train Inputting the data into a teacher network and a student network, and obtaining the output of the teacher network and the student network in a logits layer;
s62, converting the output of the logits layer into probability distribution of category by using softmax function;
s63, calculating differences of class probability distributions of the teacher network and the student network by using KL divergence, and taking the differences as soft target loss L of the student network sof t -Target (y student ,y teacher ) The calculation formula is as follows:
wherein y is student Representing a class probability distribution of student network predictions, y teacher Representing class probability distribution predicted by a teacher network;
s64, designing a knowledge validity verification weight H (y teacher Y), avoiding the student network model learning incorrect knowledge from the teacher network model:
wherein argmax (·) represents a parametrization function, and when the teacher network model predicts the result y teacher Outputting a validity verification weight value of 1 when the validity verification weight value is consistent with the label y, and predicting a result y when a teacher network model teacher When the tag y is inconsistent with the tag y, outputting a validity verification weight value of 0;
s65, defining training loss function L of student network stuedent ：
Wherein α represents hard target loss L hard-Target (y student Weight of y), beta represents soft target loss L soft-Target (y student ,y teacher ) Weights of (2);
and S66, performing iterative training on the student network according to the defined student network training loss function until the student network model converges.
9. A convolutional neural network model compression device based on structure search and knowledge distillation, comprising a memory for storing a computer program, and a processor for implementing the convolutional neural network model compression method based on structure search and knowledge distillation of any one of claims 1-8 when the computer program is executed.
10. A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program, when used in a computer, implements the convolutional neural network model compression method based on structure search and knowledge distillation as claimed in any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311073117.8A CN117114053A (en) | 2023-08-24 | 2023-08-24 | Convolutional neural network model compression method and device based on structure search and knowledge distillation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311073117.8A CN117114053A (en) | 2023-08-24 | 2023-08-24 | Convolutional neural network model compression method and device based on structure search and knowledge distillation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117114053A true CN117114053A (en) | 2023-11-24 |
Family
ID=88803395
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311073117.8A Pending CN117114053A (en) | 2023-08-24 | 2023-08-24 | Convolutional neural network model compression method and device based on structure search and knowledge distillation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117114053A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117612214A (en) * | 2024-01-23 | 2024-02-27 | 南京航空航天大学 | Pedestrian search model compression method based on knowledge distillation |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110175628A (en) * | 2019-04-25 | 2019-08-27 | 北京大学 | A kind of compression algorithm based on automatic search with the neural networks pruning of knowledge distillation |
| CN113159173A (en) * | 2021-04-20 | 2021-07-23 | 北京邮电大学 | Convolutional neural network model compression method combining pruning and knowledge distillation |
| CN115359353A (en) * | 2022-08-19 | 2022-11-18 | 广东工业大学 | Flower identification and classification method and device |
| CN115601660A (en) * | 2022-10-26 | 2023-01-13 | 广东南方数码科技股份有限公司（Cn） | Remote sensing image change detection method based on neural network structure search |
| CN116362325A (en) * | 2023-03-28 | 2023-06-30 | 南京大学 | Electric power image recognition model lightweight application method based on model compression |
| CN116362294A (en) * | 2023-05-30 | 2023-06-30 | 深圳比特微电子科技有限公司 | Neural network searching method and device and readable storage medium |
-
2023
- 2023-08-24 CN CN202311073117.8A patent/CN117114053A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110175628A (en) * | 2019-04-25 | 2019-08-27 | 北京大学 | A kind of compression algorithm based on automatic search with the neural networks pruning of knowledge distillation |
| CN113159173A (en) * | 2021-04-20 | 2021-07-23 | 北京邮电大学 | Convolutional neural network model compression method combining pruning and knowledge distillation |
| CN115359353A (en) * | 2022-08-19 | 2022-11-18 | 广东工业大学 | Flower identification and classification method and device |
| CN115601660A (en) * | 2022-10-26 | 2023-01-13 | 广东南方数码科技股份有限公司（Cn） | Remote sensing image change detection method based on neural network structure search |
| CN116362325A (en) * | 2023-03-28 | 2023-06-30 | 南京大学 | Electric power image recognition model lightweight application method based on model compression |
| CN116362294A (en) * | 2023-05-30 | 2023-06-30 | 深圳比特微电子科技有限公司 | Neural network searching method and device and readable storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117612214A (en) * | 2024-01-23 | 2024-02-27 | 南京航空航天大学 | Pedestrian search model compression method based on knowledge distillation |
| CN117612214B (en) * | 2024-01-23 | 2024-04-12 | 南京航空航天大学 | Pedestrian search model compression method based on knowledge distillation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114241282B (en) | Knowledge distillation-based edge equipment scene recognition method and device | |
| CN112001496B (en) | Neural network structure searching method and system, electronic device and storage medium | |
| CN112365885B (en) | Training method and device of wake-up model and computer equipment | |
| CN111598213B (en) | Network training method, data identification method, device, equipment and medium | |
| CN112926570B (en) | Adaptive bit network quantization method, system and image processing method | |
| CN117114053A (en) | Convolutional neural network model compression method and device based on structure search and knowledge distillation | |
| CN111782826A (en) | Knowledge graph information processing method, device, equipment and storage medium | |
| CN114819143A (en) | Model compression method suitable for communication network field maintenance | |
| US6317730B1 (en) | Method for optimizing a set of fuzzy rules using a computer | |
| CN115048870A (en) | Target track identification method based on residual error network and attention mechanism | |
| CN114863226A (en) | Network physical system intrusion detection method | |
| CN116363423A (en) | Knowledge distillation method, device and storage medium for small sample learning | |
| CN110728186A (en) | Fire detection method based on multi-network fusion | |
| CN114004152A (en) | Multi-wind-field wind speed space-time prediction method based on graph convolution and recurrent neural network | |
| CN110647917B (en) | Model multiplexing method and system | |
| CN115298671A (en) | Method and device for optimizing neural network model on equipment by using sub-kernel search module | |
| CN112395952A (en) | A unmanned aerial vehicle for rail defect detection | |
| CN112068088A (en) | Radar radiation source threat assessment method based on optimized BP neural network | |
| CN111062477B (en) | Data processing method, device and storage medium | |
| KR20220111215A (en) | Apparatus and method for predicting drug-target interaction using deep neural network model based on self-attention | |
| Guo et al. | Dynamic neural network structure: A review for its theories and applications | |
| CN114091668A (en) | Neural network pruning method and system based on micro-decision maker and knowledge distillation | |
| CN116957017A (en) | Acceleration method and device for neural network model in computer equipment | |
| CN115690416A (en) | Knowledge distillation-based BEV semantic segmentation model training method, system, equipment and medium | |
| CN117077767A (en) | AIGC model establishing method based on federal learning and architecture searching |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |