CN116245162A - Neural network pruning method and system based on improved adaptive genetic algorithm - Google Patents
Neural network pruning method and system based on improved adaptive genetic algorithm Download PDFInfo
- Publication number
- CN116245162A CN116245162A CN202211492198.0A CN202211492198A CN116245162A CN 116245162 A CN116245162 A CN 116245162A CN 202211492198 A CN202211492198 A CN 202211492198A CN 116245162 A CN116245162 A CN 116245162A
- Authority
- CN
- China
- Prior art keywords
- neural network
- network
- pruning
- value
- individuals
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013138 pruning Methods 0.000 title claims abstract description 55
- 238000000034 method Methods 0.000 title claims abstract description 51
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 49
- 238000004422 calculation algorithm Methods 0.000 title claims abstract description 38
- 230000002068 genetic effect Effects 0.000 title claims abstract description 35
- 230000003044 adaptive effect Effects 0.000 title claims description 14
- 238000004364 calculation method Methods 0.000 claims abstract description 11
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 10
- 238000009966 trimming Methods 0.000 claims abstract description 4
- 230000035772 mutation Effects 0.000 claims description 20
- 230000006870 function Effects 0.000 claims description 18
- 210000000349 chromosome Anatomy 0.000 claims description 14
- 238000011478 gradient descent method Methods 0.000 claims description 4
- 238000010187 selection method Methods 0.000 claims description 4
- 239000011159 matrix material Substances 0.000 claims description 3
- 230000006835 compression Effects 0.000 abstract description 3
- 238000007906 compression Methods 0.000 abstract description 3
- 238000003062 neural network model Methods 0.000 abstract description 2
- 108090000623 proteins and genes Proteins 0.000 description 6
- 230000000694 effects Effects 0.000 description 5
- 230000006978 adaptation Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000005094 computer simulation Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/086—Learning methods using evolutionary algorithms, e.g. genetic algorithms or genetic programming
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/12—Computing arrangements based on biological models using genetic models
- G06N3/126—Evolutionary algorithms, e.g. genetic algorithms or genetic programming
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Theoretical Computer Science (AREA)
- Biophysics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biomedical Technology (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Physiology (AREA)
- Genetics & Genomics (AREA)
- Feedback Control In General (AREA)
Abstract
The invention belongs to the technical field of neural networks, and particularly relates to a neural network pruning method and system based on an improved self-adaptive genetic algorithm, wherein the neural network pruning method comprises the following steps: s1, pre-training a convolutional neural network to obtain trained network parameter weights; s2, searching an optimal network structure of each layer by utilizing an improved self-adaptive genetic algorithm, and pruning a redundant convolution kernel; s3, a retraining process is carried out, and network precision is recovered; s4, searching the next layer, and repeating the steps S2-S3; s5, trimming all the convolution layers, and obtaining the neural network with the optimal structure. The invention reduces the number of convolution kernels of the convolution layer while ensuring the accuracy of the neural network model, reduces the number of redundant parameters, accelerates calculation and can realize model compression.
Description
Technical Field
The invention belongs to the technical field of neural networks, and particularly relates to a neural network pruning method and system based on an improved self-adaptive genetic algorithm.
Background
Neural networks have enjoyed great success in the application of computer vision, for example: image classification, face recognition, object detection, machine translation, etc. However, for complex tasks, a deeper neural network is required. As the depth of the network model increases, the number of parameters and the amount of calculation are multiplied. The increasing depth of the network model is disadvantageous to be deployed in embedded devices such as smart phones or wearable devices. While the latest hardware has an acceleration framework dedicated to neural networks, there is still a need to reduce model size to save memory space; at the same time, the influence on the model precision is not too great.
There are complex structures and huge amounts of parameters in neural networks, where pruning can remove this part of redundancy without lack of unnecessary or unimportant parameters and network structure, and can be classified into unstructured pruning and structured pruning according to the size of the pruning parameter scale.
Unstructured pruning is the finest granularity pruning, sometimes referred to as weight pruning, which is simple to implement, but there are still some limitations: deleting a small number of parameters is not significant in reducing network storage and computation; unstructured pruning tends to result in a large number of sparse matrices, which often require specialized hardware to achieve the effect of accelerating computation, which can be costly.
Structured pruning is coarser in granularity compared with unstructured pruning, and can be roughly classified into convolution kernel pruning, filter pruning and layer pruning according to the small to large pruning granularity; the method can lead the network model to generate structural sparsity, and the sparsity is very beneficial to saving the computing resources in hardware systems such as embedded computers, parallel computing systems and the like. However, the determination of the redundant structure is an NP-hard problem, and most of the existing methods are based on certain criteria, such as L1, L2 norms, etc., or introduce sparse penalty terms during training, which tend to have high computational complexity, and have limited model compression effect, so that it is difficult to find the optimal structure.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a neural network pruning method and a neural network pruning system based on an improved self-adaptive genetic algorithm, and aims to find an optimal structure of a neural network, reduce the number of parameters, reduce the network performance loss and achieve the effect of compressing a model.
In order to achieve the above object, the present invention adopts the following technical scheme:
a neural network pruning method based on an improved adaptive genetic algorithm comprises the following steps:
s1, pre-training a convolutional neural network to obtain trained network parameter weights;
s2, searching an optimal network structure of each layer by utilizing an improved self-adaptive genetic algorithm, and pruning a redundant convolution kernel;
s3, a retraining process is carried out, and network precision is recovered;
s4, searching the next layer, and repeating the steps S2-S3;
s5, trimming all the convolution layers, and obtaining the neural network with the optimal structure.
Preferably, in the step S2, the improved adaptive genetic algorithm includes the following steps:
s21, setting the maximum iteration times r, the population number m, the parent number k and the super parameter omega;
s22, coding a convolution kernel matrix of a convolution layer into binary codes, wherein each code is a chromosome, and carrying out random value taking on an initial chromosome to generate a population with m individuals;
s23, calculating the fitness value of each individual;
s24, selecting k individuals as parents by using a roulette selection method;
s25, randomly selecting two optimal individuals to perform cross operation, wherein P is arranged at each position c Is exchanged to generate two offspring;
s26, carrying out mutation operation on each sub-generation, and carrying out mutation operation on each coding positionAll have P m The probability of (2) is mutated to produce a new individual;
s27, repeating the steps S23-S26, and enabling the k individuals to enter the next iteration until the maximum iteration number is reached.
In a preferred embodiment, in the step S22, a series of binary code strings are randomly generated in the encoding process, and the number of binary code strings is equal to the number of populations, and the expression is as follows:
S＝{s 1 ,s 2 ,...,s n }
where n is the number of convolution kernels per layer, s i A code value representing the ith convolution kernel, i=1, 2, …, n; s is(s) i ∈{0,1}，s i A value of 1 means that the convolution kernel, s, is preserved i A value of 0 indicates that the convolution kernel is deleted.
Preferably, in the step S23, the fitness value f of the jth individual j The calculation formula of (2) is as follows:
wherein ρ is j For the number of encoded network parameters, ρ, of the jth individual 0 For the number of initial network parameters before coding of the jth individual, ω is a constant with a value of (0, 1) interval, representing a loss function and a weight occupied by a network structure, and L represents a loss function of model accuracy, and is obtained by calculating the output cross entropy:
wherein y is pq Representing the q-th output value of the p-th convolutional layer.
In a preferred embodiment, in the step S24, k individuals are selected as parents by using a roulette selection method, and the probability of occurrence of the individuals in the offspring is calculated according to the fitness value of the individuals, wherein the probability of selection of the individuals with larger fitness value is larger, and the individuals with smaller fitness value tend to be unselected, and the expression is as follows:
wherein P is j The probability of being selected for the jth individual.
Preferably, in step S25, the two chromosomes are swapped using a single point crossover, i.e., the right hand portion is segmented and swapped at randomly selected locations, where the crossover probability P c The expression of (2) is:
wherein k is 1 、k 2 、k 3 Are all constants between 0 and 1, k 2 >k 3 ，f c For higher fitness value in the first two parents of the crossover operation, f max And f min Is the maximum and minimum fitness in the population.
Preferably, in the step S26, the mutation probability P m The expression of (2) is:
wherein k is 4 ，k 5 ，k 6 Are all constants between 0 and 1, k 5 >k 6 ，f c For the higher fitness value of the two parents before the mutation operation, f max And f min Is the maximum and minimum fitness in the population.
Preferably, in the step S3, the retraining process includes the following steps:
s31, inputting a data set into a neural network by applying the network structure of the offspring obtained last time, and calculating a loss function value L of the data set;
s32, updating the network parameter weight by using a gradient descent method, wherein the expression is as follows:
wherein W' is the weight after updating, W is the weight before updating, eta is the learning rate,
is the partial derivative of L to W;
s33, repeating the steps S31-S32 until the loss function L converges.
The invention also provides a neural network pruning system based on the improved adaptive genetic algorithm, which applies the neural network pruning method according to any one of the schemes, and comprises the following steps:
the pre-training module is used for pre-training the convolutional neural network to obtain trained network parameter weights;
the pruning module is used for searching the optimal network structure of each layer by utilizing the improved self-adaptive genetic algorithm and pruning the redundant convolution kernel;
the retraining module is used for retraining the pruned neural network and recovering network precision;
and the output module is used for outputting the neural network with the optimal structure obtained after all the convolution layers are trimmed.
Compared with the prior art, the invention has the following advantages:
1. compared with a method for training the model by using sparse punishment items or LASSO operators and the like, the method can omit the training process and reduce the calculation time by using the pre-trained model;
2. the invention adopts the genetic algorithm to search the optimal structure of each layer, the genetic algorithm is used as a computer simulation research, has strong global searching capability, and carries out global optimization based on probability. In the existing research, most pruning methods utilize certain criteria to measure the importance of a network structure, such as calculating the L1 norm, the L2 norm, the entropy and the like of the structure, and the calculation complexity is high. Compared with the methods, the method does not need strict mathematical derivation, has no function use limitation, can adaptively adjust the search path, and has universal applicability to various problems.
3. Compared with the common genetic algorithm, the invention does not need to set the cross probability, the variation probability and other super parameters in advance, and the parameters are automatically changed according to the fitness value in the cross and variation processes, so that the parameters do not need to be manually changed, and the efficiency is higher.
4. The invention improves the calculation of the fitness value of the genetic algorithm, and is different from other pruning methods based on the genetic algorithm, generally only uses the loss function of the model as the fitness value.
Drawings
FIG. 1 is a flow chart of a neural network pruning method based on an improved adaptive genetic algorithm;
FIG. 2 is a diagram of the encoding process in the genetic algorithm of the present invention;
FIG. 3 is a cross-process diagram in the genetic algorithm of the present invention;
FIG. 4 is a graph showing the mutation process in the genetic algorithm of the present invention.
Detailed Description
In order to more clearly illustrate the embodiments of the present invention, specific embodiments of the present invention will be described below with reference to the accompanying drawings. It is evident that the drawings in the following description are only examples of the invention, from which other drawings and other embodiments can be obtained by a person skilled in the art without inventive effort.
The neural network pruning method based on the improved self-adaptive genetic algorithm comprises the steps of pre-training a convolutional neural network to obtain trained network parameter weights, searching the optimal network structure of each layer by utilizing the improved self-adaptive genetic algorithm, pruning redundant convolution kernels, retraining the model when the maximum iteration number is reached, recovering network precision, and then entering the search of the next layer until all the convolutional layers are pruned, wherein the neural network with the optimal structure is obtained. The invention mainly solves the problems of redundant parameters and large calculated amount of the neural network, and comprises the following main processes: pre-training the neural network to obtain trained parameter values; and then, coding a convolution kernel of each convolution layer by utilizing an improved self-adaptive genetic algorithm to form a plurality of combinations of possible individuals, calculating the fitness value of each individual, selecting population individuals by utilizing a selection strategy, obtaining diversified individuals through crossover and mutation operation, and then, re-entering the next layer of iteration. And (3) retraining the model to recover part of accuracy until the maximum iteration number is reached, entering a next layer of network, searching the optimal structure again by using a genetic algorithm until all convolution layers are calculated, and obtaining the final structure as an optimal solution. The invention reduces the number of convolution kernels of the convolution layer while ensuring the accuracy of the neural network model, reduces the number of redundant parameters, accelerates calculation and can realize model compression.
As shown in fig. 1, the specific process of the neural network pruning method based on the improved adaptive genetic algorithm in the embodiment of the invention comprises the following steps:
s1, the pretreatment process of the invention is as follows: the special data sets, MNIST, CIFAR-10, CIFAR-100 and the like are downloaded from an open source website, the convolutional neural network is trained, the weight is updated to be stable, and network parameters are stored locally.
S2, searching an optimal network structure of each layer by utilizing an improved self-adaptive genetic algorithm, and pruning a redundant convolution kernel; the adaptive genetic algorithm searching process of the embodiment of the invention comprises the following steps:
(1) Firstly, setting the maximum iteration times r, the population quantity m, the parent quantity k and the super parameter omega.
(2) The convolution kernel matrix of the convolution layer is coded into binary codes, each code is a chromosome, and the initial chromosome is randomly valued to generate a population with m individuals; i.e. the convolution kernels in the next layer of convolution layer are encoded, a number m of binary code strings is generated according to the number of convolution kernels of the layer, 1 and 0 are randomly distributed, as shown in fig. 2, wherein 1 represents a guardLeaving the convolution kernel (dark gray portion in fig. 2), 0 represents pruning the convolution kernel (white portion in fig. 2), and can be expressed as s= { S 1 ,s 2 ,...,s n Each code string representing an individual; where n is the number of convolution kernels per layer, s i A code value representing the ith convolution kernel, i=1, 2, …, n; s is(s) i ∈{0,1}，s i A value of 1 means that the convolution kernel, s, is preserved i A value of 0 indicates that the convolution kernel is deleted.
(3) Calculating fitness value of each individual:
wherein ρ is j For the number of encoded network parameters, ρ, of the jth individual 0 For the number of initial network parameters before coding of the jth individual, ω is a constant with a value of (0, 1) interval, representing a loss function and a weight occupied by a network structure, and L represents the loss function of the model accuracy, which can be calculated by using the output cross entropy:
wherein y is pq Representing the q-th output value of the p-th convolutional layer. The higher the fitness value, the better the model effect, the more excellent the genes of the chromosome;
(4) Selecting k individuals as parents, calculating the total number of fitness values of each individual, and calculating the probability of the individual appearing in offspring according to the fitness values, wherein the larger the fitness value is, the larger the probability of the individual being selected, and the smaller the fitness value is, the more the individual tends to be unselected, and the expression is:
wherein P is i The probability of being selected for the jth individual.
(5) Crossing two chromosomes using single point crossoverInstead, the right-hand part is segmented and swapped at randomly selected locations, as shown in FIG. 3, where the crossover probability P c For the adaptation value, the expression is:
wherein k is 1 ，k 2 ，k 3 Are all constants between 0 and 1, k 2 >k 3 ，f c For the higher fitness value of the two parents before the crossover operation, f max And f min Is the maximum and minimum fitness in the population. After crossing, two filial generations are generated, and the excellent genes of the father generation can be inherited, so that the selection and calculation of the next generation can be carried out.
(6) In the mutation strategy, mutation is performed at each individual random position, and becomes 0 if the current position is coded as 1, and becomes 1 if the current position is coded as 0, as shown in FIG. 4, wherein the mutation probability P m Is an adaptive value, and the expression is as follows:
wherein k is 4 ，k 5 ，k 6 Are all constants between 0 and 1, k 5 >k 6 ，f c For the higher fitness value of the two parents before the mutation operation, f max And f min Is the maximum and minimum fitness in the population. The mutation can change part of genes of the chromosome, so that the algorithm is prevented from falling into a local optimal solution;
(6) Performing the next iteration, calculating the fitness value of the population individuals, repeating the processes of selection, crossing and mutation, retaining excellent individuals (network results), and eliminating inferior individuals;
(7) When the maximum iteration number r is reached, the codes of k excellent individuals are obtained, the code with the largest fitness value is selected, the convolution kernel with the code value of 1 is reserved in the network structure, and the convolution kernel with the code value of 0 is removed.
S3, retraining: and training the network again, and recovering part of accuracy. The method specifically comprises the following steps:
a retraining process comprising the steps of:
s31, inputting a data set into a neural network by applying the network structure of the offspring obtained last time, and calculating a loss function value L of the data set;
s32, updating the network parameter weight by using a gradient descent method, wherein the expression is as follows:
wherein W' is the weight after updating, W is the weight before updating, eta is the learning rate,
is the partial derivative of L to W;
s33, repeating the steps S31-S32 until the loss function L converges.
S4, searching the next layer, and repeating the steps S2-S3.
And S5, calculating all convolution layers, and reserving the final structure to obtain the optimal network structure.
The neural network pruning method of the embodiment of the invention can be realized through python, and coding, calculation and the like involved in the algorithm are realized by using a numpy library.
The network structure mainly used in the embodiment of the invention is LeNet, resNet-32 and AlexNet, and the network structure can be built by using a pyrach library.
The embodiment of the invention also provides a neural network pruning system based on the improved self-adaptive genetic algorithm, which comprises a pre-training module, a pruning module, a retraining module and an output module.
Specifically, the pre-training module of the embodiment of the invention is used for pre-training the convolutional neural network to obtain trained network parameter weights, specifically, downloading a special data set, MNIST, CIFAR-10, CIFAR-100 and the like from an open source website, training the convolutional neural network, updating the weights to enable the convolutional neural network to be stable, and storing network parameters to the local.
The pruning module of the embodiment of the invention is used for searching the optimal network structure of each layer by utilizing the improved self-adaptive genetic algorithm and pruning the redundant convolution kernel.
Specifically, the adaptive genetic algorithm searching process of the embodiment of the invention is as follows:
(1) Firstly, setting the maximum iteration times r, the population quantity m, the parent quantity k and the super parameter omega.
(2) Encoding the convolution kernels in the next convolution layer, generating a binary code string with m number according to the number of the convolution kernels, and randomly distributing 1 and 0, as shown in fig. 2, wherein 1 represents that the convolution kernels (dark gray part in fig. 2) are reserved, and 0 represents that the convolution kernels are pruned (white part in fig. 2), which can be expressed as s= { S 1 ,s 2 ,...,s n Each code string representing an individual; where n is the number of convolution kernels per layer, s i A code value representing the ith convolution kernel, i=1, 2, …, n; s is(s) i ∈{0,1}，s i A value of 1 means that the convolution kernel, s, is preserved i A value of 0 indicates that the convolution kernel is deleted.
(3) Calculating fitness value of each individual:
wherein ρ is j For the number of encoded network parameters, ρ, of the jth individual 0 For the number of initial network parameters before coding of the jth individual, ω is a constant with a value of (0, 1) interval, representing a loss function and a weight occupied by a network structure, and L represents the loss function of the model accuracy, which can be calculated by using the output cross entropy:
wherein y is pq Representing the q-th output value of the p-th convolutional layer. The higher the fitness value, the better the model effect, the more excellent the genes of the chromosome;
(4) Selecting k individuals as parents, calculating the total number of fitness values of each individual, and calculating the probability of the individual appearing in offspring according to the fitness values, wherein the larger the fitness value is, the larger the probability of the individual being selected, and the smaller the fitness value is, the more the individual tends to be unselected, and the expression is:
wherein P is i The probability of being selected for the jth individual.
(5) Swapping two chromosomes with a single point crossover, partitioning at randomly selected positions and swapping right-hand parts, as shown in FIG. 3, where crossover probability P c For the adaptation value, the expression is:
wherein k is 1 ，k 2 ，k 3 Are all constants between 0 and 1, k 2 >k 3 ，f c For the higher fitness value of the two parents before the crossover operation, f max And f min Is the maximum and minimum fitness in the population. After crossing, two filial generations are generated, and the excellent genes of the father generation can be inherited, so that the selection and calculation of the next generation can be carried out.
(6) In the mutation strategy, mutation is performed at each individual random position, and becomes 0 if the current position is coded as 1, and becomes 1 if the current position is coded as 0, as shown in FIG. 4, wherein the mutation probability P m Is an adaptive value, and the expression is as follows:
wherein k is 4 ，k 5 ，k 6 Are all constants between 0 and 1, k 5 >k 6 ，f c For the higher fitness value of the two parents before the mutation operation, f max And f min Is the maximum and minimum fitness in the population. The mutation can change part of genes of the chromosome, so that the algorithm is prevented from falling into a local optimal solution;
(6) Performing the next iteration, calculating the fitness value of the population individuals, repeating the processes of selection, crossing and mutation, retaining excellent individuals (network results), and eliminating inferior individuals;
(7) When the maximum iteration number r is reached, the codes of k excellent individuals are obtained, the code with the largest fitness value is selected, the convolution kernel with the code value of 1 is reserved in the network structure, and the convolution kernel with the code value of 0 is removed.
The retraining module is used for retraining the pruned neural network and recovering network accuracy. The specific retraining process is as follows:
(1) Applying the last obtained network structure of the offspring, inputting the data set into the neural network, and calculating the loss function value L of the data set;
(2) The network parameter weight is updated by using a gradient descent method, and the expression is as follows:
wherein W' is the weight after updating, W is the weight before updating, eta is the learning rate,
is the partial derivative of L to W;
(3) Repeating the steps (1) to (2) until the loss function L converges.
The output module of the embodiment of the invention is used for outputting the neural network with the optimal structure obtained after trimming all the convolution layers.
The foregoing is only illustrative of the preferred embodiments and principles of the present invention, and changes in specific embodiments will occur to those skilled in the art upon consideration of the teachings provided herein, and such changes are intended to be included within the scope of the invention as defined by the claims.
Claims (9)
1. The neural network pruning method based on the improved adaptive genetic algorithm is characterized by comprising the following steps of:
s1, pre-training a convolutional neural network to obtain trained network parameter weights;
s2, searching an optimal network structure of each layer by utilizing an improved self-adaptive genetic algorithm, and pruning a redundant convolution kernel;
s3, a retraining process is carried out, and network precision is recovered;
s4, searching the next layer, and repeating the steps S2-S3;
s5, trimming all the convolution layers, and obtaining the neural network with the optimal structure.
2. The neural network pruning method according to claim 1, wherein in the step S2, the improved adaptive genetic algorithm comprises the following steps:
s21, setting the maximum iteration times r, the population number m, the parent number k and the super parameter omega;
s22, coding a convolution kernel matrix of a convolution layer into binary codes, wherein each code is a chromosome, and carrying out random value taking on an initial chromosome to generate a population with m individuals;
s23, calculating the fitness value of each individual;
s24, selecting k individuals as parents by using a roulette selection method;
s25, randomly selecting two optimal individuals to perform cross operation, wherein P is arranged at each position c Is exchanged to generate two offspring;
s26, carrying out mutation operation on each child generation, wherein each coding position of each child generation is provided with P m The probability of (2) is mutated to produce a new individual;
s27, repeating the steps S23-S26, and enabling the k individuals to enter the next iteration until the maximum iteration number is reached.
3. The neural network pruning method according to claim 2, wherein in the step S22, a series of binary code strings are randomly generated during the encoding process, and the number of binary code strings is equal to the number of populations, and the expression is:
S＝{s 1 ,s 2 ,...,s n }
where n is the number of convolution kernels per layer, s i A code value representing the ith convolution kernel, i=1, 2, …, n; s is(s) i ∈{0,1}，s i A value of 1 means that the convolution kernel, s, is preserved i A value of 0 indicates that the convolution kernel is deleted.
4. The neural network pruning method according to claim 3, wherein in the step S23, the fitness value f of the jth individual j The calculation formula of (2) is as follows:
wherein ρ is j For the number of encoded network parameters, ρ, of the jth individual 0 For the number of initial network parameters before coding of the jth individual, ω is a constant with a value of (0, 1) interval, representing a loss function and a weight occupied by a network structure, and L represents a loss function of model accuracy, and is obtained by calculating the output cross entropy:
wherein y is pq Representing the q-th output value of the p-th convolutional layer.
5. The neural network pruning method according to claim 4, wherein in the step S24, k individuals are selected as parents by using a roulette selection method, the probability of occurrence of individuals in offspring is calculated according to the fitness value of the individuals, the probability of selection of individuals with a larger fitness value is higher, and individuals with a smaller fitness value tend to be unselected, and the expression is:
wherein P is j The probability of being selected for the jth individual.
6. The neural network pruning method according to claim 5, wherein in the step S25, two chromosomes are swapped by single-point crossover, i.e., the right-hand part is segmented and swapped at randomly selected positions, wherein the crossover probability P c The expression of (2) is:
wherein k is 1 、k 2 、k 3 Are all constants between 0 and 1, k 2 >k 3 ，f c For higher fitness value in the first two parents of the crossover operation, f max And f min Is the maximum and minimum fitness in the population.
7. The neural network pruning method according to claim 6, wherein in the step S26, the variation probability P m The expression of (2) is:
wherein k is 4 ，k 5 ，k 6 Are all constants between 0 and 1, k 5 >k 6 ，f c For the higher fitness value of the two parents before the mutation operation, f max And f min Is the maximum and minimum fitness in the population.
8. The neural network pruning method according to claim 7, wherein in the step S3, the retraining process includes the steps of:
s31, inputting a data set into a neural network by applying the network structure of the offspring obtained last time, and calculating a loss function value L of the data set;
s32, updating the network parameter weight by using a gradient descent method, wherein the expression is as follows:
wherein W' is the weight after updating, W is the weight before updating, eta is the learning rate,
is the partial derivative of L to W;
s33, repeating the steps S31-S32 until the loss function L converges.
9. A neural network pruning system based on an improved adaptive genetic algorithm, applying the neural network pruning method according to any one of claims 1-8, characterized in that the neural network pruning system comprises:
the pre-training module is used for pre-training the convolutional neural network to obtain trained network parameter weights;
the pruning module is used for searching the optimal network structure of each layer by utilizing the improved self-adaptive genetic algorithm and pruning the redundant convolution kernel;
the retraining module is used for retraining the pruned neural network and recovering network precision;
and the output module is used for outputting the neural network with the optimal structure obtained after all the convolution layers are trimmed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211492198.0A CN116245162A (en) | 2022-11-25 | 2022-11-25 | Neural network pruning method and system based on improved adaptive genetic algorithm |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211492198.0A CN116245162A (en) | 2022-11-25 | 2022-11-25 | Neural network pruning method and system based on improved adaptive genetic algorithm |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116245162A true CN116245162A (en) | 2023-06-09 |
Family
ID=86630175
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211492198.0A Pending CN116245162A (en) | 2022-11-25 | 2022-11-25 | Neural network pruning method and system based on improved adaptive genetic algorithm |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116245162A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117668701A (en) * | 2024-01-30 | 2024-03-08 | 云南迅盛科技有限公司 | AI artificial intelligence machine learning system and method |
-
2022
- 2022-11-25 CN CN202211492198.0A patent/CN116245162A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117668701A (en) * | 2024-01-30 | 2024-03-08 | 云南迅盛科技有限公司 | AI artificial intelligence machine learning system and method |
| CN117668701B (en) * | 2024-01-30 | 2024-04-12 | 云南迅盛科技有限公司 | AI artificial intelligence machine learning system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Chen et al. | Chasing sparsity in vision transformers: An end-to-end exploration | |
| CN106919942B (en) | Accelerated compression method of deep convolution neural network for handwritten Chinese character recognition | |
| Acharya et al. | Online embedding compression for text classification using low rank matrix factorization | |
| Le et al. | A simple way to initialize recurrent networks of rectified linear units | |
| CN110175628A (en) | A kind of compression algorithm based on automatic search with the neural networks pruning of knowledge distillation | |
| CN111079899A (en) | Neural network model compression method, system, device and medium | |
| CN111340227A (en) | Method and device for compressing business prediction model through reinforcement learning model | |
| CN109523014B (en) | News comment automatic generation method and system based on generative confrontation network model | |
| CN113657421B (en) | Convolutional neural network compression method and device, and image classification method and device | |
| CN112215353B (en) | Channel pruning method based on variational structure optimization network | |
| WO2021042857A1 (en) | Processing method and processing apparatus for image segmentation model | |
| CN110826692B (en) | Automatic model compression method, device, equipment and storage medium | |
| CN110929798A (en) | Image classification method and medium based on structure optimization sparse convolution neural network | |
| CN113837376B (en) | Neural network pruning method based on dynamic coding convolution kernel fusion | |
| CN111626404A (en) | Deep network model compression training method based on generation of antagonistic neural network | |
| CN116245162A (en) | Neural network pruning method and system based on improved adaptive genetic algorithm | |
| CN111274359B (en) | Query recommendation method and system based on improved VHRED and reinforcement learning | |
| CN110717103A (en) | Improved collaborative filtering method based on stack noise reduction encoder | |
| CN113822419A (en) | Self-supervision graph representation learning operation method based on structural information | |
| CN114118369A (en) | Image classification convolution neural network design method based on group intelligent optimization | |
| CN114254108B (en) | Method, system and medium for generating Chinese text countermeasure sample | |
| Naik et al. | Survey on comparative study of pruning mechanism on mobilenetv3 model | |
| CN112860856B (en) | Intelligent problem solving method and system for arithmetic application problem | |
| Wang et al. | Efficient deep convolutional model compression with an active stepwise pruning approach | |
| CN116340763A (en) | Model training and classifying method based on decision tree |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |