CN113516240A - Neural network structured progressive pruning method and system - Google Patents
Neural network structured progressive pruning method and system Download PDFInfo
- Publication number
- CN113516240A CN113516240A CN202110697462.3A CN202110697462A CN113516240A CN 113516240 A CN113516240 A CN 113516240A CN 202110697462 A CN202110697462 A CN 202110697462A CN 113516240 A CN113516240 A CN 113516240A
- Authority
- CN
- China
- Prior art keywords
- neural network
- pruning
- layer
- training
- progressive
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to the field of computer vision, in particular to a neural network structured progressive pruning method and a system, wherein the method comprises the following steps: step S1: setting the cutting rate and the pruning standard of each layer of the neural network and the training cycle number of the neural network; step S2: inputting pictures to train a neural network, gradually increasing the cutting rate of each layer from zero to a set cutting rate in a certain training period, and determining and setting the redundant information of each layer to be 0 according to the pruning standard; step S3: after the preset cutting rate is reached, removing redundant information in the neural network and reconstructing an original network layer; step S4: and after the neural network is reconstructed, continuing training until a set neural network training period is reached. The method is simple to operate, has few steps, can achieve the aim of pruning in the normal neural network training process, does not need the fine adjustment process after pruning, can greatly reduce the processing time, and can obtain higher performance while achieving higher pruning rate compared with the prior art.
Description
Technical Field
The invention relates to the field of computer vision, in particular to a neural network structured progressive pruning method and system.
Background
At present, the neural network obtains good performance in the field of computer vision, particularly image classification, target detection and the like. Even exceeding human recognition capabilities. However, although high performance is achieved in neural networks, these neural networks usually have high floating point operations and storage, such as inputting a picture of size 224 × 224, ResNet-50 requires 4.1B floating point operations and a parameter of 25.6 MB. The large amount of computation and storage requires the operating platform to have higher computation resources and more storage resources. Therefore, these good performance neural networks cannot be deployed to resource-limited platforms such as mobile phones and embedded platforms.
In fact, there is a lot of redundant information in the neural network, and the redundant information, especially the structured channels and filters, are cut out, thereby reducing the amount of computation and memory during operation. The existing pruning method of the neural network needs to train a complete neural network, set a fixed pruning rate for each layer of the neural network after the training is finished, judge redundant information according to a certain standard and prune the information, and finally fine-tune the pruned neural network. This pruning method is cumbersome and requires a long processing time.
Disclosure of Invention
In order to solve the technical problems in the prior art, the invention provides a neural network structured progressive pruning method and system, and the specific technical scheme is as follows:
a neural network structured progressive pruning method comprises the following steps:
step S1: setting the cutting rate and the pruning standard of each layer of the neural network and the training cycle number of the neural network;
step S2: inputting pictures to train the neural network, gradually increasing the cutting rate of each layer of the neural network from zero to the cutting rate set in the step S1 in a certain training period, selecting redundant information of each layer of the neural network according to the pruning standard determined in the step S1, and temporarily setting the value of the redundant information as 0;
step S3: after the set clipping rate is reached, removing the redundant information with the value of 0 processed in the step S2 from each layer of the neural network, and reconstructing the original neural network layer;
step S4: and after the neural network is reconstructed, continuing training until the set neural network training cycle number is reached, and obtaining the pruned lightweight neural network model after training is finished.
Preferably, the cutting rate of each layer of the neural network in step S1 is specifically: each network layer sets different positive values, or the network layers needing pruning in the network set the same positive values.
Preferably, the pruning criteria in step S1 includes L p Norm, random selection, and inverse gradient-based computation.
Preferably, the number of neural network training cycles in step S1 is one cycle of training all pictures in the data set once, or training a certain number of pictures sampled from the data set once.
Preferably, the training period in step S2 is less than or equal to the number of neural network training periods set in step S1.
Preferably, the cutting rate of each layer of the neural network in the step S2 gradually increases from zero to the cutting rate set in the step S1, which can be formally expressed as
Wherein the content of the first and second substances,
representing the clipping rate of a layer at the current training period,
indicating the clipping rate set in step S1,
indicating the set initial cropping rate of each layer,
indicating the current number of training cycles,
which represents the initial number of training cycles,
the cutting interval is represented by the number of the cutting lines,
the clipping rate attenuation rate is set to be,
any positive integer value.
Preferably, in step S2, the redundant information value is temporarily set to 0, specifically, the redundant channel or the redundant filter value in the neural network is set to 0.
Preferably, in step S3, specifically, when the redundant information is structured filter information, the filter with a value of 0 in the weight of the current network layer is removed, and meanwhile, the channel of the current network layer corresponding to the removed filter is also removed, and the convolution kernel of the filter corresponding to the current network layer removed channel in the weight of the latter network layer is also removed;
and when the redundant information is the structured channel information, removing the channel with the value of 0 in the current network layer, simultaneously removing the filter corresponding to the removal channel in the current network layer, and removing the convolution kernel of the filter corresponding to the removal channel in the current network layer in the weight of the next network layer.
Preferably, in the weight-reduced neural network model after pruning in step S4, if a batch normalization operation is used, parameters of the operation are combined with convolutional layer parameters of a previous network layer.
A neural network structured progressive pruning system is characterized by comprising a parameter setting module, a progressive pruning module, a network reconstruction module and a continuous training module, wherein the parameter setting module is used for setting the pruning rate, the pruning standard and the neural network training cycle number of each layer of a neural network; the progressive pruning module is used for inputting pictures to train the neural network, gradually increasing the pruning rate of each layer of the neural network from zero to the pruning rate set by the parameter setting module in a certain training period, selecting redundant information of each layer of the neural network according to the pruning standard determined by the parameter setting module, and temporarily setting the value of the redundant information to be 0; the network reconstruction module is used for removing the redundant information with the value of 0 processed by the progressive pruning module from each layer of the neural network after the set cutting rate is reached and reconstructing an original neural network layer; and the continuous training module is used for continuously training the reconstructed model, and after the neural network is reconstructed, the training is continuously carried out until the set neural network training period number is reached, so that the light weight neural network model after pruning can be obtained after the training is finished.
Compared with the prior art, the invention has the following advantages and beneficial effects:
1. the structured progressive pruning method is simple to operate and few in steps, and can achieve the purpose of pruning in the normal neural network training process;
2. the invention does not need the fine adjustment process after pruning, thereby greatly reducing the processing time;
3. compared with the prior art, the method can achieve higher performance while achieving higher cutting rate.
Drawings
FIG. 1 is a flow chart of a neural network structured progressive pruning method in an embodiment of the present invention;
FIG. 2 is a diagram of a structured filter information with redundancy removed according to an embodiment of the present invention;
FIG. 3 is a diagram of removing redundant structured channel information in an embodiment of the present invention;
FIG. 4 is a schematic diagram of a neural network structured progressive pruning system in an embodiment of the present invention;
in the figure, 21 is a parameter setting module, 22 is a progressive pruning module, 23 is a network reconstruction module, and 24 is a continuous training module.
Detailed Description
In order to make the objects, technical solutions and technical effects of the present invention more apparent, the present invention is further described in detail below with reference to the accompanying drawings.
The first embodiment is as follows:
as shown in fig. 1, a neural network structured progressive pruning method in this embodiment includes the following steps:
step S1: setting the cutting rate and the pruning standard of each layer of the neural network and the training cycle number of the neural network;
step S2: inputting pictures to train the neural network, gradually increasing the cutting rate of each layer of the neural network from zero to the cutting rate set in the step S1 in a certain training period, selecting redundant information of each layer of the neural network according to the pruning standard determined in the step S1, and temporarily setting the value of the redundant information as 0;
step S3: after the set clipping rate is reached, removing the redundant information with the value of 0 processed in the step S2 from each layer of the neural network, and reconstructing the original neural network layer;
step S4: and after the neural network is reconstructed, continuing training until the set neural network training cycle number is reached, and obtaining the pruned lightweight neural network model after training is finished.
As a preferred embodiment, the pruning rate of each layer of the neural network in step S1 may be set to a different positive value for each network layer, or may be set to the same positive value for the network layer that needs pruning in the network.
As a preferred embodiment, the pruning criteria in step S1 includes L p Norm, random selection, or computation based on inverse gradients.
In a preferred embodiment, the neural network training cycle in step S1 is performed once for all pictures in the data set or once for a certain number of pictures sampled from the data set.
The step S2 includes both the conventional neural network training method and the step of confirming the redundant structured information to be removed during training.
In a preferred embodiment, the training period in step S2 is less than or equal to the number of neural network training periods set in step S1.
As a preferred embodiment, the cutting rate of each layer of the neural network in the step S2 gradually increases from zero to the cutting rate set in the step S1, which can be formally expressed as
Wherein the content of the first and second substances,
representing the clipping rate of a layer at the current training period,
indicating the clipping rate set in step S1,
indicating the set initial cropping rate of each layer,
indicating the current number of training cycles,
which represents the initial number of training cycles,
the cutting interval is represented by the number of the cutting lines,
the clipping rate attenuation rate is set to be,
any positive integer value.
In a preferred embodiment, the redundant structured information value in step S2 is set to 0, in particular, the redundant channels or the redundant filter values in the neural network are set to 0.
The step S3 is to perform a real pruning operation, i.e., to remove the redundant structured information determined in the step S2 from the corresponding network layer of the neural network.
As a preferred embodiment, in step S3, specifically when the redundant information is the structured filter information, the filter with a value of 0 in the weight of the current network layer is removed, and the channel of the current network layer corresponding to the removed filter also needs to be removed, and the convolution kernel of the filter corresponding to the removed channel of the current network layer in the weight of the latter network layer is also removed.
As shown in figure 2 of the drawings, in which,
is the weight of convolutional layer conv1, where FiIndicating the need for redundant filters, I1Input characteristic diagram, O, representing convolutional layer conv11Represents the output characteristic diagram of convolutional layer conv1, where CiIs represented by FiThe corresponding output channel, convolutional layer conv2, represents the layer one after convolutional layer conv1, W2Is the weight of convolutional layer conv2, where F2 iIs represented by CiCorresponding convolution kernel, then F in the reconstruction processi, CiAnd F2 iAre removed from the respective positions.
Fig. 2 is a schematic diagram of only one case, and actually, there may be a plurality of redundant filters in each layer of neural network.
As a preferred embodiment, in step S3, specifically, when the redundant information is required to be structured channel information, the channel with the value of 0 in the current network layer needs to be removed, and the filter corresponding to the removed channel in the current network layer also needs to be removed, and the convolution kernel of the filter corresponding to the removed channel in the current network layer in the latter layer of weights is also removed.
As shown in FIG. 3, O1Represents the output characteristic diagram of convolutional layer conv1, where CiIndicating redundant channels, W1Is the weight of convolutional layer conv1, where FiIs represented by CiThe corresponding filter, convolutional layer conv2, represents the layer one after convolutional layer conv1, W2Is the weight of convolutional layer conv2, where F2 iIs represented by CiCorresponding convolution kernel, then F in the reconstruction processi, CiAnd F2 iAre removed from the respective positions.
Fig. 3 is a schematic diagram of only one case, and actually, there may be a plurality of redundant channels in each layer of neural network.
The step S4: optionally, in obtaining the pruned lightweight neural network model, if a batch normalization operation is used, parameters of the operation are combined with parameters of the convolutional layer of the previous layer, thereby further reducing the parameters and the calculation amount of the network.
Example two
The embodiment of the invention also provides a neural network structured progressive pruning system, as shown in fig. 4, the system comprises a parameter setting module 21, a progressive pruning module 22, a network reconstruction module 23 and a continuous training module 24.
The parameter setting module 21 is configured to set a pruning rate and a pruning standard of each layer of the neural network, and a training cycle number of the neural network; the progressive pruning module 22 is used for inputting pictures to train the neural network, gradually increasing the pruning rate of each layer of the neural network from zero to the pruning rate set by the parameter setting module 21 in a certain training period, selecting redundant information of each layer of the neural network according to the pruning standard determined by the parameter setting module 21, and temporarily setting the value of the redundant information to be 0; the network reconstruction module 23 is configured to remove the redundant information with a value of 0 processed by the gradual pruning module 22 from each layer of the neural network after the set clipping rate is reached, and reconstruct an original network layer; and the continuous training module 24 is used for continuous training of the reconstructed model. And after the neural network is reconstructed, continuing training until the set neural network training cycle number is reached, and obtaining the pruned lightweight neural network model after training is finished.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the present invention in any way. Although the foregoing has described the practice of the present invention in detail, it will be apparent to those skilled in the art that modifications may be made to the practice of the invention as described in the foregoing examples, or that certain features may be substituted in the practice of the invention. All changes, equivalents and modifications which come within the spirit and scope of the invention are desired to be protected.
Claims (10)
1. A neural network structured progressive pruning method is characterized by comprising the following steps:
step S1: setting the cutting rate and the pruning standard of each layer of the neural network and the training cycle number of the neural network;
step S2: inputting pictures to train the neural network, gradually increasing the cutting rate of each layer of the neural network from zero to the cutting rate set in the step S1 in a certain training period, selecting redundant information of each layer of the neural network according to the pruning standard determined in the step S1, and temporarily setting the value of the redundant information as 0;
step S3: after the set clipping rate is reached, removing the redundant information with the value of 0 processed in the step S2 from each layer of the neural network, and reconstructing the original neural network layer;
step S4: and after the neural network is reconstructed, continuing training until the set neural network training cycle number is reached, and obtaining the pruned lightweight neural network model after training is finished.
2. The neural network structured progressive pruning method according to claim 1, wherein the clipping rate of each layer of the neural network in the step S1 is specifically: each network layer sets different positive values, or the network layers needing pruning in the network set the same positive values.
3. The method of claim 1, wherein the method comprises a step of performing progressive pruning in a neural network structureCharacterized in that the pruning criterion in the step S1 comprises L p Norm, random selection, and inverse gradient-based computation.
4. The neural network structured progressive pruning method of claim 1, wherein the number of neural network training cycles in step S1 is one cycle of training all pictures in the data set once, or training pictures sampled quantitatively from the data set once.
5. The method of claim 1, wherein the training period in step S2 is less than or equal to the number of training periods of the neural network set in step S1.
6. The neural network structured progressive pruning method of claim 1, wherein the gradual increase of the per-layer clipping rate of the neural network in the step S2 from zero to the clipping rate set in the step S1 is formally expressed as
Wherein the content of the first and second substances,
representing the clipping rate of a layer at the current training period,
indicating the clipping rate set in step S1,
indicating the set initial cropping rate of each layer,
representing current trainingThe number of cycles,
which represents the initial number of training cycles,
the cutting interval is represented by the number of the cutting lines,
the clipping rate attenuation rate is set to be,
any positive integer value.
7. The neural network structured progressive pruning method of claim 1, wherein the redundant information value is temporarily set to 0 in step S2, specifically, the redundant channel or the redundant filter value in the neural network is set to 0.
8. The neural network structured progressive pruning method according to claim 1, wherein in step S3, specifically, when the redundant information is filter information, the filter with a value of 0 in the weight of the current network layer is removed, and the channel of the current network layer corresponding to the removed filter is also removed, and the convolution kernel of the filter corresponding to the removed channel of the current network layer in the weight of the next network layer is also removed;
and when the redundant information is channel information, removing the channel with the value of 0 in the current network layer, simultaneously removing the filter corresponding to the removal channel in the current network layer, and removing the convolution kernel of the filter corresponding to the removal channel in the current network layer in the weight of the next network layer.
9. The neural network structured progressive pruning method of claim 1, wherein in the lightweight neural network model after pruning obtained in step S4, if a batch normalization operation is used, parameters of the operation are merged with convolutional layer parameters of a previous network layer.
10. The neural network structured progressive pruning system is characterized by comprising a parameter setting module (21), a progressive pruning module (22), a network reconstruction module (23) and a continuous training module (24), wherein the parameter setting module (21) is used for setting the pruning rate, the pruning standard and the neural network training cycle number of each layer of a neural network; the progressive pruning module (22) is used for inputting pictures to train the neural network, gradually increasing the pruning rate of each layer of the neural network from zero to the pruning rate set by the parameter setting module (21) in a certain training period, selecting redundant information of each layer of the neural network according to the pruning standard determined by the parameter setting module (21), and temporarily setting the value of the redundant information to be 0; the network reconstruction module (23) is used for removing the redundant information with the value of 0 processed by the progressive pruning module (22) from each layer of the neural network after the set clipping rate is reached and reconstructing an original neural network layer; and the continuous training module (24) is used for continuously training the reconstructed model, and after the neural network is reconstructed, the training is continuously carried out until the set neural network training cycle number is reached, and the light weight neural network model after pruning can be obtained after the training is finished.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110697462.3A CN113516240A (en) | 2021-06-23 | 2021-06-23 | Neural network structured progressive pruning method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110697462.3A CN113516240A (en) | 2021-06-23 | 2021-06-23 | Neural network structured progressive pruning method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113516240A true CN113516240A (en) | 2021-10-19 |
Family
ID=78065852
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110697462.3A Pending CN113516240A (en) | 2021-06-23 | 2021-06-23 | Neural network structured progressive pruning method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113516240A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117557857A (en) * | 2023-11-23 | 2024-02-13 | 哈尔滨工业大学 | Detection network light weight method combining progressive guided distillation and structural reconstruction |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110263841A (en) * | 2019-06-14 | 2019-09-20 | 南京信息工程大学 | A kind of dynamic, structured network pruning method based on filter attention mechanism and BN layers of zoom factor |
| CN111461322A (en) * | 2020-03-13 | 2020-07-28 | 中国科学院计算技术研究所 | Deep neural network model compression method |
-
2021
- 2021-06-23 CN CN202110697462.3A patent/CN113516240A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110263841A (en) * | 2019-06-14 | 2019-09-20 | 南京信息工程大学 | A kind of dynamic, structured network pruning method based on filter attention mechanism and BN layers of zoom factor |
| CN111461322A (en) * | 2020-03-13 | 2020-07-28 | 中国科学院计算技术研究所 | Deep neural network model compression method |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117557857A (en) * | 2023-11-23 | 2024-02-13 | 哈尔滨工业大学 | Detection network light weight method combining progressive guided distillation and structural reconstruction |
| CN117557857B (en) * | 2023-11-23 | 2024-06-04 | 哈尔滨工业大学 | Detection network light weight method combining progressive guided distillation and structural reconstruction |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108596841B (en) | Method for realizing image super-resolution and deblurring in parallel | |
| CN111950723B (en) | Neural network model training method, image processing method, device and terminal equipment | |
| CN110555847B (en) | Image processing method and device based on convolutional neural network | |
| CN111709883B (en) | Image detection method, device and equipment | |
| CN113450290B (en) | Low-illumination image enhancement method and system based on image inpainting technology | |
| CN109858613B (en) | Compression method and system of deep neural network and terminal equipment | |
| CN111951167B (en) | Super-resolution image reconstruction method, super-resolution image reconstruction device, computer equipment and storage medium | |
| CN112862681A (en) | Super-resolution method, device, terminal equipment and storage medium | |
| CN110189260B (en) | Image noise reduction method based on multi-scale parallel gated neural network | |
| CN113450288A (en) | Single image rain removing method and system based on deep convolutional neural network and storage medium | |
| CN113673675A (en) | Model training method and device, computer equipment and storage medium | |
| CN112801906A (en) | Cyclic iterative image denoising method based on cyclic neural network | |
| CN112686869A (en) | Cloth flaw detection method and device | |
| CN110619391B (en) | Detection model compression method and device and computer readable storage medium | |
| CN113516240A (en) | Neural network structured progressive pruning method and system | |
| CN110599495B (en) | Image segmentation method based on semantic information mining | |
| CN112990299B (en) | Depth map acquisition method based on multi-scale features, electronic equipment and storage medium | |
| CN116029905A (en) | Face super-resolution reconstruction method and system based on progressive difference complementation | |
| CN116468947A (en) | Cutter image recognition method, cutter image recognition device, computer equipment and storage medium | |
| CN111402140A (en) | Single image super-resolution reconstruction system and method | |
| CN112529064B (en) | Efficient real-time semantic segmentation method | |
| CN115937121A (en) | Non-reference image quality evaluation method and system based on multi-dimensional feature fusion | |
| CN114612316A (en) | Method and device for removing rain from nuclear prediction network image | |
| CN115423697A (en) | Image restoration method, terminal and computer storage medium | |
| CN113436245B (en) | Image processing method, model training method, related device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |