JP7438517B2 - Neural network compression method, neural network compression device, computer program, and method for producing compressed neural network data - Google Patents

Description

TECHNICAL FIELD This disclosure relates to compression of neural networks.

As compression methods for neural networks such as deep neural networks (DNN), there are methods that perform pruning and reconstruction.

Pruning is performed in fully connected neural networks (FCNs) by deleting neurons (and the weights connected to the neurons), and in convolutional neural networks (CNNs) by deleting channels (non-patent (See Reference 1). Deletion of a channel is performed as a deletion of the entire weight belonging to the channel being deleted.

Reconstruction is done after pruning. In reconstruction, the weight parameters of the neural network are adjusted so that the output approaches the output before pruning. For example, in an FCN, weight parameters of connections between neurons are adjusted as reconstruction, and in CNN, weight parameters in a filter (kernel) are adjusted as reconstruction.

Yihui He, Xiangyu Zhang, Jian Sun, “Channel Pruning for Accelerating Very Deep Neural Networks,” Proc. of ICCV2017, 2017

The present inventors have discovered that in conventional compression methods that perform pruning and reconstruction, there is a problem in that pruning is not performed in such a way that the error after reconstruction (reconstruction error) is minimized. found out. This issue will be explained below. Note that in the following explanation, for the sake of simplicity, the explanation will be based on the FCN.

When pruning, it is necessary to select neurons to be deleted (pruning targets). The neuron to be deleted is selected by focusing on the error it gives to the layer next to the layer in which the neuron to be deleted exists. Specifically, the neuron whose deletion gives the least error to the next layer is selected as the neuron to be deleted. For example, when neuron A ₁ is deleted, the error given to the next layer is E ₁ , and when neuron A ₂ is deleted, the error given to the next layer is E ₂ , then error E ₁ becomes error E ₂ If it is smaller than , neuron A ₁ will be selected as the neuron to be deleted in preference to neuron A ₂ .

In the reconstruction after pruning, the weight parameters in the connections from the remaining neurons to the neurons in the next layer are adjusted. Weight adjustment is performed so that the error imparted to the next layer by neuron deletion is minimized. For example, if the error imparted to the next layer by deleting neuron A ₁ in pruning is E ₁ , the weights are adjusted in reconstruction so that the error E ₁ is as small as possible. The error E ₁ minimized by adjusting the weights is called the reconstruction error E _1r .

In the conventional compression method as described above, pruning is not performed so that the reconstruction error E _1r is minimized. For example, as described above, assume that when neuron A ₁ is deleted, the error given to the next layer is E ₁ and the reconstruction error is E _1r . It is also assumed that the error given to the next layer when neuron A ₂ is deleted is E ₂ and the reconstruction error is E _2r . In this case, even if the error E ₁ is smaller than the error E ₂ , the reconstruction error E _1r may be larger than the reconstruction error E _2r . That is, even if the error caused by deletion is minimal, there is no guarantee that the reconstruction error will be minimal.

Therefore, a solution to the above problems is desired. In the present disclosure, the above problem is solved by pruning so that the reconstruction error is minimized. Further details are described in the embodiments below.

FIG. 1 is a configuration diagram of a neural network compression device and a neural network utilization device. FIG. 2 is an explanatory diagram of the configuration of the neural network and the formulation of the amount of propagation Y. FIG. 3 is a flowchart of compression processing according to a comparative example. FIG. 4 is a flowchart of compression processing according to the embodiment. FIG. 5 is a Greedy Algorithm for compression processing according to the embodiment. FIG. 6 is a diagram showing the projection of x _j onto the subspace U. FIG. 7 shows a method for calculating the residual r _j . FIG. 8 shows a method for calculating the residual r _j . FIG. 9 shows how to calculate the reconstruction error when deleting another neuron. FIG. 10 shows how to calculate the reconstruction error when deleting another neuron. FIG. 11 shows the application of REAP to convolutional layers. FIG. 12 shows a method of calculating the projection residual r _j by applying Gran-Schmidt orthogonalization calculation. FIG. 13 is a diagram showing the experimental results.

<1. Overview of neural network compression method, neural network compression device, computer program, and method for producing compressed neural network data>

(1) The neural network compression method according to the embodiment includes the steps of pruning a selected pruning target and reconstructing the pruned neural network. The pruning target is selected such that the reconstruction error caused by pruning and reconstruction is minimized. This pruns so that the reconstruction error is minimized.

(2) The pruning may be pruning of neurons in a fully connected layer.

(3) The pruning may be pruning of channels in a convolutional layer.

(4) The reconstruction error is preferably calculated based on a projection residual when the behavior vector of the pruning target is projected onto a subspace spanned by behavior vectors of other pruning units other than the pruning target.

(5) Preferably, the projection residual is calculated based on a biorthogonal basis of a behavior vector to be pruned.

(6) The projection residual may be calculated by iteratively applying Gran-Schmidt orthogonalization calculation.

(7) Preferably, the reconstruction error is accelerated by parallel calculation or the like.

(8) The neural network compression device according to the embodiment is configured to perform pruning on a selected pruning target and reconfigure the pruned neural network. The neural network compression device is configured such that the pruning target is selected such that a reconstruction error caused by pruning and reconstruction is minimized.

(9) The computer program according to the embodiment is a computer program for causing a computer to function as a neural network compression device. The neural network compression device is configured to perform pruning on a selected pruning target and reconstruct the pruned neural network, and in the neural network compression device, the pruning target is generated by pruning and reconstruction. It is chosen so that the reconstruction error is minimized.

(10) A method for manufacturing compressed neural network data according to an embodiment includes a compression step in which pruning is performed on a selected pruning target and reconstruction of the pruned neural network; and outputting data of the neural network, and in the compression step, the pruning target is selected such that a reconstruction error caused by pruning and reconstruction is minimized.

<2. Examples of neural network compression method, neural network compression device, computer program, and method for producing compressed neural network data>

FIG. 1 shows a neural network compression device (hereinafter referred to as “compression device”) 10 and a neural network utilization device (hereinafter referred to as “utilization device”) 100 according to an embodiment. The compression device 10 according to the embodiment executes a compression process 21 for compressing the neural network N1 to reduce its size. The method implemented by performing the compression process 21 is also a method for manufacturing a compressed neural network or a method for manufacturing compressed neural network data.

A neural network is an artificial computational mechanism in which multiple artificial neurons (also called "nodes") are connected. The neural network is, for example, a deep neural network (DNN). The DNN may be, for example, a fully connected neural network (FCN) or a convolutional neural network (CNN). Hereinafter, the neural network N1 to be subjected to compression processing will be referred to as the "original neural network", and the compressed neural network N2 will be referred to as the "compressed neural network". Note that the compression device 10 according to the embodiment can also execute processing for machine learning (deep learning) of the original neural network N1. The compression device 10 compresses the trained original neural network N1.

The compression device 10 is configured by a computer having one or more processors 20 and a storage device 30. One or more processors 20 include, for example, a graphics processing unit (GPU). One or more processors 20 may further include a CPU. Large-scale parallel computing mechanisms such as GPUs are suitable for large-scale calculations to perform processing related to large-scale neural networks.

The storage device 30 stores a computer program 31 executed by the processor 20. The processor 20 performs compression processing 21 by executing a computer program 31. The compression process 21 includes pruning and reconstruction.

The storage device 30 can store data (compressed neural network data) N20 representing the compressed neural network N2 manufactured by the compression process 21. The compressed neural network data N20 is data consisting of various parameters (weights, connection relationships, etc.) expressing the compressed neural network N2. The compression device 10 can output the compressed neural network data N20 to a neural network engine or the like. The compressed neural network data N20 is read into the neural network engine, thereby causing the neural network engine to function as the compressed neural network N2.

The utilization device 100 has a neural network engine that reads the compressed neural network data N20 and functions as the compressed neural network N2. The neural network engine includes, for example, a processor 200 and a storage device 300. The processor 200 may be, for example, a low power consumption CPU in an embedded system. Since the compressed neural network data N20 is smaller in size than the data of the original neural network N1, it can be processed by a CPU with low power consumption.

Embedded systems are not general-purpose computer systems, but computer systems for specific purposes, such as household devices such as smartphones and home appliances, industrial devices such as industrial robots, and various medical devices. , computer systems in vehicles such as cars and drones, and other equipment. In embedded systems, a CPU with low power consumption is often used as a processor, but the compressed neural network data N20 is easy to execute because of its small data size.

The compression neural network N2 is used for, for example, image/audio conversion, segmentation, identification, and the like. More specifically, it can be used to extract necessary information from images of objects, such as counting the number of customers at stores, analyzing gender and age groups, counting vehicles, and analyzing vehicle types. The original neural network N1 is large-scale and has a high computational cost, making it difficult to execute in an embedded system, but the compressed neural network N2 is small-scale and therefore easy to execute in an embedded system. It is.

The compression process 21 will be explained below. In the following, for ease of understanding, we will first explain the compression process 21 based on a fully connected neural network (FCN), and then explain that the same compression process 21 can be applied to a convolutional neural network (CNN). explain.

FIG. 2 shows a layer l in a fully connected neural network (FCN), which is the original neural network N1, and a layer l+1, which is the next layer after the layer l. In FIG. 2, two layers (l layer, l+1 layer) are representatively shown. Each layer in the FCN is a fully connected layer in which artificial neurons (hereinafter simply referred to as "neurons") arranged in layers are connected between layers. The circles in each layer are neurons. The number of neurons included in layer l is c, and the number of neurons included in layer l+1 is C.

Equation (1) in FIG. 2 formulates the amount of propagation Y from layer l to the next layer l+1 when data is given to the neural network. Here, Y is an N×C matrix representing the internal activation of C neurons in layer l+1. In other words, Y is also the input to layer l+1, given from layer l.

When N pieces of data (for example, N pieces of image data) are applied to c neurons of layer l, each neuron of layer l produces N outputs. The output of the i-th neuron in layer l is denoted by x _i . x _i is an N-dimensional vector. x _i indicates the output (behavior) of the i-th neuron when N pieces of data are given to the i-th neuron. That is, x _i is also the behavior vector of the i-th neuron (neuron behavior vector). Note that the data given to the neural network may be data other than images, such as audio data. Data such as image data is fed to a neural network in order to understand the behavior of each neuron.

In FIG. 2, w _i is a C-dimensional weight vector consisting of weights (weighting coefficients) of connections from the i-th neuron of the l layer to the C neurons of the l+1 layer.

In this case, the amount of propagation Y to the next layer l+1 is determined by the output x _i of the neuron in the layer l and the weight vector w _i from the layer l to the next layer l+1, as shown in equation (1) in FIG. It is formulated as follows.

The purpose of the compression process 21 according to the embodiment is to reduce the number of neurons by a desired number without changing the above Y as much as possible. Even if the number of neurons is reduced, the performance of the original neural network N1 can be maintained as long as the change in Y is small. In other words, even if the neural network is compressed, it is possible to prevent a decrease in accuracy. Note that, as described above, in FCN, the neuron is the pruning unit, but in CNN, the channel is the pruning unit. Note that the pruning unit is the unit of deletion.

Prior to describing the compression process 21 according to the embodiment, the compression process 121 according to a comparative example will be described. FIG. 3 shows compression processing 121 according to a comparative example. The compression process 121 shown in FIG. 3 includes a pruning process 122 and a reconstruction process 123. In the comparative example, the pruning step 122 and the reconstruction step 123 are completely separated.

In the pruning step 122, a neuron (pruning target) to be deleted is selected from a plurality of neurons (a plurality of pruning units) included in a certain layer l, and the selected neuron is deleted. When a neuron to be deleted is selected from layer l, the neuron that has the smallest influence on the amount of propagation Y to the next layer l+1 is selected as the neuron to be deleted (pruning target). In this selection, neurons are selected using Lasso regression according to equation (2) in FIG. 3 (see Non-Patent Document 1. In Non-Patent Document 1, channels are selected). In the comparative example, weights are not adjusted when selecting neurons.

In the comparative example, the L ₁ norm of the importance vector β is added to the penalty term regarding the error in the internal activity at the next layer l+1. When the importance vector β that minimizes the L ₁ norm of the importance vector β is obtained, β with many zero elements can be obtained while suppressing the error in the activation degree of the next layer l+1, and the neurons to be deleted can be determined. In other words, as a result of the optimization in equation (2), an optimized neuron importance vector β ^* is obtained, but if the i-th element β ^* _i of that vector is 0, the i-th neuron is unnecessary. Yes, and selected as the neuron to be deleted.

In the comparative example, the number of neurons removed is controlled by fine-tuning the hyperparameter λ. Increasing λ increases the number of neurons deleted, and decreasing λ decreases the number of neurons deleted. In the comparative example, the number of neurons to be deleted is controlled by λ, so it is difficult to control the number of neurons to be deleted.

In the reconstruction step 123, the weights are adjusted (optimized) in layer l so that the Y given by neurons remaining after pruning to the next layer l+1 approaches Y before pruning (original Y). The weight adjustment is performed according to equation (3) in FIG. Equation (3) calculates a weight vector that minimizes the reconstruction error. The reconstruction error here is based on the difference between Y before pruning and Y when the weights are adjusted after pruning.

In the comparative example, the pruning target is selected so as to minimize the error before reconstruction, and is not selected so as to minimize the error after reconstruction. In other words, in the comparative example, the selection of the pruning target is performed based on the error before reconstruction, and the reconstruction is performed based on the error after reconstruction, and pruning and reconstruction are different. It is done according to standards. Further, in the comparative example, Lasso is used, and manual fine adjustment of λ is required to control the number of neurons to be deleted. In other words, in the comparative example, it is difficult to control the number of neurons to be deleted.

FIG. 4 shows compression processing 21 according to the embodiment. Hereinafter, the compression process 21 according to the embodiment will be referred to as REAP (Reconstruction Error Aware Pruning).

In the comparative example, pruning and reconstruction were performed using different standards, whereas in REAP, pruning and reconstruction are performed using the same standard. That is, in REAP, the data is pruned and reconstructed so that the reconstruction error is minimized. In REAP, neurons to be deleted are determined according to equation (4) in FIG. Note that Z ^* in Equation (4) indicates the neuron remaining after deletion in layer l. In equation (4), the weight vector w _i is optimized before fixing Z' to Z ^* . Therefore, according to equation (4), a set of neurons that minimizes the reconstruction error of the amount of propagation Y to the next layer l+1 is found.

In REAP, neurons to be pruned are selected so that the reconstruction error is minimized, so REAP is advantageous compared to the comparative example in which the reconstruction error is not necessarily minimized.

Equation (4) is a combinatorial optimization problem and is solved by a greedy algorithm. FIG. 5 shows an algorithm for solving equation (4). First, in step S1, the j-th neuron in layer l is erased. In step S2, Y is reconstructed using only the remaining neurons in layer l, and an error (reconstruction error) is calculated. The reconstruction error is found by calculating the cost function P(Z') shown as equation (5) in FIG. Once the reconstruction error is determined, the deleted j-th neuron is restored. In the iterative loop indicated by step S3, steps S1 and S2 are repeated for all j (j∈Z), and the neuron with the smallest value of P(Z') is selected as the pruning target, and the final will be deleted.

One neuron is deleted through the iterative loop of step S3. In the repeat loop shown in step S4, the repeat loop in step S3 is executed again using only the remaining neurons. When the iterative loop of step 3 is executed again, another neuron is deleted.

How many neurons to delete from layer l depends on how many times the iterative loop of step S4 is executed. Therefore, if it is desired to delete a desired number D of neurons, it is sufficient to execute the repeat loop of step S4 D times. Therefore, in REAP, it is easy to control the number of neurons to be deleted.

In REAP, the amount of calculation increases compared to the comparative example. That is, in REAP, since the method of least squares is applied when determining the reconstruction error, it is necessary to solve simultaneous equations. There are as many simultaneous equations as there are neurons in the layer. For example, if the number of neurons in layer l is c and the number of neurons in the next layer l+1 is C, the number of weight parameters is c×C. When one neuron is deleted, the size of the coefficient matrix becomes (c-1)C×(c-1)C. Therefore, the time complexity for solving one simultaneous equation is O(c ³ C ³ ). Since this simultaneous equation needs to be solved c times, the time complexity is O(c ⁴ C ³ ).

Here, the least squares problem that we are originally trying to solve is that when the behavior vector x _j representing the behavior of a certain neuron disappears, the linear sum of the behavior vectors x _i (i∈Z') of the remaining neuron set Z' is , the problem is to approximate the original amount of propagation Y to the next layer l+1. The error caused by this approximation is due to the error r _j when x _j is expressed as a linear sum of x _i (i∈Z'), as shown in FIG. Therefore, if this error r _j is minimized, the error in the activation level (propagation Y) at the next layer l+1 can be reduced by simply multiplying the vector indicating the error r _j by the weight vector w _i ^T for the next layer l+1. Can calculate.

That is, in order to calculate the reconstruction error caused by deletion of the j-th neuron, it is necessary to calculate the cost function shown in equation (6-1) in FIG. As described later, the cost function of equation (6-1) is expressed as equation (6-2). Therefore, if it is possible to calculate the projection residual r _j when x _j is projected onto the subspace spanned by x _i (i∈Z') without using a huge coefficient matrix, the reconstruction error can be efficiently reduced. can be calculated. As an example, the residual r _j is calculated as in equation (6-3).

An example of how to obtain the residual r _j is explained in detail in FIGS. 7 and 8. The calculation method shown in FIGS. 7 and 8 utilizes the fact that the residual r _j is linearly dependent on the biorthogonal basis of x _j . That is, the residual r _j is the projection of x _{j onto the biorthogonal basis of x j .} The residual r _j is calculated based on a biorthogonal basis of x _j . According to the calculation method shown in FIGS. 7 and 8, the residual r _j can be efficiently calculated without using the coefficient matrix of the simultaneous equations. 9 and 10 also illustrate a method for efficiently calculating a reconstruction error for deleting another neuron after deleting one neuron.

In order to increase speed, calculation of reconstruction errors is preferably performed in parallel. By calculating the reconstruction error in parallel, the reconstruction error can be obtained at high speed. However, parallelization increases the amount of memory required to store the coefficient matrix when solving the least squares method.

Therefore, in order to efficiently calculate the reconstruction error, it is preferable to reduce the amount of spatial calculation equivalent to the amount of memory consumed. Here, the space calculation amount (memory consumption) is O(c ² C ² ) per one simultaneous equation, and when c calculations are performed simultaneously in parallel, it becomes O(c ³ C ² ). In order to reduce the amount of spatial calculation, it is preferable to perform the least squares calculation without using the coefficient matrix of the simultaneous equations.

FIG. 12 shows an example of finding the residual r _j by parallel calculation. In FIG. 12, the residual r _j can be efficiently calculated by repeatedly applying Gram-Schmit orthogonalization calculation. In the calculation method shown in FIG. 11, the residual r _j can be efficiently calculated in parallel without using the coefficient matrix of the simultaneous equations. When the number of neurons is very large (4096 or more, as a guide, depending on the execution environment), the Gran-Schmidt solution method is faster.

FIG. 11 illustrates that REAP can be applied to pruning and reconstruction in convolutional layers. As shown in equation (19) in FIG. 11, the sliding window operation in the convolution layer is represented by a sum of matrix multiplications. Since Equation (19) has the same format as Equation (1) for the fully connected layer, it can be seen that REAP can be applied to the convolutional layer as well as to the fully connected layer.

FIG. 13 shows the results of an experiment in which neural network compression was performed using REAP and a comparative example. In the experiment, VGG16 trained with the ImageNet dataset was used as the original neural network N1. Image recognition accuracy (accuracy rate) was determined for each case when REAP was applied as the compression process 21 for the original neural network N1 and when the comparative example was applied.

The horizontal axis in FIG. 13 shows FLOPs (floating point operations), and the vertical axis shows the correct answer rate. The smaller FLOPs indicates that the number of operations of the compression neural network N2 is smaller and the compression ratio is larger. As shown in Figure 13, in the comparative example, when the compression rate is increased (increasing the number of neurons to be deleted), the accuracy rate drops to about 0.7 (70%), whereas in REAP, the compression rate Even when increasing the number, the accuracy rate only decreased to about 0.8 (80%). Therefore, it can be seen that REAP is better able to suppress the decrease in accuracy due to compression.

<3. Additional notes>
The present invention is not limited to the above embodiments, and various modifications are possible.

10: Neural network compression device 20: Processor 21: Compression processing 30: Storage device 31: Computer program 100: Neural network utilization device 121: Compression processing 122: Pruning process 123: Reconstruction process 200: Processor 300: Storage device N1: Original Neural network N2: Compressed neural network N20: Compressed neural network data

Claims (10)

to the computer,
A step of performing pruning on the selected pruning target and reconfiguration to adjust weight parameters of the pruned neural network,
In the step, the pruning target calculates a reconstruction error caused by the reconstruction that adjusts the weight parameters of the pruned neural network based on the adjusted weight parameters, and calculates the reconstruction error so that the reconstruction error is minimized. selected ,
The pruning target selected such that the reconstruction error is minimized is determined by pruning each of the pruning targets that can be selected as the pruning target in the neural network before the selected pruning target is pruned. The reconstruction error is minimized in
How to compress neural networks. The neural network compression method according to claim 1, wherein the pruning is pruning of neurons in a fully connected layer. The neural network compression method according to claim 1, wherein the pruning is channel pruning in a convolutional layer. The reconstruction error is calculated based on a projection residual when the behavior vector of the pruning target is projected onto a subspace spanned by behavior vectors of other pruning units other than the pruning target. The neural network compression method according to item 1. The neural network compression method according to claim 4, wherein the projection residual is calculated based on a biorthogonal basis of a behavior vector to be pruned. The neural network compression method according to claim 4, wherein the projection residual is calculated by iteratively applying Gran-Schmidt orthogonalization calculation. The neural network compression method according to any one of claims 1 to 6, wherein the reconstruction error is calculated in parallel. A neural network compression device that performs pruning on a selected pruning target and reconfiguration that adjusts weight parameters of the pruned neural network,
The pruning target is selected such that the reconstruction error caused by the reconstruction that adjusts the weight parameters of the pruned neural network is calculated based on the adjusted weight parameters, and the reconstruction error is minimized. configured ,
The pruning target selected such that the reconstruction error is minimized is determined by pruning each of the pruning targets that can be selected as the pruning target in the neural network before the selected pruning target is pruned. The reconstruction error is minimized in
Neural network compression device. A computer program for causing a computer to function as a neural network compression device,
The neural network compression device is configured to perform pruning on the selected pruning target and reconfiguration to adjust weight parameters of the pruned neural network,
In the neural network compression device, the pruning target calculates a reconstruction error caused by reconstruction that adjusts the weight parameters of the pruned neural network based on the adjusted weight parameters, and calculates the reconstruction error so that the reconstruction error is minimized. selected to be ,
The pruning target is selected such that the reconstruction error is minimized when each of the pruning targets that can be selected as the pruning target is pruned in the neural network before the selected pruning target is pruned. The reconstruction error is minimized in
computer program. to the computer,
a compression step of performing pruning on the selected pruning target and reconfiguration to adjust weight parameters of the pruned neural network;
outputting the neural network data compressed by the compression step;
has
In the compression step, the pruning target calculates the reconstruction error caused by the reconstruction that adjusts the weight parameters of the pruned neural network based on the adjusted weight parameters, and calculates the reconstruction error so that the reconstruction error is minimized. selected ,
The pruning target selected such that the reconstruction error is minimized is determined by pruning each of the pruning targets that can be selected as the pruning target in the neural network before the selected pruning target is pruned. The reconstruction error is minimized in
A method for producing compressed neural network data.

Images