JP2022054660A - Neural network weight saving device, neural network weight saving method and program - Google Patents

Abstract

To improve the accuracy of a neural network while suppressing a decrease in its processing efficiency.SOLUTION: Provided is a neural network weight saving device for: introducing a first quantization function that includes a trainable first coefficient, a second quantization function that includes a trainable second coefficient and a channel attenuation function that includes a trainable third coefficient in channel units into the correction target layer of a first neural network and generating a second neural network; training the weight parameter, the first coefficient, the second coefficient and the third coefficient of the first neural network by learning based on the second neural network; re-training the weight parameter by re-learning based on the second neural network after learning; and outputting a third neural network in which the channel attenuation function and the weight parameter of redundant channel corresponding to the third coefficient in the correction target layer after training are deleted from the second neural network after re-learning.SELECTED DRAWING: Figure 1

Description

The present invention relates to a neural network weight reduction device, a neural network weight reduction method and a program.

In recent years, neural networks have been used in various fields. For example, a general neural network model for the purpose of object recognition or object position detection is known. In such a general neural network model, features and weight parameters represented by 16-32 bit floating point numbers are used for operations in the convolutional layer or the fully connected layer, respectively.

On the other hand, for example, in a binarized neural network, which is one of the forms of a quantized neural network, the feature amount and the weight parameter used for the operation in the convolution layer or the fully connected layer are represented by (-1 and 1), respectively. Can be represented by 1 bit (ie, binary). This can replace floating point operations in the convolution layer or fully connected layer with bit operations. When bit operation is used, lower power consumption and higher speed operation processing can be performed than when floating point operation is used, and memory usage can be reduced. Therefore, FPGA (Field Programmable Gate Array) can be used. ) Or mobile terminals, etc., it is known that efficient processing of deep learning models is possible even on devices with limited computing resources.

For example, a method for constructing a binarized neural network is disclosed (see, for example, Non-Patent Document 1). More specifically, in such Non-Patent Document 1, in all convolutional layers or fully connected layers, a weight parameter represented by a floating point number is converted into a binary value represented by -1 or 1 by a sign function and input. A method of converting a feature amount to be a binary value represented by -1 or 1 by a sign function is disclosed.

Further, a method for constructing a quantized neural network is disclosed (see, for example, Non-Patent Document 2). More specifically, in Non-Patent Document 2, a plurality of combinations of weight parameters having different quantization bit numbers (bit precision) and input features are defined in advance for each layer, and the convolutional neural network is used for each layer. Discloses a method of learning to select one of the most suitable combinations from a plurality of combinations.

Japanese Unexamined Patent Publication No. 2019-212206

Itay Hubara, 4 others, "Binarized Neural Networks", [online], Neural Information Processing Systems (2016), [Search on September 16, 2nd year of Reiwa], Internet <http://papers.nips.cc/paper/ 6573-binarized-neural-networks ＞ Bichen Wu, 5 others, "MixedPrecision Quantization of ConvNets via Differentiable Neural Architecture Search", [online], 2018, [Search on September 16, 2nd year of Reiwa], Internet <https://arxiv.org/abs/1812.00090 ＞ Benoit Jacob, 7 others, "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference", [online], 2017, [Search on September 16, 2nd year of Reiwa], Internet <https: // arxiv. org / abs / 1712.05877 ＞

However, according to the method disclosed in Non-Patent Document 1, an error (quantization error) caused by binarization of data (for example, features and weight parameters) input to the convolutional layer or the fully connected layer is obtained. ) Is large and the accuracy of the quantized neural network model may be significantly deteriorated. Further, according to the method disclosed in Non-Patent Document 1, since the number of channels is fixed, the quantized neural network may include redundant channels.

Further, according to the method disclosed in Non-Patent Document 2, the number of quantization bits can be estimated for each layer, while a plurality of combinations of weight parameters and input features are used in the learning process. It is necessary to load the convolution layer corresponding to the above into the memory and repeat the forward propagation and back propagation to all the convolution layers many times. Therefore, according to the method disclosed in Non-Patent Document 2, it takes a lot of time to converge, and it may be difficult to define all the combinations in advance.

Therefore, while reducing the amount of data to be prepared in advance (for example, the convolution layer corresponding to each of the plurality of combinations disclosed in Non-Patent Document 2), the neural network can improve the processing efficiency and suppress the deterioration of accuracy. It is hoped that technology that makes it possible to build a network will be provided.

In order to solve the above problem, according to a certain viewpoint of the present invention, an input unit for acquiring a first neural network including a plurality of processing layers and at least one processing layer of the plurality of processing layers are to be modified. A first quantization function containing a first trainable coefficient, a second quantization function containing a trainable second coefficient, and a channel for the layer to be modified, which are specified as layers. A modification part that introduces a channel attenuation function containing a trainable third coefficient of the unit to generate a second neural network, and learning based on the second neural network of the first neural network. The weight parameter is retrained by a learning unit that trains the weight parameter, the first coefficient, the second coefficient, and the third coefficient, and retraining based on the second neural network after learning. The learning unit and the third neural network in which the channel attenuation function and the weight parameter of the redundant channel corresponding to the third coefficient after training in the modified target layer are deleted from the second neural network after re-learning. A neural network weight reduction device including an output unit for output is provided.

The learning unit may perform the first learning for training the weight parameter and the second learning for training the first coefficient, the second coefficient, and the third coefficient one by one. ..

The channel decay function may include a process of executing a multiplication of a value corresponding to the third coefficient with respect to the input to the correction target layer on a channel-by-channel basis in the second learning.

The channel decay function includes a process of zeroing an input corresponding to a channel whose value corresponding to the third coefficient is lower than a predetermined threshold value among the inputs to the correction target layer in the first learning. But it may be.

The redundant channel may be a channel in which the value corresponding to the third coefficient after training is lower than the predetermined threshold value.

In the second learning, the channel decay function executes multiplication of a value corresponding to the third coefficient and multiplication of an adjustment parameter whose value is gradually reduced with respect to the input to the correction target layer. It may include the processing to be performed.

In the second learning, the learning unit may gradually reduce the adjustment parameter by performing learning based on the loss function in which the adjustment parameter is incorporated.

In the second learning, the learning unit may gradually reduce the adjustment parameters according to a predetermined schedule.

The modification unit introduces the channel attenuation function and the first quantization function so that the channel attenuation function and the first quantization function are applied to the input to the modification target layer. May be good.

The first quantization function may include a process of multiplying the output from the channel decay function by the first coefficient after performing the first normalization.

The first normalization may include a transformation that keeps the output from the channel decay function in the first range.

The modification unit may introduce the second quantization function so that the second quantization function is applied to the weight parameter of the modification target layer.

The second quantization function may include a process of multiplying the weight parameter of the layer to be modified by the second coefficient after performing the second normalization.

The second normalization may include a transformation that puts the weight parameter of the layer to be modified into the second range.

The modification target layer may include at least one of a convolution layer and a fully connected layer.

The re-learning unit fixes the first coefficient to the first coefficient after training, fixes the second coefficient to the second coefficient after training, and fixes the third coefficient to the second coefficient after training. The weight parameter may be retrained in a state fixed to the coefficient of 3.

Further, according to another aspect of the present invention, the first neural network including the plurality of processing layers is acquired, and at least one processing layer of the plurality of processing layers is specified as the modification target layer. A first quantization function containing a trainable first coefficient, a second quantization function containing a trainable second coefficient, and a trainable first on a channel-by-channel basis for the layer to be modified. By introducing a channel attenuation function including a coefficient of 3 to generate a second neural network and learning based on the second neural network, the weight parameter of the first neural network and the first Retraining the weighting parameters by training the coefficients, the second coefficient, and the third coefficient, and retraining based on the second neural network after training, and the second after retraining. To output a third neural network in which the channel attenuation function and the weight parameter of the redundant channel corresponding to the third coefficient after training in the modified target layer are removed from the neural network of the neural network. A weight reduction method is provided.

Further, according to another aspect of the present invention, the computer has an input unit for acquiring a first neural network including a plurality of processing layers and at least one processing layer of the plurality of processing layers as a modification target layer. A first quantization function containing a first trainable coefficient, a second quantization function containing a trainable second coefficient, and a channel-by-channel unit for the layer to be modified. The weight parameter of the first neural network by the correction part which introduces the channel attenuation function including the trainable third coefficient to generate the second neural network and the learning based on the second neural network. And a learning unit that trains the first coefficient, the second coefficient, and the third coefficient, and a re-learning unit that retrains the weight parameter by re-learning based on the second neural network after learning. And outputs a third neural network in which the channel attenuation function and the weight parameter of the redundant channel corresponding to the third coefficient after training in the modified target layer are deleted from the second neural network after retraining. A program is provided that functions as a neural network weight reduction device including an output unit.

As described above, according to the present invention, there is provided a technique capable of constructing a neural network capable of improving processing efficiency and suppressing deterioration of accuracy while reducing the amount of data to be prepared in advance. ..

It is a figure which shows the functional structure example of the neural network weight reduction apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the neural network which is the object of weight reduction. It is a figure which shows the general composition example of the convolution layer of the 1st layer. It is a flowchart which shows the operation example of the correction part. It is a figure for demonstrating the introduction example of the channel decay function and the quantization function into the convolution layer of the 1st layer. It is a flowchart which shows the operation example of a learning part. It is a figure for demonstrating the modification of the coefficient update. It is a figure which shows the hardware composition of the information processing apparatus as an example of the neural network weight reduction apparatus which concerns on embodiment of this invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

Further, in the present specification and the drawings, a plurality of components having substantially the same functional configuration may be distinguished by adding different numbers after the same reference numerals. However, if it is not necessary to distinguish each of a plurality of components having substantially the same functional configuration, only the same reference numerals are given. Further, similar components of different embodiments may be distinguished by adding different alphabets after the same reference numerals. However, if it is not necessary to distinguish each of the similar components of different embodiments, only the same reference numerals are given.

(1. Details of the embodiment)
Subsequently, the details of the embodiment of the present invention will be described.

(1-1. Explanation of configuration)
First, a configuration example of the neural network weight reduction device according to the embodiment of the present invention will be described. FIG. 1 is a diagram showing a functional configuration example of the neural network weight reduction device according to the embodiment of the present invention. As shown in FIG. 1, the neural network weight reduction device 10 according to the embodiment of the present invention includes an input unit 100, a correction unit 101, a learning unit 102, a re-learning unit 103, and an output unit 104.

The neural network weight reduction device 10 includes an arithmetic unit such as a CPU (Central Processing Unit), and a program stored in a memory (not shown) is expanded and executed by the CPU in a RAM (Random Access Memory). The function can be realized. At this time, a computer-readable recording medium on which the program is recorded may also be provided. Alternatively, the neural network weight reduction device 10 may be configured by dedicated hardware, or may be configured by a combination of a plurality of hardware.

(Input unit 100)
The input unit 100 acquires the data (learning data set) used for learning the lightening target neural network (first neural network) and the lightening target neural network. For example, the input unit 100 may acquire the neural network to be reduced in weight and the training data set by reading them from a memory (not shown). For example, the neural network to be reduced in weight may be the structure (model structure) of the neural network before learning.

FIG. 2 is a diagram showing an example of a neural network to be reduced in weight. As shown in FIG. 2, the neural network to be lightened is composed of a plurality of layers from the first layer to the Nth layer (N is an integer of 2 or more). Input data is input to the first layer, and output data is output from the Nth layer. Each layer from the first layer to the Nth layer contains a treatment layer, and an activation function is inserted in the next layer of each layer from the first layer to the Nth layer. The processing layer included in each layer from the first layer to the Nth layer outputs to the activation function included in the next layer.

In the example shown in FIG. 2, the treated layer included in each layer from the first layer to the (N-1) layer is a convolutional layer, and the treated layer included in the Nth layer is a fully connected layer. .. However, the type of the treatment layer included in each layer from the first layer to the Nth layer is not limited to the example shown in FIG. For example, the neural network to be lightened may include one or more of the convolution layer and the fully connected layer as the processing layer, and may include one or more of each of the convolution layer and the fully connected layer as the processing layer. .. Further, the neural network to be reduced in weight may include a processing layer other than the convolutional layer and the fully connected layer.

Further, FIG. 2 shows weight parameters w1 to ^wN as weight parameters used by the processing layers from the first layer to the ^Nth layer of the neural network to be reduced in weight. In the embodiment of the present invention, it is assumed that the feature amount and the weight parameter represented by the floating point of 16 to 32 bits are used for the operation by each processing layer of the neural network to be lightened. However, the format of each feature quantity and weight parameter used by each processing layer of the neural network to be lightened is not limited to such an example.

The explanation will be continued by returning to FIG. The neural network to be reduced in weight and the training data set acquired by the input unit 100 are output to the correction unit 101.

(Correction part 101)
The correction unit 101 specifies at least one processing layer included in the weight reduction target neural network as the correction target layer based on the lightening target neural network input from the input unit 100. Here, it is assumed that all of the convolutional layer included in the first layer to the (N-1) layer and the fully connected layer included in the Nth layer are specified as the modification target layer. However, the modification unit 101 may specify only a part of the convolution layer and the fully connected layer included in the neural network to be lightened as the modification target layer (that is, the modification target layer is the convolution layer and the fully connected layer. May include at least one of).

For example, the correction unit 101 may specify only a predetermined part of the convolution layer and the fully connected layer included in the neural network to be lightened as the correction target layer. As an example, the first convolution layer (that is, the convolution layer contained in the first layer) and the last convolution layer (that is, the convolution layer contained in the (N-1) layer) are the convolution layers of other layers. It does not have to be specified as a layer to be modified because it may have a greater effect on the accuracy of the neural network than.

The correction unit 101 includes a trainable coefficient γ1 (first coefficient) for the correction target layer (for each of the plurality of correction target layers when a plurality of correction target layers are specified). A first quantization function, a second quantization function containing a trainable coefficient γ2 (second coefficient), and a channel attenuation function containing a trainable coefficient α (third coefficient) on a channel-by-channel basis. And introduce. For example, a quantization function can mean a function that transforms a continuous value into a discrete value. As a result, the correction unit 101 generates a neural network (second neural network) to be trained. The neural network to be trained and the training data set generated by the correction unit 101 are output to the learning unit 102.

(Learning unit 102)
The learning unit 102 performs learning based on the training target neural network input from the correction unit 101 based on the learning data set input from the correction unit 101. For example, the learning unit 102 performs learning based on the neural network to be trained by using an error back propagation method (backpropagation) or the like. As a result, the weight parameter, the coefficient γ1 included in the first quantization function, the coefficient γ2 included in the second quantization function, and the coefficient α included in the channel attenuation function are trained.

As will be described in detail later, it is desirable that the learning unit 102 performs the first learning for training the weight parameter and the second learning for training the coefficients γ1, the coefficient γ2, and the coefficient α one by one. .. A random number may be used as the initial value of the weight parameter, but if there is a trained weight parameter of the neural network to be lightened, the trained weight parameter may be used as the initial value. The neural network and the training data set to be trained after learning by the learning unit 102 are output to the re-learning unit 103.

(Re-learning unit 103)
The re-learning unit 103 performs re-learning based on the post-learning training target neural network input from the learning unit 102 based on the learning data set input from the learning unit 102. For example, the re-learning unit 103 initializes the weight parameter and performs re-learning based on the neural network to be trained after training by using an error back-propagation method or the like. This retrains the weight parameters. The neural network to be trained after re-learning by the re-learning unit 103 is output to the output unit 104.

(Output unit 104)
The output unit 104 deletes the channel attenuation function from the retrained neural network input from the relearning unit 103, and sets the weight parameter of the redundant channel according to the post-training coefficient α in the modified target layer. Delete it to generate a neural network (third neural network) to be output. Then, the output unit 104 outputs the neural network to be output. The neural network to be output may be output in any way. For example, the output unit 104 may record the output target neural network on the recording medium by outputting the output target neural network to the recording medium. Alternatively, the output unit 104 may transmit the output target neural network to another device via the communication device by outputting the output target neural network to the communication device.

(1-2. Explanation of operation)
Subsequently, an operation example of the neural network weight reduction device 10 according to the embodiment of the present invention will be described. As described above, the input unit 100 acquires the neural network (FIG. 2) to be reduced in weight and the training data set. Here, as an example, it is assumed that a two-dimensional image is used as learning data. At this time, the operation performed by the convolution layer of the l-th layer included in the neural network to be reduced in weight is shown by the following mathematical formula (1).

Here, x ^l indicates the input feature amount to the convolutional layer of the l-th layer, and wl indicates the weight parameter used by the convolutional layer of the ^l -th layer, and the subscripts i, j, n , M indicate an output channel, an input channel, an image width, and an image height, respectively, and f () indicates an activation function. However, in the formula (1), the subscripts indicating the width and height of the image corresponding to the input feature amount x ^{il + 1} to the ( _l + 1) layer are omitted. As shown in equation (1), the activation function is applied after the inner product of the weight parameter and the input feature is calculated. For example, a ramp function or the like may be used as the activation function. Also, batch normalization may be applied before applying the activation function.

FIG. 3 is a diagram showing a general configuration example of the convolutional layer of the first layer. Referring to FIG. 3, the convolutional layer 202 of the first layer is shown. In the convolution layer 202 of the first layer, data to which the activation function is applied to the output from the previous layer is input as an input feature amount x ^l . Further, the convolutional layer 202 of the first layer has a weight parameter ^wl . The convolutional layer 202 of the first layer calculates the inner product of the input feature amount x ^l and the weight parameter w ^l . The calculation result is output to the next layer. The input unit 100 outputs the neural network to be reduced in weight and the learning data set to the correction unit 101.

FIG. 4 is a flowchart showing an operation example of the correction unit 101. The correction unit 101 identifies the correction target layer based on the neural network of the weight reduction target input from the input unit 100 (S100). Here, it is assumed that all of the convolutional layer included in the first layer to the (N-1) layer and the fully connected layer included in the Nth layer are specified as the modification target layer. The modification unit 101 can train the modification target layer in units of a channel, a first quantization function including a trainable coefficient γ1, a second quantization function including a trainable coefficient γ2, and a channel unit. A channel attenuation function including the coefficient α is introduced (S101). As an example, an example of introducing a channel decay function and a quantization function into the convolutional layer of the first layer will be described.

FIG. 5 is a diagram for explaining an example of introducing a channel decay function and a quantization function into the convolution layer of the first layer. Referring to FIG. 5, the convolutional layer 202 of the first layer is shown. Further, referring to FIG. 5, the input feature amount x ^l is shown as the input to the convolution layer 202 of the first layer, and the weight parameter ^wl of the convolution layer 202 of the first layer is shown.

As shown in FIG. 5, the correction unit 101 has a channel attenuation function 204 and a quantization function 205 (first quantization function) with respect to an input (input feature amount x ^l ) to the convolution layer 202 of the first layer. ) Is applied, and the channel attenuation function 204 and the quantization function 205 are introduced. The channel decay function 204 includes a trainable coefficient α ^l for each channel. The quantization function 205 includes a trainable coefficient γ ^{1 l} .

On the other hand, the correction unit 101 introduces the quantization function 206 so that the quantization function 206 (second quantization function) is applied to the weight parameter ^wl of the convolution layer 202 of the first layer. The quantization function 206 includes a trainable coefficient γ 2 ^l .

The channel attenuation function 204 is a function that attenuates the value corresponding to each channel of the input (input feature amount x ^l ) to the convolution layer 202 of the first layer. As will be described later, the first learning for training the weight parameter w (hereinafter, also simply referred to as “weight parameter training”) and the second training for the coefficient α ^l , the coefficient γ1 ^l , and the coefficient ^{γ2 l .} Learning (hereinafter, also simply referred to as "coefficient training") is performed one by one. The channel decay function 204 includes a process applied during weight parameter training and a process applied during coefficient training.

More specifically, the channel attenuation function 204 executes the multiplication of the input (input feature amount x ^l ) to the convolution layer 202 of the first layer by the value corresponding to the coefficient α ^l on a channel-by-channel basis during the coefficient training. Including processing. Further, the channel attenuation function 204 is a process of executing the multiplication of the adjustment parameter η ^l whose value gradually decreases with respect to the input (input feature amount x ^l ) to the convolution layer 202 of the first layer during the coefficient training. including.

For example, assuming that the number of channels of the input feature amount x ^l is C, x ^l is expressed as x _i ^l (i = 1, 2, ..., C), and the coefficient α ^l is the input feature amount x. It can be expressed as a vector α il ( _i = ¹ , 2, ..., C) having the same number of elements as the number of channels C of ^l . As an example of the value corresponding to the coefficient α ^l , the value obtained by applying the softmax function to the coefficient α ^l can be used. At this time, the value corresponding to the coefficient α ^il corresponding to the channel _{i can be expressed as softmax i (} α ^l ). As an example, the channel attenuation function 204 includes a process expressed by the following mathematical formula (2) as a process at the time of coefficient training.

As will be explained later, as α ^l is trained, the value of α ^l will differ between channels. More specifically, a channel having a value of α ^l closer to 0 can be regarded as a channel having a smaller contribution to the accuracy of the neural network (more likely to be a redundant channel). Further, as will be described later, the value of the adjustment parameter η ^l gradually decreases within the range of 0 or more at the time of coefficient training. As the adjustment parameter η ^l becomes smaller, it is considered that the difference in the values of α ^l generated between the channels becomes larger, so that it is expected that it becomes easier to identify the redundant channel.

On the other hand, the channel attenuation function 204 corresponds to a channel in which the value corresponding to the coefficient α ^l is lower than the predetermined threshold value δ among the inputs (input feature amount x ^l ) to the convolution layer 202 of the first layer during the weight parameter training. Includes a process to make the input (input feature amount x ^l ) to be zero. The threshold δ may be a given non-negative value. As an example, the channel decay function 204 includes a process expressed as the following mathematical formula (3) as a process at the time of weight parameter training.

That is, when softmax _i (α ^l ) falls below the threshold value δ, the channel i is considered to be a redundant channel, and the input feature amount x _i ^l corresponding to the channel i is set to zero.

The quantization function 205 includes a process of performing normalization (first normalization) on the output X ^l from the channel attenuation function 204 and then multiplying the coefficient γ ^{1 l} . The normalization for the output X ^l from the channel decay function 204 may include a transformation that puts the output X ^l from the channel decay function 204 into a predetermined range (first range). Here, as a conversion within a predetermined range, an operation of dividing the output X ^l from the channel decay function 204 by max | X ^l |, which is the maximum value of the absolute value of the output X ^l in all channels in the first layer, is used. Imagine a case.

As an example, when the output X ^l from the channel attenuation function 204 is quantized to a k-bit signed integer by the quantization function 205, the quantization function 205 is expressed by the following equation (4). Including processing.

In formula (4), the Round function is a function that rounds a value to an integer (eg, by rounding). β1 is the reciprocal of 2 ^k-1 / max (| X ^l |) (that is, max (| X ^l | / 2 ^k-1 ). That is, the quantize function shown in the equation (4) has a value. It has a form of rounding to an integer and then multiplying it by β1 expressed by a floating point number to return it to a floating point number. For example, the quantization function 205 may have such a form at the learning stage. The output to the next layer does not change even if the multiplication of is applied after the calculation by the convolution layer 202. Therefore, in the inference stage, β1 may be moved to the rear stage of the convolution layer 202. An integer after the value is rounded by the Round function is input to 202, and the load due to the convolution operation can be reduced.

Although there is a description about quantization in the above-mentioned Non-Patent Document 3, in the quantization already disclosed as described above, γ1 = 1 (fixed value). On the other hand, in the quantization shown in the equation (4), unlike the quantization already disclosed as described above, the trainable γ1 is included in the quantize function. The optimum number of quantization bits can be estimated by training γ1. As an example, when γ1 = 1 and k = ⁸ bits, the maximum value after the Round function is applied is 27-1. On the other hand, when γ1 = ^{2 -4} and k = 8 bits, the maximum value after the Round function is applied is 23, and the value after the Round function is applied can be expressed by 4 bits. Will be.

In the above, it is assumed that the quantization function 205 is used to quantize the k-bit into a signed integer. However, the quantization function 205 may quantize k-bits into unsigned integers. In such a case, 2 ^k-1 in the mathematical formula (4) may be replaced with 2 ^k .

The quantization function 206 includes a process of performing normalization (second normalization) on the weight parameter w ^l and then multiplying the coefficient γ 2 ^l . The normalization for the weight parameter ^wl may include a transformation that puts the weight parameter ^wl within a predetermined range (second range). Here, it is assumed that the weight parameter ^wl is divided by max | ^wl |, which is the maximum value of all channels in the first layer of the absolute value of ^wl , as a conversion to be within a predetermined range.

As an example, when the weight parameter ^wl is quantized to a k-bit signed integer by the quantization function 206, the quantization function 206 includes a process expressed as the following equation (5).

In the formula (5), the Round function has the same characteristics as the Round function shown in the formula (4). β2 is the reciprocal of 2 ^k-1 / max (| ^wl |) (that is, max (| wl | / 2 ^{k-1 )} . The quantize function shown in the formula (5) is also the formula (4). Similar to the quantize function shown in, the value is rounded to an integer and then multiplied by β2 expressed by the floating point to return to the floating point. For example, the quantization function 206 is used in the learning stage. It may have such a form. Further, in the reasoning stage, β2 may be moved to the subsequent stage of the folding layer 202.

The quantization function 206 may be quantized to an unsigned integer of k bits in the same manner as the quantization function 205. In such a case, 2 ^k-1 in the mathematical formula (5) may be replaced with 2 ^k .

The explanation will be continued by returning to FIG. If there is a layer to be modified that does not introduce the channel attenuation function 204, the quantization function 205, and the quantization function 206 (“NO” in S102), the modification unit 101 still quantizes with the channel attenuation function 204. S101 is executed for the modification target layer in which the function 205 and the quantization function 206 are not introduced. On the other hand, when the correction unit 101 finishes introducing the channel decay function 204, the quantization function 205, and the quantization function 206 for all the correction target layers (“YES” in S102), the correction unit 101 ends the correction. do.

The explanation will be continued by returning to FIG. The correction unit 101 outputs the neural network to be trained and the training data set generated by the introduction of the channel attenuation function 204, the quantization function 205, and the quantization function 206 to the learning unit 102. As described above, the learning unit 102 performs learning based on the training target neural network input from the correction unit 101 based on the learning data set input from the correction unit 101. As a result, the weight parameter w, the coefficient α, the coefficient γ1 and the coefficient γ2 are trained.

FIG. 6 is a flowchart showing an operation example of the learning unit 102. As described above, the learning unit 102 performs weight parameter training and coefficient training one by one. First, the learning unit 102 initializes the weight parameter w of the neural network to be trained (S110), and performs weight parameter training. More specifically, the learning unit 102 is weighted by an error backpropagation method based on the loss function (for example, a stochastic gradient descent method based on the error backpropagation method) in a state where the coefficient α, the coefficient γ1 and the coefficient γ2 are fixed. The parameter w is updated (S111). In the weight parameter training, the input features corresponding to the redundant channels are set to zero (formula (3)).

The loss function used in the embodiment of the present invention is not limited to a specific function, and a loss function similar to the loss function used in a general neural network may be used. For example, the learning unit 102 may calculate the difference between the output value from the neural network to be trained and the correct answer value based on the training data set, and calculate the mean square error based on the difference as a loss function. ..

Subsequently, the learning unit 102 determines whether or not the number of updates of the weight parameter w has reached a predetermined number of times (S112). For example, the number of updates of the weight parameter w may be the number of iterations, and the predetermined number of times may be the threshold value of the number of iterations (for example, 5 times). When the learning unit 102 determines that the number of updates of the weight parameter w has not reached a predetermined number of times (“NO” in S112), the learning unit 102 returns to S111.

On the other hand, when it is determined that the number of updates of the weight parameter w has reached a predetermined number of times (“YES” in S112), the learning unit 102 performs coefficient training. More specifically, the learning unit 102 uses an error backpropagation method based on a loss function to which a regularization term is added (for example, a stochastic gradient descent method based on the error backpropagation method) in a state where the weight parameter w is fixed. The coefficient α, the coefficient γ1, and the coefficient γ2 are updated (S113). For example, the loss function to which the regularization term is given can be expressed by the following mathematical formula (6).

In the equation (6), the loss function L, which is the first term, is not limited in the same manner as the loss function of the weight parameter training. Each of the second, third and fourth terms is a regularization term. λ ₁ , λ ₂ and λ ₃ are coefficients that determine the strength of the regularization and may be given non-negative values. The second term contains the sum of the L1 norms of the adjustment parameter η ^l over the entire layer to be modified. The learning unit 102 can gradually reduce the adjustment parameter η by performing learning based on the loss function to which the adjustment parameter η is given.

However, the method of gradually reducing the adjustment parameter η is not limited to such an example. For example, the learning unit 102 may gradually reduce the adjustment parameter η according to a predetermined schedule during the coefficient training. As an example, the learning unit 102 may reduce the adjustment parameter η by a predetermined width for each iteration a predetermined number of times (for example, the adjustment parameter η may be reduced by 0.001 for each iteration). As described above, it is expected that gradually reducing the adjustment parameter η will help identify the redundant channel.

The third term includes the sum of the L1 norms of the coefficient γ1 ^l included in the quantization function 205 in the entire correction target layer. That is, the third term is a constraint term regarding the coefficient γ1 included in the quantization function 205. Similarly, the fourth term contains the sum of the L1 norms of the coefficient γ2 ^l included in the quantization function 206 in the entire correction target layer. That is, the fourth term is a constraint term regarding the coefficient γ2 included in the quantization function 206.

It is considered that the loss function L becomes smaller as the number of quantization bits increases. Therefore, if the coefficient γ1 and the coefficient γ2 are simply updated based on the loss function L, the coefficient γ1 and the coefficient γ2 become large, and it is considered that the number of quantization bits cannot be suppressed. However, by applying such a constraint term to the loss function L, it is possible not only to suppress the deterioration of the accuracy of the neural network but also to estimate the number of quantization bits suppressed to a necessary degree.

As described above, in the coefficient training by the learning unit 102, the number of channels (that is, the number of channels other than redundant channels) and the number of quantization bits can be estimated at the same time. Therefore, the optimum solution for the number of channels and the number of quantization bits can be obtained while considering the trade-off relationship existing between the number of channels and the number of quantization bits. As a result, when the number of channels and the number of quantization bits are estimated independently (for example, when quantization is performed on the model after channel reduction, or channel reduction is performed on the model that has been quantized. It can be expected to construct a neural network that suppresses the deterioration of processing efficiency while suppressing the deterioration of accuracy.

Subsequently, the learning unit 102 determines whether or not the number of updates of the coefficient γ1, the coefficient γ2, and the coefficient α has reached a predetermined number of times (S114). For example, the number of updates of the coefficient γ1, the coefficient γ2, and the coefficient α may be the number of iterations, and the predetermined number of times may be the threshold value of the number of iterations (for example, 3 times). When the learning unit 102 determines that the number of updates of the coefficient γ1, the coefficient γ2, and the coefficient α has not reached a predetermined number of times (“NO” in S114), the learning unit 102 returns to S113.

On the other hand, when the learning unit 102 determines that the number of updates of the weight parameter w has reached a predetermined number (“YES” in S114), the learning unit 102 determines whether or not the loss function to which the regularization term is given has converged. Judgment (S115). When the learning unit 102 determines that the loss function to which the regularization term is added has not converged (“NO” in S115), the learning unit 102 returns to S111. On the other hand, when it is determined that the loss function to which the regularization term is added has converged (“YES” in S115), the learning unit 102 ends the training of the neural network to be trained. For example, when the loss function with the regularization term or its change becomes smaller than the threshold value, it may be determined that the loss function with the regularization term has converged.

The neural network to be trained and the data set for training after training are output to the re-learning unit 103.

The re-learning unit 103 performs re-learning based on the post-learning training target neural network input from the learning unit 102 based on the learning data set input from the learning unit 102. More specifically, the re-learning unit 103 initializes the weight parameter w, fixes the coefficient γ1 to the coefficient γ1 after training by the learning unit 102, and fixes the coefficient γ2 to the coefficient γ2 after training by the learning unit 102. In a state where α is fixed to the coefficient α after training by the learning unit 102, the weight parameter w is updated by an error back propagation method based on the loss function (for example, a stochastic gradient descent method based on the error back propagation method). As a result, the optimum weight parameter w in the state where the number of channels and the number of quantization bits are specified is acquired, and further improvement in the accuracy of the neural network can be expected.

The neural network to be trained after re-learning by the re-learning unit 103 is output to the output unit 104.

The output unit 104 deletes the channel attenuation function 204 introduced for the modification target layer from the retraining target neural network input from the relearning unit 103, and the output unit 104 deletes the training coefficient in the modification target layer. Delete the weight parameter of the redundant channel according to α. As a result, the neural network to be output is generated. The redundant channel may be a channel whose value corresponding to the coefficient α after training is below the threshold value δ. For example, if the softmax _i (α ^l ) after training is below the threshold value δ, the channel i is considered to be a redundant channel in the first layer, and the weight parameter ^il of the channel _i is deleted from the first layer. To.

The neural network to be output is not limited to this example, and various modifications may be applied. For example, the output unit 104 may integrate the coefficient γ1 ^l after training and the number of quantization bits k set as the initial value (for example, in equation (4), γ1 ^l = 2 ^-4 and k = 8). If, (2 ^-4 ) x ( ^{2 8-1} ) -1 may be integrated into 23). Similarly, the output unit 104 may integrate the post-training coefficients γ2 ^l and k.

Further, as described above, the output unit 104 may move β1 included in the quantization function 205 in the learning stage to the subsequent stage of the convolution layer 202. As a result, in the inference stage, the convolution layer 202 does not include β1 represented by a floating point number, so that the load of the convolution operation by the convolution layer 202 can be reduced. Similarly, the output unit 104 may move β2 included in the quantization function 206 in the learning stage to the subsequent stage of the convolution layer 202. The output unit 104 outputs the neural network to be output generated in this way.

(1-3. Explanation of the effect)
According to an embodiment of the present invention, there is provided a neural network weight reduction device 10 including an input unit 100, a correction unit 101, a learning unit 102, a re-learning unit 103, and an output unit 104. The input unit 100 acquires a neural network to be reduced in weight including a plurality of processing layers. Then, the correction unit 101 identifies at least one processing layer of the plurality of processing layers as the correction target layer, and for the correction target layer, a quantization function 205 including a trainable coefficient γ1 and a trainable coefficient. A quantization function 206 including γ2 and a channel attenuation function 204 including a trainable coefficient α for each channel are introduced to generate a neural network to be trained.

The learning unit 102 trains the weight parameter w, the coefficient γ1, the coefficient γ2, and the coefficient α of the neural network to be lightened by learning based on the neural network to be trained. The re-learning unit 103 retrains the weight parameter w by re-learning based on the neural network to be trained after learning. The output unit 104 outputs the output target neural network in which the channel attenuation function 204 and the redundant channel weight parameter w according to the post-training coefficient α in the correction target layer are deleted from the retrained target neural network. do.

According to such a configuration, the number of channels (that is, the number of channels other than redundant channels) and the number of quantization bits can be estimated at the same time. Therefore, the optimum solution for the number of channels and the number of quantization bits can be obtained while considering the trade-off relationship existing between the number of channels and the number of quantization bits. This makes it possible to construct a neural network that suppresses a decrease in processing efficiency while suppressing a deterioration in accuracy as compared with a case where the number of channels and the number of quantization bits are estimated independently.

The details of the embodiment of the present invention have been described above.

(2. Various modifications)
Although the preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, the present invention is not limited to these examples. It is clear that a person having ordinary knowledge in the field of technology to which the present invention belongs can come up with various modifications or modifications within the scope of the technical ideas described in the claims. , These are also naturally understood to belong to the technical scope of the present invention.

For example, in the above, the example in which the learning unit 102 independently updates the coefficient α, the coefficient γ1, and the coefficient γ2 has been mainly described. However, the learning unit 102 may generate a coefficient α, a coefficient γ1, and a coefficient γ2 based on a neural network (fourth neural network) different from the neural network to be trained. A modified example of such coefficient update will be described with reference to FIG. 7.

FIG. 7 is a diagram for explaining a modified example of coefficient update. Referring to FIG. 7, a neural network 209 (fourth neural network) is provided separately from the neural network to be trained. The configuration of the neural network 209 is not particularly limited. For example, the neural network 209 may include at least one of a convolution layer and a fully connected layer. The learning unit 102 updates the weight parameter of the neural network 209 by the error back propagation method based on the loss function to which the regularization term is given in the coefficient training.

The learning unit 102 causes the neural network 209 to input data based on the input to the correction target layer (input feature amount x ^l ), and based on the output from the neural network 209 corresponding to the data, the coefficient α and the coefficient γ1 And the coefficient γ2 may be generated. In such a case, the coefficient α, the coefficient γ1, and the coefficient γ2 depend on the input to the correction target layer. At this time, the same data as the input to the correction target layer may be input to the neural network 209, or a uniquely determined statistic such as the average value of the input to the correction target layer is input to the neural network 209 as a representative value. You may.

(3. Hardware configuration example)
Subsequently, a hardware configuration example of the neural network weight reduction device 10 according to the embodiment of the present invention will be described. Hereinafter, as a hardware configuration example of the neural network weight reduction device 10 according to the embodiment of the present invention, a hardware configuration example of the information processing apparatus 900 will be described. The hardware configuration example of the information processing device 900 described below is only an example of the hardware configuration of the neural network weight reduction device 10. Therefore, as for the hardware configuration of the neural network weight reduction device 10, an unnecessary configuration may be deleted from the hardware configuration of the information processing apparatus 900 described below, or a new configuration may be added.

FIG. 8 is a diagram showing a hardware configuration of an information processing device 900 as an example of the neural network weight reduction device 10 according to the embodiment of the present invention. The information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, and an interface 907. , An input device 908, an output device 909, a storage device 910, and a communication device 911.

The CPU 901 functions as an arithmetic processing device and a control device, and controls the overall operation in the information processing device 900 according to various programs. Further, the CPU 901 may be a microprocessor. The ROM 902 stores programs, calculation parameters, and the like used by the CPU 901. The RAM 903 temporarily stores a program used in the execution of the CPU 901, parameters that appropriately change in the execution, and the like. These are connected to each other by a host bus 904 composed of a CPU bus or the like.

The host bus 904 is connected to an external bus 906 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 905. It is not always necessary to separately configure the host bus 904, the bridge 905, and the external bus 906, and these functions may be implemented in one bus.

The input device 908 includes input means for the user to input information such as a mouse, keyboard, touch panel, buttons, microphones, switches and levers, and an input control circuit that generates an input signal based on the input by the user and outputs the input signal to the CPU 901. It is composed of etc. By operating the input device 908, the user who operates the information processing device 900 can input various data to the information processing device 900 and instruct the processing operation.

The output device 909 includes, for example, a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, a display device such as a lamp, and an audio output device such as a speaker.

The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, a deleting device for deleting data recorded on the storage medium, and the like. The storage device 910 is composed of, for example, an HDD (Hard Disk Drive). The storage device 910 drives a hard disk and stores programs and various data executed by the CPU 901.

The communication device 911 is a communication interface composed of, for example, a communication device for connecting to a network. Further, the communication device 911 may support either wireless communication or wired communication.

The hardware configuration example of the neural network weight reduction device 10 according to the embodiment of the present invention has been described above.

10 Neural network weight reduction device 100 Input unit 101 Correction unit 102 Learning unit 103 Re-learning unit 104 Output unit

Claims (18)

An input unit that acquires a first neural network that includes multiple processing layers,
At least one processing layer of the plurality of processing layers is specified as a modification target layer, and a first quantization function including a trainable first coefficient and a trainable second processing layer are specified for the modification target layer. A modification part that introduces a second quantization function containing the coefficients of and a channel attenuation function containing a trainable third coefficient on a channel-by-channel basis to generate a second neural network.
A learning unit that trains the weight parameter of the first neural network, the first coefficient, the second coefficient, and the third coefficient by learning based on the second neural network.
A re-learning unit that retrains the weight parameters by re-learning based on the second neural network after learning,
An output unit that outputs a third neural network in which the channel attenuation function and the weight parameter of the redundant channel corresponding to the third coefficient after training in the modified target layer are deleted from the second neural network after retraining. When,
A neural network weight reduction device equipped with. The learning unit performs a first learning for training the weight parameter and a second learning for training the first coefficient, the second coefficient, and the third coefficient, one by one.
The neural network weight reduction device according to claim 1. The channel decay function includes a process of executing a multiplication of a value corresponding to the third coefficient with respect to an input to the correction target layer on a channel-by-channel basis in the second learning.
The neural network weight reduction device according to claim 2. The channel decay function includes a process of zeroing an input corresponding to a channel whose value corresponding to the third coefficient is lower than a predetermined threshold value among the inputs to the correction target layer in the first learning. ,
The neural network weight reduction device according to claim 3. The redundant channel is a channel in which the value corresponding to the third coefficient after training is lower than the predetermined threshold value.
The neural network weight reduction device according to claim 4. In the second learning, the channel decay function executes multiplication of a value corresponding to the third coefficient and multiplication of an adjustment parameter whose value is gradually reduced with respect to the input to the correction target layer. Including processing to do,
The neural network weight reduction device according to any one of claims 3 to 5. In the second learning, the learning unit gradually reduces the adjustment parameter by performing learning based on the loss function in which the adjustment parameter is incorporated.
The neural network weight reduction device according to claim 6. In the second learning, the learning unit gradually reduces the adjustment parameters according to a predetermined schedule.
The neural network weight reduction device according to claim 6. The modification unit introduces the channel attenuation function and the first quantization function so that the channel attenuation function and the first quantization function are applied to the input to the modification target layer.
The neural network weight reduction device according to any one of claims 1 to 8. The first quantization function includes a process of multiplying the output from the channel decay function by the first coefficient after performing the first normalization.
The neural network weight reduction device according to claim 9. The first normalization comprises transforming the output from the channel decay function into the first range.
The neural network weight reduction device according to claim 10. The modification unit introduces the second quantization function so that the second quantization function is applied to the weight parameter of the modification target layer.
The neural network weight reduction device according to any one of claims 1 to 11. The second quantization function includes a process of multiplying the weight parameter of the layer to be modified by the second coefficient after performing the second normalization.
The neural network weight reduction device according to claim 12. The second normalization includes a transformation that puts the weight parameter of the layer to be modified into the second range.
The neural network weight reduction device according to claim 13. The modification target layer includes at least one of a convolution layer and a fully connected layer.
The neural network weight reduction device according to any one of claims 1 to 14. The re-learning unit fixes the first coefficient to the first coefficient after training, fixes the second coefficient to the second coefficient after training, and fixes the third coefficient to the second coefficient after training. The weight parameter is retrained in a state fixed to the coefficient of 3.
The neural network weight reduction device according to any one of claims 1 to 15. Obtaining a first neural network containing multiple processing layers,
At least one processing layer of the plurality of processing layers is specified as a modification target layer, and a first quantization function including a trainable first coefficient and a trainable second processing layer are specified for the modification target layer. To generate a second neural network by introducing a second quantization function containing the coefficients of and a channel decay function containing a trainable third coefficient on a per-channel basis.
By learning based on the second neural network, the weight parameter of the first neural network, the first coefficient, the second coefficient, and the third coefficient are trained.
By retraining based on the second neural network after training, the weight parameter is retrained, and
To output a third neural network in which the channel attenuation function and the weight parameter of the redundant channel corresponding to the third coefficient after training in the modified target layer are deleted from the second neural network after retraining. ,
How to reduce the weight of neural networks, including. Computer,
An input unit that acquires a first neural network that includes multiple processing layers,
At least one processing layer of the plurality of processing layers is specified as a modification target layer, and a first quantization function including a trainable first coefficient and a trainable second processing layer are specified for the modification target layer. A modification part that introduces a second quantization function containing the coefficients of and a channel attenuation function containing a trainable third coefficient on a channel-by-channel basis to generate a second neural network.
A learning unit that trains the weight parameter of the first neural network, the first coefficient, the second coefficient, and the third coefficient by learning based on the second neural network.
A re-learning unit that retrains the weight parameters by re-learning based on the second neural network after learning,
An output unit that outputs a third neural network in which the channel attenuation function and the weight parameter of the redundant channel corresponding to the third coefficient after training in the modified target layer are deleted from the second neural network after retraining. When,
A program that functions as a neural network weight reduction device.

Images