KR102247896B1 - Convolution neural network parameter optimization method, neural network computing method and apparatus - Google Patents

Abstract

The present invention relates to a convolutional neural network parameter optimization method, a convolutional neural network calculation method, and an apparatus thereof using a shape transformation of a learned parameter. A neural network parameter optimization method suitable for hardware implementation according to the present invention comprises: Converting an existing parameter of the neural network into a code parameter and a size parameter having a single value per channel using a negative and a size parameter conversion unit; And generating an optimized parameter by pruning only a size parameter from among the shape-transformed parameters using a parameter pruning unit constituting the parameter optimization unit, wherein the shape conversion and pruning are performed in a neural network convolution operation. You can target the parameters of the root layer.
Accordingly, the present invention is effective in providing a neural network parameter optimization method, a neural network computation method, and an apparatus for obtaining a minimum loss of accuracy and maximum computation speed by effectively optimizing a large amount of computation and parameters of a convolutional neural network for hardware implementation. There is.

Description

Convolutional neural network parameter optimization method using shape transformation of learned parameters, convolutional neural network calculation method, and apparatus thereof {Convolution neural network parameter optimization method, neural network computing method and apparatus}

The present invention relates to an operation technique in an artificial neural network, and more particularly, to a neural network parameter optimization method suitable for hardware implementation, a neural network operation method, and an apparatus thereof.

With the recent development of artificial intelligence technology, various industries are applying and introducing artificial intelligence technology.

According to this trend, the demand to implement an artificial neural network such as a convolutional neural network as real-time hardware is also increasing.

However, the artificial neural network-related computation method according to the prior art is not easy to actually implement because of the large number of parameters and computational amount of the convolutional neural network.

To solve this problem, parameter optimization methods have been proposed to reduce the amount of computation compared to accuracy, such as a method of omitting unnecessary parameters through learning and a method of reducing the number of bits of a parameter.

However, since parameter optimization methods according to the prior art are limited to software calculation methods that do not consider hardware implementation, there is a limit to implementing a convolutional neural network in real-time hardware by applying only such a method.

Therefore, there is an urgent need for a practical and applicable technology for an artificial neural network-related computation technology suitable for hardware implementation.

Publication Patent Publication No. KR 10-2011-0027916 (published date 2011.03.17)

The present invention was conceived to solve the above problem, and the present invention is a method for optimizing neural network parameters to obtain a minimum loss of accuracy and maximum operation speed by effectively optimizing a large amount of computation and parameters of a convolutional neural network in hardware implementation, It is to provide a neural network computation method and its device.

In addition, it is to provide a neural network parameter optimization method, a neural network computation method, and an apparatus for correcting the accuracy through a minimum computational speed loss.

A method for optimizing a neural network parameter according to an embodiment of the present invention includes the steps of converting an existing parameter of a neural network into a code parameter and a size parameter having a single value per channel, and a parameter optimized by pruning the shape-converted size parameter. And generating.

The sign parameter determines the direction of elements for each channel of the existing parameter, and the size parameter is a value obtained by optimizing the weights with a single representative value per channel of the existing parameter.

In the step of converting the shape, a size parameter may be generated by calculating an average of the absolute values of elements for each channel of the existing parameter.

In the shape conversion step, if a predetermined element constituting the channel of the existing parameter is less than 0, the element value of the corresponding sign parameter is expressed as 0, and if it is greater than or equal to 0, the element value of the corresponding sign parameter is expressed as 1. Sign parameters can be created.

In the pruning step, a reference value is calculated by using the average value and size distribution of the size parameter for each input channel or the average value and size distribution of the size parameter for each input and output channel, and having a value less than the calculated reference value. By setting the value of the size parameter to 0, the convolution operation of the corresponding channel can be omitted.

When pruning using the average value and size distribution of the size parameters of the input channels constituting a predetermined layer, a reference value is calculated by multiplying the average value of the size parameters of the input channels by a constant for each layer, and a predetermined input If the channel size parameter value is smaller than the reference value, pruning is performed by changing the size parameter value of the corresponding channel to 0, and pruning is performed using the average value and size distribution of the size parameters of the input and output channels constituting a predetermined layer. In this case, a reference value is calculated by multiplying the average value of the size parameters of the input channels and output channels by a constant for each layer, and if the size parameter value of a predetermined input channel is less than the reference value, the size parameter value of the corresponding channel is calculated. It can be pruned by changing it to 0.

The constant value for each layer may be determined according to a distribution of convolution parameters for each layer.

The neural network parameter optimization method may further include converting the shape of an existing parameter of the neural network into a ratio parameter.

The step of converting the shape to the ratio parameter may include variably allocating bits of the ratio parameter, and quantizing the range and weight of the value of the ratio parameter element by user selection.

A neural network operation method according to another embodiment of the present invention includes the steps of loading an existing parameter and input channel data of a neural network into a memory, converting the shape of the existing parameter into a sign parameter and a size parameter having a single value per channel, and converting the shape. Pruning the sized parameter to generate an optimized parameter, convolutional calculation of the optimized parameter and input channel data to infer, correcting the optimized parameter, and converting the existing parameter to the corrected optimization parameter. It may include the step of updating.

The neural network operation method includes determining whether a learned parameter exists, generating an initial parameter through parameter initialization if there is no learned parameter, and generating an optimized parameter for the initial parameter, and the learned parameter. If there is a parameter, the step of loading an existing parameter may be further included.

The step of inferring by convolutional operation of the optimized parameter and input channel data includes loading the optimized parameter into a memory, and determining whether a value of a size parameter included in the optimized parameter loaded into the memory is 0. And, if the value of the size parameter is 0, the step of omitting the convolution operation process may be included.

Inferring the optimized parameter and input channel data by convolutional operation, if the value of the size parameter is not 0, determining the direction of the data by bitwise operation of a sign parameter and input channel data, and a convolution parameter filter It may include performing a sum operation between input channel data by a size, and performing a single multiplication operation on the size parameter and the input channel data.

The step of inferring by convolutional operation of the optimized parameter and input channel data may further include reducing an error in the operation result by differentially reflecting a weight to a size parameter using a ratio parameter if a ratio parameter exists. .

A neural network computing device according to another embodiment of the present invention includes a code parameter conversion unit that converts the shape of an existing parameter into a code parameter, a size parameter conversion unit that converts the shape of the existing parameter into a size parameter having a single value per channel, And a parameter pruning unit configured to generate an optimized parameter by pruning the shape-transformed size parameter.

The parameter pruning unit calculates a reference value using the average value and size distribution of the size parameter for each input channel or the average value and size distribution of the size parameter for each input and output channel, and a size having a value less than the calculated reference value. By setting the parameter value to 0, the convolution operation of the corresponding channel can be omitted.

When pruning by using the average value and size distribution of the size parameter of the input channel constituting a predetermined layer, the parameter pruning unit calculates a reference value by multiplying the average value of the size parameter of the input channels by a constant for each layer. Calculation, and if the size parameter value of a predetermined input channel is less than the reference value, the size parameter value of the corresponding channel is changed to 0 and pruned, and the average value and size of the size parameters of the input and output channels constituting the predetermined layer In the case of pruning using a distribution, a reference value is calculated by multiplying the average value of the size parameters of the input channels and output channels by a constant for each layer, and if the value of the size parameter of a predetermined input channel is less than the reference value, the corresponding You can prune by changing the channel size parameter value to 0.

The neural network computing device may further include a ratio parameter conversion unit for converting the shape of the existing parameter into a ratio parameter.

The neural network computing device may further include an inference unit for inferring by convolutional calculation of optimized parameters and input channel data.

The inference unit may determine whether the value of the size parameter included in the optimization parameter is 0, and if the value of the size parameter is 0, the convolution operation process may be omitted.

If there is a ratio parameter, the reasoning unit may reduce an error in an operation result by differentially reflecting the weight to the size parameter using the ratio parameter.

As described above, the present invention provides a neural network parameter optimization method, a neural network computation method, and a device thereof that can effectively optimize a large amount of computation and parameters of a convolutional neural network to obtain a minimum loss of accuracy and maximum computation speed. Has the effect of providing.

In addition, according to the present invention, by optimizing the parameters of the neural network, low power and high performance required by hardware implementing a convolutional neural network operating in real time can be satisfied. When the neural network parameter optimization technique according to an embodiment is applied, the calculation for each channel may be omitted according to the size of the channel in the convolution calculation. In addition, the hardware does not determine whether to omit or perform an operation for each element in a channel, but determines whether to omit an operation for each channel, so there is an effect of reducing the operation by a multiple of the number of channel elements.

In addition, the present invention can correct the accuracy through a minimum loss of operation speed. For example, by separating the scale parameter separately, different weights can be effectively applied according to the range of values, so performance can be efficiently improved when implementing hardware.

1 is a diagram illustrating a convolution parameter used in a neural network convolution operation according to an embodiment of the present invention;
2 is a diagram illustrating a learning process for generating an optimization parameter in a convolution neural network (CNN) according to an embodiment of the present invention;
3 is a diagram illustrating in detail a learning process for generating an optimization parameter of FIG. 2 according to an embodiment of the present invention;
4 is a diagram showing in detail an inference process according to an embodiment of the present invention;
5 is a diagram showing an application example of a convolution parameter optimization process according to an embodiment of the present invention;
6 is a diagram comparing a general convolution operation and a convolution operation according to an embodiment of the present invention;
7 is a diagram showing an advantage of a convolution operation after parameter optimization according to an embodiment of the present invention;
8 is a diagram showing an application example of a convolution parameter optimization process according to another embodiment of the present invention;
9 is a diagram comparing a parameter distribution to which only a size parameter is applied and a parameter distribution to which a ratio parameter is additionally applied to explain ratio parameter optimization;
10 is a diagram showing an example of a convolution operation using an optimized parameter according to an embodiment of the present invention;
11 is a diagram showing the configuration of a neural network computing device according to an embodiment of the present invention;
12 is a diagram showing a detailed configuration of the parameter optimization unit of FIG. 11 according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments make the disclosure of the present invention complete, and the general knowledge in the technical field to which the present invention pertains. It is provided to completely inform the scope of the invention to the possessor, and the invention is only defined by the scope of the claims. The same reference numerals refer to the same elements throughout the specification.

In describing the embodiments of the present invention, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted, and terms to be described later are in the embodiment of the present invention. These terms are defined in consideration of the functions of the user and may vary according to the intention or custom of users or operators. Therefore, the definition should be made based on the contents throughout the present specification.

Combinations of each block of the attached block diagram and each step of the flowchart may be executed by computer program instructions (execution engine), and these computer program instructions are used on a processor of a general-purpose computer, special purpose computer, or other programmable data processing device. As can be mounted, the instructions executed by the processor of a computer or other programmable data processing device generate means for performing the functions described in each block of the block diagram or each step of the flowchart.

These computer program instructions may also be stored in a computer-usable or computer-readable memory that can be directed to a computer or other programmable data processing device to implement a function in a particular manner, so that the computer-usable or computer-readable memory It is also possible to produce an article of manufacture in which the instructions stored in the block diagram contain instruction means for performing the functions described in each block of the block diagram or each step of the flowchart.

In addition, since computer program instructions can be mounted on a computer or other programmable data processing device, a series of operation steps are performed on a computer or other programmable data processing device to create a computer-executable process. It is also possible that the instructions for performing the data processing apparatus provide steps for executing the functions described in each block in the block diagram and in each step in the flowchart.

In addition, each block or each step may represent a module, segment, or part of code containing one or more executable instructions for executing specified logical functions, and in some alternative embodiments mentioned in the blocks or steps. It should be noted that it is also possible for functions to occur out of order. For example, two blocks or steps shown in succession may in fact be performed substantially simultaneously, and the blocks or steps may be performed in the reverse order of a corresponding function as necessary.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the embodiments of the present invention exemplified below may be modified in various other forms, and the scope of the present invention is not limited to the embodiments described below. Embodiments of the present invention are provided to more completely describe the present invention to those of ordinary skill in the art to which the present invention pertains.

1 is a diagram illustrating a convolution parameter used in a neural network convolution operation according to an embodiment of the present invention.

As shown in the figure, referring to FIG. 1, if the total number of convolutional layers of a neural network is l, k output channels (O ₁ ,…, O _k ) exist for each layer, and each The output channel is composed of j input channels (I ₁ , I ₂ ,…, O _j ). Each input channel has i weight parameters (W ₁ , W ₂ ,…, W _i ). In the present invention, the optimization technique is mathematically described based on the definition of the convolution parameter defined above.

2 is a diagram illustrating a learning process for generating an optimization parameter in a convolution neural network (CNN) according to an embodiment of the present invention.

2, the learning process may include an existing parameter and input data loading step (S200), a parameter optimization step (S210), an inference step (S220), a correction step (S230), and a parameter update step (S240). .

More specifically, if the parameter optimization step S210 is omitted, it is the same as a general deep learning learning process, which means that the present invention can be easily applied in a general deep learning learning environment.

The convolutional neural network optimization technique according to an embodiment of the present invention can be performed by adding only the parameter optimization step (S210) to the existing learning environment when there are existing learned parameters. In this case, if the previously learned parameter does not exist, the parameter optimization step S210 may be applied after initial learning is performed through learning by omitting the parameter optimization step S210. On the other hand, if the learning is performed by applying the parameter optimization step (S210) from the beginning, the result accuracy may be lowered compared to the same amount of operation reduction, so after loading the existing learned convolution parameters and input data into the memory (S200), The convolution parameter optimization technique proposed in the present invention is applied to the loaded existing convolution parameter (S210).

Next, the parameter to which the optimization has been applied and the input data are subjected to a convolution operation to perform inference (S220), and the existing parameters are updated through a parameter correction step (S230) (S240). The parameter correction step (S230) includes a process of obtaining backpropagation. After learning is complete, the parameters are stored by applying a convolutional parameter optimization technique. The parameter optimization step S210, the inference step S220, the parameter correction step S230, and the update step S240 may be repeatedly performed until the target number of times is reached.

3 is a diagram illustrating a detailed learning process for generating an optimization parameter of FIG. 2 according to an embodiment of the present invention.

Referring to FIG. 3, the neural network computing apparatus according to an embodiment of the present invention first determines whether a learned parameter exists (S300). Next, if the learned parameter exists, it is loaded into the memory (S302). Next, it is determined whether to apply the convolution parameter optimization (S303) of the present invention (S304). The case where the convolution parameter optimization (S303) is not applied is the same as the general deep learning learning process.

The convolutional parameter optimization step (S303) includes a size parameter generation step (S306), a sign parameter generation step (S308), determining whether to apply a ratio parameter optimization (S310), and applying a ratio parameter optimization. If it is determined that the ratio parameter is generated (S312) may be further included.

According to an embodiment of the present invention, when the convolutional parameter optimization step (S303) is completed, the neural network computing device performs inference with the parameter and input data to which the optimization has been applied through the inference step (S316), and the correction step (S318) Through the process, the existing parameters are updated with the optimized and corrected parameters (S320). At this time, it is determined whether the preset target number has been reached (S322), and if not, the optimization parameter optimization step (S303), the inference step (S316), the correction step (S318), and the parameter update step (S320) are repeatedly performed.

4 is a diagram illustrating in detail an inference process according to an embodiment of the present invention.

4, the neural network computing device according to an embodiment of the present invention loads an optimization parameter into a memory (S400), and determines whether a size parameter value exists, that is, whether the size parameter value is 0 (zero). (S410). If the value of the size parameter is 0, the amount of inference computation can be reduced as subsequent processes are omitted.

Next, if the size parameter value is not 0, the input data is loaded into the memory (S420), and the sign parameter and the input data are bitwise calculated (S430). Then, the sum operation between the input data is applied (S460), and the multiplication operation of the size parameter and the input data (S470) is performed. After the bit operation (S430), when the ratio parameter is applied (S440), a step (S470) of applying the ratio parameter correspondence constant to the size parameter with reference to the ratio parameter may be further included.

5 is a diagram illustrating an application example of a convolution parameter optimization process according to an embodiment of the present invention.

Referring to FIG. 5, a parameter optimization process according to an embodiment is largely composed of two parts: a shape change step 510 and a pruning step 520. The parameter shape conversion 510 refers to a process of reducing the amount of calculation of the existing convolution parameter and converting it into a shape that is easy to pruning. For example, as shown in FIG. 5, the existing parameter 50 is divided into a Magnitude Parameter 52 representing a magnitude and a Signed Parameter 54 representing a direction. 5 shows an example in which the existing parameter has three input channels 50-1, 50-2, and 50-3. Each of the input channels 50-1, 50-2 and 50-3 consists of a number of parameter elements. The sign parameter 54 determines the directions of elements for each channel of the existing parameter 50, and the size parameter 52 optimizes the weights with a single representative value per channel of the existing parameter 50.

The user may arbitrarily set the number of bits of each element of the size parameter 52, and may be expressed as an integer and a floating number. As the number of bits increases, the result is the same as the original convolution operation result. The number of bits of each element in the sign parameter 54 is 1 bit, which indicates only the positive (+) direction and the negative (-) direction. Various methods can be applied to separate the size parameter 52 and the sign parameter 54 from the existing parameter 50. For example, in the existing parameter 50, shape conversion may be performed through the average of the input channels of the convolutional layer and the most significant bit of each parameter.

One of the methods of converting the shape to the size parameter 52 is to use the average value of the elements (weight parameters) of the input channel. For example, i elements W _i , j input channels I _j , k output channels O _k , and size parameter M = {M ₁ , M ₂ ,… , M _j }, M _j is the same as in Equation 1.

One of the method of converting the shape to the sign parameter 54 is a method of assigning a value of 0 when the channel element (weight parameters) is greater than or equal to 0, and assigning a value of 1 when it is less than 0. That is, the sign parameter S = {S ₁ , S ₂ ,… , S _i }, S _i is equal to Equation 2.

After performing the shape conversion process, a pruning step 520 of removing unnecessary parameters is performed. In the pruning step 520, a parameter of low importance is selected and its value is set to 0 (zero) so that the operation can be omitted, thereby obtaining an effect of reducing the total amount of calculation. At this time, an optimized parameter is generated by pruning the size parameter according to the size. For example, if the predetermined size parameter value is less than a preset reference value, the size parameter of the corresponding channel is pruned.

The important point here is that this pruning process is applied only to the shape-transformed size parameter 52, not to the entire parameter of the convolutional layer. By applying the pruning 520 only to the size parameter 52, the omitted parameter affects all parameters of each input channel of the convolution layer, thereby maximizing the effect of the pruning 520.

Various methods can be applied to omit unnecessary parameters. For example, pruning is performed using the average value and distribution of each input channel and output channel of the convolution layer. Each layer has a user-determined constant, and this constant is determined according to the distribution of convolution parameters for each layer. As a reference value for omitting the value, an average value of the size parameters of the input channel or a value obtained by multiplying the average of the size parameters of the input channel and the output channel by a weight may be used. The layer constant has a value of 0 to 1. When the convolution parameter for each layer is assumed to be a continuous uniform distribution or a normal distribution, the layer constant may be 0.5 to 0.7 in order to omit the 50% size parameter. By referring to this value, consider the distribution of actual parameters to determine the layer constant.

M _LOI = {M ₁₁₁ , M ₁₁₂ ,… , M _lkj }, L = {C ₁ , C ₂ ,… , C _l ), and C is a layer constant, the size parameter pruning method to which the reference for each input channel is applied is as shown in Equation 3 below.

The standard for each output channel is applied, but the size parameter pruning method is as shown in Equation 4 below.

Hereinafter, an example of a convolution parameter optimization calculation to which Equations 1 to 4 described above are applied will be described.

It is assumed that a size parameter and a sign parameter are calculated for the existing convolution parameters with i=4, j=2, and k=2, and pruning is performed on the size parameters. At this time, O ₁ = {I ₁₁ , I ₁₂ } ={[W ₁₁₁ , W ₁₁₂ , W ₁₁₃ , W ₁₁₄ }, [W ₁₂₁ , W ₁₂₂ , W ₁₂₃ , W ₁₂₄ ]]} = {[-0.5, 1.0 , 0.5, -1.0], [1.0, -1.5, 1.0, -1.5]}, O ₂ = {I ₂₁ , I ₂₂ } ={[W ₂₁₁ , W ₂₁₂ , W ₂₁₃ , W ₂₁₄ }, [W ₂₃₁ , Assume that W ₂₂₂ , W ₂₂₃ , W ₂₂₄ ]]} = {[1.5, -2.0, -1.5, 2.0], [-2.0, -2.5, 2.0, 2.5]}.

The size parameter calculated by applying Equation 1 is M={M ₁₁ , M ₁₂ , M ₂₁ , M ₂₂ } = {0.75, 1.25, 1.75, 2.25}. That is, M ₁₁ = 1/4 (0.5 + 1.0 + 0.5 + 1.0) =0.75, M ₁₂ = 1/4 (1.0 + 1.5 + 1.0 + 1.5) =1.25, M ₂₁ = 1/4 (1.5 + 2.0 + 1.5) + 2.0) =1.75, M ₂₂ = 1/4 (2.0 + 2.5 + 2.0 + 2.5) =2.25.

Sign parameter calculated by applying Equation 2 S = {S ₁ , S ₂ , S _3, S ₄ } = {[1, 0, 0, 1], [0, 1, 0, 1], [0, 1, 1, 0], [1, 1, 0, 0]}.

An example of applying pruning to the size parameter is as follows. In the case of the existing parameter with j=2, k=2, l=1, the size parameter M = {M ₁₁₁ , M ₁₁₂ , M ₁₂₁ , M ₁₂₂ } = {0.75, 1.25, 1.75, 2.25} separated from the existing parameter If the layer constant L ₁ =0.9 is applied for each input channel of Equation 3, the pruned size parameter M'= {M ₁₁₁ , M ₁₁₂ , M ₁₂₁ , M ₁₂₂ } = {0, 1.25, 0, 2.25 }to be. For example, in _{the case of M 112} , since 1.25 <1/2(0.75+1.25)·0.9, M ₁₁₂ = 0 by Equation 3.

When applied to each input channel mill output channel of Equation 4, the pruned size parameter M'= {M ₁₁₁ , M ₁₁₂ , M ₁₂₁ , M ₁₂₂ } = {0, 0, 1.75, 2.25}. In _{the case of M 112} , since 1.25 <1/4 (0.75+1.25+1.75+2.25)·0.9, M ₁₁₂ = 0 by Equation 4. Whether to apply for each input channel or for each input and output channel may be determined according to the distribution.

6 is a diagram illustrating a comparison between a general convolution operation and a convolution operation according to an embodiment of the present invention.

Referring to FIG. 6, in a general convolution operation 60, a multiplication operation 600 and a sum operation 602 exist as much as the convolution parameter filter size. However, in the convolution operation 62 to which parameter optimization according to an embodiment is applied, the direction of data is determined through the bit operation 620, the sum operation 622 is performed by the size of the filter, and then only one multiplication operation 624 is performed. You can derive the results. For example, as shown in FIG. 6, it is confirmed that the multiplication operation 600 is 9 times in the general convolution operation 60, but the convolution operation 62 to which the parameter optimization according to an embodiment is applied is only 1 time. I can. This means that there is a product operation gain as much as the filter size.

In addition, when pruning is applied without changing the shape of the convolution parameter, a comparison operation equal to the filter size is required to determine whether or not an operation should be omitted. In the case of the general convolution operation 60 in the example of FIG. 6, a total of 9 comparison operations are required. However, in the case of the convolution operation 62 to which the parameter optimization technique of the present invention is applied, it can be determined whether or not to skip the operation with only one comparison per input channel of the convolution layer. This reduction in the amount of computation requires fewer operators than the conventional convolution operation 60. Therefore, it is possible to design the hardware efficiently.

7 is a diagram illustrating an advantage of a convolution operation after parameter optimization according to an embodiment of the present invention.

Referring to FIG. 7, output data is generated by performing a convolution operation of input data, a size parameter, and a sign parameter. If the size parameter value is 0, there is no need to perform an operation, so that the amount of calculation in a channel unit can be greatly reduced.

8 is a diagram illustrating an application example of a convolution parameter optimization process according to another embodiment of the present invention.

The goal of optimizing convolution parameters is to achieve maximum accuracy with maximum optimization (reduction of computation). Operation reduction and accuracy are generally inversely proportional to each other. In the actual application stage, there are performances (speed and accuracy) to be satisfied, but it is not easy for the user to adjust them. To solve this, the present invention proposes an optimization method using a scale parameter. This is a ratio parameter added to the optimization method described above with reference to FIG. 5.

In the shape conversion process 510, the shape of the existing parameter 50 is converted into a size parameter 52, a sign parameter 54, and a ratio parameter 58, and pruning 520 is applied only to the size parameter 52. do. During inference, a weight is applied to the size parameter by referring to the ratio parameter 58 to reduce the error of the existing operation result.

The ratio parameter 58 indicates how important the size parameter 52 is. In other words, it is a method of differentially applying a constant by comparing the importance between the input channel parameters of the convolution layer. For one element of the ratio parameter 58, the user can allocate a desired bit, through which the user can adjust the degree of optimization. The ratio parameter 58 does not mean a weight, and there are constant values referring to the ratio parameter 58. These values are assigned values that are easy to implement in hardware, so they can be implemented only with bit operations. If the value of the size parameter 52 is not immediately applied, but the weight through the ratio parameter 58 is applied to perform optimization, the similarity with the existing pre-optimization parameter increases, thereby increasing the accuracy.

9 is a diagram comparing a parameter distribution to which only a size parameter is applied and a parameter distribution to which a ratio parameter is additionally applied to explain ratio parameter optimization.

8 and 9, various methods can be applied to generate the ratio parameter 58. In the present invention, the ratio parameter 58 is generated through the distribution of the convolution parameter. Assuming that the number of ratio parameter bits is b, there are 2 ^b user-defined constants, which are referred to as ratio parameter correspondence constants. The ratio parameter correspondence constant can be arbitrarily determined by the user based on the convolution parameter distribution, and is a constant corresponding to the ratio parameter.

An example of calculating the ratio parameter 58 is as follows.

Ratio parameter number of bits = b,

λ = {λ ₁ , λ ₂ ,… , λ _t }, t = {1, 2,… , 2 ^b -1}, λ _t > λ _t+1 , and the ratio parameter SC = {SC ₁ , SC ₂ ,… If ,SC _i }, then SC _i is as in Equation 5 below.

In inference, the value of the ratio parameter correspondence constant multiplied by the magnitude parameter is used as the actual calculation value.

Hereinafter, an example of calculating ratio parameter optimization to which Equation 5 is applied will be described.

Assume that the ratio parameter is calculated for the existing convolution parameters with i=4, j=2, and k=2. Assume that the ratio parameter number of bits b = 2, t = {1, 2, 3}, and λ = {0.9, 0.7, 0.5}. At this time, O ₁ = {I ₁₁ , I ₁₂ } ={W ₁₁₁ , W ₁₁₂ , W ₁₁₃ , W ₁₁₄ , W ₁₂₁ , W ₁₂₂ , W ₁₂₃ , W ₁₂₄ } = {-0.5, 1.0, 0.5, -1.0, 1.0, -1.5, 1.0, -1.5}, O ₂ = {I ₂₁ , I ₂₂ } ={W ₂₁₁ , W ₂₁₂ , W ₂₁₃ , W ₂₁₄ , W ₂₃₁ , W ₂₂₂ , W ₂₂₃ , W ₂₂₄ } = {1.5 , -2.0, -1.5, 2.0, -2.0, -2.5, 2.0, 2.5}.

By Equation 5, SC ₁₁₁ = 2. 1/4(0.5 + 1.0 + 0.5 + 1.0) 0.7> |-0.5| > 1/4(0.5 + 1.0 + 0.5 + 1.0)·0.5. _{The ratio parameter SC i} = (SC ₁₁₁ , SC ₁₁₂ , SC ₁₁₃ , SC ₁₁₄ , SC ₁₂₁ , SC ₁₂₂ , SC ₁₂₃ , SC ₁₂₄ , SC ₂₁₁ , SC ₂₁₂ , SC ₂₁₃ , SC ₂₁₄ , SC ₂₂₁ , calculated in this way) SC ₂₂₂ , SC ₂₂₃ , SC ₂₂₄ } = {2, 0, 2, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0}.

10 is a diagram illustrating an example of a convolution operation using an optimized parameter according to an embodiment of the present invention.

As shown in the figure, the size parameter (M), the sign parameter (S), the ratio parameter (SC), and the ratio parameter correspondence constant (C) exist, and the convolution when the input data (In) is input. The operation is as shown in FIG. 10. At this time, M = {0.75}, S = {0, 0, 1, 1}, SC = {0, 0, 1, 1}, C = {1.0, 0.5}, and input data In = {1.2, -2.0 , 0.5, 3.1} as an example.

In the example with reference to FIG. 10, the ratio parameter correspondence constant (C) is set as a multiple of 2 so that the result can be derived by shift bit operation rather than general arithmetic operation. In this case, there is an advantage in hardware implementation.

11 is a diagram illustrating a configuration of a neural network computing device according to an embodiment of the present invention.

As shown in the drawing, the neural network computing device 1 according to the embodiment of the present invention may include an input unit 10, a processor 12, a memory 14, and an output unit 16.

The input unit 10 receives data including a user command, and the input data is stored in the memory 18. The processor 12 may perform a neural network operation, store the operation result in the memory 18, and output through the output unit 16. Input data and parameters may be loaded into the memory 14. The processor 12 according to an embodiment includes a parameter optimization unit 120, an inference unit 122, a correction unit 124, and an update unit 126.

The parameter optimizing unit 120 converts the shape of the existing parameter and prunes the size parameter among the shape-converted parameters. Existing parameters may be parameters that have already been learned, and are loaded into memory for computation.

The parameter optimizer 120 according to an embodiment of the present invention converts an existing parameter into a size parameter and a sign parameter, or converts the form into a size parameter, a sign parameter, and a ratio parameter. The sign parameter determines the direction of elements for each channel of the existing parameter, and the size parameter optimizes the weights with a single representative value per channel of the existing parameter. The ratio parameter indicates how important the size parameter is, and a differential weight can be applied to the size parameter by referring to the ratio parameter during inference.

In an embodiment of the present invention, the pruning is performed for only the size parameter among the shape-transformed parameters. Pruning means selecting a size parameter of low importance and making its value 0 (zero). For pruned parameters, the convolution operation can be omitted for each channel, thereby reducing the total amount of computation. The important point is that this pruning process affects all parameter elements constituting each input channel of the convolutional layer, thus maximizing the effect of pruning.

In addition, the inference unit 122 infers by performing a convolution operation on the optimized parameters and input channel data. Here, the inference unit 122 may determine whether the value of the size parameter included in the optimization parameter is 0, and if the value of the size parameter is 0, the convolution operation process may be omitted. In addition, if the ratio parameter exists, the inference unit 122 may reduce an error in the calculation result by differentially reflecting the weight to the size parameter using the ratio parameter. In an embodiment of the present invention, the correction unit 124 corrects the optimized parameter, and the update unit 126 updates the existing parameter with the corrected optimization parameter.

12 is a diagram showing a detailed configuration of the parameter optimization unit of FIG. 11 according to an embodiment of the present invention.

11 and 12, the parameter optimization unit 120 according to an embodiment of the present invention includes a size parameter conversion unit 1200, a sign parameter conversion unit 1202, a parameter pruning unit 1204, and a ratio parameter conversion. It may include a part 1206.

The sign parameter conversion unit 1202 converts an existing parameter into a sign parameter. The size parameter conversion unit 1200 converts the existing parameter into a size parameter having a single value per channel. The ratio parameter conversion unit 1206 converts the existing parameter into a ratio parameter.

In addition, the parameter pruning unit 1204 prunes the shape-converted size parameter to generate an optimized parameter. For example, if the predetermined size parameter value is less than a preset reference value, the size parameter of the corresponding channel is pruned.

The parameter pruning unit 1204 according to an embodiment of the present invention calculates and calculates a reference value using the average value and size distribution of the size parameter for each input channel or the average value and size distribution of the size parameter for each input and output channel. By making the value of the size parameter with a value less than the specified reference value to 0, the convolution operation of the corresponding channel is omitted.

In addition, when pruning using the average value and size distribution of the size parameter of the input channel constituting a predetermined layer, the parameter pruning unit 1204 multiplies the average value of the size parameter of the input channels by a constant for each layer. A reference value is calculated from the value, and if the size parameter value of a predetermined input channel is smaller than the reference value, the size parameter value of the corresponding channel may be changed to 0 and pruned.

In addition, the parameter pruning unit 1204 is an average value of the size parameters of the input channels and output channels when pruning by using the average value and size distribution of the size parameters of the input and output channels constituting a predetermined layer. A reference value is calculated by multiplying by a constant for each layer, and if the size parameter value of a predetermined input channel is smaller than the reference value, the size parameter value of the corresponding channel may be changed to 0 and pruned.

So far, the present invention has been looked at around the embodiments. Those of ordinary skill in the art to which the present invention pertains will be able to understand that the present invention may be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from a descriptive point of view rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the above description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

10: input
12: processor
14: memory
16: output
120: parameter optimization unit
122: reasoning unit
124: correction unit
126: update unit
1200: size parameter conversion unit
1202: sign parameter conversion unit
1204: parameter pruning unit
1206: ratio parameter conversion unit

Claims (18)

Converting an existing parameter of the neural network into a code parameter and a size parameter having a single value per channel using a code parameter conversion unit and a size parameter conversion unit constituting the parameter optimization unit; And
Generating an optimized parameter by pruning only a size parameter from among the shape-transformed parameters using a parameter pruning unit constituting a parameter optimization unit, and
The shape transformation and the pruning are performed by targeting a parameter of a convolution layer used for a neural network convolution operation.
The method of claim 1,
The sign parameter determines the direction of elements for each channel of the existing parameter,
The size parameter is a neural network parameter optimization method, characterized in that the weights are optimized with a single representative value per channel of the existing parameter.
The method of claim 1, wherein the step of converting the shape,
A method for optimizing a neural network parameter, comprising generating a size parameter by calculating an average of the absolute values of elements for each channel of an existing parameter.
The method of claim 1, wherein the step of converting the shape,
If a predetermined element constituting the channel of the existing parameter is less than 0, the element value of the corresponding sign parameter is expressed as 0, and if it is greater than or equal to 0, the element value of the corresponding sign parameter is expressed as 1 to generate a sign parameter. Neural network parameter optimization method using.
The method of claim 1, wherein the pruning step,
Calculate the reference value using the average value and size distribution of the size parameter for each input channel or the average value and size distribution of the size parameter for each input and output channel, and make the size parameter value with a value less than the calculated reference value to 0. A neural network parameter optimization method, characterized in that omitting a convolution operation of a corresponding channel.
The method of claim 1, wherein the neural network parameter optimization method,
Converting an existing parameter of the neural network into a ratio parameter using a ratio parameter conversion unit constituting the parameter optimization unit;
Neural network parameter optimization method, characterized in that it further comprises.
The method of claim 6, wherein the step of converting the shape to the ratio parameter,
Variably allocating bits of the ratio parameter; And
Quantizing the range and weight of the value of the ratio parameter element by user selection;
Neural network parameter optimization method comprising a.
Loading existing parameters and input channel data of a neural network into a memory using a processor constituting a neural network computing device;
Code parameter change constituting the parameter optimization unit included in the processor
The size of the previously learned parameters of the neural network is converted into a code parameter and a size parameter having a single value per channel using the affected part and the size parameter conversion unit, and the size of the shape converted parameters using a parameter pruning unit constituting the parameter optimization unit. Generating optimized parameters by pruning only the parameters;
Inferring by performing a convolution operation on the optimized parameter and input channel data using an inference unit included in the processor;
Correcting the optimized parameter using a correction unit included in the processor; And
Including; updating an existing parameter to a corrected optimization parameter using an update unit included in the processor,
The shape transformation and the pruning are performed by targeting a parameter of a convolution layer used in a neural network convolution operation.
The method of claim 8, wherein the neural network operation method comprises:
Determining whether the learned parameter exists;
Generating an initial parameter through parameter initialization if there is no learned parameter;
Generating optimized parameters for the initial parameters; And
Loading an existing parameter if there is the learned parameter;
Neural network calculation method, characterized in that it further comprises.
The method of claim 8, wherein the step of inferring by convolutional calculation of optimized parameters and input channel data using the inference unit,
Loading the optimized parameter into a memory;
Determining whether a value of a size parameter included in the loaded optimized parameter is 0; And
If the value of the size parameter is 0, omitting the convolution operation process;
Neural network operation method comprising a.
The method of claim 8, wherein the step of inferring by convolutional calculation of optimized parameters and input channel data using the inference unit,
If the value of the size parameter is not 0, determining a direction of data by bitwise operation of a sign parameter and input channel data;
Calculating a sum of input channel data by the size of the convolution parameter filter; And
Performing a single multiplication operation on the size parameter and input channel data;
Neural network operation method comprising a.
The method of claim 10, wherein the step of inferring by convolving the optimized parameter and input channel data comprises:
Reducing the error of the calculation result by differentially reflecting the weight to the size parameter using the ratio parameter if there is a ratio parameter;
Neural network calculation method, characterized in that it further comprises.
A sign parameter conversion unit converting the shape of an existing parameter into a sign parameter;
A size parameter conversion unit converting the shape of the existing parameter into a size parameter having a single value per channel; And
Includes; a parameter pruning unit for generating an optimized parameter by pruning only a size parameter from among the shape-converted parameters,
The shape transformation and the pruning are performed by targeting a parameter of a convolutional layer used in a neural network convolution operation.
The method of claim 13, wherein the parameter pruning unit
Calculate the reference value using the average value and size distribution of the size parameter for each input channel or the average value and size distribution of the size parameter for each input and output channel, and make the size parameter value with a value less than the calculated reference value to 0. Neural network computing device, characterized in that to omit the convolution operation of the corresponding channel.
The method of claim 13, wherein the neural network computing device
A ratio parameter conversion unit converting the shape of an existing parameter into a ratio parameter;
Neural network computing device, characterized in that it further comprises.
The method of claim 13, wherein the neural network computing device
An inference unit for inferring by convolutional operation of optimized parameters and input channel data;
Neural network computing device, characterized in that it further comprises.
The method of claim 16, wherein the reasoning unit
A neural network computing device, comprising: determining whether a value of the size parameter included in the optimization parameter is 0, and if the value of the size parameter is 0, skipping a convolution operation process.
The method of claim 16, wherein the reasoning unit
If there is a ratio parameter, the error of the calculation result is reduced by differentially reflecting the weight to the size parameter using the ratio parameter.

Images