KR102271254B1 - Device and method for enhancing deduction speed of binarized convolutional neural network - Google Patents

Abstract

An apparatus and method for improving the inference speed of a binarized convolutional neural network are disclosed, and as a method for improving the inference speed of a binarized convolutional neural network according to an embodiment of the present application, (a) binarizing the input data to the convolutional layer to obtain the binarized input data generating, binarizing the parameters of the convolutional layer to generate a binarized parameter, and performing a binarized convolution operation between the binarized input data and the doubling parameter, (b) multiplying the binarized parameter by performing a doubling generating a binarization parameter, performing a multiplication operation by multiplying the binarized convolutional operation result by the multiplication parameter, and performing batch normalization on the multiplication operation result, and (c) binarizing the batch normalized result value and calculating a binarization operation result value by doing so, and (d) performing inference through maximum value pooling using the binarization operation result value as an input element.

Description

DEVICE AND METHOD FOR ENHANCED DEDUCTION SPEED OF BINARIZED CONVOLUTIONAL NEURAL NETWORK

The present application relates to an apparatus and method for improving inference speed of a binarized convolutional neural network.

A binarized convolutional neural network (BCNN) is a type of convolutional neural network in which each element of a weight and feature is represented by a single bit. The binarized convolutional neural network can reduce the computation and memory requirements of CNNs.

Meanwhile, recently, BCNN, which can reduce the difference in accuracy with existing CNNs while implementing an efficient neural network due to binarization, is being studied. For example, studies on BinaryNet that guarantees the accuracy that is acceptable through classification of small data sets even if weights and features are binarized, and studies on improving classification accuracy by binarizing weights and features with multiple binary bases have been conducted. .

However, research on technology to improve inference speed through binarized convolutional neural networks is insufficient, and the level of development is not adequate in relation to the technology for neural network models that can be applied to various neural networks including BCNN to improve inference speed. it is not the case.

The technology that is the background of the present application is disclosed in Korean Patent Application Laid-Open No. 10-2018-0138441.

The present application is to solve the problems of the prior art described above, and to provide an apparatus and method for improving the inference speed of a binarized convolutional neural network that can reduce the inference time while maintaining the image analysis accuracy of the existing binarized convolutional neural network do it with

An object of the present application is to provide an apparatus and method for improving the inference speed of a binarized convolutional neural network, which can improve the inference speed by changing the process of the binarized convolutional neural network.

However, the technical problems to be achieved by the embodiment of the present application are not limited to the technical problems as described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, the method for improving the inference speed of a binarized convolutional neural network according to an embodiment of the present application includes (a) binarizing input data for a convolutional layer to generate binarized input data, and A step of binarizing a parameter of a convolutional layer to generate a binarized parameter, and performing a binarized convolution operation between the binarized input data and the binarized parameter, (b) performing doubling on the binarized parameter to generate a binarized parameter, , performing a doubling operation by multiplying the binarized convolutional operation result by the doubling parameter, and performing batch normalization on the doubling operation result, (c) binarizing the batch normalized result value to a binarization operation result value and (d) performing inference through maximal pooling using the result of the binarization operation as an input element.

According to an embodiment of the present application, the method for improving the inference speed of a binarized convolutional neural network further comprises, before the step (a), performing learning of the binarized convolutional neural network for the inference, and performing the learning. The step may include performing the batch normalization before the maximum value pooling, and binarizing the batch normalized result value after the maximum value pooling.

According to an embodiment of the present application, the input element corresponding to the result of the binarization operation has a value of -1 or +1, and in step (d), +1 of the elements in the pooling window by the result of the binarization operation is If included, the output value of the corresponding pooling window may be determined to be +1.

According to an embodiment of the present application, in step (d), if a +1 value is confirmed during the sequential maximum pooling calculation for the input elements in the pooling window, the calculation for the remaining input elements is omitted and the maximum value for the pooling window is It can be determined that the output value of value pooling is +1.

According to an embodiment of the present application, the pooling window may have k2 input elements as a K x K window.

According to an embodiment of the present application, the binarized input data can be calculated by Equation 1 below,

[Equation 1]

Here, sign(·) is an operation that returns the sign of the input, and A ^b x,y,z represents the (x, y, z) position component ^{of A b} , the input of the binarized convolution.

According to an embodiment of the present application, the binarized convolution operation can be calculated by Equation 2 below,

[Equation 2]

Here, W ^b m,x,y,z represents the (x, y, z) position component of the ^{m-th filter W b m.}

According to an embodiment of the present application, the doubling coefficient may be calculated by Equation 3 below.

[Equation 3]

According to an embodiment of the present application, the batch normalization can be calculated by Equation 4 below,

[Equation 4]

는 각각 m 번째 채널의 평균, 분산, 스케일링 가중치, 바이어스 가중치를 나타낸다.where X _m represents the input of the batch normalization process,

denotes the average, variance, scaling weight, and bias weight of the m-th channel, respectively.

The apparatus for improving inference speed of a binarized convolutional neural network according to an embodiment of the present application generates binarized input data by binarizing input data for a convolutional layer, and binarizing a parameter of the convolutional layer to generate a binarized parameter, and the binarized input a convolution operation unit for performing a binarized convolution operation between data and the binarization parameter; A normalization unit that performs doubling on the binarization parameter to generate a doubling parameter, multiplies the binarized convolutional operation result by the doubling parameter, performs a doubling operation, and performs batch normalization on the doubling operation result. ; a binarization operation unit for binarizing the batch normalized result value to calculate a binarization result value; and an inference unit for performing inference through maximum value pooling using the result of the binarization operation as an input element.

According to an embodiment of the present application, the apparatus for improving the inference speed of a binarized convolutional neural network further includes a learning unit configured to perform learning of the binarized convolutional neural network for the inference, wherein the learning unit is configured to perform the batch normalization prior to the maximum pooling. , and after pooling the maximum value, the batch normalized result value may be binarized.

According to an embodiment of the present application, when the input element corresponding to the result of the binarization operation has a value of -1 or +1, and the reasoning unit includes +1 among the elements in the pooling window by the result of the binarization operation, , an output value of the corresponding pooling window may be determined to be +1.

According to an embodiment of the present application, the inference unit, if a +1 value is confirmed during the sequential maximum pooling calculation for the input elements within the pooling window, omit the calculation for the remaining input elements and pool the maximum value for the pooling window It can be determined that the output value of +1 is +1.

According to an embodiment of the present application, the pooling window may have k2 input elements as a K x K window.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present application. In addition to the exemplary embodiments described above, additional embodiments may exist in the drawings and detailed description.

According to the above-described problem solving means of the present application, it is possible to provide an apparatus and method for improving the inference speed of a binarized convolutional neural network, which can reduce the inference time while maintaining the image analysis accuracy of the existing binarized convolutional neural network.

The present application provides an apparatus and method for improving the inference speed of a binarized convolutional neural network that can improve the inference speed by changing the process of the binarized convolutional neural network to solve the problems of the prior art described above.

1 is a diagram illustrating a process of a conventional binarized convolution.
2 is a diagram illustrating an example of maximum pooling.
3 is a diagram illustrating the configuration of an apparatus for improving inference speed of a binarized convolutional neural network according to an embodiment of the present application.
4 is a diagram illustrating a process for learning a binarized convolutional neural network according to an embodiment of the present application.
5 is a diagram illustrating a process for inference of a binarized convolutional neural network according to an embodiment of the present application.
6 is a diagram comparing analysis performance by an apparatus for improving inference speed of a binarized convolutional neural network according to an embodiment of the present application.
7 is a diagram illustrating a flow of a method for improving inference speed of a binarized convolutional neural network according to an embodiment of the present application.

Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present application pertains can easily implement them. However, the present application may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present application in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout this specification, when a part is said to be "connected" with another part, it includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element interposed therebetween. do.

Throughout this specification, when it is said that a member is positioned "on", "on", "on", "under", "under", or "under" another member, this means that a member is positioned on the other member. It includes not only the case where they are in contact, but also the case where another member exists between two members.

Throughout this specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

1 is a diagram illustrating a process of a conventional binarized convolution.

Inference through a conventional binary convolutional neural network (BCNN) (hereinafter, conventional BCNN) is associated with a plurality of multiply-and-accumulate (MAC) operations. On the other hand, in the conventional reasoning through BCNN, the MAC operation can be equally realized by a simple XNOR-bitcount operation, so that rapid inference can be made in the existing CPU-based system. Therefore, the conventional BCNN has been studied to implement artificial intelligence in a computation limited device (Computation Limited Device) that is not equipped with a GPU or dedicated hardware.

Referring to FIG. 1, the conventional BCNN studied to reduce the accuracy difference with the existing convolutional neural network while securing efficiency through binarization is binarized convolution (S11), maximum value pooling (S12), multiplication and batch normalization. It can be arranged in the order of (S13) and binarization (S14), and when each process is expressed as a block, BCNN can be expressed as a set of blocks in which each block is serially connected. An apparatus for improving the inference speed of a binarized convolutional neural network, which will be described later, may improve the inference speed while maintaining the image analysis accuracy of the binarized convolutional neural network by changing the block structure. First, looking at the conventional BCNN, the binarized convolution S11 may binarize input data through binarization of a previous layer, and parameters of the convolution layer may also be binarized. The convolution operation between the input of the binarized convolution and the binarization parameter can be performed by bitwise XNOR and accumulation.

2 is a diagram illustrating an example of maximum pooling.

After the convolution operation, maximum pooling ( S12 ) may be performed. Maximum pooling means reducing the size of input data by selecting a maximum value for values within the pooling window 10 formed by the local area KXK. Therefore, K ² input values need to be calculated. Exemplarily, FIG. 2 shows the maximum pooling result when the stride value is 2 and the area is 3×3. After pooling the maximum value, multiplication and batch normalization (S13) may be performed. Since the fold normalization and batch normalization can be efficiently implemented as an affine transformation, the fold normalization and batch normalization can be merged into one block. Thereafter, the output of the layer may be calculated through binarization ( S14 ) of the batch normalized result value.

In the conventional BCNN, in order to calculate one component value of the output of maximum pooling, all k2 input values have to be calculated, which is inefficient. Accordingly, the apparatus for improving the inference speed of a binarized convolutional neural network according to an embodiment of the present application may change the structure of the BCNN so that the input element of the maximum value pooling becomes a binary number, thereby terminating the maximum value pooling operation early.

3 is a diagram illustrating the configuration of an apparatus for improving inference speed of a binarized convolutional neural network according to an embodiment of the present application, and FIG. 4 is a diagram illustrating a process for learning a binarized convolutional neural network according to an embodiment of the present application to be.

Referring to FIG. 3 , the apparatus 100 for improving the inference speed of a binarized convolutional neural network includes a learning unit 110 , a convolution operation unit 120 , a normalization unit 130 , a binarization operation unit 140 , and an inference unit 150 . can do. Referring to FIG. 4 , the convolution operator 120 may generate binarized input data by binarizing input data for a convolutional layer. The binarized input data may be calculated through Equation (1).

[Equation 1]

Here, sign(·) is an operation that returns the sign of the input, and A ^b x,y,z represents the (x, y, z) position component ^{of A b} , which is the input of the binarized convolution. When input data is binarized, analysis performance may be degraded due to data loss due to binarization, but since convolution is replaced with binarized convolution, memory requirements for parameter storage and computational complexity may be reduced.

Also, the convolution operation unit 120 may generate a binarization parameter by binarizing the parameter of the convolution layer. The convolution operator 120 may generate a binarization parameter through Equation (2).

[Equation 2]

Here, (W ₁ ) ^b _m,x,y,z represents the binarized parameter of the first layer, and (W ₁ ) _m,x,y,z represents the parameter before binarization of the first layer.

Also, the convolution operation unit 120 may perform a binarized convolution operation between the binarized input data and a binarization parameter (eg, a binarization parameter in which a weight is considered) ( S41 ). The binarized convolution operation may be calculated through Equation (3).

[Equation 3]

Here, W ^b m,x,y,z represents the (x, y, z) position component of the ^{m-th filter W b m.} Also, the binarization parameter may mean a parameter in which the weight of the corresponding convolutional layer is considered.

The normalizer 130 may generate a multiplied parameter by performing multiplication on the binarization parameter. The multiplication operation may be performed to reduce the decrease in analysis performance due to the binarization process of the weights. In other words, the doubling operation refers to an operation capable of approximating a binarized parameter to a parameter before binarization by introducing a doubling coefficient e. The multiplication coefficient may be calculated by Equation (4). The multiplication coefficient may be defined as the average of the absolute values of the corresponding channel parameters for each channel, and may be generated as many as the number of channels of the corresponding layer.

[Equation 4]

Here, e _m represents the multiplication factor.

Also, the normalizer 130 may perform a multiplication operation by multiplying the binarized convolutional operation result by the multiplication parameter. The normalization unit 130 may approximate the binarized parameter value to the parameter value prior to binarization as the multiplication operation is performed as shown in Equation (5).

[Equation 5]

That is, doubling is an operation that multiplies each component of the binarized convolution result O by e, and requires w·h·M floating-point multiplication operations. consumes

The normalization unit 130 may perform a multiplication operation by multiplying the binarized convolutional operation result by the doubling parameter, and may perform batch normalization on the doubling operation result ( S42 ). Batch normalization can be calculated by Equation (6). formula

[Equation 6]

는 각각 m 번째 채널의 평균, 분산, 스케일링 가중치, 바이어스 가중치를 나타낸다. 또한, 상기 m 번째 채널의 평균, 분산, 스케일링 가중치, 바이어스 가중치 각각은 필터 수, 즉 입력의 채널 수 만큼의 개수를 가질 수 있다. 보편적으로 2ⅹ2, 혹은 3ⅹ3의 국부 영역 내의 값들의 평균값 또는 최댓값을 선택하여 입력의 크기를 (w/2)ⅹ(h/2)ⅹM 으로 줄인다.where X _m represents the input of the batch normalization process,

denotes the average, variance, scaling weight, and bias weight of the m-th channel, respectively. Also, each of the average, variance, scaling weight, and bias weight of the m-th channel may have the same number of filters, that is, as many as the number of input channels. In general, the size of the input is reduced to (w/2)×(h/2)×M by selecting the average or maximum value of the values in the local region of 2×2 or 3×3.

A learning process may precede inference through the apparatus 100 for improving the inference speed of a binarized convolutional neural network. The learning refers to the process of calculating the optimal value while updating the parameter (weighted parameter) value of the binarized convolutional neural network through input data (learning data) for learning, and 'learning performed by the learning unit 110' step'. On the other hand, inference through the apparatus 100 for improving the inference speed of the binarized convolutional neural network described above is a process of performing convolution operation, multiplication and batch normalization, binarization, and maximum pooling operation of the binarized convolutional neural network using the learned parameters. It can be called an inference step that includes That is, both the learning unit 110 and the inference unit 150 perform convolutional operations, multiples and batch normalization, binarization and maximum pooling operations, but the learning unit 110 learns to find the optimal value of the parameter through learning. process, and the reasoning unit 150 may be said to perform an inference process that actually uses the parameters learned by the learning unit 110 .

은 항상 양의 값을 유지하며 학습시키기 때문에, 배치 정규화 연산은 단조 증가 함수이다. 또한, 배수화 계수 e_m은 정의상 항시 양의 값을 가지며, 이진화 함수 또한 단조 증가 함수이기 때문에 프로세스 순서의 변화는 최종 출력값의 영향을 미치지 않을 수 있다.Inference through the apparatus 100 for improving the inference speed of a binarized convolutional neural network is derived through binarization and maximum pooling after doubling and batch normalization (S42), but the learning of the binarized convolutional neural network is as shown in FIG. Maximum pooling (S43) may precede binarization (S44). Specifically, unlike the conventional BCNN in which multiples and batch normalization (S13) are performed after maximum pooling (S12), the learning unit 110 performs batch normalization (S42) before maximum pooling (S43) and , after the maximum value pooling (S43), the batch normalized result value may be binarized (S44). This can be considered as applying a distributively increasing binarization function to each element of the pooling window in a distributed manner, so that changes in the process order do not affect the binarized convolutional neural network. Specifically, the scaling weights of batch normalization

Batch regularization is a monotonically increasing function, since it always learns while maintaining a positive value. In addition, the multiplication coefficient e _m always has a positive value by definition, and since the binarization function is also a monotonically increasing function, a change in the process order may not affect the final output value.

5 is a diagram illustrating a process for inference of a binarized convolutional neural network according to an embodiment of the present application.

In learning through the binarized convolutional neural network, the process of performing the binarization (S45) before the maximum value pooling (S46) according to the inference process of the binarized convolutional neural network, which will be described with reference to FIG. 5, cannot be applied. Specifically, when 'learning' with the structure of 'convolution-batch normalization-binarization-maximum pooling', there may be a problem that there are two or more maximum elements and the maximum element selection for gradient backpropagation is non-deterministic ( For example, if there are two +1s in a kxk window). Accordingly, a method in which the maximum pooling input has only a value of -1 or +1 by performing binarization immediately before maximum pooling, such as the inference process of a binarized convolutional neural network, cannot be used in the 'learning' stage. That is, in the learning process, if binarization is performed immediately before maximum pooling, the maximum pooling input has only a value of -1 or +1, so it is highly likely that two or more maximum elements within the pooling window exist, Accordingly, since one maximum factor required for backpropagating the gradient is selected ambiguously, there is a possibility that an inappropriate gradient is selected, so the process for inference cannot be applied to the learning process. Therefore, in the learning process, as shown in FIG. 4 , the process in which binarization ( S44 ) is performed after maximum pooling ( S43 ) can be followed, whereas in the inference process, the maximum value after binarization ( S45 ) as shown in FIG. 5 . A process in which the pulling ( S46 ) is performed may be followed. Unlike the learning process, according to the inference process in which the maximum value pooling (S46) is performed after the binarization (S45), since the elements in the pooling window are binarized, the maximum value pooling can be performed quickly, and as a result, the inference is also quickly performed. can be derived.

Specifically, the binarization operation unit 140 may binarize the batch normalized result value to calculate the binarization operation result value. The reasoning unit 150 may perform inference through maximum value pooling using the result of the binarization operation as an input element. The maximum value pooling performed by the inference unit 150 may be performed on the result value of the operation binarized by the binarization operation unit 140 . That is, the input element corresponding to the result value of the binarization operation may have a value of -1 or +1. In addition, the pooling window of the maximum pooling performed by the inference unit 150 may have ^{K 2 input elements as a K x K window.} As described above, since the K ² input elements of the maximum pooling are binarized values, the reasoning unit 150 calculates the output value of the corresponding pooling window when +1 of the elements in the pooling window according to the binarization operation result is included. It can be decided by +1. Specifically, the reasoning unit 150 omits the calculation of the remaining input elements when a +1 value is confirmed during the sequential maximum pooling calculation for the input elements within the pooling window, and adds the output value of the maximum pooling for the pooling window to + 1 can be determined. That is, the maximum value pooling performed by the inference unit 150 is sequential for ^{K 2} elements including -1 and +1 in the pooling window, unlike the conventional BCNN in which ^{all K 2 elements in the pooling window are calculated.} If the +1 value is confirmed during the maximum pooling calculation, the redundant calculation required to calculate the remaining elements can be omitted, so that the maximum pooling can be terminated early, thereby enabling rapid inference. In addition, since maximum value pooling is performed on the binarized input element, rapid inference may be possible even if the size of the K x K pooling window increases.

The ratio of -1 to +1 within the pooling window affects the speed of inference. The process for inference of a binarized convolutional neural network according to an embodiment of the present application builds an algorithm such that the operations of multiplication and batch normalization (S42) and binarization (S45) are included in the maximum pooling operation, so -1 in the pooling window The higher the ratio, the longer the inference time may be. Specifically, in the case of an inference process according to 'convolution-batch normalization-binarization-maximum pooling', the number of operations may increase because batch normalization (including multiplication) and binarization are included in the maximum pooling calculation. However, in this case, if the ratio of -1 in the pooling window is increased, the inference time may be rather long. That is, when the convolution operation time reduced by applying the process for inference of a binarized convolutional neural network is smaller than the operation time of batch normalization and binarization in which the number of operations is increased, the inference time may rather increase. Exemplarily, if the size of the input is 8 x 8, the size of the pooling window is 3 x 3, and the stride is 2, the size of the output is 4 x 4, and for calculations for each element of the output ( Multiplexing, batch normalization, and binarization are performed 3 x 3 - skip) times (here, skip is an operation that is omitted for the remaining input elements when a value of +1 is confirmed during the sequential maximum pooling calculation for input elements within the pooling window) can mean the number of ). That is, in order to calculate all 4 x 4 output, (4 x 4 x 3 x3 - skip total amount) times the doubling, batch normalization and binarization are performed. On the other hand, in the case of a conventional BCNN that learns and infers through the processes of convolution operation, maximum pooling, multiplication, batch normalization, and binarization, since maximum pooling is performed before doubling, batch normalization, and binarization, doubling, batch Normalization and binarization can only be performed 4 x 4 times. However, even if the computation time in the multiplication, batch normalization, and binarization operation is increased compared to the conventional BCNN as described above according to a change in the process, the convolution operation takes a relatively long time compared to the multiplication, batch normalization, and binarization operation. This skip can be shortened as much as the total amount of skips, and as a result, the reasoning time can be significantly shortened compared to the prior art. The effect of improving the speed of reasoning according to the change of such a process is to be verified through an experiment to be described later.

Meanwhile, in the learning process of the binarized convolutional neural network, as shown in FIG. 4 , maximum pooling (S43) and binarization (S44) are performed after multiplication and batch normalization (S42) in order to improve the learning speed. Since the multiplication and batch normalization ( S42 ) have a different order from that of the conventional BCNN, the results of the multiplication and batch normalization may have different values from the weights in the learning process of the conventional BCNN. Because batch normalization equalizes the distribution of results, it is unlikely that the ratio of -1 within the pooling window will be relatively high.

It is intended to verify the effectiveness of the binarized convolutional neural network through an experiment comparing the inference time by the inference speed improvement apparatus 100 and the accuracy of the classification task. ImageNet and CIFAR-10, which are color image data sets used for experiments, consider large and small data sets as classification targets, respectively. CIFAR10 is a small 32x32 color image data set, including 50,000 training sets and 10,000 test sets for 10 object classifications. ImageNet is a large color image dataset with an average size of 469×387, including 1.2 million training sets and 50,000 test sets for 1000 object classifications. In the conventional BCNN model, Binarized ConvNet (for CIFAR-10) and Binarized AlexNet (for ImageNet) are used. The Binarized ConvNet is trained over 250 epochs with a batch size of 512 based on an SGD optimizer with an initial training rate of 0.1 and momentum of 0.9. Binarized AlexNet uses the same optimization tools and batch size settings as the Binarized ConvNet above, and is trained over 40 epex. Also, any pre-processing or data augmentation techniques that may affect classification accuracy are not used.

6 is a diagram comparing analysis performance by an apparatus for improving inference speed of a binarized convolutional neural network according to an embodiment of the present application.

The classification experiment comparing the inference speed improvement apparatus 100 of the conventional BCNN and the binarized convolutional neural network used C language, and the runtime environment in which the inference time is measured utilizes an 800 MHz ARM Cortex-A9 processor and 1 GB SDRAM. Fig. 6(a) shows the average inference time in milliseconds, and Fig. 6(b) shows the highest classification accuracy. Referring to FIG. 6 , Binarized ConvNet and Binarized AlexNet for image classification of ImageNet of the conventional BCNN and binarized convolutional neural network inference speed improvement apparatus 100 that do not take into account redundant operation omission of the pooling window when pooling the maximum value. As a result of comparing the speed, it can be confirmed that the Binarized ConvNet decreases by 27.5% and the Binarized AlexNet decreases by 42.3%, which is faster than the conventional BCNN compared to the inference speed improvement apparatus 100 of the binarized convolutional neural network. The apparatus 100 for improving the inference speed of a binarized convolutional neural network can effectively reduce the speed of inference while maintaining the image analysis accuracy of the conventional BCNN. In addition, when the upper limit of the maximum pooling is set in advance, the inference speed may be improved by applying the inference speed improving apparatus 100 of the binarized convolutional neural network to various neural networks, not limited to BCNN.

7 is a diagram illustrating a flow of a method for improving inference speed of a binarized convolutional neural network according to an embodiment of the present application.

The method for improving the inference speed of the binarized convolutional neural network in the embodiment of the present application shown in FIG. 7 is performed by the apparatus for improving the inference speed of the binarized convolutional neural network according to the embodiment of the present application described above with reference to FIGS. 3 to 6 . can be Therefore, even if omitted below, the description of the apparatus for improving the inference speed of a binarized convolutional neural network according to an embodiment of the present application through FIGS. 3 to 6 may be equally applied to FIG. 7 .

Referring to FIG. 7 , in operation S710 , the convolution operation unit 120 may binarize input data for a convolution layer to generate binarized input data. When input data is binarized, analysis performance may be degraded due to data loss due to binarization, but since convolution is replaced with binarized convolution, memory requirements for parameter storage and computational complexity may be reduced. Also, the convolution operation unit 120 may generate a binarization parameter by binarizing the parameter of the convolution layer. Also, the convolution operation unit 120 may perform a binarized convolution operation between the binarized input data and the multiplier parameter.

In step S720, the normalization unit 130 may generate a multiplier parameter by performing multiplication on the binarization parameter. The multiplication operation may be performed to reduce the decrease in analysis performance due to the binarization process of the weights. In other words, the doubling operation refers to an operation capable of approximating a binarized parameter to a parameter before binarization by introducing a doubling coefficient e. Also, the normalizer 130 may perform a multiplication operation by multiplying the binarized convolutional operation result by the multiplication parameter. In addition, the normalization unit 130 may perform a multiplication operation by multiplying the binarized convolutional operation result by the multiplication parameter, and may perform batch normalization on the multiplication operation result.

In step S730, the binarization operation unit 140 may binarize the batch normalized result value to calculate a binarization operation result value.

In step S740 , the inference unit 150 may perform inference through maximum value pooling using the result of the binarization operation as an input element. The maximum value pooling performed by the inference unit 150 may be performed on the result value of the operation binarized by the binarization operation unit 140 . That is, the input element corresponding to the result value of the binarization operation may have a value of -1 or +1. In addition, the pooling window of the maximum pooling performed by the inference unit 150 may have K2 input elements as a K x K window. As described above, since the K2 input elements of the maximum pooling are binarized values, the reasoning unit 150 increases the output value of the corresponding pooling window when +1 among the elements in the pooling window according to the binarization operation result is included. 1 can be determined. Specifically, the reasoning unit 150 omits the calculation of the remaining input elements when a +1 value is confirmed during the sequential maximum pooling calculation for the input elements within the pooling window, and adds the output value of the maximum pooling for the pooling window to + 1 can be determined.

According to an embodiment of the present application, before the step S710, the learning unit 110 may learn the binarized convolutional neural network for the inference. The learner 110 may perform batch normalization before the maximum value pooling ( S43 ), and may binarize the batch normalized result value after the maximum value pooling.

The improved binarization method of the first layer of the convolutional neural network according to an embodiment of the present application may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and carry out program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The above description of the present application is for illustration, and those of ordinary skill in the art to which the present application pertains will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present application.

10: pooling window
100: Inference speedup device of binarized convolutional neural network
110: study department
120: convolution operation unit
130: normalization unit
140: binary operation unit
150: reasoning unit

Claims (15)

A method for improving the inference speed of a binarized convolutional neural network performed by a device for speeding up inference of a binarized convolutional neural network,
(a) generating binarized input data by binarizing the input data for the convolutional layer, generating binarized parameters by binarizing the parameters of the convolutional layer, and performing a binarized convolution operation between the binarized input data and the binarized parameters ;
(b) perform doubling on the binarization parameter to generate a doubling parameter, multiply the binarized convolutional operation result by the doubling parameter to perform a doubling operation, and perform batch normalization on the doubling operation result to do;
(c) binarizing the batch normalized result value to calculate a binarization result value; and
(d) performing inference through maximum pooling using the result of the binarization operation as an input element,
The input element corresponding to the result of the binarization operation has a value of -1 or +1,
In step (d),
If +1 is included among the elements in the pooling window according to the binarization result, the output value of the corresponding pooling window is determined as +1,
In step (d), if a value of +1 is confirmed during the sequential maximum pooling calculation for the input elements within the pooling window, the calculation for the remaining input elements is omitted and the output value of the maximum pooling for the pooling window is +1 A method for improving the inference speed of a binarized convolutional neural network to determine According to claim 1,
Prior to the step (a), further comprising the step of performing the learning of the binarized convolutional neural network for the inference,
In the performing of the learning, the batch normalization is performed before the maximum value pooling, and the batch normalized result value is binarized after the maximum value pooling. How to improve the inference speed of a binarized convolutional neural network. delete delete According to claim 1,
The pooling window is a K x K window and has k ² input elements. The method for improving inference speed of a binarized convolutional neural network. According to claim 1,
The binarized input data is calculated by Equation 1 below,
[Equation 1]

Here, sign(·) is an operation that returns the sign of the input, and A ^b x,y,z is the (x, y, z) position component of ^{A b , the input of the binarized convolutional binarized convolutional neural network.} How to speed up your reasoning. According to claim 1,
The binarized convolution operation is calculated by Equation 2 below,
[Equation 2]

Here, W ^b m,x,y,z is ^{the (x, y, z) position component of the mth filter W b} m, the method for improving the inference speed of a binarized convolutional neural network. According to claim 1,
The method for improving the inference speed of a binarized convolutional neural network, wherein the multiplication parameter is calculated by Equation 3 below.
[Equation 3]

According to claim 1,
The batch normalization is calculated by Equation 4 below,
[Equation 4]

where X _m represents the input of the batch normalization process,

are the mean, variance, scaling weight, and bias weight of the m-th channel, respectively. A method for improving inference speed of a binarized convolutional neural network. An inference speedup device for a binarized convolutional neural network, comprising:
a convolution operator for binarizing input data for a convolutional layer to generate binarized input data, generating a binarized parameter by binarizing a parameter of the convolutional layer, and performing a binarized convolution operation between the binarized input data and the binarized parameter;
A normalization unit that performs doubling on the binarization parameter to generate a doubling parameter, multiplies the binarized convolutional operation result by the doubling parameter, performs a doubling operation, and performs batch normalization on the doubling operation result. ;
a binarization operation unit for binarizing the batch normalized result value to calculate a binarization result value; and
and a reasoning unit for performing inference through maximal pooling using the result of the binarization operation as an input element,
The input element corresponding to the result of the binarization operation has a value of -1 or +1,
The reasoning unit is
When +1 is included among the elements in the pooling window according to the binarization result, the output value of the corresponding pooling window is determined as +1, but during the sequential maximum pooling calculation for the input elements in the pooling window, the +1 value is If it is confirmed, the calculation for the remaining input elements is omitted and the output value of the maximum pooling for the pooling window is determined as +1, the apparatus for improving the inference speed of a binarized convolutional neural network. 11. The method of claim 10,
Further comprising a learning unit that performs learning of the binarized convolutional neural network for the inference,
The learning unit performs the batch normalization before the maximum value pooling, and after the maximum value pooling, the apparatus for improving the inference speed of a binarized convolutional neural network to binarize the batch normalized result value. delete delete 11. The method of claim 10,
The pooling window is a K x K window, which has k ² input elements, an apparatus for improving inference speed of a binarized convolutional neural network. A computer-readable recording medium recording a program for executing the method of any one of claims 1, 2, and 5 to 9 on a computer.

Images