KR102597079B1 - Method and device for compression of convolution neural network using n-mode tensor product operation - Google Patents

Abstract

The disclosed technology relates to a method and device for compressing a convolutional neural network using an n-dimensional tensor product operation, comprising: selecting a specific layer among a plurality of layers included in a convolutional neural network (CNN) by a computing device; Calculating importance for each of the plurality of weight filters by performing an n-dimensional tensor product operation on each of the plurality of weight filters included in the specific layer, by the computing device; and removing, by the calculation device, some weight filters whose importance is less than a standard among the plurality of weight filters according to the calculation result.

Description

Convolutional neural network compression method and device using n-dimensional tensor product operation {METHOD AND DEVICE FOR COMPRESSION OF CONVOLUTION NEURAL NETWORK USING N-MODE TENSOR PRODUCT OPERATION}

The disclosed technology relates to a method and device for compressing a convolutional neural network by calculating the influence of a weight filter of a specific layer within the convolutional neural network on the next layer through an n-dimensional tensor product operation.

Recently, various machine learning and computer vision technologies have achieved very high performance by utilizing deep convolutional neural networks. However, due to the structural characteristics of convolutional neural networks, many previously developed deep convolutional neural network-based technologies have essential problems that require a high level of computing power. To solve the problem of high memory and computational complexity of deep convolutional neural networks, a filter selection method is used to select and remove redundant weight filters that do not or have only a minor effect on the performance of the neural network among the weight filters that make up the convolutional layer. Neural network compression techniques have been proposed.

Existing filter selection methods measure the norms of the weight filters constituting a specific convolution layer, remove weight filters with low norms, and conversely, maintain weight filters with high norms. Since this neural network compression technique processes each convolution layer independently, the effect of the filter on the later convolution layer is not considered in the process of evaluating the importance of the weight filter. In other words, the existing filter selection method had the problem of generating a local optimal problem from the perspective of neural network compression.

Korean Patent Publication No. 10-2020-0052200

The disclosed technology provides a method and device for compressing a convolutional neural network by calculating the influence of a weight filter of a specific layer in the convolutional neural network on the next layer through an n-dimensional tensor product operation.

The first aspect of the technology disclosed to achieve the above technical problem is a step of selecting a specific layer from a plurality of layers included in a convolution neural network (CNN) by a computing device, and selecting a specific layer by the processing device in the specific layer. Calculating importance for each of the plurality of weight filters by performing an n-dimensional tensor product operation on each of the plurality of weight filters included, and the calculation device selecting the weight filters among the plurality of weight filters according to the calculation result. The aim is to provide a convolutional neural network compression method that includes removing some weight filters whose importance is less than a standard.

The second aspect of the technology disclosed to achieve the above technical problem is a storage device for storing a convolution neural network (CNN) including a plurality of layers and a specific layer among the plurality of layers included in the convolution neural network. A convolutional neural network including an operation unit that selects and performs an n-dimensional tensor product operation on the weight filters included in the specific layer to calculate the importance of the weight filter, and removes the weight filter according to the calculation result. The purpose is to provide a compression device.

Embodiments of the disclosed technology may have effects including the following advantages. However, since this does not mean that the embodiments of the disclosed technology must include all of them, the scope of rights of the disclosed technology should not be understood as being limited thereby.

A convolutional neural network compression method and device using an n-dimensional tensor product operation according to an embodiment of the disclosed technology has the effect of compressing a convolutional neural network by performing an n-dimensional tensor product operation for a specific weight filter.

In addition, it has the effect of reducing memory consumption and computational complexity by efficiently compressing the convolutional neural network.

In addition, it has the effect of preventing local optimality problems in neural networks from occurring by calculating the impact of a specific weight filter on the next layer.

Figure 1 is a diagram illustrating a process for compressing a convolutional neural network using an n-dimensional tensor product operation according to an embodiment of the disclosed technology.
Figure 2 is a flowchart of a convolutional neural network compression method using an n-dimensional tensor product operation according to an embodiment of the disclosed technology.
Figure 3 is a block diagram of a convolutional neural network compression device using an n-dimensional tensor product operation according to an embodiment of the disclosed technology.
Figure 4 is a diagram showing the structure of a conventional convolutional neural network.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present invention.

Terms such as first, second, A, B, etc. may be used to describe various components, but the components are not limited by the terms, and are only used for the purpose of distinguishing one component from other components. It is used only as For example, a first component may be referred to as a second component, and similarly, the second component may be referred to as a first component without departing from the scope of the present invention. The term and/or includes any of a plurality of related stated items or a combination of a plurality of related stated items.

As used herein, singular expressions should be understood to include plural expressions unless the context clearly dictates otherwise. And terms such as "include" mean the presence of the described features, numbers, steps, operations, components, parts, or combinations thereof, but one or more other features, numbers, steps, operations, components, parts, etc. It should be understood that it does not exclude the existence or addition of combinations thereof.

Before providing a detailed description of the drawings, it would be clarified that the division of components in this specification is merely a division according to the main function each component is responsible for. That is, two or more components, which will be described below, may be combined into one component, or one component may be divided into two or more components for more detailed functions.

In addition to the main functions it is responsible for, each of the components described below may additionally perform some or all of the functions handled by other components, and some of the main functions handled by each component may be performed by other components. Of course, it can also be carried out exclusively by . Therefore, the presence or absence of each component described throughout this specification should be interpreted functionally.

Figure 1 is a diagram illustrating a process of compressing a convolution neural network (CNN) using an n-dimensional tensor product operation according to an embodiment of the disclosed technology. Referring to Figure 1, the computing device stores a convolutional neural network for image analysis. Convolutional neural networks are pre-trained for image classification and can be equipped with convolutional neural networks of various structures. The computing device can input the image input by the user into a convolutional neural network and classify the image according to the results output through the convolutional neural network. Of course, in addition to classifying images, it is also possible to extract features by inputting text or voice. Below, image classification is explained as an example.

Meanwhile, a convolutional neural network has a structure in which multiple layers are stacked. The performance of feature extraction can be improved by increasing the depth of the network. However, due to the above-mentioned structural characteristics, a computing device equipped with a convolutional neural network may experience problems of increased memory complexity and computational complexity when analyzing an input image. To solve this problem, the computing device can compress the convolutional neural network by calculating the importance of the weight filters included in the plurality of layers constituting the convolutional neural network and removing weight filters that are below the standard.

Meanwhile, in the process of compressing a convolutional neural network, conventionally, the importance of weight filters included in a specific layer was independently determined. For example, the neural network was compressed by measuring the L1 norm or L2 norm of a weight filter, maintaining weight filters with a high norm and removing weight filters with a low norm. However, this method only judges the importance of the layer to which the weight filter belongs and does not consider the influence of the weight filter on subsequent layers, resulting in a local optimality problem. Therefore, this technology discloses a technique for compressing a convolutional neural network by considering the influence of the weight filter on subsequent layers.

Meanwhile, the computing device selects a specific layer among a plurality of layers included in the convolutional neural network in order to perform the convolutional neural network compression in the above-described manner. Here, the process of selecting a specific layer can be processed by randomly selecting one of the plurality of layers that make up the convolutional neural network. After calculating the importance of a plurality of weight filters included in the corresponding layer, another layer can be selected. Of course, you can calculate the importance of the weight filter by selecting it sequentially from the first layer, or you can randomly select both the order of the layers of the convolutional neural network and the order of the weight filter. For example, you can select the jth weight filter of the ith layer, calculate the importance, and then select the sth weight filter of the kth layer.

Meanwhile, the computing device calculates the importance of each of the plurality of weight filters by performing an n-dimensional tensor product operation on each of the plurality of weight filters included in the selected specific layer. Before calculating the importance of the weight filter, the memory complexity of the convolutional neural network is calculated according to Equation 1 below: and computational complexity is defined as follows.

From here is the ith layer of the convolutional neural network refers to the width and height of the weight filter that constitutes means the number of weight filters. and means the width of the i+1th feature map, means the height of the i+1th feature map. In the neural network compression technique that removes specific weight filters, the number of weight filters cast The goal is to reduce it to Therefore, the memory complexity of a compressed convolutional neural network is and computational complexity is defined as follows.

Meanwhile, the memory compression rate of a neural network can be defined as the memory complexity of the compressed neural network compared to the memory complexity of the original neural network. Likewise, the computational complexity compression ratio of a neural network can be defined as the computational complexity of the compressed neural network compared to the computational complexity of the original neural network. The memory compression rate and computational complexity compression rate of this compressed neural network can be defined by Equation 3.

Meanwhile, in the norm-based filter selection technique, the j-th weight filter included in the i-th layer The importance of can be expressed as Equation 4 below.

Here, the importance of the jth weight filter included in the ith layer Is Same as in other words, Is means the norm operator of . In the conventional technique, since a specific weight determines only whether the norm of the filter is high or low, the importance is determined independently for the layer to which the weight filter belongs. In other words, a local optimality problem occurs because the weight filter does not consider how it will affect subsequent layers.

To solve this problem, the computing device can calculate the importance of the weight filter by performing an n-dimensional tensor product operation according to Equation 5 below.

From here is the ith layer and i+1th layer It refers to the n-dimensional multiplication process between and means a weight filter in which the jth weight filter of the ith layer is set to 0. The n-dimensional multiplication is calculated using Equation 6 below.

From here is the ith layer second It refers to a dimension conversion function that changes the form of . The method proposed in the disclosed technology is defined in the form of a reconstruction loss function of a new high-dimensional tensor calculated according to an n-dimensional tensor product operation. New tensor data obtained through an n-dimensional tensor product operation expresses the influence of a specific layer on the subsequent layer, and the importance of the weight filter can be quantified through the reconstruction loss value of replacing a specific weight filter with 0.

Meanwhile, the computing device can calculate the importance of each weight filter of a specific layer according to the above-described calculation process. And, depending on each result, some weight filters whose importance is below the standard can be removed. And by applying this process to all layers in the convolutional neural network, the convolutional neural network can be compressed by removing weight filters of low importance for each layer. In this process, redundant parameters among the many parameters included in the convolutional neural network are removed, thereby reducing the complexity of the calculation process and memory complexity during calculation.

Figure 2 is a flowchart of a convolutional neural network compression method using an n-dimensional tensor product operation according to an embodiment of the disclosed technology. Referring to FIG. 2, the convolutional neural network compression method includes steps 210 to 230. Each step can be performed sequentially through a computing device.

In step 210, the computing device selects a specific layer among a plurality of layers included in a convolution neural network (CNN). One of the plurality of layers that make up the convolutional neural network may be selected in a random manner. Since the computing device cannot know the importance of the weight filters for each layer, it may sequentially select all layers included in the neural network and perform an n-dimensional tensor product operation for each layer.

In step 220, the computing device calculates the importance of each of the plurality of weight filters by performing an n-dimensional tensor product operation on each of the plurality of weight filters included in a specific layer. The process of calculating the importance does not calculate the importance of the weight filter independently for each layer, as previously explained in Figure 1, but calculates the importance by considering the influence on the subsequent layer.

In step 230, the computing device removes some weight filters whose importance is less than the standard among the plurality of weight filters according to the calculation result. The processing unit may repeat steps 210 to 230 a certain number of times to remove weight filters of low importance from the convolutional neural network.

Figure 3 is a block diagram of a convolutional neural network compression device using an n-dimensional tensor product operation according to an embodiment of the disclosed technology. Referring to FIG. 3, the convolutional neural network compression device 300 includes a storage device 310 and an arithmetic device 320.

The storage device 310 stores the convolutional neural network. Since the convolutional neural network increases the number of layers to create a deeper network, the storage device 310 can use a memory with a capacity sufficient to store the convolutional neural network.

The computing unit 320 selects a specific layer from among the plurality of layers included in the convolutional neural network and calculates the importance of the weight filter by performing an n-dimensional tensor product operation on the weight filter included in the specific layer. Then, the weight filter is removed according to the calculation results. The computing device 320 may be a processor or an AP of the convolutional neural network compression device 300. The calculation unit 320 may remove the weight filter if the result of subtracting the tensor data of the weight filters calculated according to the n-dimensional tensor product operation and the tensor data obtained by replacing the weight filter with 0 is less than the standard. The computing device may apply this calculation process to each of the remaining weight filters included in a specific layer to remove at least one weight filter whose importance is less than the standard. By repeating this calculation process, the convolutional neural network stored in the storage device 310 can be compressed as a whole.

Meanwhile, the convolutional neural network compression device 300 described above may be implemented as a program (or application) including an executable algorithm that can be executed on a computer. The program may be stored and provided in a temporary or non-transitory computer readable medium.

A non-transitory readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as registers, caches, and memories. Specifically, the various applications or programs described above include CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM (read-only memory), PROM (programmable read only memory), and EPROM (Erasable PROM, EPROM). Alternatively, it may be stored and provided in a non-transitory readable medium such as EEPROM (Electrically EPROM) or flash memory.

Temporarily readable media include Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), and Enhanced SDRAM (Enhanced RAM). It refers to various types of RAM, such as SDRAM (ESDRAM), Synchronous DRAM (Synclink DRAM, SLDRAM), and Direct Rambus RAM (DRRAM).

Figure 4 is a diagram showing the structure of a conventional convolutional neural network. Referring to Figure 4, VGG-16 and ResNet-18, which are types of convolutional neural networks, have different structures. First, VGG-16 is a structure that forms a neural network by stacking 3x3 weight filters, and ResNet-18 is basically the same as VGG-16, but has the added structural feature of skip connection. Referring to Table 1 below, as a result of applying the disclosed technology to the above two types of convolutional neural networks, it was confirmed that compression performance was improved compared to the conventional norm-based filter removal technique. Table 1 shows the performance of VGG-16 and ResNet-18 when classifying images using CIFAR-100, a public dataset, performance when compressing using the L1 Norm filter selection method, performance when compressing using the L2 Norm filter selection method, and applying the disclosed technology. This shows the performance during compression.

# of params Accuracy (%) VGG-16 14723136 72.48 L1 Norm 1833621
(12.45%) 63.94
(-8.54) L2 Norm 1833621
(12.45%) 62.59
(-9.89) Ours 1811760
(12.31%) 65.73
(-6.75) ResNet-18 6448192 76.38 L1 Norm 929660
(14.4%) 68.17
(-8.21) L2 Norm 929660
(14.4%) 67.29
(-9.09) Ours 914923
(14.2%) 71.45
(-4.93)

Referring to Table 1 above, the number of parameters (# of params) of the original neural network of VGG-16 was 14723136, and the accuracy was 72.48%. Here, as a result of applying the L1 Norm-based filter selection method, the number of parameters was reduced to 1833621 and the accuracy was 63.94%. The results of applying the L2 Norm-based filter selection method also showed that the number of parameters was 1833621 and the accuracy was 62.59%. Both techniques showed similar performance. And as a result of applying the disclosed technology to VGG-16, the number of parameters was reduced to 1811760 compared to the conventional technique, and the accuracy was confirmed to be increased to 65.73%.

Meanwhile, the number of parameters (# of params) of the original neural network of ResNet-18 was 6448192, and the accuracy was 76.38%. Here, as a result of applying the L1 Norm-based filter selection method, the number of parameters was reduced to 929660 and the accuracy was 68.17%. The results of applying the L2 Norm-based filter selection method showed a number of parameters of 929660 and an accuracy of 67.29%. Likewise, both techniques showed similar performance. And as a result of applying the disclosed technology to ResNet-18, it was confirmed that the number of parameters was reduced to 914923, which was more than the conventional method, and the accuracy was actually increased to 71.45%. In other words, regardless of the structure of the convolutional neural network, it was confirmed that the result of compressing the neural network by applying the disclosed technology shows higher performance compared to the conventional L1 and L2 Norm-based filter selection methods. Therefore, by applying the disclosed technology regardless of the type or structure of the convolutional neural network, the neural network can be efficiently compressed and memory complexity and computational complexity can be reduced.

The convolutional neural network compression method and device using an n-dimensional tensor product operation according to an embodiment of the disclosed technology has been described with reference to the embodiment shown in the drawings to aid understanding, but this is merely illustrative and is a method commonly used in the field. Those skilled in the art will understand that various modifications and other equivalent embodiments are possible. Therefore, the true technical protection scope of the disclosed technology should be determined by the appended patent claims.

Claims (10)

Selecting, by a computing device, a specific layer from among a plurality of layers included in a convolution neural network (CNN);
Calculating importance for each of the plurality of weight filters by performing an n-dimensional tensor product operation on each of the plurality of weight filters included in the specific layer, by the computing device; and
Including, by the calculation device, removing some weight filters whose importance is less than the standard among the plurality of weight filters according to the calculation result,
A convolutional neural network compression method in which the calculation unit removes the weight filter if the result of subtracting the tensor data of the plurality of weight filters calculated according to the n-dimensional tensor multiplication operation and the tensor data obtained by replacing the weight filter with 0 is less than a standard. . According to claim 1,
The step of selecting a specific layer is a convolutional neural network compression method in which one of a plurality of layers included in the convolutional neural network is randomly selected. delete According to claim 1,
A convolutional neural network compression method in which the importance is calculated according to Equation 5 below.
[Equation 5]

(From here means the importance of the jth weight filter of the ith layer, is the ith layer and i+1th layer It means n-dimensional multiplication between and means the width and height of the weight filter included in the ith layer, means the number of weight filters in the ith layer. and means replacing the jth weight filter of the ith layer with a weight filter consisting of 0.) According to claim 4,
A convolutional neural network compression method in which the n-dimensional multiplication is calculated according to Equation 6 below.
[Equation 6]

(From here is the ith layer second refers to a dimension conversion function that changes the form.) A storage device for storing a convolution neural network (CNN) including a plurality of layers; and
Select a specific layer from among the plurality of layers included in the convolutional neural network, calculate the importance of the weight filter by performing an n-dimensional tensor product operation on the weight filter included in the specific layer, and calculate the importance of the weight filter according to the calculation result. Including an arithmetic unit for removing the weight filter,
The calculation device is a convolutional neural network compression device that removes the weight filter when the result of subtracting the tensor data of the weight filters calculated according to the n-dimensional tensor multiplication operation and the tensor data obtained by replacing the weight filter with 0 is less than a standard. delete According to claim 6,
The computing device is a convolutional neural network compression device that calculates the importance according to Equation 5 below.
[Equation 5]

(From here means the importance of the jth weight filter of the ith layer, is the ith layer and i+1th layer It means n-dimensional multiplication between and means the width and height of the weight filter included in the ith layer, means the number of weight filters in the ith layer. and means replacing the jth weight filter of the ith layer with a weight filter consisting of 0.) According to claim 8,
The computing device is a convolutional neural network compression device that calculates the n-dimensional multiplication according to Equation 6 below.
[Equation 6]

(From here is the ith layer second refers to a dimension conversion function that changes the form.) According to claim 6,
A convolutional neural network compression device in which the calculation unit calculates the importance of each of the remaining weight filters included in the specific layer and removes at least one weight filter whose importance is less than a standard.