KR102315617B1 - Apparatus and method for neural network pruning considering structure of graphic processing device - Google Patents

Abstract

An objective of the present application is to provide a device and method for neural network pruning considering a structure of a graphic processing device capable of accelerating inference speed while minimizing a loss of accuracy of neural network inference performed in the graphic processing device. According to a first embodiment of the present application, the method for neural network pruning may comprise: a step of performing GEMM transformation of a weighted kernel of a neural network operating through a graphic processing unit (GPU) including a plurality of computation units; and a step of pruning the GEMM-transformed weighted kernel using a block, which is a division unit of a matrix multiplication operation performed in parallel through a plurality of operation units, as a unit.

Description

Apparatus and method for neural network pruning considering the structure of graphic processing unit

The present application relates to a neural network pruning apparatus and method in consideration of the structure of a graphic processing apparatus.

Convolutional Neural Network (CNN) is very good at extracting features of an applied image, so it can perform computer vision such as image classification, segmentation, and recognition. ), and the application of artificial neural networks (ANNs) such as natural language processing (NLP) and autonomous driving. In particular, in the field of natural language processing and autonomous driving, the general purpose computing on graphics processing units (GPGPU) of GPUs is common, and through this, the computational acceleration of data stream processing utilizing the massive parallelism of the GPU is utilized. .

In particular, the performance of the convolutional neural network has been maximized through the construction of a deep neural network (DNN) in which several convolutional layers are deeply stacked, but this performance improvement requires a huge amount of computation and memory usage. Moreover, it can be seen that the operation is biased in the convolutional layer even though 99% of all operations of the convolutional neural network have a small number of weights compared to the all-connected layer to be performed in the convolutional layer.

To overcome these limitations of convolutional neural networks, various neural network lightweighting techniques have been studied, and among them, the pruning technique compresses the neural network through the process of removing unnecessary weights existing in the neural network. In general, the internal weights of the convolutional neural network are over-parameterized, so considering that there are redundant weights that do not significantly affect the accuracy even if they are removed, the pruning method uses these extra weights. Through the process of making the value of 0 to 0, the amount of computation and memory usage can be reduced.

In this regard, the pruning technique is divided into unstructured/structured pruning according to the weight unit to be removed. First, the weight region of the convolutional layer removed through the unstructured pruning technique has irregular sparseness, which is a special computation library. Alternatively, there is a limit in that it is limited to obtain a performance gain on the GPU without a separate hardware accelerator.

Meanwhile, FIG. 1 is a conceptual diagram illustrating a kernel-channel pruning technique, which is one of the conventional structural pruning techniques. Referring to FIG. 1 , since the structural pruning technique removes the regular weight region, performance improvement can be obtained only with the pruning technique, but the removal of the regular weight causes the accuracy deterioration of the convolutional neural network that cannot be ignored, and In an accelerator with massive parallelism, there is a problem that the removal of some regular weights may cause a decrease in computational speed.

The technology that is the background of the present application is disclosed in Korean Patent Application Laid-Open No. 10-2020-0145648.

The present application provides a neural network pruning apparatus and method in consideration of the structure of a graphics processing apparatus capable of accelerating the inference speed while minimizing the loss of accuracy of neural network inference performed in the graphics processing apparatus in order to solve the problems of the prior art described above. intended to provide.

However, the technical problems to be achieved by the embodiments of the present application are not limited to the technical problems as described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, the neural network pruning method in consideration of the structure of the graphic processing device according to the first embodiment of the present application is a graphic processing unit including a plurality of computation units (Compute Unit). GEMM transformation of the weight kernel of the neural network operating through the unit, GPU) and the GEMM-transformed weight kernel by using the block as a division unit of the matrix multiplication operation performed in parallel through the plurality of operation units as a unit. It may include the step of looping.

Also, the graphic processing apparatus may include a global memory and a plurality of local memories provided for each of the plurality of operation units.

Also, the block may correspond to a partitioned portion of the GEMM-transformed weight kernel copied from the global memory to the local memory in order to perform the matrix multiplication operation.

In addition, each of the plurality of operation units may perform the matrix multiplication operation on the plurality of preset blocks.

In addition, the pruning may include removing blocks of a preset ratio among the plurality of blocks associated with each of the plurality of operation units.

In addition, the block removed by the pruning may be not copied from the global memory to the local memory in the reasoning process of the neural network.

Also, the pruning may include calculating a weight importance for each block and determining a block to be removed at the preset ratio based on the calculated weight importance.

On the other hand, the neural network pruning method in consideration of the structure of the graphic processing device according to the second embodiment of the present application includes a plurality of arithmetic units (Compute Unit) each including a plurality of arithmetic processing elements (PE). Transforming the weight kernel of a neural network operating through a graphic processing unit (GPU) to GEMM and using the microblock, which is a division unit of a matrix multiplication operation performed in parallel through the plurality of operation processing elements, as a unit and pruning the GEMM-transformed weight kernel.

Also, the graphic processing apparatus may include a global memory, a plurality of local memories provided for each of the plurality of arithmetic units, and a plurality of private memories provided for each of the plurality of arithmetic processing elements.

Also, the microblock may correspond to a weight row included in a block that is a partitioned part of the GEMM-transformed weight kernel copied from the global memory to the local memory to perform the matrix multiplication operation.

Also, in the pruning, a preset ratio of the fine blocks among the plurality of fine blocks associated with each of the plurality of arithmetic processing elements may be removed.

In addition, by the pruning, the computational load of each of the plurality of computational processing elements may be equally determined.

Also, the pruning may include calculating a weight importance for each of the fine blocks and determining the fine blocks to be removed at the preset ratio based on the calculated weight importance.

On the other hand, the neural network pruning apparatus in consideration of the structure of the graphics processing apparatus according to the first embodiment of the present application is a neural network operating through a graphics processing unit (GPU) including a plurality of computation units (Compute Unit). A weight conversion unit for GEMM transformation of the weight kernel, and a pruning unit for pruning the GEMM-transformed weight kernel using a block as a division unit of a matrix multiplication operation performed in parallel through the plurality of operation units as a unit. have.

Also, the pruning unit may calculate weight importance for each of the blocks, and determine blocks to be removed at the preset ratio based on the calculated weight importance.

On the other hand, the neural network pruning apparatus in consideration of the structure of the graphic processing apparatus according to the second embodiment of the present application includes a plurality of operation units (Compute Unit) each including a plurality of processing elements (Processing Element, PE) A weight conversion unit that transforms a weight kernel of a neural network operating through a graphic processing unit (GPU) into a GEMM and a fine block that is a division unit of a matrix multiplication operation performed in parallel through the plurality of operation processing elements It may include a pruning unit for pruning the GEMM-transformed weight kernel.

In addition, the pruning unit may equally determine the computational load of each of the plurality of arithmetic processing elements by removing a preset ratio of the fine blocks from among the plurality of fine blocks associated with each of the plurality of arithmetic processing elements.

Also, the pruning unit may calculate a weight importance for each of the fine blocks, and determine the fine blocks to be removed at the preset ratio based on the calculated weight importance.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present application. In addition to the exemplary embodiments described above, additional embodiments may exist in the drawings and detailed description.

According to the above-described problem solving means of the present application, it is possible to provide a neural network pruning apparatus and method in consideration of the structure of a graphic processing unit capable of accelerating the inference speed while minimizing the loss of accuracy of the neural network inference performed in the graphic processing unit. .

According to the above-described problem solving means of the present application, it is possible to obtain a gain in operation speed while showing a much smaller loss of accuracy compared to the structural weight pruning technique.

However, the effects obtainable herein are not limited to the above-described effects, and other effects may exist.

1 is a conceptual diagram illustrating a kernel-channel pruning technique, which is one of the conventional structural pruning techniques.
2 is a schematic configuration diagram of a neural network pruning apparatus in consideration of the structure of the graphic processing apparatus according to the first embodiment of the present application.
3 and 4 are conceptual diagrams illustrating a process of transforming an input feature map and a weight kernel of a neural network into a GEMM.
5 is a conceptual diagram for explaining a division unit of a matrix multiplication operation using a hierarchical structure of a graphic processing device and a GEMM-transformed weight kernel.
6 is a conceptual diagram illustrating a blocking/tiling process associated with a matrix multiplication operation using a GEMM-transformed weight kernel.
7 is a schematic configuration diagram of a neural network pruning apparatus in consideration of the structure of a graphic processing apparatus according to a second embodiment of the present application.
8 is a conceptual diagram for explaining a pruning technique in units of fine blocks performed by a neural network pruning device considering the structure of a graphic processing device according to a second embodiment of the present application.
9A to 9C are views showing comparison of the pruning results by the conventional pruning technique and the pruning technique disclosed herein.
10 is an operation flowchart of a neural network pruning method in consideration of the structure of the graphic processing apparatus according to the first embodiment of the present application.
11 is an operation flowchart of a neural network pruning method in consideration of the structure of the graphic processing apparatus according to the second embodiment of the present application.

Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present application pertains can easily implement them. However, the present application may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present application in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout this specification, when a part is "connected" with another part, it is not only "directly connected" but also "electrically connected" or "indirectly connected" with another element interposed therebetween. "Including cases where

Throughout this specification, when it is said that a member is positioned "on", "on", "on", "under", "under", or "under" another member, this means that a member is positioned on the other member. It includes not only the case where they are in contact, but also the case where another member exists between two members.

Throughout this specification, when a part "includes" a component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

The present application relates to a neural network pruning apparatus and method in consideration of the structure of a graphic processing apparatus.

2 is a schematic configuration diagram of a neural network pruning apparatus in consideration of the structure of the graphic processing apparatus according to the first embodiment of the present application.

Referring to FIG. 2 , the neural network pruning apparatus 100 (hereinafter referred to as 'pruning apparatus 100') in consideration of the structure of the graphic processing apparatus according to the first embodiment of the present application is a weight conversion unit 110 . , it may include a pruning unit 120 and a recovery retraining unit 130 .

On the other hand, in the description of the embodiment of the present application, the neural network pruning apparatus 100' considering the structure of the pruning apparatus 100 according to the first embodiment of the present application and the graphic processing apparatus according to the second embodiment of the present application ( Hereinafter, referred to as 'pruning device 100'.) All of the convolutional operations in the learning process of a neural network in the convolution layer of a convolutional neural network are performed by a graphic processing unit (Graphic Processing Unit) (hereinafter, ' When performed through the GPU 200 '.), the convolution operation is divided in consideration of the characteristic that the convolution operation is divided by a plurality of sub-operation means included in the GPU 200 and processed in parallel. It is common in that pruning is performed using the unit to be calculated as the unit of pruning, but may be classified based on the type of division unit (ie, pruning unit) of the operation.

In summary, the pruning apparatus 100 according to the first embodiment of the present application performs pruning by 'block', and the pruning apparatus 100 ' according to the second embodiment of the present application is a 'fine block'. The pruning may be performed using .

For reference, the neural network herein may include a convolutional neural network (CNN). However, the present invention is not limited thereto, and the neural network to which the present application is applied includes various neural networks that have been previously known or developed in the future, such as a recurrent neural network (RNN), which includes a trained neural network, an untrained neural network, etc.) may include.

On the other hand, in pruning, when inferring a result by multiplying a value received as an input of a neural network by a learned weight, the values of all weights do not significantly affect the result inference. Therefore, even if some of the values of various weights constituting the neural network are removed, the accuracy of the inference result is not affected, so it can be used to reduce the size of the neural network or reduce the amount of computation. There is a method that uniformly removes all weights below the threshold with this method, but this method reduces the size of the neural network due to irregular sparsity, but there is a limitation that the computation speed cannot be improved if there is no special computation technique or hardware. have. As another technique, the technique of removing a specific weight group can improve computational speed, but since the weight affecting the result is also easily removed, there is a limit in that the accuracy of the inference result is greatly reduced.

Hereinafter, the function and operation of the pruning apparatus 100 according to the first embodiment of the present application will be described with reference to FIGS. 3 to 6 first.

3 and 4 are conceptual diagrams illustrating a process of transforming an input feature map and a weight kernel of a neural network into a GEMM.

3 and 4 , the weight transform unit 110 may perform GEMM transformation on the weight kernel of the neural network. For reference, GEMM (General Matrix Multiply) transformation is a matrix transformation method widely used in today's deep learning inference engines. In this paper, by performing pruning based on a transformation matrix utilizing such GEMM transformation, the conventional pruning technique is CSR Because special sparse matrices such as (Compressed Sparse Row), CSC (Compressed Sparse Column), and COO (Coordinate List) are used, the problem that could not be applied by overlapping with other types of pruning techniques was solved.

In this regard, in each layer constituting the neural network (eg, the convolutional layer of CNN), a predetermined interval (eg, a preset stride unit) with respect to an input image or feature map, rather than simple multiplication between matrices. As the weight kernel sequentially slides, a complex operation that multiplies and sums multiple values is processed several times. In order to transform the kernel into a two-dimensional matrix, the above-described GEMM transform may be utilized.

More specifically, referring to FIG. 4 , the weight transform unit 110 converts a two-dimensional Im2Col (Image to Column) matrix of a three-dimensional input feature map to perform a convolution operation by converting it into a general matrix product operation, and One-dimensional vectorize transformation may be performed on the weight filter of the convolutional layer. Here, Im2Col transformation is performed on the three-dimensional input feature map, and all image patches when the weights of the input feature map and the convolution layer are calculated in this process can be copied. The input feature map may be transformed into a two-dimensional matrix as shown in FIG. 4 . Also, the weight transform unit 110 may perform one-dimensional vector transformation on the three-dimensional weight kernel of the convolutional layer.

In other words, the convolution operation, which occupies most of the computational amount of neural networks (especially convolutional neural networks), is generally performed as a general matrix multiplication (GEMM) through a separate transformation process, which is highly optimized matrix multiplication. A convolution operation can be performed using a library, and in a large-scale parallel processing system such as the GPU 200, the general matrix product operation is performed more efficiently by utilizing blocking and tiling, which will be described later. can be

5 is a conceptual diagram for explaining a division unit of a matrix multiplication operation using a hierarchical structure of a graphic processing device and a GEMM-transformed weight kernel.

Referring to FIG. 5 , the GPU 200 may include a plurality of computation units 211 . In addition, each of the plurality of arithmetic units 211 may include a plurality of arithmetic processing elements 212 .

Specifically, each of the operation processing elements 212 included in the GPU 200 may be a unit of operation processing in which an actual operation is performed, and the operation unit 211 may be a set of a plurality of operation processing elements 212 . With respect to the work (task) performed through the GPU 200 , a single thread performed by one operation processing element 212 may be referred to as a work item, and one operation unit A set of work items performed in parallel in step 211 may be referred to as a work group.

5 for better understanding, Output(O '), the area partitioned into each work group may be divided into output parts individually calculated by each operation unit 211 included in the GPU 200 .

In addition, referring to FIG. 5 in relation to the hierarchical memory structure of the GPU 200 , the GPU 200 may include a global memory 221 , a plurality of local memories 222 , and a plurality of private memories 223 . have.

Specifically, the GPU 200 has various memory hierarchical structures as shown in FIG. 5 , and the global memory 221 may mean an off-chip DRAM memory area inside the GPU 200 , , it may occupy the largest area of internal memory of the GPU 200 .

In addition, the local memory 222 may refer to an on-chip SRAM memory area existing corresponding to each of the plurality of arithmetic units 211 , and is provided in any one arithmetic unit 211 . It may be a shared memory area commonly used by the plurality of arithmetic processing elements 212 .

In addition, the private memory 223 is a register memory region that exists for each arithmetic processing element 212 and may have a characteristic that the memory region is not shared with other arithmetic processing elements 212 .

In relation to the hierarchical memory structure of the GPU 200, when a predetermined GPU kernel (a set of instructions performed through the GPU 200) is executed in an external host, the GPU ( 200) Data required in the GPU kernel is copied to the internal global memory 221 , and data required for the operation of each operation processing element 212 is copied from the global memory 221 , and an instruction is executed. In addition, the size of the memory decreases in the order of the global memory 221 , the local memory 222 , and the private memory 223 , and the memory access speed may be fast.

On the other hand, the general matrix multiplication operation (GEMM operation) performed in the GPU 200 is performed in parallel using a plurality of operation processing elements 212 inside the GPU 200, and each work item is a general matrix product operation ( GEMM operation) is divided and responsible for the output feature map area, and general matrix multiplication operation is performed for each allocated area. Both the GEMM-transformed two-dimensional input feature map and the convolution weight matrix are copied to the global memory 221 area when the GPU kernel is performed, and a single input feature map and weight kernel located (copied) in the global memory 221 are single. The process of calculating the output feature map area through the cross product operation corresponding to the work item may be performed in parallel in each operation processing element 212 for all work items. A general matrix multiplication operation for the output feature map may be calculated.

6 is a conceptual diagram illustrating a blocking/tiling process associated with a matrix multiplication operation using a GEMM-transformed weight kernel.

Referring to FIG. 6 , the external operation process performed in parallel through the plurality of operation processing elements 212 of the GPU 200 is a local memory 222 and a private memory through blocking and tiling processes. This is performed after being partially copied to the memory 223, and each block divided through this blocking/tiling process is copied to the local memory 222, and the copied block is again divided into several tiles (fine blocks) to the private memory ( 223) can be copied. As such, the cross product operation on the block of the input feature map and the weight kernel copied to the local memory 222 is divided by the work group in each operation unit 211 inside the GPU 200 and copied to the private memory 223 . The input feature map and the weight kernel may be performed by a work item in each arithmetic processing element 212 inside the GPU 200 .

The pruning unit 120 is a GEMM-transformed weight kernel using a 'block', which is a division unit of a matrix multiplication operation (general matrix multiplication operation) performed in parallel through a plurality of operation units 211 of the GPU 200 as a unit. can be pruned. Here, the 'block', which is the pruning unit according to the first embodiment of the present application, is copied from the global memory 221 to the local memory 222 of each operation unit 211 in order to perform a matrix multiplication operation based on the GEMM transformation. It may correspond to a partitioned part of the GEMM-transformed weight kernel.

According to the first embodiment of the present application, when each of the plurality of operation units 211 of the GPU 200 performs a matrix multiplication operation on a plurality of preset blocks, the pruning unit 120 includes the plurality of operation units 211 . ) of a plurality of blocks associated with each of the blocks at a preset ratio may be removed for each operation unit 211 at an equal ratio.

개 일 때, 프루닝 비율 p를 적용하여 프루닝 후의 각 연산 유닛(211)에 대한 잔여 블록의 수가

개가 되도록 할 수 있다.In other words, the pruning unit 120 may remove the same amount of blocks from the individual work groups so that the work groups corresponding to each of the plurality of operation units 211 equally maintain the computational workload after pruning. In other words, the number of blocks that each operation unit 211 processes through an assigned work group

, the number of remaining blocks for each arithmetic unit 211 after pruning by applying the pruning ratio p

You can make it a dog.

In this regard, as the unit to be pruned by the pruning unit 120 is determined in block units according to the first embodiment of the present application, the blocks removed by pruning are stored from the global memory 221 to the local memory ( 222) may not be copied. In other words, according to the first embodiment of the present application, the hierarchical structure of the GPU 200 and the unit in which data (divided/divided weights) are copied when parallel operation is performed through the GPU 200 are matched with the pruning unit. Since the copy process itself between the memories of the blocks removed by pruning can be omitted, it is possible to substantially improve the operation speed through pruning.

Specifically, according to the first embodiment of the present application, the pruning unit 120 may calculate the weight importance for each of the plurality of blocks allocated to the corresponding operation unit 211 for each of the plurality of operation units 211 . .

Specifically, the pruning unit 120 may calculate the weight importance for each block based on the l2-norm value based on Equation 1 below.

[Equation 1]

개만큼의 블록만이 남도록 프루닝을 수행할 수 있다. 즉, 상위

개에 포함되지 않은 하위 가중치 블록은 값이 모두 0으로 대체될 수 있다.Here, each of l2 _k is a l2-norm value a weight the importance of each block, fruit ningbu 120 ratio previously set based on the calculated weighted priority (p) a plurality of operation unit 211, the block to be removed as a top with the highest l2-norm value to determine

Pruning can be performed so that only as many blocks are left. that is, upper

All lower weight blocks not included in the dog can be replaced with 0 values.

In summary, the pruning apparatus 100 according to the first embodiment of the present application uses a 'block' divided in the block/tiling process for performing a GEMM transform-based general matrix multiplication operation as a unit of pruning and a weight to be removed It is possible to perform block unit pruning to select .

On the other hand, the weight region removed by applying block unit pruning according to the first embodiment of the present application is identified as simple block location metadata within the GPU kernel, so that the copying process and calculation process for the local memory are performed. may be omitted. In this regard, the pruning unit 120 may replace all weight values of the weighted region removed through block-by-block pruning with 0, and generate block position metadata that is metadata for the removed weighted region. That is, the block location metadata contains location information of blocks that have not been removed, and in order to generate this, the pruning unit 120 first generates binarized masked pruning metadata to remove the pruning metadata. Block position metadata can be generated through a process of setting a block to have a value of 0 and counting the number of 0s between blocks that have not been removed. When this block location metadata is passed to the GPU kernel, the GPU 200 can easily determine the location where the actual operation is to be performed by simply excluding the pruned weight in the inference process of the neural network.

The recovery retraining unit 130 may recover at least a portion of the inter-node weight kernel in the pruned neural network, and retrain the recovered inter-node weight kernel.

7 is a schematic configuration diagram of a neural network pruning apparatus in consideration of the structure of a graphic processing apparatus according to a second embodiment of the present application.

Referring to FIG. 7 , the pruning apparatus 100 ′ may include a weight conversion unit 110 ′, a pruning unit 120 ′, and a recovery retraining unit 130 ′. On the other hand, in the case of the weight conversion unit 110' and the recovery retraining unit 130', the weight conversion unit 110 and the recovery retraining unit 130' of the pruning apparatus 100 according to the first embodiment of the present application described above. Since it can be understood to perform the same function as , hereinafter, the function of the pruning unit 120' of the pruning apparatus 100' will be mainly described.

The pruning unit 120' uses the 'fine block', which is a division unit of the matrix multiplication operation performed in parallel through the plurality of operation processing elements 212, as a unit, and the weights converted to the GEMM by the weight conversion unit 110'. You can prune the kernel. Here, the 'fine block', which is the pruning unit according to the second embodiment of the present application, is copied from the global memory 221 to the local memory 222 of each operation unit 211 in order to perform a matrix multiplication operation based on the GEMM transformation. It may correspond to each weight row included in a block that is a partitioned part of the GEMM-transformed weight kernel.

That is, the pruning apparatus 100 ′ according to the second embodiment of the present application sets the pruning unit as a smaller unit, that is, a fine block, compared to the pruning apparatus 100 according to the first embodiment of the present application, so that it is relatively fine. A pruning technique that determines the weight to be removed can be applied.

According to the second embodiment of the present application, the pruning unit 120 ′ removes for each individual arithmetic processing element 212 a predetermined ratio of fine blocks among a plurality of fine blocks associated with each of the plurality of arithmetic processing elements 212 . can As such, the pruning unit 120 ′ removes fine blocks of equal proportion from the plurality of blocks allocated to each of the arithmetic processing elements 212 , so that the arithmetic load in the reasoning step of each of the plurality of arithmetic processing elements 212 is equal. can be decided

개의 블록을 포함하고 각각의 블록이

개의 미세 블록으로 이루어질 때 단일 워크 그룹과 관련하여

개의 프루닝 후보가 존재하는 것으로 이해될 수 있다.In this regard, the work group in charge of each operation unit 211 includes a plurality of consecutive blocks, and each operation unit 211 may repeatedly process the consecutive blocks one by one. In addition, each of the plurality of blocks may include a plurality of micro blocks. More specifically, each work group

contains blocks, and each block is

With respect to a single workgroup when made up of two microblocks

It can be understood that there are two pruning candidates.

개의 미세 블록이 제거되는 것일 수 있다.In addition, since the same pruning ratio is applied to all work groups so that an even computational load (overhead) between the arithmetic processing elements 212 can be maintained, when the preset pruning removal ratio is p, each work group about

It may be that the number of fine blocks is removed.

8 is a conceptual diagram for explaining a pruning technique in units of fine blocks performed by a neural network pruning device considering the structure of a graphic processing device according to a second embodiment of the present application.

Referring to FIG. 8 , the pruning unit 120 ′ is configured so that the work items included in the work group can be allocated to the plurality of computation processing elements 212 with the same computational load so that each work group has the same computational load. The number of fine blocks removed from the copy area corresponding to the individual work item must be equally maintained for each operation processing element 212 .

개의 미세 블록을 각 미세 블록을 처리하는 연산 처리 소자(212)를 기준으로 그룹핑(재배열)한 후

개의 그룹으로 그룹핑된 각 그룹에서 같은 수의 미세 블록이 제거되도록 제거할 그룹별로 제거할 미세 블록을 결정할 수 있다.To this end, the pruning unit 120' is processed corresponding to a single work group.

After grouping (rearranging) the microblocks based on the arithmetic processing element 212 that processes each microblock

The microblocks to be removed may be determined for each group to be removed so that the same number of microblocks are removed from each group grouped into groups.

번째)에 해당하는 미세 블록은

번째 연산 처리 소자(212)에서 처리되는 워크 아이템에 대응하는 것일 수 있다.More specifically, referring to the [Block] structure in the upper right of FIG. 8 , each of the plurality of fine blocks constituting one block may be processed as one work item by the individual arithmetic processing element 212 . For example, the fine block corresponding to the first row of the block shown in the upper right of FIG. 8 corresponds to the work item processed by the first operation processing element 212 , and the fine block corresponding to the second row of the block corresponds to the second operation Corresponding to the work item being processed by the processing element 212, furthermore, the last row of the block (

The microblock corresponding to the second) is

It may correspond to the work item processed by the th operation processing element 212 .

개의 그룹으로 재배열하고,

개의 그룹에 각각 포함된

개의 미세 블록 중 동일한 비율로 미세 블록을 제거하여 연산 처리 소자(212) 각각이 처리하는 미세 블록의 수가 프루닝 후 동등하게 유지되도록 할 수 있는 것이다.In addition, referring to the [Load balancing in micro-block pruning] part at the lower right of FIG. 8 , the pruning unit 120 ′ groups rows disposed at the same position in each of a plurality of blocks into one group.

rearranged into groups of dogs,

included in each group

The number of microblocks processed by each of the arithmetic processing elements 212 can be maintained equally after pruning by removing the microblocks at the same ratio among the number of microblocks.

According to the second embodiment of the present application, the pruning unit 120 ′ may calculate the weight importance for each of the plurality of fine blocks associated with each of the plurality of arithmetic processing elements 212 .

In this regard, the pruning unit 120' may calculate the weight importance for each of the fine blocks based on the l2-norm value based on Equation 2 below.

[Equation 2]

Here, l2 _{i, j} are l2-norm values that are weight importance of each fine block, and the pruning unit 120' performs a plurality of calculations on the fine blocks to be removed at a preset ratio p based on the calculated weight importance. Pruning may be performed in units of fine blocks in an order of increasing l2-norm values to determine corresponding to each of the devices 212 . That is, all values of the removed lower weighted fine blocks may be replaced with 0.

The fine block unit pruning by the pruning unit 120 ′ also uses metadata similar to block unit pruning to omit the local memory copy and general matrix multiplication operation. To generate such metadata, the pruning unit 120' generates pruning metadata of a binarization mask having a value of 0 corresponding to the fine block removed from the pruning-retrained weight region, and calculates the amount of metadata. In order to reduce it, an integer encoding process may be performed. For example, if pruning metadata of (0100 1000 0000 00012) is encoded using an 8-bit unsigned character data type, it can be expressed as two data types of (73, 1). For example, the pruning unit 120' can encode and use 32 pruning metadata as a single integer using a 32-bit integer type, and the encoded metadata undergoes a decoding process inside the operation of the GPU Kernel, and is then decoded. The metadata may be utilized to calculate the next access position to the global memory 221 for each work item through an operation process of counting the number of zeros inside the GPU kernel.

The recovery retraining unit 130 ′ may recover at least a portion of the inter-node weight kernel in the pruned neural network, and retrain the recovered inter-node weight kernel.

In summary, the pruning apparatus 100 according to the first embodiment of the present application and the pruning apparatus 100 ′ according to the second embodiment of the present application are common in the inference process of the neural network operating through the GPU 200 GPU. Each sub-operation module is processed through parallel operation by removing weights at the same level (ratio) from a number of lower-level operation modules (operation unit 211, operation processing element 212, etc.) included in the hierarchical structure of 200. By pruning so that the amount of computation to be performed is evenly distributed, the actual computational speed gain can be obtained.

That is, the GPU 200 can quickly process parallel operations through a structure in which a plurality of sub-operation modules are arranged in parallel compared to the central processing unit (CPU), but has fewer control units compared to the central processing unit (CPU). Considering that the individual operation itself is optimized for parallel operation due to the structural feature that takes longer than the central processing unit (CPU), the pruning technique disclosed herein is an operation (convolution operation to General matrix multiplication operation according to GEMM transformation) is partitioned in a parallel form to be allocated to individual sub-operation modules, and the unit of division/allocation is set as a unit of pruning, so that optimization targeted to the parallel characteristics of the GPU 200 You can get the speed gain.

On the other hand, in the description of the embodiment of the present application above, the pruning technique (block unit pruning) according to the first embodiment of the present application and the pruning technique (fine block unit pruning) according to the second embodiment of the present application are separately performed. However, according to the embodiment of the present application, after performing block unit pruning according to the first embodiment of the present application, the weight kernel that is not removed is subsequently secondarily performed according to the second embodiment of the present application. Of course, a step-by-step pruning process of performing fine block unit pruning may be performed.

9A to 9C are diagrams showing comparison of pruning results by the conventional pruning technique and the pruning technique disclosed herein.

In the experimental example associated with the present application shown in FIGS. 9A to 9C, block unit pruning according to the first embodiment of the present application and fine block unit according to the second embodiment of the present application to a VGG16 neural network trained with CIFAR100 data by way of example Pruning was applied and compared with the conventional pruning technique.

Specifically, FIG. 9A is a diagram illustrating the weight values of the third convolutional layer of the neural network to which each pruning technique is applied at a 50% pruning ratio (p=0.5) as a two-dimensional array. Here, FIG. 9A (a) shows a case in which the conventional kernel-channel pruning is applied, and FIG. 9A (b) shows a case in which block unit pruning according to the first embodiment of the present application is applied, and FIG. 9a (c) shows a case in which the fine block unit pruning according to the second embodiment of the present application is applied, and FIG. 9a (d) shows a case in which the conventional weight pruning is applied.

The black region in FIG. 9A is a region from which the internal weight of the convolutional neural network has been removed, and has a value of 0. The third convolutional layer has 128 64×3×3 size filters, and a single filter is shown to be arranged in one column. In the case of kernel-channel pruning, it can be seen that 64 filters out of 128 filters have been removed, and in filters 64 to 128 that are actually removed, the weight of the neural network does not exist as a value of 0, but it is the same as the weight of other pruning techniques. It is plotted with a value of 0 for comparison in shape.

9B is a graph comparing the accuracy of each pruning technique. The horizontal axis of the graph of FIG. 9B is the pruning ratio (p) applied to each convolutional layer, and the vertical axis is the accuracy of the neural network that has gone through the pruning-retraining process. In addition, FIG. 9C is a graph comparing inference speeds for each pruning technique. The horizontal axis represents the number of frames processed per second, which is an index indicating the inference speed, and the vertical axis represents the accuracy of the neural network. Referring to FIG. 9C , the rates for all pruning ratios were measured in relation to block pruning and micro tile pruning disclosed herein, and weight pruning is the sparsest Only the speed of the neural network (p=0.9) was measured.

9B and 9C , both pruning techniques (block unit pruning and fine block unit pruning) disclosed herein are structural pruning techniques at the same pruning ratio Kernel-Channel pruning (Kernel-Channel pruning) ), it has less accuracy deterioration than the technique and verified that the image processing speed per second of the neural network is faster. In addition, although both pruning techniques have greater accuracy degradation compared to the weight pruning technique, which is an unstructured pruning technique, weight pruning and On the contrary, it can be seen that the two pruning techniques disclosed herein show an increase in operation speed compared to the existing neural network even at a low pruning ratio (p=0.25).

Accordingly, by applying the two pruning techniques disclosed herein to a system utilizing the GPU 200, it is possible to efficiently utilize the generated irregular sparsity while maintaining the high accuracy of the existing neural network. The inference processing rate in processing applications can be dramatically improved.

Hereinafter, an operation flow of the present application will be briefly reviewed based on the details described above.

10 is an operation flowchart of a neural network pruning method in consideration of the structure of the graphic processing apparatus according to the first embodiment of the present application.

The neural network pruning method in consideration of the structure of the graphic processing apparatus according to the first embodiment of the present application shown in FIG. 10 may be performed by the pruning apparatus 100 described above. Therefore, even if omitted below, the description of the pruning apparatus 100 may be equally applied to the description of the neural network pruning method in consideration of the structure of the graphic processing apparatus according to the first embodiment of the present application.

Referring to FIG. 10 , in step S11 , the weight conversion unit 110 generates a weight kernel of a neural network that operates through a graphic processing unit (GPU) 200 including a plurality of computation units (Compute Unit) 211 . GEMM can be converted.

Next, in step S12, the pruning unit 120 prunes the GEMM-transformed weight kernel in step S11 by using a block, which is a division unit of a matrix multiplication operation performed in parallel through a plurality of operation units 211, as a unit. can

In the above description, steps S11 and S12 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present application. In addition, some steps may be omitted as necessary, and the order between steps may be changed.

11 is an operation flowchart of a neural network pruning method in consideration of the structure of the graphic processing apparatus according to the second embodiment of the present application.

The neural network pruning method in consideration of the structure of the graphic processing apparatus according to the second embodiment of the present application shown in FIG. 11 may be performed by the above-described pruning apparatus 100'. Therefore, even if omitted below, the description of the pruning apparatus 100' may be equally applied to the description of the neural network pruning method in consideration of the structure of the graphic processing apparatus according to the second embodiment of the present application.

Referring to FIG. 11 , in step S21 , the weight converter 110 ′ performs graphic processing including a plurality of computation units (Compute Units) 211 each including a plurality of processing elements (PEs) 212 . A weight kernel of a neural network operating through a device (Graphic Processing Unit, GPU; 200) may be transformed into a GEMM.

Next, in step S22 , the pruning unit 120 ′ uses a fine block that is a division unit of a matrix multiplication operation performed in parallel through a plurality of arithmetic processing elements 212 as a unit to obtain the GEMM-converted weight kernel in step S21 . can be pruned.

In the above description, steps S21 and S22 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present application. In addition, some steps may be omitted as necessary, and the order between steps may be changed.

The neural network pruning method in consideration of the structure of the graphic processing device according to the embodiment of the present application may be implemented in the form of a program instruction that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and carry out program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In addition, the neural network pruning method in consideration of the structure of the above-described graphic processing device may be implemented in the form of a computer program or application executed by a computer stored in a recording medium.

The above description of the present application is for illustration, and those of ordinary skill in the art to which the present application pertains will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present application.

100, 100': Neural network pruning device considering the structure of the graphic processing unit
110, 110': weight conversion unit
120, 120': Pruning part
130, 130': Recovery Retraining Department
200: Graphics Processing Unit (GPU)
211: Compute Unit
212: arithmetic processing element (PE)
221: global memory
222: local memory
223: private memory

Claims (17)

In the neural network pruning method in consideration of the structure of the graphic processing device performed by the neural network pruning device,
GEMM transformation of a weighted kernel of a neural network operating through a graphic processing unit (GPU) including a plurality of computation units; and
Pruning the GEMM-transformed weight kernel using a block, which is a division unit of a matrix multiplication operation performed in parallel through the plurality of operation units, as a unit;
including,
The graphic processing device,
A global memory and a plurality of local memories provided for each of the plurality of operation units,
The block is
Corresponding to a partitioned portion of the GEMM transformed weight kernel copied from the global memory to the local memory to perform the matrix multiplication operation. delete According to claim 1,
Each of the plurality of operation units performs the matrix multiplication operation for a plurality of preset blocks,
The pruning step is
Of the plurality of blocks associated with each of the plurality of arithmetic units, a block of a preset ratio is removed, a neural network pruning method. According to claim 1,
The block removed by the pruning is characterized in that it is not copied from the global memory to the local memory in the reasoning process of the neural network. 4. The method of claim 3,
The pruning step is
calculating weight importance for each of the blocks; and
determining a block to be removed at the preset ratio based on the calculated weight importance;
That comprising a, neural network pruning method. In the neural network pruning method in consideration of the structure of the graphic processing device performed by the neural network pruning device,
Transforming the weight kernel of a neural network operating through a graphic processing unit (GPU) including a plurality of computation units (Compute Units) each including a plurality of processing elements (PE) to GEMM ; and
Pruning the GEMM-transformed weight kernel by using a fine block, which is a division unit of a matrix multiplication operation performed in parallel through the plurality of operation processing elements, as a unit;
including,
The graphic processing device,
a global memory, a plurality of local memories provided for each of the plurality of arithmetic units, and a plurality of private memories provided for each of the plurality of arithmetic processing elements,
The microblock is
Which corresponds to a weight row included in a block that is a partitioned part of the GEMM transformed weight kernel copied from the global memory to the local memory to perform the matrix multiplication operation. delete 7. The method of claim 6,
The pruning step is
Of the plurality of microblocks associated with each of the plurality of arithmetic processing elements, a method for pruning a neural network that removes a preset ratio of microblocks. 9. The method of claim 8,
By the pruning step,
A neural network pruning method, characterized in that the computational load of each of the plurality of computational processing elements is determined equally. 9. The method of claim 8,
The pruning step is
calculating weight importance for each of the fine blocks; and
determining the fine blocks to be removed at the preset ratio based on the calculated weight importance;
That comprising a, neural network pruning method. In the neural network pruning device considering the structure of the graphic processing device,
a weight conversion unit for GEMM conversion of a weight kernel of a neural network operating through a graphic processing unit (GPU) including a plurality of computation units; and
A pruning unit for pruning the GEMM-transformed weight kernel using a block, which is a division unit of a matrix multiplication operation performed in parallel through the plurality of operation units, as a unit;
including,
The graphic processing device,
A global memory and a plurality of local memories provided for each of the plurality of operation units,
The block is
Corresponding to the partitioned portion of the GEMM transformed weight kernel copied from the global memory to the local memory to perform the matrix multiplication operation, a neural network pruning apparatus. delete 12. The method of claim 11,
The pruning unit,
A neural network pruning apparatus that calculates weight importance for each of the blocks, and determines blocks to be removed at a preset ratio based on the calculated weight importance. In the neural network pruning device considering the structure of the graphic processing device,
A weight for GEMM transformation of a weighted kernel of a neural network operating through a graphic processing unit (GPU) including a plurality of computation units, each including a plurality of processing elements (PE). conversion unit; and
A pruning unit for pruning the GEMM-transformed weight kernel by using a fine block, which is a division unit of a matrix multiplication operation performed in parallel through the plurality of operation processing elements, as a unit;
including,
The graphic processing device,
a global memory, a plurality of local memories provided for each of the plurality of arithmetic units, and a plurality of private memories provided for each of the plurality of arithmetic processing elements,
The microblock is
Which corresponds to a weight row included in a block that is a partitioned part of the GEMM transformed weight kernel copied from the global memory to the local memory to perform the matrix multiplication operation, a neural network pruning apparatus. delete 15. The method of claim 14,
The pruning unit,
A neural network pruning device that equally determines the computational load of each of the plurality of arithmetic processing elements by removing a predetermined ratio of the fine blocks among the plurality of fine blocks associated with each of the plurality of arithmetic processing elements. 17. The method of claim 16,
The pruning unit,
A neural network pruning apparatus that calculates weight importance for each of the fine blocks, and determines the fine blocks to be removed at the preset ratio based on the calculated weight importance.

Images