KR102455310B1 - Apparatus and Method for Convolutional Neural Network Quantization Inference - Google Patents

Abstract

A convolutional neural network quantization inference apparatus and method are disclosed. The convolutional neural network quantization inference method according to an embodiment of the present invention receives the characteristics and convolutional layer information of the input data to be inferred from the pre-trained convolutional neural network, by analyzing the characteristics and convolutional layer information of the input data. and setting OpenCL parameters and a kernel optimized for the hardware platform of the embedded system, and performing quantization inference on the convolutional layer using the set OpenCL parameters and kernel.

Description

Apparatus and Method for Convolutional Neural Network Quantization Inference

The described embodiment relates to a technology for real-time reasoning using a convolutional neural network (CNN) in an embedded system such as a smartphone.

With the recent development of deep learning (DNN) technology, convolutional neural networks (CNNs), which are used for object recognition in images, are in the spotlight. In particular, as an embedded system in the form of a personal terminal such as a smartphone is widely used recently, the demand for a real-time inference technology using a convolutional neural network in an embedded system rather than a general desktop environment is rapidly increasing.

In particular, since embedded systems have limited system resources and performance compared to high-end desktops, a technology that can lighten and accelerate convolutional neural networks is essential for real-time reasoning in embedded systems.

Korean Patent Publication No. 10-2019-0034985

The described embodiment aims to lighten and accelerate a convolutional neural network for real-time inference in an embedded system having limited system resources, such as a smartphone.

The convolutional neural network quantization inference method according to the embodiment receives the characteristics and convolutional layer information of the input data to be inferred from the pre-trained convolutional neural network, and analyzes the characteristics of the input data and the convolutional layer information of the embedded system. and setting OpenCL parameters and a kernel optimized for a hardware platform, and performing quantization inference on a convolutional layer using the set OpenCL parameters and kernel.

According to an embodiment, real-time reasoning is enabled by maximizing the effect of accelerating the reasoning operation based on the GPU in an embedded system having limited system resources.

According to an embodiment, by using OpenCL, quantization inference can be performed in various embedded systems without being limited to a specific hardware platform.

According to an embodiment, by automatically setting OpenCL parameters and a kernel optimized for a hardware platform and data size, it is possible to utilize the quantization inference function even without specialized knowledge of OpenCL.

1 is a block diagram of an embedded system including a convolutional neural network quantization inference apparatus according to an embodiment.
2 is a signal flow diagram illustrating a method for inferring quantization of a convolutional neural network according to an embodiment.
FIG. 3 is a flowchart for explaining an optimal OpenCL parameter and kernel setting step ( S230 ) shown in FIG. 2 .
4 is a diagram for explaining an operation process of a convolutional layer using vectorization.
FIG. 5 is a flowchart for explaining in detail the step S250 of performing the convolutional layer parallel processing shown in FIG. 2 .
6 is an exemplary diagram of performing interleaving based on a 4x4 reference matrix in an 8x8 matrix.
7 is an exemplary diagram of performing order transformation based on a 1x4 reference matrix in an 8x8 matrix.
8 is a flowchart illustrating a quantization speculation operation according to an embodiment.
9 is a diagram showing the configuration of a computer system according to an embodiment.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in a variety of different forms, only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention belongs It is provided to fully inform the possessor of the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout.

Although "first" or "second" is used to describe various elements, these elements are not limited by the above terms. Such terms may only be used to distinguish one component from another. Accordingly, the first component mentioned below may be the second component within the spirit of the present invention.

The terminology used herein is for the purpose of describing the embodiment and is not intended to limit the present invention. In this specification, the singular also includes the plural, unless specifically stated otherwise in the phrase. As used herein, “comprises” or “comprising” implies that the stated component or step does not exclude the presence or addition of one or more other components or steps.

Unless otherwise defined, all terms used herein may be interpreted with meanings commonly understood by those of ordinary skill in the art to which the present invention pertains. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless clearly specifically defined.

Hereinafter, a convolutional neural network quantization inference apparatus and method according to an embodiment will be described in detail with reference to FIGS. 1 to 9 .

1 is a block diagram of an embedded system including a convolutional neural network quantization inference apparatus according to an embodiment.

Referring to FIG. 1 , a convolutional neural network quantization inference apparatus (hereinafter referred to as 'device') 100 uses a convolutional neural network 10, a pre-trained model, OpenCL Deivce 20 such as a GPU in an embedded system. to infer

In this case, the learning of the convolutional neural network 10 and the transformation of the learning model may be performed in advance on a general desktop having less resource limitations than an embedded system. Here, the convolutional neural network 10 is generally composed of a convolutional layer and a fully connected layer. Since the inference speed for the convolutional layer may be lowered in an embedded system with limited resources, the device 100 according to the embodiment. may perform quantization inference on the convolutional layer in the convolutional neural network 10 .

In this case, the device 100 accelerates the reasoning operation by using OpenCL to support various embedded systems without being dependent on specific hardware. In this case, OpenCL is an open parallel computing language and is a standard specification that can perform parallel processing in various embedded systems regardless of hardware platforms.

The apparatus 100 according to the embodiment includes an application unit 110 , an analysis unit 120 , and a parallel processing unit 130 .

The application unit 110 performs a user interface function with the convolutional neural network 10 . That is, the application unit 110 transmits the input data characteristics of the convolutional neural network 10 and the information of the convolutional layer to the analysis unit 120 , and receives the operation result from the parallel processing unit 130 to receive the convolutional neural network 10 . ) is passed to

The analyzer 120 automatically sets OpenCL parameters optimized for a hardware platform on which inference is to be performed so as to maximize the parallel processing effect of OpenCL. In addition, by automatically selecting the optimal OpenCL kernel according to the size of the data to be inferred, the OpenCL operation optimized for the hardware platform is performed.

The parallel processing unit 130 converts the convolutional neural network quantization reasoning device 100 into a signed 8-bit integer type (= Inference is performed by converting to signed char).

Detailed operation descriptions of the application unit 110 , the analysis unit 120 , and the parallel processing unit 130 will be described in detail in the convolutional neural network quantization inference method shown in FIGS. 2 to 7 .

2 is a signal flow diagram illustrating a method for inferring quantization of a convolutional neural network according to an embodiment.

Referring to FIG. 2 , the application unit 110 receives characteristics of input data and convolutional layer information to be inferred from the convolutional neural network 10 ( S210 ), and transmits them to the analysis unit 120 ( S220 ). ).

In this case, the characteristics of the input data to be inferred may be horizontal and vertical sizes of the image or image, the number of channels of the input image or image, and information on the arrangement order of channels, for example, RGB, BGR, and the like.

At this time, the information of the convolutional layer includes at least one of the memory address of the input layer, the number of filters, the size of the filter, the size of the filter padding, the size of the filter stride, and whether to use a bias. may be included.

The analysis unit 120 analyzes the convolutional layer and input data characteristic information transmitted through the application unit 110, analyzes the characteristics of the OpenCL device in the hardware platform to be inferred, and OpenCL parameters optimized for the current system and The kernel is automatically set (S230). A detailed description thereof will be described later with reference to FIG. 3 .

The analysis unit 120 transmits the input data received from the application unit 110 and information on the convolution layer based on the set OpenCL parameter and the kernel to request a parallel processing operation (S240).

The parallel processing unit 130 performs a parallel processing operation based on the OpenCL parameter and the kernel set using the received input data and information of the convolutional layer (S250). A detailed description thereof will be described later with reference to FIGS. 5 to 8 .

The parallel processing unit 130 transmits the result of a general matrix multiply (GEMM) operation performed in the convolutional layer to the application unit 110 ( S260 ), and the application unit 110 performs the parallel processing unit 130 . The GEMM operation result transmitted from the GEMM is transmitted to the convolutional neural network 10 (S270).

Then, although not shown in the figure, the convolutional neural network 10 uses the operation result of the convolutional layer as an input of a back-end layer connected to the convolutional layer, for example, a fully connected layer. You can make your inferences complete.

FIG. 3 is a flowchart for explaining the optimal OpenCL parameter and kernel setting step ( S230 ) shown in FIG. 2 .

Referring to FIG. 3 , the analysis unit 120 analyzes input data and convolutional layer information ( S231 ).

In this case, the analysis unit 120 may search for a storage unit (not shown) that stores information on the convolutional layer according to the data arrangement type of the managed input layer.

That is, an identifier (ID) is assigned and stored for each convolutional layer based on the data arrangement type of the input layer and the convolutional layer information, and the OpenCL parameter and the kernel set for each identifier (ID) are mapped in the storage unit.

In this case, the data arrangement form of the input layer may include the maximum number of data of the corresponding layer, the channel of the corresponding data, the height of the corresponding data, and the width of the corresponding data.

Therefore, the analysis unit 120 analyzes the input data and the convolutional layer information, determines whether there is an identifier corresponding to the data arrangement form of the input layer and the convolutional layer in the storage unit, so that the input data and the convolutional layer information are It is determined whether it is first performed (S232).

As a result of the determination in S232, when the corresponding convolutional layer is first performed, the analysis unit 120 newly sets the optimal OpenCL parameter and the kernel through interworking with the parallel processing unit 130 (S233). At this time, the analyzer 120 automatically sets the OpenCL kernel according to the data size of the corresponding convolutional layer. That is, in order to minimize the number of memory reads when the operation of the convolutional layer is performed, the OpenCL kernel for the quantization inference operation of the convolutional layer is dynamically selected according to the size of the data.

At this time, since a performance difference may occur for the OpenCL kernel even in the same data arrangement form for each hardware platform, the analysis unit 120 compares the operation speed of the OpenCL kernel selectable through the parallel processing unit 130 at the time of initial execution to optimize OpenCL Select the kernel. That is, by comparing the execution time, the fastest parameter and kernel are set in the corresponding convolutional layer.

In this case, the set parameter storage may selectively utilize a file system or memory according to the characteristics of the hardware platform to be inferred.

In this case, the OpenCL parameter is a value for vectorization in the OpenCL kernel and may be set to an optimal value for maximizing parallelism among values 4, 8, and 16 that can be set in OpenCL.

Here, vectorization is a parallel processing function of OpenCL that can process multiple data simultaneously through one instruction, and can be used through vload n or vstore n ( n is 4, 8, 16) among OpenCL built-in functions.

4 is a diagram for explaining an operation process of a convolutional layer using vectorization.

Referring to FIG. 4 , the vectorized value n represents 4. When using the vectorization function of OpenCL, multiple elements can be accessed at once by using the vload n function for matrix B corresponding to the input data of the convolutional layer, and the output result can also be stored at once.

In addition, the OpenCL kernel configurable by the analyzer 120 may include a kernel to which only vectorization is applied and a kernel to which interleave and transpose are applied after adding padding according to the amount of computation.

At this time, the kernel to which only vectorization is applied targets data with a relatively small amount of computation, and is implemented using only vload n or vstore n . In addition, the kernel to which padding, interleaving, and order transformation are applied targets data with a relatively large amount of computation.

Referring back to FIG. 3 , as a result of the determination in S232 , if there is a history in which the corresponding convolution layer has been previously performed, the analysis unit 120 uses the OpenCL parameter and kernel preset in the convolution layer used when it was previously performed. set (S234).

FIG. 5 is a flowchart for explaining in detail the step S250 of performing the convolutional layer parallel processing shown in FIG. 2 .

Referring to FIG. 5 , the parallel processing unit 130 selectively proceeds to S252 or S253 depending on whether the OpenCL kernel applies only vectorization ( S251 ).

When the OpenCL kernel applies only vectorization, the parallel processing unit 130 performs a parallel processing operation based on the OpenCL parameter and the kernel set using the input data received from the analysis unit 120 and the information of the convolution layer ( S252).

On the other hand, if the OpenCL kernel does not apply only vectorization, that is, if it is a kernel to which padding, interleaving, and order transformation are applied, the parallel processing unit 130 determines that the matrix to be operated in the convolutional layer is not a multiple of 4. Padding is added to each matrix so as to be a multiple of 4 (S263). In this case, the step of adding padding is performed using OpenCL vectorization, and the vectorization value is set to an optimal value among 4, 8, and 16. In this case, the data storage unit in the OpenCL kernel uses char n , which is a vector type of signed 8-bit integer type, and n is determined according to the vectorized value.

Next, the parallel processing unit 130 performs interleaving (S264). That is, after dividing the entire matrix region based on the matrix size specified for the matrix A (see FIG. 4 ) storing the weights of the convolutional layer, the values are rearranged in column order in each divided region.

6 is an exemplary diagram of performing interleaving based on a 4x4 reference matrix in an 8x8 matrix.

Here, the size of the reference matrix during interleaving can be selected as 4x4, 8x8, 16x16, etc. according to the data size. The interleaving step is performed using OpenCL vectorization, and the vectorization value is set to an optimal value among 4, 8, and 16 according to the size of the reference matrix. In addition, the data storage unit in the OpenCL kernel uses char n , which is a vector type of signed 8-bit integer type, and n is determined according to the vectorized value.

Next, referring back to FIG. 5 , the parallel processing unit 130 converts the order ( S265 ). That is, after dividing the entire matrix region based on the matrix size specified for the matrix B (see FIG. 4 ) storing the input data of the convolutional layer, the values are rearranged in row order in each divided region.

7 is an exemplary diagram of performing order transformation based on a 1x4 reference matrix in an 8x8 matrix.

When converting the order, the size of the reference matrix can be selected as 1x4, 1x8, 1x16, etc. depending on the data size. The step of performing the order transformation is performed using OpenCL vectorization, and the vectorization value is set to an optimal value among 4, 8, and 16 according to the size of the reference matrix. In addition, the data storage unit in the OpenCL kernel uses char n , which is a vector type of signed 8-bit integer type, and n is determined according to the vectorized value.

Next, the parallel processing unit 130 performs a convolutional layer operation (S266). Convolutional layer operation is performed on matrices A and B on which padding addition, interleaving, and order transformation have been performed.

The parallel processing unit 130 removes the added padding from the result of performing the convolutional layer operation in order to ensure the accuracy of the result (S268). At this time, the vectorization of OpenCL is utilized, the vectorization value is set to 4, and the data storage unit in the OpenCL kernel uses int 4, a vector type of a signed 32-bit integer type (=signed int).

8 is a flowchart illustrating a quantization speculation operation according to an embodiment.

Referring to FIG. 8 , the parallel processing unit 130 quantizes input data ( S710 ). At this time, the input data is a single-precision 32-bit data type, and an operation of multiplying the input data and the activation scale variable obtained through the learning model transformation process is performed. Here, the existing single-precision 32-bit data type is converted into a signed 8-bit integer type through quantization.

The parallel processing unit 130 analyzes the information of the convolutional layer (S720). This is to perform an optimized operation according to the filter size of the convolutional layer. That is, when the horizontal/vertical size of the filter of the convolutional layer is 1, the horizontal/vertical size of the padding is 0, and the horizontal/vertical size of the stride is 1, the 1x1 filter check variable is set to true.

The parallel processing unit 130 checks whether the 1x1 filter check variable is true (S730).

If it is determined in S730 that the 1x1 filter check variable is not true, the parallel processing unit 130 performs an IM2COL (Image-to-Column) operation on the weight ( S740 ). In this case, the IM2COL operation is an operation that transforms the image data arrangement to speed up the GEMM operation in the convolutional layer, and can be implemented using a publicly available algorithm. In addition, the IM2COL operation receives 8-bit signed integer data as input data, and the output is also signed 8-bit integer data.

As a result of the check in S730, the 1x1 filter check variable is true or after performing S740, the parallel processing unit 130 performs a GEMM operation on the weights (S650).

Here, an example of the GEMM operation may be as in Equation 1 below.

<Equation 1>

C = αAB + βC (A, B, C are matrices, α, β are scalars)

In this case, the sizes of matrices A, B, and C are composed of M, N, and K. That is, matrix A may have a size of M * K, matrix B may have a size of K * N, and matrix C may have a size of M * N.

At this time, to prevent overflow of the GEMM operation result, the output of the GEMM operation result uses a signed 32-bit integer type (=int). In addition, the OpenCL kernel for performing the GEMM operation uses the kernel set by the analysis unit 120 .

The parallel processing unit 130 dequantizes the GEMM operation result (S760). In this case, dequantization performs an operation of dividing the GEMM operation result into an activation scale variable and a weight scale variable. In addition, dequantization receives a signed 32-bit integer type and outputs a single-precision 32-bit data type.

Next, the parallel processing unit 130 checks whether a bias exists in the convolutional layer (S770). If it is determined in S770 that there is no bias in the convolutional layer, the quantization inference process is terminated.

On the other hand, if bias exists in the convolutional layer as a result of checking in S770, the parallel processing unit 130 performs a GEMM operation on the bias (S780). At this time, the GEMM operation for the bias is performed based on a single-precision 32-bit data type. In addition, the GEMM operation for bias is performed using only vectorization because the amount of computation is relatively small.

9 is a diagram showing the configuration of a computer system according to an embodiment.

The convolutional neural network quantization reasoning apparatus according to the embodiment may be implemented in the computer system 1000 such as a computer-readable recording medium.

Computer system 1000 may include one or more processors 1010 , memory 1030 , user interface input device 1040 , user interface output device 1050 , and storage 1060 that communicate with each other via bus 1020 . can In addition, computer system 1000 may further include a network interface 1070 coupled to network 1080 . The processor 1010 may be a central processing unit or a semiconductor device that executes programs or processing instructions stored in the memory 1030 or storage 1060 . The memory 1030 and the storage 1060 may be a storage medium including at least one of a volatile medium, a non-volatile medium, a removable medium, a non-removable medium, a communication medium, and an information delivery medium. For example, the memory 1030 may include a ROM 1031 or a RAM 1032 .

Although embodiments of the present invention have been described above with reference to the accompanying drawings, those of ordinary skill in the art to which the present invention pertains can implement the present invention in other specific forms without changing its technical spirit or essential features. You will understand that there is Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

100: convolutional neural network quantization inference device
110: application unit 120: analysis unit
130: parallel processing unit

Claims (10)

In the convolutional neural network quantization inference method performed by the convolutional neural network quantization inference apparatus,
As the pre-trained convolutional neural network receives the characteristics and convolutional layer information of the input data to be inferred, it analyzes the characteristics of the input data and the convolutional layer information to set the OpenCL parameters and kernel optimized for the hardware platform of the embedded system Steps to do: and
Comprising the step of performing quantization inference on the convolutional layer using the set OpenCL parameter and kernel,
The step of setting the OpenCL parameter and the kernel is performed using the OpenCL data arrangement form of the input layer including at least one of the maximum number of data in the input layer, the channel of the data, the height of the data, and the width of the data. Decide whether to set new parameters and kernels,
The step of performing the quantization inference comprises:
adding padding to each matrix so that, when the OpenCL kernel does not apply only vectorization, the matrix to be operated on in the convolutional layer is not a multiple of 4;
dividing an entire matrix region based on a specified matrix size for a matrix storing weights of the convolutional layer, and then performing an interleaving step of rearranging values in each divided region in column order;
A method comprising: dividing an entire matrix region based on a specified matrix size with respect to a matrix storing input data of a convolutional layer, and performing order transformation in which values are rearranged in row order in each divided region;
performing a convolutional layer operation on matrices A and B on which padding addition, interleaving, and order transformation have been performed; and
A convolutional neural network quantization inference method, comprising removing padding added from a result of performing a convolutional layer operation. The method of claim 1, wherein the characteristics of the input data are
A convolutional neural network quantization inference method, comprising at least one of horizontal and vertical sizes of the input image or image, the number of channels of the input image or image, and arrangement order information of the channels. The method of claim 1, wherein the information of the convolutional layer,
Convolutional neural network quantization inference method, including at least one of the memory address of the input layer, the number of filters, the size of the filter, the size of the filter padding, the size of the filter stride, and whether to use a bias . The method of claim 1, wherein setting OpenCL parameters and a kernel comprises:
An identifier (ID) is assigned and stored for each convolutional layer based on the data arrangement form of the input layer and the convolutional layer information, and a storage unit to which the OpenCL parameter set for each identifier (ID) and the kernel are mapped,
analyzing the input data and the convolutional layer information to determine whether an identifier corresponding to the data arrangement type of the input layer and the convolutional layer exists in the storage unit;
setting new optimal OpenCL parameters and kernels by comparing operation speeds for selectable OpenCL kernels when input data and convolutional layer information are first performed, when the corresponding convolutional layer is first performed; and
A convolutional neural network quantization inference method, comprising the step of setting OpenCL parameters and a kernel preset in the convolutional layer used when the convolutional layer has been previously performed, if there is a history of performing the corresponding convolutional layer previously. delete The method of claim 1, wherein performing quantization inference comprises:
When the OpenCL kernel applies only vectorization, a convolutional neural network quantization inference method that performs parallel processing based on the OpenCL parameter and kernel set using the received input data and information of the convolutional layer. delete The method of claim 1, wherein performing quantization inference comprises:
A convolutional neural network quantization inference method that performs IM2COL (Image-to-Column) operation on weights when the set filter check variable is not true. The method of claim 1, wherein performing quantization inference comprises:
A quantization inference method for a convolutional neural network that performs a matrix multiplication (GEMM: General Matrix Multiply, hereinafter GEMM) operation on weights. a memory in which at least one program is recorded; and
a processor for executing a program;
program,
As the pre-trained convolutional neural network receives the characteristics and convolutional layer information of the input data to be inferred, it analyzes the characteristics of the input data and the convolutional layer information to set the OpenCL parameters and kernel optimized for the hardware platform of the embedded system Steps to do: and
performing quantization inference on the convolutional layer using the set OpenCL parameters and kernel;
The step of setting the OpenCL parameter and the kernel includes the OpenCL parameter using the data arrangement form of the input layer including at least one of the maximum number of data of the corresponding layer, the channel of the corresponding data, the height of the corresponding data, and the width of the corresponding data. and determining whether to set a new kernel,
The step of performing the quantization inference comprises:
adding padding to each matrix so that, when the OpenCL kernel does not apply only vectorization, the matrix to be operated on in the convolutional layer is not a multiple of 4;
dividing an entire matrix region based on a specified matrix size for a matrix storing weights of the convolutional layer, and then performing an interleaving step of rearranging values in each divided region in column order;
A method comprising: dividing an entire matrix region based on a specified matrix size with respect to a matrix storing input data of a convolutional layer, and performing order transformation in which values are rearranged in row order in each divided region;
performing a convolutional layer operation on matrices A and B on which padding addition, interleaving, and order transformation have been performed; and
Comprising the step of removing the added padding from the result of performing the convolutional layer operation
In convolutional neural network quantization reasoning device.

Images