KR20220041404A - Apparatus and method for compressing trained deep neural networks - Google Patents

Abstract

Disclosed are an apparatus and method for compressing a trained deep neural network for describing, analyzing, and processing multimedia content to minimize degradation in performance of an artificial neural network. According to one embodiment of the present invention, the apparatus comprises: a parameter reduction unit reducing the size of a weight matrix describing a trained deep neural network for processing multimedia content; a quantization unit quantizing the weight matrix reduced by the parameter reduction unit; and an entropy coding unit scanning the weight matrix quantized by the quantization unit in a predetermined direction and sequentially entropy-coding and outputting the scanned weights as a bitstream. The entropy coding unit encodes the scanned weights according to an encoding mode determined on the basis of characteristics of the scanned weights.

Description

Apparatus and method for compressing trained deep neural networks}

The present invention relates to an artificial neural network (ANN), and more specifically, to a technology for compression of a trained deep neural network (DNN) for description, analysis or processing of multimedia content. it's about

Attempts have been made to utilize artificial intelligence (AI) in various industries. In particular, the recent artificial intelligence technology utilizes a neural network (NN), an information processing system that has a specific performance common to a biological neural network, and its performance is greatly improved. there is.

Such a neural network (NN) is also called an 'artificial' neural network (ANN). Artificial neural networks (ANNs) are distributed parallel information processing models that mimic the behavioral characteristics of animal neurons. An ANN has a large number of nodes (also called neurons) that are connected to each other. ANNs have two characteristics: 1) Each neuron computes a weighted input from other adjacent neurons through a specific output function (also called an activation function). 2) The strength of information transmission between neurons is measured by so-called “weights”, which weights can be adjusted by self-learning of specific algorithms.

ANNs can use different architectures to specify the variables and topological relationships involved in neural networks. A parameter included in the neural network may be a weight of a connection between neurons along with activity of the neuron. There are two types of neural network topologies: a feed forward network and a backward propagation neural network. In the former, nodes in each layer connected to each other in the same layer are supplied to the next stage, and include some form of 'learning rule' that modifies the weight of the connection according to the provided input pattern. The latter allows for backward error propagation of weighted adjustments, and is a more advanced neural network than the former.

A deep neural network (DNN) can compactly express highly nonlinear and highly changing functions corresponding to a neural network having multiple levels of interconnected nodes. Nevertheless, the computational complexity for DNNs rises sharply with the number of nodes associated with multiple layers. Until recently, efficient computational methods for learning or training such a DNN have been developed. As the learning speed of DNN has increased dramatically, it has been successfully applied to various and complex tasks such as speech recognition, image segmentation, object detection, and face recognition.

Compression and decompression of multimedia content, for example, video is also one of the fields in which the application of the DNN is attempted. High Efficiency Video Coding (HEVC) as the current next-generation video coding was developed by the joint video project of the ITU-T (Video Coding Experts Group) and ISO/IEC MPEG (Movie Experts Group) standardization organizations and adopted as an international standard. It is known that by applying DNN to a new video coding standard such as HEVC, it is possible to further improve its performance. One such attempt is disclosed in Korean Patent Application Laid-Open No. 10-2018-0052651, "Method and apparatus for neural network-based processing in video coding".

However, the scale of neural networks is exploding due to rapid development in recent years. Some advanced neural network models will have hundreds of layers and billions of connections. And its implementation is both computation-centric and memory-centric.

Since the neural network is getting bigger, it is very important to make the neural network model small, that is, to compress it, in order to apply it to a device that has restrictions on storage capacity, processor performance, communication bandwidth, etc., such as a mobile terminal. In particular, in order to apply a neural network model to a video coding/playback application for production and consumption of multimedia content used as an important application in a mobile terminal, a small-sized neural network model is essential. However, until now, sufficient research has not been made with respect to a technique for compressing a neural network model and reconstructing a compressed neural network model.

Korean Patent Application Laid-Open No. 10-2018-0052651, "Method and apparatus for neural network-based processing in video coding"

One problem to be solved by the present invention is the processing of multimedia contents (multimedia technology, To provide an apparatus and method for compression of a trained deep neural network for analysis and processing).

An embodiment of the present invention for solving the above problems is an apparatus for compression of a trained deep neural network for processing multimedia content, and describes a trained deep neural network for processing multimedia content a parameter reduction unit for reducing the size of a weight matrix, a quantization unit for quantizing the weight matrix reduced by the parameter reduction unit, and and an entropy coding unit for scanning a weight matrix in a predetermined direction and then sequentially entropy-coding the scanned weights to output them as a bitstream, wherein the entropy coding unit includes: The scanned weights are encoded according to an encoding mode determined based on .

According to one aspect of the embodiment, the entropy coding unit selects an encoding mode for encoding a specific weight from a set of encoding modes, wherein the set of encoding modes includes intrinsic absolute value encoding, arithmetic sequence encoding, importance map encoding, and eigenvalue coordinate encoding. and two or more selected from average value encoding.

Another embodiment of the present invention for solving the above problem is a compression method of a trained deep neural network for processing multimedia content, for reducing the size of a weight matrix describing a trained deep neural network for processing multimedia content. A parameter reduction step, a quantization step for quantizing the weight matrix reduced in the parameter reduction step, and scanning the weight matrix quantized in the quantization step in a predetermined direction, sequentially entropy-coding the scanned weights, and outputting them as a bitstream and an entropy coding step to encode the scanned weights according to an encoding mode determined based on characteristics of the scanned weights in the entropy coding step.

According to an embodiment of the present invention, it is possible to effectively compress the trained deep neural network so that the deep neural network can be applied to the processing of multimedia content, even in devices with restrictions on storage, processor performance, and communication bandwidth, such as a mobile terminal. Do.

1 is a block diagram showing a specific configuration of a computer system in which a compression apparatus of a trained deep neural network for processing multimedia contents according to an embodiment of the present invention is implemented.
2 is a block diagram showing an example of the configuration of the deep neural network compression apparatus 152 according to an embodiment of the present invention.
3 is a diagram showing a convolutional neural network (CNN) as an example of a trained deep neural network according to an embodiment of the present invention.
4 is a flowchart showing an example of a processing process according to the pruning technique.
5 is a flowchart showing an example of a matrix decomposition process.
6 is a flowchart illustrating an example of an adaptive quantization process.
7 is a flowchart illustrating an example of a quantization process.
8A shows a layer syntax.
8B shows a quantization parameter syntax.
8C shows a quantized weight tensor syntax.
8D shows a quantized weight syntax.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments and examples of the present invention will be described with reference to the drawings. However, the following embodiments and examples are merely illustrative of preferred configurations of the present invention, and the scope of the present invention is not limited to these configurations. And in the following description, the hardware configuration and software configuration of the device, processing flow, manufacturing conditions, size, material, shape, etc. are not intended to limit the scope of the present invention to these unless specifically stated otherwise. .

1 is a block diagram showing a specific configuration of a computer system 100 in which a compression apparatus of a trained deep neural network for processing multimedia content according to an embodiment of the present invention is implemented.

Referring to FIG. 1 , the computer system 100 includes one or more processors 110 , an input/output device interface 120 , a network interface 130 , an interconnector (BUS, 140 ), a memory 150 , and a storage 160 . includes The computer system 100 may be a single specific device configured as a single computing device, or may be a set of multiple devices configured including one or more processors and one or more associated memories.

The processor 110 fetches and executes programming instructions stored in the memory 150 or the storage 160 . Similarly, the processor 110 stores or retrieves application data in the memory 150 . The input/output device interface 120 is for connecting the input/output device 12 such as a keyboard, a display, and a mouse device to the computer system 100 . The network interface 130 is for communicating with an external network such as an internal network (intranet), the Internet, or a wireless communication network through a wired or wireless communication network, and transmits data through the data communication network 14 .

The interconnector 140 performs a function of transmitting programming commands and application data between the processor 110 and the input/output device interface 120 , the storage 160 , the network interface 130 , and the memory 150 , respectively. do. Such an interconnector 140 may be one or more buses (BUS). Processor 110 may be a single central processing unit (CPU) or may be implemented as a single CPU having multiple CPUs, and in various implementations multiple processing cores. According to one aspect, the processor 110 may be a digital signal processor (DSP).

The memory 150 generally includes a random access memory such as static random access memory (SRAM), dynamic random access memory (DRAM), or flash. Storage 160 generally includes connections to hard disk drives, solid state devices (SSDs), removable memory cards, optical storage, flash memory devices, network attached storage (NAS) or storage area devices (SANs), etc. Includes non-volatile memory.

The computer system 100 may include one or more Operating Systems (OS) 164 . A portion of the operating system 164 may be stored in the memory 150 and the remaining portion may be stored in the storage 160 . Alternatively, the operating system 164 may be entirely stored in the memory 150 or may be stored in the storage 160 . The operating system 164 provides an interface between various hardware resources, such as the processor 110 , the input/output device interface 110 , the network interface 130 , and the like. In addition, the operating system 164 provides common services such as time functions for application programs.

The deep neural network compression device 152 compresses a trained deep neural network, for example, a deep convolutional neural network for description, analysis, and processing of multimedia content, and outputs it as a bitstream. The technology, analysis, and processing of multimedia contents by deep neural network are compression and restoration of video using deep neural network technology, motion detection for facial, iris, fingerprint, voice recognition, monitoring, etc. for security, diagnosis or analysis of diseases , including various applications that use multimedia contents such as moving images, still images, and voices, such as autonomous driving.

For example, an apparatus for compression or restoration of a moving picture or a coding tool constituting the same is a representative field to which processing of multimedia contents by a deep neural network can be applied. According to this, the deep neural network compression device 152 is a means for compressing the deep neural network of the learned or trained coding tool, and describing it in a compatible format. That is, the compressed deep neural network resulting from the deep neural network compression device 152 corresponds to an interoperable compressed representation of neural networks of the coding tool.

In order to compress the trained deep neural network, the deep neural network compression device 152 performs a series of processes including parameter reduction, quantization, and entropy coding for the trained deep neural network. Thus, the encoded bitstream is output. According to this embodiment, a matrix decomposition technique is used in the parameter reduction process. In some embodiments, before parameter reduction by matrix decomposition, some weights constituting the deep neural network are removed or A pruning process to simplify the connection relationship may be additionally performed.

2 is a block diagram showing an example of the configuration of the deep neural network compression apparatus 152 according to an embodiment of the present invention. Referring to FIG. 2 , the deep neural network compression apparatus 152 includes a parameter reduction unit 22 , a quantization unit 24 , and an entropy coding unit 26 . As described above, the deep neural network compression device 152 compresses the trained deep neural network (ie, weights and parameters describing the trained deep neural network) for analysis of multimedia content using a predetermined algorithm, and then, Finally, the encoded bitstream is output.

The input to the deep neural network compression device 152 includes various information and parameters describing the trained deep neural network.

First, the input to the deep neural network compression device 152 includes various high-level information. For example, when the corresponding deep neural network is applied as a coding tool for coding a video, the higher-level information may include information indicating the type of the deep neural network-based coding tool. Two types of deep trust-based coding tools may exist. More specifically, there are two types of deep neural network-based coding tools: a first type coding tool that implements functions essential to both the encoder and decoder, and a second type coding tool that implements functions essential to only one of the encoder and decoder. . This is, in image/video coding, some coding tools, i.e. type 1 coding tools, are required for both encoder and decoder, and the other part coding tools i.e. type 2 coding tools are required only for either encoder or decoder. Because it is a function. For example, in the video coding process, the in-loop filtering process corresponds to the function of the first type coding tool performed in both the encoder and the decoder, while the intra mode prediction process corresponds to the function of the second type coding tool performed only in the encoder. , and only the determined prediction mode information is sent to the decoder. Therefore, two types of DNN-based coding tools should be considered, and the types of these DNN-based coding tools must be displayed as high-level information of a trained deep neural network-based coding tool.

And, the higher level information includes information about the overall configuration of the DNN-based multimedia content processing device (hereinafter, referred to as 'DNN-based processing device'). More specifically, as high-level information related to the configuration of the DNN-based processing unit, the target application in terms of basic functions of the corresponding neural network, such as recognition, classification, generation, discrimination, etc. (target application) information, information indicating the type of DNN-based processing unit, applying a trained neural network to an inference engine when an encoder performs a specific encoding process, and applying a standardized image or video encoding tool Information indicating which of the selections is made, information on a customized content type, autoencoder, convolutional neural network (CNN), generative adversarial network (GAN), recurrent neural network (RNN), etc. Basic information about the algorithm of a trained deep neural network-based neural network, basic information about training data and/or test data, information about the capabilities required for an inference engine in terms of memory capacity and computing power, information about model compression, etc. include

The input to the deep neural network compression device 152 also includes various parameters describing the trained deep neural network. Various parameters include weights of kernels, neurons, and connections. Hereinafter, with reference to the architecture of a convolutional neural network (CNN), which is an example of a deep neural network, this will be described in more detail.

3 is a diagram showing an example of a trained deep neural network according to an embodiment of the present invention, in the case of a convolutional neural network (CNN). Referring to FIG. 3 , the CNN 200 includes an input layer 210 , a convolutional layer 215 , a subsampling layer 220 , a convolutional layer 225 , a subsampling layer 230 . , including fully connected (FC) layers 235 and 240 and an output layer 245 . In the example shown in Fig. 3, the input layer 210 is configured to accept a 32x32 pixel image, and the convolutional layer 215 generates six 28x28 feature maps from the input layer.

As described above, although the configuration of a specific CNN is shown in FIG. 3, more generally, a CNN includes one or more convolutional layers each having a sub-sampling step, and one or more fully connected layers. layer) is included. In general, the CNN architecture shown is designed to exploit the two-dimensional structure of the input image. For example, CNNs achieve this by using local concatenation and concatenated weights followed by some form of pooling. In general, compared to full chain networks with a similar number of hidden units, CNNs are easier to train and tend to have fewer parameters.

In general, a CNN includes a convolutional layer, a subsampling layer, and a fully connected layer following it. According to one embodiment, the CNN accepts an image of x×y×z from the convolutional layer as input, where x and y represent the height and width of the image, respectively, and z represents the channel in the image. For example, an RGB image has a channel of z=3. A convolutional layer may include a filter (kernel) of size a×b×c, where a×b is less than x×y and c is less than or equal to z. In general, the size of the filter k results in a locally coupled structure, which is convolved with the image to generate k feature maps. In addition, each feature map is subsampled over adjacent regions of various sizes.

Continuing to refer to FIG. 2 , the parameter reduction unit 22 performs a process of reducing a set of parameters describing the deep neural network. The parameter reduction unit 22 may apply a sparsity technique such as pruning and/or a matrix decomposition technique. The sparing technique refers to reducing the frequency of weights by setting the weights to '0' or to simplifying the neural network by cutting the connections. And the matrix decomposition technique refers to decomposing a parameter tensor of a neural network, that is, a weight matrix, into smaller-sized matrices. The parameter reduction unit 22 applies either the sparsity technique and the matrix decomposition technique successively (eg, first spars the weights and then decomposes the matrix of the sparse weights) or applies only one of the two techniques. You may.

By reducing the set of parameters describing the deep neural network by the parameter reduction unit 22, the deep neural network compression apparatus 100 as well as a device that restores and uses the deep neural network compressed thereby, for example, a mobile device, performs a multiplication operation (multiplication) and memory load can be reduced. However, since reducing the set of parameters corresponds to a lossy process, the performance of the reconstructed deep neural network is inevitably degraded to some extent. Therefore, it is preferable that the parameter reduction unit 22 can reduce the size of the weight matrix while minimizing the performance degradation of the deep neural network to be reconstructed.

In general, in the compression process of a trained deep neural network, the parameter reduction process is performed in terms of preprocessing of the quantization process, and is an arbitrary process. Accordingly, input data (parameters describing the initial deep neural network model) may be directly input to the quantization unit 24 without going through the parameter reduction unit 22 . However, when input data is input to the parameter reduction unit 22, information (metadata, etc.) related to the performance of the parameter reduction process is generated, and may be encoded together with other data to restore the neural network and included in the bitstream. . For example, information (flag) indicating whether the pruning technique and/or the matrix decomposition technique is performed together with information related to the performance of the corresponding technique may be the output of the parameter reduction unit 22 as metadata. It will be described later.

The input data to the parameter reduction unit 22 are parameters and architecture (structure definition) describing the neural network specified by a particular deep learning framework. And the output data from the parameter reduction unit 22 is also the same as the said input data. There is no particular limitation on the format (representation format) for describing the input data and the output data, but Keras/Tensorflow or Pytorch may be applied.

In one process that can be performed in the parameter reduction unit 22, the 'sparse or pruning process', the learned deep neural networks are reduced in parameter size. In the pruning, the importance of a weight in each unit, for example, one weight or a channel unit, is obtained, and a weight having a lower importance is set to '0' or a connection relationship is removed (a parameter value is set to 0).

In this pruning process, a problem arises in setting a good threshold so that each layer of neural networks prunes possible networks while maintaining the original performance. For neural networks composed of multiple layers, especially considering that a threshold of one layer may be dependent on thresholds of another layer, a brute force to find a threshold may not be practical. Also, pruning may require retraining of the network to restore its original performance. It can take a significant amount of time for the pruning process to be confirmed as efficient. As described herein, in various embodiments, automatic selection of thresholds along with methods for retraining the network may be used to prune the neural network to reduce parameters.

In order to cut out a neuron having a weak response or an unnecessary parameter in the neural network, a criterion for determining the importance of the corresponding parameter may be changed according to an algorithm. Such a criterion may be defined as an objective function, as shown in FIG. 4 . Referring to FIG. 4 , the pruning technique may proceed as follows: data type decision -> object function decision & parameter input -> pruning the decision value ( pruning decision value -> pruned NN model.

The data type determination process determines which units (individual or channel units) are to be pruned. And in the process of determining the objective function and inputting parameters, the objective function has the importance of the parameters and the objective function is defined. Parameters for defining the objective function include a weight value threshold, a scaling factor, a gradient, and the like. And the extrinsic parameters include a pruning rate and/or an arbitrary hyperparameter for re-training.

The pruning technique can be divided into two methods depending on whether or not it is learned. In the first method, an additional learning process is not required, and the pruning technique is performed based on the already learned weight values. For example, the threshold of the weight values is set in advance, the importance of the weight values is determined according to the threshold, and then pruning is performed. The second method requires an additional learning process, in which case hyperparameters and learning data required for learning must be input. In addition, the pruning technique can additionally set the compression ratio as well, and the objective function indicates the pruning technique method. For example, the second pruning method learns only a scaling factor while maintaining the learned weight value and learns only a threshold that is as close to a specified performance (task-based loss) as possible. There is a way to do it.

In relation to such a pruning technique, the above-described parameters or set thresholds may be output as metadata for restoration of a neural network, and as metadata for restoration of a neural network. For example, mask information (information indicating a position with a weight of 0) and/or threshold information additionally required in the pruning technique may be output as metadata of the pruning or sparing process. In addition, information on whether the number of parameters has actually decreased in the process (eg, a flag indicating whether the dimension value of the neural network model of reconstruction and the dimension value of the neural network model before compression are the same) It can be output as metadata of the beating or sparing process.

In the 'matrix decomposition process', which is another process that can be performed in the parameter reduction unit 22, one weight matrix, which is an initial weight matrix, is converted into N (N is an integer greater than or equal to 2) low-dimensional weight matrices. disassemble As a result, N indexes are generated for each index of one layer, and a model different from the original model is generated. Accordingly, not only can the number of weight parameters be reduced, but also a runtime reduction effect can be expected. This matrix decomposition technique is generally applicable to a fully-connected layer and a convolutional layer.

An example of the matrix decomposition technique is a low rank (Low Rank, LR) approximation process or a low displacement rank (LDR) approximation process. According to the lower rank approximation process or the lower displacement rank approximation process, the weights of the neural network expressed in a multidimensional matrix form are expressed as a plurality of matrices of lower dimensions, for example, a row matrix and/or a column matrix, thereby reducing the amount of computation. it is possible

For example, the lower rank approximation applied to the convolutional layer can be performed as follows: Convolutional weights trained in 2D-CNN are of 4D type (filter size, filter size, number of input channels, number of output channels). Equation 1 shows an approximate equation of each channel. The dimension of W_st is expressed as d x d, where d is the filter size of the convolutional layer. In addition, the dimensions of the matrices Us and Vt that can be approximated as lower ranks are expressed as d x R. where R is the target value of the size for the lower rank approximation.

According to an example of lower rank approximation, in order to reduce the data amount of weights, a 2D kernel matrix of weights in each convolutional layer may be transformed into two 1D kernel matrices.

Therefore, the objective of this method is to find (U,V) which gives the minimum difference between W _st and (U _s ×V _t ). A cost function for optimizing filter reconstruction is Equation (2).

According to another example of lower rank approximation, in order to reduce the data amount of weights, the 2D kernel matrix of weights in each convolutional layer may be transformed into three kernel matrices (eg, CANDECOMP/PARAFAC (CP) decomposition) ). That is, one convolutional weight is decomposed into three weights. The basic formula of such CP decomposition can be expressed by Equation (3).

Here, R indicates the number of components, which is a tensor rank.

In general, the weights of trained convolutional layers in 2D-CNN are 4D type. A convolutional kernel tensor K having a size of T × S × D × D includes T corresponding to the output channel, S corresponding to the input channel, and D corresponding to the kernel size.

In CP decomposition, the convolutional kernel tensor K is approximated with rank R. This can be expressed by the following Equation (4).

및

은 각각 크기

및

를 갖는 3개의 컴포넌트이다. here,

and

are each size

and

There are three components with

Accordingly, the decomposed result may be mapped as in Equation 5 below.

In the case of general convolution, convolution operation should be performed as much as the number of input channels × output channels.

In this case, a cost function for optimizing filter reconstruction is expressed in Equation 6 below.

이다.here,

am.

According to the lower rank approximation method according to this embodiment, it is possible to reduce the number of weight parameters as well as improve the operation speed.

In general, the outputs from the matrix decomposition process are matrices A and B (where matrices A and B are matrix factors that are circulant operators used for matrix decomposition), rank values, and integers. base), a reshaping mode for filter reconstruction (integer-based), and the dimensions and shape of the original weight matrix.

Among them, the reshaping mode W' may have one or a plurality of modes. As an example, the reshaping mode W' may have four modes as in Equation 7, which is merely exemplary. And, as described above, whether a mode is applied in the matrix decomposition process is reshaping mode information and must be transmitted for neural network restoration.

Equation 8 describes the weight matrix of the all-connected layer for the lower rank approximation technique.

Equation 9 describes the weight matrix of the convolutional layer for the lower rank approximation technique.

In the lower displacement approximation technique, the decomposition process is the same as the lower rank approximation technique, but the input of the decomposition process is different from the lower rank approximation technique. Equation 10 describes a lower displacement approximation technique.

Here, the f value is set arbitrarily. In addition, it can be seen that matrices A and B must also be outputs of the matrix decomposition unit in order to restore the neural network model.

The output of this matrix decomposition process may vary depending on whether the corresponding parameter matrix is for a fully concatenated (FC) layer or a convolutional layer. In addition, the output of the matrix decomposition unit may vary depending on whether the applied technique is a low rank approximation technique or a low displacement approximation technique.

An example of this is shown in FIG. 5 . If the lower rank approximation technique is applied to all connected layers (LR & FC layers), the output from the matrix decomposition process is information on whether the number of parameters is actually reduced in the process (eg, the neural network model of reconstruction). A flag indicating whether or not the dimension value of the neural network model before compression is the same), a rank value, Reshape_mode=none, the dimension of the initial matrix, etc. may be included. Instead, when the lower rank approximation technique is applied to the convolutional layer (LR & Conv layers), the output from the matrix decomposition process is information on whether the number of parameters actually decreased in the process (eg, the neural network of reconstruction). A flag indicating whether the dimension value of the model and the dimension value of the neural network model before compression are the same), rank value, Reshape_mode= 1 (or 2, 3, 4), the dimension of the initial matrix, etc. may be included. In addition, when a lower displacement approximation technique is applied to the convolution layer (LDR & Conv layers), the output from the matrix decomposition process is information on whether the number of parameters is actually reduced in the process (eg, restoration A flag indicating whether the dimension value of the neural network model of , and the dimension value of the neural network model before compression are the same), rank value, Reshape_mode= 1 (or 2, 3, 4), factor A, B, the dimensions of the original matrix, etc. may be included.

According to an embodiment of the matrix decomposition process, if the k-th layer is a convolutional layer, the initial weight matrix is a four-dimensional weight matrix. The dimension of the weight matrix can be decomposed using the number of input channels, the number of output channels, and the size of the two-dimensional filter kernel. Accordingly, the 4D weight matrix may be decomposed into a plurality of 2D weight matrices, and the size and components of the generated 2D matrix may be determined based on the reshaping mode.

With continued reference to FIG. 2 , the quantization unit 24 performs quantization on the outputs of the parameter reduction unit 22 , for example, a plurality of low-dimensional weight matrices. To this end, quantization is performed based on input quantization bits, and quantized weights are output by extracting maximum/minimum values (Max/Min values).

According to an aspect of the present embodiment, in order to reduce a quantization error by considering the distribution of parameters input for quantization, the quantization unit 24 may perform adaptive quantization. In the adaptive quantization process, a compressed integer weight file and a codebook are input, and a quantization level (r _k ) and a quantization region boundary (d _k ) may be expressed by Equation (11).

An example of such an adaptive quantization process is shown in FIG. 6 .

In the adaptive quantization process as shown in FIG. 6, if the quantization error is not sufficiently low, non-uniform quantization may be replaced with uniform quantization. According to this, the input includes a configuration file indicating the number of layers. And if the magnitude of the quantization error is larger than the uniform quantization, uniform quantization may be used instead of non-uniform quantization.

7 is a flowchart illustrating a quantization process according to another embodiment of the present invention. The quantization process shown in FIG. 7 is an addition/supplementation of the existing quantization process of storing a quantized value (integer, integer) of a weight value of a neural net in a binarized form. According to the quantization process shown in FIG. 7 , lossy coding may use other techniques as well as quantization, and may be, for example, a method of partitioning coding in a weight value matrix (with high coding efficiency (RD) (estimated by rate distortion, etc.) to determine the segmentation map). Additionally, a binary file is created by entropy coding (arithmetic coding, CABAC, palette, index map coding, etc.) which is lossless coding for quantized values. In addition, in the decoding (or decompression) process, if a binary file (bitstream) is input as an input value, reconstruction is performed. At this time, information on whether to perform the entire or only partial reconstruction of the neural net model may be included. there is.

According to one embodiment, the input to the quantization unit 24 is an item represented as a numpy array, e.g., an array of weight matrices (usually in the form of a tensor), e.g. in the format numpy.float32, all of the network It is a dictionary containing parameter tensors and their names as keys. The name of each parameter tensor must match the nomenclature specified by each model definition provided by the coordinator of the corresponding model (eg, nomenclature in the test model).

The output from the quantization unit 24 contains three keys: “parameters”, “int_parameters” and “metadata”. Here, "parameters" is a dictionary form (float-) having all parameters that are not modified during the approximation process (that is, parameters independent of the approximation process) as values and their respective names as keys. stored as a point) (eg, layer_type). These can be represented, for example, as numpy arrays in the format numpy.int32. And "int_parameters" are integer-based parameters generated through the approximation process, that is, all parameter tensors that are modified during the approximation process, their respective integer representations are items, and their respective names are Contains a dictionary represented by a key.

"metadata" includes (stores) all metadata generated during the approximation process. This metadata includes all necessary information required for decoding/reconstruction of the compressed deep neural network. That is, in the decoding/reconstruction process, it should be possible to express, for example, all numpy.int32 format parameter tensors of the network in the numpy.float32 format. Metadata may be shared by all parameters (global, global) or shared by specific parameters (eg, a codebook or step_size generated when performing quantization) or specific parameters (eg, local). For example, metadata that changes for each layer characteristic, such as the per_layer parameter). In the latter case, a separate mapping between parameter names and metadata has to be provided for the reconstruction process. In order to secure flexibility between different approximation techniques, a specific format of metadata may be arbitrarily selected according to a specific technique used. However, to ensure consistency, a prior structure is preferable.

The metadata includes, for example, step_size and codebook information generated in the quantization process. In addition, additionally necessary parameters (eg, lambda value when calculating RD or hyperparameters required when learning and finding step_size) can also be expressed as metadata.

Continuing to refer to FIG. 2 , the entropy coding unit 26 encodes each of the quantized weights and indexes according to a predetermined algorithm (eg, entropy encoding), and as a result, a compressed deep neural network bit stream is output. . According to the present embodiment, there is no particular limitation on the specific process of entropy encoding, and if it is known in the art, it may be applied without limitation, unless it is an algorithm that is essentially impossible to apply due to the characteristics of entropy encoding.

According to an embodiment, the entropy coding unit 26 may identify a data format capable of encoding and a data format of which encoding cannot be performed. All data that the entropy coding unit 26 cannot interpret (understand) is classified as "auxiliary data" and is delivered directly in the bitstream in a serialized manner. The input of the entropy coding unit 26 has the same format as the output of the quantization unit 24 . And the output from the entropy coding unit 26 is, for example, a bitstream represented by a numpy array in the numpy.float32 format. This bitstream contains information about all items in the input dictionary. Other information such as key names or dictionary structures do not need to be encoded into the bitstream.

In entropy coding unit 26, it is possible to start scanning the weight parameters from the top left, for example. That is, after scanning in a row-first manner from left to right, rows are scanned from top to bottom. Then, the quantized index values are binarized and stored in a binarized form. For example, the quantized index value may be expressed by binarizing it using the values of “Sig_flag”, “Sign_flag”, “AbsGrXFlag”, “MaxNumNoRem” and/or “RemAbs”. Here, "Sig_flag" indicates whether the index value is 0 or not. "Sign_flag" indicates the sign of a non-zero index value. "AbsGrXFlag" indicates the size of the index value. If AbsGrXFlag==1, the corresponding index value means that the index value is greater than the X value (generally expressed as an exponential power of 2). Accordingly, when the corresponding flag is 0 branch should continue to appear. "MaxNumNoRem" represents AbsGrX to be displayed as the maximum, and "RemAbs" represents the remaining values generated when the AbsGrXFlag value having the index value MaxNumNoRem is 0 in the form of a binary representation. In this case, bins representing the remaining values are coded as bypass. For example, if MaxNumNoRem==8, if the index absolute value is 5~8, the value of RemAbs should be sent. In the context model, "Sig_flag", "Sign_flag", "AbsGrXFlag", etc. used in the binarization process correspond.

According to another embodiment, the entropy coding unit 26 may adaptively encode data output from the quantization unit 24 based on an encoding mode determined based on a characteristic of the corresponding data. To this end, the entropy coding unit 26 may include an encoding mode determiner 26a for determining an encoding mode. According to an embodiment, the encoding mode determined by the encoding mode determiner 26a may be any one selected from a plurality of encoding mode sets. The plurality of encoding mode sets may be, for example, two or more selected from eigen absolute value encoding, arithmetic sequence encoding, importance map encoding, eigenvalue coordinate encoding, and average value encoding.

This embodiment of the present invention considers that various encoding methods may exhibit different compression efficiencies depending on the type of data to be encoded. To this end, the encoding mode determiner 26a of the entropy coding unit 26 may analyze the data to be encoded and then determine the encoding mode according to the analysis result.

For example, the encoding mode determiner 26a may determine the encoding mode based on data characteristics such as a dynamic range of data to be encoded. In this case, data having a very low dynamic range can be encoded using mean encoding. Data with a very small number of eigenvalues can be encoded using an eigenvalue encoding technique. For example, unique absolute value table encoding can be used as the default encoding method, and the efficiency of the unique absolute value table encoding method for specific types of data can be further increased by additionally using importance map encoding or table encoding. there is.

8A to 8D show an example of a syntax for a bitstream generated as a result of entropy coding in the entropy coding unit 26 . This bitstream syntax may be utilized by an entropy decoding unit that reconstructs (decodes) the bitstream.

More specifically, FIG. 8A shows a layer syntax. In FIG. 8A , the parameter “end_of_quant_layer_one_bit” indicates a terminating bit (“1”), and the parameter “nseting_zero_bit” is 1 bit set to 0.

8B shows a quantization parameter syntax. In FIG. 8B , the parameter “qp” indicates a quantization parameter.

8C shows a quantized weight tensor syntax.

8D shows a quantized weight syntax. In FIG. 8D , the parameter “sig_flag” indicates whether the quantized weight (QunatWeight[i]) is not 0. If the value is '0', it indicates that the quantized weight (QunatWeight[i]) is 0. The parameter "sign_flag" indicates whether the quantized weight (QunatWeight[i]) is positive or negative. If the value is '1', it indicates that the quantized weight (QunatWeight[i]) is negative. The parameter "abs_level_greater_x[j]" indicates whether the absolute value of the quantized weight (QunatWeight[i]) is greater than 'j+1', and "abs_level_greater_x2[j]" is the unary part of the exponential golomb remainder ( unary part), and the parameter "abs_remainder" indicates the remainder of the fixed length.

As described above, the above description is merely an example and should not be construed as being limited thereto. The technical idea of the present invention should be specified only by the invention described in the claims to be described later, and all technical ideas within an equivalent range should be interpreted as being included in the scope of the present invention. Therefore, it is apparent to those skilled in the art that the above-described embodiment can be modified and implemented in various forms.

Claims (3)

In the compression apparatus of a trained deep neural network for processing multimedia content,
a parameter reduction unit for reducing the size of a weight matrix describing a trained deep neural network for processing multimedia content;
a quantization unit for quantizing the weight matrix reduced by the parameter reduction unit; and
an entropy coding unit for scanning the weight matrix quantized by the quantization unit in a predetermined direction, and then sequentially entropy-coding the scanned weights and outputting them as a bitstream;
The entropy coding unit encodes the scanned weights according to an encoding mode determined based on characteristics of the scanned weights.
The method of claim 1,
The entropy coding unit selects the encoding mode to encode a specific weight in the encoding mode set,
The encoding mode set is a compression apparatus for a trained deep neural network, characterized in that it includes two or more selected from eigen absolute value encoding, arithmetic sequence encoding, importance map encoding, eigenvalue coordinate encoding, and average value encoding.
In the compression method of a trained deep neural network for processing multimedia content,
a parameter reduction step for reducing the size of a weight matrix describing a trained deep neural network for processing multimedia content;
a quantization step for quantizing the weight matrix reduced in the parameter reduction step; and
an entropy coding step for scanning the weight matrix quantized in the quantization step in a predetermined direction, entropy-coding the scanned weights sequentially, and outputting them as a bitstream;
In the entropy coding step, the compression method of a trained deep neural network, characterized in that encoding the scanned weights according to an encoding mode determined based on the characteristics of the scanned weights.

Images