KR20220143276A - Training method and training apparatus of deep learning model - Google Patents

Abstract

The present invention relates to a method and learning device for learning a deep learning model, wherein the method comprises: a step of selecting N number of layers of a learned deep learning model; a step of setting each threshold corresponding to a sparsification parameter according to the sparsification parameter, for N number of layers preceding the selected N number of layers; and a step of tuning the deep learning model using each threshold set for the preceding N number of layers. Therefore, the present invention is capable of having an effect of improving an inference speed.

Description

TRAINING METHOD AND TRAINING APPARATUS OF DEEP LEARNING MODEL

The present invention relates to a method and a learner for learning a deep learning model, and more specifically, to a method and a learner for learning a deep learning model that can increase inference speed on a mobile platform by increasing the sparsity of an activation map without reducing inference accuracy.

As deep learning technology becomes more common, the need to perform deep learning directly on a device is increasing. A wide range of devices, such as autonomous vehicles, smart cameras, robots, and smart IoT, prefer to perform deep learning directly on the device rather than sending large amounts of data to the cloud and receiving inference results.

Deep learning on device eliminates the need for network connectivity, response latency, modem power consumption, and transmission of secure data to cloud servers, so more and more devices are using deep learning (e.g., inference) directly on the device itself. are performing

Efficient on-device reasoning studies in the mobile platform of devices can be mainly classified into two directions: algorithm optimization and system optimization. As an algorithm optimization, a lightweight CNN structure such as MobileNet has been proposed. The lightweight CNN structure uses a lightweight convolution operation such as depthwise 1*1 convolution, or techniques such as pruning and quantization.

As a system optimization, research on NPU (Neueral Processing Unit: NPU) or a dedicated deep learning accelerator is continuing. NPU is fast and energy efficient, but has a small SRAM, so if the latency of DRAM is not effectively hidden, it causes a lot of loss in efficiency. Unlike CPU or GPU (Graphics Processing Unit: GPU), the development environment for custom layer for NPU users is extremely limited or not available.

As the need to perform deep learning on devices increases, it is expected that multiple deep neural networks (or models) will run concurrently on devices in the future. Therefore, it is expected that deep learning inference will continue to be performed not only on NPUs but also on GPUs in the future.

As deep learning networks (or models) become lightweight, it is becoming more and more suitable to perform lightweight deep learning models on mobile platforms (eg, smartphones, tablet PCs, etc.). Sparsity-aware reasoning enables efficient reasoning on mobile platforms. In sparsity-considered reasoning, time and energy can be saved by skipping many zero elements of a weight (or filter) in the convolution operation as a result of weight pruning.

According to a recent study, it is known that the pruned sparse model has the same or higher accuracy than the dense model that does not consider the operation according to the sparsity.

Unlike NPUs dedicated to deep learning models, GPU performance is mainly affected by uncoalesced memory accesses and divergent branches. When the processing elements (PE) of the GPU individually access arbitrary locations, the performance of the GPU deteriorates significantly. This is also why sparsity-considered inference is not actively tried on GPUs.

Also, in addition to weight pruning, a lot of sparseness is found in layer-by-layer feature maps (feature maps or activation maps), since activation functions such as ReLU change negative numbers of convolution or pooling function outputs to zero.

Therefore, it is very important to efficiently utilize the sparsity of the deep learning model in the mobile GPU constituting the mobile platform. A typical GPU employs a SIMD (Single Instruction Mule Data) structure to implement a large amount of fine-grain data parallelism, so there are many problems in efficiently using unstructured sparsity.

It is known that studies on the sparseness of the weights of the layers determined in the learning stage of a deep learning model can be efficiently reused in the inference stage.

On the other hand, there is no known study for efficiently dealing with activation sparsity caused by activation or activation functions such as the ReLU function. Unlike the weight fixed in the learning phase, although the ReLU function converts more than 50% of the elements to zero, due to the non-normality of the feature map or activation map generated by the ReLU function dynamically changing according to the input data of the inference It is never easy to deal with this efficiently.

As such, a deep learning model inference method is needed to efficiently handle the sparsity of the activation map of the deep learning model.

The present invention has been devised to solve the above problems, and the purpose of the present invention is to provide a deep learning model learning method and a learner that can improve the inference speed of a deep learning model performed on a mobile platform without lowering inference accuracy. have.

In addition, an object of the present invention is to provide a deep learning model learning method and learner capable of improving the inference speed in the GPU or NPU of a mobile platform by maximizing the sparsity of the activation map of the deep learning model.

In addition, the present invention provides a deep learning model learning method that can improve the inference speed on a mobile platform by increasing the sparsity ratio of a plurality of layers without reducing accuracy by using the correlation between the execution time for each layer of the deep learning model and the sparsity ratio and to provide a learning machine.

The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the description below. will be able

A deep learning model learning method according to an aspect of the present invention includes the steps of (a) selecting N layers of a trained deep learning model, (b) for N layers preceding the selected N layers, to the sparsity parameter. and (c) tuning the deep learning model using each threshold set for the preceding N layers, wherein N is an integer greater than or equal to 1, respectively.

In the deep learning model learning method described above, further comprising calculating the accuracy of the tuned deep learning model and determining whether the calculated accuracy is greater than or equal to a threshold accuracy, when the calculated accuracy is less than the threshold accuracy, learning the deep learning model The method reduces the sparing parameter and repeats steps (b) and (c).

In the deep learning model training method described above, when the calculated accuracy is equal to or greater than the threshold accuracy, the deep learning model training method increases N by 1 and performs steps (a) to (c) from the sparsity parameter corresponding to the calculated accuracy. Repeat.

In the deep learning model learning method described above, after performing steps (a) to (d), storing the tuned deep learning model and the corresponding tuning parameters, and performing inference from the input data using the tuning parameters The tuned deep learning model performs the activation function of the preceding N layers using a threshold set through a tuning parameter.

In the deep learning model learning method described above, step (a) selects N layers based on the execution time of all layers according to profiling of the learned deep learning model.

In the above-described deep learning model learning method, step (a) is to order each of the entire layers according to the execution time and select the N layers having the longest execution time or to reduce the execution time according to the sparsity of each of the entire layers. Accordingly, all layers are ordered and N layers with the highest reduction in execution time are selected.

In the above-described deep learning model learning method, step (b) is performed in each of the N layers preceding each input value corresponding to the sparing parameter in each activation function input distribution according to the profiling of the preceding N layers. set to the threshold of

In addition, the deep learning model learner according to an aspect of the present invention includes a control unit for executing a command of the program, a deep learning model learning program and a storage unit for storing the learned deep learning model, and a control unit for performing a deep learning model learning program selects N layers of the trained deep learning model, sets thresholds corresponding to the sparsation parameters according to the sparsization parameters for N layers preceding the selected N layers, respectively, and sets the N layers preceding the selected N layers. Tune the deep learning model using each threshold set for , where N is an integer greater than or equal to 1.

In the deep learning model learner, the control unit calculates the accuracy of the tuned deep learning model, determines whether the calculated accuracy is greater than or equal to a threshold accuracy, and reduces the sparing parameter when the calculated accuracy is less than the threshold accuracy, For the preceding N layers, a corresponding threshold is set according to the reduced sparsity parameter, respectively, and the tuning of the deep learning model is repeated using each threshold set to be reduced for the preceding N layers.

In the above-described deep learning model learner, when the calculated accuracy is greater than or equal to the threshold accuracy, the control unit increases N by 1, selects N layers increased by 1 of the learned deep learning model, and N data preceding the N layers Thresholds corresponding to the sparing parameters corresponding to the calculated accuracy for the layers are respectively set, and the tuning of the deep learning model is repeated using the respective thresholds set for the preceding N layers.

In the deep learning model learner described above, the control unit stores the tuned deep learning model and the corresponding tuning parameters in the storage unit, and the tuned deep learning model that performs inference from the input data using the tuning parameters is performed through the tuning parameters. The activation function of the preceding N layers is performed using the set threshold.

In the deep learning model learner, the control unit orders each of the entire layers according to the execution time according to the profiling of the learned deep learning model and selects N layers having the longest execution time or each of the entire layers All layers are ordered according to the reduction in execution time according to the sparsity, and N layers with the highest reduction in execution time are selected.

In the above-described deep learning model learner, the control unit sets each input value corresponding to the sparing parameter in each activation function input distribution according to the profiling of the preceding N layers to each threshold of the preceding N layers. set

The deep learning model learning method and learner according to the present invention as described above has the effect of improving the inference speed of the deep learning model performed on the mobile platform without degrading the inference accuracy.

In addition, the deep learning model learning method and learner according to the present invention as described above has the effect of maximizing the scarcity of the activation map of the deep learning model to improve the inference speed in the GPU or NPU of the mobile platform.

In addition, the deep learning model learning method and learner according to the present invention as described above uses the correlation between the execution time for each layer of the deep learning model and the sparsity ratio to increase the sparsity ratio of a plurality of layers without lowering accuracy in the mobile platform. It has the effect of improving the inference speed.

The effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those of ordinary skill in the art from the following description. will be.

1 is a diagram illustrating an exemplary block diagram of a deep learning model learner.
2 is a diagram illustrating an exemplary internal structure of a deep learning model.
3 is a diagram illustrating a main process for tuning a deep learning model.
4 is a diagram illustrating a control flow for tuning a deep learning model.
5 is a diagram illustrating an exemplary pseudo code for tuning a deep learning model.
6 is a diagram illustrating an example of an input distribution in specific activation functions.
7 is a diagram illustrating an exemplary block diagram of a deep learning model reasoner.
8 is a diagram illustrating an example of an algorithm for generating an auxiliary data structure.
9 is a diagram illustrating an example of performing a convolution operation using an auxiliary data structure.

The above-described objects, features and advantages will become more clear through the detailed description described below in detail with reference to the accompanying drawings, and accordingly, those of ordinary skill in the art to which the present invention pertains can understand the technical spirit of the present invention. can be easily implemented. In addition, in the description of the present invention, when it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating an exemplary block diagram of a deep learning model learner 100 .

Referring to FIG. 1 , the deep learning model learner 100 includes a communication unit 110 , a storage unit 150 , and a control unit 190 . The block diagram of Fig. 1 preferably shows a functional block diagram, each functional block having a corresponding hardware block. For example, the deep learning model learner 100 may be configured as a personal computer, a laptop computer, or a server. The deep learning model learner 100 may further include other blocks not shown in FIG. 1 .

Looking at the deep learning model learner 100 through FIG. 1 , the communication unit 110 includes a communication chipset and an antenna for communication such as wireless LAN, wired LAN, optical LAN, etc., and various data through a broadband network such as a local network or the Internet. send and receive For example, the communication unit 110 receives the (pre)trained deep learning model 300 from other devices, devices, devices, etc. through the Internet or a local network, or tunes the (pre)trained deep learning model 300 . One deep learning model 300 may be transmitted to another device, apparatus, device, or the like. The communication unit 110 may transmit the tuned deep learning model 300 to a mobile device such as a smartphone or a tablet PC.

The storage unit 150 stores various data and programs. The storage unit 150 includes a volatile memory, a non-volatile memory and/or a mass storage medium such as a hard disk, at least a deep learning model learning program for tuning the deep learning model 300 and a pre-trained deep learning model and learning Stores the deep learning model 300 tuned for the deep learning model.

The controller 190 controls the deep learning model learner 100 . The control unit 190 is configured to execute the command of the program, and to perform the deep learning model learning program of the storage unit 150 to tune the deep learning model 300 previously learned according to the present invention. The controller 190 may generate and store the deep learning model 300 and tuning parameters tuned for the deep learning model through the execution of the deep learning model learning program in the storage unit 150 .

The control unit 190 for executing the program instruction includes a first processing unit 191 and a second processing unit 195 . The first processing unit 191 may be, for example, a CPU, NPU, processor, etc. and the second processing unit 195 may be a GPU or the like.

The control unit 190 for performing the deep learning model learning program executes the deep learning model 300 (program of) learned in the first processing unit 191 and the second processing unit 195, and various information according to profiling to acquire and tune the deep learning model according to the acquired information, generate the tuned deep learning model and tuning parameters, store it in the storage 150, and then transmit the tuned deep learning model (and tuning parameters) to a mobile device, etc. Or it can be mounted on a mobile device or the like.

At least, the controller 190 tunes to increase the sparsity of the three-dimensional (or more) tensor generated in the layers 310 of the first-learned deep learning model 300 . In the tuning process, the output tensor of the specific layer 310 is set to have 0 greater than or equal to the set sparsity ratio. In addition, the controller 190 tunes the weight (or filter) of each layer 310 so that there is no loss of accuracy or is set within a certain ratio even when the rarity is forcibly increased in the layer 310 .

According to tuning for sparsity, as the deep learning model 300 is sparse, the deep learning model reasoner 200 performing the deep learning model 300 is dynamically changed or changed according to the input data for inference. Inference speed can be improved by using a sparse tensor.

A detailed control flow performed in the deep learning model learner 100 (controller 190 of) in relation to tuning will be looked at in more detail through FIGS. 2 to 5 .

2 is a diagram illustrating an exemplary internal structure of the deep learning model 300 .

According to FIG. 2 , the deep learning model 300 includes a plurality of layers 310 therein, and each layer 310 performs a convolution operation, an activation operation, and a pooling operation. It generates and outputs each input tensor as an output tensor. One or more specific layers 310 may be configured by omitting a pooling operation.

The deep learning model 300 is any known or to be known deep learning model, and may be, for example, MobileNet, ResNet, Inception, or the like.

Each deep learning model may have a different number of layers 310 according to a design example, and each layer 310 includes a convolution between an input tensor and weights and a subsequent activation operation 319 and further An output tensor generated through the pooling operation 315 is provided as an input tensor of subsequent layers 310 .

The number of weights (or filters) of each layer 310 may be different (see L, M, N in FIG. 2 ), and the subsequent layers 310 by performing convolution, activation, and pooling functions for each weight. ) to output a two-dimensional array (tensor) as many as the number of weights.

The execution time of the deep learning model 300 is mainly consumed by the convolution operation 311 . The convolution operation 311 requires a lot of time because it consists of the product between the data element of the input tensor and the data element of the weight and the dot product of the products. While the weight is fixed and used in the reasoner 200 , the input tensor and the output tensor are generated based on input data and have variable characteristics in the reasoning process.

When the majority of input tensors that are input factors to the convolution operation 311 of the layer 310 are 0, the multiplication operation may be omitted from the convolution operation 311, thereby improving the inference speed.

The input tensor provided to a specific layer 310 (see ⓐ in FIG. 2 ) is the output tensor of the immediately preceding layer 310 (see ⓑ in FIG. 2 ), and the output tensor of the immediately preceding layer 310 is the activation of the immediately preceding layer 310 . It is created through a function (operation). The output tensor of the immediately preceding layer 310 is generated by an activation function or a subsequent pooling function with an activation function.

The activation function of the activation operation 319 is a function for filtering the convolution data after the convolution operation 311, and may be, for example, a ReLU function or a sigmoid function. If the convolutional data input is less than 0, the ReLU function sets it to 0 and outputs it. The output tensor output through the activation function has zeros within 50% of the profiling result, and even output tensors with zeros over 90% are also found.

When the output tensor output through the activation function (and furthermore the pooling function) has a high ratio of zeros, the product operation between the zero element and the weight element of the input tensor in the convolution operation 311 of the subsequent layer 310 may be omitted, which may provide high-speed inference performance in the second processing unit 293 and/or the third processing unit 295 including the NPU or GPU. Moreover, it would be highly desirable to provide high-speed inference performance without compromising accuracy.

3 is a diagram illustrating a main process for tuning the deep learning model 300 .

The main process of tuning in FIG. 3 is performed by the deep learning model learner 100 and preferably through the control unit 190 for performing a deep learning model learning program or other program.

First, the deep learning model learner 100 (control unit 190 of) profiles (S1) the deep learning model 300 in which primary learning has already been performed. The control unit 190 for performing a deep learning model learning program performs a deep learning model on a test set in the first processing unit 191 and the second processing unit 195 according to the program of the deep learning model, and deep learning The execution time of each layer 310 of the model is measured. In addition, the controller 190 measures an input distribution input to the activation function of each layer 310 .

The controller 190 may measure an average execution time of each layer 310 by performing a deep learning model on a plurality of test sets (eg, 256 test sets). In addition, the controller 190 measures an input distribution input to the activation function of each layer 310 and configures an input distribution table.

The controller 190 may configure an input distribution by ordering input data input to the activation function of each layer 310 for each test set according to sizes. Also, the controller 190 may determine the input distribution by averaging the input distribution for each test set.

The input distribution table can be set with the input values corresponding to the lower N % of the distribution, for example, the input values corresponding to the lower 50%, the lower 51% in the total input distribution (the order of the total input data used for the test). The corresponding input value may be set as an input value corresponding to the lower N% (N is an integer less than or equal to 99). It is confirmed that the input distribution of each layer 310 is approximately the same or extremely similar even during real-time inference according to the filtering characteristics of the layer 310 .

The input distribution of the activation function of each layer 310 may vary according to the position of each layer 310 , characteristics of the preceding and subsequent layers 310 , and the like. For example, a specific layer 310 may have a value between -150 and +250 (refer to (a) of FIG. 6) and another specific layer 310 may have a value between -5 and 5 (see FIG. 6(a)). See (b)) The shape of the distribution may also be different. According to the profiling, the controller 190 may store the execution time corresponding to each layer 310 and the distribution table of each layer 310 in the storage 150 .

Furthermore, the control unit 190 uses the distribution table of each layer 310 generated first to sparsize the output of the activation function (activation operation 319) of each layer 310, and to reduce it accordingly. The execution time can be measured.

For example, the control unit 190 sets the threshold value for setting the activation function of each layer 310 to a specified sparing ratio other than 0 according to the distribution table of each layer 310 . The control unit 190 sets the input value corresponding to the specified sparsity ratio (eg, 90%) in the distribution table of each layer 310 as a threshold value, then performs the deep learning model 300 and sets the threshold value accordingly. The execution time of the layer 310 following the set activation function is measured.

The control unit 190 may match the execution time reduction rate between the execution time before sparsity and the execution time after the activation cancellation to each layer 310 to further store it in the storage unit 150 or store it instead of the execution time.

After profiling the deep learning model in order to improve the performance speed of the deep learning model 300 through sparing and maintain the performance accuracy, the deep learning model learner 100 tunes the first pre-trained deep learning model ( S2).

The control unit 190 fine-tunes the deep learning model previously trained with the test data set over a plurality of epochs in consideration of the application of sparsity. Fine tuning (or transfer learning) is performed by the control unit 190 performing a deep learning model learning program.

4 is a diagram illustrating a control flow for tuning (S2) of the deep learning model 300, and FIG. 5 is a diagram illustrating an exemplary pseudo code for tuning the deep learning model 300.

The pseudo code of FIG. 5 is a type of code representing the control flow of FIG. 4 , and codes corresponding to some control flows may be partially different. The control flow of FIG. 4 and the pseudo code of FIG. 5 are performed by the deep learning model learner 100 and preferably by the control unit 190 performing a deep learning model learning program.

First, the deep learning model learner 100 (controller 190 of) initializes the N value, which is the number of the sparsation parameters and the number of layers 310 selected for sparsation (S201). The controller 190 initializes the sparsity parameter to 99% and initializes the N value to 1 (refer to ① in FIG. 5 ). Also, the controller 190 may calculate in advance (base_acc in FIG. 5 ) the (reference) accuracy of the learned deep learning model 300 .

Then, the deep learning model learner 100 (control unit 190 of) selects N (N is an integer greater than or equal to 1) layers 310 among all the layers 310 of the learned deep learning model 300 ( S203) (refer to ② in FIG. 5). The controller 190 selects N layers 310 based on the execution time of all layers 310 of the deep learning model according to profiling (see S1 of FIG. 3 ) for the learned deep learning model.

The control unit 190 orders the corresponding execution times of the layers 310 stored in the storage unit 150 according to the profiling from the longest time to the shortest time, and N layers 310 having the longest execution time. can be selected. Alternatively, the controller 190 orders the execution time reduction rate of each of the layers 310 stored in the storage unit 150 from highest to lowest according to the profiling and N layers 310 having the highest execution time reduction rate. can be selected.

The deep learning model learner 100 (control unit 190 of) for the N layers 310 preceding the selected N layers 310 (eg, in FIG. 2 , when the ⓐ layer 310 is selected) According to the set sparing parameter (for the ⓑ layer 310 that precedes the ⓐ layer 310 ), threshold values corresponding to the sparing parameters are respectively set ( S205 ) (see ③ in FIG. 5 ).

The control unit 190 controls each threshold value of the N layers 310 preceding each input value corresponding to the currently set sparing parameter in each activation function input distribution according to the profiling of the preceding N layers 310 . set to

The controller 190 sets thresholds (input values) matching the values of the sparsization parameters in the distribution table stored in the storage 150 of each of the preceding N layers 310 as the respective thresholds. Accordingly, the threshold of each preceding layer 310 is set differently according to the input distribution of the preceding layer 310 .

The deep learning model learner 100 (control unit 190 of) uses different thresholds corresponding to the same sparing parameters according to the input distribution set for the preceding N layers 310 to the deep learning model 300. Tune (S207) (refer to ④ in FIG. 5).

In the tuning process using different thresholds, the control unit 190 learns the deep learning model 300 by reflecting the sparsity of specific N layers 310, and updates the weights of all layers 310 with feedback according to the learning. do.

The deep learning model learner 100 (control unit 190 of) calculates the accuracy of the tuned deep learning model (S209) (⑤ in FIG. 5) and compares the calculated accuracy with the threshold accuracy (S211) (⑥ in FIG. 5) see).

The deep learning model learner 100 (control unit 190 of) calculates the accuracy of the threshold accuracy based on the base accuracy of the deep learning model 300 before tuning (ratio or difference to the base accuracy of the deep learning model before tuning) less than In this case, the set sparsity parameter may be decreased ( S213 ), and steps S205 and lower may be repeatedly performed for the reduced sparsity parameter.

For example, the control unit 190 reduces the currently set sparing parameter by a specified offset (eg, 1 %) to the deep learning model 300 by sparing the preceding layer 310 according to the reduced sparing parameter. can be retuned and the accuracy can be calculated.

If the calculated accuracy is greater than or equal to the threshold accuracy, the deep learning model learner 100 (controller 190 of) stores the tuned deep learning model 300 and the tuning parameters in the storage unit 150 (S217) (FIG. 5) Refer to ⑦). The control unit 190 controls the array of weights of the layers 310 that are changed according to tuning, the layer IDs selected according to N where the sparsity is made, and the threshold value of the activation function of the N layers 310 immediately before the selected layer 310 ( threshold according to the input distribution) may be stored as a tuning parameter or included in the deep learning model 300 .

Other deep learning models may further include a modified layer program-module used in the layer 310 in which speed is improved according to sparsity. The input tensor of the selected layer 310 and the output tensor output from the selected layer 310 are changed in consideration of the sparing according to the specified sparsization parameter, or the layer program-module so that the layer program-module can be changed or the sparsation element can be specified. it is composed

According to the calculated accuracy is greater than or equal to the threshold accuracy, following the storage of the tuned deep learning model 300 and the tuning parameters, the deep learning model learner 100 (controller 190 of) increases N by 1 (S219), The process below step S203 is repeatedly performed from the sparsity parameter (refer to ⑦ in FIG. 5) corresponding to the calculated accuracy equal to or greater than the threshold accuracy.

When the reduced sparsity parameter is the minimum parameter (eg, 50%) (S215), the deep learning model learner 100 (controller 190 of) ends the control flow of FIG. 4 (S250) and final tuning The stored final deep learning model 300 and the corresponding final tuning parameters may be stored in the storage unit 150, and the stored final deep learning model 300 and the final tuning parameters may be provided to a mobile device or the like.

The tuned deep learning model 300 performed by the control unit 290 such as a mobile device uses different thresholds according to the input distribution set through the provided tuning parameters, and N preceding the N layers 310 are selected. configured to sparse the activation function of the layer 310 .

The last stored tuning parameter increases the sparsity rate by more than a specified ratio (spacing parameter) in the activation function of the layer 310 immediately before the maximum number of N selected layers 310 determined according to FIGS. 4 and 5, The performance speed of the N selected layers 310 affected by the performance speed may be improved according to the sparsity. The tuned final deep learning model 300 will be described in more detail in the descriptions of FIGS. 7 to 9 related to the reasoning machine 200 .

7 is a diagram illustrating an exemplary block diagram of a deep learning model reasoner 200 .

According to FIG. 7 , the deep learning model reasoner 200 includes an input unit 210 , a communication unit 230 , a storage unit 250 , an output unit 270 , and a control unit 290 . According to a design example, the deep learning model reasoner 200 may be configured by further including other blocks or omitting some blocks of FIG. 7 .

The deep learning model reasoner 200 of FIG. 7 shows an example of a mobile device such as a smart phone or a tablet PC. The deep learning model reasoner 200 uses the internal GPU and/or NPU instead of requesting inference to the server through the communication network and receiving the inference response according to the request, including the GPU and/or NPU therein. It is configured to be able to infer according to the directly tuned deep learning model 300 .

The deep learning model reasoner 200 according to the present invention sparses the input tensor (or the output tensor) that dynamically changes for each layer 310 dynamically according to the input data to improve the inference performance according to the sparsity accuracy. can be improved without a decrease in

Referring to the deep learning model reasoner 200 through FIG. 7 , the input unit 210 receives a user input of the deep learning model reasoner 200 . The input unit 210 receives various user inputs including a button, a touch screen, a touch pad, a microphone, and the like.

The communication unit 230 transmits and receives various data through the Internet or a broadband network of a mobile communication network, including a communication chipset and an antenna for connection to a wireless LAN or a mobile communication network. The communication unit 230 may receive the tuned deep learning model 300 and tuning parameters from the deep learning model learner 100 or another server.

The storage unit 250 stores various data and programs. The storage unit 250 stores at least a deep learning program for performing the tuned deep learning model 300 using a tuning parameter including a volatile memory, a nonvolatile memory and/or a cache. The deep learning program may be a deep learning model itself or a program capable of executing the deep learning model.

The deep learning program may be received from the deep learning model learner 100 or another server built into the deep learning model reasoner 200 (mobile device of) during production or through the communication unit 230 . The deep learning program may be a program executed in the background or a program driven according to a user input through the input unit 210 .

The storage unit 250 may store data (eg, images) used as input data to other deep learning programs.

The output unit 270 outputs various audio and/or video signals. The output unit 270 may output an audio signal or a video signal generated by the control unit 290 including a display, a speaker, an LED, and the like. The output unit 270 may convert the inference result generated by the control unit 290 into an audio or video signal and output it.

The controller 290 controls the deep learning model reasoner 200 . The control unit 290 is configured to execute the command of the program, and is configured to infer the input data according to the deep learning model 300 and the tuning parameters tuned by executing the deep learning program of the storage unit 250 .

The tuned deep learning model and tuning parameters at least sparse the output (tensor) of the layer 310 preceding the selected specific layer 310 by more than a certain percentage (for example, 90%, etc.) It is possible to improve the overall inference speed by improving the execution speed of the selected specific layer 310).

The control unit 290 for executing the instructions of the program includes a first processing unit 291 and further includes a second processing unit 293 and/or a third processing unit 295 . The first processing unit 291 may be a CPU, an MPU, a processor, or the like. The second processing unit 293 may be a GPU or the like. The second processing unit 293 is a mobile GPU used in a mobile device, and may be, for example, a Mali GPU or an Ardeno GPU.

Mali GPU or Ardeno GPU has an internal SIMD structure and is tied according to a scheduling unit called a warp or wavefront. If there is a divergent branch), performance is degraded due to thread waiting or synchronization.

In addition, when adjacent threads refer to a memory in a location adjacent to each other, the mobile GPU can access a memory of a certain unit size instead of an individual memory access, thereby increasing efficiency. The deep learning model 300 needs to be configured in consideration of the characteristics of the mobile GPU.

The third processing unit 295 may be an NPU. The control unit 290 for performing the deep learning program dynamically or statically transfers the specific layer 310 to the second processing unit 293 and/or the third processing unit 295 according to the control procedure of the tuned deep learning model. Inference can be performed on the input data by assigning

Referring to the performance of the deep learning model reasoner 200 of the deep learning program in more detail through FIG. 2 , the internal scheduler (control unit 290) of the deep learning program is sequentially layered with respect to the input data for inference. perform processing. The internal scheduler may process a different number of layers configured according to the deep learning model, and the deep learning program may output an inference result to the output unit 270 . The control unit 290 executing the deep learning program may process the layer through the deep learning program and output the result.

The deep learning program of the tuned deep learning model 300 is configured to increase the inference speed by using the sparsity of the tensor input to the layer 310, and in particular, to the output tensor of the preceding layer 310 The inference speed can be further improved by increasing the sparsity ratio.

The internal scheduler of the deep learning program sequentially executes the layers 310 and provides the output tensor generated in the previous layer 310 as the input tensor of the subsequent layer 310 . For example, the deep learning program provides a tensor output by application according to the weight array L in the layer 310 of FIG. 2 as an input tensor to the next array, ⓐ of FIG. 2 . In addition, according to the sequential execution of the layer 310, a tensor output by application according to the weight array N in the ⓒ layer 310 of FIG. 2 is provided as the input tensor of ⓓ of FIG. 2, which is the next array.

Hereinafter, the layer 310 that generates the auxiliary data structure representing the zero data element among the data elements included in the tensor output from the deep learning model tuned through the deep learning program of the controller 290 is referred to as a 'generation layer' Hereinafter, a layer 310 that processes a layer operation in consideration of scarcity by using the auxiliary data structure of the generation layer 310 of the previous stage is referred to as a 'producing layer' is hereinafter referred to as a 'consumer layer'.

In FIG. 2 , the ⓑ layer 310 is a generation layer for the ⓐ layer 310 , and the ⓐ layer 310 is a consumption layer that consumes the tensor output from the ⓑ layer 310 . In addition, the ⓒ layer 310 is a generation layer for the ⓓ layer 310 , and the ⓓ layer 310 is a consumption layer that consumes the tensor output from the ⓓ layer 310 .

In the tuned deep learning model, all layers 310 except for the first and last layers may be a generation layer 310 and a consumption layer 310 . Alternatively, a specific one or more selected layers 310 among all the layers 310 may be configured as the consumption layer 310 , and one or more layers 310 preceding the selected layer 310 may be configured as the generation layer 310 .

Consumption layers 310 of the tuned deep learning model 300 are selected according to the execution time measured according to profiling among the layers 310 of the deep learning model in the deep learning model learner 100 ( 310) ( 4).

4 , the tuning parameter is stored and provided by changing the threshold to have a sparseness of a predetermined ratio or more in the generation layer 310 preceding the selected consumption layer 310 .

In addition, the deep learning model learner 100 changes the program (module) of the generation layer 310 to generate an auxiliary data structure in the generation layer 310 of the deep learning model 300 and assists in the consumption layer 310 The program (module) of the consumption layer 310 is changed to increase the layer execution speed by utilizing the data structure.

Looking more specifically through the example of FIG. 2, the tuned deep learning model of FIG. 2 includes, for example, two consumption layers 310 (ⓐ, ⓓ) and two generation layers 310 (ⓑ, ⓒ). have The deep learning model reasoner 200 (controller 290 of) that performs a deep learning program according to the tuned deep learning model has an output tensor and data in the output tensor in one preceding generation layer 310 (ⓑ). An auxiliary data structure indicating whether an element is 0 or not is created (refer to ① in FIG. 2).

The generation layer 310 (ⓑ) is internally configured to sequentially perform a convolution operation 311 and an activation operation 319 and further a pooling operation. The convolution operation 311 is performed with respect to the generation layer 310 using a weight array L tuned in consideration of the sparseness of the input and output tensors, and the activation operation 319 is the output of the generation layer 310 . An activation function according to a threshold set according to the input distribution after the convolution operation 311 of the corresponding generation layer 310 (ⓑ) is performed so as to have 0 equal to or greater than the sparsity ratio set in the tensor. The threshold applied to the activation function of the generation layer 310 (ⓑ) is set differently according to the threshold applied to the activation function of another generation layer 310 (eg, ⓒ) and the input distribution thereof.

The activation function loads (uses) a threshold value corresponding to the generation layer 310 (ⓑ) of the tuning parameter and sets the convolution data element output from the convolution operation 311 as an output tensor by using the threshold value. For example, the activation function sets convolutional data elements below (less than) a threshold to zero in the output tensor. The activation function, which is a ReLU function, generates an output tensor with a high ratio of zeros by filtering the convolutional data with a threshold greater than or equal to zero to have a high ratio of zeros in the output according to its input distribution.

In addition, the activation function or a specific function following the activation function (for example, a pooling function or an arbitrary function) is a tensor output from the activation function or a pooling function (when a pooling operation exists in the corresponding layer 310). It stores data indicating whether each data element is zero or not in a tensor as an auxiliary data structure.

FIG. 8 is a diagram illustrating an example of an algorithm for generating an auxiliary data structure, wherein the algorithm of FIG. 8 is performed in an activation function or after an activation function (eg, a pooling function or any function).

FIG. 8(a) shows an example of generating a bit-mask array having an output tensor size, and FIG. 8(b) shows an example of an index table storing an index indicating the position of a non-zero data element.

In Fig. 8(a), the activation function or the subsequent function is a bit-mask depending on whether 0 at each data element position of the output tensor in an array of the same dimension as the sparse output tensor (nz_mask in Fig. 8(a)). 0 or 1 is written to the corresponding position in the array (see lines 10 and 11 in Fig. 8(a)).

In (b) of FIG. 8, the activation function or the subsequent function determines the position representing the non-zero data element according to whether it is 0 at each data element position of the output tensor in the index table (nz_idx, 10, 11 in Fig. 8(b)). line) is recorded.

The control unit 290 stores the output tensor generated in the generation layer 310 (ⓑ) and an auxiliary data structure indicating whether the data element is 0 in the storage unit 250 .

Subsequent to the generation of the output tensor and auxiliary data structure in one generation layer 310 (ⓑ), the deep learning model reasoner 200 (controller 290 of) performing a deep learning program is the generation layer 310 In the consumption layer 310 (ⓐ) following (ⓑ), an operation is performed on the tensor from the preceding generation layer 310 using the auxiliary data structure generated in the preceding generation layer 310 (see FIG. 2 ). ②).

The consumption layer 310 (ⓐ) is configured to sequentially perform a convolution operation 311 , an activation operation 319 , and further (additionally) a pooling operation 315 therein. The convolution operation 311 applies a sparsity ratio to generate a convolution operation 311 between the tensor output from the generation layer 310 and the weight array M of the consumption layer 310 (ⓐ) generation layer 310 It is performed using an auxiliary data structure that is dynamically output from

The consumption layer 310 (ⓐ) is configured to perform the convolution operation 311 in consideration of a high proportion of zero data elements of the input tensor.

FIG. 9 is a diagram illustrating an example of performing the convolution operation 311 by using the auxiliary data structure provided from the generation layer 310 in the consumption layer 310 (ⓐ).

9A illustrates an example of performing the convolution operation 311 according to the mask value of the data element position of the bit-mask array provided from the generation layer 310 . When the mask value of the bit-mask array is 1, the consuming layer 310 (ⓐ) performs the convolution operation 311, and when the mask value is 0, the convolution operation 311 is skipped. 9(a) has the advantage that it can be processed using the representation of the existing matrix as it is without conversion into a representation considering the sparse matrix between the layers 310, whereas the conditional branch in the GPU, etc. As a result, some performance degradation may occur.

9B is an example of performing a convolution operation 311 between a data element of a non-zero input tensor and a weight array using the index table provided by the generation layer 310 . The example of FIG. 9 (b) performs only the required number of loops, whereas 'c' (see line 7) of the index table is stored in a random order, so that the stride of memory access in the iteration of the loop is There may be significant drawbacks.

The tensor generated by the generation layer 310 of the tuned deep learning model and consumed by the consumption layer 310 is data generated by input data for inference at the mobile device. Unlike the weight array, such a tensor is dynamically changed as the inference target data according to the input data is changed.

On the other hand, the output tensor output by the generation layer 310 is composed of a sparse matrix having zeros greater than or equal to a certain ratio, and the deep learning model 300 is pre-tuned in consideration of sparing, so that its accuracy does not decrease.

8 and 9 show examples of generating an auxiliary data structure for each data element of the tensor and performing the convolution operation 311 depending on whether one data element of the auxiliary data structure is 0 or not.

Alternatively, the creation layer 310 creates an auxiliary data structure that indicates whether all of a series of data elements of a specified size are zero in a specified size unit. For example, the generation layer 310 has a size (for example, 4 or 8, etc.) and create an auxiliary data structure according to whether or not all of the grouped consecutive data elements are 0.

The consumption layer 310 performs a convolution operation 311 using an auxiliary data structure that is grouped by a specified size and indicates whether data elements for each group are 0 or not. The consumption layer 310 performs a convolution operation 311 through a vector operation of a specified size.

For example, when all of group data of a specified size is not 0, the consumption layer 310 reads consecutive data elements of a specified size from contiguous memory and performs a convolution operation 311 even if some of them are 0. do. The consumption layer 310 may skip performing the convolution operation 311 when all group data is 0 through the group bit mask or group index of the auxiliary data structure.

By vector arithmetic processing for group data of a specified size, data in an adjacent memory can be efficiently loaded and operation processing can be performed in units of a specified size, thereby improving the execution speed of the consumption layer 310 .

In particular, the generation layer 310 and the consuming layer 310, which have high sparsity according to the sparsity by the deep learning model learner 100, even when the sparsity ratio is decreased due to grouping for vector operation, high performance speed improvement brings

The deep learning model reasoner 200 (controller 290 of) performs the operation of the other layers 310 following the generation layer 310 (ⓑ) and the consumption layer 310 (ⓐ), and the layer 310 According to the execution order of the other generation layer 310 (ⓒ) and the other consumption layer 310 (ⓓ) is performed (refer to ③ and ④ in FIG. 2).

The following generation layer 310 (©) and the following consumption layer 310 (ⓓ) have different thresholds than the previous generation layer 310 (ⓑ) and consumption layer 310 (ⓐ), and dynamically generate tensors create and compute

Since the generation layer 310 (ⓑ) and the consumption layer 310 (ⓐ) have already been described in detail, the differences will be mainly discussed here.

The deep learning model reasoner 200 (controller 290 of) generates and outputs an output tensor using a threshold different from the threshold used in the previous generation layer 310 (ⓑ) in the generation layer 310 (ⓒ). Creates an auxiliary data structure indicating whether each data element in the tensor is zero or not.

The deep learning model reasoner 200 (control unit 290 of) sets the threshold of the activation function of the generation layer 310 (ⓒ) according to the provided tuning parameter, and according to the set threshold, convolution operation data below the threshold value set to 0.

The threshold of the generation layer 310 (ⓒ) and the threshold of the generation layer 310 (ⓑ) are different from each other, and the threshold of the generation layer 310 (ⓒ) is an input greater than or equal to the scarcity ratio specified by the consumption layer 310 (ⓓ). It is a threshold set according to an input distribution according to profiling for the generation layer 310 (©) to provide a tensor.

In addition, the threshold of the generation layer 310 (ⓑ) is set according to the input distribution according to profiling for the generation layer 310 (ⓑ) so as to provide an input tensor equal to or greater than the sparsity ratio specified by the consumption layer 310 (ⓐ). is the threshold to be

The deep learning model reasoner 200 (control unit 290 of) is an auxiliary data structure generated in the generation layer 310 (©) in the consumption layer 310 (ⓓ) subsequent to the generation layer 310 (©). is used to perform an operation on the input tensor.

The deep learning model reasoner 200 (control unit 290 of) sequentially performs a convolution operation 311 and an activation operation 319 inside the consumption layer 310 (ⓓ) and further (additionally) a pooling operation ( 315). The consumption layer 310 (ⓓ) applies the sparsity ratio to perform a convolution operation 311 between the tensor output from the generation layer 310 (©) and the weight array N of the consumption layer 310 (ⓓ). This is performed using an auxiliary data structure dynamically output from the generation layer 310 (©).

The deep learning model 300 tuned in this way has one or more (preferably two or more) generation layers 310 and consumption layers 310, and each generation layer 310 has an output tensor (or activation operation 319). ) in the output tensor of ), it is sparsed through an activation function, etc. according to the input distribution so as to have a certain ratio or more of 0 data elements, and the consumption layer 310 performs inference performance by performing a convolution operation 311 on the sparse tensor. can improve

The weights of the tuned deep learning model 300 are tuned according to learning in consideration of the layer 310 sparsity of the deep learning model 300 . The tuned deep learning model 300 is configured such that its accuracy is maintained. In addition, the overall inference speed may be improved by selecting the consuming layer 310 in consideration of the execution time and sparse the preceding generation layer 310 .

Compared to the existing deep learning model, it was confirmed through experiments that the inference speed improved by more than 1.7 times without deterioration in inference accuracy. have.

The present invention described above, various substitutions, modifications and changes are possible within the scope without departing from the technical spirit of the present invention for those of ordinary skill in the art to which the present invention pertains. It is not limited by the drawings.

100: Deep Learning Model Learner
110: communication unit 150: storage unit
190: control unit
191 first processing unit 195 second processing unit
200: deep learning model reasoner
210: input unit 230: communication unit
250: storage unit 270: output unit
290: control unit
291 first processing unit 293 second processing unit
295: third processing unit
300: deep learning model
310: layer
311 : Convolution operation
315: Pooling operation
319: Activation operation

Claims (13)

(a) selecting N layers of the trained deep learning model;
(b) for N layers preceding the selected N layers, respectively setting thresholds corresponding to the sparsization parameters according to the sparsization parameters; and
(c) tuning the deep learning model by using each threshold set for the preceding N layers;
Wherein N is an integer of 1 or more,
How to train a deep learning model. According to claim 1,
(d) calculating the accuracy of the tuned deep learning model and determining whether the calculated accuracy is greater than or equal to a threshold accuracy; further comprising,
When the calculated accuracy is less than the threshold accuracy, the deep learning model training method reduces the sparsity parameter and repeats steps (b) and (c),
How to train a deep learning model. 3. The method of claim 2,
When the calculated accuracy is equal to or greater than the threshold accuracy, the deep learning model training method increases N by 1 and repeats steps (a) to (c) from the sparsity parameter corresponding to the calculated accuracy,
How to train a deep learning model. 4. The method of claim 3,
After performing the steps (a) to (d), storing the tuned deep learning model and the corresponding tuning parameters,
The tuned deep learning model performing inference from input data using the tuning parameter performs the activation function of the preceding N layers using a threshold set through the tuning parameter,
How to train a deep learning model. The method of claim 1,
The step (a) is to select N layers based on the execution time of all layers according to the profiling of the learned deep learning model,
How to train a deep learning model. 6. The method of claim 5,
In step (a), each of the layers is ordered according to the execution time, and N layers having the longest execution time are selected, or the entire layers are ordered according to the execution time reduction rate according to the sparsity of each of the entire layers and performed the most. Selecting N layers with a high time reduction rate,
How to train a deep learning model. According to claim 1,
The step (b) is to set each input value corresponding to the sparing parameter in each activation function input distribution according to the profiling of the preceding N layers to the respective thresholds of the N preceding layers,
How to train a deep learning model. a control unit that executes a program command; and
Including; a deep learning model learning program and a storage unit for storing the learned deep learning model;
The control unit for performing the deep learning model training program selects N layers of the learned deep learning model, and a threshold value corresponding to the sparsation parameter according to the sparing parameter for N layers preceding the selected N layers. , and tuning the deep learning model using each threshold set for the preceding N layers,
Wherein N is an integer of 1 or more,
Deep learning model learner. 9. The method of claim 8,
The control unit calculates the accuracy of the tuned deep learning model, determines whether the calculated accuracy is equal to or greater than a threshold accuracy, and if the calculated accuracy is less than the threshold accuracy, reduces the sparsization parameter, and for the preceding N layers Each setting a corresponding threshold according to the reduced sparing parameter and repeating the tuning of the deep learning model using each threshold set to be reduced for the preceding N layers,
Deep learning model learner. 10. The method of claim 9,
When the calculated accuracy is greater than or equal to the threshold accuracy, the control unit increases N by 1, selects N layers increased by 1 of the learned deep learning model, and calculates the N layers preceding the N layers Each of the thresholds corresponding to the sparing parameters corresponding to the obtained accuracy are set, and the tuning of the deep learning model is repeated using each threshold set for the preceding N layers,
Deep learning model learner. 11. The method of claim 10,
The control unit stores the tuned deep learning model and the corresponding tuning parameters in the storage unit,
The tuned deep learning model performing inference from input data using the tuning parameter performs the activation function of the preceding N layers using a threshold set through the tuning parameter,
Deep learning model learner. 9. The method of claim 8,
The control unit, according to the profiling of the learned deep learning model, orders each of the entire layers according to the execution time and selects N layers having the longest execution time or a reduction rate of the execution time according to the sparsity of each of the entire layers order the entire layers according to , and select N layers with the highest reduction in execution time,
Deep learning model learner. 9. The method of claim 8,
The control unit sets each input value corresponding to the sparing parameter in each activation function input distribution according to the profiling of the preceding N layers to each threshold value of the preceding N layers,
Deep learning model learner.

Images