KR20230076075A - Method and apparatus for optimal layer-wise execution policy prediction based on supervised learning method in systolic array based npu environment for inference - Google Patents

Abstract

Disclosed are a method and device for predicting an optimal execution policy for each layer based on a supervised learning technique in an NPU environment for inference with a systolic array structure. A method for predicting execution policy for each layer includes the steps of: analyzing kernel parameters for each layer of a target deep neural network (DNN) application and the available resources of the target NPU; searching for an optimal factor for each situation or each layer through a model learned based on the layer information of the target DNN application and the target NPU information; and applying an optimal execution policy predicted based on the optimal factor to DNN application runtime.

Description

Method and Apparatus for Predicting Optimal Layer-by-Layer Execution Policy Based on Supervised Learning Technique in Neural Network Operation Device Environment for Inference of Systolic Array Structure INFERENCE}

The present invention provides a method for predicting an optimal layer-wise execution policy based on a supervised learning method in a neural processing unit (NPU) environment for reasoning with a systolic array structure and a framework to which the same is applied. it's about

Some prior art mentions that the optimal execution policy for each layer may be different according to the structure of target hardware in deep learning-based applications and the parameters of each layer constituting the application. .

However, in a systolic array-based neural processing unit (NPU) environment, parameters such as optimal dataflow and clustering factor for each layer of deep learning applications are set differently. However, an appropriate method for how to apply an optimal parameter for each layer has not yet been proposed.

The present invention is a map capable of predicting and applying various execution policies presented in the prior art according to the target environment of a neural processing unit (NPU) for inference having a systolic array structure for each layer of a neural network. The purpose of this study is to provide a method and device for predicting the optimal execution policy for each layer based on learning techniques.

Another object of the present invention is a method for predicting an execution policy for each layer, capable of predicting an optimal policy for each layer through a supervised learning method in various environments without being limited to a single NPU or a single application when performing an NPU environment-based deep learning application, and device is provided.

Another object of the present invention is to provide an execution policy prediction method for each layer and an application framework capable of flexibly applying the optimal execution policy for each layer in real time to a deep learning application in an NPU environment.

In order to solve the above technical problem, a supervised learning method-based layer-by-layer execution policy prediction method according to an aspect of the present invention is guided in a neural processing unit (NPU) environment for inference having a systolic array structure. As a learning method-based optimal layer-wise execution policy prediction method, analysis of kernel parameters for each layer of a target deep neural network (DNN) application and available resources of a target neural processing unit (NPU) doing; Searching for an optimal factor, which is a factor representing the best performance for each situation or each layer, through a model learned based on layer information and target NPU information of the target DNN application; and applying the optimal execution policy predicted based on the optimal factor to the DNN application runtime.

In one embodiment, the analyzing may include obtaining kernel features including layer configuration of DNN kernels, arrangement of layers, and neural network parameters.

In one embodiment, the analyzing may include processing element (PE) resources of the target NPU, memory limitations of on-chip scratchpad memory and off-chip memory. limits) may include acquiring NPU characteristics.

In one embodiment, the step of searching for the optimal factor for each layer comprises deriving each tiling factor for each DNN layer, deriving each clustering factor, and deriving each dataflow. may include derivation.

In one embodiment, the applying to the DNN application runtime may include extracting a matrix multiplication operation and convolution operation layer of the target DNN application; predicting an optimal policy set based on the target NPU and layer characteristics; Compiling by applying a policy set derived for each layer; and performing a DNN application through the compiled layer.

In one embodiment, the supervised learning method-based execution policy prediction method for each layer further includes, prior to the searching step, learning a prediction model through a neural network composed of fully connected layers based on data obtained through indiscriminate investigation. can do.

In one embodiment, the step of learning may include classifying computational features or kernel features and NPU features for training the predictive model; Building an artificial neural network composed of fully connected layers; and training a predictive model through a DNN based on the synthetic data set and the classified feature set.

In one embodiment, the supervised learning method-based execution policy prediction method for each layer may further include, prior to the learning step, performing a brute-force search for model learning through the supervised learning method. .

In one embodiment, the step of conducting the indiscriminate search may be performed in a range of data sets having all correct answers available for learning the predictive model.

According to another aspect of the present invention for solving the above technical problem, an apparatus for predicting an execution policy for each layer based on a supervised learning method is provided in a neural processing unit (NPU) environment for inference having a systolic array structure. An apparatus for predicting an optimal layer-wise execution policy based on a supervised learning method, comprising: a processor; and a memory storing at least one instruction executed by the processor. Analyzing, by the at least one instruction, kernel parameters for each layer of a target deep neural network (DNN) application and available resources of a target neural processing unit (NPU), by the processor, layer of the target DNN application Searching for an optimal factor, which is a factor that exhibits the best performance for each situation or each layer, through a model learned based on information and target NPU information, and DNN application runtime, which predicts an optimal execution policy based on the optimal factor It can be configured to perform steps that apply to (runtime).

In one embodiment, in the analyzing step, the processor may be configured to obtain kernel characteristics including layer configuration of DNN kernels, arrangement of layers, and neural network parameters.

In one embodiment, the processor, in the analyzing step, PE (processing element) resources of the target NPU, on-chip scratchpad memory and off-chip memory It can be configured to acquire NPU characteristics for memory limits.

In one embodiment, the processor derives each tiling factor for each DNN layer, derives each clustering factor, and each data flow in the step of searching for the optimal factor for each layer. It can be configured to derive (dataflow).

In one embodiment, the processor extracts matrix multiplication operation and convolution operation layers of the target DNN application in the step of applying to the DNN application runtime, and an optimal policy set based on the target NPU and layer characteristics. It may be configured to perform a predicting step, a step of performing compilation by applying a set of policies derived for each layer, and a step of performing a DNN application through the compiled layer.

In one embodiment, the supervised learning method-based execution policy prediction apparatus for each layer is configured such that the processor, prior to the searching step, learns a predictive model through a neural network composed of fully connected layers based on data obtained through indiscriminate investigation. It can be configured to perform further steps.

In one embodiment, the processor, in the step of learning, classifying a computational feature or kernel feature and NPU feature for training the predictive model; Building an artificial neural network composed of fully connected layers; and training a predictive model via a DNN based on the synthetic data set and the classified feature set.

In one embodiment, in the supervised learning method-based execution policy prediction apparatus for each layer, the processor, prior to the learning step, for model learning through the supervised learning method, sets a range of data sets having all correct answers usable for predictive model learning. It may be configured to further perform a step of performing a brute-force search in .

According to the present invention, an optimal execution policy according to a given neural processing unit (NPU) and deep neural network (DNN) application layer can be predicted in real time and applied according to circumstances.

In addition, according to the present invention, it is possible to derive an optimal execution policy for each layer in various environments without being limited to a single NPU or a single application. That is, it is possible to provide an effective solution that overcomes the limitation of not being able to flexibly change the execution policy of the NPU or DNN according to the target environment in the prior art.

In addition, according to the present invention, it is possible to provide a solution that can be utilized as an optimization technique capable of maximizing performance such as throughput per unit time in object recognition, real-time license plate recognition applications, and the like.

1 is an exemplary view showing a performance difference according to an execution policy in an actual deep neural network (DNN) application in relation to optimization of a neural processing unit (NPU) environment according to an embodiment of the present invention. am.
2A and 2B show an optimal layer-wise execution policy prediction technique based on a supervised learning technique in an NPU environment having a systolic array (SA) structure according to an embodiment of the present invention. and an overall structural diagram illustrating the application framework.
3 is a flowchart schematically illustrating an entire execution sequence of the framework of FIG. 2A.
FIG. 4 is a flowchart for explaining a detailed procedure for a brute force search process among the entire execution sequence of the framework of FIG. 3 .
5 is a flowchart for explaining detailed procedures for a predictive model training process among the entire execution sequence of the framework of FIG. 3 .
FIG. 6 is a flowchart for explaining a detailed procedure for applying the framework of FIG. 3 to a DNN application among the entire execution sequence.
7 is an exemplary diagram for explaining a DNN kernel that can be employed in the framework of FIG. 2A.
8 is an exemplary view for explaining a target NPU that can be employed in the framework of FIG. 2A.
9 is an exemplary diagram for explaining an MLP model training process that can be employed in the framework of FIG. 2A and optimal parameter configuration according to this process.
10 is a schematic block diagram of main components that can be employed in a framework that applies an optimal layer-by-layer execution policy prediction method based on a supervised learning method in an NPU environment of an SA structure according to another embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. Like reference numerals have been used for like elements throughout the description of each figure.

Terms such as first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention. The term 'and/or' includes a combination of a plurality of related recited items or any one of a plurality of related recited items.

In embodiments of the present application, 'at least one of A and B' may mean 'at least one of A or B' or 'at least one of combinations of one or more of A and B'. Also, in the embodiments of the present application, 'at least one of A and B' may mean 'at least one of A or B' or 'at least one of combinations of one or more of A and B'.

It is understood that when a component is referred to as being 'connected' or 'connected' to another component, it may be directly connected or connected to the other component, but other components may exist in the middle. It should be. On the other hand, when a component is referred to as being 'directly connected' or 'directly connected' to another component, it should be understood that no other component exists in the middle.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as 'comprise' or 'having' are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this application, it should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is an exemplary view showing a performance difference according to an execution policy in an actual deep neural network (DNN) application in relation to optimization of an NPU environment according to an embodiment of the present invention.

That is, FIG. 1 quantitatively shows the gain in terms of performance that can be expected when the NPU environment is well optimized according to the present embodiment. In addition, in FIG. 1, for two cases of S-CNN (sentimantal convolutional neural network) and Transformer of DNN application, output stationary (OS), weight stationary (WS) and input The performance difference according to the three execution policies of value fixation (input stationary, IS) is shown, and the other two execution policies, WS and IS, are normalized and shown for the OS execution policy.

Output fixation (OS) is a structure designed to minimize energy consumption when reading and writing partial sums by reusing outputs. The weight fixation (WS) is a structure that increases reusability by maintaining filter values or weights. That is, weight fixation (WS) minimizes energy consumption when reading weights by maximizing convolutional and filter reuse by minimizing fetching weights from register files. It is a designed structure. In contrast to weight fixation, the input value lock (IS) is a structure in which an input feature map (fmap) value is stored in an array and a convolution operation is performed by passing the weight to the array.

In this embodiment, the S-CNN model may correspond to an example of a DNN application consisting of 4 convolution layers, and the Transformer model may correspond to another example of a DNN application consisting of 10 convolution layers.

Here, the S-CNN model refers to a sentimental classification convolutional neural network (Sentimental CNN), which is a model that classifies information expressing a human emotional state collected through a social network service (SNS) or the like through a convolutional neural network. The S-CNN model is a model that processes, trains, and learns lexical expressions representing positives and negatives from massive input data into data that the model can recognize.

The Transformer model does not use a recurrent neural network (RNN), but maintains an encoder-decoder structure that receives an input sequence from an encoder and outputs an output sequence from a decoder like the structure of an existing seq2seq (sequence to sequence) model. can be configured. In addition, in the existing seq2seq structure, each RNN in the encoder and decoder has t time steps, but in the Transformer model, the unit of encoder and decoder is N units (10 in this embodiment). can have

According to this embodiment, an optimal case among three different execution policies may appear slightly different according to a convolution layer. In particular, in some cases, the performance difference according to the execution policy appears close to 16 times or 30 times.

Therefore, in this embodiment, among several execution policies applicable to the NPU environment for reasoning with a systolic array structure, the optimal execution policy in the target environment can be effectively derived and applied using deep learning. By doing so, it is expected to have the effect of maximizing the performance of NPU and applications in various situations.

On the other hand, the prediction method and application framework of this embodiment, in addition to the above-mentioned three execution policies, are systolic arrays such as no local reuse (NLR), row stationary (RS), tiling, and clustering. Of course, it can be applied to all execution policies that can be applied to the NPU environment for structure inference.

Here, the NLR has a structure that reduces memory access such as DRAM by directly transferring data from a global buffer to a processing element (PE) array.

Row fixation (RS) is a structure that maximizes reuse for various data such as weights, pixels, and subtotals, compared to optimizing only weights and subtotals in case of weight fixation or output fixation. That is, row fixation (RS) puts enough input feature map (fmap) values to be used for operation in a specific part of local memory, calculates the partial sum corresponding to the output feature map location, , RS has a structure in which the same weight is used in all processes, that is, the weight is reused, and only one value is stored in the corresponding part of the local memory in each process. This RS structure can perform calculations with a small amount of local memory and reuse of height data, and as a result, it is the most energy efficient compared to other execution policies.

Tiling is a structure in which a certain portion of at least one or more layers of classifier layers, convolution layers, and pooling layers is divided into tiles and operated, and values brought from memory are reused as much as possible. It is configured to operate as

Clustering basically refers to grouping similar data instances into a set according to certain criteria. Clustering may refer to an unsupervised learning method that preserves as much information as possible while mapping the original data space to a low-dimensional space. In this embodiment, clustering may be applied in a form combined with a supervised learning method.

In the description below, the execution policy may be referred to as a policy for short, or may be replaced with a factor. For example, the aforementioned OS policy, WS policy, and IS policy may be referred to as an OS factor, a WS factor, and an IS factor in the order in which they are described.

2A and 2B show a method for predicting an optimal layer-wise execution policy based on a supervised learning technique in an NPU environment having a systolic array (SA) structure according to an embodiment of the present invention. and an overall structural diagram illustrating the application framework. And Figure 3 is a flow chart schematically showing the entire execution sequence of the framework of Figure 2a.

Referring to FIGS. 2A, 2B, and 3 , detailed operations of a method for predicting a layer-wise execution policy may proceed in the following order.

First, kernel parameters for each layer of various DNN applications and available resources of the target NPU are analyzed.

This step is to set kernel parameters for each layer applicable to the target NPU in consideration of available resources of the target NPU by analyzing kernel parameters for each layer available for DNN application and storing kernel parameter information for each layer.

In addition, in order to analyze the available resources of the target NPU, this step is performed in such a way that an execution policy prediction device for each layer reads specific information recorded in the memory of the target NPU connected to the prediction device, or when the two devices are connected It may be performed in a manner in which specific information is transferred from the target NPU to the prediction device according to a preset procedure.

Kernels for each layer include deep learning (DL) kernels, and the deep learning kernels may include a general matrix multiplication (GEMM) kernel, a convolution (CONV) kernel, and the like. The GEMM kernel can be implemented in the form of the C=A*B+C operation. A, B, X and C represent matrices. For example, the GEMM kernel may consist of three main parts: data movers, transpose and buffers, and a systolic array. The GEMM kernel may have a size corresponding to a GEMM layer having a size such as 2048 × 256 × 1600. In addition, the CONV kernel is for extracting features by giving weights for each element of the convolution layer, and may have a size of 3 × 5 × 5 for a convolution layer having a size of 3 × 3 × 256 × 256, etc. .

Next, brute-force search is performed for model learning through supervised learning techniques (S40).

The supervised learning technique is one of the machine learning techniques for classification problems, which learns a model through a large number of classification results acquired from prior knowledge and performs classification on new data. it is a technique In this embodiment, the supervised learning technique is used as an approach for predicting the optimal execution policy according to the target environment. The parameters of the NPU structure and DNN application are defined as features, and the available execution policies are defined as classification labels, and the results of deriving optimal labels from various features are made into training data and used for model learning. Through the learned model, it predicts the optimal execution policy in various NPU and DNN application environments.

Indiscriminate search, also referred to as exhaustive search, is the simplest way to find a result value by searching all possible numbers or all possible paths. In this embodiment, indiscriminate search can be performed in the range of the data set with all possible correct answers for learning the predictive model of this embodiment.

Next, a prediction model is trained through a neural network composed of a fully-connected layer (FCL) based on data obtained through indiscriminate investigation (S50).

Prediction model learning (S50) may include multilayer perceptron (MLP) model training (model training) (50). In this case, the predictive model learning (S50) includes the first learning 30 that infers the first policy set including the execution policy of each of the convolution (CONV) layers or the execution policy of each of the GEMM layers It may be configured to perform the second learning 40 for inferring the second policy set based on a supervised learning technique.

Next, through the model learned based on the layer information of the target DNN application and the information of the target NPU 20, the optimal factor search for each situation or each layer is searched, and the optimal execution policy policy), and the predicted optimal execution policy can be flexibly applied to the DNN application runtime (S60).

That is, in application to DNN application (S60), convolution is performed through a trained model (80) for target layers of each of a plurality of NPUs (NPU #0, NPU #1, and NPU #2). A policy set for each solution layer (CONV #0, CONV #1, GEMM #0) can be predicted and applied to DNNs to perform optimal DNN execution.

Here, the target NPU 20 may include a dynamic random access memory (DRAM) and a plurality of processing element (PE) clusters (PE cluster 0, PE cluster 1, and Pe cluster 2) connected to the DRAM. . And the target layers are input layers, convolution layers, and pooling layers of real-world DNNs (70), except for output layers including softmax or fully connected layers (FCL). ), etc. may include the remaining layers.

FIG. 4 is a flowchart for explaining a detailed procedure for a brute force search process among the entire execution sequence of the framework of FIG. 3 .

Referring to FIG. 4 , in a brute-force search process, first, a composite data set composed of a matrix multiplication operation and a convolution operation is defined (S41).

Next, all available policy sets applicable to the NPU environment are classified (S42).

Next, the NPU performs calculations on all policies available for the synthetic data set (S43).

Next, the synthesized data set is reconstructed into neural network training data based on the kernel feature or NPU feature derived according to the NPU operation (S44).

5 is a flowchart for explaining detailed procedures for a predictive model training process among the entire execution sequence of the framework of FIG. 3 .

Referring to FIG. 5, in the process of predictive model training, first, computational features (or kernel features) and NPU features for predictive model training are classified (S51).

Next, an artificial neural network composed of fully connected layers is built (S52). Artificial neural networks include deep neural networks (DNNs).

Next, a predictive model is trained through a DNN based on the synthetic data set and the classified feature set (S53).

The trained model may be verified with a verification data set (S54). The validation data set may include data sets with correct answers.

FIG. 6 is a flowchart for explaining a detailed procedure for applying a DNN application among the entire execution sequence of the framework of FIG. 3 .

Referring to FIG. 6 , in the process of applying to DNN applications, matrix multiplication operation and convolution operation layers of the target DNN application are first extracted (S61).

Next, an optimal policy set is predicted based on the target NPU and layer characteristics (S62).

Next, compilation is performed by applying the policy set derived for each layer (S63). Compilation may refer to defining parameters required for a training process before training a predictive model. During the compilation process, training parameters can be passed to the compilation method. A compilation method may correspond to a set of codes or commands defining a compilation function.

Next, DNN application is performed through the compiled layer (S64).

7 is an exemplary diagram for explaining a DNN kernel that can be employed in the framework of FIG. 2A. 8 is an exemplary view for explaining a target NPU that can be employed in the framework of FIG. 2A. 9 is an exemplary diagram for explaining an MLP model training process that can be employed in the framework of FIG. 2A and optimal parameter configuration according to this process.

That is, FIGS. 7 to 9 are methods according to another embodiment of the present invention, which describe a method of predicting an optimal layer-wise execution policy based on a supervised learning method in a systolic array-structured NPU. drawings for

The layer-by-layer execution policy prediction method first analyzes various parameters according to deep learning applications and target NPU structures, as shown in FIG. 7 . In the parameter analysis process, kernel characteristics including layer configuration (Layer configs) of DNN kernels (12), batch of layers, neural network (NN) parameters (parameters, params), etc. (kernel features).

In addition, in the parameter analysis process, as shown in FIG. 8, processing element (PE) resources of the target NPU 22, on-chip scratchpad memory and off-chip ) may be configured to obtain NPU features for memory limits, etc. of memory.

Next, as shown in FIG. 9, the method of predicting the execution policy for each layer proceeds with indiscriminate investigation for the application of the supervised learning technique (52), and builds based on the data obtained through the indiscriminate investigation. A model can be trained through a neural network. Model learning may be configured to infer the best configuration based on the trained model (82).

Inferred optimal parameter configuration (optimal parameter configuration, 84), for example, each tiling factor (tiling factor) in the order described for each DNN layer (CONV #0, CONV 31, CONV #2) (8, 8), (4, 16), (16, 8), deriving each clustering factor as 2, 8, and 4, and deriving each dataflow as OS, WS, and IS. can

After model learning is completed, the optimal execution policy in the new environment that was not considered during training can be predicted through the model, and the predicted optimal execution policy can be applied to the target NPU application runtime.

As such, the optimal policy set for each layer classified by the prediction method of the present embodiment may include a tiling policy, a clustering policy, a data flow, and the like.

The tiling policy may correspond to a tiling factor, and is an optimization policy in which an entire matrix multiplication or convolution operation is divided into tiles of the same size and divided into respective operation cores. Optimum values of the tiling parameters may vary depending on the size and shape of matrices constituting the target operation.

The clustering policy, which may be referred to as a clustering factor, is an optimization policy that divides computational cores of a target NPU into clusters of the same size and allocates computations to each cluster. The clustering parameter may have different optimal values depending on the target NPU resource and the size and shape of the target operation.

Dataflow is a policy for the order or method of allocating input values necessary for operation to the target NPU. There are data flow policies such as output stationary, filter value fixed or weight stationary, and input stationary, and the optimal value may differ depending on the target matrix multiplication operation or convolution operation. .

10 is a device according to another embodiment of the present invention, which is a schematic diagram of a device for predicting an optimal execution policy for each layer based on a supervised learning method in an NPU having a systolic array structure (hereinafter, simply referred to as a 'prediction device'). is a block diagram.

The prediction device of this embodiment may also include a framework for applying the predicted execution policy. Therefore, the device of this embodiment may also be referred to as a prediction and application device.

Referring to FIG. 10 , the prediction device 100 may include at least one processor 110 and a memory 120. In addition, the prediction device 100 may further include a transceiver 130. In addition, the prediction device 100 may optionally further include a storage device 140, an input interface device 150, an output interface device 160, and the like. Each component included in the prediction device 100 may be connected by a bus to communicate with each other.

The processor 110 may execute a program command stored in at least one of the memory 120 and the storage device 140 . The program command is a command to analyze the kernel parameters of each layer of various DNN applications and the available resources of the target NPU, and a range of data sets with all possible correct answers for model learning through supervised learning techniques. A command to perform a brute-force search in , a command to train a prediction model through a neural network composed of a fully-connected layer (FCL) based on data obtained through brute-force search, and a target DNN application A command to search for an optimal factor for each situation or each layer through a model learned based on layer information and target NPU information, a command to predict an optimal execution policy, and a DNN application of the predicted optimal execution policy It may include commands that are flexibly applied at runtime. Also, the program command may include instructions for performing each step of FIGS. 4 to 6 .

The above-described processor 110 may be a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor for performing at least one of the methods according to an embodiment of the present invention. can mean

Each of the memory 120 and the storage device 140 may include at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory 120 may include at least one of a read only memory (ROM) and a random access memory (RAM).

The transceiver 130 may include a communication subsystem capable of performing communication in a wired or wireless communication method. The sub-communication system may be configured to support at least one communication method such as intranet, Internet, wireless LAN, mobile communication, satellite communication, and the like.

The input interface device 150 may include at least one input means selected from a keyboard, microphone, touch pad, touch screen, etc., and an input signal processing unit that maps or processes a signal input through the at least one input means with a pre-stored command. can

In addition, the output interface device 160 includes an output signal processing unit that maps or processes the signal output under the control of the processor 110 into a pre-stored signal type or level, and a signal of the output signal processing unit in the form of vibration or light. It may include at least one output means for outputting a signal or information. The at least one output means may include at least one selected from among output means such as a speaker, a display device, a printer, an optical output device, and a vibration output device.

The operation of the method according to the embodiment of the present invention can be implemented as a computer readable program or code on a computer readable recording medium. A computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. In addition, computer-readable recording media may be distributed to computer systems connected through a network to store and execute computer-readable programs or codes in a distributed manner.

In addition, the computer-readable recording medium may include hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. The program command may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine code generated by a compiler.

Although some aspects of the invention have been described in the context of an apparatus, it may also represent a description according to a corresponding method, and a unit or apparatus may correspond to a method step or characteristics of a method step. Similarly, aspects described in the context of a method may also be represented by corresponding blocks or components or features of a device performing a corresponding function. Some or all of the method steps may be performed by (or using) a hardware device such as, for example, a microprocessor, programmable computer, or electronic circuitry. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

In embodiments, a programmable logic device, such as a field programmable gate array (FPGA), may be used to perform some or all of the functions of the methods described herein. Further, in embodiments, an application specific semiconductor such as an FPGA, application-specific integrated circuit (ASIC), application-specific standard parts (ASSP), system on chip (SoC), or the like is used as a microcontroller for performing one of the methods described herein. It can work with processors. Generally, methods are preferably performed by some hardware device.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art can variously modify and change the present invention within the scope not departing from the spirit and scope of the present invention described in the claims below. will be able to understand

Claims (16)

As a method for predicting an optimal layer-wise execution policy based on a supervised learning method in a neural processing unit (NPU) environment for reasoning with a systolic array structure,
Analyzing kernel parameters for each layer of a target deep neural network (DNN) application and available resources of a target neural processing unit (NPU);
Searching for an optimal factor, which is a factor representing the best performance for each situation or each layer, through a model learned based on layer information and target NPU information of the target DNN application; and
A method for predicting execution policies for each layer based on a supervised learning method comprising: applying an execution policy predicted based on the optimal factor to a DNN application runtime. The method of claim 1,
Wherein the analyzing step includes acquiring kernel characteristics including layer configuration of DNN kernels, arrangement of layers, and neural network parameters. The method of claim 1,
The analyzing step may include processing element (PE) resources of the target NPU or memory limits of an on-chip scratchpad memory and an off-chip memory of the NPU. A method for predicting execution policies for each layer based on a supervised learning method, including acquiring features. The method of claim 1,
The step of searching for the optimal factor for each layer includes deriving each tiling factor for each DNN layer, deriving each clustering factor, and deriving each dataflow. , A method for predicting execution policies by layer based on supervised learning techniques. The method of claim 1,
The step of applying to the DNN application runtime,
extracting matrix multiplication operation and convolution operation layers of the target DNN application;
predicting an execution policy set based on the target NPU and layer characteristics;
Compiling by applying an execution policy set derived for each layer; and
A method for predicting execution policy for each layer based on supervised learning techniques, including the step of performing DNN application through compiled layers. The method of claim 1,
Prior to the searching step, further comprising the step of learning a prediction model through a neural network composed of fully connected layers based on data obtained through indiscriminate investigation. The method of claim 6,
The learning step is
Classifying computational features or kernel features and NPU features for training the predictive model;
Building an artificial neural network composed of fully connected layers; and
A method for predicting an execution policy for each layer based on a supervised learning method, comprising training a predictive model through a DNN based on a synthetic data set and a classified feature set. The method of claim 7,
Prior to the learning step, the step of performing a brute-force search for model learning through a supervised learning method, further comprising a supervised learning method based execution policy prediction method for each layer. The method of claim 8,
The step of performing the indiscriminate investigation is performed in a range of data sets having all correct answers available for learning the predictive model, a method for predicting execution policies for each layer based on a supervised learning technique. A device for predicting an optimal layer-wise execution policy based on a supervised learning method in a neural processing unit (NPU) environment for reasoning with a systolic array structure,
at least one processor; and
A memory storing at least one instruction executed by the at least one processor,
According to the at least one instruction, the at least one or more processors,
Analyze kernel parameters for each layer of the target deep neural network (DNN) application and available resources of the target neural processing unit (NPU),
Searching for an optimal factor, which is a factor showing the best performance for each situation or each layer, through a model learned based on the layer information and target NPU information of the target DNN application,
An apparatus for predicting an execution policy for each layer based on a supervised learning method, which applies an execution policy predicted based on the optimal factor to a DNN application runtime. The method of claim 10,
Wherein the processor acquires kernel characteristics including layer configuration of DNN kernels, arrangement of layers, and neural network parameters in the process of analyzing the available resources. The method of claim 11,
The processor, in the process of analyzing the available resources, the PE (processing element) resources of the target NPU, memory limitations of on-chip scratchpad memory and off-chip memory A device for predicting execution policies for each layer based on a supervised learning method that acquires NPU characteristics for (memory limits). The method of claim 11,
In the process of searching for the optimal factor for each layer, the processor derives each tiling factor for each DNN layer, derives each clustering factor, and derives each dataflow. A device for predicting execution policies for each layer based on supervised learning techniques. The method of claim 11,
The processor, in the process of applying to the DNN application runtime,
Extract matrix multiplication operation and convolution operation layers of the target DNN application;
Predicting an execution policy set based on the target NPU and layer characteristics,
Compilation is performed by applying the set of execution policies derived for each layer,
Execution policy prediction device for each layer based on supervised learning method that performs DNN application through compiled layer. The method of claim 11,
the processor,
Before searching for the optimal factor for each layer, a predictive model is trained through a neural network composed of fully connected layers based on data obtained through indiscriminate investigation,
In the process of learning the predictive model, computational features or kernel features and NPU features for training the predictive model are classified, an artificial neural network composed of fully connected layers is built, and a synthetic data set and classified A device for predicting execution policy for each layer based on a supervised learning method that trains a prediction model through DNN based on a feature set. The method of claim 15
The processor, before learning the predictive model, performs a brute-force search in the range of a data set having all correct answers usable for predictive model learning for model learning through a supervised learning technique. Execution policy prediction device for each layer based on technique.

Images