KR102487535B1 - Method and apparatus for operating deep learning by using the systolic array - Google Patents

Abstract

Provided is an artificial intelligence (AI) learning method and apparatus for learning output data by performing a convolution operation on input data through a systolic array. According to an embodiment of the present disclosure, a method for reducing the amount of power consumed in performing a convolution operation and increasing the operation speed through mapping to change the sequence of data input to a systolic array operator and devices.

Description

Method and apparatus for performing deep learning operation using systolic array {METHOD AND APPARATUS FOR OPERATING DEEP LEARNING BY USING THE SYSTOLIC ARRAY}

The present disclosure provides a method and apparatus for performing a convolution operation between an input feature map and a filter using a systolic array in a deep learning operation based on a convolutional neural network (CNN). it's about

An artificial intelligence (AI) system is a computer system that implements human-level intelligence, and unlike existing rule-based smart systems, machines learn, judge, and become smarter on their own. The more AI systems are used, the higher the recognition rate and the more accurate understanding of user preferences. Existing rule-based smart systems are gradually being replaced by deep learning-based AI systems.

Artificial intelligence technology consists of machine learning (deep learning) and element technologies using machine learning.

Machine learning is an algorithm technology that classifies/learns the characteristics of input data by itself, and element technology is a technology that uses machine learning algorithms such as deep learning to mimic the functions of the human brain, such as cognition and judgment. It consists of technical fields such as understanding, inference/prediction, knowledge expression, and motion control.

The various fields where artificial intelligence technology is applied are as follows. Linguistic understanding is a technology for recognizing and applying/processing human language/characters, including natural language processing, machine translation, dialogue systems, question and answering, voice recognition/synthesis, and the like. Visual understanding is a technology for recognizing and processing objects like human vision, and includes object recognition, object tracking, image search, person recognition, scene understanding, space understanding, image improvement, and the like. Inference prediction is a technique of reasoning and predicting logically by judging information, and includes knowledge/probability-based reasoning, optimization prediction, preference-based planning, and recommendation. Knowledge expression is a technology that automatically processes human experience information into knowledge data, and includes knowledge construction (data creation/classification) and knowledge management (data utilization). Motion control is a technology for controlling the autonomous driving of a vehicle and the movement of a robot, and includes motion control (navigation, collision, driving), manipulation control (action control), and the like.

When performing deep learning operations, in the case of convolution with a large amount of calculation, giga units of operation or equivalent capabilities are required, especially when using 32-bit or 16-bit input data and filter weight buses. In this case, there is a large amount of data that is toggled inside the processor even if only inference is made. As the amount of data to be toggled increases, the amount of power consumed by the processor increases. In a recent mobile device environment, there is a problem due to a battery capacity limit.

The present disclosure relates to a deep learning operation that learns output data by performing a convolution operation on input data through a systolic array, and operates through mapping to change the order of data input to a systolic array operator. It is an object of the present invention to provide a method and apparatus for reducing the amount of power consumed in performing and increasing an operation speed.

In order to solve the above-described technical problem, an embodiment of the present disclosure is an input feature map such that data having the same value is repeated a predetermined number of times according to a sequential clock and input to a multiplication addition operator in a systolic array operator changing the order of the data of , mapping the data of the input feature map into a systolic array, inputting the mapped data of the input feature map and the weight value of the filter into the systolic array calculator to perform convolution operation Deep learning using a systolic array, including performing convolution and generating an output feature map by accumulating the convolution-operated output values (deep learning) Provides a way to perform operations.

For example, in the mapping step, the input order of the input data may be changed so that the same two input data values of the input feature map are repeatedly input to the systolic array calculator.

For example, the filter has a size of N by N (NXN), and the mapping step is such that the data of the input feature map is repeatedly input to the systolic array operator the same number of times as the size (N) of the filter. The input order of the data of the input feature map can be changed.

For example, the generating of the output feature map may include performing the convolution operation in consideration of a stride value of the filter and data of the input feature map overlapping in a first direction and a second direction due to the stride. The method may include selecting output values, and generating an output feature map resulting from a combination of the input feature map and the filter by cumulatively summing the selected output values.

For example, the method may further include identifying data having a value of 0 among data of the input feature map.

For example, in the mapping step, data having a value of 0 may be grouped so that the data having a value of 0 is repeatedly input to the systolic operator.

For example, the filter has a size of N by N (NXN), and in the mapping step, the 0 value is repeatedly input to the systolic array operator the same number of times as the size (N) of the filter. Data having values may be collected and grouped by the size (N) of the filter.

For example, the method further comprises storing in a memory row and column positions of data having a value of 0 on the input feature map, and the convolution The performing of the operation may include reading the stored location information from the memory and skipping multiplication and addition operations for data having a value of 0 by referring to the read location information.

For example, the method may further include, when data identified as 0 is input, acquiring location information on the systolic array of the identified 0-value data, and generating the output feature map Output values for each region of the input feature map may be cumulatively summed based on the obtained location information of the 0-value data.

In order to solve the above technical problem, an embodiment of the present disclosure includes a processor that performs a convolution operation between data of an input feature map and a weight value of a filter using a systolic array, and the processor performs a sequential clock ( A mapper that maps the data of the input feature map into a systolic array by changing the order of the data of the input feature map so that data having the same value is repeated a predetermined number of times and input to the multiplication and addition operator according to the clock) , A systolic array operator including a plurality of multiplication and addition operators performing a convolution operation on the mapped input feature map data or the weight value of the filter, and accumulating and summing the output values obtained by the convolution operation and outputting the result. An electronic device for performing deep learning operations, including an adder for generating an output feature map, is provided.

For example, the mapper may change the input order of the input data such that the same two input data values of the input feature map are repeatedly input to the systolic array operator.

For example, the filter has a size of N by N (NXN), and the mapper repeats the data of the input feature map the same number of times as the size (N) of the filter. The multiply-add operator in the systolic array operator It is possible to change the input order of data of the input feature map so that it is input to .

For example, the accumulator selects the output values subjected to the convolution operation in consideration of a stride value of the filter and data of the input feature map overlapping in a first direction and a second direction due to the stride, and An output feature map for each region of the input feature map may be generated by cumulatively summing the selected output values.

For example, the mapper may identify data having a value of 0 among data of the input feature map.

For example, the mapper may group data having a value of 0 so that the identified data having a value of 0 is repeatedly input to the systolic operator.

For example, the filter has a size of N by N (NXN), and the mapper sets the 0 value to the systolic array operator so that data is repeatedly input to the systolic array operator the same number of times as the size (N) of the filter. It is possible to collect and group data having as much as the size (N) of the filter.

For example, the electronic device further includes a memory for storing row and column positions of the data having a value of 0, and the processor reads the stored position information from the memory and , multiplication and addition operations may be skipped for data having a value of 0 with reference to the read location information.

For example, the electronic device further includes an unmapper that obtains location information on the systolic array of data identified as a value of 0 among data of the input feature map mapped to the systolic array by the mapper. wherein the unmapper comprises an adder-tree control circuit for controlling the accumulator to cumulatively add output values for each region of the input feature map based on the acquired location information of the 0-value data. can include

In order to solve the above technical problem, an embodiment of the present disclosure provides a computer-readable recording medium, wherein the recording medium repeats data having the same value a predetermined number of times according to a sequential clock. mapping the data of the input feature map into a systolic array by changing the order of the data of the input feature map so that the input feature map data is input to the multiplication and addition operator in the systolic array calculator, and the mapped data and filters of the input feature map In the step of performing a convolution operation by inputting the weight value of to the systolic array operator, and in the step of generating an output feature map by accumulating the output values obtained by the convolution operation It may contain related instructions.

This disclosure may be readily understood in combination with the detailed description that follows and the accompanying drawings, wherein reference numerals denote structural elements.
1A is a diagram illustrating a deep learning operation method using an artificial neural network, and FIG. 1B is a diagram illustrating data and a filter of an input feature map provided as an input in a deep learning operation.
1C is a diagram schematically illustrating a process of performing a convolution operation.
2 is a conceptual diagram for explaining a method of performing a convolution operation using a conventional systolic array.
3A is a diagram illustrating a method of performing a deep learning-based convolution operation according to an embodiment of the present disclosure, and FIG. 3B is a diagram illustrating an input order of data of an input feature map according to an embodiment of the present disclosure. It is a schematic diagram of a method of mapping to a systolic array.
3C is a diagram illustrating a method of generating an output feature map by performing a convolution operation using a systolic array operator and accumulating the calculated result values according to an embodiment of the present disclosure.
4 is a flowchart illustrating a method of performing a deep learning operation using a systolic array according to an embodiment of the present disclosure.
5A is a diagram illustrating a method of performing a deep learning-based convolution operation according to an embodiment of the present disclosure, and FIG. 5B is a diagram illustrating data of an input feature map and weights of a filter according to an embodiment of the present disclosure. This is a schematic diagram of how to do it.
6 is a block diagram illustrating components of an apparatus for performing a convolution operation using a systolic array according to an embodiment of the present disclosure.
7A and 7B are diagrams illustrating a method of changing the input order of input feature map data and mapping them into a systolic arrangement according to an embodiment of the present disclosure.
8 is a block diagram illustrating components of an apparatus for performing a convolution operation using a systolic array according to an embodiment of the present disclosure.
9A is a diagram illustrating data stored in a buffer of a memory, which is a component of a device according to an embodiment of the present disclosure, and FIG. 9B is a diagram illustrating convolution using a systolic array according to an embodiment of the present disclosure. It is a flow diagram showing how to perform the operation.
10A is a diagram illustrating data stored in a buffer of a memory, which is a component of a device according to an embodiment of the present disclosure, and FIG. 10B performs a convolution operation using a systolic array according to an embodiment of the present disclosure. Here is a flow chart showing how to do it.
FIG. 11A is a diagram illustrating an arithmetic circuit that performs a convolution operation using a systolic array according to an embodiment of the present disclosure, and FIG. 11B is an output feature map generated according to a convolution operation between an input feature map and a filter. It is a schematic drawing of

The terms used in the embodiments of this specification have been selected from general terms that are currently widely used as much as possible while considering the functions of the present disclosure, but they may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technologies, and the like. . In addition, in a specific case, there is also a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding embodiment. Therefore, terms used in the present disclosure should be defined based on the meaning of the term and the general content of the present disclosure, not simply the name of the term.

When it is said that a certain part "includes" a certain component throughout the specification, it means that it may further include other components without excluding other components unless otherwise stated. In addition, terms such as "...unit" and "...module" described in the specification mean a unit that processes at least one function or operation, which is implemented as hardware or software or a combination of hardware and software. It can be.

Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily carry out the present disclosure. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

1A is a schematic diagram of a deep learning operation method using an artificial neural network, and FIG. 1B is data and filters 110-1 to 110 of the input feature map 100 provided as input in the deep learning operation. -n) is a schematic diagram.

Artificial intelligence (AI) algorithms including deep learning, etc. input input data 10 to an artificial neural network (ANN), and output data 30 through operations such as convolution characterized by learning. An artificial neural network may refer to a computer science architecture modeled on a biological brain. In an artificial neural network, nodes corresponding to brain neurons are connected to each other and collectively operate to process input data. Examples of various types of neural networks include Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Deep Belief Networks (DBNs), and Restricted Boltzman Machines (Restricted Boltzman Machines). Machine, RBM) method, etc., but is not limited thereto. In a feed-forward neural network, neurons in the neural network have links to other neurons. Such connections may extend through the neural network in one direction, for example in a forward direction.

Referring to FIG. 1A, input data 10 is input to an artificial neural network, and output data 30 is generated through a convolutional neural network (CNN) 20 including one or more layers. The output structure is shown. The artificial neural network may be a deep neural network having two or more layers.

A convolutional neural network 20 may be used to extract “features” such as borders, line colors, etc. from input data 10 . The convolutional neural network 20 may include a plurality of layers. Each layer may receive data, and may process data input to the corresponding layer to generate data output from the corresponding layer. The data output from the layer may be a feature map generated by convolving an image or a feature map input to the convolutional neural network 20 with weight values of one or more filters. there is. Initial layers of convolutional neural network 20 may be operated to extract low-level features such as edges or gradients from the input. Subsequent layers of the convolutional neural network 20 may extract progressively more complex features such as eyes, nose, etc. in the image.

Referring to FIG. 1B , an input feature map 100 provided as an input to an artificial neural network and a plurality of filters 110-1 to 110-n are shown. The input feature map 100 may be a set of pixel values or numerical data of an image input to the artificial neural network, but is not limited thereto. In FIG. 1B , the input feature map 100 may be defined as a pixel value of an image to be learned through an artificial neural network. For example, the input feature map 100 may have 256×256 pixels and a depth of K. However, the value is an example, and the pixel size of the input feature map 100 is not limited to the example.

The number of filters 110-1 to 110-n may be N. Each of the plurality of filters 110-1 to 110-n may include a weight value of n by n (n×n). For example, each of the plurality of filters 110-1 to 110-n may have 3×3 pixels and K depth values. However, the size of the filter is exemplary, and the size of each of the plurality of filters 110-1 to 110-n is not limited to the above example.

1C is a diagram illustrating a process of performing a convolution operation based on deep learning.

Referring to FIG. 1C, in the process of performing the convolution operation in the artificial neural network, an output value is generated by performing multiplication and addition operations between the input feature map 100 and the filter 110 in each layer, and the output value This may refer to a process of generating the output feature map 120 by accumulating and summing.

The process of performing the convolution operation is a process of performing multiplication and addition operations by applying a filter 110 of a constant size, that is, n×n size, from the upper left corner to the lower right corner of the input feature map 100 in the current layer. Hereinafter, a process of performing a convolution operation when the size of the filter 110 is 3×3 will be described.

For example, first, in the first region 101 at the upper left of the input feature map 100, a total of 9 data (x ₁₁ ) including 3 × 3, that is, 3 data in the first direction and 3 data in the second direction to x ₃₃ ) and weight values (w ₁₁ to w ₃₃ ) of the filter 110 are multiplied. Afterwards, the output values of the multiplication operation, i.e. x ₁₁ *w ₁₁ , x ₁₂ *w ₁₂ , x ₁₃ *w ₁₃ , x ₂₁ *w ₂₁ , x ₂₂ *w ₂₂ , x ₂₃ *w ₂₃ , x ₃₁ *w ₃₁ , x ₃₂ *w ₃₂ , and x ₃₃ *w ₃₃ are accumulated and summed to generate the 1-1st output data y ₁₁ of the output feature map 120 .

Thereafter, calculation is performed while moving from the first area 101 at the upper left of the input feature map 100 to the second area 102 by a unit of data. In this case, the number of data movements in the input feature map 100 during the convolution operation is called a stride, and the size of the generated output feature map 120 may be determined according to the size of the stride. For example, when the stride is 1, an operation of multiplying a total of 9 input data (x ₁₂ to x ₃₄ ) included in the second region 102 by weight values (w ₁₁ to w ₃₃ ) of the filter 110 is performed. and the output values of the multiplication operation are x ₁₂ *w ₁₁ , x ₁₃ *w ₁₂ , x ₁₄ *w ₁₃ , x ₂₂ *w ₂₁ , x ₂₃ *w ₂₂ , x ₂₄ *w ₂₃ , x ₃₂ *w ₃₁ , When x ₃₃ *w ₃₂ and x ₃₄ *w ₃₃ are accumulated and summed, first-second output data y ₁₂ of the output feature map 120 is generated.

When generating the output feature map 120 by performing the convolution operation shown in FIG. it becomes a lot In the case of an image file having a large pixel size, an operation in units of gigabytes per second must be performed, which increases the amount of time and power required for the operation.

2 is a conceptual diagram for explaining a method of performing a convolution operation using a conventional systolic array.

Referring to FIG. 2, each data of the input feature map 200 is sequentially inputted to the multiplication and addition operators 211 to 219 according to a clock having a certain latency. Mapping into a systolic array It can be. For example, at the first clock, the 1-1 data (x ₁₁ ) of the first row (①) of the systolic array may be input to the first multiplication and addition operator 211 . Although not shown in FIG. 2 , the 1-1 data (x ₁₁ ) may be multiplied by the weight value of w ₁₁ at the first clock. Then, at the second clock, the 1-1 data (x ₁₁ ) is input to the second multiplication-add operator 212, and the 2-1 data (x ₂₁ ) is input to the first multiplication-add operator 211, The 1-2 data (x ₁₂ ) may be input to the fourth multiplication operator 214 . Similarly, at the third clock, the 1-1 data (x ₁₁ ) is input to the 3 multiplication addition operator 213, and the 2-1 data (x ₂₁ ) is input to the 2 multiplication addition operator 212, The 1-2 data (x ₁₂ ) may be input to the fifth multiplication and addition operator 215 . At the third clock, the 3-1st data (x ₃₁ ) is input to the first multiplication-adding operator 211, the 2-2nd data (x ₂₂ ) is input to the 4th multiplication-adding operator 214, and the first The -3 data (x ₁₃ ) may be input to the seventh multiplication and addition operator 217 .

As described above, the input data 200 is input to each multiplication and addition operator in the multiplication and addition operators 211 to 219 according to sequential clocks, and multiplication and addition operations are performed with the input weight value according to each clock. there is. An output feature map may be generated by cumulatively summing values output through multiplication and addition operations of respective data of the input feature map 200 input sequentially and weight values.

The convolution operation using the systolic arrangement shown in FIG. 2 may have a higher computational speed than the convolution operation shown in FIG. 1C.

3A is a diagram illustrating a method of performing a deep learning-based convolution operation.

Referring to FIG. 3A , a process of performing multiplication and addition operations by applying a filter 310 of a constant size, that is, a 3×3 size, from the upper left corner to the lower right corner of the input feature map 300 is illustrated.

In the convolution operation according to an embodiment of the present disclosure, first, in the first area 301 at the top left of the input feature map 300, 3 × 3, that is, 3 data in the first direction and 3 data in the second direction An operation of multiplying a total of 9 data (x ₁₁ to x ₃₃ ) including the weight value (w ₁₁ to w ₃₃ ) of the filter 310 is performed. Afterwards, the output values of the multiplication operation, i.e. x ₁₁ *w ₁₁ , x ₁₂ *w ₁₂ , x ₁₃ *w ₁₃ , x ₂₁ *w ₂₁ , x ₂₂ *w ₂₂ , x ₂₃ *w ₂₃ , x ₃₁ *w ₃₁ , x ₃₂ *w ₃₂ , and x ₃₃ *w ₃₃ are accumulated and summed to generate the 1-1st output data y ₁₁ of the output feature map 320 .

Thereafter, stride by 1 from the first area 301 at the upper left of the input feature map 300 to the second area 302, and a total of 9 input data (x ₁₂ to x ₃₄ ) is multiplied by the weight values (w ₁₁ to w ₃₃ ) of the filter 310, and the output values of the multiplication operation are x ₁₂ *w ₁₁ , x ₁₃ *w ₁₂ , x ₁₄ *w ₁₃ , x ₂₂ * When w ₂₁ , x ₂₃ *w ₂₂ , x ₂₄ *w ₂₃ , x ₃₂ *w ₃₁ , x ₃₃ *w ₃₂ , and x ₃₄ *w ₃₃ are accumulated and summed, the 1-2 outputs of the output feature map 320 Data (y ₁₂ ) is created. When the size of the stride is 1, overlapping data of the input feature map 300 between the first area 301 and the second area 302 and used for calculation are x ₁₂ , x ₁₃ , x ₂₂ , x This is a total of 6 data including ₂₃ , x ₃₂ , x ₃₃ .

When the stride is continued to reach the third area 303, a total of 9 data included in the third area 303, that is, x21 to x43 are multiplied by the weight values (w ₁₁ to w ₃₃ ) of the filter 310 The operation is performed, and the output values of the multiplication operation are x ₂₁ *w ₁₁ , x ₂₂ *w ₁₂ , x ₂₃ *w ₁₃ , x ₃₁ *w ₂₁ , x ₃₂ *w ₂₂ , x ₃₃ *w ₂₃ , x ₄₁ *w When ₃₁ , x ₄₂ *w ₃₂ , and x ₄₃ *w ₃₃ are accumulated and summed, the 2-1st output data y ₂₁ of the output feature map 320 is generated. Among the data of the input feature map 300 between the third area 303 and the first area 301, the overlapped data used for calculation are x ₂₁ , x ₂₂ , x ₂₃ , x ₃₁ , x ₃₂ , x ₃₃ A total of 6 data included.

That is, when the stride is 1, the input data (x ₂₂ , x ₂₃ , x ₃₂ , x ₃₃ , x ₄₂ , x ₄₃ ) included in the specific area 300A among the input data are duplicated 3 times in total to perform multiplication and addition operations. will be entered Here, the number of repetitions may be determined according to the size of the filter and the value of the stride. As the data of the input feature map 300 is sequentially input as x ₁₁ , x ₁₂ , x ₁₃ , x ₁₂ , x ₁₃ , x ₁₄ , x ₁₃ , x ₁₄ , x ₁₅ ,..., data for each clock must be toggled, which increases power consumption.

3B is a diagram illustrating a method of changing the input sequence of data of the input feature map 300 and mapping them into a systolic arrangement according to an embodiment of the present disclosure.

Referring to FIG. 3B, data of the input feature map 300 is mapped into a systolic array in which data is input according to sequential clocks, and data having the same value of the input feature map 300 is repeated to form a systolic array. The order of data of the input feature map 300 may be changed to be input to each of the multiplication and addition operators 311 to 319 in the Tolik array operator. For example, in the first row (①) of the input feature map 300 mapped to the systolic array, data input to the systolic array operator according to the first clock, the second clock, and the third clock are all the same x ₁₁ (300-1). In addition, in the first row (①), data input to the systolic array calculator according to the fourth clock, fifth clock, and sixth clock are all the same x ₁₂ (300-2), and the seventh clock and the eighth clock and data input to the systolic array operator according to the ninth clock may be the same x ₁₃ (300-3).

Similarly, data input to the systolic array operator according to the second clock, the third clock, and the fourth clock in the second row (②) of the input feature map 300 mapped to the systolic array are all the same x ₁₂ (300 -4), and all data input to the systolic array calculator according to the third clock, fourth clock, and fifth clock in the third row (③) may be the same x ₁₃ (300-5).

Looking at each multiplication-adding operator 311 to 319 in the systolic array operator, the first multiplication-adding operator 311 has x ₁₁ data 300-1 of the input feature map 300 according to the first clock to the third clock. ) may be repeatedly input. Similarly, x ₁₂ data 300-4 of the input feature map 300 is repeatedly input to the fourth multiplication operator 314 according to the second to fourth clocks, and to the seventh multiplication operator 317 According to the third to fifth clocks, x ₁₃ data 300-5 may be repeatedly input.

In the embodiment shown in FIG. 3B , the input feature map 300 may be mapped into a systolic array such that data having the same value are repeatedly input to the systolic array operator a predetermined number of times. Here, the preset number of times may be the same as the size of the filter 310 . That is, when the size of the filter is n×n, the input feature map 300 may be mapped such that data having the same value are repeatedly input to the systolic array calculator n times. For example, when the size of the filter 310 is 3 × 3, x ₁₁ data (300-1), x ₂₁ data (300-2), x ₃₁ data (300-3) of the input feature map 300 , x ₁₂ data 300-4 and x ₁₃ data 300-5 may be repeated three times and input to the systolic array calculator, so that the input sequence may be changed.

However, the number of repetitions of data of the input feature map 300 is not limited to the size of the filter. In one embodiment, the number of times the data of the input feature map 300 is repeated and inputted to the systolic array operator may be twice. This will be described with reference to FIGS. 5A and 5B.

3C is a diagram illustrating a method of generating an output feature map by performing a convolution operation using a systolic array operator and accumulating the calculated result values according to an embodiment of the present disclosure.

Referring to FIG. 3C, the input feature map and filter weight values are input to the systolic array calculator, and operations of multiplying and adding the input feature map data and filter weight values are performed according to sequential clocks. is shown For example, at the first clock, the 1-1 data (x ₁₁ ) and the 1-1 weight (w ₁₁ ) may be multiplied by the 1-1 operator 1-1. At the second clock, the 1-1 data (x ₁₁ ) and the 1-2 weight (w ₁₂ ) are multiplied by the 1-1 operator (1-1), and x ₁₁ * calculated at the first clock Can be added with w ₁₁ . Similarly, at the second clock, the 1-1 data (x ₁₁ ) is multiplied by the 2-1 weight (w ₂₁ ) and the 2-1 operator 2-1, and the 1-2 data (x ₁₂ ) may be multiplied with the 2-1 weight (w ₂₁ ) by the 2-2 operator 2-2. At the third clock, the 1-1 data (x ₁₁ ) and the 1-3 weight (w ₁₃ ) are multiplied by the 1-1 operator (1-1), and the pre-calculated x ₁₁ *w ₁₁ + Can be added to x ₁₁ *x ₁₂ . In addition, at the third clock, the 1-1 data (x ₁₁ ) and the 2-2 weight (w ₂₂ ) are multiplied by the 2-2 operator 2-2, and the pre-calculated x ₁₁ *w ₂₁ and can be added. Referring to the lower part of FIG. 3C , each data of the input feature map and the weight value of the filter may be sequentially multiplied and added according to each cycle by a plurality of systolic array operators and outputted according to the above method.

In the embodiment shown in FIG. 3C , an output value calculated by the systolic array operator is selected based on a combination of weight values of the filter and data of the input feature map that is changed as the filter is strided on the input feature map, and the selected An output feature map may be generated by cumulative summing of the output values. That is, it is calculated by a systolic array operator based on a combination between data of the input feature map and weights of the filter, which are changed as the filter is strided in the first direction (horizontal direction) and the second direction (vertical direction) on the input feature map. When selected output values are selected and the selected output values are cumulatively summed, an output feature map identical to that obtained by performing the convolution operation may be obtained.

For example, x ₁₁ *w ₁₁ multiplied by the 1-1 operator (1-1), x ₁₂ *w ₁₂ multiplied by the 1-2 operator (1-2), and 1-3 x ₁₃ *w ₁₃ multiplied by operator 1-3, x ₂₁ *w ₂₁ multiplied by 2-1 operator 2-1, multiplied by 2-2 operator 2-2 x ₂₂ *w ₂₂ calculated, x ₂₃ *w ₂₃ multiplied by the 2-3 operator (2-3), x ₃₁ *w ₃₁ multiplied by the 3-1 operator (3-1), When x ₃₂ *w ₃₂ multiplied by the 3-2 operator 3-2 and x ₃₃ *w ₃₃ multiplied by the 3-3 operator 3-3 are selectively cumulatively summed, an output A 1-1st output value (y ₁₁ ) of the feature map may be generated. Similarly, x ₁₂ *w ₁₁ multiplied by the 1-2 operator 1-2, x ₁₃ *w ₁₂ multiplied by the 1-3 operator 1-3, and the 1-4 operator ( 1-4) multiplication operation x ₁₄ *w ₁₃ , multiplication operation by the 2-2 operator (2-2) x ₂₂ *w ₂₁ , multiplication operation by the 2-3 operator (2-3) x ₂₃ *w ₂₂ , x ₂₄ *w ₂₃ multiplied by the 2-4 operator (2-4), x ₃₂ *w ₃₁ multiplied by the 3-2 operator (3-2), the third When x ₃₃ *w ₃₂ multiplied by the -3 operator 3-3 and x ₃₄ *w ₃₃ multiplied by the 3-4 operator 3-4 are selectively cumulatively summed, the output feature map A 1-2 output value (y ₁₂ ) of may be generated.

Since the conventional systolic array operation uses a method of sequentially inputting the data of the input feature map 300 without considering input data that is operated repeatedly, the value of the input data must be toggled according to all clocks. do. For example, the data of the input feature map 300 is input as x ₁₁ , x ₁₂ , x ₁₃ , x ₁₂ , x ₁₃ , x ₁₄ , ... , according to sequential clocks, and the bus is input at every clock. Data should be toggled. In general, in deep learning operations using convolutional neural networks, data operations in giga units per second must be performed. power will increase. In a mobile device, application, or computing environment that must use only limited power, deep learning through convolution operation is inevitably limited.

Referring back to FIG. 3A, as the filter 310 is strided by 1 in the first direction (horizontal direction) and the second direction (vertical direction) on the input feature map 300, x of the data of the input feature map 300 ₂₂ , x ₂₃ , x ₃₂ , x ₃₃ , x ₄₂ , x ₄₃ can be multiplied three times in total. In an embodiment of the present disclosure, the data of the input feature map 300 is overlapped by the number of times that the data of the input feature map 300 is input to multiplication and addition operations and reused, thereby repeatedly systolic arrangement Data of the input feature map 300 may be mapped to be input to the calculator. In one embodiment, the number of repetitions of data input to the systolic array operator may be determined by the size of the filter 310 . For example, when the size of the filter 310 is n×n, data of the input feature map 300 may be repeated n times and mapped to be input to a systolic array operator. For example, the order of data input to the multiplication and addition operators in the systolic array operator according to sequential clocks is x ₁₁ , x ₁₁ , x ₁₁ , x ₂₁ , x ₂₁ , x ₂₁ , x ₃₁ , x ₃₁ , x Data of the input feature map 300 may be mapped so that it is repeated three times in the order of ₃₁ , ... , .

In addition, referring to the embodiment shown in FIG. 3C, the data of the mapped input feature map 300 is repeatedly input according to each clock, and the output value is multiplied and added by the input data and the weight value of the filter. may optionally be cumulatively summed to generate an output feature map.

Therefore, according to the embodiments shown in FIGS. 3B and 3C, since data having the same value among the input feature maps 300 is repeatedly input to the systolic array operator, the amount of memory access and Since the amount of bus data toggle is significantly reduced, the amount of power consumed inside a processor or chip can be reduced.

4 is a flowchart illustrating a method of performing a deep learning operation using a systolic array according to an embodiment of the present disclosure. Each step (S410 to S430) shown in FIG. 4 includes a computing device or application processor (AP) including a processor or chip that performs a deep learning operation based on a convolutional neural network. It can be performed by a mobile device that does. Hereinafter, the term 'electronic device' encompassing a computing device and a mobile device will be used.

In step S410, the electronic device maps data of the input feature map into a systolic array so that data having the same value is repeated a predetermined number of times and input to a multiplication and addition operator in the systolic array operator. The electronic device may map data of the input feature map into a systolic array so as to be sequentially input to the multiplication operator according to a clock having a certain latency. In an embodiment, the electronic device may change the input order of input feature map data in the systolic array so that input data input to the multiplication and addition operator in the systolic array is repeatedly input at least twice. In one embodiment, the number of repetitions of data input to the multiplication and addition operator in the systolic array operator may be determined by the size of the filter. For example, when the size of the filter is n×n, the electronic device may map data of the input feature map such that the data of the input feature map is repeated n times and input to the systolic array calculator.

In step S420, the electronic device performs a convolution operation by inputting the mapped input feature map data and the weight value of the filter to a systolic array calculator. In one embodiment, the electronic device repeatedly inputs the data of the input feature map to the multiplication-adding operator in the systolic array operator a predetermined number of times according to sequential clocks, and the weight value of the filter is input to the multiplication-adding operator according to each clock. Convolution operation can be performed while inputting.

In step S430, the electronic device generates an output feature map by accumulating the output values obtained by the convolution operation. In an embodiment, the electronic device may generate output data by accumulating and summing input feature map data calculated by each of a plurality of multiplication and addition operators in the systolic array operator and the weight product of the filter. The electronic device may selectively cumulatively add a plurality of output data calculated by a plurality of multiplication operators by referring to a stride of the filter on the input feature map. In an embodiment, the electronic device may generate an output feature map by selecting a plurality of output values based on a combination between filter weights and data of an input feature map according to a stride value and cumulatively summing the selected output values. .

5A is a diagram illustrating a method of performing a deep learning-based convolution operation.

Referring to FIG. 5A , a filter 510 having a size of 3×3 is shown that multiplies and adds data values of the input feature map 500 while moving by a stride of 1 on the input feature map 500 .

In the embodiment shown in FIG. 5A, a total of 9 data including 3 data in the first direction and 3 data in the second direction is 3×3 on the first area 501 at the top left of the input feature map 500. An operation of multiplying each of the data (x ₁₁ to x ₃₃ ) with weight values (w ₁₁ to w ₃₃ ) of the filter 510 is performed. The output values of the multiplication operation, i.e. x ₁₁ *w ₁₁ , x ₁₂ *w ₁₂ , x ₁₃ *w ₁₃ , x ₂₁ *w ₂₁ , x ₂₂ *w ₂₂ , x ₂₃ *w ₂₃ , x ₃₁ *w ₃₁ , x When ₃₂ *w ₃₂ and x ₃₃ *w ₃₃ are accumulated and summed, the 1-1st output data y ₁₁ of the output feature map 520 is generated.

Thereafter, a stride is performed by 1 from the first area 501 of the input feature map 500 to the second area 502, and a total of 9 input data (x ₁₂ to x ₃₄ ) included in the second area 502 is When an operation of multiplying with weight values (w ₁₁ to w ₃₃ ) of the filter 510 is performed, x ₁₂ *w ₁₁ , x ₁₃ *w ₁₂ , x ₁₄ *w ₁₃ , x ₂₂ *w ₂₁ , which are output values of the multiplication operation, x ₂₃ *w ₂₂ , x ₂₄ *w ₂₃ , x ₃₂ *w ₃₁ , x ₃₃ *w ₃₂ , x ₃₄ *w ₃₃ are cumulatively summed to obtain 1-2 output data (y ₁₂ ) of the output feature map 520 is created

In the above-described embodiment, when the size of the stride is 1, among data of the input feature map 500 overlapping between the first area 501 and the second area 502, data used for calculation are x ₁₂ , x It is a total of 6 data including ₁₃ , x ₂₂ , x ₂₃ , x ₃₂ , x ₃₃ . That is, while multiplication and addition operations for the first area 501 and multiplication and addition operations for the second area 502 are sequentially calculated, x ₁₂ , x ₁₃ , x ₂₂ of the data of the input feature map 500 , x ₂₃ , x ₃₂ , x ₃₃ are each used twice. Here, the meaning of 'used' may mean that each value of the input data 500 is sequentially read according to a clock within a processor or chip. Since data values are changed according to each clock, bus data must be toggled for each sequential clock inside a processor or chip, so that a large amount of processing is required and thus, a large amount of power may be consumed.

5B is a diagram illustrating a method of performing a convolution operation between data of an input feature map and weights of a filter using a systolic array according to an embodiment of the present disclosure.

Referring to FIG. 5B , the input feature map 500 may be mapped into a systolic arrangement in which data is input to the systolic operator 530 according to sequential clocks. In one embodiment, the data of the input feature map 500 may be mapped such that data having the same value are repeated twice and input into the systolic operator 530 according to a clock. For example, in the first row of the input feature map 500 mapped to a systolic array, the input data is x ₁₁ , x ₁₂ , x ₁₂ , x ₁₃ , x ₁₃ , x ₁₄ ,... , in order of systolic It can be input to the calculator 530. In the first row, the input data x ₁₂ (500-1) may be repeated twice and then input to the systolic operator 530. For example, x ₁₂ may be input to the systolic operator 530 at the second clock, and x ₁₂ may be input to the systolic operator 530 at the third clock. Similarly, the input data x ₁₃ (500-2) in the first row may also be repeated twice and input to the systolic operator 530. For example, x ₁₃ may be input to the systolic operator 530 at the fourth clock, and x ₁₃ may be input to the systolic operator 530 at the fifth clock.

Also in the second row of the input feature map 500 mapped in the systolic array, the input data may be repeated twice according to the sequential clock and input to the systolic operator 530 . For example, the data of the input feature map 500 in the second row of the systolic array is x ₁₃ , x ₁₄ , x ₁₄ , x ₁₅ , x ₁₅ , x ₁₆ , ... , in order of the systolic operator 530 ) can be entered. Similar to the first row, in the second row, x ₁₄ (500-3) of the input data is repeated twice and inputted to the systolic operator 530, and x ₁₅ (500-4) is also repeated twice and inputted to the systolic operator 530. (530).

Each data of the input feature map 500 shown in FIG. 5B may be multiplied and added with the filter 510 input to the systolic operator 530 according to sequential clocks. The weight value of the filter 510 may be input to the systolic operator 530 by overlapping twice according to sequential clocks.

In the embodiment shown in FIG. 5B , the input feature map 500 may be mapped in a systolic array such that data having the same value are repeated twice and input to the systolic operator 530 . According to the embodiment of FIG. 5B, values of data input to the systolic operator 530 are duplicated twice according to sequential clocks, so data input to the systolic operator 530 is changed according to all conventional clocks. Compared to the case, the number of toggles of bus data can be remarkably reduced. Accordingly, power consumption inside a processor or chip that performs multiplication and addition operations through the systolic calculator 530 can be reduced.

6 is a block diagram illustrating components of an electronic device 600 that performs a convolution operation using a systolic array according to an embodiment of the present disclosure.

Referring to FIG. 6 , an electronic device 600 may include a processor 610 . Here, the electronic device 600 may be a computing device or a mobile device that performs a deep learning operation based on a convolutional neural network.

The processor 610 may include a processing element (PE) that performs multiplication and addition operations. The processor 610 may include at least one of a central processing unit, a microprocessor, and a graphic processing unit, but is not limited thereto. In one embodiment, when the electronic device 600 is a mobile device such as a smart phone or a tablet PC, the processor 610 may be an application processor (AP) that executes an application.

In one embodiment, the processor 610 may access memory to read data required for calculation and to store the calculation result again. For example, the processor 610 may execute a load instruction for reading data from memory and a store instruction for storing data in memory.

The processor 610 may include a mapper 611 , a systolic operator 612 , and an adder 613 .

The mapper 611 repeats data having the same value with respect to the data of the input feature map or the weight value of the filter according to sequential clocks and inputs the data of the input feature map or the weight value of the filter to the systolic operator 612. It is possible to perform mapping to change the input sequence of . In one embodiment, the mapper 611 performs an input order of data of the input feature map so that the data of the input feature map is repeatedly input to the multiplication-add operator in the systolic operator 611 the same number of times as the size (N) of the filter ( sequence) can be changed. For example, if the size of the filter is 3 × 3, the mapper 611 generates an input feature map so that each input data x ₁₁ , x ₁₂ , x ₁₃ , ..., is repeatedly input to the multiplication-add operator three times. can be mapped to a systolic array. However, the mapper 611 is not limited thereto, and the mapper 611 may map the input feature map so that the data of the input feature map is repeated twice and input to the multiplication operator.

The systolic operator 612 may include a plurality of multiplication and addition operators (Processing Elements). Each of the plurality of multiplication and addition operators may be configured as a flip-flop circuit. Each of the plurality of multiplication and addition operators may perform an operation of multiplying input data and a weight value of a filter according to sequential clocks having a constant latency and adding the multiplied values.

The accumulator 613 may generate an output feature map by accumulating and summing output values obtained by multiplying the data of the input feature map by the weight value of the filter. The accumulator 613 calculates the value of the multiplication-add operator of the systolic array operator 612 based on a combination of the weight value of the filter and the data of the input feature map, which is changed as the filter strides on the input feature map. You can select the calculated output value. The accumulator 613 may generate an output feature map by cumulatively summing the selected output values.

7A and 7B are diagrams illustrating a method of changing the input order of input feature map data and mapping them into a systolic arrangement according to an embodiment of the present disclosure.

Referring to FIG. 7A , data of the input feature map 700 may be mapped into a systolic arrangement in which data is input according to sequential clocks. The input feature map 700 may include data having a value of 0. For example, when the input feature map 700 is an image to be learned through deep learning, a pixel value of a black pixel may be 0. In the example shown in FIG. 7A , pixel values of x ₂₂ and x ₃₃ of the data of the input feature map 700 may be 0.

If a pixel value of the data of the input feature map 700 is 0, even if a multiplication operation is performed with the weight of the filter, there is no need to perform the operation because the output value is 0. Therefore, a method of identifying data having a value of 0 among the data of the input feature map 700 and not performing multiplication and addition operations with reference to the position of the identified 0 data is called zero-skipping.

In the embodiment shown in FIG. 7A, x ₂₂ having a value of 0 at the third clock is input to the multiplication-adding operator in the systolic array operator 710, and x ₂₂ having a value of 0 at the 5th clock is input to the multiplication-adding operator can be entered. Similarly, x ₃₃ having a value of 0 at the seventh and ninth clocks may be input to the multiplication and addition operator in the systolic array operator 710 . In the first row (①) of the input feature map 700 mapped in a systolic array, x ₁₂ having a value other than 0 is input at the fourth clock, but x ₂₂ having a value 0 at the fifth clock is a systolic array It is input to the calculator 710. Thereafter, x ₃₂ having a value other than 0 may be input to the systolic array calculator 710 according to the sixth clock in the first row ①. In the second row (②), data x ₂₂ and x ₃₃ having a value of 0 are alternately inputted to the systolic array calculator along with data having a value other than 0 according to sequential clocks.

When data having a value of 0 and data having a value other than 0 are input alternately according to sequential clocks, a toggle of bus data occurs with respect to all clocks, allowing the processor or chip to access the memory. This increases the amount of power, and thus power consumption may increase.

7B is a diagram illustrating a method of changing an input sequence of data of an input feature map 700 and mapping them into a systolic arrangement according to an embodiment of the present disclosure.

Referring to FIG. 7B, data of the input feature map 700 is mapped into a systolic array in which data is input according to sequential clocks, and data having the same value of the input feature map 700 is repeated to obtain a systolic array. The order of data of the input feature map 700 may be changed to be input to each multiplication and addition operator in the Tolik array operator 710. In addition, data (x ₂₂ and x ₃₃ ) having a value of 0 among data of the input feature map 700 may be grouped so as to be repeatedly input to the systolic operator 710 . For example, data x ₂₂ having a value of 0 may be grouped into a first group 701 to be repeated a predetermined number of times and input to the systolic array operator 710 . Likewise, data x ₃₃ having a value of 0 may be grouped into a second group 702 such that it is repeated the same number of times as the number of repetitions of x ₂₂ and then input to the systolic array operator 710 .

In one embodiment, the number of repetitions of data having a value of 0 may be determined by the size of the filter. For example, when the size of the filter is n×n, a value of 0 may be repeated n times and mapped to be input to the systolic operator 710. In the embodiment shown in FIG. 7B, since the filter has a size of 3×3, data x ₂₂ and x ₃₃ having a value of 0 may be repeated three times and mapped to be input to the systolic operator 710.

Referring to FIG. 7A, in the conventional convolution operation method using a systolic array, data having a value of 0 among the data of the input feature map 700 are sporadically scattered, and thus, the memory output for the input data relates to the convolution operation. Combining the cumulative summation was a complex structure. However, in the convolution operation method according to the embodiment shown in FIG. 7B of the present disclosure, data (x ₂₂ , x ₃₃ ) having a value of 0 among the data of the input feature map 700 are repeated a predetermined number of times to obtain a systolic operator. 710, it is possible to skip multiplication and addition operations of at least two or more input data at a time, thereby reducing memory access and power consumption of the processor. A method of reducing memory access due to grouping of input data having a value of 0 will be described in detail with reference to FIGS. 9A and 10A and 10B.

8 is a block diagram illustrating components of an electronic device 800 that performs a convolution operation using a systolic array according to an embodiment of the present disclosure. The systolic operator 812 and accumulator 813 shown in FIG. 8 are components that perform the same functions and operations as the systolic operator 612 and accumulator 613 shown in FIG. 6, respectively. omit explanation.

Referring to FIG. 8 , an electronic device 800 may include a processor 810 and an unmapper 820. In one embodiment, the processor 810 includes a central processing unit (CPU) including Processing Elements (PEs) that perform multiplication and addition operations, a microprocessor, a graphics processing unit (Graphic Processing Unit), and an application processor ( AP), but is not limited thereto.

The mapper 811 may identify data having a value of 0 among data of the input feature map. The mapper 811 may also group data having a value of 0 so that the identified data having a value of 0 is repeatedly input to the systolic operator 812 .

The mapper 811 may change the input sequence of the 0 value so that data having the identified 0 value is input to the systolic operator 812 a predetermined number of times. The predetermined number of times may be determined by the size of the filter. In one embodiment, the mapper 811 maps the data of the input feature map such that data having a value of 0 among the data of the input feature map are repeatedly input to the systolic operator 812 as many times as the size (N) of the filter. can do. For example, if the size of the filter is 3×3, the mapper 811 may map the input feature map into a systolic array so that data having a value of 0 is repeatedly input to the multiplication operator three times.

The systolic operator 812 may include a plurality of multiplication and addition operators (Processing Elements). Each of the plurality of multiplication and addition operators included in the systolic operator 812 multiplies the input data and the weight value of the filter according to sequential clocks having a constant latency, and performs an operation of adding the multiplied values. can The systolic operator 812 may skip multiplication and addition operations when data having a value of 0 is input.

An adder 813 may generate an output feature map by accumulating and summing output values obtained by multiplying the data of the input feature map by the weight value of the filter. The accumulator 813 is controlled by the accumulator control unit 822 of the unmapper 820 to receive location information (address) of data having a value of 0 among input data, and selectively accumulate and add output values by referring to this. can

The unmapper 820 may store location information of each data of the input feature map mapped by the mapper 811, and may unmap the data of the input feature map into an array before mapping based on the location information. . The relationship between the mapper 811 and the unmapper 820 is similar to that between an encoder and a decoder. The mapper 811 performs mapping to change the input sequence so that data of the input feature map are repeatedly input to the systolic operator 812 a predetermined number of times, and the unmapper 820 performs mapping of the mapped input feature map. Performs the operation of restoring back to the array before mapping.

The unmapper 820 may include a memory 821 and an accumulator controller 822 .

The memory 821 may store a look-up table (LUT) for storing location information of data having a value of 0 identified by the mapper 811 among data of the input feature map. The memory 821 may include at least one of, for example, dynamic RAM (DRAM), static RAM (SRAM), and synchronous dynamic RAM (SDRAM), but is not limited thereto. In one embodiment, the memory 821 may store information about the position of data to be accumulated and summed within the size (n × n) of the filter among output values resulting from multiplication and addition operations of input data and weight values of the filter. there is. That is, the memory 821 may store position information in row and column units of data having a value of 0 among data of the input feature map.

An adder-tree controller 822 may control the accumulator 813 to generate an output feature map by accumulating and summing output values with reference to location information of data having a value of 0 among data of the input feature map. there is. In one embodiment, the accumulator control unit 822 reads position information in units of rows and columns of data identified as 0 among the data of the input feature map stored in the memory 821 to obtain zero Information on the number of zero-skips may be referred to.

In another embodiment, the accumulator control unit 822 transfers information on how many cycles after which data having a value of 0 among the input feature map data input to the systolic operator 812 is calculated to the mapper 811 in real time. The accumulator 813 may be received from , and the accumulator 813 may be controlled to accumulate and add output values with reference to the received information. This is due to the fact that when the operation of data having a 0 value is skipped in a convolution operation using a systolic array, the output value is shifted to the top by the number of times the 0 value data is skipped. it is based on That is, the accumulator control unit 822 refers to shift information of output values according to the number of times that data having a value of 0 is skipped in multiplication and addition operations, accumulates and sums the output values, and through this, output characteristics You can create maps.

9A is a diagram illustrating data stored in a buffer of a memory, which is a component of a device according to an embodiment of the present disclosure, and FIG. 9B is a diagram illustrating convolution using a systolic array according to an embodiment of the present disclosure. It is a flow diagram showing how to perform the operation.

Assuming that there are a total of four buffers 901 to 904 in the memory 900 in FIG. 9A , non-zero input data is stored in the first buffer 901, the third buffer 903, and the fourth buffer 904. and data having a value of 0 may be stored in the second buffer 902 .

In one embodiment of the present disclosure, data having a value of 0 may not be stored in a buffer of the memory 900 . In the embodiment shown in FIG. 9A, if data having a value of 0 is not stored in the memory 900, input data x ₃₂ stored in the conventional third buffer 903 and input stored in the fourth buffer 904 Data x ₃₃ may be shifted to the second buffer 902 and the third buffer 903 and stored.

Referring to FIG. 9B , in step S910, the electronic device identifies data having a value of 0 (zero) among the data of the input feature map. In one embodiment, the mapper in the processor may identify data having a value of 0 among data of an input feature map input to the systolic array operator.

In step S920, the electronic device groups data having a value of 0 so that the data having a value of 0 is repeatedly input to the systolic array calculator. In one embodiment, the mapper allows data having a value of 0 identified in the input feature map to be inputted repeatedly to a systolic array operator according to sequential clocks having a constant latency. can be grouped.

In step S930, the electronic device stores row and column positions of data having a value of 0 in a memory. In one embodiment, the electronic device may include a memory that stores row and column position information of data having a value of 0 identified on the input feature map. In one embodiment, the memory may store row and column positions of input data having a value of 0 in the form of a look-up table (LUT). At this time, the memory may not store input data having a value of 0 in the buffer.

In step S940, the electronic device reads the stored location information from the memory, and skips multiplication and addition operations for data having a value of 0 by referring to the read location information.

In the embodiments shown in FIGS. 9A and 9B , data having a value of 0 among the data of the input feature map is not stored in the buffer of the memory 900, and row and column of the value 0 data are generated during convolution operation. column) location information, it is possible to secure more memory storage space. In addition, by reducing memory access, the amount of power consumed by a processor or chip in an electronic device may be reduced.

10A is a diagram illustrating data stored in a buffer of a memory, which is a component of a device according to an embodiment of the present disclosure, and FIG. 10B performs a convolution operation using a systolic array according to an embodiment of the present disclosure. Here is a flow chart showing how to do it.

Referring to FIG. 10A , among the four buffers 1001 to 1004 of the memory 1000, data having a value of 0 is stored in the first buffer 1001 to the third buffer 1003, and the fourth buffer 1004 In , data having a value of 0, x ₃₃ , may be stored. Unlike the embodiment shown in FIG. 9A , in the embodiment shown in FIG. 10A , data having a value of 0 is also stored in the buffer of the memory 1000 . However, even if 0-value data is stored in the memory 1000, when multiplication and addition operations are performed, the operation may be skipped through location information of the 0-value data. This will be described with reference to the flowchart of FIG. 10B.

Referring to FIG. 10B , in step S1010, the electronic device identifies data having a value of 0 (zero) among the data of the input feature map. In one embodiment, the mapper in the processor may identify data having a value of 0 among data of an input feature map input to the systolic array operator.

In step S1020, the electronic device groups data having a value of 0 so that the data having a value of 0 is repeatedly input to the systolic array operator.

In step S1030, the electronic device acquires location information of the 0-value data on the systolic array as the 0-value data is input. In one embodiment, the unmapper (see 820 in FIG. 8 ), which is a component included in the electronic device, is information about how many cycles after which zero-value data among input feature map data input to the systolic operator is calculated. may be received from the mapper (see 811 of FIG. 8 ) in real time, and the accumulator (see 813 of FIG. 8 ) may be controlled to accumulate and add output values with reference to the received information. When 0-value data is input to the systolic operator, the unmapper may receive location information of data having a 0 value on the systolic array in real time from the mapper.

In step S1040, the electronic device controls an adder-tree that accumulates and sums output values for each region of the input feature map based on location information of the 0-value data. In one embodiment, the electronic device acquires location information of the 0-value data identified in the input feature map mapped in a systolic array in real time from the mapper (811 in FIG. 8 ), and based on the location information of the obtained 0-value data and an accumulator controller ( 822 of FIG. 8 ) that controls the accumulator (see 813 of FIG. 8 ) to selectively accumulate and add output values multiplied and added by the systolic operator.

The embodiments shown in FIGS. 10A and 10B relate to a method for skipping the operation of data having a 0 value in a convolution operation using a systolic array, and the number of times 0 value data is skipped. It is a method that focuses on shifting the output value to the upper level as much as possible. That is, even if 0-value data is stored in the buffer (see 1001 to 1003 in FIG. 10A), the processor in the electronic device does not read the buffers 1001 to 1003 in which the 0-value data is stored in the memory 1000. Convolution operation values due to multiplication and addition operations are changed. In an embodiment of the present disclosure, in order to prevent an output value from being changed due to 0-value data, as 0-value data is input in multiplication and addition operations through a systolic operator, location information of input 0-value data is obtained in real time And, based on the location information of the obtained zero-value data, the output values are cumulatively summed. In addition, even if 0-value data is stored in the buffers 1001 to 1003 of the memory 1000, since the processor does not access the 0-value data, power consumption due to memory access can be reduced.

FIG. 11A is a diagram illustrating an arithmetic circuit that performs a convolution operation using a systolic array according to an embodiment of the present disclosure, and FIG. 11B is an output feature map generated according to a convolution operation between an input feature map and a filter. It is a schematic drawing of

Referring to FIG. 11A, the systolic operator 1110 includes five flip-flop circuits 1110-1 to 1110-5, a multiplication operator 1120, and two addition operators 1130-1 and 1130. -2), and two clock gate circuits 1140-1 and 1140-2. However, this is exemplary, and the flip-flop circuits 1110-1 to 1110-5 included in the systolic operator 1110, the addition operators 1130-1 and 1130-2, and the clock gate circuit 1140-1 , 1140-2) is not limited to that shown in FIG. 11A.

The first clock gate circuit 1140-1 and the second clock gate circuit 1140-2 may generate sequential clocks with a constant latency. According to sequential clocks, the weight value of the filter may be input to the first flip-flop circuit 1110-1, and the data of the input feature map may be input to the second flip-flop circuit 1110-2. In one embodiment, the weight value of the filter may be repeated twice according to a sequential clock and then input to the first flip-flop circuit 1110-1. In addition, the data of the input feature map may be mapped in a systolic arrangement so that it is repeated twice according to a sequential clock and inputted to the second flip-flop circuit 1110-2.

The weight value input to the first flip-flop circuit 1110-1 and the data input to the second flip-flop circuit 1110-2 may be multiplied through the multiplication operator 1120. The multiplication of input data output from the multiplication operator 1120 and the weight value according to sequential clocks is transferred to the first addition operator 1130-1 and the second addition operator 1130-2. At this time, the third flip-flop circuit 1110-3 may perform a binary count function by the clock gate circuits 1140-1 and 1140-2. That is, when the third flip-flop circuit 1110-3 is high, the product of the input data and the weight value is transferred to the first adder 1130-1, and the third flip-flop circuit 1110-3 In the case of low, the product of the input data and the weight value may be transmitted to the second addition operator 1130-2. The fourth flip-flop circuit 1110-4 receives a value obtained by adding the product of the input data transmitted to the first adder 1130-1 and the weight value, and the fifth flip-flop circuit 1110-5 receives the second adder 1110-5. A value obtained by adding the product of the input data transmitted to the calculator 1130-2 and the weight value may be delivered. For example, the fourth flip-flop circuit 1110-4 may output S11 (x ₁₁ *w ₁₁ ) at the first clock and output S13 (x ₁₂ *w ₁₂ ) at the third clock. In addition, the fifth flip-flop circuit 1110-5 may output S12 (x ₁₂ *w ₁₁ ) at the second clock and output S14 (x ₁₃ *w ₁₂ ) at the fourth clock. Data of an output feature map may be obtained when output values calculated according to sequential clocks are cumulatively summed in the above-described manner.

The embodiment shown in FIG. 11A requires one more combination of an addition operator and a flip-flop circuit than the conventional systolic array operation circuit. However, the amount of operation of the processor is not doubled by the increased addition operator and flip-flop circuit in the embodiment of the present disclosure because the third flip-flop circuit 1110-3 performs a binary count function.

Referring to FIG. 11B , data x ₁₁ , x ₁₂ , x ₁₃ of the input feature map 100 in the first area P1 on the input feature map FMap 1 are w ₁₁ , w ₁₂ , w of the filter 110 ₁₃ is multiplied, and the multiplied values are cumulatively summed to generate 1-1st output data (x ₁₁ *w ₁₁ +x ₁₂ *w ₁₂ +x ₁₃ *w ₁₃ ) of the output feature map 120 . In addition, in the second area P2 on the input feature map FMap 1, data x ₁₂ , x ₁₃ , and x ₁₄ of the input feature map 100 are multiplied by w ₁₁ , w ₁₂ , and w ₁₃ of the filter 110, respectively. 1-2 output data (x ₁₂ *w ₁₁ +x ₁₃ *w ₁₂ +x ₁₄ *w ₁₃ ) of the output feature map 120 is generated by accumulating and summing the values obtained from the operation and multiplication. This is the same as the result value output by the systolic calculator 1110 shown in FIG. 11A. That is, when S ₁₁ , S ₁₃ , and S ₁₅ are cumulatively added in FIG. 11A , the 1-1 output data x ₁₁ *w ₁₁ +x ₁₂ *w ₁₂ +x ₁₃ *w ₁₃ of FIG. 11B is obtained. Similarly, when S ₁₂ , S ₁₄ , and S ₁₆ are cumulatively added in FIG. 11A , the first-second output data x ₁₂ *w ₁₁ +x ₁₃ *w ₁₂ +x ₁₄ *w ₁₃ of FIG. 11B is obtained.

An electronic device according to the above-described embodiments includes a processor, a memory for storing and executing program data, a permanent storage unit such as a disk drive, a communication port for communicating with an external device, a touch panel, a key, and a button. It may include user interface devices such as the like. Methods implemented as software modules or algorithms may be stored on a computer-readable recording medium as computer-readable codes or program instructions executable on the processor. Here, the computer-readable recording medium includes magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM) ), and DVD (Digital Versatile Disc). A computer-readable recording medium may be distributed among computer systems connected through a network, and computer-readable codes may be stored and executed in a distributed manner. The medium may be readable by a computer, stored in a memory, and executed by a processor.

Embodiments of the present disclosure may be presented as functional block structures and various processing steps. These functional blocks may be implemented with any number of hardware or/and software components that perform specific functions. For example, embodiments may include integrated circuit configurations such as memory, processing, logic, look-up tables, etc., that may execute various functions by means of the control of one or more microprocessors or other control devices. can employ them. Similar to components that can be implemented as software programming or software elements, the present embodiments include data structures, processes, routines, or various algorithms implemented as combinations of other programming constructs, such as C, C++, Java ( It can be implemented in a programming or scripting language such as Java), assembler, or the like. Functional aspects may be implemented in an algorithm running on one or more processors. In addition, this embodiment may employ conventional techniques for electronic environment setting, signal processing, and/or data processing. Terms such as "mechanism", "element", "means" and "composition" may be used broadly and are not limited to mechanical and physical components. The term may include a meaning of a series of software routines in connection with a processor or the like.

Claims (19)

In the method of performing a deep learning operation using a systolic array,
The order of data of the input feature map is changed so that data having the same value is repeated a predetermined number of times according to a sequential clock and inputted to the multiplication and addition operator in the systolic array operator. mapping to an array;
performing a convolution operation by inputting the mapped data of the input feature map and weight values of the filter into the systolic array calculator; and
generating an output feature map by accumulating the output values obtained by the convolution operation;
Including, method.
According to claim 1,
In the mapping step, the input order of the input data is changed so that the same two input data values of the input feature map are repeated and input to the systolic array operator.
According to claim 1,
The filter has a size of N by N (NXN),
Wherein the mapping step changes the input order of the data of the input feature map so that the data of the input feature map is repeatedly input to the systolic array calculator a number of times equal to the size (N) of the filter.
According to claim 1,
Generating the output feature map comprises:
selecting output values obtained by the convolution operation in consideration of a stride value of the filter and data of the input feature map overlapping in a first direction and a second direction due to the stride; and
generating an output feature map resulting from a combination of the input feature map and the filter by cumulatively summing the selected output values;
Including, method.
According to claim 1,
identifying data having a value of 0 among data of the input feature map; Further comprising a method.
◈Claim 6 was abandoned when the registration fee was paid.◈ According to claim 5,
In the mapping step, the data having a value of 0 are grouped so that the data having a value of 0 is repeatedly input to the systolic array calculator.
◈Claim 7 was abandoned when the registration fee was paid.◈ According to claim 5,
The filter has a size of N by N (NXN),
In the mapping step, the data having a value of 0 is collected by the size (N) of the filter and grouped so that the data having the value of 0 is repeatedly input to the systolic array operator the same number of times as the size (N) of the filter. , Way.
◈Claim 8 was abandoned when the registration fee was paid.◈ According to claim 5,
storing row and column positions of the data having a value of 0 on the input feature map in a memory; Including more,
The performing of the convolution operation may include reading the stored location information from the memory and skipping multiplication and addition operations for data having a value of 0 by referring to the read location information. Way.
◈Claim 9 was abandoned when the registration fee was paid.◈ According to claim 5,
obtaining location information on the systolic array of the zero-value data when the data having the zero-value is identified; Including more,
In the generating of the output feature map, output values for each region of the input feature map are cumulatively summed based on the acquired location information of the 0-value data.
In an electronic device that performs deep learning calculations,
A processor performing a convolution operation between data of an input feature map and a weight value of a filter using a systolic array;
the processor,
Mapping the data of the input feature map into a systolic array by changing the order of data of the input feature map so that data having the same value is repeated a predetermined number of times according to a sequential clock and input to the multiplication and addition operator mapper;
a systolic array operator including a plurality of multiplication and addition operators performing a convolution operation on the mapped input feature map data or the weight value of the filter; and
an adder generating an output feature map by accumulating and summing the convolutional output values;
Including, electronic device.
According to claim 10,
The mapper changes an input order of the input data such that two identical input data values of the input feature map are repeated and input to the systolic array operator.
According to claim 10,
The filter has a size of N by N (NXN),
The mapper changes the input order of the data of the input feature map so that the data of the input feature map is repeatedly input to the multiplication operator in the systolic array operator as many times as the size (N) of the filter. Device.
According to claim 10,
The accumulator selects the output values subjected to the convolution operation in consideration of a stride value of the filter and data of the input feature map overlapping in a first direction and a second direction due to the stride, and selects the selected output values An electronic device generating an output feature map for each region of the input feature map by cumulative summation.
According to claim 10,
The mapper identifies data having a value of 0 among the data of the input feature map.
◈Claim 15 was abandoned when the registration fee was paid.◈ According to claim 14,
The mapper groups data having a value of 0 so that the identified data having a value of 0 is repeatedly input to the systolic array calculator.
◈Claim 16 was abandoned when the registration fee was paid.◈ According to claim 14,
The filter has a size of N by N (NXN),
The mapper collects and groups the data having a value of 0 by the size (N) of the filter so that the data having the value of 0 is repeatedly input to the systolic array operator the same number of times as the size (N) of the filter. electronic device.
◈Claim 17 was abandoned when the registration fee was paid.◈ According to claim 14,
a memory for storing row and column positions of the data having a value of 0; Including more,
The electronic device of claim 1 , wherein the processor reads the stored location information from the memory and skips multiplication and addition operations for data having a value of 0 by referring to the read location information.
◈Claim 18 was abandoned when the registration fee was paid.◈ According to claim 10,
an unmapper for obtaining location information on the systolic array of data identified as a value of 0 among data of the input feature map mapped to the systolic array by the mapper; Including more,
The unmapper includes an adder-tree control circuit for controlling the accumulator to accumulate and sum output values for each region of the input feature map based on the obtained location information of the 0-value data. electronic device.
◈Claim 19 was abandoned when the registration fee was paid.◈ A computer-readable recording medium recording a program for executing the method of claim 1 on a computer.

Images