KR20210059623A - Electronic device and method for inference Binary and Ternary Neural Networks - Google Patents

Abstract

An embodiment provides a method of calculating a dot product for binary data, ternary data, non-binary data, and non-ternary data using an electronic device. The method includes designing to compute the dot product on ternary data. The method includes designing a fused bitwise data path to support dot product calculations for binary data and ternary data. The method includes designing an FPL data path to compute a dot product between one of the non-binary data and the non-ternary data and one of the binary data and the ternary data. The method includes distributing the dot product of the binary data and the ternary data and the dot product of one of the non-binary data and the non-ternary data and one of the binary data and the ternary data to the fused bitwise data path and the FPL data path.

Description

Electronic device and method for inference Binary and Ternary Neural Networks}

The present disclosure relates to an electronic device, and more particularly, to an electronic device and a method for inferring a binary neural network (BNN) and a ternary neural network (TNN).

Convolutional Neural Network (CNN) is a kind of deep neural network most commonly applied in computer vision. However, in conventional central processing unit/graphic processing unit (CPU/GPU) based systems using CNNs, the high accuracy of computing is achieved by significant energy consumption and overhead. Model quantization technology is used to alleviate the problem caused by CNN-based computing. Binary neural networks (BNN) and three-dimensional neural networks (TNN) are quantization techniques that can express model data (ie, parameters, activations, etc.) in 1 bit and 2 bits, respectively. Various BNN and TNN models (ie, data types) and operands used in each model may be shown in Table 1.

Data type Operand 1 Operand 2 BNN_A {0,1} {0,1} BNN_B {-1,+1} {-1,+1} TNN {-1,0,+1} {-1,0,+1} TNN_BNN_A {-1,0,+1} {0,1} TNN_BNN_B {-1,0,+1} {-1,+1}

In general, a hardware accelerator in an electronic device can process specific data types. However, hardware accelerators cannot handle all data types. Therefore, multiple hardware accelerators are required in an electronic device to process multiple data types. To implement multiple hardware accelerators on the electronic chip of the electronic device, more areas are required, which undesirably increases the overall size of the electronic device. In addition, a large number of components are required to implement multiple hardware accelerators in an electronic device, which undesirably increases the manufacturing cost of the electronic device. In addition, overall power consumption of the electronic device further increases due to power consumption for operating multiple hardware accelerators in the electronic device. Accordingly, it is desirable to solve the aforementioned drawbacks or at least provide a useful alternative.

The main object of the embodiments of the present specification is to provide a method and an electronic device for calculating a dot product for binary data, ternary data, non-binary data, and non ternary data.

Another object of the embodiments of the present specification is to provide a fused bitwise data path for a binary neural network (BNN) model and a three-dimensional neural network (TNN) model.

Another object of the embodiments of the present specification is to calculate the dot product for struck out data.

Another object of the embodiments of the present specification is to provide a fused bitwise data path to support dot product calculation for binary data and ternary data.

Another object of the embodiments of the present specification is to provide a full precision layer (FPL) data path in order to calculate a dot product between non-binary data or non-struck data and binary data or struck data.

Another object of the embodiments of the present specification is to calculate the dot product for binary data and ternary data, and distribute the dot product of one of the non-binary and non ternary data and one of the binary and ternary data to the fused data path and the FPL data path. It is to do.

Another object of the embodiments of the present specification is to configure an electronic device to process multiple data types by achieving more power efficiency and area efficiency.

Another object of the embodiments of the present specification is to provide an electronic device for supporting full precision of a first layer as well as a bit-wise dot product operation of a hidden layer for binary data or ternary data.

Another object of the embodiments of the present specification is to process struck out data in a bit-wise manner.

Another object of the embodiments of the present specification is to combine individual data paths required for a BNN operating on two different data types (eg, {0,1}, {-1,+1}) into a single fused data path. .

According to an aspect, a method of calculating a dot product for binary data, ternary data, non-binary data, and non ternary data may include: designing, by an electronic device, to calculate a dot product for the ternary data; Designing, by the electronic device, a fused bit-wise data path for supporting dot product calculation of the binary data and the ternary data; Designing, by the electronic device, a full precision layer (FPL) data path for calculating a dot product between one of the non-binary data and the non-three data, and one of the binary data and the ternary data ; And by the electronic device, calculating a dot product of the binary data and the ternary data, one of the non-binary data and the non ternary data, and a dot product between the binary data and the ternary data, the fused data path, and And distributing to the FPL data path.

According to another aspect, an electronic device for calculating a dot product for binary data, ternary data, non-binary data, and non ternary data includes: a static random access memory (SRAM); At least one controller configured to transmit an address to the SRAM; Processing engine (PE) array; And a dispatcher configured to receive at least one SRAM data and deliver the at least one SRAM data to the PE array, wherein the PE array comprises a PE array controller, an output feature map (OFM) combiner, and a fused data path engine. Including, the fused data path engine is configured to provide a fused data path for a binary neural network (BNN) model and a three-dimensional neural network (TNN) model.

1 shows an example scenario for determining the dot product for the BNN_A data type.
2A shows an example scenario for determining a dot product in a BNN_B data type based on a representation of an arithmetic calculation.
2B shows an example scenario for determining a dot product for a BNN_B data type based on a bitwise implementation.
3A is a flowchart illustrating a method of performing a MAC operation on binary data belonging to the BNN_A data type.
3B is a flowchart illustrating a method of performing a MAC operation on binary data belonging to the BNN_B data type.
4 shows an exemplary scenario in which ternary data including a sign is modified into a 2-bit representation.
5A illustrates a scenario for determining a dot product for a TNN data type based on an expression of arithmetic calculation according to an embodiment.
5B is a diagram illustrating a scenario of determining a dot product for a TNN data type based on a bit-wise implementation according to an embodiment.
6 is a flowchart illustrating a method of performing a MAC operation on ternary data belonging to a TNN data type according to an embodiment.
7A is a schematic diagram of an electronic device for calculating a dot product for binary data, ternary data, non-binary data, and non ternary data, according to an embodiment.
7B is a schematic diagram of a PE array of an electronic device according to an exemplary embodiment.
8 is a block diagram of an electronic device for calculating a dot product for binary data, struck out data, non-binary data, and non-struck out data, according to an exemplary embodiment.
9A is a block diagram of a fused data path engine for performing MAC operations, according to an embodiment.
9B is a diagram illustrating a TNN model used by a fusion data path engine to perform a MAC operation on triplet data in a triplet mode, according to an embodiment.
9C illustrates a BNN model used by a fused data path engine to perform a MAC operation on binary data in a binary mode, according to an embodiment.
10 is a schematic diagram of a traversal in a loop format according to an embodiment.
11A is a schematic diagram of a data path of a PE according to an embodiment.
11B is a flowchart illustrating a method of calculating a dot product for binary data, struck out data, non-binary data, and non-struck out data using an electronic device, according to an exemplary embodiment.
12A is a schematic diagram illustrating steps performed by a dispatcher for data transfer using a full precision layer data path, according to an embodiment.
12B shows a data distribution method for a fused bit-wise data path.

Embodiments of the present specification and various features and details thereof are described more fully with reference to non-limiting embodiments illustrated in the accompanying drawings and described in detail in the following description. Descriptions of well-known components and processing techniques are omitted so as not to unnecessarily obscure the embodiments of the present specification. Further, since some embodiments may be combined with one or more other embodiments to form a new embodiment, the various embodiments described herein are not necessarily mutually exclusive. As used herein, the term “or” is used non-exclusively unless otherwise specified. The examples used herein are merely intended to facilitate understanding of the manner in which the embodiments of the present specification may be practiced, and to enable those skilled in the art to practice the embodiments of the present specification. Accordingly, the examples should not be construed as limiting the scope of the embodiments herein.

Embodiments may be described and illustrated in terms of a described function or a block performing a function. These blocks, which may be referred to herein as managers, units, modules, hardware components, etc., include logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard wired circuits, etc. It may be physically implemented by analog and/or digital circuitry, or may be optionally driven by firmware. For example, the circuit may be implemented on one or more semiconductor chips or a substrate support, such as a printed circuit board. The circuit constituting the block may be implemented by a combination of dedicated hardware or a processor (eg, one or more programmed microprocessors and related circuits) or a dedicated hardware that performs some functions of the block and a processor that performs other functions of the block. . Each block of the embodiment may be physically separated into two or more interactions and individual blocks without departing from the scope of the present disclosure. Likewise, the blocks of the embodiment may be physically combined into more complex blocks without departing from the scope of the present disclosure.

The dot product is a calculation for multiply and accumulate (MAC) operations in neural networks. A lot of expensive hardware multiplication logic is required to perform MAC operations on higher precision data. However, in order to perform MAC operations on extremely low-precision data such as binary data and ternary data, inexpensive hardware logic that performs bit operations is sufficient. The dot product of a vector pair is calculated by summing all the multiplication results calculated after element-wise multiplication. For example, it is calculated as sum(Ai x Bi), where A and B represent a pair of data vectors of length L, and Ai and Bi represent the i-th element of vector A and B, respectively. In an example, the elements of A and B may be real or complex. Binary data can be of two types: BNN_A ({0,1}) and BNN_B ({-1, +1}). Both binary data types can be represented using 1 bit. In the case of BNN_B, 0 may represent -1 and 1 may represent +1. This methodology is described in the context of a neural network for clarity in later descriptions. In the present disclosure, an input feature map (IFM) vector and two input vectors, which are weights or kernel (W) vectors, are used for the dot product operation. “Data”, “data vector”, “IFM vector”, “bit vector”, and “IFM data” are used interchangeably in this disclosure and have the same meaning.

1 shows an example scenario for determining the dot product for the BNN_A data type.

According to the definition of the dot product, the final result is obtained by multiplying each element between the W vector and the IFM vector. The multiplication hardware has a large area and high power overhead. The "W AND IFM" vector represents the multiplication of each element between the W vector and the IFM vector. Therefore, instead of using multiplication hardware, element-by-element AND (bitwise AND) operations for W vectors and IFM vectors are used. The final result is obtained by counting the number of 1s in the "W AND IFM" vector. In one example, the dot product result between the W vector and the IFM vector is 3 as follows.

Dot product between W and IFM

= #1 in W AND IFM bit vector

= popcount(#1 in bit vector)

= 3

2A shows an example scenario for determining a dot product in a BNN_B data type based on a representation of an arithmetic calculation.

W vector and IFM vector are denoted by the BNN_B data type. The "W X IFM" vector represents a multiplication for each element between a W vector and an IFM vector. The final result of the dot product is obtained by subtracting B from A. Where A is the number of "1" in the "W X IFM" vector and B is the number of "-1" in the "W X IFM" vector. In one example, the final result is -2. In "W X IFM", the number of 1 is indicated by #1, and in "W X IFM" the number of -1 is indicated by #-1.

Dot product between W and IFM

= #1 + (#-1) * -1

= 3-5

= -2

2B shows an example scenario for determining a dot product for a BNN_B data type based on a bitwise implementation.

Fig. 2b shows the same final result (i.e. -2) as in the case of using multipliers. The W vector and the IFM vector are denoted by 1 bit. The bit-wise dot product operation is performed as in steps a) to c).

a) Performs a bit-wise XNOR operation (W XNOR IFM).

b) Count the number of 1s in the "W XNOR IFM" vector. Counting 1 in a bit vector is called POPCOUNT, and this value is called v.

c) If the length of the bit vector of W and IFM is L, the dot product result is determined as (2 X v)-L, and can be expressed as (v << 1)-L. Here, “<<” represents a bitwise left shift.

In one example, step c) represents the result as -2, since v is 3 and L is 8. Therefore, the dot product calculated in the bitwise manner is the same as the operation by the definition of "dot product". Steps a) to c) above may be illustrated in the form of a flowchart of FIG. 3B. In Fig. 3B, the output result of step B304 is added in step B305. This addition operation corresponds to the accumulation in the MAC operation.

Dot product between W and IFM

= #1 + (#0) * -1

= #1-#0

= #1-(BIT_VECTOR_LENGTH-#1)

= 2 * (#1)-BIT_VECTOR_LENGTH

= (popcount << 1)-BIT_VECTOR_LENGTH

= (3 << 1)-8

= -2

3A is a flowchart illustrating a method of performing a MAC operation on binary data belonging to the BNN_A data type.

In step A301, a bitwise AND (AND) operation is performed between the W vector and the IFM vector. In step A302, a popcount is detected in response to performing a bitwise AND operation on each bit pair and determining a dot product value between an IFM vector (Bit vector IFM) and a W vector (Bit vector W). In step A303, popcount is added for the accumulation operation, where the MAC operation is completed after step A303.

3B is a flowchart illustrating a method of performing a MAC operation on binary data belonging to the BNN_B data type.

In step B301, a bitwise XNOR (XNOR) operation is performed between a W vector and a bit vector IFM. In step B302, a popcount is detected in response to performing a bitwise XNOR operation on each bit pair. In step B303, a left shift is performed on popcount. In step B304, the length of the IFM vector or the W vector (BIT_VECTOR_LENGTH) is subtracted from the left-shifted popcount, and a dot product value between the IFM vector and the W vector is determined. In step B305, a pop count is added for an accumulation operation, where the MAC operation is completed after step B305.

4 shows an exemplary scenario in which ternary data including a sign is modified into a 2-bit representation.

In an example, a vector of ternary data is a of FIG. 4. Each ternary data is expressed as 1 bit for a mask, 1 bit for a value, and 2 bits using Table 2. The mask bit indicates whether the ternary data is 0. The value bit represents ternary data less than 0 as 0, otherwise it is represented as 1.

Data Mask (m) Value (v) 2 bit representation
msb: m lsb: v 0 0 0 00 -One One 0 10 +1 One One 11

In this way, the ternary bit vector can be represented as shown in FIG. 4B. 4C and 4D show mask vectors and value vectors extracted from the ternary bit vector (b), respectively. "01" (i.e., mask bit 0 and value bit 1) is not allowed in this expression method. Preferred embodiments are shown in Figs. 5A to 12B. Hereinafter, a description will be given with reference to FIGS. 5A to 12B together.

5A illustrates a scenario for determining a dot product for a TNN data type based on an expression of arithmetic calculation according to an embodiment.

The operand of the TNN (Ternary Neural Network) data type is represented by {-1, 0, 1}. Consider the W vector (W) and the IFM vector (IFM) expressed using the TNN data type as shown in FIG. 5A. The dot product of the W vector (W) and the IFM vector (IFM) is determined by performing element-by-element multiplication between the W vector and the IFM vector, and then performing the sum of each bit of the multiplication result by element. The "W X IFM" vector represents an element-by-element multiplication between W and IFM. The final result of the dot product for struck out data is to subtract B from A. Where B is the number of "-1" in the "W X IFM" vector and A is the number of "1" in the "W X IFM" vector. In one example, the final result is -2. In "W X IFM", the number of 1 is indicated by #1, and in "W X IFM" the number of -1 is indicated by #-1.

Dot product between W and IFM

= #1 + (#-1) * -1

= 2-4

= -2

5B is a diagram illustrating a scenario of determining a dot product for a TNN data type based on a bit-wise implementation according to an embodiment. In addition, FIG. 5B shows an example of a dot product operation for ternary data.

According to Table 2, the W vector (W) and the IFM vector (IFM) are modified to a 2-bit representation. The mask vector (m1) and the value vector (v1) for W are shown in a of Fig. 5B. The mask vector (m2) and the value vector (v2) for the IFM are indicated in b of FIG. 5B. In addition, a bitwise AND operation between the m1 vector and the m2 vector is performed to form an m3 vector. In addition, a v3 vector is formed by performing a bit-wise XNOR operation between the v1 vector and the v2 vector. Also, the bitwise AND operation uses m3 vectors and v3 vectors to determine the dot product. popcount is the number of 1s in the vector. This shows that the final result obtained by the bit-wise implementation is the same as the multiplication method described in FIG. 5A.

Dot product between W and IFM

= (2*m3 AND v3 popcount)-m3 popcount

= (m3 AND v3 popcount << 1)-m3 popcount

= 4-6

= -2

6 is a flowchart illustrating a method of performing a MAC operation on ternary data belonging to a TNN data type according to an embodiment.

In step 601, a bitwise AND (AND) operation between m1 and m2 is performed. In step 602, the output from step 601 is used to determine which ternary data is non-zero after an element-by-element operation. In step 603, a bitwise XNOR (XNOR) operation between v1 and v2 is performed. The output of step 603 is the result of an element-by-element operation between value bit vectors. If steps a) to c) are performed with the BNN_B data type, the inner result may be wrong. In the ternary representation, the value bit is 0 because it indicates 0 or less than 0. Consider a scenario where two strikeout data, d_a = 00 and d_b = 00 are used for the dot product. In this case, 1 is obtained from the bit-wise XNOR of the value bits of d_a and d_b. Considering this value, the final dot product is 1, not 0, resulting in an incorrect result. Therefore, an AND operation must be additionally performed using a mask so that only non-zero values can be considered. Accordingly, in step 604, a bitwise AND (AND) operation is performed between the outputs from steps 601 and 603. The output of step 604 holds only non-zero values after the bitwise operation between v1 and v2. In step 605, the number of 1s in the generated vector is counted as the output of step 604. Left shifting the output of step 605 is equivalent to multiplying by 2. If the length of the bit vector of the ternary data is subtracted from the output of step 606, the result is incorrect. This is because the output of step 604 may contain zero data. Thus, in order to maintain the dot product result generated only by non-zero data in step 607, the number of non-zero values (output of step 602) is subtracted from the output of step 606 instead of the length of the vector. The output of step 607 determines the dot product of the two ternary vectors. To perform the cumulative operation, the dot product of the two ternary vectors is added in step 608.

Examine the accuracy of examples of dot product operations on ternary data. There are two "1s" in the final vector after m3 AND v3. The number of "1" in m3 is 6. Therefore, the result of (2 << 1)-6 is -2, which is the same as the dot product result performed using element-by-element calculation as shown in FIG. 5A.

7A is a schematic diagram of an electronic device for calculating a dot product for binary data, ternary data, non-binary data, and non ternary data, according to an embodiment.

Examples of the electronic device 1000 include a smart phone, a tablet computer, a personal computer, a desktop computer, a personal digital assistant (PDA), a multimedia device, an Internet of Things (IoT), or a device such as, but is not limited thereto. The electronic device 1000 is a hardware accelerator that performs inference in an extremely low level quantized neural network, that is, a BNN and a TNN. In one embodiment, the electronic device 1000 transmits a command to the host central processing unit (CPU) 2000 to receive a command for performing a layer operation such as convolution, pooling, and fully connected. Connected. In an embodiment, the electronic device 1000 is connected to a dynamic random-access memory (DRAM) 3000 to receive and store a data vector.

In one embodiment, the electronic device 1000 includes a controller 100, a dispatcher 200, a static random access memory (SRAM) 300 (eg, on-chip SRAM), a processing engine (PE) array ( 400). The controller 100 controls the PE array 400 and the dispatcher 200. The dispatcher 200 is coupled with the controller 100 and the on-chip SRAM 300. The controller 100 is configured to transmit an address to the SRAM 300. Dispatcher 200 is configured to receive SRAM data and deliver SRAM data to PE array 400. In one embodiment, the dispatcher 200 is one of two data paths of each PE of the PE array 400 (ie, a full precision layer (FPL) data path and a fused bitwise (BW) data path). It is configured to pass data into one. In one embodiment, the FPL data path performs a dot product between a full precision (non-ternary and non-binary) IFM data vector and one of a binary or ternary kernel data vector. In one embodiment, the output of one of the FPL data path and the BW data path is selected based on the type of one of a full precision layer and a hidden layer for accumulation. In one embodiment, the fused data path performs a dot product between a pair of vectors (i.e., IFM and kernel) for ternary data and two pairs of vectors (i.e., IFM and kernel) for binary data.

In one embodiment, the PE array 400 includes a PE array controller 100, an output feature map (OFM) combiner 402, and a plurality of fused data path engines 403. The fused data path engine 403 is configured to provide a fused data path for the BNN model and the TNN model in full precision mode or bitwise mode. In one embodiment, the dot product is calculated for MAC operations in the BNN model and the TNN model. In one embodiment, the fused data path engine 403 is configured to support at least one of binary data used by the BNN model and ternary data used by the TNN model. In one embodiment, the fused data path engine 403 is configured to combine the FPL data path and the fused bitwise data path into a single fused data path. In one embodiment, the fused data path engine 403 is configured to process struck out data in a bit-wise manner. In one embodiment, the fused data path engine 403 is configured to combine the data paths for the BNN model and the TNN model into a single fused data path. In one embodiment, the BNN model is used to calculate the dot product for a pair of binary data and the TNN model is used to calculate the dot product for the ternary data.

In another embodiment, the PE array 400 includes a plurality of PEs (i.e., PE 0 to PE N-1), a PE array controller 401, and an OFM coupler 402, wherein each PE is a fused It includes a data path engine 403, an adder (indicated by "+"), an accumulator (indicated by ACC), and a comparator (indicated by ">=0"). The fused data path engine 403 provides a fused data path by combining individual data paths for the BNN model and the TNN model. In one embodiment, the PE array 400 performs a MAC operation on data provided by the dispatcher 200. Each PE performs a MAC operation, a key part of neural network inference. This MAC operation consists of two sub-operations: multiply or dot product calculation and accumulate. PE supports dot product in various data types, namely 1-bit binary data used in BNN and 2-bit ternary data used in TNN. In BNN and TNN, the input data of the first layer tends to be kept with higher precision (eg 8 bits), and the weights (kernel) with low precision (eg 1 bit or 2 bits). Therefore, to reflect this trend, PE supports dot products for operands with high precision and other operands with low precision (ie, binary or ternary).

In one embodiment, the PE array 400 is designed as a two-dimensional array of single instruction multiple data (SIMD) based PEs to share input data and increase throughput of the electronic device 1000. The PE array 400 can perform multiple dot product calculations in parallel. In the context of a neural network, two inputs are used, the IFM and the kernel to generate the output (e.g. OFM). IFM is shared across multiple columns of PE array 400 while kernels are shared across multiple rows of PE array 400. All input data is stored in the staging registers of the PE array 400. The dispatcher 200 transfers input data from staging registers to each PE of the PE array 400 according to a mode (ie, data path selection, fused bit unit or FPL) selection without additional data storage overhead.

In an embodiment, the host CPU 2000 provides a command required to the controller 100, where the controller 100 establishes communication between the host CPU 2000 and the electronic device 1000. In one embodiment, the controller 100 processes a loop traversal to generate an OFM tensor. During the loop traversal, the controller 100 generates an SRAM request using two controllers (i.e., an IFM controller and a kernel controller) and transmits the address of the data vector to the SRAM 300. In addition, the controller 100 transmits a command to the dispatcher 200 to transfer data vectors from the SRAM 300 to a plurality of PEs. The dispatcher 200 transfers the data vector received from the SRAM 300 to the PE array 400 based on a data path selection (ie, FPL or fused bit unit) in order to perform a MAC operation by each PE. In addition, the PE's fused data path engine 403 receives a data vector from the dispatcher 200. The controller 100 provides a command to the PE array controller 401 to transmit mode signals of a plurality of PEs. The PE array controller 401 transmits a mode signal to the plurality of PEs to select an operation mode (ie, mode selection) in response to receiving a command from the controller 100. In one embodiment, the operation mode is one of a binary operation mode or a ternary operation mode. The PE's fused data path engine 403 performs a MAC operation on a data vector based on an operation mode for dot product calculation.

In one embodiment, the fused data path engine 403 receives one of the ternary data and the binary data to calculate the dot product for the binary data or ternary data. In addition, the fused data path engine 403 determines the operation mode for the ternary data and the binary data. Further, the fused data path engine 403 processes at least one of ternary data and binary data in the XNOR gate and the AND gate using at least one popcount logic based on the determined operation mode. Further, the fused data path engine 403 receives at least one of the processed ternary data and the processed binary data as an accumulator. The final result of the MAC operation is stored in the accumulator. The value of the accumulator is further compared with the threshold value and converts the output value to the final ternary value or the final binary value. The PE provides data output in the form of 00, 10, or 11 to the OFM combiner 402. In addition, the OFM combiner 402 writes data to the on-chip SRAM 300. The PE array controller 401 transmits the done_pe signal to the controller 100 in response to completing the MAC operation required by the layer of the BNN or TNN. In addition, the controller 100 transmits a completion signal to the host CPU 2000 when the operation of all layers is completed.

Although FIG. 7A illustrates hardware components of the electronic device 1000, it should be understood that other embodiments are not limited thereto. In another embodiment, the electronic device 1000 may include fewer or more components. Also, the labels or names of components are used for illustrative purposes only and do not limit the scope of the present invention. One or more components may be combined to perform the same or substantially similar function to calculate the dot product.

7B is a schematic diagram of a PE array of an electronic device according to an exemplary embodiment.

A PE array 400 having dimensions M X N is shown in FIG. 7B. Here, M PEs exist in each column of the PE array 400 and N PEs exist in each row of the PE array 400. In one example, PE(0, 5) represents PEs in the 0th row and 5th column of the PE array 400. In another example, PE(3, 2) represents the PE in the 3rd row and 2nd column of the PE array 400. On the left edge of the PE array 400, M registers are used to store different IFM data. On the upper edge of the PE array 400, N registers are used to store different kernel data. To maximize data sharing for PEs, IFM vectors are shared among PEs in a row of PE array 400, and kernel vectors are shared among PEs in a row of PE array 400. Each PE produces an OFM pixel. In one embodiment, the PE array 400 performs a plurality of MAC operations in parallel on data provided by the dispatcher 200.

8 is a block diagram of an electronic device for calculating a dot product for binary data, struck out data, non-binary data, and non-struck out data, according to an exemplary embodiment.

In an embodiment, the electronic device 1000 includes a controller 100, a dispatcher 200, an IFM memory space 301, a kernel memory space 302, an OFM memory space 303, a PE array 400, and a pooling controller. (Pooling Controller, PC) 500, and a Transform Engine (TE) 600. In one embodiment, the on-chip SRAM 300 includes an IFM memory space 301, a kernel memory space 302, and an OFM memory space 303. The PE array 400 includes a plurality of PEs and a dispatcher 200, where each PE includes an FPL data path and a fused bitwise data path. The host CPU 2000 is connected to the controller 100 of the electronic device 1000. Further, the DRAM 3000 is connected to the IFM memory space 301, the kernel memory space 302, and the OFM memory space 303 of the electronic device 1000.

The host CPU 2000 provides the controller 100 with instructions necessary to perform a layer operation (eg, convolution, pooling, fully connected, etc.). In addition, the controller 100 establishes communication between different components. During loop traversal, the controller 100 generates an SRAM request using two controllers (i.e., an IFM controller and a kernel controller). In addition, the controller 100 transfers data received from the kernel memory of the kernel memory space 302 and the IFM memory of the IFM memory space 301 to the dispatcher 200 using a buffer (Buff). Also, the dispatcher 200 distributes IFM data and kernel data to a plurality of PEs. In one embodiment, the dispatcher 200 distributes data to a plurality of staging registers at the left and upper edges of the PE array 400 to store IFM and kernel data, respectively. The dispatcher 200 distributes IFM data shared among PEs in different columns of a specific row. The dispatcher 200 distributes kernel data shared between PEs in different rows in a specific column. Each PE performs a MAC operation on data received from the dispatcher 200. In addition, the PE array 400 transmits an output (ie, an ACC vector) to the TE 600 to perform tasks such as batch normalization and comparison with a threshold value. Further, the TE 600 writes the output back to the OFM memory 1 of the OFM memory space 303. The PC 500 reads data (ie, OFM vector) from OFM memory 1 and performs a pooling operation. In addition, the PC 500 records the final data (ie, OFM vector after pooling) in the OFM memory 2 of the OFM memory space 303. In addition, data of the OFM memory 1 and OFM memory 2 are written back to the DRAM 3000. In addition, the controller 100 transmits a completion signal to the host CPU 2000 when operations of all layers are completed.

9A is a block diagram of a fused data path engine for performing MAC operations, according to an embodiment.

In one embodiment, the fused data path engine includes three AND gates (AND1-AND3), two exclusive NOR (XNOR) gates (XNOR1-XNOR2), six multiplexers (MUX1-MUX6), and two subtractors (SUB1- SUB2), two pop counters (POPCOUNT1-POPCOUNT2), two adders (ADD1-ADD2), two accumulators (ACC-ACC_D), and a comparator (COMPARATOR).

The first XNOR gate (XNOR1) is an output of the first XNOR gate (XNOR1), in order to generate a product vector of length N bits, 1 bit of the first vector (BIT VECTOR 1) and the second vector (BIT VECTOR 2). Receive 1 bit. The first AND gate AND1 receives 1 bit of the first vector and 1 bit of the second vector in order to generate a product vector of length N bits as an output of the first AND gate AND1. The second XNOR gate (XNOR2) receives 1 bit of the third vector (BIT VECTOR 3) and 1 bit of the fourth vector (BIT VECTOR 4), and is output of the second XNOR gate (XNOR2) with a length of N bits. Create a product vector. The second AND gate AND2 receives 1 bit of the third vector and 1 bit of the fourth vector, and generates a product vector having an N-bit length as an output of the second AND gate AND2.

The output of the first XNOR gate XNOR1 and the output of the first AND gate AND1 are provided as inputs of the first multiplexer MUX1. The output of the first AND gate AND1 and the output of the second XNOR gate XNOR2 are provided as inputs of the third AND gate AND3. The second multiplexer MUX2 receives an input from the output of the second XNOR gate XNOR2, the output of the second AND gate AND2, and the output of the third AND gate AND3. The output of the second multiplexer MUX2 includes only non-zero elements of the dot product operation on two ternary vectors, or all elements of the dot product operation on two binary vectors (BIT VECTOR3 and BIT VECTOR4). The output of the first multiplexer MUX1 is supplied to the input of the first pop counter POPCOUNT1, where the first pop counter 1 is 1 at the output of the first XNOR gate XNOR1 or the first AND gate AND1. The number of is calculated, which is transferred to the third multiplexer MUX3 and the fourth multiplexer MUX4. The third multiplexer MUX3 receives an input from the first multiplexer MUX1 through a first bit length BIT LENGTH 1 and a pop counter POPCOUNT1. The second pop counter POPCOUNT2 calculates the number of 1s at the output of the second multiplexer MUX 2, which is transferred to the fourth multiplexer MUX4. Here, the output of the second pop counter POPCOUNT2 is shifted by 1 bit to the left.

The fourth multiplexer MUX4 receives an input from the second multiplexer through the second bit length BIT LENGTH 2 and the second pop counter POPCOUNT2. In addition, in the first subtractor SUB1, the output shifted to the left of the first pop counter POPCOUNT1 and the output of the third multiplexer MUX3 are subtracted. In addition, the output shifted to the left of the second pop counter (POPCOUNT2) and the output of the fourth multiplexer (MUX4) are subtracted by the second subtractor (SUB2), where the output of the second subtractor (SUB2) is 2 in the case of strikeout data. It represents the dot product between two ternary vectors, and in the case of binary data, it represents the dot product between two binary vectors (BIT VECTOR3 and BIT VECTOR4). The output of the first subtractor SUB1 represents the dot product of BIT VECTOR1 and BIT VECTOR2 in the case of binary data, and the output of the first subtractor SUB1 is not used in the case of ternary data. The output of the second subtractor SUB2 and the output of the first subtractor SUB1 are added by the first adder ADD1. Here, in the case of binary data, the output of the first adder ADD1 is a combination of two pairs of vectors. Represents the dot product result (i.e. pair 1 is {BIT VECTOR1, BIT VECTOR2} and pair 2 is {BIT VECTOR2, BIT VECTOR3}). The fifth multiplexer MUX5 selects the output of the second subtractor SUB2 as an output in the case of ternary data, and selects the output of the first adder ADD1 as the output in the case of binary data. In order to perform the accumulation operation, the output of the fifth multiplexer MUX5 from the second adder ADD2 is added together with the first accumulator ACC. The output of the second adder is stored in the first accumulator (ACC). In addition, an output value is generated by comparing the output of the second adder with a threshold value of a comparator. In an embodiment, the second accumulator ACC_D stores the output of the second adder ADD2 and compares the output value of the second adder ADD2 with a threshold value of the comparator COMPARATOR. In one embodiment, the sixth multiplexer MUX6 receives inputs from the second adder and the second accumulator ACC_D, and the comparator COMPARATOR receives the input from the second accumulator ACC_D.

The hardware resources required for MAC operation on ternary data can also be used for binary data. Since binary data is represented by 1 bit (instead of 2 bits of ternary data), mask and value pair vectors are used as two independent vector pairs in the TNN to perform dot product operations on binary data. In the case of binary data, processing of two independent vectors produces one output each, so the throughput of the BNN is twice that of the TNN. In addition, the PE supports two data types BNN_A({0,1}) and BNN_B({-1, +1}) for MAC operation. With appropriate control signals, the fused data path of the PE can support MAC operations on binary and ternary data.

An additional data path, the FPL data path, is included in the PE to support dot product operations on either input with high precision. In the case of the FPL data path, one of the input data vectors consists of high-precision data and the other input vector consists of extremely low (ie, binary or ternary) precision data. Therefore, there are two data paths in the PE, namely, a bit unit data path and an FPL data path, and have separate outputs for the dot product. Depending on the data path selection, one of the outputs (internal results) of the two data paths is added to the accumulator to complete the MAC operation.

The fused bitwise data path is one of the two data paths inside the PE. The hardware components used to calculate the dot product of ternary data can also be used for binary data. For example, there are POPCOUNT logic (both BNN_A and BNN_B), bitwise XNOR logic (for BNN_B), and bitwise AND logic (for BNN_A). The outputs of the two POPCOUNT logic shown in FIG. 6 (steps 602 and 605) can be used individually, and the two independent pairs of binary vectors can be processed for dot product operation. For example, in the case of ternary data in FIG. 6, the dot product is calculated using m1, m2, v1, and v2 vectors. In the proposed method, for binary data, the dot product of a pair of {m1, m2} and {v1, v2} vectors is computed in parallel. These two inner results can be added, and the addition result can be added additionally in the accumulator. In this way, the throughput of dot product calculations on binary data is doubled compared to ternary data.

The microarchitecture of the fused bitwise data path is shown in Fig. 9A. 9A shows a data path having a vector length of 16. However, the proposed method can be extended to any vector length. Input data in the data path is given as follows, where detailed control signals are shown in Fig. 11A.

a) BIT VETCOR1

b) BIT VECTOR2

c) BIT VECTOR3

d) BIT VECTOR4

e) BIT LENGTH1

f) BIT LENGTH2

As shown in Figure 9a. The four inputs a)-d) are 16-bit input vectors, and e)-f) are the bit vector lengths for the input pairs {a, b} and {c, d} respectively (i.e., a 16-bit vector should be considered. There are two additional data representing the amount). Components MUX, ADD, SUB, AND, XNOR, and POPCOUNT of FIG. 9A and subsequent drawings (ie, FIGS. 9B and 9C) are a multiplexer, an adder, a subtractor, a bitwise AND, a bitwise XNOR, and a POPCOUNT logic, respectively. The number added as a suffix to these logic components represents the instance number (eg AND3 represents the third instance of the bitwise AND logic). ACC and ACC_D are two accumulators. ACC is updated every week and ACC_D is updated when all MAC operations on the output data (i.e. OFM) are complete. 9B and 9C show the active components of the fused data path (highlighted in dark gray) when operating on ternary and binary data, respectively.

9B is a diagram illustrating a TNN model used by a fusion data path engine to perform a MAC operation on triplet data in a triplet mode, according to an embodiment.

The first AND gate AND1 receives mask vectors BIT VECTOR1 and BIT VECTOR2 of two ternary vectors each having a length of N. In addition, the second XNOR gate XNOR2 receives value vectors BIT VECTOR3 and BIT VECTOR4 of two ternary vectors each having a length of N. Also, the output of the first AND gate AND1 is provided as an input of the first multiplexer MUX1. In addition, the output of the first AND gate AND1 and the output of the second XNOR gate XNOR2 are provided as inputs of the third AND gate AND3. Also, the second multiplexer MUX2 receives an input from the output of the second XNOR gate XNOR2 and the output of the third AND gate AND3. The output of the second multiplexer MUX2 selects the output of the third AND gate AND3. Here, the output of the second multiplexer MUX2 includes a non-zero element in the dot product. In addition, the fourth multiplexer MUX4 inputs the second multiplexer MUX2 through the second bit length BIT LENGTH 2 and the second pop counter POPCOUNT2, and the first multiplexer through the first pop counter POPCOUNT1. MUX1) input is received.

The second pop counter POPCOUNT2 calculates the number of 1s from the output of the second multiplexer MUX2 delivered to the fourth multiplexer MUX4. The output of the second pop counter POPCOUNT2 is shifted to the left by 1 bit. The first pop counter POPCOUNT1 calculates a non-zero element in the dot product operation transmitted to the fourth multiplexer MUX4. Here, the fourth multiplexer MUX4 selects the output of the first pop counter POPCOUNT1 as the output of the fourth multiplexer MUX4. The output shifted to the left of the second pop counter POPCOUNT2 and the output of the fourth multiplexer MUX4 are subtracted by the second subtractor SUB2. The output of the second subtractor SUB2 represents the dot product between two ternary vectors. The output of the second subtractor SUB2 is provided to perform an accumulation operation together with the first accumulator ACC using the fifth multiplexer MUX5 and the second adder. The output of the second adder is stored in the first accumulator (ACC). The output of the second adder ADD2 is additionally stored in the second accumulator ACC_D and compared with a threshold value of the comparator COMPARATOR to generate an output value.

In one embodiment, the PE performs dot products on a pair of data vectors in parallel to improve throughput. PE performs a dot product on a vector pair of ternary data (in the case of TNN). Each ternary data has two bits: a mask bit (indicating whether the data is 0 or not) and a value bit (whether the data represents -1 or +1, represented by 0 and 1, respectively). In this way, three possible values of the ternary data (0, -1 and +1) are encoded in 2 bits (00, 10, and 11, respectively). Also, the 2-bit ternary data vector is divided into two separate 1-bit vectors for mask and value, respectively. Therefore, in order to perform dot product on the ternary data, the PE receives 4 1-bit vectors as inputs. The pair of input mask bit vectors and input value bit vectors are processed separately to determine the dot product of the two ternary bit vectors. Since the input data to the PE is transferred in the form of a bit vector, an area- and power-efficient bit-wise operation logic (eg, bit-wise OR, bit-wise AND, bit-wise XNOR, etc.) is used instead of a multiplier.

The mask vectors of the ternary data are transferred to BIT VECTOR1 and BIT VECTOR2. The value vectors of the ternary data are passed to BIT VECTOR3 and BIT VECTOR4. In the case of the bit lengths BIT LENGTH1 and BIT LENGTH2, an arbitrary value may be transferred. In the case of ternary data, the number of non-zero elements is internally calculated. In AND1, a bitwise AND operation is performed between BIT VECTOR1 and BIT VECTOR2. In XNOR1, a bit-wise XNOR operation is performed between BIT VECTOR3 and BIT VECTOR4. The outputs of AND1 and XNOR1 are bitwise ANDed using AND3. The output of AND3 has non-zero strikeout data and is delivered through MUX2. POPCOUNT2 counts the number of 1s in the output bit vector of MUX2. The output of POPCOUNT2 is shifted to the left by 1 bit. The output of this left shift logic is one of the inputs of the SUB2 subtractor. Another input to SUB2 is generated at the MUX4 output. MUX4 passes the POPCOUNT1 output (counting the number of 1s at the output of MUX2, i.e. counting the number of 1s at the output of AND1). The output of MUX4 represents the number of non-zero elements in the resulting vector after the dot product between the two ternary vectors. Therefore, the output of SUB2 produces the final result of the dot product calculation for the ternary data. The output of SUB2 is additionally provided by MUX5. MUX5 receives inputs from SUB2 (which generates the result of the dot product operation on struck out data) and ADD1 (which generates the result of the dot product operation on binary data). Depending on the mode (i.e. binary or ternary), one of the outputs (i.e., SUB2 or ADD1) is delivered through MUX5. Therefore, in the case of three-step mode, the output of SUB2 is transmitted. The output of MUX5 is the result of the multiplication step of the MAC operation. The accumulation step is performed by adding the output of MUX5 together with ACC in ADD2. When all MAC tasks are completed, the output of ACC is saved in ACC_D. ACC_D stores the final value of the OFM that is further compared with the threshold value and generates 2-bit struck out data (ie 00, 10 or 11).

9C illustrates a BNN model used by a fused data path engine to perform a MAC operation on binary data in a binary mode, according to an embodiment.

The first XNOR gate (XNOR1) receives 1 bit of the first vector (BIT VECTOR1) and 1 bit of the second vector (BIT VECTOR2) and generates a product vector of length N bits as the output of the first XNOR gate (XNOR1). do. In addition, the second XNOR gate XNOR2 receives 1 bit of the third vector BIT VECTOR3 and 1 bit of the fourth vector BIT VECTOR4, and outputs the product vector of length N bits as the output of the second XNOR gate XNOR2. Create In addition, the output of the first XNOR gate XNOR1 is supplied as an input of the first multiplexer MUX1. In addition, the output of the second XNOR gate XNOR2 is supplied to the input of the second multiplexer MUX2.

In addition, the third multiplexer (MUX3) receives the input from the first multiplexer through the first bit length (BIT LENGTH1) and the first pop counter (POPCOUNT1), and the output of the first multiplexer is a first pop counter (POPCOUNT1). Is supplied as an input of. The first pop counter (POPCOUNT1) calculates the number of 1s in the result vector after performing a dot product between BIT VECTOR1 and BIT VECTOR2 on binary data. The calculated number of 1s is transferred to the third multiplexer MUX3 and the fourth multiplexer MUX4. The fourth multiplexer MUX4 receives a second bit length BIT LENGTH2 and an input from the first multiplexer through a second pop counter POPCOUNT2. The second pop counter (POPCOUNT2) performs a dot product between BIT VECTOR3 and BIT VECTOR4 on the transferred binary data, and then calculates the number of 1s in the result vector (that is, the output of the second multiplexer MUX2). The calculated number of 1s is transferred to the fourth multiplexer MUX4. The output of the second pop counter POPCOUNT2 is shifted to the left by one. The output shifted to the left of the first pop counter POPCOUNT1 and the output of the third multiplexer MUX3 are subtracted by the first subtractor SUB1. Here, the output of the first subtractor SUB1 represents the dot product between BIT VECTOR1 and BIT VECTOR2 for binary data.

In addition, the output shifted to the left of the second pop counter POPCOUNT2 and the output of the fourth multiplexer MUX4 are subtracted by the second subtractor SUB2. Here, the output of the second subtractor SUB2 represents the dot product between BIT VECTOR3 and BIT VECTOR4 in binary data. The output of the second subtractor SUB2 and the output of the first subtractor SUB1 are added by the first adder ADD1, and accumulated using the fifth multiplexer (MUX5) and the second adder together with the first accumulator (ACC). Perform the operation. The output of the second adder is stored in the first accumulator (ACC). The output of the second adder is compared with a threshold value of a comparator to generate an output value.

In an embodiment, when the binary data type is {0,1} (ie, BNN_A), the output of the first pop counter (POPCOUNT1) is transmitted through the left shifter and the third multiplexer (MUX3). The output of the second pop counter POPCOUNT2 is transmitted through the left shifter and the fourth multiplexer MUX4. In addition, the output shifted to the left by the first pop counter POPCOUNT1 and the output of the third multiplexer are subtracted by the first subtractor SUB1. In addition, the output shifted to the left by the second pop counter POPCOUNT2 and the output of the fourth multiplexer are subtracted by the second subtractor SUB2. In this way, the outputs of the first subtractor SUB1 and the second subtractor SUB2 provide POPCOUNT1 and POPCOUNT2 values, which are dot product results for two binary vectors of the BNN_A data type.

In one embodiment, when the binary data type is {-1,1}, the output of the first pop counter (POPCOUNT1) passes through the left shifter, the first bit length (BIT LENGTH1) passes through the third multiplexer, and 2 The output of the pop counter (POPCOUNT2) passes through the left shifter and the second bit length (BIT LENGTH2) passes through the fourth multiplexer. In addition, the first subtractor SUB1 calculates a dot product between the first bit vector and the second bit vector by subtracting the output of the third multiplexer from the output shifted to the left of the first pop counter POPCOUNT1. In addition, the second subtractor SUB2 calculates the dot product between the third bit vector and the fourth bit vector by subtracting the output of the fourth multiplexer from the output shifted to the left of the second pop counter POPCOUNT2. As such, the outputs of the first subtractor (SUB1) and the second subtractor (SUB2) are ((POPCOUNT1 << 1)-BIT LENGTH1) values and ((POPCOUNT2 << 1), which are the dot product results of two binary vectors of the BNN_B data type. -Provides BIT LENGTH2) value.

In an exemplary scenario, the electronic device 1000 includes a fused data path that is more efficient in area with respect to reference hardware having 2X2 integer multiplier hardware to perform binary dot product and triple dot product. In an exemplary scenario, the power efficiency of the electronic device 1000 with respect to the existing accelerator increases as the size of the PE array 400 of the electronic device 1000 increases.

Two independent vector pairs {BIT VECTOR1, BIT VECTOR2} and {BIT VECTOR3, BIT VECTOR4} are used to calculate the dot product for binary data. Dot product for the BNN_B data type: The active components (highlighted in dark gray) are used to calculate the dot product of the BNN_B data type. In a fused data path, two vector pairs are processed independently in two separate data flows. One data flow (left of BIT LENGTH1) is the same as the sequence of XNOR1, MUX1, POPCOUNT1, left shift logic, and SUB1. The other data flow (right of BIT LENGTH2) is the same as the sequence of XNOR2, MUX2, POPCOUNT2, left shift logic, and SUB2. The outputs of these two data flows are added at ADD1. For the BNN_B data type, the bit length is required to determine the effective bit length. Thus, for both sequences, BIT LENGTH1 and BIT LENGTH2 are passed through MUX3 and MUX4, respectively.

Dot product for the BNN_A data type: For the BNN_A data type, a bitwise AND operation is required instead of a bitwise XNOR operation (see Fig. 1). This slightly changes the two data flows mentioned for BNN_B. The left data flows of the BNN_A data type are AND1, MUX1, POPCOUNT1, left shift logic, and SUB1. The right data flows of the BNN_A data type are AND2, MUX2, POPCOUNT2, left shift logic, and SUB2. For the BNN_A data type, the bit length and left shift logic are not required because the output of the POPCOUNT logic itself produces a dot product result. To eliminate the effect of the left shift logic on the output of POPCOUNT, the POPCOUNT value is shifted left and subtracted by the previous value. For example, in the case of the left flow, the output of POPCOUNT1 is subtracted from the output of the left shift logic. Therefore, for the left and right data flows, the outputs of POPCOUNT1 and POPCOUNT2 are passed through MUX3 and MUX4. Similar to the BNN_B data type, the outputs of SUB1 and SUB2 are added in ADD1.

The output of ADD1 represents the result of the dot product for binary data (either BNN_A or BNN_B). In binary mode, the output of ADD1 is delivered through MUX5. The output of MUX5 is the result of the multiplication step of the MAC operation. The accumulation step is performed by adding the output of MUX5 together with ACC in ADD2. When all MAC operations are completed, the output of ACC is stored in ACC_D. ACC_D stores the final value of the OFM, which is further compared to the threshold to generate 1-bit binary data (ie, 0 or 1).

10 is a schematic diagram of a traversal in a loop format according to an embodiment.

OFM generation by the PE array is shown in FIG. 10. The controller 100 performs a loop traversal to generate a sequence of SRAM addresses to fetch IFM and kernel data. Loop traversal is shown in Fig. Explained by 10. The steps to perform traversal in the loop form are as follows.

Step 1: for ofm_ch in range (0, C, OCH)

Step 2: for ofm_h in range(0, 1, OH)

Step 3: for ofm_w in range (0, R, OW)

Step 4: if BNN, ch_step = vector_length X 2

Step 5: else ch_step = vector_length

Step 6: for ifm_ch in range (0, ch_step, ICH)

Step 7: for k_h in range (0, 1, KH)

Step 8: for k_w in range (0, 1, KW)

Step 9: OFM[ofm_ch: (ofm_ch +C), ofm_h, ofm_w: (ofm_w+R)] += matrix_mul (IFM[ifm_ch: (ifm_ch +ch_step), ifm_h + k_h, (ifm_w +k_w): (ifm_w + k_w) +R], kernel[ofm_ ch: (ofm_ch +C), ifm_ch: (ifm_ch +ch_step), k_h, k_w])

In step 1, the OFM channel update is performed (indicated by 3 in the OFM tensor in FIG. 10). In step 2, an OFM height update is performed (indicated by 2 in the OFM tensor of FIG. 10). In step 3, an OFM width update is performed (indicated by 1 in the OFM tensor of FIG. 10). In step 4, channel step update is performed for processing two streams of binary data (ie, in the case of BNN). In step 5, channel step update is performed for one stream processing for the three-way data (ie, in the case of TNN). In step 6, an IFM channel update is performed (indicated by 3 in the IFM tensor of FIG. 10). In step 7, a kernel height update is performed (indicated by 2 in the IFM tensor in Fig. 10). In step 8, the kernel width update is performed (indicated by 1 in the IFM tensor of FIG. 10). In step 9, an OFM tile of size 1 X R X C is created.

OFM three-dimensional tensors are created in a tile format with a maximum dimension of 1 X R X C, where R and C are the rows and columns of the PE array. The IFM tensor of size KW X KH X ICH is loaded from IFM by performing steps 6-8 to generate OFM tiles. Here, KW, KH, and ICH represent a kernel width, a kernel height, and an IFM channel, respectively. The channel level used for BNN is twice the number used for TNN, and each ternary data is represented by 2 bits. The remaining OFM tensors are generated for the OFM channels of size C in the main order by performing steps 2-3, and then the OCH size of the OFM channels is completed.

11A is a schematic diagram of a data path of a PE according to an embodiment.

In order to support the FPL and the bit unit layer, respectively, an FPL data path and a fused bit unit data path (BW data path) can be used in each PE. The FPL data path performs a dot product between the full-precision IFM data and one of the binary kernel data and the ternary kernel data. The output of either the FPL data path and the fused data path (BW data path) is selected for accumulation based on the layer type, where the layer type is Full precision layer or Hidden layer. layer). The fused data path (BW data path) performs a dot product between kernel data and IFM data. Here, kernel data and IFM data are expressed as binary data or ternary data.

In FPL, IFM vectors are used in full precision (i.e., not binary or not ternary), whereas kernel vectors are used in binary or ternary format. Full precision IFM vectors are typically used in the input layer with a smaller number of channels (i.e. 3 for image processing). In the bitwise layer, both IFM and kernel vector are used in binary or ternary format. Inputs to PE include Mode, Layer type, Datatype, IFM stream, Kernel stream, Vector len. 1, and Vector len. 2 is included. Mode (Op. mode) is a binary operation mode or a ternary operation mode. The layer type is an FPL or bit-based layer. The data type is the BNN_A data type or the BNN_B data type. The IFM stream provides a vector of IFM data to the dispatcher 200. The kernel stream provides a vector of kernel data to the dispatcher 200. Vector length 1 (Vector len. 1) is the vector length of the IFM stream. Vector length 2 (Vector len. 2) is the vector length of the kernel stream. The dispatcher 200 provides necessary data for both data paths. The mechanism of operation of the dispatcher 200 is further described in FIGS. 12A-12B. In the case of a fused bitwise data path, the electronic device compresses full-precision IFM data using the already used IFM and kernel staging registers. Therefore, no additional storage is required to store full-precision IFM data.

For the FPL data path, the dot product operation is expressed using the following equation.

Dot product (A,W) = sum (Ai X Wi)

Here, i=1 to n, Ai R and Wi {-1, 0, +1} (for ternary data) or [{-1, +1} (for BNN_B) or {0,1} (for BNN_A) Occation)].

Multiplication can be replaced by updating and summing the Ai values, the three possible values of Ai are 0, -Ai, and + Ai. Accordingly, as shown in FIG. 11A, three inputs (ie, an IFM vector, 0, and an IFM vector of opposite signs) are provided to the multiplexers MUX8, MUX9, and MUX L-1 of the FPL data path. Control of the multiplexers (MUX8, MUX9, MUX L-1) is based on the value of the kernel vector (ie, Wi). Further, the outputs of the multiplexers MUX8, MUX9, and MUX L-1 are summed using an adder tree (ADDER TREE) of the FPL data path. The output of the ADDER TREE is the dot product. In order to perform the accumulation operation, the FPL data path (that is, the output of the adder tree) and the output of the fused data path are selected using a multiplexer MUX6 selected according to the layer type. In addition, an accumulator value is added to the output of the multiplexer MUX6 to generate a new accumulator value.

PE's FPL data path supports typical dot product operations (one of the operands is represented with higher precision). A vector of high-precision IFM data and a vector of lower-precision (binary or ternary) kernel data are received as inputs in the FPL data path. In the case of high-precision IFM data, when element-by-element multiplication is performed with the precision of binary (bi) or ternary (ti) between IFM (Ai) and the kernel, the output of the multiplication is the same (Ai), opposite sign (-Ai). , Or can be 0. According to this principle, each multiplexer (MUX8, MUX9, MUX L-1) passes one of the IFM values. The output of each multiplexer is added using an adder tree (ADDER TREE). The output of the ADDER TREE represents the result of the dot product for high-precision IFM data.

Figure 11a shows two data paths inside the PE. The output of each data path is selected using a MUX6 multiplexer. In Figs. 9A, 9B, and 9C, the logic units under MUX5 (ie, ADD2, ACC, MUX6, ACC_D) are indicated for the integrity of the fused bitwise data path of the PE. In the final design of the PE, these units are shared across both data paths. Thus, these logical units are placed outside of the two data paths (ie, below MUX6 in Fig. 11A).

11B is a flowchart illustrating a method of calculating a dot product for binary data, struck out data, non-binary data, and non-struck out data using an electronic device, according to an exemplary embodiment.

In step 1101, the method includes designing to calculate a dot product for the ternary data. At step 1102, the method includes designing a fused bitwise data path to support dot product computation for binary data and ternary data. In step 1103, the method includes designing an FPL data path to calculate a dot product between one of the non-binary data and the non-ternary data and one of the binary data and the ternary data. In step 1104, the method comprises calculating a dot product for the binary data and ternary data and distributing the dot product between one of the non-binary and non ternary data and one of the binary and ternary data to the fused data path and the FPL data path. Includes.

In an embodiment, the electronic device 1000 receives one of the strikeout data and the binary data, determines an operation mode for the strikeout data and the binary data, and determines at least one pop count logic (POPCOUNT1-POPCOUNT2) based on the determined operation mode. ) To process at least one of the ternary data and the binary data in the XNOR gate (XNOR1- XNOR2) and the AND gate (AND1- AND3), and receive at least one of the processed ternary data and the processed binary data to the accumulator, The fused data path is designed to support dot product calculations on binary data and ternary data by generating at least one of a final ternary value and a final binary value.

In one embodiment, when the fused data path engine 403 of the electronic device 1000 is configured to support binary data used in the BNN model and ternary data used in the TNN model, the electronic device 1000 is operated At least one of the ternary data and binary data is processed in the XNOR gates XNOR1-XNOR2 and the AND gates AND1-AND3 using at least one pop count logic (POPCOUNT1-POPCOUNT2) based on the mode. The electronic device generates a product vector having a length of N bits as an output of the first XNOR gate XNOR1 by receiving 1 bit of the first vector and 1 bit of the second vector using the first XNOR gate XNOR1.

The electronic device 1000 receives 1 bit of the first vector and 1 bit of the second vector using the first AND gate AND1, and converts a product vector having a length of N bits as an output of the first AND gate AND1. Generate. The electronic device 1000 receives 1 bit of the third vector and 1 bit of the fourth vector using the second XNOR gate XNOR2, and converts a product vector having a length of N bits as an output of the second XNOR gate XNOR2. Generate.

The electronic device 1000 generates a product vector of length N bits as an output of the second AND gate AND2 by receiving 1 bit of the third vector and 1 bit of the fourth vector using the second AND gate AND2. do. The electronic device 1000 supplies the output of the first XNOR gate XNOR1 and the output of the first AND gate AND1 as inputs of the first multiplexer MUX1, where the output of the first multiplexer MUX1 is three-dimensional data. In the case of, the mask vector of the dot product between two ternary vectors or the dot product vector of the first binary vector pair (BIT VECTOR 1 and BIT VECTOR 2) in case of binary data is included.

The electronic device 1000 supplies the output of the first AND gate AND1 and the output of the second XNOR gate XNOR2 as inputs of the third AND gate AND3. The electronic device 1000 receives an input from the output of the second XNOR gate XNOR2, the output of the second AND gate AND2, and the output of the third AND gate AND3 using the second multiplexer MUX2. Here, in the case of ternary data, the output of the second multiplexer MUX2 includes a dot product value vector of two ternary vector pairs that are affected only by a non-zero element pair in the value vector of the input ternary vector pair. In the case of binary data, the output of the second multiplexer MUX2 includes the dot product vector of the second binary vector pair (BIT VECTOR 3 and BIT VECTOR 4).

The electronic device 1000 receives a first bit length and an input from the first multiplexer MUX1 through a first pop counter (POPCOUNT1), and the output of the first multiplexer (MUX1) is a first pop counter (POPCOUNT1). Provided, the first pop counter (POPCOUNT1) calculates the number of 1s of the mask vector in case of ternary data and the number of 1s of the dot product vector in case of binary data. The calculation result is transferred to the third multiplexer MUX3 in case of binary data, and is transferred to the fourth multiplexer MUX4 in case of ternary data.

The electronic device 1000 provides a second bit length, an output of the first multiplexer MUX1, and a second pop counter (POPCOUNT2) to the fourth multiplexer (MUX4), wherein the second pop counter (POPCOUNT2) is a fourth The number of 1 is calculated from the output of the second multiplexer (MUX2) delivered to the multiplexer (MUX4), the output of the second pop counter (POPCOUNT2) is shifted to the left by 1, and the output of the fourth multiplexer (MUX4) is struck. In the case of data, the output of the first pop counter (POPCOUNT1) or the second bit length in the case of binary data type B, or the output of the second pop counter (POPCOUNT2) in case of binary data type A are included.

The left-shifted output of the first pop counter (POPCOUNT1) and the output of the third multiplexer (MUX3) are subtracted by the first subtractor (SUB1), where the output of the first subtractor (SUB1) is two binary vector pairs (BIT VECTOR). 1 and BIT VECTOR2). The left-shifted output of the second pop counter (POPCOUNT2) and the output of the fourth multiplexer (MUX4) are subtracted by the second subtractor (SUB2), where the output of the second subtractor (SUB2) is two ternary vectors in the case of ternary data. In the case of the dot product between pairs or binary data, it represents the dot product between the second binary vector pair (BIT VECTOR 3 and BIT VECTOR4).

In the case of binary data, the output of the second subtractor SUB2 and the output of the first subtractor SUB1 are added to the first adder ADD1, where the output of the fifth multiplexer MUX5 is the first adder ( In the case of the output of ADD1) and the triplet data, the output of the second subtractor SUB2 is selected, and the output of the fifth multiplexer MUX5 is added together with the first accumulator ACC using the second adder ADD2. . The output of the second adder ADD2 is stored in the first accumulator ACC, where the output of the second adder ADD2 is compared with a threshold value of the comparator COMPARATOR to generate an output value.

In an embodiment, when the fused data path engine 403 of the electronic device 1000 is configured to support the struck out data used in the TNN model 900B, the electronic device 1000 At least one of ternary data and binary data is processed in the XNOR gates XNOR1-XNOR2 and AND gates AND1-AND3 using pop count logic POPCOUNT1-POPCOUNT2. The electronic device 1000 receives 1 bit of the first vector and 1 bit of the second vector using the first AND gate AND1, and receives a product vector having a length of N bits as an output of the first AND gate AND1. Generate.

The electronic device 1000 generates a product vector of length N bits as an output of the second XNOR gate (XNOR2) by receiving 1 bit of the third vector and 1 bit of the fourth vector using the second XNOR gate (XNOR2). do. The electronic device 1000 provides an output of the first AND gate AND1 as an input of the first multiplexer MUX1. The electronic device 1000 provides the output of the first AND gate AND1 and the output of the second XNOR gate XNOR2 as inputs of the third AND gate AND3. The electronic device 1000 receives an input from the output of the second XNOR gate XNOR2, and receives the output of the third AND gate AND3 using the second multiplexer MUX2, where the second multiplexer MUX2 The output of contains the elements of a bit vector obtained from a bitwise operation between two ternary vectors.

The electronic device 1000 includes a second pit length using a fourth multiplexer MUX4, an input from the second multiplexer MUX2 through a second pop counter (POPCOUNT2), and a first pop counter from the first multiplexer MUX1. Receives an input through (POPCOUNT1), wherein the second pop counter (POPCOUNT2) calculates the number of 1s of the second multiplexer (MUX2) provided to the fourth multiplexer (MUX4), and calculates the number of 1s of the second pop counter (POPCOUNT2). The output is shifted to the left by 1, and the first pop counter (POPCOUNT1) calculates the number of 1s in the bit vector obtained after a bitwise AND operation between the mask vectors of two ternary data transmitted to the fourth multiplexer (MUX4). , Where the first bit vector and the second bit vector are mask vectors, where the number of 1 represents the number of non-zero values obtained after a dot product operation between two ternary vectors.

The influence of zero elements from the output of the second subtractor SUB2 is removed by subtracting the output shifted to the left of the second pop counter POPCOUNT2 and the output of the fourth multiplexer MUX4 in the second subtractor SUB2. Here, the output of the second subtractor SUB2 is a result of a dot product operation performed on two ternary vectors. The output of the second subtractor SUB2 is provided together with the first accumulator ACC to perform an accumulation operation using the fifth multiplexer MUX5 and the second adder ADD2. The output of the second adder ADD2 is stored in the first accumulator ACC, where the output of the second adder ADD2 is compared with a threshold value of the comparator COMPARATOR to generate an output value.

In an embodiment, when the fusion data path engine 403 of the electronic device 1000 is configured to support binary data used in the BNN model 900C, the electronic device 1000 At least one of ternary data and binary data is processed in the XNOR gates XNOR1-XNOR2 and AND gates AND1-AND3 using count logic POPCOUNT1-POPCOUNT2. The electronic device 1000 generates a product vector of length N bits as an output of the first XNOR gate (XNOR1) by receiving 1 bit of the first vector and 1 bit of the second vector using the first XNOR gate (XNOR1). do. The electronic device 1000 receives 1 bit of the third vector and 1 bit of the fourth vector using the second XNOR gate XNOR2, and converts a product vector having a length of N bits as an output of the second XNOR gate XNOR2. Generate.

The electronic device 1000 provides an output of the first XNOR gate XNOR1 as an input of the first multiplexer MUX1. The electronic device 1000 provides the output of the second XNOR gate XNOR2 as an input of the second multiplexer MUX2, where the output of the second multiplexer MUX2 is a bit-wise XNOR operation between the third and fourth bits. It contains the bit vector obtained after. The electronic device 1000 provides the output of the first multiplexer MUX1 to the third multiplexer MUX3 through a first bit length and a first pop counter POPCOUNT1, wherein the output of the first multiplexer MUX1 is zero. 1 is provided as an input of a pop counter (POPCOUNT1), and the first pop counter (POPCOUNT1) calculates the number of 1s obtained from the bit vector at the output of the first multiplexer (MUX1), and the calculation result is the third multiplexer (MUX3). And a fourth multiplexer MUX4.

The electronic device 1000 receives a second bit length, an input from the first multiplexer MUX1 through the first pop counter POPCOUNT1, and an input from the second multiplexer MUX2 through the second pop counter POPCOUNT2. Receive, where the second pop counter (POPCOUNT2) counts the number of 1s at the output of the second multiplexer (MUX2) delivered to the fourth multiplexer (MUX4), and the output of the second pop counter (POPCOUNT2) is left by 1 Shifted to. The output shifted to the left of the first pop counter POPCOUNT1 and the output of the third multiplexer MUX3 are subtracted by the first subtractor SUB1. Here, the output of the first subtractor SUB1 represents the dot product of the first bit vector and the second bit vector. The output shifted to the left of the second pop counter POPCOUNT2 and the output of the fourth multiplexer MUX4 are subtracted by the second subtractor SUB2. Here, the output of the second subtractor SUB2 represents the dot product of the third bit vector and the fourth bit vector. The output of the second subtractor SUB2 and the output of the first subtractor SUB1 are added by the first adder ADD1, and the fifth multiplexer (MUX5) and the second adder (ADD2) are used together with the first accumulator (ACC). To perform the cumulative operation. The output of the second adder ADD2 is stored in the first accumulator ACC, where the output of the second adder ADD2 is compared with a threshold value of the comparator COMPARATOR to generate an output value.

In one embodiment, a two-dimensional data path is designed to form a PE by combining a bit-wise data path and an FPL data path, and using the dispatcher 200 to support both the bit-wise data path and the FPL data path without additional storage overhead. By distributing the necessary data to multiple PEs of the PE array, the electronic device 1000 calculates the dot product of binary data and ternary data, and combines the dot product between one of the non-binary data and the non stern data and one of the binary data and ternary data. Allocate to path and FPL data path. The PE computes the dot product between binary or ternary data pairs in a bitwise data path or between one full precision data and one binary or ternary data in the FPL data path.

In the flowchart 1100, various operations, blocks, steps, etc. may be performed in the order presented, in a different order, or simultaneously. Further, in some embodiments, some of operations, blocks, steps, etc. may be omitted, added, modified, skipped, and the like without departing from the scope of the present invention.

12A is a schematic diagram illustrating steps performed by a dispatcher for data transfer using a full precision layer data path, according to an embodiment.

The PE array 400 has two sets of registers for storing an IFM stream and a kernel stream, that is, a set 0 and a set 1. The number of registers for storing the IFM vector and the kernel vector of the set is equal to the row and column of the PE array 400, respectively. Consider the electronic device 1000 in which the PE array has a dimension of 8 X 8, a vector length of 16 bits, and a full-precision IFM bit width of 8. The PE array 400 includes 2X8 (ie, Set 0 and Set 1) IFM registers and Kernel registers. Each register is 16 bits in size. The IFM register and the kernel register are shown vertically and horizontally in Fig. 12A, respectively. IFM data is broadcast through rows of PE array 400 and kernel data is broadcast through columns of PE array 400.

An IFM vector broadcast in a row consists of two registers, and a kernel vector broadcast in a column consists of one register. For example, Row 0 receives IFM vectors from R0 and R1 in Set 0. Also, Row 1 receives IFM vectors from R2 and R3 of Set 0. Similarly, Row 7 receives IFM vectors from R6 and R7 of Set 1. Since the width of the IFM bits is 8, each IFM register contains 2 IFM data, and the length of the IFM vector is 4 (ie 2 in each register). To perform dot product, the length of the kernel vector must also be equal to the length of the IFM vector, i.e. 4. Thus, 4 or 8 bits of the binary or ternary format of kernel data are stored in each kernel register. In one example, any one of the sets is used for kernel registers. Column 0 receives kernel data from C0. Column 1 receives kernel data from C1. Similarly, column 7 receives kernel data from the C7 register. So, in one example, each PE can perform 4 dot product operations every cycle.

In order to improve performance and data reuse, multiple PEs are configured into a two-dimensional array. Since the PE array 400 follows a single instruction multiple data (SIMD) based architecture and adjacent PEs of the PE array 400 do not communicate with each other, a data flow based architecture (e.g., systolic array) ) Do not follow. To improve input data reuse, IFM data vectors are shared across multiple columns of PE array 400 while kernel vectors are shared across multiple rows of PE array 400. All input data is stored in staging registers. Staging registers are disposed on the edges (left and top) of the PE array 400. In the PE array 400 of the R X C dimension (where R and C are the rows and columns of the PE array 400), there are the number of R and C staging registers that store IFM and kernel data, respectively. In one embodiment, there are two sets of staging registers (Set 0 and Set 1). Dispatcher 200 distributes input data from staging registers to each PE of PE array 400 based on data path selection (ie, FPL or fused bit unit) without additional storage overhead. 12A and 12B show data distribution schemes for FPL and fused bitwise data paths, respectively. For example, the bit width of each staging register is considered to be 16 and the precision of IFM data is considered to be 8 bits. The dimension of the PE array is considered to be 8 X 8. However, the proposed method works at the bit width of all registers, IFM data precision, and PE array level.

12A shows a data distribution scheme for the FPL data path.

According to an example, two 8-bit IFM data are stored together in each 16-bit IFM staging registers. 4 IFM data is formed to generate IFM vectors and is shared across all columns of a specific row of PE array 400. For example, R0 and R1 of Set 0 generate IFM vectors for the 0th row of the PE array 400. Similarly, R2 and R3 of Set 0 are the IFM vectors for the first row of the PE array 400, and R6 and R7 of Set 1 are the IFM vectors for the 7th row of the PE array 400. Generate. In the case of the kernel, each staging register uses 4 (binary) or 8 (ternary) bits of the total 16 bits. Therefore, in one example, only one set of staging registers is used to supply kernel data. In one example, C0 of the Set 0 register provides kernel data to all rows of the 0th column of the PE array 400. Similarly, C1 of Set 0 is supplied to the first column, and C7 of Set 0 is supplied to the 7th column of the PE array 400. Therefore, in an example, each PE of the FPL data path performs four element-specific operations (ie, IFM and kernel data pairs) in parallel.

12B is a schematic diagram illustrating steps performed by a dispatcher for data transfer using a fused bitwise data path, according to an embodiment.

The PE array 400 has two sets of registers for storing an IFM stream and a kernel stream, that is, a set 0 and a set 1. The number of registers for storing the IFM vector and the kernel vector in the set is equal to the column and row of the PE array 400, respectively. Consider an electronic device 1000 in which the PE array 400 has a dimension of 8 X 8, a vector length of 16 bits, and a full precision IFM bit width of 8. The PE array 400 includes 2X8 (ie, Set 0 and Set1) IFM registers and Kernel registers. Each register is 16 bits in size. IFM registers and Kernel registers are shown vertically and horizontally in Fig. 12B, respectively. IFM data broadcast through a row of the PE array 400 and kernel data broadcast through a column of the PE array 400 are stored in two registers. For example, Row 0 receives the IFM vector in the R0 register of Set 0 and Set 1. Similarly, Row 7 receives the IFM vector from the R7 registers of Set 0 and Set 1. Column 0 obtains a kernel vector from C0 of Set 0 and Set 1. Similarly, Column 7 fetches the kernel vector from C7 of Set 0 and Set 1. Since each register is 16 bits in size, for each of binary data and ternary data, each PE can perform 32 (2 X 16) and 16 dot product operations in every cycle.

12B shows a data distribution method for a fused bit-wise data path.

Unlike the FPL data path, in the case of a fused bitwise data path, two sets of staging registers are used to supply data to the PE array 400. Since each PE needs to receive 4 input data (2 for IFM and 2 for kernel), 2 registers are used in the IFM and kernel to provide the required data. In one example, two registers are used to supply two independent IFM vectors (for binary) or one IFM vector (for ternary) to every column of a particular row of PE array 400. Similarly, two registers are used to provide two independent kernel vectors (for binary data) or one kernel vector (for ternary data) for every row of a particular column of PE array 400. For example, {RO, R1} of set 0 (Set0) provides IFM data to the 0th row of the PE array, and {R6, R7} of set 1 (Set1) is the 7th row of the PE array (400). Provides IFM data. In the case of kernel data, {C0, C0} of Set 0 and Set 1 are supplied to the 0th column, and {C1, C1} of Set 0 are supplied to the 1st column. In an example, each PE of the fused bitwise data path performs an operation for each element of 32 (for binary data) or 16 (for ternary data) in parallel.

The foregoing description of the present specification is for illustrative purposes only, and those of ordinary skill in the art to which the content of the present specification belongs will understand that it is possible to easily transform it into other specific forms without changing the technical spirit or essential features of the present invention. I will be able to. It is to be understood that the phraseology or terminology used herein is for description and not limitation. Accordingly, although the embodiments of the present specification have been described in terms of preferred embodiments, those of ordinary skill in the art will recognize that the embodiments of the present specification may be modified and implemented within the scope of the embodiments described herein.

Claims (22)

As a method of calculating the dot product for binary data, ternary data, non-binary data, and non ternary data,
Designing, by an electronic device, to calculate a dot product for the strikeout data;
Designing, by the electronic device, a fused bit-wise data path for supporting dot product calculation of the binary data and the ternary data;
Designing, by the electronic device, a full precision layer (FPL) data path for calculating a dot product between one of the non-binary data and the non-three data, and one of the binary data and the ternary data ; And
By the electronic device, the fused data path and the dot product between the binary data and the ternary data and one of the non-binary data and the struck out data, and the dot product between the binary data and the struck out data are calculated. Distributing to the FPL data path. The method of claim 1,
Designing the fused bitwise data path to support dot product calculation for the binary data and the ternary data:
Receiving, by the electronic device, one of the struck out data and the binary data;
Determining, by the electronic device, an operation mode for the struck out data and the binary data;
Processing, by the electronic device, at least one of the ternary data and the binary data in an XNOR gate and an AND gate using at least one pop count logic based on the determined operation mode;
Receiving, by the electronic device, at least one of the processed struck out data and the processed binary data as an accumulator; And
Generating, by the electronic device, at least one of a final ternary value and a final binary value. The method of claim 2,
When the fused data path engine of the electronic device is configured to support a binary data model used in a Binary Neural Network (BNN) model and a ternary data model used in a TNN (Ternary Neural Network) model, by the electronic device, Processing at least one of the ternary data and the binary data in the XNOR gate and the AND gate using the at least one pop count logic based on the determined operation mode,
Receiving 1 bit of a first vector and 1 bit of a second vector using a first XNOR gate, and generating a product vector of length N bits as an output of the first XNOR gate;
Receiving the 1 bit of the first vector and the 1 bit of the second vector using a first AND gate, and generating a product vector of length N bits as an output of the first AND gate;
Receiving 1 bit of a third vector and 1 bit of a fourth vector using a second XNOR gate, and generating a product vector of length N bits as an output of the second XNOR gate;
Receiving the 1 bit of the third vector and the 1 bit of the fourth vector using a second AND gate, and generating a product vector of length N bits as an output of the second AND gate;
Supplying an output of the first XNOR gate and an output of the first AND gate as an input of a first multiplexer, wherein the output of the first multiplexer is a dot product between two ternary vectors or of the binary data in the case of the ternary data. Case contains the mask vector of the dot product of the first binary vector pair;
Providing an output of the first AND gate and an output of the second XNOR gate as inputs of a third AND gate;
Using a second multiplexer, receiving an input from an output of the second XNOR gate, an output of the second AND gate, and an output of the third AND gate, wherein the output of the second multiplexer is Case contains the dot product value vector of two ternary vector pairs affected only by a non-zero element pair from the value vector of the input ternary vector pair, where the output of the second multiplexer is a second binary vector pair in the case of the binary data Contains the dot product vector of;
Receiving a first bit length and an input from the first multiplexer through a first pop counter, wherein the output of the first multiplexer is provided as an input of the first pop counter, wherein the first pop counter is the struck out In the case of data, the number of 1s in the mask vector is calculated, in the case of binary data, the number of 1s in the dot product vector is calculated. In the case of binary data, the calculated result is transferred to a third multiplexer, and in the case of the ternary data The calculated result is transferred to a fourth multiplexer; And
Receiving a second bit length provided to the fourth multiplexer, an output of the first multiplexer, and an output of a second pop counter, wherein the second pop counter is provided to the fourth multiplexer. 2 Calculate the number of 1s at the output of the multiplexer, where the output of the second pop counter is shifted to the left by 1, where the output of the fourth multiplexer is the output of the first pop counter in the case of the struck data, binary Including the output of the second bit length for data type B, or the second pop counter for ternary data type A;
The left-shifted output of the first pop counter and the output of the third multiplexer are subtracted by a first subtractor, where the output of the first subtractor represents the dot product of two bit-wise vector pairs,
The left-shifted output of the second pop counter and the output of the fourth multiplexer are subtracted by a second subtractor, where the output of the second subtractor represents the dot product between two ternary vector pairs in the case of the ternary data or the For binary data, it represents the dot product between two binary vector pairs,
In the case of the binary data, the output of the second subtraction and the output of the first subtractor are added by a first adder, where the output of the fifth multiplexer selects the output of the first adder in case of binary data or in the case of ternary data Select the output of the second subtractor, where the output of the fifth multiplexer is added together with the first accumulator using a second adder,
The output of the second adder is stored in the first accumulator, wherein the output of the second adder is compared to a threshold value of a comparator to produce an output value. The method of claim 2,
When the fused data path engine of the electronic device is configured to support the ternary data model used in the TNN (Ternary Neural Network) model, the at least one pop count logic based on the determined operation mode by the electronic device Processing at least one of the ternary data and the binary data in the XNOR gate and the AND gate by using,
Receiving 1 bit of a first vector and 1 bit of a second vector using a first AND gate, and generating a product vector of length N bits as an output of the first AND gate;
Receiving 1 bit of a third vector and 1 bit of a fourth vector using a second XNOR gate, and generating a product vector of length N bits as an output of the second XNOR gate;
Supplying an output of the first AND gate as an input of a first multiplexer;
Supplying an output of the first AND gate and an output of the second XNOR gate as inputs of a third AND gate;
Receiving an input from an output of the second XNOR gate and an output of the third AND gate using a second multiplexer, wherein the output of the second multiplexer is a bit vector obtained in a bit-wise operation between two ternary vectors. Contains elements; And
Receiving a second bit length using a fourth multiplexer, an input from the second multiplexer through a second pop counter, and an input from the first multiplexer through a first pop counter, wherein the first The 2 pop counter calculates the number of 1s of the output of the second multiplexer provided to the fourth multiplexer, where the output of the second pop counter is shifted to the left by 1, where the first pop counter is the first 4 Calculate the number of 1s of the stroked bit vectors after a bitwise AND operation between the mask vectors of two ternary data provided by the multiplexer, where the first bit vector and the second bit vector are mask vectors, where the number of 1s is Represents the number of non-zero values obtained after the dot product of the two ternary vectors;
The left-shifted output of the second pop counter and the output of the fourth multiplexer are subtracted by a second subtractor to remove the influence of the zero element from the output of the second subtractor, where the output of the second subtractor is the Represents the result of the dot product operation of two ternary vectors,
The output of the second subtractor is provided with a first accumulator to perform an accumulation operation using a fifth multiplexer and a second adder,
Wherein the output of the second adder is stored in the first accumulator, and the output of the second adder is compared to a threshold value of a comparator to generate an output value. The method of claim 2,
When the fused data path engine of the electronic device is configured to support binary data used in a binary neural network (BNN) model, the at least one pop count logic is performed by the electronic device based on the determined operation mode. Processing at least one of the ternary data and the binary data in the XNOR gate and the AND gate by using,
Receiving 1 bit of a first vector and 1 bit of a second vector using a first XNOR gate, and generating a product vector of length N bits as an output of the first XNOR gate;
Receiving 1 bit of a third vector and 1 bit of a fourth vector using a second XNOR gate, and generating a product vector of length N bits as an output of the second XNOR gate;
Providing an output of the first XNOR gate as an input of a first multiplexer;
Providing an output of the second XNOR gate as an input of a second multiplexer, wherein the output of the second multiplexer includes a bit vector obtained after a bitwise XNOR operation between the third vector and the fourth vector;
Receiving an input from the first multiplexer via a first bit length and a first pop counter using a third multiplexer, wherein the output of the first multiplexer is supplied as an input of the first pop counter and Wherein the first pop counter calculates the number of 1s of the bit vector from the outputs of the first multiplexer transmitted to the third and fourth multiplexers; And
Receiving a second bit length, an input from the first multiplexer through the first pop counter, and an input from the second multiplexer through a second pop counter, wherein the second pop counter is the Calculating the number of 1s from the output of the second multiplexer provided to a fourth multiplexer, wherein the output of the second pop counter is shifted to the left by 1;
The left-shifted output of the first pop counter and the output of the third multiplexer are subtracted by a first subtractor, where the output of the first subtractor represents the dot product between the first vector and the second vector,
The left-shifted output of the second pop counter and the output of the fourth multiplexer are subtracted by a second subtractor, where the output of the second subtractor represents the dot product of the third vector and the fourth vector,
The output of the second subtractor and the output of the first subtractor are added by a first adder, and the output of the first adder is provided together with a first accumulator to perform an accumulation operation using a fifth multiplexer and a second adder. ,
Wherein the output of the second adder is stored in the first accumulator, and the output of the second adder is compared to a threshold value of a comparator to generate an output value. The method of claim 3,
The dot product performs a calculation for a multiply and accumulate (MAC) operation in a BNN model and a TNN model. The method of claim 1,
The FPL data path performs a dot product between full precision input feature map (IFM) data and one of binary kernel data and ternary kernel data. The method of claim 1,
The output of one of the FPL data path and the fused data path is selected for accumulation based on a layer type, and the layer type is a full precision layer or a hidden layer. The method of claim 1,
The fused data path performs a dot product between kernel data and IFM data, and the kernel data and the IFM data are represented by the binary data or the ternary data. The method of claim 1,
By the electronic device, the fused data path and the dot product between the binary data and the ternary data and one of the non-binary data and the struck out data, and the dot product between the binary data and the struck out data are calculated. Distributing to the FPL data path,
Designing, by the electronic device, to form a processing element (PE) by combining the bit-wise data path and the FPL data path, wherein the PE is a dot product between binary or ternary data pairs in the bit-wise data path. Calculating or calculating the dot product between one full precision data and one binary or ternary data in the FPL data path; And
Distributing, by the electronic device, necessary data to multiple PEs of a two-dimensional PE array using a dispatcher to support both the bitwise data path and the FPL data path without additional storage overhead. As an electronic device for calculating the dot product for binary data, ternary data, non-binary data, and non ternary data,
Static random access memory (SRAM);
At least one controller configured to transmit an address to the SRAM;
Processing engine (PE) array; And
A dispatcher configured to receive at least one SRAM data and deliver the at least one SRAM data to the PE array,
The PE array includes a PE array controller, an output feature map (OFM) combiner, and a fused data path engine,
The electronic device, wherein the fused data path engine is configured to provide a fused data path for a binary neural network (BNN) model and a ternary neural network (TNN) model. The method of claim 11,
The BNN model is used to calculate a dot product for a pair of binary data, and the TNN model is used to calculate a dot product of ternary data. The method of claim 11,
The fused data path engine,
Three AND gates;
Two XNOR gates;
6 multiplexers;
Two subtractors;
Two pop counters;
Two adders;
Two accumulators; And
Includes a comparator,
In the two XNOR gates, a first XNOR gate receives 1 bit of a first vector and 1 bit of a second vector,
In the two XNOR gates, a second XNOR gate receives 1 bit of a third vector and 1 bit of a fourth vector,
In the three AND gates, a first AND gate receives the 1 bit of the first vector and the 1 bit of the second vector,
In the three AND gates, a second AND gate receives the 1 bit of the third vector and the 1 bit of the fourth vector,
In the three AND gates, a third AND gate receives an input from the first AND gate and the second XNOR gate,
In the six multiplexers, a first multiplexer receives an input from the first XNOR gate and the first AND gate,
In the six multiplexers, a second multiplexer receives an input from the second XNOR gate, the second AND gate, and the third AND gate,
In the six multiplexers, a third multiplexer receives a first bit length and an input from the first multiplexer through a first pop counter in the two pop counters,
In the six multiplexers, a fourth multiplexer has a second bit length, an input from the first multiplexer through the first pop counter, and from the second multiplexer through a second pop counter in the two pop counters. Receive input,
In the two subtractors, a first subtractor receives an output shifted to the left of the first pop counter and an output of the third multiplexer,
In the two subtractors, a second subtractor receives an output shifted to the left of the second pop counter and an output of the fourth multiplexer,
In the two adders, a first adder receives an input from the first subtractor and the second subtractor,
In the six multiplexers, a fifth multiplexer receives inputs from the first adder and the second subtractor,
A second adder in the two adders receives an input from a first accumulator in the fifth multiplexer and the two accumulators,
In the six multiplexers, a sixth multiplexer receives an input from the second adder and a second accumulator in the two accumulators,
The comparator receives an input from the second accumulator,
The fused data path engine
Supporting at least one of binary data used in the BNN model and ternary data used in the TNN model,
Combining the data paths for the BNN model and the TNN model into a single fused data path,
Processing the ternary data in a bit manner, and
The electronic device, configured to perform at least one of combining the data path for the BNN model and the TNN model operating on a data type into the single fused data path. The method of claim 13,
If the fused data path engine is configured to support the binary data used by the BNN model and the ternary data used by the TNN model,
The first XNOR gate receives the 1 bit of the first vector and the 1 bit of the second vector to generate a product vector of length N bits as an output of the first XNOR gate,
The first AND gate receives the 1 bit of the first vector and the 1 bit of the second vector to generate a product vector of length N bits as an output of the first AND gate,
The second XNOR gate receives the 1 bit of the third vector and the 1 bit of the fourth vector to generate a product vector of length N bits as an output of the second XNOR gate,
The second AND gate receives the 1 bit of the third vector and the 1 bit of the fourth vector to generate a product vector of length N bits as an output of the second AND gate,
The output of the first XNOR gate and the output of the first AND gate are provided as inputs of the first multiplexer, where the output of the first multiplexer includes the output of the first XNOR gate for binary data type B or binary In the case of data type A and ternary data, including the output of the first AND gate,
The output of the first AND gate and the output of the second XNOR gate are supplied to the input of the third AND gate,
The second multiplexer receives an input from an output of the second XNOR gate, an output of the second AND gate, and an output of the third AND gate, wherein the output of the second multiplexer is the third in the case of the ternary data. 3 includes the output of the AND gate or includes the output of the second XNOR gate in case of the binary data type B, or includes the output of the second AND gate in case of the binary data type A,
The third multiplexer receives a second bit length and an input from the first multiplexer through the first pop counter, and the output of the first multiplexer is provided to the first pop counter, where the first pop counter Calculates a non-zero element using a bit vector obtained by performing a bitwise operation by the first XNOR gate or performing a bitwise operation by the first AND gate, and the calculation result is the third multiplexer and Provided as the fourth multiplexer,
The fourth multiplexer receives a second bit length, an input from the first multiplexer through the first pop counter, and an input from the second multiplexer through the second pop counter, wherein the second pop counter Calculates a number of 1s from the output of the second multiplexer provided to the fourth multiplexer, where the output of the second pop counter is shifted left by one.
The left-shifted output of the first pop counter and the output of the third multiplexer are subtracted by the first subtractor,
The left-shifted output of the second pop counter and the output of the fourth multiplexer are subtracted by the second subtractor, where the output of the second subtracter represents the dot product between two ternary vector pairs in the case of the ternary data or the For binary data, it represents the dot product between two binary vector pairs,
The output of the second subtractor and the output of the first subtractor are added by the first adder, and an accumulation operation is performed using the fifth multiplexer and the second adder together with the first accumulator, wherein the first The output of the adder supports the binary data used in the BNN model, the output of the second subtracter is the input of the fifth multiplexer for supporting the ternary data used in the TNN model,
The electronic device, wherein the output of the second adder is stored in the first accumulator, and the output of the second adder is compared with a threshold value of the comparator to generate an output value. The method of claim 13,
If the fused data path engine is configured to support the ternary data used by the TNN model,
The first AND gate receives the 1 bit of the first vector and the 1 bit of the second vector to generate a product vector of length N bits as an output of the first AND gate,
The second XNOR gate receives the 1 bit of the third vector and the 1 bit of the fourth vector to generate a product vector of length N bits as an output of the second XNOR gate,
An output of the first AND gate is provided as an input of the first multiplexer,
The output of the first AND gate and the output of the second XNOR gate are provided as inputs of the third AND gate,
The second multiplexer receives an input from an output of the second XNOR gate and an output of the third AND gate, wherein the output of the second multiplexer is non-zero of a bit vector obtained in a bit-wise operation between two ternary vectors. Contains an element,
The fourth multiplexer receives a second bit length, an input from the second multiplexer through the second pop counter, and an input from the first multiplexer through the first pop counter, wherein the second pop counter Calculates the number of 1 from the output of the second multiplexer transmitted to the fourth multiplexer, the output of the second pop counter is shifted to the left by 1, and the first pop counter is transmitted to the fourth multiplexer. Compute a non-zero element of a bit vector obtained from a bitwise operation between two ternary vectors,
The left-shifted output of the second pop counter and the output of the fourth multiplexer are subtracted by the second subtractor to remove the influence of zero elements from the output of the second subtractor, where the output of the second subtractor is Is the result of the dot product performed on the two ternary vectors, the output of the fourth multiplexer is the output of the first pop counter,
The output of the second subtractor is provided to perform an accumulation operation using the fifth multiplexer and the second adder together with the first accumulator,
The electronic device, wherein the output of the second adder is stored in the first accumulator, and the output of the second adder is compared with a threshold value of the comparator to generate an output value. The method of claim 13,
If the fused data path engine is configured to support binary data used by the BNN model,
The first XNOR gate receives the 1 bit of the first vector and the 1 bit of the second vector to generate a product vector of length N bits as an output of the first XNOR gate,
The second XNOR gate receives the 1 bit of the third vector and the 1 bit of the fourth vector to generate a product vector of length N bits as an output of the second XNOR gate,
The output of the first XNOR gate is provided as an input of the first multiplexer, and the output of the first multiplexer includes a bit vector after a bit-wise XNOR operation on the first vector and the second vector,
The output of the second XNOR gate is provided as an input of the second multiplexer, and the output of the second multiplexer includes a bit vector obtained after a bit-wise XNOR operation on the third vector and the fourth vector,
The third multiplexer receives a first bit length and an input from the first multiplexer through the first pop counter, and the output of the first multiplexer is provided to the first pop counter, where the first pop counter Calculates the number of 1s in the bit vector obtained after bit-wise XNOR operation between the first vector and the second vector in a dot product operation provided by the third multiplexer and the fourth multiplexer,
The fourth multiplexer receives a second bit length, an input from the first multiplexer through the first pop counter, and an input from the first multiplexer through the second pop counter, wherein the second pop counter is Calculate the number of 1s of bit vectors obtained after bit-wise XNOR operation between the third vector and the fourth vector from the output of the second multiplexer provided to the fourth multiplexer, wherein the output of the second pop counter is Shifted left by 1,
The left-shifted output of the first pop counter and the output of the third multiplexer are subtracted by the first subtractor, where the output of the first subtractor represents the dot product between the first vector and the second vector, and the The output of the third multiplexer is the first bit length,
The left-shifted output of the second pop counter and the output of the fourth multiplexer are subtracted by the second subtractor, where the output of the second subtractor represents the dot product between the third vector and the fourth vector, and the The output of the fourth multiplexer is the second bit length,
The output of the second subtractor and the output of the first subtractor are added by the first adder, and the output of the first adder is performed by using the fifth multiplexer and the second adder together with the accumulator to perform an accumulation operation. Is provided,
The electronic device, wherein the output of the second adder is stored in the first accumulator, and the output of the second adder is compared with a threshold value of the comparator to generate an output value. The method of claim 14,
The dot product calculates a multiply and accumulate (MAC) operation on the binary data and the ternary data. The method of claim 11,
The dispatcher is configured to deliver data to the FPL (Full Precision Layer) data path and the fused data path, and the FPL data path and the fused data path are combined together to form a PE in the PE array. Device. The method of claim 18,
The electronic device, wherein the FPL data path performs a dot product between a full precision IFM layer and one of a binary kernel stream and a ternary kernel stream. The method of claim 18,
An output of one of the FPL data path and the fused data path is selected for accumulation based on a layer type, wherein the layer type is a full precision layer or a hidden layer. The method of claim 11,
The electronic device, wherein the PE array performs a plurality of MAC operations in parallel on data provided by the dispatcher. The method of claim 18,
The fused data path performs a dot product between kernel data and IFM data, wherein the kernel data and the IFM data are represented by the binary data or the ternary data.

Images