KR20190030564A - Neural network accelerator including bidirectional processing element array - Google Patents

Abstract

According to an embodiment of the present invention, a neural network accelerator for performing an operation of a neural network including layers may comprise: a kernel memory storing kernel data related to a filter; a feature map memory storing feature map data, which is output of the layers; and a PE array including processing elements (PEs) arranged along first and second directions. Each of the PEs may perform an operation using the feature map data transmitted in the first direction from the feature map memory and kernel data transmitted in the second direction from the kernel memory, and may transmit an operation result to the feature map memory in a third direction opposite to the first direction.

Description

TECHNICAL FIELD [0001] The present invention relates to a neural network accelerator including a bidirectional processing element array,

The present invention relates to a neural network accelerator, and more particularly to a neural network accelerator including a bidirectional processing element array.

An Artificial Neural Network (ANN) can process data or information in a manner similar to a biological neural network. Convolutional Neural Network, which is one of Deep Neural Network techniques, is being studied as a technology for image recognition. In particular, CNN can provide effective performance for recognizing various objects such as objects, characters, handwriting, images, and the like.

CNNs composed of a plurality of layers can be implemented in a hardware form using a semiconductor device. When a CNN is implemented in a hardware form, the amount of hardware resources required to implement CNN may increase due to the amount of processing to be performed in each of the plurality of layers constituting CNN. In addition, the frequency of memory accesses required to perform operations of CNN may also increase. Therefore, there is a need for a technology that can reduce constraints in implementing CNN in hardware form.

SUMMARY OF THE INVENTION The present invention is directed to a neural network accelerator including a bi-directional processing element array.

A neural network accelerator for performing an operation of a neural network including layers according to an embodiment of the present invention includes a kernel memory for storing kernel data related to a filter, a feature map memory for storing feature map data, Direction and a second direction, wherein each of the PEs includes feature map data transmitted in a first direction from the feature map memory and feature map data transmitted in a second direction from the kernel memory The operation may be performed using kernel data and the operation result may be transferred to the feature map memory in a third direction opposite to the first direction.

A neural network accelerator for performing a computation of a neural network including layers according to another embodiment of the present invention includes a kernel memory for storing kernel data related to a filter, a feature map memory for storing feature map data, And a PE array comprising processing elements disposed along one direction and a second direction and activation units disposed between the PEs and the feature map memory in a second direction, each of the PEs comprising: Performing a first operation using feature map data transmitted in a first direction and kernel data transmitted in a second direction from a kernel memory and outputting a first operation result to a activation unit in a third direction opposite to the first direction And the activation units perform a second operation on the first operation result, 2 The operation result can be transferred to the feature map memory.

The neural network accelerator according to the embodiment of the present invention can perform the convolution operation, the activation operation, the normalization operation, and the pulling operation within the bi-directional PE array to output the operation result to the memory at one time. In addition, the neural network accelerator may receive a partial sum that is the result of the intermediate operation from a bi-directional PE array or may transmit a partial sum to a bi-directional PE array. Therefore, the neural network accelerator can minimize the access to the memory and perform the neural network operation efficiently.

1 illustrates an exemplary CNN according to an embodiment of the present invention.
2 is a block diagram illustrating an exemplary neural network accelerator for performing the CNN operation of FIG.
Figure 3 is a block diagram illustrating the PE array of Figure 2 in more detail.
4 is a block diagram showing the PE of FIG. 3 in more detail.
5 is a flowchart exemplarily showing a method of receiving the calculation result in the X-axis direction in response to an output instruction of the PE of FIG.
FIG. 6 exemplarily shows a first PE and a second PE receiving an output command in the X-axis direction.
7 is a flowchart exemplarily showing how the PE of FIG. 4 transmits a partial sum in a direction opposite to the X axis in response to a load partial sum instruction or a path partial sum instruction.
FIG. 8 exemplarily shows first through third PEs receiving a load partial sum instruction and a path partial sum instruction in a direction opposite to the X axis.
Figure 9 is a block diagram illustrating an exemplary PE array of Figure 2 in accordance with another embodiment of the present invention.

In the following, embodiments of the present invention will be described in detail and in detail so that those skilled in the art can easily carry out the present invention.

The present invention relates to a neural network accelerator for performing an operation of a neural network. The neural network of the present invention may be an artificial neural network (ANN) capable of processing data or information in a manner similar to a biological neural network. The neural network may comprise a number of layers including artificial neurons similar to biological neurons. Neural networks can be used in the field of artificial intelligence to recognize and classify objects. Hereinafter, a Convolutional Neural Network (CNN) for image recognition will be exemplarily described. However, the neural network accelerator of the present invention is not limited to CNN but can also be used to implement a neural network other than CNN.

In general, a convolution operation may represent an operation for detecting a correlation between two functions. In CNN, a convolution operation between input data or a kernel indicating a specific feature and specific variables (e.g., weight, bias, etc.) is repeatedly performed, Or the characteristics of the image can be extracted.

1 illustrates an exemplary CNN according to an embodiment of the present invention. Referring to FIG. 1, the CNN 100 may include first through fifth layers L1 through L5.

The CNN may receive the first data (DATA1) from the outside (e.g., host, memory, etc.). The first data (DATA1) may represent an input image or input data provided to the CNN (100). For example, an input image may be generated from an imaging device comprising a plurality of pixels. The input image may have a size of W1 X H1 X D1. As another example, the first data DATA1 may be an output of another layer.

The first layer L1 may perform a convolution operation on the first data DATA1 using the first kernel K1 and may generate the second data DATA2. The first kernel K1 may represent a filter, a mask, a window, and the like used for extracting a correlation for spatially adjacent values in the first data DATA1. The first layer (L1) can extract a feature of the first data (DATA1). The first data DATA1 and the second data DATA2 output from the first layer L1 may be referred to as feature maps, respectively.

The first layer L1 performs a multiplication operation on the input values of the first data DATA1 and the weight values of the first kernel K1, adds all the multiplication results, i.e., performs a convolution operation, Then, an activation operation can be performed on the addition result. The first layer L1 may perform an activation operation using a non-linear function ReLU (Rectified Linear Unit), a Leaky ReLU function, a Sigmoid function, and a tanh (Hyperbolic Tangent) function. In addition, the first layer L1 may perform a normalization operation on the results of the activation operation. The ranges of the output values of the first layer L1 may be matched or the distributions of the output values of the first layer L1 may be similar through the normalization operation.

The second data DATA2 may have a size of W2 X H2 X D2 and the size of the second data DATA2 may be equal to or different from the size of the first data DATA1. For example, if there is data padding in the edge region of the first data DATA1, the size of the second data DATA2 is smaller than that of the first data DATA1, Size.

The second layer L2 may perform a pooling operation on the second data DATA2 using the second kernel K2. The pooling operation may be referred to as a sub-sampling operation. The second layer L2 selects the value of the pixel at the fixed position among the values of the second data DATA2 corresponding to the second kernel K2 or selects the value of the second data DATA2 corresponding to the second kernel K2 (I.e., Average Pooling), or to select the largest value of the second data (DATA2) corresponding to the second kernel K2 Max Pooling). The third data DATA3 output from the second layer L2 may be referred to as a feature map and the third data DATA3 may have the size of W3 X H3 X D3. For example, the size of the third data (DATA3) may be smaller than the size of the second data (DATA2).

The third layer L3 may perform a convolution operation, an activation operation, a normalization operation, and the like for the third data (DATA3) using the third kernel K3. Next, the fourth layer L4 may perform a pooling operation on the fourth data (DATA4) using the fourth kernel K4. The third layer L3 may perform an operation similar to the first layer L1. For example, the first and third layers L1 and L3 may be convolution layers. The fourth layer L4 may perform an operation similar to the second layer L2. For example, the second and fourth layers L2 and L4 may be pooling layers.

In the embodiment, the sizes (W2XH2XD2, W3XH3XD3, W4X) of the second through fifth data (DATA2 through DATA5) output from the first through fourth layers (L1 through L4) H4 X D4, W5 X H5 X D5) may be the same or different from each other. Although CNN 100 is illustratively shown in FIG. 1 as including four layers L1 through L4, actual CNN 100 may include more layers.

The size of the first data (DATA1) may decrease while passing through the first to fourth layers (L1 to L4). The fifth data (DATA5) may include features extracted while the first data (DATA1) passes through the first to fourth layers (L1 to L4). The feature included in the fifth data (DATA5) may be a feature that can represent the first data (DATA1). In addition, when the first to fourth kernels K1 to K4 of the first to fourth layers L1 to L4 are variously set, various features of the first data DATA1 can be extracted. 1, the sizes of the first to fourth kernels K1 to K4 are all the same, but the sizes of the first to fourth kernels K1 to K4 may be the same or different from each other have.

The fifth layer L5 may perform a full connection operation to the fifth data DATA5 and generate output data OUTPUT DATA. For example, the fifth layer L5 may be a fully connected layer. The output data (OUTPUT DATA) may indicate the result that the first data (DATA1) is recognized or classified. Although CNN 100 is illustratively shown in FIG. 1 as including one fifth layer L5, actual CNN 100 may include more layers. The feature maps DATA1 to DATA5 and kernels K1 to K4 of the first to fourth layers L1 to L4 of the CNN 100 have been briefly described. Hereinafter, an apparatus for implementing the CNN 100 in hardware will be described.

2 is a block diagram illustrating an exemplary neural network accelerator for performing the CNN operation of FIG. Fig. 2 will be described with reference to Fig. The neural network accelerator 1000 includes a processing element array (PE array) 1100, a feature map memory 1200, a kernel memory 1300, and a controller 1400 can do. The neural network accelerator 1000 may be a computing device that is hardware implemented to perform neural network operations such as a neural network device, a neural network circuit, a hardware accelerator, a processing unit, and the like. For example, the neural network accelerator 1000 may include a system on chip (SoC), an application specific integrated circuit (ASIC), a central processing unit (CPU), a graphics processing unit (GPU), a vision processing unit (VPU) Processing Unit), and the like.

The PE array 1100 may include processing elements (PEs) arranged along the X-axis and Y-axis directions. The PEs are described below with reference to FIG. The PE array 1100 may be a systolic array that performs operations according to a synchronization signal (e.g., a clock signal). For example, the operations may include the convolution operation, the activation operation, the normalization operation, and the pulling operation described above in FIG.

The PE array 1100 can receive feature map data transmitted in the X axis direction from the feature map memory 1200. [ The X-axis direction can be directed from the feature map memory 1200 to the PE array 1100. The PE array 1100 can receive kernel data transmitted from the kernel memory 1300 in the Y axis direction perpendicular to the X axis direction. The Y axis direction can be directed from the kernel memory 1300 to the PE array 1100. [ Here, the feature map data may represent the first to fifth data (DATA1 to DATA5, feature maps) of Fig. The kernel data may be associated with the first through fourth kernels (K1 through K4, or filters, windows, masks) of FIG.

The PE array 1100 can perform operations using feature map data and kernel data. The PE array 1100 can transfer the result of the operation to the feature map memory 1200 in a direction opposite to the X axis. The neural network accelerator 1000 may include a PE array 1100 and a feature map memory 1200 for processing various operations of the CNN 100 since various operations of the CNN 100 may be processed within the PE array 1100. [ Circuits, units, and the like between the memory cells. The operation result of the PE array 1100 can be transferred to the feature map memory 1200 immediately. Since the PE array 1100 receives the feature map data in the X axis direction and outputs the calculation results in the opposite direction to the X axis, the PE array 1100 can be a bidirectional PE array based on the X axis.

In the embodiment, the calculation result of the PE array 1100 may be feature map data corresponding to the second to fifth data (DATA2 to DATA5) of Fig. Alternatively, the computation result of the PE array 1100 may be a partial sum, which is the intermediate computation result for generating feature map data. The PE array 1100 may send a partial sum for generating new feature map data to the feature map memory 1200 before transferring the new feature map data to the feature map memory 1200 due to constraints such as the number of PEs, .

The feature map memory 1200 may store input data or feature map data corresponding to the first to fifth data (DATA1 to DATA5) in Fig. In addition, the feature map memory 1200 can receive and store computation results transmitted in the direction opposite to the X axis from the PE array 1100. [ The feature map memory 1200 may transmit feature map data to the PE array 1100 in the X axis direction or may transfer the results of previous operations of the PE array 1100 previously stored in the X axis direction to the PE array 1100 have.

The kernel memory 1300 may store kernel data corresponding to the first to fourth kernels K1 to K4 of FIG. The kernel memory 1300 can transmit kernel data to the PE array 1100 in the Y-axis direction. The kernel data may include kernel weight values used in the convolution operation. The weight values may represent the coupling strength between any and all artificial neurons in any layer. In an embodiment, feature map memory 1200 and kernel memory 1300 may be implemented in different memory devices, respectively, or may be implemented in different regions of a memory device, respectively.

In an embodiment, the locations of feature map memory 1200 and kernel memory 1300 for PE array 1100 are not limited to those shown in FIG. For example, the feature map memory 1200 may be located on the right, above, or below the PE array 1100 and the kernel memory 1300 may be located below, to the left, or to the right of the PE array 1100 have. In either case, regardless of the locations of the feature map memory 1200 and the kernel memory 1300, each of the feature map memory 1200 and the kernel memory 1300 may be relatively < RTI ID = 0.0 > To the remote PEs.

Controller 1400 may generate instructions to control PE array 1100, feature map memory 1200, and kernel memory 1300. [ Controller 1400 may generate a (global) clock signal to which PE array 1100, feature map memory 1200, and kernel memory 1300 are synchronized. The PE array 1100 and feature map memory 1200 may exchange feature map data and computation results based on the clock signal. The kernel memory 1300 may send kernel data to the PE array 1100 based on the clock signal.

Figure 3 is a block diagram illustrating the PE array of Figure 2 in more detail. Fig. 3 will be described with reference to Figs. 1 and 2. Fig. The PE array 1100 may include PEs 1110, Feature Map I / O Units 1120, and Kernel Load Units 1130.

The PEs 1110 may be disposed along the X-axis and Y-axis directions and may constitute a two-dimensional array. Each of the PEs 1110 can perform the operation using the feature map data transmitted in the X axis direction from the feature map memory 1200 and the kernel data transmitted in the Y axis direction from the kernel memory 1300. Each of the PEs 1110 may transmit the computation result to the feature map memory 1200 in a direction opposite to the X axis.

Each of the PEs 1110 may be synchronized to a clock signal generated by the controller 1400 of FIG. Each of the PEs 1110 may transmit data to adjacent PEs (or next PEs) in the X and Y axis directions based on the clock signal. More specifically, each of the PEs 1110 may transmit feature map data to the neighbor PE in the X-axis direction based on the clock signal. Each of the PEs 1110 can transmit the calculation result to the adjacent PE in the direction opposite to the X axis based on the clock signal. Each of the PEs 1110 can send kernel data to the neighbor PE based on the Y-axis based on the clock signal. The PEs 1110 may constitute a systolic array and may operate concurrently based on the clock signal. The number of PEs 1110 may be determined based on the area, speed, power, amount of computation of the CNN 100, and the like of the neural network accelerator 1000.

The feature map input / output units 1120 may be disposed along the Y axis direction. Each of the feature map input / output units 1120 may be arranged along the X axis direction and exchange data with the PEs 1110 corresponding to one row. For example, each of the feature map input / output units 1120 may receive feature map data stored in the feature map memory 1200. Each of the feature map input / output units 1120 may transmit the feature map data received in the X axis direction to the PEs 1110. [

PEs 1110 disposed along the X-axis direction may perform operations on the received feature map data and generate an operation result (new feature map data or a subset to generate new feature map data). Each of the feature map input / output units 1120 may then receive the result of the operation from the PEs 1110 in the opposite direction to the X axis and send the result of the operation received to the feature map memory 1200. Each of the feature map input / output units 1120 may receive a partial sum from the feature map memory 1200 and transmit the received partial sum to the PEs 1110 in the X axis direction.

The kernel load units 1130 may be disposed along the X-axis direction. Each of the kernel load units 1130 may receive kernel data stored in the kernel memory 1300. Each of the kernel load units 1130 may transmit the received kernel data to the PEs 1110 arranged along the Y-axis direction and corresponding to one column. The direction in which the feature map data is transmitted (the direction opposite to the X axis direction or the X axis) and the direction in which the kernel data is transferred (the Y axis direction) may be perpendicular to each other.

4 is a block diagram showing the PE of FIG. 3 in more detail. Fig. 4 will be described with reference to Figs. 1 to 3. Fig. The PE 1110 of FIG. 4 may be any one of the PEs in the PE array 1100 of FIG. That is, the PEs in the PE array 1100 may be implemented identically. The PE 1110 may include a control register 1111, a feature map register 1112, a kernel register 1113, a multiplier 1114, an adder 1115, an accumulation register 1116, and an output register 1117 have.

The control register 1111 can store an instruction to be transmitted in the X-axis direction. An instruction may be generated in the controller 1400 of FIG. The control register 1111 is coupled to other components in the PE 1110 (feature map register 1112, kernel register 1113, multiplier 1114, adder 1115, accumulation register 1116, And output register 1117). The control register 1111 may receive the command in any cycle and may transmit the command stored in the X axis direction to another PE (e.g., the PE on the right) in the next cycle. Here, the cycle may represent any one period of the above-mentioned clock signal. In another example, the control register 1111 may receive an instruction at the rising or falling edge of the clock signal.

Feature map register 1112 may receive and store feature map data transmitted in the X axis direction in any cycle. Feature map register 1112 may provide stored feature map data to multiplier 1114. Feature map register 1112 may transfer feature map data stored in the X axis direction to another PE (e.g., PE on the right) in the next cycle.

The kernel register 1113 can receive and store kernel data transmitted in the Y axis direction in any cycle. The kernel register 1113 may provide stored kernel data to the multiplier 1114. The kernel register 1113 may transmit the kernel data stored in the Y axis direction to another PE (e.g., the lower PE) in the next cycle. The control register 1111, feature map register 1112, and kernel register 1113 may be implemented in hardware using at least one flip-flop, at least one latch, at least one logic gate, and so on.

The multiplier 1114 may perform a multiplication operation on the input value of the received feature map data and the weight value of the received kernel data. For example, the multiplication result of multiplier 1114 may be accumulated in accumulation register 1116 without being immediately transmitted to another PE. For example, when the PE 1110 performs an activation operation, a normalization operation operation, and a pulling operation of the CNN 100, the operation result is transmitted to an adjacent PE (for example, a left PE) Lt; / RTI >

The adder 1115 may perform an addition operation on the multiplication result of the multiplier 1114 and the previous operation result accumulated in the accumulation register 1116 in the previous cycle. Accumulation register 1116 may accumulate or store the addition result of adder 1115. [ When the accumulation register 1116 receives the new addition result of the adder 1115, the addition result stored therein can be the previous calculation result. That is, the multiplication result of the multiplier 1114 can be accumulated in the accumulation register 1116.

In an embodiment, the PE 1110 may perform a Multiply Accumulate (MAC) operation using a multiplier 1114, an adder 1115, and an accumulation register 1116. The multiplier 1114, the adder 1115, and the accumulation register 1116 can be implemented in hardware using at least one flip-flop, at least one latch, at least one logic gate, and the like. PE 1110 may perform the convolution operation, the activation operation, and the normalization operation described above in FIG. 1 by repeatedly performing the multiply-accumulate operation.

The convolution operation is an operation that multiplies the feature map data and the kernel data and adds all of the multiplication results. The activation and normalization operations may include operations that multiply or add the result of a convolution operation stored in accumulation register 1116 with a particular value. That is, the result of the convolution operation stored in the accumulation register 1116 may be provided by a multiplier 1114 (see a solid line arrow) and an adder 1115 (see a dotted line arrow). For example, a PE 1110 that includes a multiplier 1114, an adder 1115, and an accumulation register 1116 may perform an activation operation using the ReLU function or the Leaky ReLU function.

The output register 1117 can store the addition result of the adder 1115. [ The addition result stored in the output register 1117 is the result of operation transmitted from the PE 1110 in the direction opposite to the X axis. The output register 1117 may store the result of the operation transmitted in the opposite direction from the X axis from another PE (e.g., PE on the right).

In an embodiment, PE 1110 may further perform a pooling operation of CNN 100 using multiplier 1114, adder 1115, accumulation register 1116, and output register 1117. The PE 1110 may compare the results of the new operation that performed the convolution operation, the activation operation, and the normalization operation with the results of the previous operation stored in the output register 1117. Here, a new operation result can be generated in the adder 1115. [ The adder 1115 may further include a comparator 1115_1 for performing a comparison operation on the new operation result and the previous operation result.

In the case of the Max-Pulling operation, based on the comparison result of the comparator 1115_1, the output register 1117 may store a new operation result or a larger result of the previous operation result. For example, a new operation result that is larger than the previous operation result may be updated in the output register 1117, or a previous operation result that is larger than the new operation result may be maintained in the output register 1117.

In the case of the overhead pulling operation, the adder 1115 may add a previous operation result and a new operation result and perform a division operation (for example, a shift operation). For example, the result of the division operation may be updated in the output register 1117.

The PE 1110 according to an embodiment of the present invention may internally perform or process both the convolution operation, the activation operation, the normalization operation, and the pulling operation of the CNN 100. [ Thus, the number of data exchanges between the PE array 1100, the feature map memory 1200, and the kernel memory 1300 of FIG. 2 can be minimized. The power consumption of the neural network accelerator 1000 and the areas of the feature map memory 1200 and the kernel memory 1300 can be improved since the access frequency to the feature map memory 1200 and the kernel memory 1300 is reduced.

5 is a flowchart exemplarily showing a method of receiving the calculation result in the X-axis direction in response to an output instruction of the PE of FIG. Fig. 5 will be described with reference to Figs. 1 to 4. Fig.

In step S110, the PE 1110 of FIG. 4 may receive an instruction to be transmitted in the X-axis direction. In step S120, the PE 1110 can determine whether the received command is an output command. When the PE 1110 receives the output command, step S130 may be performed. If the PE 1110 does not receive an output command, the PE 1110 may process the operation corresponding to another command other than the output command.

In step S130, the PE 1110 may activate a valid flag bit. For example, the PE 1110 may set the valid flag bit to one. Of course, the logical value of the activated valid flag bit is not limited to the example described above. The output register 1117 may further store valid flag bits in addition to the operation result of the PE 1110. [ The valid flag bit stored in the output register 1117 can be transmitted to another PE in the opposite direction to the X axis together with the operation result of the PE 1110. [ The valid flag bit of PE 1110 can be used to determine whether a neighboring PE located next to PE 1110 along the opposite direction of the X axis will store the result of the operation of PE 1110 in its own output register have. The valid flag bit of PE 1110 may indicate whether the operation result of PE 1110 is valid.

In step S140, the PE 1110 may determine whether its location is located in the last column. Here, the last column may represent a column that is farthest in the X-axis direction from the column in which the feature map input / output units 1120 of FIG. 3 are disposed among the columns in which the PEs 1110 of FIG. 3 are disposed. 3, feature map input / output units 1120 may be arranged in any row parallel to the X axis. In this case, in step S140, the PE 1110 may determine whether its location is located in the last row.

For example, the PE 1110 may determine its location based on the address information provided by the controller 1400 of FIG. Alternatively, the above address information may be pre-programmed into the PEs placed in the last column. The PE 1110 disposed in the last column may perform step S150. PE 1110 that is not placed in the last column may perform step S160.

In step S150, the PE 1110 may activate a Last Flag Bit. For example, PE 1110 may set the last flag bit to one. Of course, the logical value of the activated last flag bit is not limited to the example described above. The output register 1117 may further store the last flag bits in addition to the operation result and valid flag bits of the PE 1110. [ The last flag bit stored in the output register 1117 can be transmitted to another PE in the opposite direction to the X axis together with the operation result of the PE 1110 and the valid flag bit. The last flag bit may indicate whether the PE 1110 is located in the farthest column from the feature map memory 1200 in the X-axis direction.

In step S160, the PE 1110 can store the operation result, the valid flag bit, and the last flag bit stored in the output register of the adjacent PE located next to the PE 1110 along the X-axis direction in its own output register . The PE 1110 may receive the operation result, the valid flag bit, and the last flag bit transmitted from the neighbor PE in the direction opposite to the X axis.

In step S170, the PE 1110 may determine whether the received last flag bit is activated. For example, the PE 1110 may determine whether the logical value of the received last flag bit is one. If the last flag bit is not activated, the PE 1110 may perform step S160 again. If the last flag bit is activated, the PE 1110 may perform step S180. That is, the PE 1110 repeatedly receives the operation result, the valid flag bit, and the last flag bit stored in the output register of the adjacent PE transmitted in the direction opposite to the X axis until receiving the last flag bit activated from the adjacent PE And store it.

In step S180, the PE 1110 may deactivate both its valid flag bit and the last flag bit based on the activated last flag bit. For example, the PE 1110 may set its own valid flag bit and the last flag bit to zero. The PE 1110 performing step S180 may no longer receive the operation result from the adjacent PE. PE 1110 may deactivate its own valid flag bit and last flag bit to receive another output command. That is, the PE 1110 may reset its valid flag bit and the last flag bit.

FIG. 6 exemplarily shows a first PE and a second PE receiving an output command in the X-axis direction. Fig. 6 will be described with reference to Figs. 1 to 5. Fig. 6, the first PE and the second PE may be disposed along the X axis. The first PE may be positioned next to the second PE along the direction opposite to the X axis while the second PE may be positioned alongside the first PE along the X axis. The first and second PEs may be adjacent to each other. Each of the first and second PEs may be implemented in the same manner as the PE 1110 of FIG.

The first PE may comprise a first output register which is an output register 1117 and the second PE may comprise a second output register which is an output register 1117. [ The first output register may store a first valid flag bit and a first last flag bit. The second output register may store a second valid flag bit and a second last flag bit.

In the first cycle, the first PE may receive an output command. The first PE can activate the first valid flag bit based on the output instruction (see step S130).

In the second cycle, the adjacent PE or feature map input / output unit 1120 located next to the first PE along the direction opposite to the X axis, based on the first valid flag bit and the first last flag bit, , A first valid flag bit, and a first last flag bit (see step S160). The first PE may send an output command to the second PE and the second PE may receive the output command. The second PE may activate the second valid flag bit or the second last flag bit based on the output instruction (see steps S130 and S150).

In the third cycle, the first PE receives, based on the second valid flag bit and the second last flag bit, the second PE data including the second operation result, the second valid flag bit, and the second last flag bit (See step S160). The second PE can send an output command to the adjacent PE (e.g., the third PE) located next to the second PE along the X axis direction.

In an embodiment, the first PE does not necessarily activate the first valid flag bit in the first cycle. For example, the first PE may activate the first valid flag bit in the second cycle or between the first and second cycles. Similarly, the second PE does not necessarily activate the second valid flag bit or the second last flag bit in the second cycle necessarily. For example, the second PE may activate the second valid flag bit or the second last flag bit in the third cycle or between the second and third cycles.

In the fourth cycle, the second PE can receive and store the third PE data of the neighboring PE located next to the second PE along the X-axis direction (see step S160). Another PE or feature map input / output unit 1120 located next to the first PE may receive and store the second PE data transmitted to the first PE (see step S160).

Referring to FIG. 6, PE data is not output every cycle in the first PE. The first PE data and the second PE data may be output in the first and second cycles, respectively. In summary, the operation result of the PE array 1100 can be transferred to the feature map memory 1200 for as many cycles as twice the number of columns in which the PEs 1110 are disposed.

7 is a flowchart exemplarily showing how the PE of FIG. 4 transmits a partial sum in a direction opposite to the X axis in response to a load partial sum instruction or a path partial sum instruction. Fig. 7 will be described with reference to Figs. 1 to 4. Fig. The PE 1110 sends a partial sum in response to a Load Partial Sum command or a Pass Partial Sum command and the direction in which the PE 1110 sends the partial sum in response to the output command Are opposite.

In step S210, the PE 1110 of FIG. 4 may receive an instruction to be transmitted in the X-axis direction. In step S220, the PE 1110 may determine whether the received instruction is a load partial sum instruction. If the PE 1110 receives the load partial sum command, the process proceeds to step S230. If the PE 1110 does not receive the load partial sum command, step S250 may be performed.

In step S230, the PE 1110, in response to the load partial sum instruction, may store a partial sum transmitted along with the load partial sum instruction in the X axis direction. Here, the accumulated register 1116, rather than the feature map register 1112 of the PE 1110, may store the received partial sum. Feature map register 1112 may store feature map data and accumulation register 1116 may store a sub-sum that is the result of an intermediate operation to generate new feature map data.

In step S240, the PE 1110 may receive a path partial sum command transmitted in the X axis direction in the next cycle after receiving the command in step S210. PE 1110 may temporarily store the partial sum sent with the path partial sum instruction in response to the path partial sum instruction. The partial sum transmitted with the path partial sum instruction is for any one of the other PEs located along the X axis direction. The control register 1111 of the PE 1110 may transmit the partial sum instruction temporarily stored in the load partial sum instruction to the adjacent PE in the X axis direction instead of the path partial sum instruction in the next cycle after receiving the path partial sum instruction.

In step S250, the PE 1110 may determine whether the instruction received in step S210 is a path partial sum instruction. If the PE 1110 receives the path partial sum command, the process proceeds to step S260. Here, the point of time when the path partial sum command is transmitted to the PE 1110 in step S250 (i.e., the cycle) and the point in time when the path partial sum command is transmitted to the PE 1110 in step S240 are different. For example, the PE 1110 may receive a path partial sum instruction in the next cycle after the load partial sum instruction in step S240. PE 1110 may further receive the path partial sum instruction in the next cycle (i.e., in step S250) after receiving the path partial sum instruction in step S240. If PE 1110 does not receive the path partial sum instruction, PE 1110 may process the operation corresponding to the instruction other than the path partial sum instruction.

In step S260, the PE 1110 may transmit the path partial sum instruction and the partial sum to the adjacent PE in the X axis direction. As described above, the partial sum transmitted with the path partial sum instruction is for any one of the other PEs located along the X-axis direction, not for the PE 1110. [ In step S240, the control register 1111 may transmit the load partial sum instruction and the partial sum to the adjacent PE. However, in step S260, the control register 1111 may transmit the partial sum instruction and the partial sum to the adjacent PE.

FIG. 8 exemplarily shows first through third PEs receiving a load partial sum instruction and a path partial sum instruction in a direction opposite to the X axis. Fig. 8 will be described with reference to Figs. 1 to 4 and Fig. In Fig. 8, the first to third PEs may be arranged along the X-axis. Along the opposite direction to the X axis, the first PE can be positioned next to the second PE and the second PE can be positioned next to the third PE. Along the X-axis direction, the second PE can be positioned next to the first PE and the third PE can be positioned next to the second PE. The first and second PEs may be adjacent to each other and the second and third PEs may be adjacent to each other. Each of the first to third PEs may be implemented in the same manner as the PE 1110 in FIG.

In the first cycle, the first PE may receive a first load partial sum instruction LC1 and a first partial sum PS1 transmitted in the X-axis direction. The first PE may store the first partial sum PS1 in its accumulation register 1116 (see step S230).

In the second cycle, the first PE may receive a second pass partial sum instruction PC2 and a second partial sum PS2 that are transmitted in the X-axis direction. The second partial sum PS2 is for the second PE rather than the first PE. The first PE may temporarily store the second partial sum PS2 such that the second partial sum PS2 is transmitted to the second PE.

In the third cycle, the first PE sends a second load partial sum instruction LC2 and a second partial sum PS2 in the X-axis direction instead of the second pass partial sum instruction PC2 received in the second cycle to the second PE (refer to step S240). The second PE may store the received second partial sum PS2 (see step S230). In addition, the first PE can receive the third-path partial sum instruction PC3 and the third partial sum PS3 in the X-axis direction. The third subset PS3 is for the third PE, not the first and second PEs. The first PE may temporarily store the third partial sum PS3 so that the third partial sum PS3 is transmitted to the third PE (see step S260).

In the fourth cycle, the second PE may receive the third pass partial sum command PC3 and the third partial sum PS3 transmitted from the first PE in the X axis direction. Similarly, the second PE may temporarily store the third partial sum PS3 so that the third partial sum PS3 is transmitted to the third PE.

In the fifth cycle, the second PE transfers the third load partial sum instruction LC3 and the third partial sum PS3 in the X-axis direction instead of the third-path partial sum instruction PC3 received in the fourth cycle to the third PE (refer to step S240). The third PE may store the received third partial sum PS3 (see step S230).

Referring to FIG. 8, load partial sum instructions are not transmitted to the first to third PEs every cycle. The first PE may receive the first load partial sum instruction LC1 in the first cycle, the second PE may receive the second load partial sum instruction LC2 in the third cycle, and the third PE Lt; / RTI > may receive a third load partial sum instruction LC3 in the fifth cycle. In summary, a partial sum stored in the feature map memory 1200 may be transferred to the PE array 1100 for cycles twice as many as the number of columns in which the PEs 1110 are located.

Figure 9 is a block diagram illustrating an exemplary PE array of Figure 2 in accordance with another embodiment of the present invention. Figure 9 will be described with reference to Figures 1-8. PE array 2100 may include PEs 2110, feature map input / output units 2120, kernel load units 2130, activation units 2140, and multiplexing units 2150. The PEs 2110, feature map input / output units 2120, and kernel load units 2130 are connected to the PEs 1110, feature map input / output units 1120, kernel load units 1130, But may be practically the same.

Activation units 2140 may be disposed along the Y axis between feature map input / output units 2120 and PEs 2110. Activation units 2140 may receive computation results from PEs 2110 based on the steps according to the flowchart of FIG. The activation units 2140 may perform an activation operation using a SigLoid function or a tanh (Hyperbolic Tangent) function as well as an ReLU function or Leaky ReLU function used by each of the PEs 2110. [ Activation units 2140 may send the result of the activation operation to multiplexing units 2150.

Multiplexing units 2150 may be disposed along the Y axis between activation units 2140 and feature map input / output units 2120. Multiplexing units 2150 are operable to receive a result of an operation from PEs 2110 or an operation result of activation units 2140 based on control of controller 1400 of Figure 2 in accordance with an operation performed by CNN 100. [ You can choose either one. The selected operation result may be sent to feature map input / output units 2120.

The above description is a concrete example for carrying out the present invention. The present invention includes not only the above-described embodiments, but also embodiments that can be simply modified or easily changed. In addition, the present invention includes techniques that can be easily modified by using the above-described embodiments.

1000: neural network accelerator;
1100: PE array;
1200: Feature map memory;
1300: kernel memory;
1400: controller;

Claims (20)

1. A neural network accelerator for performing an operation of a neural network including layers, the neural network accelerator comprising:
A kernel memory for storing kernel data related to the filter;
A feature map memory for storing feature map data which are outputs of the layers; And
A PE array comprising processing elements (PEs) disposed along a first direction and a second direction,
Wherein each of the PEs performs an operation using the feature map data transferred in a first direction from the feature map memory and the kernel data transferred in a second direction from the kernel memory, And transfers the computation result to the feature map memory in a third direction. The method according to claim 1,
Wherein the operation includes a multiplication operation, an addition operation, an activation operation, a normalization operation, and a pooling operation. 3. The method of claim 2,
The PE array includes:
Kernel load units for transmitting the kernel data to the PEs in the second direction; And
Further comprising feature map input / output units for transmitting the feature map data to the PEs in the first direction, receiving the result of the operation being transmitted in the third direction, and transmitting the result of the operation to the feature map memory A neural network accelerator. The method of claim 3,
Each of the PEs comprises:
A control register for storing an instruction to be transmitted in the first direction;
A kernel register for storing the kernel data;
A feature map register for storing the feature map data;
A multiplier for performing a multiplication operation on data stored in the kernel register and the feature map register;
An adder for performing an addition operation on a multiplication result of the multiplier and a previous operation result;
An accumulation register for accumulating the previous operation result or the addition result of the adder; And
And an output register for storing the result of the addition or an operation result transmitted from the other PE in the third direction. 5. The method of claim 4,
Wherein each of the PEs performs the activation operation using a ReLU (Rectified Linear Unit) function or a Leaky ReLU function. 5. The method of claim 4,
Wherein the adder compares the operation result stored in the output register with the cumulative result stored in the cumulative register and updates the operation result stored in the output register based on the comparison result. 5. The method of claim 4,
Wherein the feature map input / output unit further receives, based on the output instruction, a new feature map data from the PEs or a partial sum for generating the new feature map data and sends the new feature map data or the partial sum to the feature map A neural network accelerator that transfers further into memory. 8. The method of claim 7,
Wherein the output register comprises: a valid flag bit (Valid Flag Bit) indicating whether or not the result of the operation is valid; and a valid flag bit indicating whether the PE is valid from the feature map memory And further stores a Last Flag Bit indicating whether or not to be placed in a farther column. 9. The method of claim 8,
The PEs include a first PE and a second PE positioned next to the first PE in the first direction, and
Wherein the first PE receives the operation result of the second PE, the valid flag bit, and the last flag bit in the third direction based on the valid flag bit and the last flag bit of the second PE. 10. The method of claim 9,
Wherein the first PE repeatedly receives the operation result of the second PE, the valid flag bit, and the last flag bit in the third direction until the last flag bit of the second PE is activated. 8. The method of claim 7,
Wherein the feature map input / output unit further receives the partial sum from the feature map memory in the first direction based on a Load Partial Sum instruction and a Pass Partial Sum instruction, And further transmits the neural network accelerator to the PEs. 12. The method of claim 11,
Wherein the cumulative register of each of the PEs stores the partial sum transmitted in the first direction in response to the load partial sum instruction being sent to the control register. 13. The method of claim 12,
The PEs include a first PE and a second PE positioned next to the first PE in the first direction, and
Wherein the first PE sends the load partial sum instruction to the second PE instead of the received path partial sum instruction upon receiving the load partial sum instruction and then receiving the path partial sum instruction. 14. The method of claim 13,
Wherein the first PE sends a partial sum to be sent to the second PE together with the received path partial sum instruction in the first direction. 1. A neural network accelerator for performing an operation of a neural network including layers, the neural network accelerator comprising:
A kernel memory for storing kernel data related to the filter;
A feature map memory for storing feature map data which are outputs of the layers; And
Processing elements (PEs) disposed along a first direction and a second direction, and activation arrays disposed between the PEs and the feature map memory in the second direction. However,
Each of the PEs performs a first operation using the feature map data transferred in a first direction from the feature map memory and the kernel data transferred in a second direction from the kernel memory, Transmits a first operation result to the activation units in a third direction opposite to the first operation result, and
Wherein the activation units perform a second operation on the first operation result and transmit a second operation result to the feature map memory. 16. The method of claim 15,
Wherein the first operation comprises a multiplication operation and an addition operation. 17. The method of claim 16,
The second operation may include a neural network including an activation operation using a ReLU (Rectified Linear Unit) function, a Leaky ReLU function, a Sigmoid function, or a tanh (Hyperbolic Tangent) function, a normalization operation, and a pooling operation Accelerator. 18. The method of claim 17,
The PE array includes:
Kernel load units for transmitting the kernel data to the PEs in the second direction;
The processor sends the feature map data to the PEs in the first direction, receives the first operation result or the second operation result transmitted in the third direction, and transmits the received operation result to the feature map memory To < RTI ID = 0.0 > a < / RTI > And
And multiplexing units for selecting any one of the first calculation result and the second calculation result. 19. The method of claim 18,
When each of the PEs performs the activation operation using the ReLU function or the Leaky ReLU function, the multiplexing units select the first operation result, and the received operation result is input to the neural network accelerator . 19. The method of claim 18,
Wherein if the PEs do not perform the activation operation, the multiplexing units select the second operation result and the received operation result is the second operation result.

Images