KR102667134B1 - Hardware accelerator for neural network including a single port memory and operation method thereof - Google Patents

Abstract

The present invention configures the on-chip memory of the convolution layer in the CNN hardware accelerator as a single-port memory without double buffering, and can greatly reduce the area of the on-chip memory without reducing accelerator performance compared to the prior art.

Description

Neural network hardware accelerator including single port memory and method of operation thereof {HARDWARE ACCELERATOR FOR NEURAL NETWORK INCLUDING A SINGLE PORT MEMORY AND OPERATION METHOD THEREOF}

The present invention separates memory read and write access timings by changing the raster scan order, which is the operation order used by conventional neural network hardware accelerators.

The present invention configures a CNN accelerator with a single-port memory instead of the dual-port memory used by a conventional CNN (Convolutional Neural Network) accelerator, thereby reducing the area occupied by the on-chip memory among the overall chip area of the accelerator.

CNN hardware accelerator is used to accelerate the large amount of calculations of CNN. CNN hardware accelerators can efficiently accelerate CNNs by operating with small hardware resources and low power compared to GPUs (Graphic processing units) for generally operating CNNs.

CNN hardware accelerators require on-chip memory to temporarily store input data to be used in convolution operations or convolution operation results.

The on-chip memory is located close to the convolution operator, allowing convolution operations to be performed at high operating speeds, but the size of the on-chip memory within the hardware is very small compared to external memory.

In general, as a larger amount of data is stored in the on-chip memory of a CNN hardware accelerator, external memory access is reduced, thereby increasing the operation speed of the hardware accelerator.

Because the conventional CNN hardware accelerator performs convolution operations in the raster scan order, which is the order in which images are input, an operation to read data from the on-chip memory and a write operation to store data received from the previous convolution layer are required in each clock cycle. They occur simultaneously and use dual port memory.

Even when using single-port memory, double buffering is essential, so there is a problem that it takes up the same area as dual-port memory.

1 is a block diagram showing a conventional hardware accelerator.

In Figure 1, Li represents the width of the input feature map input to the ith convolutional layer, and ifmap and ofmap represent the input feature map and output feature map, respectively.

Ni represents the number of channels in the ifmap in Li, Mi represents the number of filters, and Hi and Wi represent the height and width of the convolution filter.

ifmap SRAM temporarily stores the input ifmap and transfers it to the ifmap register (REG).

ifmap SRAM includes Hi-1 banks, and each bank stores Li x Ni words included in the feature map.

The ifmap register (REG) stores Hi x Wi x Ni features in a memory composed of registers for storing ifmaps used in convolution operations.

The filter register (REG) stores Hi x Wi x Ni x Mi filter coefficients for convolution operation.

The features and coefficients stored in the ifmap register and the filter register are provided to the MAC (Multiplication and Accumulation) unit to perform a convolution operation and generate a feature map of the Mi-channel as a result.

The ofmap register stores the output feature map (ofmap), which is provided as input to the next convolutional layer.

FIG. 2 is an algorithm explaining a convolution operation method performed in the convolution layer of FIG. 1.

In the algorithm of Figure 2, Ki represents the height of Li's ifmap.

In the conventional ifmap SRAM structure, ifmap streams are stored in raster scan order in ifmap SRAM and output in the same order.

In this ifmap SRAM, read operations are performed in Hi-2 SRAM banks, and write and read operations are performed in one SRAM bank.

Each bank uses dual-port SRAM to satisfy the throughput requirements of write and read operations.

However, among Hi-1 write ports, only one write port can be accessed at a time, and Hi-2 write ports must be in an idle state, resulting in significant waste of hardware resources.

The present invention provides a CNN hardware accelerator including a single port memory and a method of operating the same so that the area of the on-chip memory is reduced.

The present invention provides a memory structure and access scheduling technology that can prevent memory port waste in order to prevent hardware resource waste while maintaining real-time processing speed.

A hardware accelerator according to an embodiment of the present invention includes a single port memory.

The present invention configures the on-chip memory of the convolution layer in the CNN hardware accelerator as a single-port memory without double buffering, and can greatly reduce the area of the on-chip memory without reducing accelerator performance compared to the prior art.

1 is a block diagram showing a hardware accelerator according to the prior art.
Figure 2 is an algorithm showing a convolution operation according to the prior art.
3 is an explanatory diagram illustrating a convolution operation according to an embodiment of the present invention.
Figure 4 is an algorithm showing a convolution operation according to an embodiment of the present invention.
Figure 5 is a block diagram showing an ifmap register according to an embodiment of the present invention.
Figure 6 is a block diagram showing a hardware accelerator according to an embodiment of the present invention.
7 and 8 are explanatory diagrams illustrating a convolution operation according to an embodiment of the present invention.
9 is a graph illustrating the effect of the present invention.

Hereinafter, embodiments of the present invention will be disclosed with reference to the attached drawings. The following disclosure is an example for explaining the present invention, and the scope of the present invention is not limited to the examples below.

A. Partially-vertical order convolution

In order to reduce the size of the buffer for storing temporary calculation results, the present invention changes the layer processing order of the CNN hardware accelerator.

Figure 3 shows the operation of a convolutional layer applying partial vertical order.

Figures 3 (a) and (b) illustrate the input and output sequences, respectively.

For convenience of explanation, it is assumed that Ki = 12, Li = 5, the filter size is 3x3, that is, Hi = Wi = 3, and zero padding is performed when there is no input data at the image boundary.

In Figures 3 (a) and (b), numbers in boxes indicate processing cycles and arrows indicate processing orders, respectively.

Figure 3(c) shows read and write operations in a buffer (e.g. SRAM) during clock cycles 30 - 45.

The buffer stores the input data (ifmap), which is filtered through the 3x3 kernel of the convolutional layer.

In cycles 30 and 31, the input features (6, 3) and (7, 3) are stored in the ifmap SRAM, which are output from the previous layer.

From cycles 32 to 35, the input features (0,4) to (5,4) are stored in the ifmap register, and the ifmap SRAM is in the idle state.

For synchronous SRAM, one read operation is performed per cycle, so in cycle 36 the input feature (6, 0) is read from ifmap SRAM, which is stored in ifmap REG in cycle 37 and processed by a 3x3 filter.

In the same cycle, the input feature (4,4) is output from the previous layer and stored in ifmap REG.

In cycle 37, input feature (7,0) is read from ifmap SRAM and input feature (5,4) is read from the previous layer and stored in ifmap REG.

Input features (6,4), (7,4) are read from ifmap SRAM in cycles 38 and 39 and stored in ifmap REG.

In cycles 40 to 43, input features (8,0) to (11,0) are stored in ifmap REG and ifmap SRAM is idled.

In cycles 44 and 45, the input features (6,1) and (7,1) are read from ifmap SRAM and stored in ifmap REG.

In cycle 48, a 3x3 filtering operation is performed using (6,0), (7,0), (8,0), (6,1), (7,1), and (8,1) to obtain the output characteristics (7,0) is output.

Zero padding is performed for (6,-1), (7, -1), and (8, -1) as shown in Figure 3(b).

The filtering results are passed to the next convolutional layer in cycle 49.

In cycle 49, a filtering operation is performed on (8,0), and the result is passed to the next convolution layer in cycle 50, where the result is stored in ofmap REG.

The input feature (6,0) is used to generate (5,0), (6,0), and (7,0).

The processing sequence in Figure 3(b) represents a convolution operation in which (5,0) and (6,0) are processed sequentially in cycles 14 and 15, and (7,0) is not processed sequentially. Instead, (7,0) is processed in cycle 33 after (6,0) is processed.

In the operation to generate (7,0), (6, 0), (7,0) and (8,0) are needed.

Since we need (6,0) and (7,0) to generate (5,0) and (6,0), they are stored back in ifmap SRAM in ifmap REG. Similarly, (7,0) is also stored back in ifmap SRAM.

In Figure 3(a), input features in gray boxes are stored in ifmap SRAM, and input features in white boxes are not stored in ifmap SRAM.

One read or write is performed in each cycle.

In the case of Figure 3, read and write operations are repeated every 8 cycles and the convolution operation in vertical order is performed for 8 cycles.

However, there are 4 idle cycles for ifmap SRAM, and these 4 idle cycles can be eliminated by reducing the partial vertical order convolution cycle to 4.

The partial vertical order convolution cycle is expressed as pd _conv , which means the number of clock cycles that perform the convolution operation for each column.

B. Proposed Algorithm

Figure 4 shows the partial vertical order convolution algorithm for the ith convolution layer.

For the ith convolutional layer, the filter size is expressed as Hi x Wi, where Hi is the height of the filter (number of rows) and Wi is the width of the filter (number of columns).

At this time, the batch size is expressed as Mi and Ni, which is assumed to be 1 for convenience.

h_temp indicates a change in processing order in partial vertical order convolution.

h_temp increases from 0 to pd _conv , which is chosen as 2 x (max(Hi)-1) (i>=1). At this time, max(Hi) indicates the maximum value of the filter kernel among all convolutional layers except the 0th convolutional layer.

The value of pd _conv should be chosen to ensure one write or read operation per cycle in the ifmap SRAM.

For all inner and last convolutional layers, Hi-1 lines among the pd _conv lines of the feature map are stored in the ifmap SRAM.

Equation 1 represents the write and read timing for the two-dimensional coordinates (h,w) of the ifmap SRAM in this embodiment.

In Equation 1, mod(a,b) refers to the modulo operation that represents the remainder when a is divided by b.

In this embodiment, Hi-1 lines are stored, which is the same as the conventional case in terms of number.

However, by using a single-port SRAM, the area can be greatly reduced compared to using a conventional dual-port SRAM.

C. Register that stores input to the MAC unit

For each convolution layer, ifmap features are stored in a register, and ifmap REG is used for input to provide to the MAC unit for convolution operation.

As mentioned above, some of the ifmap features are provided by the previous convolutional layer and some are read from the ifmap SRAM.

The number of words stored in ifmap REG is the same as Equation 2.

Figure 5 is a diagram explaining the structure of ifmap REG for output feature (4,0). At this time, the output feature (4,0) is indicated by a box with a blue diagonal line.

The solid arrow indicates the convolution operation order for the current column, and the dotted arrow indicates the convolution operation order for the next column. At this time, max(Hi), Hi, and Wi are 3.

The blue box displays the input features within the 3x3 convolution window, pd _conv is chosen as 4 and the batch size Mi, and Ni are 1.

The input features shown in gray boxes are provided by the ifmap SRAM and the rest are provided by the previous layer.

Since the convolution operation is performed in partial vertical order, the features stored in ifmap REG are updated from top left to bottom right.

For example, the input feature (0,0) of the ifmap REG is invalidated and replaced by the input feature (2,2) provided by the ifmap SRAM.

After the convolution operation on the output feature (4,0), the feature at the (1,0) position of the ifmap REG is invalidated and replaced by the input feature (3,2), which is indicated by a blue check pattern on a gray background. .

For the convolution operation of the output feature (5,0), the input feature (6,1) is required. The input feature (6,1) is stored at location (4,2) of ifmap REG and is indicated by a diagonal line on a gray background. This comes from the ifmap stream.

The input feature (7,1) comes from the ifmap stream stored at position (5,2), corresponds to the gray diagonal box, and is used in the convolution operation of the output feature (6,0).

When the convolution operation is completed for one column, a partial vertical order convolution operation is performed for the next column.

D. Memory structure for the best and last convolutional layers.

Typically, input images are received in raster scan order, so partial vertical order convolution operation requires changing the processing order.

Figure 6 shows an example of the ifmap SRAM structure for the best layer of CNN in this embodiment.

In this embodiment, the convolution layer is assumed to include a 3x3 filter.

Each SRAM block includes H ₀ single-port SRAM banks, and each SRAM block stores H ₀ - 1 + pd _conv lines of input data.

The data stored in the first and last (H ₀ - 1)/2 banks in each SRAM block is data for boundary processing of the convolution operation.

Therefore, a read operation for (H ₀ - 1)/2 banks from the top of one SRAM block is performed in parallel with a read operation for the same number of banks from the bottom.

Each of these SRAM banks for boundary processing requires storage space for L ₀ x N ₀ words.

SRAM bank (H ₀ - 1)/2 stores pd _conv x L ₀ x Nx words and this SRAM changes the address to convert the streaming order from raster scan order to partial vertical order.

The last H ₀ - 1 lines from the input data stored in one block are also stored in another block for the convolution operation for the first convolution layer.

The ifmap SRAM of the best convolutional layer includes multiple blocks with multiple buffers to prevent conflicts in the input and output order of that layer.

The number of blocks of ifmap SRAM in the best convolutional layer is equal to Equation 3.

On the right side of Equation 3, the numerator represents the number of clock cycles required for write and read operations, and the denominator represents the number of clock cycles required for read operations.

Single-port SRAM shares write and read ports, so (H ₀ -1+pd _conv ) x L ₀ clock cycles are required to write input data to SRAM, and pd _conv x L0 clock cycles are required to read input data from SRAM. need.

Therefore, one SRAM block requires (H ₀ - 1 + 2 x pd _conv ) x L ₀ clock cycles to write the next pd _conv x L ₀ data.

When using one block, the number of clock cycles required to write data to SRAM is greater than the number of cycles required to read data, resulting in memory overflow.

Therefore, in order to maintain read throughput greater than write throughput, SRAM with multiple blocks is required.

The final layer also requires SRAM to convert the partial vertical processing sequence into a raster scan sequence for the CNN's final output.

The rasterizer includes two blocks from SRAM by replacing H ₀ and max(Hi) with 1 in Equation 3.

Each block contains one singleport SRAM bank.

Each rasterizer SRAM bank stores pd _conv lines in ofmap, changes addresses, and outputs data in raster scan order.

SISR (Single Image Super Resolution) and noise removal CNN structure applies a rasterizer to make the input and output order the same as the raster scan order.

However, in the case of CNN for image classification, a rasterizer is not required because the ofmap of the final convolution layer is input to the fully connected layer regardless of the output order of the feature map.

E. On-chip SRAM storing multi-channel ifmap

If one ifmap SRAM is shared with multi-channel ifmap, the number of ports in ifmap SRAM is reduced and the total SRAM area in the CNN is also reduced.

There are idle cycles between lines of input data, and these idle cycles are used to share the ifmap SRAM to multi-channel ifmap.

Figure 7 shows an example of separating channels in the output feature map in the first convolutional layer.

Figures 7(a) and 7(b) illustrate the input and output sequences, respectively, where Ki and Li were chosen to be 12 and 5, respectively.

The batch size is 1, Ni is 1, Mi is 2, Hi and Wi are 3, and assumes zero padding.

The numbers in the boxes in FIGS. 7(a) and 7(b) indicate processing cycles and the arrows indicate the processing order.

Figure 7(c) shows write and read operations in the buffer during clock cycles 0 to 59.

In FIG. 7(a), the ifmap is input in raster scan order, and the same five idle cycles for Li exist between each row of the ifmap.

Considering the edge case of 3x3 convolution, each block stores 6 lines for input features.

Therefore, block 1 stores the input features from (3,0) to (8,0), and blocks 0 and 2 store 5 lines of input features due to boundaries without data.

In Figure 7(a), the input features from (3,0) to (4,4) are stored in both blocks 0 and 1, and the input data from (7,0) to (8,4) are stored in blocks 1 and 1. 2 are all saved.

The input features stored in the two SRAM blocks are marked with gray boxes.

In Figure 7(c), input data from (0,0) to (4,4) from cycles 0 to 49 are stored in ifmap SRAM.

Among these 50 cycles, there are 25 idle cycles in ifmap SRAM, which indicates that 2 cycles are available for convolution processing for one output feature.

During cycles 46 to 59, the input features from (0,0) to (2,1) are read from the ifmap SRAM of the first convolutional layer.

ifmap SRAM's read and idle states occur in an alternating manner to match the SRAM's input and output throughput.

The output feature located at (0,0) is output as follows.

In cycles 46 to 55, input data from (0,0) to (0,1) are read from ifmap SRAM and stored in ifmap REG.

In cycle 56, the input feature (1,1) is read from ifmap SRAM and in cycle 57 it is stored in ifmap REG.

In cycle 57, input features (0,0), (1,0), (0,1) and (1,1) are available and an output feature located at (0,0) is generated.

In this cycle, there are two output features (0,0), i.e. (0,0) ₀ and (0,0) ₁ , for channels 0 and 1.

Channels 0 and 1 of the output characteristics can be distinguished by adding one cycle of delay to channel 1, as shown in FIG. 7(b).

Figure 8 explains an example of sharing ifmap SRAM for ifmap of two channels.

Figures 8(a) and 8(b) show examples of input and output sequences, respectively.

At this time, Ki, Li, Hi, Wi, and batch size are the same as the example in Figure 7, and Mi and Ni are assumed to be 2.

In Figures 8(a) and 8(b), numbers inside boxes indicate processing cycles and arrows indicate processing orders.

Figure 8(c) shows write and read operations in the buffer during clock cycles 32 to 59.

The buffer stores an ifmap, which is filtered by the 3x3 kernel of the convolutional layer.

The order of outputting the features of the (3,0) position is as follows.

In Figure 8(a), channels 0 and 1 for ifmap are input at different clock cycles and share a buffer.

In cycles 32 and 33, the features (2,0) ₀ and (2,0) ₁ are read from ifmap SRAM and they are stored in ifmap REG in cycles 33 and 34.

In cycles 34 and 35, the input features (3,0) ₀ and (3,0) ₁ are stored in ifmap REG.

During cycles 36 to 39, the input features (2,4) ₀ , (2,4) ₁ , (3,4) ₀ , and (3,4) ₁ , indicated by gray boxes, are stored in the ifmap SRAM.

During cycles 40 to 43 the input features (2,1) ₀ , (2,1) ₁ , (3,1) ₀ and (3,1) ₁ are read from the ifmap SRAM and during cycles 41 to 44 they are Save it to REG.

In cycles 48 and 49, we store the input features (4,1) ₀ and (4,1) ₁ provided by the previous layer in ifmap REG.

In cycle 49, all features of both channels around the input feature (3,0) are made available and the convolution operation is performed.

To match the order of the output feature map with ifmap, the output feature (3,0) ₁ is delayed by 1 cycle.

Accordingly, in cycles 50 and 51, the output features (3,0) ₀ and (3,1) ₁ are passed to the next convolution layer.

In cycles 51, 53 and 55 the output features (4,0), (5,0) and (6,0) are calculated.

During cycles 52 to 57 the output features (4,0) ₀ , (5,0) ₀ and (6,0) ₀ , (4,0) ₁ , (5,0) ₁ and (6,0) ₁ This is passed on to the next layer.

Equation 4 represents the write and read timings in the (h,w) _Ch coordinates of ifmap SRAM sharing a multi-channel ifmap. At this time, Ch represents the ifmap channel and is 0 or 1.

In Equation 4, N ^s represents the number of channels of ifmap stored in the same ifmap SRAM. At this time, pd ^s _conv and L ^s _i are the values of pd _conv and Li multiplied by N ^s , respectively. Also, h _Ch is the same as N ^s xh + Ch.

The final convolutional layer delivers the output feature map to the rasterizer without distinguishing channels through delay.

Therefore, if ifmap SRAM is shared by two channels ifmap, an idle cycle of the rasterizer occurs and the number of idle cycles becomes the same as the number of write cycles of the rasterizer. Through this, the number of SRAM banks in the rasterizer can be reduced to one.

Figure 9 is a graph showing the effect of the present invention.

Figures 9(a) and 9(b) show the areas of ifmap SRAM and ifmap REG required by CNN used for image enhancement and image classification.

As shown in FIG. 9(a), the SRAM area was reduced by an average of 1.84 times in this embodiment compared to the conventional technology, and the ifmap REG area was increased by 2.3 times in this embodiment.

However, since the size of ifmap REG is very small, about 117.95 times that of ifmap SRAM, the overall on-chip memory area is greatly reduced.

In the case of FIG. 9(b), it can be seen that the memory area is reduced in this embodiment compared to the prior art.

1, 100: hardware accelerator

Claims (6)

Pertaining to an accelerator that performs a convolutional filtering operation on an image map with multiple rows and multiple columns,
a register that stores data while scanning a portion of the image map including a predetermined number of consecutive rows among the plurality of rows in column order; and
a memory that stores data corresponding to some rows of the portion when the portion is scanned in column order;
Including,
The convolution filtering operation is performed using data stored in the register, and some data that is not stored in the register among the data required for the convolution filtering operation is read from the memory and stored in the register, and then the convolution filtering operation is performed. accelerator. The accelerator according to claim 1, wherein the minimum value of the certain number corresponds to twice the value obtained by subtracting 1 from the maximum value among the number of rows of one or more filters for the convolutional filtering operation. The method of claim 1, wherein scanning is performed in a direction in which the row number increases within one column of the portion, and when all data included in one column is scanned, a scanning operation is performed on the next column of the one column to perform the convolution filtering operation. Accelerator that performs. The accelerator according to claim 1, wherein when all data included in the portion of the image map is scanned, the convolution filtering operation is performed while a scan operation is performed on the remaining portion of the image map in column order. The accelerator according to claim 1, wherein the memory is a single-port SRAM. The accelerator of claim 1, wherein the subset of rows includes the last row of the portion.