KR102334473B1 - Adaptive Deep Learning Accelerator and Method thereof - Google Patents

Abstract

Provided are an apparatus and method for accelerating deep learning that can adaptively control hardware speed even when an access pattern of an external memory cannot be predicted by sharing an external memory with other devices. A deep learning acceleration apparatus according to an embodiment of the present invention directly accesses an external memory, and reads data for a deep learning operation from the external memory (Read Direct Memory Access) (RDMA); a first buffer in which data read from RDMA is stored; a checker for understanding the status of RDMA and the first buffer; and an operator for performing a deep learning operation with the data stored in the first buffer; It includes; based on the situation identified in the checker, the controller for varying the operating speed of the calculator.
Accordingly, it is possible to adaptively control the hardware speed even in a state where the access pattern of the external memory cannot be predicted by sharing the external memory with other devices.

Description

Adaptive Deep Learning Accelerator and Method thereof

The present invention relates to image processing and SoC (System on Chip) technology, and more particularly, to a deep learning acceleration apparatus and method capable of controlling hardware speed during deep learning operation.

Most research and development for deep learning hardware accelerators is focused on processing as many convolution parameters (weight) as possible in addition to input data (feature map).

However, there is a limitation of the external memory allowable Bandwidth, which is a physical constraint on the external memory that the deep learning hardware accelerator must access. That is, even if the performance of the deep learning hardware accelerator is excellent and the operator allocation is sufficient, if the data from the external memory is not supplied quickly, it is in a state in which fast operation cannot be performed.

If the deep learning hardware accelerator uses an external memory alone, the bandwidth situation of the accelerator can be predicted in advance, so that the data supply from the external memory can be optimized as long as it does not exceed the peak memory bandwidth.

However, in the case of a small system, it is very rare for a deep learning hardware accelerator to use an external memory alone, and it must be used by sharing the external memory with another interface or main processor. In this case, there is a problem in that the above design method cannot be applied because the access condition of the external memory cannot be predicted.

The present invention has been devised to solve the above problems, and an object of the present invention is to adaptively control the hardware speed even in a state where the access pattern of the external memory cannot be predicted by sharing the external memory with other devices. It is to provide an apparatus and method for accelerating deep learning that can do this.

According to an embodiment of the present invention for achieving the above object, a deep learning acceleration apparatus directly accesses an external memory, and reads data for a deep learning operation from the external memory (Read Direct Memory Access) (RDMA); a first buffer in which data read from RDMA is stored; a checker for understanding the status of RDMA and the first buffer; and an operator for performing a deep learning operation with the data stored in the first buffer; It includes; based on the situation identified in the checker, the controller for varying the operating speed of the calculator.

A deep learning acceleration apparatus according to the present invention includes: a second buffer in which data calculated by deep learning in the calculator is stored; The method further includes a write direct memory access (WDMA) that directly accesses the external memory and writes the data stored in the second buffer to the external memory, and the checker may further understand the status of the WDMA and the second buffer.

The checker determines the bandwidth status of RDMA and WDMA, and the controller can control whether the operator operates according to the identified bandwidth status of RDMA and WDMA.

The controller may control the enable interval of the operator according to the identified RDMA and WDMA bandwidth conditions.

The controller may control the enable signal interval of the operator according to the identified bandwidth conditions of RDMA and WDMA.

The controller may control the length of the enable signal of the operator according to the identified bandwidth conditions of RDMA and WDMA.

The controller may control the speed of the clock applied to the calculator according to the identified bandwidth conditions of RDMA and WDMA.

According to another aspect of the present invention, RDMA (Read Direct Memory Access), by directly accessing an external memory, reading data for a deep learning operation from the external memory; storing, by the first buffer, data read by the RDMA; performing, by an operator, a deep learning operation with data stored in a first buffer; checkers. determining the status of the RDMA and the first buffer; And, the controller, based on the situation identified in the checker, the step of varying the operating speed of the operator; is provided a deep learning acceleration method comprising a.

As described above, according to embodiments of the present invention, it is possible to adaptively control the hardware speed even in a state where the access pattern of the external memory cannot be predicted by sharing the external memory with other devices.

In particular, according to embodiments of the present invention, it is possible to control the operation speed according to the access of a large-capacity external memory for data processing by the same channel/weight each time in the deep learning accelerator, and the operation speed or operation rate of the operator in a peak bandwidth situation It is possible to minimize the effect of the data buffering speed of the accelerator by properly distributing the .

1 is a view showing a deep learning acceleration device to which the present invention is applicable;
2 is a block diagram showing the structure of a deep learning acceleration apparatus according to an embodiment of the present invention;
3 is a block diagram showing the detailed structure of the operator 130 shown in FIG. 2;
4 is a flowchart provided for explaining an operation control method of a deep learning acceleration device according to another embodiment of the present invention;
5 is a block diagram conceptually illustrating the configuration of the deep learning acceleration apparatus shown in FIG. 2 .

Hereinafter, the present invention will be described in more detail with reference to the drawings.

1 is a diagram illustrating a deep learning acceleration device to which the present invention is applicable. The illustrated deep learning acceleration device comprises a read direct memory access (RDMA) 110 , an input buffer 120 , an operator 130 , an output buffer 140 , and a write direct memory access (WDMA) 150 . do.

The deep learning acceleration device receives data from the external memory 10 , performs a deep learning operation, and outputs and stores the operation result to the external memory 10 .

The data input from the external memory 10 is IFmap (Input Feature map: feature data of the input image) and Weight (convolution parameter of the deep learning model), and the deep learning operation result output to the external memory 10 is OFmap ( output feature map).

The operator 130 performs a deep learning operation with data stored in the input buffer 120 . In this process, the operator 130 sequentially calculates and sums each channel for filtering, applies simultaneous filtering to multiple channels, and may simultaneously process images of multiple channels with filters.

However, when simply using the above techniques, when there is a lot of IFmap data for channel increase and operation, it takes a lot of time to input/output data from the external memory 10 .

That is, when a lot of time is spent on data input/output, the data preparation rate for performing the operation of the calculator 130 decreases, resulting in a decrease in speed during hardware operation (the core hold state continues).

This requires a reduction in overall speed or a redesign of the hardware structure in consideration of bandwidth.

As described above, in the structure for performing a number of operations, IFmap and Weight require data input from the external memory 10 , and the OFmap requires storage in the external memory 10 .

That is, input/output bandwidth for sufficient data input/output is very important. In general, when requesting input data in advance, peak bandwidth is required, so it is necessary to distribute the corresponding peak value.

Various hardware implementation methods have been proposed in consideration of the corresponding peak value, but most are for the internal structure of the deep learning accelerator and are limited only to the operation of the entire core.

Accordingly, in an embodiment of the present invention, for deep learning operation acceleration processing using input data, a technique for continuously managing the hardware resource state and adaptively controlling the hardware speed in performing a deep learning operation is proposed. do.

Specifically, in the embodiment of the present invention, the hardware speed is adapted even in a state where the access pattern of the external memory 10 cannot be predicted at all due to the environment in which the external memory 10 is shared by other processors or other interfaces in the system. We present the hardware structure of an adaptive deep learning accelerator that can be controlled efficiently.

2 is a block diagram illustrating the structure of a deep learning acceleration apparatus according to an embodiment of the present invention. The deep learning acceleration apparatus according to an embodiment of the present invention, as shown in FIG. 2, RDMA 110, input buffer 120, operator 130, output buffer 140, WDMA 150, checker ( Checker) (160) and a controller (Controller) (170) is configured to include.

The RDMA 110 is a module that directly accesses the external memory 10 and reads data for a deep learning operation from the external memory 10 . The input buffer 120 is a buffer in which data read from the RDMA 110 is stored.

Data that the RDMA 110 reads from the external memory 10 and stores in the input buffer 120 are IFmap and Weight.

The calculator 130 is a module for performing a deep learning operation with data stored in the input buffer 120 . FIG. 3 is a block diagram illustrating a detailed structure of the calculator 130 shown in FIG. 2 .

As shown, the operator 130 includes a convolution operation module 131, an address tree module 132, a batch normalization module 133, an Add Bias module 134, and an Activation module 135 necessary for a deep learning operation. and a Maxpool module 136 .

The output buffer 140 is a buffer in which OFmap, which is data calculated by deep learning by the operator 130, is stored. The WDMA 150 directly accesses the external memory 10 and writes the data stored in the output buffer 140 to the external memory 10 .

The checker 160 checks the RDMA 110 and the input buffer 120 and checks the output buffer 140 and the WDMA 150 to determine the bandwidth situation of the RDMA 110 and the bandwidth situation of the WDMA 150 .

The controller 170 controls the operation speed of the operator 130 to vary based on the DMA bandwidth situation (channel situation) identified by the checker 160 .

Specifically, the controller 170 may control the interval of the enable signal of the operator 130, the length of the enable signal, etc. according to the DMA bandwidth situation. Furthermore, the controller 170 may control the speed of the clock applied to the operator 130 according to the DMA bandwidth situation.

It is assumed that the specification of the deep learning acceleration device according to the embodiment of the present invention is designed as follows.

Bitwidth of AXI interface, which is an interface between external memory 10 and DMA (110, 150): 512 bits

burst : 16

multiple outstanding : 8

Kernel : 3×3

Fmap : 17bits~32bits

Weight: 16 bits

If 16-channel input data is processed and 16-channel output data is calculated, the data is requested in advance to the RDMA 110 in advance to be stored in the DMA cache, and this is retrieved from the input buffer 120 and line memory because it is line memory-based. make up

The data that can be acquired with one multiple outstanding request is 2048 data (based on 32bits) = 16 pixels × 16channel data within a maximum of 32 clocks, and 2,304 (3×3×16ch(in)×16ch(out)) of 2,304(3×3×16ch(in)×16ch(out)) ) can be simultaneously processed and stored in the output buffer 140 .

For the above operation, if all data is buffered, a peak bandwidth of Pcal×n×m×Input Channel×Output Channel×2 is required according to the number of parallel processing (Pcal). Requires an addition buffer twice the size of

However, if the delay for input/output is increased because there is not enough room in the AXI channel, or the speed of the RDMA 110 and the WDMA 150 are different, the speed of the calculator 130 is reduced or does not operate at all in order to handle this. If there is no function, a phenomenon of insufficient data or an overflow of the output buffer 140 occurs.

However, in the deep learning acceleration device according to an embodiment of the present invention, Go/Stop control of the calculator 130 is possible according to the DMA bandwidth situation for input/output, and also by controlling the clock speed applied to the calculator 130, the calculator The operation/speed of 130 can be adaptively controlled according to the DMA bandwidth situation.

If it is processed in the conventional way, other AXI channels are held until the input/output of the corresponding data is finished, so the response according to the bus channel conditions will be slow.

4 is a flowchart provided to explain a method for controlling an operation of a deep learning acceleration apparatus according to another embodiment of the present invention.

As shown, the RDMA 110 of the deep learning acceleration device reads data for a deep learning operation from the external memory 10 and stores it in the input buffer 120 (S210).

Next, the calculator 130 performs a deep learning operation with the data stored in step S210 and stores the result of the deep learning operation in the output buffer 140 (S220).

Then, the WDMA 150 stores the data stored in step S220 in the external memory 10 (S230).

While steps S210 to S230 are performed, the checker 160 checks the RDMA 110 and the input buffer 120 and checks the output buffer 140 and the WDMA 150 to determine the DMA bandwidth situation (S240) ).

Then, the controller 170 controls the operating speed of the operator 130 to be variable based on the DMA bandwidth situation identified in step S240 (S350). In step S350, the speed control of the operator 130 is performed by adjusting the interval of the enable signal, the length of the enable signal, or the like, or controlling the speed of the applied clock.

FIG. 5 is a diagram conceptually illustrating the configuration of the deep learning acceleration device shown in FIG. 2 . As shown, the deep learning acceleration apparatus according to an embodiment of the present invention is configured to include a communication unit 101 , a processor 102 and a storage unit 103 .

The communication unit 101 is a communication means for transferring data to and from the external memory 10 , and the above-described RDMA 110 and WDMA 150 correspond to these.

The processor 102 is a configuration necessary for performing calculations and control required in the deep learning acceleration device, and the aforementioned calculator 130 , the checker 160 and the controller 170 correspond to them.

The storage unit 103 is a storage in which data input from the external memory 10 and data to be output to the external memory 10 are temporarily stored, and the above-described input buffer 120 and output buffer 140 correspond to this.

So far, the adaptive deep learning acceleration apparatus and method have been described in detail with reference to preferred embodiments.

In the embodiment of the present invention, low-power processing is possible with appropriate speed control through a structure that changes the hardware speed by continuously checking the preparation status of input data and input convolution parameters, and various operations can be performed by controlling the detailed operation of the assigned operator. applicable to the network.

Specifically, the adaptive deep learning acceleration apparatus according to an embodiment of the present invention is designed in a structure that can operate even in an unpredictable situation of large-capacity external memory access, applies a low-power hardware processing technique, and can be used in various situations in an actual system. may be applicable.

In addition, in the embodiment of the present invention, a hardware structure capable of controlling the operation speed according to an external large-capacity memory access is adopted for data processing by the same channel/weight every time in the deep learning acceleration device.

Furthermore, while increasing data reusability, the effect of the data buffering speed of the accelerator can be minimized by properly distributing the operation speed or operation rate of the operator in the peak bandwidth situation.

On the other hand, it goes without saying that the technical idea of the present invention can be applied to a computer-readable recording medium containing a computer program for performing the functions of the apparatus and method according to the present embodiment. In addition, the technical ideas according to various embodiments of the present invention may be implemented in the form of computer-readable codes recorded on a computer-readable recording medium. The computer-readable recording medium may be any data storage device readable by the computer and capable of storing data. For example, the computer-readable recording medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical disk, hard disk drive, or the like. In addition, the computer-readable code or program stored in the computer-readable recording medium may be transmitted through a network connected between computers.

In addition, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention as claimed in the claims In addition, various modifications are possible by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

110 : RDMA (Read Direct Memory Access)
120: input buffer
130: operator
131: convolution operation module
132: address tree module
133: batch normalization module
134 : Add Bias module
135: Activation module
136: Maxpool module 140: output buffer
150: WDMA (Write Direct Memory Access)
160: Checker
170: controller

Claims (8)

RDMA (Read Direct Memory Access) that directly accesses external memory and reads data for deep learning operation from external memory;
a first buffer in which data read from RDMA is stored;
a checker for understanding the status of RDMA and the first buffer; and
an operator for performing a deep learning operation with data stored in the first buffer;
Based on the situation identified in the checker, a controller for varying the operating speed of the calculator; Deep learning acceleration device comprising a.
The method according to claim 1,
a second buffer in which data calculated by deep learning in the calculator is stored;
WDMA (Write Direct Memory Access) for writing the data stored in the second buffer to the external memory by directly accessing the external memory; further comprising,
checkers,
Deep learning acceleration device, characterized in that it further grasps the situation of WDMA and the second buffer.
3. The method according to claim 2,
checkers,
Understand the bandwidth status of RDMA and WDMA,
The controller is
A deep learning accelerator, characterized in that it controls whether the calculator operates according to the identified RDMA and WDMA bandwidth conditions.
4. The method according to claim 3,
The controller is
A deep learning accelerator, characterized in that it controls the enable interval of the calculator according to the identified RDMA and WDMA bandwidth conditions.
5. The method according to claim 4,
The controller is
A deep learning accelerator, characterized in that it controls the enable signal interval of the calculator according to the identified RDMA and WDMA bandwidth conditions.
5. The method according to claim 4,
The controller is
A deep learning accelerator, characterized in that it controls the length of the enable signal of the calculator according to the identified RDMA and WDMA bandwidth conditions.
The method according to claim 1,
The controller is
A deep learning accelerator, characterized in that it controls the speed of the clock applied to the calculator according to the identified RDMA and WDMA bandwidth conditions.
RDMA (Read Direct Memory Access), by directly accessing an external memory, reading data for deep learning operation from the external memory;
storing, by the first buffer, data read by the RDMA;
performing, by an operator, a deep learning operation with data stored in a first buffer;
checkers. determining the status of the RDMA and the first buffer; and
Deep learning acceleration method comprising a; by the controller, based on the situation identified in the checker, varying the operating speed of the operator.

Images