KR20230134747A - Neural processing device and method for controlling same - Google Patents

Abstract

The present invention discloses a neural processing device and a control method thereof. The neural processing device includes at least one CGRA engine group each including at least one CGRA engine, an L2 memory shared by the at least one CGRA engine group, and an L2 memory shared between the at least one CGRA engine group and the L2 memory. Receive monitoring information about performance between an L2 interconnection and the at least one CGRA engine for exchanging data and the L2 interconnection and the at least one CGRA engine, and according to the monitoring information, the at least one CGRA engine, the It includes a sequencer that individually provides hardware resources for at least one of the L2 memory and the L2 interconnection.

Description

Neural processing device and method for controlling same}

The present invention relates to a neural processing device and a control method thereof. Specifically, the present invention relates to a neural processing device capable of efficient distribution of hardware resources and a control method thereof.

Over the past few years, artificial intelligence (AI) technology has been mentioned as the most promising technology worldwide as a core technology of the 4th Industrial Revolution. The biggest problem with these artificial intelligence technologies is computing performance. For artificial intelligence technology that realizes human learning, reasoning, perception, and natural language translation abilities, the most important thing is to quickly process a lot of data.

The central processing unit (CPU) or graphics processing unit (GPU) of existing computers were used for deep learning learning and inference in early artificial intelligence, but deep learning learning and inference tasks with a high workload were used. Due to its limitations, the Neural Processing Unit (NPU), which is structurally specialized for deep learning tasks, is gaining attention.

This neural network processing device contains a large number of processing elements and processor structures internally, and may have a hierarchical structure of several levels so that each structure can perform tasks optimally. This hierarchical structure can achieve the highest efficiency when composed of units optimized for deep learning tasks.

Registered Patent Publication No. 10-2258566

The object of the present invention is to provide a neural processing device that controls reconfigurable hardware resources in real time.

Additionally, another object of the present invention is to provide a control method for a neural processing device that controls reconfigurable hardware resources in real time.

The objects of the present invention are not limited to the objects mentioned above, and other objects and advantages of the present invention that are not mentioned can be understood by the following description and will be more clearly understood by the examples of the present invention. Additionally, it will be readily apparent that the objects and advantages of the present invention can be realized by the means and combinations thereof indicated in the patent claims.

A neural processing device according to some embodiments of the present invention for solving the above problems includes at least one CGRA engine group each including at least one CGRA engine, an L2 memory shared by the at least one CGRA engine group, and Receive monitoring information about an L2 interconnection for exchanging data between at least one CGRA engine group and the L2 memory and performance between the at least one CGRA engine and the L2 interconnection and the at least one CGRA engine, and a sequencer that individually provides hardware resources to at least one of the at least one CGRA engine, the L2 memory, and the L2 interconnection according to the monitoring information.

Additionally, the monitoring information may include at least one of bandwidth, latency, supply power, and temperature of the at least one CGRA engine.

Additionally, the sequencer includes a monitoring module configured to detect performance problems by receiving monitoring information about the at least one CGRA engine and monitoring information about traffic between the L2 interconnection and the at least one CGRA engine, and When the performance problem is limited by the computational performance, a processor controller that performs performance increase control of the at least one CGRA engine, and when the performance problem is a bandwidth limitation, the L2 memory or an off-chip memory that exchanges data with the L2 memory. It may include a compression activator that performs traffic downward control, and an interconnect controller that performs performance increase control of the L2 interconnection when the performance problem is bandwidth limitation.

Additionally, the processor controller may generate a processor control signal that increases at least one of the supply power and frequency of the at least one CGRA engine.

Additionally, the compression activator may generate a memory control signal that activates at least one of a traffic compression engine that compresses traffic in the L2 memory or the off-chip memory and a traffic decompression engine that decompresses the traffic.

Additionally, the sequencer may receive operation characteristics of an operation task and adjust at least one of voltage and frequency for at least one of the at least one CGRA engine, the L2 memory, and the L2 interconnection according to the operation characteristics.

Additionally, the sequencer may receive virtual device status and adjust at least one of voltage and frequency for at least one of the at least one CGRA engine, the L2 memory, and the L2 interconnection according to the virtual device status.

In addition, it further includes a CGRA Engine Cluster including the at least one CGRA engine group, wherein the sequencer includes an upper sequencer that manages the at least one CGRA engine group within the CGRA engine cluster, and the at least It may include a lower sequencer that manages the at least one CGRA engine within one CGRA engine group.

In addition, the CGRA engine cluster may include different first and second CGRA engine groups, and the lower sequencer may include first and second lower sequencers corresponding to the first and second CGRA engine groups, respectively. .

A control method of a neural processing device according to some embodiments of the present invention for solving the above other problems includes first and second CGRA engine groups each including at least one CGRA engine, and the first and second CGRA engines. Receive monitoring information related to at least one of an L2 memory shared by a group and an L2 interconnection that transmits data between the L2 memory and the first and second CGRA engine groups, and detect performance problems through the monitoring information. And, when the performance problem is the computational performance limitation, it may include performing performance increase control of the at least one CGRA engine.

Additionally, the method may further include determining whether the performance problem is an off-chip memory limitation, and downwardly controlling the off-chip memory traffic if the performance problem is an off-chip memory limitation.

Additionally, downwardly controlling the off-chip memory traffic may include activating a compression engine of the off-chip memory traffic.

Additionally, the method may further include determining whether the performance problem is a limitation of the L2 memory and, if the performance problem is a limitation of the L2 memory, downwardly controlling the L2 memory traffic.

Additionally, downward controlling the L2 memory traffic may include activating a compression engine of the L2 memory traffic.

Additionally, if the performance problem is not the L2 memory limitation, the method may further include controlling the performance of the L2 interconnection to increase.

Additionally, controlling the performance of the L2 interconnection to increase may include overdriving the frequency of the L2 interconnection.

A neural processing device according to some embodiments of the present invention for solving the above problems is at least one CGRA engine group, each including at least one CGRA engine, wherein the at least one CGRA engine includes at least one VP (virtual processor) ), at least one CGRA engine group, an L2 memory shared by the at least one CGRA engine group, an L2 interconnection for exchanging data between the at least one CGRA engine group and the L2 memory, and the VP and a sequencer that scales the voltage and/or frequency of the at least one CGRA engine in real time according to the status of.

Additionally, the number of the at least one CGRA engine may be different from the number of the at least one VP.

Additionally, the status of the VP may be determined according to the correspondence relationship between the at least one VP and the at least one CGRA engine.

The neural processing device of the present invention determines the individual importance of processors with various hierarchical structures and provides hardware resources, so it is possible to perform tasks according to importance.

Additionally, the efficiency of computational tasks can be maximized by using limited hardware resources with optimal efficiency.

In addition to the above-described content, specific effects of the present invention are described below while explaining specific details for carrying out the invention.

1 is a block diagram for explaining a neural processing system according to some embodiments of the present invention.
FIG. 2 is a block diagram for explaining the neural processing device of FIG. 1 in detail.
FIG. 3 is a block diagram for explaining the neural core SoC of FIG. 2 in detail.
FIG. 4 is a block diagram for explaining in detail the CGRA Engine Group of FIG. 3.
FIG. 5 is a conceptual diagram for explaining the hardware structure of the CGRA engine group of FIG. 3.
Figure 6 is a conceptual diagram to explain the hierarchical structure of the neural core SoC.
Figure 7 is a conceptual diagram for explaining a neural processing device according to some embodiments of the present invention.
Figure 8 is a conceptual diagram for explaining a neural processing device according to some embodiments of the present invention.
Figure 9 is a conceptual diagram for explaining a neural processing device according to some embodiments of the present invention.
FIG. 10 is a conceptual diagram for explaining the operation of the sequencer of FIG. 3.
FIG. 11 is a block diagram for explaining monitoring and control operations of the sequencer of FIG. 3.
FIG. 12 is a conceptual diagram for explaining Dynamic Voltage Frequency Scaling (DVFS) according to the operation characteristics of the sequencer of FIG. 3.
FIG. 13 is a conceptual diagram for explaining DVFS according to the virtual device state of the sequencer of FIG. 3.
FIG. 14 is a block diagram for explaining in detail the structure of the sequencer of FIG. 3.
FIG. 15 is a block diagram for explaining in detail the structure of the CGRA engine of FIG. 4.
FIG. 16 is a conceptual diagram for explaining the instruction memory of FIG. 15 in detail.
FIG. 17 is a diagram for explaining the processing element of FIG. 15 in detail.
FIG. 18 is a diagram illustrating the Instruction Set Architecture (ISA) of a neural processing device according to some embodiments of the present invention.
FIG. 19 is a block diagram for explaining the operation of the instruction queue in the CGRA engine of FIG. 4.
FIG. 20 is a block diagram for explaining the LSU of FIG. 15 in detail.
FIG. 21 is a block diagram for explaining the L0 memory of FIG. 15 in detail.
FIG. 22 is a block diagram for explaining the L0 memory bank of FIG. 21 in detail.
FIG. 23 is a block diagram illustrating the software hierarchy of a neural processing device according to some embodiments of the present invention.
FIG. 24 is a block diagram to explain in detail the structure of the CGRA compiler of FIG. 23.
Figure 25 is a block diagram to explain in detail the structure of the CGRA engine scheduler of Figure 24.
FIG. 26 is a block diagram for explaining the CGRA engine compiled according to the restriction module of FIG. 25.
Figure 27 is a block diagram to explain in detail the structure of the front-end compiler of Figure 23.
FIG. 28 is a block diagram for explaining in detail the structure of the backend compiler of FIG. 23.
Figure 29 is a conceptual diagram to explain a deep learning operation performed by a neural processing device according to some embodiments of the present invention.
FIG. 30 is a conceptual diagram illustrating learning and inference operations of a neural network of a neural processing device according to some embodiments of the present invention.
31 is a flowchart illustrating a control method of a neural processing device according to some embodiments of the present invention.
Figure 32 is a flowchart for explaining a compilation method of a neural processing device according to some embodiments of the present invention.
FIG. 33 is a flowchart for explaining the storage step of FIG. 32 in detail.
FIG. 34 is a flowchart illustrating the scheduling step of the storage step of FIG. 33 in detail.
FIG. 35 is a flowchart illustrating in detail the binary code generation step of FIG. 32.

Terms or words used in this specification and patent claims should not be construed as limited to their general or dictionary meaning. According to the principle that the inventor can define terms or word concepts in order to explain his or her invention in the best way, it should be interpreted with a meaning and concept consistent with the technical idea of the present invention. In addition, the embodiments described in this specification and the configurations shown in the drawings are only one embodiment of the present invention and do not completely represent the technical idea of the present invention, so they cannot be replaced at the time of filing the present application. It should be understood that there may be various equivalents, variations, and applicable examples.

Terms such as first, second, A, and B used in the present specification and claims may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, a first component may be named a second component, and similarly, the second component may also be named a first component without departing from the scope of the present invention. The term 'and/or' includes any of a plurality of related stated items or a combination of a plurality of related stated items.

The terms used in the specification and claims are merely used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "include" or "have" should be understood as not precluding the existence or addition possibility of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification. .

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the technical field to which the present invention pertains.

Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and unless clearly defined in the present application, should not be interpreted in an ideal or excessively formal sense. No.

Additionally, each configuration, process, process, or method included in each embodiment of the present invention may be shared within the scope of not being technically contradictory to each other.

Hereinafter, a neural processing device according to some embodiments of the present invention will be described with reference to FIGS. 1 to 30.

1 is a block diagram for explaining a neural processing system according to some embodiments of the present invention.

Referring to FIG. 1, a neural processing system (NPS) according to some embodiments of the present invention may include a first neural processing device 1, a second neural processing device 2, and an external interface 3. .

The first neural processing device 1 may be a device that performs calculations using an artificial neural network. For example, the first neural processing device 1 may be a device specialized for performing deep learning calculation tasks. However, this embodiment is not limited to this.

The second neural processing device 2 may be a device that has the same or similar configuration as the first neural processing device 1. The first neural processing device 1 and the second neural processing device 2 may be connected to each other through an external interface 3 and share data and control signals.

Although FIG. 1 shows two neural processing devices, the neural processing system (NPS) according to some embodiments of the present invention is not limited thereto. That is, in the neural processing system (NPS) according to some embodiments of the present invention, three or more neural processing devices may be connected to each other through the external interface 3. Also, conversely, a neural processing system (NPS) according to some embodiments of the present invention may include only one neural processing device.

FIG. 2 is a block diagram for explaining the neural processing device of FIG. 1 in detail.

Referring to FIG. 2, the first neural processing device 1 includes a neural core SoC 10, a CPU 20, an off-chip memory 30, a first non-volatile memory interface 40, and a first volatile memory interface ( 50), a second non-volatile memory interface 60, and a second volatile memory interface 70.

The Neural Core SoC 10 may be a System on Chip device. The Neural Core SoC (10) is an artificial intelligence computing device and may be an accelerator. The neural core SoC 10 may be, for example, one of a graphics processing unit (GPU), a field programmable gate array (FPGA), and an application-specific integrated circuit (ASIC). However, this embodiment is not limited to this.

The neural core SoC (10) can exchange data with other external computing devices through the external interface (3). Additionally, the neural core SoC 10 may be connected to the non-volatile memory 31 and the volatile memory 32 through the first non-volatile memory interface 40 and the first volatile memory interface 50, respectively.

The CPU 20 may be a control device that controls the system of the first neural processing device 1 and executes program operations. The CPU 20 is a general-purpose computing device and may have low efficiency in performing parallel simple operations commonly used in deep learning. Accordingly, the neural core SoC 10 can achieve high efficiency by performing calculations on deep learning inference and learning tasks.

The CPU 20 can exchange data with other external computing devices through the external interface 3. Additionally, the CPU 20 may be connected to the non-volatile memory 31 and the volatile memory 32 through the second non-volatile memory interface 60 and the second volatile memory interface 70, respectively.

The off-chip memory 30 may be memory placed outside the chip of the neural core SoC 10. Off-chip memory 30 may include non-volatile memory 31 and volatile memory 32.

The non-volatile memory 31 may be a memory that continues to retain stored information even when power is not supplied. The non-volatile memory 31 includes, for example, Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Alterable ROM (EAROM), Erasable Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Memory (EEPROM). Read-Only Memory (e.g., NAND Flash memory, NOR Flash memory, UVEPROM (Ultra-Violet Erasable Programmable Read-Only Memory), FeRAM (Ferroelectric Random Access Memory), MRAM (Magnetoresistive Random Access Memory), PRAM (Phase-change Random Access Memory), SONOS (silicon-oxide-nitride-oxide-silicon), RRAM (Resistive Random Access Memory), NRAM (Nanotube Random Access Memory), magnetic computer memory It may include at least one of (for example, hard disk, diskette drive, magnetic tape), optical disk drive, and 3D XPoint memory, but the present embodiment is not limited thereto.

Unlike the non-volatile memory 31, the volatile memory 32 may be a memory that continuously requires power to maintain stored information. The volatile memory 32 may include, for example, at least one of Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Synchronous Dynamic Random Access Memory (SDRAM), and Double Data Rate SDRAM (DDR SDRAM). there is. However, this embodiment is not limited to this.

The first non-volatile memory interface 40 and the second non-volatile memory interface 60 are, for example, Parallel Advanced Technology Attachment (PATA), Small Computer System Interface (SCSI), Serial Attached SCSI (SAS), and SATA ( It may include at least one of Serial Advanced Technology Attachment) and PCIe (PCI Express). However, this embodiment is not limited to this.

The first volatile memory interface 50 and the second volatile memory interface 70 may be configured to perform, for example, Single Data Rate (SDR), Double Data Rate (DDR), Quad Data Rate (QDR), and eXtreme Data Rate (XDR), respectively. , Octal Data Rate). However, this embodiment is not limited to this.

FIG. 3 is a block diagram for explaining the neural core SoC of FIG. 2 in detail.

2 and 3, the neural core SoC 10 includes a CGRA Engine Cluster 100, an L2 memory 300, a Direct Memory Access (DMA) 400, and a non-volatile memory controller 500. ), a volatile memory controller 600, and an L2 interconnection 700.

The CGRA engine cluster 100 may include a plurality of CGRA engine groups 110. Although the CGRA engine cluster 100 is shown as one in the drawing, the present embodiment is not limited thereto.

The CGRA engine group 110 may be a computing device that directly performs computational tasks. If there are multiple CGRA engine groups 110, calculation tasks may be assigned to each CGRA engine group 110. Each CGRA engine group 110 may be connected to each other through an L2 interconnection 700.

The sequencer 200 may individually provide hardware resources to the CGRA engine group 110. That is, the importance of the work of the CGRA engine group 110 is determined according to the judgment of the sequencer 200, and hardware resources may be provided differentially accordingly. At this time, hardware resources may include at least one of voltage, power, frequency, and bandwidth. However, this embodiment is not limited to this.

The sequencer 200 may monitor the operation of several CGRA engine groups 110 within the CGRA engine cluster 100 and provide hardware resources to each CGRA engine group 110 accordingly. The sequencer 200 may monitor various performance parameters of the CGRA engine group 110. The sequencer 200 may detect performance problems determined by monitoring and provide hardware resources accordingly. Accordingly, the CGRA engine group 110 can efficiently perform various calculation tasks according to instructions from the sequencer 200.

The importance of the sequencer 200 can be determined based on various criteria. First, the sequencer can determine importance based on QoS (Quality of Service). That is, a priority selection method to ensure a specific level of performance can be used by the sequencer 200.

Additionally, the sequencer 200 can determine importance according to Service Level Objectives (SLO). This SLO can be set to an appropriate value in advance and can be updated later in various ways.

That is, the sequencer 200 can determine the importance of a task based on criteria such as QoS and/or SLO and provide hardware resources accordingly.

The L2 memory 300 may be memory shared by several CGRA engine groups 110. The L2 memory 300 may store data of each CGRA engine group 110. Additionally, the L2 memory 300 may receive data from the off-chip memory 30, temporarily store it, and transfer it to each CGRA engine group 110. Conversely, the L2 memory 300 may receive data from the CGRA engine group 110, temporarily store it, and transfer it to the off-chip memory 30.

The L2 memory 300 may require relatively fast memory. Accordingly, the L2 memory 300 may include, for example, SRAM. However, this embodiment is not limited to this. That is, the L2 memory 300 may include DRAM.

The L2 memory 300 may be a memory corresponding to the SoC level, that is, level 2 (L2). That is, the L2 memory 300 can operate at level 2 of the hierarchical structure. The hierarchical structure will be explained in more detail later.

The DMA 400 can directly control data movement without the need for the CGRA engine group 110 to control input/output of data. Accordingly, the DMA 400 can control data movement between memories to minimize the number of interrupts of the CGRA engine group 110.

The DMA 400 can control data movement between the L2 memory 300 and the off-chip memory 30. The non-volatile memory controller 500 and volatile memory controller 600 can move data through the authority of the DMA 400.

The non-volatile memory controller 500 can control read or write operations on the non-volatile memory 31. The non-volatile memory controller 500 can control the non-volatile memory 31 through the first non-volatile memory interface 40.

The volatile memory controller 600 can control read or write operations on the volatile memory 32. Additionally, the volatile memory controller 600 may perform a refresh operation on the volatile memory 32. The volatile memory controller 600 can control the non-volatile memory 31 through the first volatile memory interface 50.

The L2 interconnection 700 may connect at least one CGRA engine group 110, L2 memory 300, DMA 400, non-volatile memory controller 500, and volatile memory controller 600 to each other. Additionally, the external interface 3 may also be connected to the L2 interconnection 700. L2 interconnection 700 is between at least one CGRA engine group 110, L2 memory 300, DMA 400, non-volatile memory controller 500, volatile memory controller 600, and external interface 3. This may be the path through which data moves.

The L2 interconnection 700 can transmit signals for transmission and synchronization of control signals as well as data. That is, in the neural processing device according to some embodiments of the present invention, each CGRA engine group 110 can directly transmit and receive the synchronization signal, rather than having a separate control processor manage the synchronization signal. Accordingly, the latency of the synchronization signal generated by the control processor can be blocked.

That is, when there are multiple CGRA engine groups 110, there may be a dependency of individual tasks in which the task of one CGRA engine group 110 must be completed before the next CGRA engine group 110 can start a new task. Accordingly, in the neural processing device according to some embodiments of the present invention, each CGRA engine group 110 may directly transmit a synchronization signal to another CGRA engine group 110 according to the dependency of the task, instead of a control processor. In this case, compared to the method managed by the control processor, multiple CGRA engine groups 110 can perform synchronization tasks in parallel, thereby minimizing latency due to synchronization.

FIG. 4 is a block diagram for explaining the CGRA engine group of FIG. 3 in detail.

Referring to FIGS. 3 and 4 , the CGRA engine group 110 may include at least one CGRA engine 111, L1 memory 120, and L1 interconnection 130.

At least one CGRA engine 111 may share and perform the work of the CGRA engine group 110. The CGRA engine 111 may be a type of processor. That is, the CGRA engine 111 can perform calculation tasks and derive calculation results.

There may be multiple CGRA engines 111. However, this embodiment is not limited to this. Although several CGRA engines 111 are shown in FIG. 4 as being included in the CGRA engine group 110, the present embodiment is not limited thereto. That is, the CGRA engine group 110 may be composed of only one CGRA engine 111.

The L1 memory 120 may be a memory shared by each CGRA engine 111 within the CGRA engine group 110. The L1 memory 120 can store data of each CGRA engine 111. Additionally, the L1 memory 120 can receive data from the L2 memory 300, temporarily store it, and transmit it to each CGRA engine 111. Conversely, the L1 memory 120 may receive data from the CGRA engine 111, temporarily store it, and transfer it to the L2 memory 300.

The L1 memory 120 may be a memory corresponding to the CGRA engine group level, that is, level 1 (L1). That is, the L2 memory 300 may be shared by the CGRA engine group 110, and the L1 memory 120 may be shared by the CGRA engine 111.

The L1 interconnection 130 may connect at least one CGRA engine 111 and the L1 memory 120 to each other. The L1 interconnection 130 may be a path through which data moves between at least one CGRA engine 111 and the L1 memory 120. The L1 interconnection 130 is connected to the L2 interconnection 700 and can transmit data.

The L1 interconnection 130 may have relatively higher latency sensitivity compared to the L2 interconnection 700. That is, data transmission of the L1 interconnection 130 can be performed faster than that of the L2 interconnection 700.

Conversely, the L2 interconnection 700 may have greater bandwidth than the L1 interconnection 130. Since more data must be transmitted in the L2 interconnection 700 than in the L1 interconnection 130, a bottleneck phenomenon may occur if the bandwidth is small and the overall performance of the device may be reduced. Accordingly, the L1 interconnection 130 and the L2 interconnection 700 can be designed with a focus on different performance parameters.

Additionally, the L2 interconnection 700 may be an expandable structure. That is, the dimensions of the CGRA engine 111 or the dimensions of the CGRA engine group 110 can be fixed to some extent for optimization of operations. On the other hand, the larger the dimension of the CGRA engine cluster 100 as much as hardware resources allow, the better, so the scalability of the L2 interconnection 700 may be one of the very important characteristics.

Here, dimension may mean the scale of the CGRA engine 111 or the CGRA engine group 110. That is, since the CGRA engine group 110 includes at least one CGRA engine 111, the dimension of the CGRA engine group 110 may be determined depending on the number of CGRA engines 111 included in the CGRA engine group 110. . Similarly, since the CGRA engine 111 also includes at least one internal component such as a processing element, an instruction memory, an L0 memory, and an LSU, the dimension of the CGRA engine 111 may be determined depending on the number of these components.

FIG. 5 is a conceptual diagram for explaining the hardware structure of the CGRA engine group of FIG. 3.

Referring to FIG. 5, the CGRA engine cluster 100 may include at least one CGRA engine group 110. Each CGRA engine group 110 can transmit data to each other through a local interconnection 701. The local interconnection 701 may be an interconnection formed separately from the L2 interconnection 700. Alternatively, the local interconnection 701 may be a separate dedicated channel for communication between the CGRA engine group 110 within the L2 interconnection 700.

Each CGRA engine group 110 may include at least one CGRA engine 111. The CGRA engine 111 may be a processing unit optimized for deep learning calculation tasks. The CGRA engine 111 may be a processing unit corresponding to one operation of a deep learning calculation task. In other words, deep learning computational tasks can be expressed as a sequential or parallel combination of multiple operations. The CGRA engine 111 is a processing unit that can each process one operation, and may be the minimum operation unit that can be considered for scheduling from a compiler's perspective.

The neural processing device according to this embodiment can achieve fast and efficient scheduling and performance of calculation tasks by configuring the scale of the minimum calculation unit and hardware processing unit considered from a compiler scheduling perspective to be the same.

In other words, if the processing unit that can be divided into hardware is too large compared to the computational task, inefficiency in the computational task may occur when driving the processing unit. Conversely, it is not appropriate to always schedule a processing unit smaller than the operation, which is the compiler's minimum scheduling unit, because scheduling inefficiencies may occur and hardware design costs may increase.

Therefore, in this embodiment, the scale of the compiler's scheduling unit and the hardware processing unit can be similarly adjusted to simultaneously satisfy fast computational task scheduling and efficient computational task performance without wasting hardware resources.

Figure 6 is a conceptual diagram to explain the hierarchical structure of the neural core SoC.

Referring to FIG. 6, the neural core SoC 10 may include at least one CGRA engine cluster 100 at the top level. Each CGRA engine cluster 100 may include at least one CGRA engine group 110 therein. Furthermore, each CGRA engine group 110 may include at least one CGRA engine 111 therein.

At this time, the level of the CGRA engine 111, which is the lowest level, can be defined as L1, that is, the first level. Accordingly, the level of the CGRA engine group 110 above it can be defined as L2, that is, the second level, and the level of the CGRA engine cluster 100 above it can be defined as L3, that is, the third level.

Although FIG. 6 shows a hierarchical structure of a neural processing device according to some embodiments of the present invention at three levels, the present embodiment is not limited thereto. That is, this embodiment can define a higher-level cluster larger than the CGRA engine cluster 100, so it is possible to have a hierarchical structure with four or more levels.

The sequencer 200 can control all CGRA engine clusters 100, CGRA engine groups 110, and CGRA engines 111 at the top level. Specifically, the sequencer 200 controls the distribution and task performance of computational tasks between the CGRA engine clusters 100, and this is performed by controlling the distribution and task performance of computational tasks between each CGRA engine group 110. It can be. Furthermore, the sequencer 200 may control the CGRA engine group 110 by controlling the distribution and execution of computational tasks between the CGRA engines 111. Since the sequencer 200 can control all CGRA engine clusters 100, the overall task performance can be smoothly controlled.

That is, the sequencer 200 can control all levels of L1, L2, and L3. Additionally, monitoring at all levels can be performed.

Figure 7 is a conceptual diagram for explaining a neural processing device according to some embodiments of the present invention.

Referring to FIGS. 3 and 7 , there may be a plurality of sequencers 200 so that each CGRA engine cluster 100 can be divided and managed in L3. That is, the sequencer 200 may include a first sequencer 210, a second sequencer 220, and a third sequencer 230 that manage different CGRA engine clusters 100. Although FIG. 7 shows three CGRA engine clusters 100 and first to third sequencers 210, 220, and 230, the present embodiment is not limited thereto. The number of CGRA engine clusters 100 and the corresponding number of sequencers 210, 220, and 230 may vary.

Each CGRA engine cluster 100 may include a first CGRA engine group 110a, a second CGRA engine group 110b, and a third CGRA engine group 110c. The first CGRA engine group 110a, the second CGRA engine group 110b, and the third CGRA engine group 110c may each be plural or one. At this time, the first sequencer 210 may control and monitor the operation of the first CGRA engine group 110a and the CGRA engine 111 within the first CGRA engine group 110a. Similarly, the second sequencer 220 may control and monitor the operation of the second CGRA engine group 110b and the CGRA engine 111 within the second CGRA engine group 110b. The third sequencer 230 may control and monitor the operation of the third CGRA engine group 110c and the CGRA engine 111 within the third CGRA engine group 110c.

This embodiment can distribute the overhead concentrated on one sequencer 200. Accordingly, latency caused by the sequencer 200 or performance degradation of the entire device can be prevented, and parallel control for each CGRA engine cluster 100 can be possible.

Figure 8 is a conceptual diagram for explaining a neural processing device according to some embodiments of the present invention.

Referring to FIGS. 3 and 8 , a plurality of sequencers 210a, 210b, and 210c may exist for one CGRA engine cluster 100. That is, the sequencer 200 may include a first region sequencer 210a, a second region sequencer 210b, and a third region sequencer 210c. At this time, the number of first to third region sequencers 210a, 210b, and 210c may vary.

The first area sequencer 210a manages the first CGRA engine group 110a corresponding to the first area of one CGRA engine cluster 100 and the CGRA engine 111 within the first CGRA engine group 110a. You can. The second area sequencer 210b may manage the second CGRA engine group 110b corresponding to the second area of the CGRA engine cluster 100 and the CGRA engine 111 within the second CGRA engine group 110b. . The third area sequencer 210c manages the third CGRA engine group 110c corresponding to the third area of one CGRA engine cluster 100 and the CGRA engine 111 within the third CGRA engine group 110c. You can.

In this embodiment, the work of the sequencer can be divided by simply dividing the area without separate hardware design for configuring the CGRA engine cluster 100. That is, the overhead concentrated in one sequencer 200 can be distributed while minimizing hardware resources. Accordingly, latency caused by the sequencer 200 or performance degradation of the entire device can be prevented, and parallel control for each CGRA engine cluster 100 can be possible.

Figure 9 is a conceptual diagram for explaining a neural processing device according to some embodiments of the present invention.

Referring to FIGS. 3 and 9, each CGRA engine cluster 100 may include a first CGRA engine group 110a, a second CGRA engine group 110b, and a third CGRA engine group 110c, respectively. . The first CGRA engine group 110a, the second CGRA engine group 110b, and the third CGRA engine group 110c may each be plural or one. At this time, the first sequencer 210 may control and monitor the operation of the first CGRA engine group 110a and the CGRA engine 111 within the first CGRA engine group 110a. Similarly, the second sequencer 220 may control and monitor the operation of the second CGRA engine group 110b and the CGRA engine 111 within the second CGRA engine group 110b. The third sequencer 230 may control and monitor the operation of the third CGRA engine group 110c and the CGRA engine 111 within the third CGRA engine group 110c.

At this time, the first sequencer 210, the second sequencer 220, and the third sequencer 230 are upper sequencers and can control the operation of the CGRA engine groups 110. The first lower sequencer 211, the second lower sequencer 221, and the third lower sequencer 231 exist for each CGRA engine group 110, and the CGRA engine 111 located at the bottom of each CGRA engine group 110 ) operation can be controlled. The first sequencer 210, the second sequencer 220, and the third sequencer 230 may be linked to the first lower sequencer 211, the second lower sequencer 221, and the third lower sequencer 231, respectively. .

This sequencer, divided into upper and lower parts, distributes operation control for each level, thereby reducing overhead and improving the speed of the entire device through parallel control.

FIG. 10 is a conceptual diagram for explaining the operation of the sequencer of FIG. 3.

Referring to FIG. 10, the sequencer 200 monitors the input parameter (In_p) to control at least one of the CGRA engine 111, L2 interconnection 700, L2 memory 300, and off-chip memory 30. You can. The sequencer 200 can control parameters such as bandwidth or latency of the CGRA engine 111, L2 interconnection 700, L2 memory 300, and off-chip memory 30. It is also possible to control the sequencer 200 as well as the L1 memory 120, L1 interconnection 130, and local interconnection 701. However, for convenience of explanation, only the control of the CGRA engine 111, L2 interconnection 700, L2 memory 300, and off-chip memory 30 will be described below.

At this time, the input parameter (In_p) may include at least one of bandwidth, latency, supplied power, and temperature.

At this time, bandwidth may mean the size of the CGRA engine 111 and external data transmission traffic over time. In the case of bandwidth, the CGRA engine 111 and the corresponding memory, that is, the L2 memory 300 or off-chip memory 30, or the traffic of the L2 interconnection 700 connecting the two may be related in various ways. there is.

At this time, latency is one of the parameters of the calculation performance of the CGRA engine 111 and may mean a period during which the results processed by the CGRA engine 111 are delayed. Latency can be minimized by increasing the frequency of the CGRA engine 111 or increasing the power supplied to the CGRA engine 111. Supply power and temperature are parameters related to the hardware operating environment, and by controlling them, the performance of the hardware can be maximized.

The sequencer 200 controls the operation of at least one of the CGRA engine 111, L2 interconnection 700, L2 memory 300, and off-chip memory 30 using the above input parameter (In_p) and resolves performance issues. can be solved.

FIG. 11 is a block diagram for explaining monitoring and control operations of the sequencer of FIG. 3.

Referring to FIG. 11, each CGRA engine 111 may be mapped to a virtual device (VP). In other words, a virtual device (VP) can be implemented to efficiently provide the necessary hardware resources according to the characteristics of the computational task. Two or more CGRA engines 111 may be mapped to one virtual device (VP), and in this case, two or more CGRA engines 111 mapped to one virtual device (VP) may operate like one unit. there is.

Accordingly, the number of actual CGRA engines 111 and the number of virtual devices (VP) may be different. At this time, the number of virtual devices (VP) may be equal to or less than the actual number of CGRA engines 111.

A virtual device (VP) may exchange data with the L2 interconnection 700. Data exchange (Ex) can be recorded through the virtual device (VP) and L2 interconnection 700, and can be monitored by the sequencer 200.

The sequencer 200 can monitor the operation of the CGRA engine 111. At this time, the latency, supplied power, and temperature of the CGRA engine 111 can be monitored. Additionally, the sequencer 200 can monitor the bandwidth between the CGRA engine 111 and the L2 interconnection 700. That is, the sequencer 200 can check the bandwidth by monitoring data exchange (Ex). At this time, the sequencer 200 can receive monitoring information (Im) in real time. At this time, the monitoring information (Im) includes the latency of the CGRA engine 111, the supply power of the CGRA engine 111, the temperature of the CGRA engine 111, and the bandwidth between the CGRA engine 111 and the L2 interconnection 700. It can contain at least one.

The sequencer 200 can detect performance problems by receiving this monitoring information (Im). The performance problem may mean that hardware latency or bandwidth is detected below a preset standard. Specifically, the performance problem may be at least one of a bandwidth limitation problem or a computational performance limitation problem.

Correspondingly, the sequencer 200 may generate and transmit at least one of a processor control signal (Proc_Cont), a memory control signal (Mem_Cont), and an interconnection control signal (Inter_Cont). The sequencer 200 may transmit at least one of a processor control signal (Proc_Cont), a memory control signal (Mem_Cont), and an interconnection control signal (Inter_Cont) to the CGRA engine 111 and the L2 interconnection 700. The processor control signal (Proc_Cont), memory control signal (Mem_Cont), and interconnection control signal (Inter_Cont) will be described in more detail later.

FIG. 12 is a conceptual diagram for explaining Dynamic Voltage Frequency Scaling (DVFS) according to the operation characteristics of the sequencer of FIG. 3.

Referring to FIG. 12, the sequencer 200 may receive characteristics of the input calculation task (Task), that is, task characteristics (T_st). Task characteristics (T_st) may include the operation and order of the calculation task (Task), type and number of operands, etc.

The sequencer 200 can optimize hardware performance by adjusting the voltage and/or frequency in real time when a computation task is assigned to each CGRA engine 111 according to the task characteristic (T_st). At this time, the hardware controlled by the sequencer 200 may include at least one of the CGRA engine 111, the L2 interconnection 700, the L2 memory 300, and the off-chip memory 30. Of course, the hardware controlled by the sequencer 200 may include at least one of the L1 interconnection 130, the L1 memory 120, and the local interconnection 701.

FIG. 13 is a conceptual diagram for explaining DVFS according to the virtual device state of the sequencer of FIG. 3.

Referring to FIGS. 11 and 13 , the sequencer 200 may receive the status of the virtual device (VP), that is, the virtual device status (V_st). The virtual device status (V_st) may be information depending on which CGRA engine 111 is being used as which virtual device (VP).

When a computation task is assigned to each CGRA engine 111 according to the virtual device status (V_st), the sequencer 200 can optimize hardware performance by adjusting the voltage and/or frequency in real time. In other words, real-time scaling is possible, such as lowering the power supplied to the memory corresponding to the CGRA engine 111 that is not used in the virtual device status (V_st) and increasing the power supplied to the most actively used CGRA engine 111 or memory. You can.

At this time, the hardware controlled by the sequencer 200 may include at least one of the CGRA engine 111, the L2 interconnection 700, the L2 memory 300, and the off-chip memory 30. Of course, the hardware controlled by the sequencer 200 may include at least one of the L1 interconnection 130, the L1 memory 120, and the local interconnection 701.

FIG. 14 is a block diagram for explaining in detail the structure of the sequencer of FIG. 3.

Referring to FIGS. 11 and 14 , the sequencer 200 may include a monitoring module 250, a processor controller 260, a compression activator 270, and an interconnect controller 280.

The monitoring module 250 may receive monitoring information (Im). The monitoring module 250 can detect any performance problems through monitoring information (Im). For example, it is possible to analyze whether bandwidth is limited or computational performance is limited. When bandwidth is limited, it can be identified whether the off-chip memory 30, the L2 memory 300, or the L2 interconnection 700 is limited.

The processor controller 260 may generate a processor control signal (Proc_Cont) that increases the supply power or frequency of the CGRA engine 111 when computational performance is limited. The processor controller 260 may transmit a processor control signal (Proc_Cont) to the CGRA engine 111.

The compression activator 270 may perform compression and decompression of data when the bandwidth is limited and the off-chip memory 30 or L2 memory 300 is limited. In other words, when the off-chip memory 30 is limited, the compression activator 270 can compress the traffic of the off-chip memory 30 and generate a memory control signal (Mem_Cont) to decompress it again. . Through this, the compression activator 270 can solve the traffic problem of the off-chip memory 30. The memory control signal (Mem_Cont) can perform compression and decompression by activating the compression engine and decompression engine. At this time, compression and decompression are only one example of traffic downward control, and the present embodiment is not limited thereto.

Additionally, when the L2 memory 300 is limited, the compression activator 270 may compress traffic of the L2 memory 300 and generate a memory control signal (Mem_Cont) to decompress it again. Through this, the compression activator 270 can solve the traffic problem of the L2 memory 300. At this time, compression and decompression are only one example of traffic downward control, and the present embodiment is not limited thereto.

The interconnect controller 280 sends an interconnection control signal (Inter_Cont) to overdrive the frequency of the L2 interconnection 700 when the bandwidth is limited and the off-chip memory 30 or L2 memory 300 is limited. can be created. The interconnection control signal (Inter_Cont) can solve the bandwidth limitation problem by increasing the frequency of the L2 interconnection 700. At this time, frequency overdrive is only an example of interconnection performance increase control, and the present embodiment is not limited thereto.

FIG. 15 is a block diagram for explaining in detail the structure of the CGRA engine of FIG. 4.

Referring to FIG. 15, the CGRA engine 111 may include an instruction memory 111_1, an L0 memory 111_2, a PE array 111_3, and a Load/Store Unit (LSU) 111_4.

FIG. 16 is a conceptual diagram for explaining the instruction memory of FIG. 15 in detail.

Referring to FIG. 16, the instruction memory 111_1 can receive and store instructions. The instruction memory 111_1 may sequentially store instructions internally and provide the stored instructions to the PE array 111_3. At this time, the instruction may direct the operation of the processing element 111_3a included in each PE array 111_3.

Referring again to FIG. 15, the L0 memory 111_2 is a memory located inside the CGRA engine 111, and the CGRA engine 111 can receive all input data required for work from the outside and temporarily store them. Additionally, the L0 memory 111_2 can temporarily store output data calculated by the CGRA engine 111 in order to transmit it to the outside. The L0 memory 111_2 may serve as a cache memory for the CGRA engine 111.

The L0 memory 111_2 can transmit and receive data with the PE array 111_3. The L0 memory 111_2 may be a memory corresponding to L0 (level 0), which is lower than L1. At this time, the L0 memory may be a private memory of the CGRA engine 111 that is not shared, unlike the L1 memory 120 and L2 memory 300. The L0 memory (111_2) can transmit data such as activation or weight and programs to the PE array (111_3).

The PE array 111_3 may be a module that performs calculations. The PE array 111_3 can perform not only one-dimensional operations but also two-dimensional or more matrix/tensor operations. The PE array 111_3 may include a plurality of processing elements 111_3a and a specific processing element 111_3b therein.

The processing element 111_3a and the specific processing element 111_3b may be arranged in rows and columns. The processing element 111_3a and the specific processing element 111_3b may be arranged in m columns. Additionally, the processing element 111_3a may be arranged in n rows, and the specific processing element 111_3b may be arranged in l rows. Accordingly, the processing element 111_3a and the specific processing element 111_3b may be arranged in (n+l) rows and m columns.

The LSU 111_4 may receive at least one of data, control signals, and synchronization signals from the outside through the L1 interconnection 130. The LSU 111_4 may transmit at least one of the received data, control signal, and synchronization signal to the L0 memory 111_2. Similarly, the LSU 111_4 may transmit at least one of data, a control signal, and a synchronization signal to the outside through the L1 interconnection 130.

The CGRA engine 111 may have a CGRA (Coarse Grained Reconfigurable Architecture) structure. Accordingly, the CGRA engine 111 configures each processing element 111_3a and the specific processing element 111_3b of the PE array 111_3 at least one of the L0 memory 111_2, the instruction memory 111_1, and the LSU 111_4. can be connected with That is, the processing element 111_3a and the specific processing element 111_3b do not have to be connected to all of the L0 memory 111_2, the instruction memory 111_1, and the LSU 111_4, but may be connected to some of them.

Additionally, the processing element 111_3a and the specific processing element 111_3b may be different types of processing elements. Accordingly, among the L0 memory 111_2, the instruction memory 111_1, and the LSU 111_4, elements connected to the processing element 111_3a and elements connected to the specific processing element 111_3b may be different.

The CGRA engine 111 of the present invention, which has a CGRA structure, is capable of high-level parallel computation and enables direct data exchange between the processing element 111_3a and the specific processing element 111_3b, so power consumption can be low. Additionally, optimization according to various computational tasks may be possible by including two or more types of processing elements 111_3a.

For example, if the processing element 111_3a is a processing element that performs a two-dimensional operation, the specific processing element 111_3b may be a processing element that performs a one-dimensional operation. However, this embodiment is not limited to this.

FIG. 17 is a diagram for explaining the processing element of FIG. 15 in detail.

Referring to FIG. 17, the processing element 111_3a includes an instruction queue (IQ), a first register (R1), a second register (R2), a third register (R3), an input formatter (I_Form), and an output formatter (O_Form). may include.

The instruction queue (IQ) may receive instructions received from the instruction memory 111_1, divide them, and sequentially provide them to the first register (R1), the second register (R2), and the third register (R3). The first register (R1) can receive source (src) information and converting (CVT) information. The second register (R2) can receive opcode information. The third register (R3) can receive destination (dst) information and converting (CVT) information.

At this time, the opcode may mean the operation code of the corresponding instruction, that is, the operator. Opcodes may include, for example, arithmetic operations such as ADD, SUB, MUL, DIV, and arithmetic shift, and logical operations such as AND, OR, NOT, XOR, logical shift, rotation shift, Complement, and Clear.

The input formatter (I_Form) can determine the operand by receiving source (src) information from the first register (R1). Additionally, the input formatter (I_Form) can receive conversion (CVT) information from the first register (R1) and convert the precision of the operand. In other words, the precision of the input data and the precision required for calculation may be different, so the input formatter (I_Form) can convert the precision. At this time, the source (src) information may include at least one of North (N), East (E), South (S), West (W), Global Register File (GRF), and Bypass. Bypass may be a path transmitted from the output formatter (O_Form).

The second register (R2) can receive opcode information and generate an operator. An operator can use operands to generate output, which is the result of the operation. The output formatter (O_Form) can receive output. The output formatter (O_Form) can receive destination (dst) information from the third register (R3) and transmit output. Additionally, the output formatter (O_Form) can receive conversion (CVT) information from the third register (R3) and convert the output precision. In other words, the precision required for calculation and the precision required for output may be different, so the output formatter (O_Form) can convert the precision.

At this time, destination (dst) information may include at least one of North (N), East (E), South (S), and West (W). Additionally, the output formatter (O_Form) can transfer output to the input formatter (I_Form) through bypass.

The processing element according to this embodiment does not have a separate precision conversion device but performs precision conversion directly from the instruction queue, thereby improving hardware efficiency.

FIG. 18 is a diagram illustrating the Instruction Set Architecture (ISA) of a neural processing device according to some embodiments of the present invention.

17 and 18, the ISA of the neural processing device according to some embodiments of the present invention includes precision, opcode, source (src0~2), and destination (dst) information. can do.

Precision can be included to generate conversion (CVT) information in the input formatter (I_Form) and output formatter (O_Form). The opcode is used to determine the operator, the source is used to determine the operand, and the destination can be included in the ISA for transmission of output.

FIG. 19 is a block diagram for explaining the operation of the instruction queue in the CGRA engine of FIG. 4.

17 to 19, the instruction queue (IQ) may be loaded through the LSU 111_4 and transmitted to each processing element 111_3a and a specific processing element 111_3b. Each processing element 111_3a and the specific processing element 111_3b may receive an instruction queue (IQ) and perform calculation tasks.

FIG. 20 is a block diagram for explaining the LSU of FIG. 15 in detail.

Referring to FIG. 20, the LSU 111_4 includes a local memory load unit (LMLU), a local memory store unit (LMSU), a neural core load unit (NCLU), a neural core store unit (NCSU), a load buffer (LB), and a store unit. It may include a buffer (SB), load engine (LE), store engine (SE), and translation lookaside buffer (TLB).

The local memory load unit (LMLU) may fetch a load instruction for the L0 memory 111_2 and issue the load instruction. When the local memory load unit (LMLU) provides an issued load instruction to the load buffer (LB), memory access requests can be sequentially transmitted to the load engine (LE) according to the order in which the load buffer (LB) was input.

Additionally, the local memory store unit (LMSU) may fetch a store instruction for the L0 memory 111_2 and issue the store instruction. When the local memory store unit (LMSU) provides the issue store instruction to the store buffer (SB), memory access requests can be sequentially transmitted to the store engine (SE) according to the order in which the store buffer (SB) was input.

The neural core load unit (NCLU) can fetch load instructions for the CGRA engine 111 and issue load instructions. When the neural core load unit (NCLU) provides an issued load instruction to the load buffer (LB), memory access requests can be sequentially transmitted to the load engine (LE) according to the order in which the load buffer (LB) was input.

Additionally, the neural core store unit (NCSU) can fetch store instructions for the CGRA engine 111 and issue store instructions. When the neural core store unit (NCSU) provides the issue store instruction to the store buffer (SB), memory access requests can be sequentially transmitted to the store engine (SE) according to the order in which the store buffer (SB) was input.

The load engine (LE) may receive a memory access request and load data through the L2 interconnection 700. At this time, the load engine (LE) can quickly find data using the translation table of recently used virtual addresses and physical addresses in the translation index buffer (TLB). If the virtual address of the load engine (LE) is not in the translation lookaside buffer (TLB), the address translation information can be found in other memory.

The store engine (SE) may receive a memory access request and load data through the L2 interconnection 700. At this time, the store engine (SE) can quickly find data using the translation table of recently used virtual addresses and physical addresses in the translation index buffer (TLB). If the virtual address of the store engine (SE) is not in the translation lookaside buffer (TLB), the address translation information can be found in other memory.

FIG. 21 is a block diagram for explaining the L0 memory of FIG. 15 in detail.

Referring to FIG. 21, the L0 memory 111_2 may include an arbiter (Arb) and at least one memory bank (bk).

When data is stored in the L0 memory 111_2, the arbiter (Arb) can receive data from the load engine (LE). At this time, data can be allocated to a memory bank (bk) in a round robin manner. Accordingly, data may be stored in any one of at least one memory bank (bk).

Conversely, when data is loaded from the L0 memory 111_2, the arbiter (Arb) may receive data from the memory bank (bk) and transfer it to the store engine (SE). The store engine (SE) can store data externally through the local interconnection 200.

FIG. 22 is a block diagram for explaining the L0 memory bank of FIG. 21 in detail.

Referring to FIG. 22, the memory bank (bk) may include a bank controller (bkc) and a bank cell array (bkca).

The bank controller (bkc) can manage read and write operations through the address of data stored in the memory bank (bk). In other words, the bank controller (bkc) can manage the overall input and output of data.

The bank cell array (bkca) may have a structure in which cells in which data is directly stored are aligned in rows and columns. The bank cell array (bkca) can be controlled by the bank controller (bkc).

Figure 23 is a block diagram to explain the software hierarchy of a neural processing device according to some embodiments of the present invention.

Referring to FIG. 23, the software layer structure of a neural processing device according to some embodiments of the present invention may include a DL framework 10000, a compiler stack 20000, and a backend module 30000.

The DL framework (10000) may refer to a framework for a deep learning model network used by users. For example, a trained neural network, that is, a deep learning graph, can be created using programs such as TensorFlow or PyTorch. A deep learning graph can be a code-type representation of a computational task.

The compiler stack 20000 may include a CGRA compiler (CGCP) and a main compiler (Mcp). The CGRA compiler (CGCP) can perform CGRA engine level compilation. That is, the CGRA compiler (CGCP) can perform internal optimization of the CGRA engine 111. The CGRA compiler (CGCP) can store operation codes in the compute library 22000 through CGRA engine level compilation.

In contrast, the main compiler (Mcp) can perform L2 level compilation, that is, CGRA engine group level compilation. That is, the main compiler (Mcp) can perform compilation such as task scheduling between the CGRA engine group 110, L2 memory 300, and L2 interconnection 700. In this embodiment, optimization can be performed twice through CGRA compilation and main compilation.

The main compiler (Mcp) may include an adaptation layer (21000), a compute library (22000), a front-end compiler (23000), a back-end compiler (24000), and a runtime driver (25000).

The adaptation layer (21000) may be a layer in contact with the DL framework (10000). The adaptation layer (21000) can quantize the user's neural network model, that is, the deep learning graph, created in the DL framework (10000) and generate a quantized model. Additionally, the adaptation layer 21000 can convert the model type into a required type. The quantization model may also be in the form of a deep learning graph.

The front-end compiler (23000) can convert the quantization model received from the adaptation layer (21000) into a certain intermediate representation (IR). The converted IR may be a preset expression that is easy to handle later in the backend compiler 24000.

The IR of the front-end compiler 23000 can be optimized in advance at the graph level. Additionally, the front-end compiler 23000 can ultimately generate the IR by converting it into a hardware-optimized layout.

The backend compiler (24000) optimizes the IR converted from the frontend compiler (23000) and converts it into a binary file so that the runtime driver can use it. The back-end compiler (24000) can generate optimized code by dividing the job at a scale that matches the details of the hardware.

The compute library 22000 can store template operations designed in a form suitable for hardware among various operations. The compute library 22000 provides several template operations that require hardware to the backend compiler 24000, allowing optimized code to be generated. At this time, the compute library 22000 can receive the operation code from the CGRA compiler (CGCP) and save it as a template operation. Accordingly, in this embodiment, the template operation that has already been optimized is optimized again through the back-end compiler 24000, so it can be seen that optimization is performed twice.

The runtime driver 25000 may continuously perform monitoring while driving to drive the neural network device according to some embodiments of the present invention. Specifically, it may be responsible for executing the interface of the neural network device.

The backend module 30000 may include an application specific integrated circuit (ASIC) 31000, a field programmable gate array (FPGA) 32000, and a C-model (33000). ASIC (31000) may refer to a hardware chip determined according to a predetermined design method. FPGA 32000 may be a programmable hardware chip. C-model (33000) may refer to a model implemented by simulating hardware in software.

The backend module 30000 can perform various tasks and derive results using binary code generated through the compiler stack 20000.

FIG. 24 is a block diagram to explain in detail the structure of the CGRA compiler of FIG. 23.

5 and 24, the CGRA compiler (CGCP) may include a CE dimension determiner (26000) and a CE scheduler (27000).

The CE dimension determiner 26000 may determine the scale of the CGRA engine 111 according to the input calculation task. That is, the CE dimension determiner 26000 determines the number of processing elements 111_3a and specific processing elements 111_3b included in the CGRA engine 111 to perform optimal calculation tasks.

Furthermore, the CE dimension determiner 26000 may determine the number of CGRA engines 111 included in the CGRA engine group 110. That is, the dimensions of the CGRA engine 111 and the dimensions of the CGRA engine group 110 are determined, so that the unit structure and cluster structure of the final hierarchical structure can be confirmed.

The CE scheduler (27000) can perform CP level scheduling. The CE scheduler 27000 may perform job scheduling of the processing element 111_3a and the specific processing element 111_3b within the CGRA engine 111. Accordingly, an operation code for the operation of each task can be generated.

Figure 25 is a block diagram to explain in detail the structure of the CGRA engine scheduler of Figure 24.

Referring to FIG. 25, the CGRA engine scheduler (27000) includes a Control-Flow Graph (CFG) generation module (27100), an unrolling module (27200), a hyperblocking module (27300), a limiting module (27500), and a scheduling module (27400). ) may include.

The CFG generation module 27100 may receive a deep learning graph from the deep learning DL framework 10000. Deep learning graphs can be expressed in the form of code written by a DL framework. The CFG generation module 27100 can convert a deep learning graph into a CFG (CFG) consisting of nodes and edges in operation units. CFG (CFG) may include a loop that is processed repeatedly a specified number of times, and may also include a conditional branch structure that branches according to a condition.

The unrolling module 27200 can unroll a loop included in CFG (CFG). Additionally, the unrolling module may perform loop peeling, loop flattening, and inlining. The unrolling module 27200 may generate an unrolled CFG (UCFG) by unrolling a loop included in the CFG (CFG).

The hyperblocking module 27300 can generate a hyperblock by receiving an unroll CFG (UCFG) and reconstructing the conditional branch structure. Hyperblocks can be created by merging blocks with the same conditions among different blocks. The hyperblocking module 27300 can generate hyperblocking CFG (HCFG).

The constraint module 27500 may store a hardware constraint (Cst) created based on pre-written experts' knowledge. Hardware limits (Cst) may include information designed in advance to optimize specific operations. In other words, hardware limitations can serve as a guideline on how to reconfigure the CGRA engine 111 when performing a specific input operation.

The scheduling module 27400 may receive a hyperblocking CFG (HCFG) and receive a hardware limit (Cst). The scheduling module 27400 can generate hyperblocking CFG (HCFG) by converting it into operation code (SC) based on the hardware limit (Cst).

FIG. 26 is a block diagram for explaining the CGRA engine compiled according to the restriction module of FIG. 25.

Referring to FIG. 26, when performing matrix multiplication, the PE array 111_3 of the CGRA engine 111 may configure the processing element 111_3a as a multiplier and configure a specific processing element 111_3b as an accumulator. . This configuration can be set through the history of existing hardware implementation. In other words, hardware limitations (Cst) can provide a guide on how to structure operands and operators.

Figure 27 is a block diagram to explain in detail the structure of the front-end compiler of Figure 23.

Referring to FIG. 27, the front-end compiler 23000 may include an L2 scheduler 23100.

The L2 scheduler 23100 can perform scheduling at the L2 level, that is, the CGRA engine group level. That is, the L2 scheduler 23100 can receive the deep learning graph, tile the computational tasks, and perform scheduling at the level of each CGRA engine cluster 100 and CGRA engine group 110. In this embodiment, both CGRA engine level scheduling and CGRA engine group level scheduling exist, thereby maximizing optimization efficiency.

FIG. 28 is a block diagram to explain in detail the structure of the backend compiler of FIG. 23.

Referring to FIG. 28, the backend compiler 24000 may include a code generator 24100 and a CE code generator 24200.

The code generator 24100 may refer to the compute library 22000. The code generator 24100 may generate partial binary code based on the operation code (SC) stored in the compute library 22000. The partial binary code may be a code that is later added to form a binary code. Since operation codes (SC) are stored based on operations, partial binary codes can also be generated based on operations.

The CE code generator 24200 can receive a partial binary code. The CE code generator 24200 can generate a final binary code by summing several partial binary codes. The CE code generator 24200 can transmit binary code to the runtime driver 25000.

Figure 29 is a conceptual diagram to explain a deep learning operation performed by a neural processing device according to some embodiments of the present invention.

Referring to FIG. 29, the artificial neural network model 40000 is an example of a machine learning model, and in machine learning technology and cognitive science, a statistical learning algorithm implemented based on the structure of a biological neural network or an algorithm thereof. It is a structure that executes .

In the artificial neural network model (40000), as in a biological neural network, nodes, which are artificial neurons that form a network through the combination of synapses, repeatedly adjust the weights of the synapses to determine the difference between the correct output corresponding to a specific input and the inferred output. By learning to reduce the error of , a machine learning model with problem-solving capabilities can be expressed. For example, the artificial neural network model 40000 may include random probability models, neural network models, etc. used in artificial intelligence learning methods such as machine learning and deep learning.

A neural processing device according to some embodiments of the present invention may perform calculations by implementing this type of artificial neural network model (40000). For example, the artificial neural network model 40000 may receive an input image and output information about at least a portion of the object included in the input image.

The artificial neural network model (40000) is implemented as a multilayer perceptron (MLP) consisting of multiple layers of nodes and connections between them. The artificial neural network model 40000 according to this embodiment can be implemented using one of various artificial neural network model structures including MLP. As shown in FIG. 25, the artificial neural network model 40000 includes an input layer 41000 that receives an input signal or data 40100 from the outside, and an output layer that outputs an output signal or data 40200 corresponding to the input data. (44000), located between the input layer 41000 and the output layer 44000, receives n signals from the input layer 41000, extracts the characteristics, and transmits them to the output layer 44000 (where n is a positive integer). It consists of a hidden layer (42000 to 43000). Here, the output layer 44000 receives signals from the hidden layers 42000 to 43000 and outputs them to the outside.

The learning methods of the artificial neural network model (40000) include supervised learning, which learns to optimize problem solving by inputting teacher signals (correct answers), and unsupervised learning, which does not require teacher signals. ) There is a way.

The neural processing device can directly generate learning data for training the artificial neural network model (40000) through simulation. In this way, a plurality of input variables and a plurality of output variables corresponding to the input layer 41000 and the output layer 44000 of the artificial neural network model 40000 are matched, respectively, and the input layer 41000, hidden layers 42000 to 43000, and By adjusting the synapse values between nodes included in the output layer 44000, learning can be done so that the correct output corresponding to a specific input can be extracted. Through this learning process, the characteristics hidden in the input variables of the artificial neural network model (40000) can be identified, and the nodes of the artificial neural network model (40000) can be used to reduce the error between the output variables calculated based on the input variables and the target output. You can adjust the synapse value (or weight) between them.

FIG. 30 is a conceptual diagram illustrating learning and inference operations of a neural network of a neural processing device according to some embodiments of the present invention.

Referring to FIG. 30, in the training phase, a number of learning data (TD) may be forwarded to an artificial neural network model (NN) and then forwarded back again. Through this, the weights and biases of each node of the artificial neural network model (NN) are adjusted, and through this, learning can be performed to produce increasingly accurate results. Through this learning process (Training Phase), the artificial neural network model (NN) can be converted into a learned neural network model (NN_T).

In the inference phase, new data (ND) can be input back into the learned neural network model (NN_T). The learned neural network model (NN_T) can take new data (ND) as input and derive result data (RD) through already learned weights and biases. This result data (RD) may be important in terms of which learning materials (TD) were used in the training process (Training Phase) and how much learning materials (TD) were used.

Hereinafter, a control method of a neural processing device according to some embodiments of the present invention will be described with reference to FIGS. 11, 14, and 31. Parts that overlap with the above-described embodiments are omitted or simplified.

31 is a flowchart illustrating a control method of a neural processing device according to some embodiments of the present invention.

Referring to FIG. 31, monitoring information is received and performance problems are detected (S100).

Specifically, referring to FIGS. 11 and 14 , the sequencer 200 may detect performance problems by receiving this monitoring information (Im). Specifically, the performance problem may be at least one of a bandwidth limitation problem or a computational performance limitation problem.

The monitoring module 250 may receive monitoring information (Im). The monitoring module 250 can detect any performance problems through monitoring information (Im). For example, it is possible to analyze whether bandwidth is limited or computational performance is limited. When bandwidth is limited, it can be identified whether the off-chip memory 30, the L2 memory 300, or the L2 interconnection 700 is limited.

Again, referring to FIG. 31, it is determined whether the bandwidth is limited (S200).

If the bandwidth is not limited, it is determined whether the computational performance is limited (S300). If so, control to increase CGRA engine performance is performed (S500).

Specifically, referring to FIG. 14 , the processor controller 260 may generate a processor control signal (Proc_Cont) that increases the supply power or frequency of the CGRA engine 111 when computational performance is limited. The processor controller 260 may transmit a processor control signal (Proc_Cont) to the CGRA engine 111.

Again, referring to FIG. 31, if the bandwidth is limited in step S200, it is determined whether the off-chip memory is limited (S400). If so, off-chip memory traffic is controlled downward (S600).

Specifically, referring to FIG. 14, when the off-chip memory 30 is limited, the compression activator 270 compresses the traffic of the off-chip memory 30 and sends a memory control signal ( Mem_Cont) can be created. Through this, the compression activator 270 can solve the traffic problem of the off-chip memory 30. The memory control signal (Mem_Cont) can perform compression and decompression by activating the compression engine and decompression engine. At this time, compression and decompression are only one example of traffic downward control, and the present embodiment is not limited thereto.

Again, referring to FIG. 31, if the off-chip memory is not restricted in step S400, it is determined whether the L2 memory is restricted (S700). If so, L2 memory traffic is controlled downward (S800).

Specifically, referring to FIG. 14, when the L2 memory 300 is limited, the compression activator 270 compresses the traffic of the L2 memory 300 and sends a memory control signal (Mem_Cont) to decompress it again. can be created. Through this, the compression activator 270 can solve the traffic problem of the L2 memory 300. At this time, compression and decompression are only one example of traffic downward control, and the present embodiment is not limited thereto.

Again, referring to FIG. 31, if the L2 memory is not limited in step S700, control to increase interconnection performance is performed (S900).

Specifically, referring to FIG. 14, the interconnect controller 280 overdrives the frequency of the L2 interconnection 700 when the bandwidth is limited and the off-chip memory 30 or L2 memory 300 is limited. An interconnection control signal (Inter_Cont) can be generated. The interconnection control signal (Inter_Cont) can solve the bandwidth limitation problem by increasing the frequency of the L2 interconnection 700. At this time, frequency overdrive is only an example of interconnection performance increase control, and the present embodiment is not limited thereto.

Hereinafter, a control method of a neural processing device according to some embodiments of the present invention will be described with reference to FIGS. 23 to 25, 27, and 32 to 35. Parts that overlap with the above-described embodiments are omitted or simplified.

FIG. 32 is a flowchart for explaining a compilation method of a neural processing device according to some embodiments of the present invention, and FIG. 33 is a flowchart for explaining the storage step of FIG. 32 in detail. FIG. 34 is a flowchart for explaining in detail the storage step of FIG. 33 and the scheduling step, and FIG. 35 is a flowchart for explaining the binary code generation step of FIG. 32 in detail.

Referring to Figure 32, a deep learning graph created with a deep learning framework is received (S1100).

Specifically, referring to FIG. 23, the DL framework 10000 may refer to a framework for a deep learning model network used by a user. For example, a trained neural network, that is, a deep learning graph, can be created using programs such as TensorFlow or PyTorch. A deep learning graph can be a code-type representation of a computational task.

Again, referring to FIG. 32, the operation code through CGRA compilation is stored in the compute library (S1200).

In detail, referring to FIG. 33, the dimensions of the CGRA engine are determined (S1210).

Specifically, referring to FIG. 24, the CE dimension determiner 26000 may determine the scale of the CGRA engine 111, that is, the dimension, according to the input calculation task. That is, the CE dimension determiner 26000 determines the number of processing elements 111_3a and specific processing elements 111_3b included in the CGRA engine 111 to perform optimal calculation tasks.

Furthermore, the CE dimension determiner 26000 may determine the number of CGRA engines 111 included in the CGRA engine group 110. That is, the dimensions of the CGRA engine 111 and the dimensions of the CGRA engine group 110 are determined, so that the unit structure and cluster structure of the final hierarchical structure can be confirmed.

Again, referring to FIG. 33, CGRA engine level scheduling is performed (S1220).

Referring to FIG. 34 in detail, CFG is created (S1221).

Specifically, referring to FIG. 25, the CFG generation module 27100 may receive a deep learning graph from the deep learning DL framework 10000. Deep learning graphs can be expressed in the form of code written by a DL framework. The CFG generation module 27100 can convert a deep learning graph into a CFG (CFG) consisting of nodes and edges in operation units. CFG (CFG) may include a loop that is processed repeatedly a specified number of times, and may also include a conditional branch structure that branches according to a condition.

Again, referring to FIG. 34, CFG unrolling is performed (S1222).

Specifically, referring to FIG. 25, the unrolling module 27200 can unroll a loop included in CFG (CFG). Additionally, the unrolling module may perform loop peeling, loop flattening, and inlining. The unrolling module 27200 may generate an unrolled CFG (UCFG) by unrolling a loop included in the CFG (CFG).

Again, referring to FIG. 34, a hyperblock is formed (S1223).

Specifically, referring to FIG. 25, the hyperblocking module 27300 may receive an unroll CFG (UCFG) and reconstruct the conditional branch structure to generate a hyperblock. Hyperblocks can be created by merging blocks with the same conditions among different blocks. The hyperblocking module 27300 can generate hyperblocking CFG (HCFG).

Again, referring to FIG. 34, CGRA engine level scheduling is performed according to preset hardware limitations (S1224). Next, an operation code is generated (S1225).

Specifically, referring to FIG. 25, the constraint module 27500 may store a hardware constraint (Cst) generated based on pre-written experts' knowledge. Hardware limitations (Cst) may be information designed in advance about how to implement a specific operation when optimizing it. In other words, hardware limitations can serve as a guideline on how to reconfigure the CGRA engine 111 when performing a specific input operation.

The scheduling module 27400 may receive a hyperblocking CFG (HCFG) and receive a hardware limit (Cst). The scheduling module 27400 can generate hyperblocking CFG (HCFG) by converting it into operation code (SC) based on the hardware limit (Cst). The CGRA compiler (CGCP) can store operation codes in the compute library 22000 through CGRA engine level compilation.

Again, referring to FIG. 32, the deep learning graph is optimized to generate IR (S1300).

Specifically, referring to FIG. 23, the front-end compiler 23000 can convert various neural network models and graphs received from the adaptation layer 21000 into a certain intermediate representation (IR). The converted IR may be a preset expression that is easy to handle later in the backend compiler 24000.

Again, referring to FIG. 32, L2 level scheduling is performed according to IR (S1400).

Referring to FIG. 27, the L2 scheduler 23100 may perform scheduling at the L2 level, that is, the CGRA engine group level. That is, the L2 scheduler 23100 can receive the deep learning graph, tile the calculation tasks, and perform scheduling at the level of each CGRA engine cluster 100 and CGRA engine group 110. In this embodiment, both CGRA engine level scheduling and CGRA engine group level scheduling exist, thereby maximizing optimization efficiency.

Again, referring to FIG. 32, binary code according to the compute library is generated (S1500).

In detail, referring to FIG. 35, a partial binary code is generated (S1510).

Referring to FIG. 28, the code generator 24100 may refer to the compute library 22000. The code generator 24100 may generate partial binary code based on the operation code (SC) stored in the compute library 22000. The partial binary code may be a code that is later added to form a binary code. Since operation codes (SC) are stored based on operations, partial binary codes can also be generated based on operations.

Again, referring to FIG. 35, binary code is generated (S1520).

Referring to FIG. 28, the CE code generator 24200 can receive a partial binary code. The CE code generator 24200 can generate a final binary code by summing several partial binary codes. The CE code generator 24200 can transmit binary code to the runtime driver 25000.

The above description is merely an illustrative explanation of the technical idea of this embodiment, and those skilled in the art will be able to make various modifications and variations without departing from the essential characteristics of this embodiment. Accordingly, the present embodiments are not intended to limit the technical idea of the present embodiment, but rather to explain it, and the scope of the technical idea of the present embodiment is not limited by these examples. The scope of protection of this embodiment should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of rights of this embodiment.

Claims (20)

At least one CGRA Engine Group, each including at least one CGRA Engine;
L2 memory shared by the at least one CGRA engine group;
an L2 interconnection for exchanging data between the at least one CGRA engine group and the L2 memory; and
Receive monitoring information about performance between the at least one CGRA engine and the L2 interconnection and the at least one CGRA engine, and perform a function among the at least one CGRA engine, the L2 memory, and the L2 interconnection according to the monitoring information. Comprising a sequencer that individually provides hardware resources for at least one,
Neural processing device.
According to claim 1,
The monitoring information includes at least one of bandwidth, latency, supply power, and temperature of the at least one CGRA engine.
Neural processing device.
According to claim 1,
The sequencer,
a monitoring module configured to detect performance problems by receiving monitoring information about the at least one CGRA engine and monitoring information about traffic between the L2 interconnection and the at least one CGRA engine;
a processor controller that performs performance increase control of the at least one CGRA engine when the performance problem is a limitation in computational performance;
When the performance problem is bandwidth limitation, a compression activator that performs traffic downward control of the L2 memory or an off-chip memory that exchanges data with the L2 memory;
When the performance problem is bandwidth limitation, comprising an interconnect controller that performs performance increase control of the L2 interconnection,
Neural processing device.
According to clause 3,
The processor controller generates a processor control signal to increase at least one of the supply power and frequency of the at least one CGRA engine,
Neural processing device.
According to clause 3,
The compression activator generates a memory control signal that activates at least one of a traffic compression engine that compresses traffic in the L2 memory or the off-chip memory, and a traffic decompression engine that decompresses the traffic,
Neural processing device.
According to clause 3,
The interconnect controller generates an interconnection control signal that increases the frequency of the L2 interconnection.
Neural processing device.
According to claim 1,
The sequencer receives operational characteristics of the computational task and adjusts at least one of voltage and frequency for at least one of the at least one CGRA engine, the L2 memory, and the L2 interconnection according to the operational characteristics,
Neural processing device.
According to claim 1,
The sequencer receives virtual device status and adjusts at least one of voltage and frequency for at least one of the at least one CGRA engine, the L2 memory, and the L2 interconnection according to the virtual device status,
Neural processing device.
According to claim 1,
Further comprising a CGRA Engine Cluster including the at least one CGRA engine group,
The sequencer,
an upper sequencer that manages the at least one CGRA engine group within the CGRA engine cluster;
Comprising a lower sequencer that manages the at least one CGRA engine in the at least one CGRA engine group,
Neural processing device.
According to clause 9,
The CGRA engine cluster includes different first and second CGRA engine groups,
The lower sequencer includes first and second lower sequencers corresponding to the first and second CGRA engine groups, respectively.
Neural processing device.
First and second CGRA engine groups each including at least one CGRA engine, L2 memory shared by the first and second CGRA engine groups, and between the L2 memory and the first and second CGRA engine groups Receive monitoring information related to at least one of the L2 interconnections transmitting data,
Detect performance problems through the monitoring information,
When the performance problem is the computational performance limitation, performing performance increase control of the at least one CGRA engine,
Control method of a neural processing device.
According to claim 11,
Determine whether the performance issue is an off-chip memory limitation,
Further comprising downward controlling the off-chip memory traffic when the performance problem is the off-chip memory limitation,
Control method of a neural processing device.
According to claim 12,
Downwardly controlling the off-chip memory traffic,
Including activating a compression engine of the off-chip memory traffic,
Control method of a neural processing device.
According to claim 12,
Determine whether the performance problem is the L2 memory limitation,
If the performance problem is the L2 memory limitation, further comprising downward controlling the L2 memory traffic,
Control method of a neural processing device.
According to claim 14,
Downward controlling the L2 memory traffic,
Including activating a compression engine of the L2 memory traffic,
Control method of a neural processing device.
According to claim 14,
If the performance problem is not the L2 memory limitation, further comprising controlling the performance of the L2 interconnection to increase,
Control method of a neural processing device.
According to claim 16,
Increasing the performance of the L2 interconnection is controlled by:
Including overdriving the frequency of the L2 interconnection,
Control method of a neural processing device.
At least one CGRA engine group, each including at least one CGRA engine, wherein the at least one CGRA engine implements at least one virtual processor (VP);
L2 memory shared by the at least one CGRA engine group;
an L2 interconnection for exchanging data between the at least one CGRA engine group and the L2 memory; and
Comprising a sequencer that scales the voltage and/or frequency of the at least one CGRA engine in real time according to the status of the VP,
Neural processing device.
According to clause 18,
The number of the at least one CGRA engine is different from the number of the at least one VP,
Neural processing device.
According to clause 18,
The status of the VP is determined according to the correspondence relationship between the at least one VP and the at least one CGRA engine,
Neural processing device.

Images