KR20230157811A - Processing element and Neural processing device including same - Google Patents

Abstract

The present invention discloses a processing element and a neural processing apparatus comprising a processing element. The processing element receives a weight register for receiving and storing a weight, an input activation register for storing an input activation, receives a first sub-weight of the first precision included in the weight, and receives the first precision included in the input activation. Receive a first sub-input activation, and according to the first sub-weight and the first sub-input activation, apply the first sub-weight and the first sub-input activation to the first precision or a second precision different from the first precision. It includes a flexible multiplier that generates result data by performing a multiplication operation with one of two precisions, and a saturating adder that generates a partial total using the result data.

Description

Processing element and neural processing device including same}

The present invention relates to a processing element and a neural processing device including the same. Specifically, the present invention relates to a processing element that selects a multiplier to perform an operation according to the size of weight and input activation and a neural processing device including the same.

Over the past few years, artificial intelligence (AI) technology has been mentioned as the most promising technology worldwide as a core technology of the 4th Industrial Revolution. The biggest problem with these artificial intelligence technologies is computing performance. For artificial intelligence technology that realizes human learning, reasoning, perception, and natural language translation abilities, the most important thing is to quickly process a lot of data.

The central processing unit (CPU) or graphics processing unit (GPU) of existing computers were used for deep learning learning and inference in early artificial intelligence, but deep learning learning and inference tasks with a high workload were used. Due to its limitations, the Neural Processing Unit (NPU), which is structurally specialized for deep learning tasks, is gaining attention.

A neural network processing unit can generally use data of a certain precision. The larger the number of bits of data, the more precisely the data can be expressed, but it may require more hardware resources.

Registered Patent Publication No. 10-2258566

The object of the present invention is to provide a processing element that can reduce power consumption by performing operations using various procedures depending on the size of data.

Another object of the present invention is to provide a neural processing device that can reduce power consumption by performing operations using various procedures depending on the size of data.

The objects of the present invention are not limited to the objects mentioned above, and other objects and advantages of the present invention that are not mentioned can be understood by the following description and will be more clearly understood by the examples of the present invention. Additionally, it will be readily apparent that the objects and advantages of the present invention can be realized by the means and combinations thereof indicated in the patent claims.

The processing element according to some embodiments of the present invention for solving the above problem includes a weight register for receiving and storing a weight, an input activation register for storing input activation, and a first sub-weight of the first precision included in the weight. Receive, receive the first sub-input activation of the first precision included in the input activation, and according to the first sub-weight and the first sub-input activation, the first sub-weight and the first sub-input activation It includes a flexible multiplier that generates result data by performing a multiplication operation with either the first precision or a second precision different from the first precision, and a saturating adder that generates a partial sum using the result data.

In some embodiments, the flexible multiplier includes a pass decision unit that generates a pass decision signal based on the first sub weight and the first sub input activation, and a first multiplier that performs a multiplication operation with the first precision. A plier, a second multiplier that performs a multiplication operation with the second precision, and according to the pass decision signal, the first sub weight and the first sub input activation are applied to any of the first multiplier and the second multiplier. It may include a demultiplexer that provides one.

In some embodiments, the pass decision unit, when the size of at least one of the first sub weight and the first sub input activation is greater than a predetermined first size, sends the pass decision signal to the first sub weight and the first sub input activation. If the first sub-input activation is generated as a first signal for providing to the first multiplier, and the size of each of the first sub-weight and the first sub-input activation is smaller than or equal to the first size, The pass decision signal may be generated as a second signal for providing the first sub weight and the first sub input activation to the second multiplier.

In some embodiments, the pass determination unit divides the weight into the first precision or the second precision units according to the operation mode signal, generates the first sub-weight, and performs the input activation into the first precision unit. Divided into precision or second precision units, bit division logic for generating the first sub-input activation, the operation mode signal, the first sub-weight, and the pass decision signal according to the first sub-input activation It may include path selection logic for generating and conversion logic for converting the first sub weight and the precision of the first sub input activation.

In some embodiments, the number of first multipliers may be k, and the number of second multipliers may be 2k.

In some embodiments, the first precision may be 2N bits, and the second precision may be N bits.

In some embodiments, the first precision may be INT4 and the second precision may be INT2.

In some embodiments, the weight includes the first sub-weight and the second sub-weight, the input activation includes the first sub-input activation and the second sub-input activation, and the flexible multiplier includes the first sub-input activation. 1 Based on the sub-weight and the first sub-input activation, generate a first pass decision signal, and based on the second sub-weight and the second sub-input activation, generate a second pass decision signal, and the first The first pass decision signal and the second pass decision signal may be generated independently from each other.

In some embodiments, the weight includes the first sub-weight and the second sub-weight, the input activation includes the first sub-input activation and the second sub-input activation, and the flexible multiplier includes the first sub-input activation. The pass decision signal may be generated based on the 1 sub weight, the second sub weight, the first sub input activation, and the second sub input activation.

In some embodiments, the flexible multiplier may include a control pipeline that synchronizes reception of the first sub-weight and the first sub-input activation, and generation of the result data.

A processing element according to some embodiments of the present invention for solving the above problem includes a weight register for receiving and storing a weight, an input activation register for storing input activation, an operation mode signal, and a size of the weight and the input activation. Accordingly, a flexible multiplier that generates result data by multiplying the weight and the input activation by either a first precision or a second precision smaller than the first precision, and a saturating function that generates a partial sum using the result data. Includes adder.

In some embodiments, the flexible multiplier includes error detection logic that checks whether overflow or underflow occurs according to a multiplication operation of the weight and the input activation and generates a detection result, k first precision A first multiplier, 2k second multipliers of the second precision, and a pass selection logic that selects one of the first multiplier and the second multiplier according to the size of the weight and the input activation. It can be included.

In some embodiments, the pass selection logic is configured to: when the operation mode signal is associated with the first precision, based on whether at least one of the weight and the input activation is greater than a maximum value of the second precision, the first Any one of the multiplier and the second multiplier can be selected.

In some embodiments, the error detection logic generates a first result when an overflow or underflow occurs in the multiplication operation of the weight and the input activation, and the error detection logic generates a first result when an overflow or underflow occurs in the multiplication operation of the weight and the input activation. If is not generated, a second result is generated, and if each of the weight and the input activation is less than the maximum value of the second precision, the pass selection logic generates the first multi If a plier is selected and the detection result is the second result, the second multiplier can be selected.

In some embodiments, the pass selection logic may select one of the first multiplier and the second multiplier according to the detection result when the operation mode signal is related to the second precision.

In some embodiments, the error detection logic generates a first result when an overflow or underflow occurs in the multiplication operation of the weight and the input activation, and the error detection logic generates a first result when an overflow or underflow occurs in the multiplication operation of the weight and the input activation. If is not generated, a second result is generated, and the pass selection logic may select the first multiplier if the detection result is the first result, and select the second multiplier if the detection result is the second result. there is.

A neural processing device according to some embodiments of the present invention for solving the above problems includes at least one neural core, wherein the neural core includes a processing unit that performs an operation and stores input/output data of the processing unit. and an L0 memory, wherein the processing unit includes a PE array including at least one processing element, wherein the PE array receives a weight and an input activation and, depending on the size of the weight and the input activation, performs a first It includes a flexible multiplier that generates result data by performing a multiplication operation with 1 precision or a second precision that is smaller than the first precision, and a saturating adder that receives the result data and generates a partial sum.

In some embodiments, the flexible multiplier performs a multiplication operation on the weight and the input activation by the first precision when the size of at least one of the weight and the input activation is greater than the maximum value of the second precision, and If the size of each of the weight and the input activation is less than or equal to the maximum value of the second precision, the weight and the input activation can be multiplied by the second precision.

In some embodiments, the weights include first and second sub-weights, the input activation includes first and second sub-input activations, and the flexible multiplier includes the first sub-weights and the first sub-weights. Depending on the sub-input activation, the first sub-weight and the first sub-input activation are multiplied by the first precision or the second precision, and depending on the size of the second sub-weight and the second sub-input activation, The second sub weight and the second sub input activation may be multiplied by the first precision or the second precision.

In some embodiments, the weights include first and second sub-weights, the input activation includes first and second sub-input activations, and the flexible multiplier includes: the first sub-weight, the second Depending on the respective sizes of the sub-weight, the first sub-input activation, and the second sub-input activation, the weight and the input activation may be multiplied by the first precision or the second precision.

The processing element of the present invention and the neural processing device including it can reduce power consumption by performing an operation with appropriate precision according to the size of data.

Additionally, when overflow or underflow occurs, precision conversion is performed to increase precision.

In addition to the above-described content, specific effects of the present invention are described below while explaining specific details for carrying out the invention.

1 is a block diagram for explaining a neural processing system according to some embodiments of the present invention.
FIG. 2 is a block diagram for explaining the neural processing device of FIG. 1 in detail.
FIG. 3 is a block diagram for explaining the neural core SoC of FIG. 2 in detail.
FIG. 4 is a structural diagram for explaining the global interconnection of FIG. 3 in detail.
FIG. 5 is a block diagram for explaining the neural processor of FIG. 3 in detail.
FIG. 6 is a diagram illustrating the hierarchical structure of a neural processing device according to some embodiments of the present invention.
FIG. 7 is a block diagram for explaining the neural core of FIG. 5 in detail.
FIG. 8 is a block diagram for explaining the LSU of FIG. 7 in detail.
FIG. 9 is a block diagram for explaining the processing unit of FIG. 7 in detail.
FIG. 10 is a block diagram for explaining the processing element of FIG. 9 in detail.
FIG. 11 is a block diagram for explaining the flexible multiplier of FIG. 10 in detail.
Figure 12 is a diagram for explaining the structure and operation of a flexible multiplier according to some embodiments of the present invention.
Figure 13 is a diagram for explaining the structure and operation of a path determination unit according to some embodiments of the present invention.
Figure 14 is a diagram for explaining the configuration of pass selection logic according to some embodiments of the present invention.
Figure 15 is a diagram for explaining the structure and operation of a flexible multiplier according to some other embodiments of the present invention.
Figure 16 is a diagram for explaining the configuration of pass selection logic according to some other embodiments of the present invention.
Figure 17 is a diagram for explaining the configuration of a flexible multiplier according to another embodiment of the present invention.
Figure 18 is a diagram for explaining the configuration of a flexible multiplier according to another embodiment of the present invention.
FIG. 19 is a block diagram for explaining the L0 memory of FIG. 7 in detail.
FIG. 20 is a block diagram for explaining the local memory bank of FIG. 19 in detail.
Figure 21 is a block diagram to explain in detail the structure of a neural processing device according to some embodiments of the present invention.
Figure 22 is a block diagram for explaining memory reorganization of a neural processing system according to some embodiments of the present invention.
Figure 23 is a block diagram showing an example of memory reorganization of a neural processing system according to some embodiments of the present invention.
Figure 24 is an enlarged block diagram of part A of Figure 22.
FIG. 25 is a diagram for explaining the first memory bank of FIG. 24 in detail.
Figure 26 is a block diagram to explain the software hierarchy of a neural processing device according to some embodiments of the present invention.
Figure 27 is a conceptual diagram to explain a deep learning operation performed by a neural processing device according to some embodiments of the present invention.
FIG. 28 is a conceptual diagram illustrating learning and inference operations of a neural network of a neural processing device according to some embodiments of the present invention.
Figure 29 is a diagram for explaining an operation method of a neural processing device according to some embodiments of the present invention.
Figure 30 is a diagram for explaining a calculation method of a neural processing device according to some other embodiments of the present invention.

Terms or words used in this specification and patent claims should not be construed as limited to their general or dictionary meaning. According to the principle that the inventor can define terms or word concepts in order to explain his or her invention in the best way, it should be interpreted with a meaning and concept consistent with the technical idea of the present invention. In addition, the embodiments described in this specification and the configurations shown in the drawings are only one embodiment of the present invention and do not completely represent the technical idea of the present invention, so they cannot be replaced at the time of filing the present application. It should be understood that there may be various equivalents, variations, and applicable examples.

Terms such as first, second, A, and B used in the present specification and claims may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, a first component may be named a second component, and similarly, the second component may also be named a first component without departing from the scope of the present invention. The term 'and/or' includes any of a plurality of related stated items or a combination of a plurality of related stated items.

The terms used in the specification and claims are merely used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "include" or "have" should be understood as not precluding the existence or addition possibility of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification. .

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the technical field to which the present invention pertains.

Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and unless clearly defined in the present application, should not be interpreted in an ideal or excessively formal sense. No.

Additionally, each configuration, process, process, or method included in each embodiment of the present invention may be shared within the scope of not being technically contradictory to each other.

Hereinafter, a neural processing device according to some embodiments of the present invention will be described with reference to FIGS. 1 to 28.

1 is a block diagram for explaining a neural processing system according to some embodiments of the present invention.

Referring to FIG. 1, a neural processing system (NPS) according to some embodiments of the present invention may include a first neural processing device 1, a second neural processing device 2, and an external interface 3. .

The first neural processing device 1 may be a device that performs calculations using an artificial neural network. For example, the first neural processing device 1 may be a device specialized for performing deep learning calculation tasks. However, this embodiment is not limited to this.

The second neural processing device 2 may be a device that has the same or similar configuration as the first neural processing device 1. The first neural processing device 1 and the second neural processing device 2 may be connected to each other through an external interface 3 and share data and control signals.

Although FIG. 1 shows two neural processing devices, the neural processing system (NPS) according to some embodiments of the present invention is not limited thereto. That is, in the neural processing system (NPS) according to some embodiments of the present invention, three or more neural processing devices may be connected to each other through the external interface 3. Also, conversely, a neural processing system (NPS) according to some embodiments of the present invention may include only one neural processing device.

FIG. 2 is a block diagram for explaining the neural processing device of FIG. 1 in detail.

Referring to FIG. 2, the first neural processing device 1 includes a neural core SoC 10, a CPU 20, an off-chip memory 30, a first non-volatile memory interface 40, and a first volatile memory interface ( 50), a second non-volatile memory interface 60, and a second volatile memory interface 70.

The Neural Core SoC 10 may be a System on Chip device. The neural core SoC (10) is an artificial intelligence calculation unit and may be an accelerator. The neural core SoC 10 may be, for example, one of a graphics processing unit (GPU), a field programmable gate array (FPGA), and an application-specific integrated circuit (ASIC). However, this embodiment is not limited to this.

The neural core SoC (10) can exchange data with other external computational units through the external interface (3). Additionally, the neural core SoC 10 may be connected to the non-volatile memory 31 and the volatile memory 32 through the first non-volatile memory interface 40 and the first volatile memory interface 50, respectively.

The CPU 20 may be a control device that controls the system of the first neural processing device 1 and executes program operations. The CPU 20 is a general-purpose calculation unit and may be inefficient in performing parallel simple calculations commonly used in deep learning. Accordingly, the neural core SoC 10 can achieve high efficiency by performing calculations on deep learning inference and learning tasks.

The CPU 20 can exchange data with other external computational units through the external interface 3. Additionally, the CPU 20 may be connected to the non-volatile memory 31 and the volatile memory 32 through the second non-volatile memory interface 60 and the second volatile memory interface 70, respectively.

The off-chip memory 30 may be memory placed outside the chip of the neural core SoC 10. Off-chip memory 30 may include non-volatile memory 31 and volatile memory 32.

The non-volatile memory 31 may be a memory that continues to retain stored information even when power is not supplied. The non-volatile memory 31 includes, for example, Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Alterable ROM (EAROM), Erasable Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Memory (EEPROM). Read-Only Memory (e.g., NAND Flash memory, NOR Flash memory), UVEPROM (Ultra-Violet Erasable Programmable Read-Only Memory), FeRAM (Ferroelectric Random Access Memory), Magnetoresistive Random Access Memory (MRAM), Phase-change Random Access Memory (PRAM), silicon-oxide-nitride-oxide-silicon (SONOS), Resistive Random Access Memory (RRAM), Nanotube Random Access Memory (NRAM), magnetic computer memory It may include at least one of a device (eg, hard disk, diskette drive, magnetic tape), optical disk drive, and 3D XPoint memory. However, this embodiment is not limited to this.

Unlike the non-volatile memory 31, the volatile memory 32 may be a memory that continuously requires power to maintain stored information. The volatile memory 32 may include, for example, at least one of Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Synchronous Dynamic Random Access Memory (SDRAM), and Double Data Rate SDRAM (DDR SDRAM). there is. However, this embodiment is not limited to this.

The first non-volatile memory interface 40 and the second non-volatile memory interface 60 are, for example, Parallel Advanced Technology Attachment (PATA), Small Computer System Interface (SCSI), Serial Attached SCSI (SAS), and SATA ( It may include at least one of Serial Advanced Technology Attachment) and PCIe (PCI Express). However, this embodiment is not limited to this.

The first volatile memory interface 50 and the second volatile memory interface 70 may be configured to perform, for example, Single Data Rate (SDR), Double Data Rate (DDR), Quad Data Rate (QDR), and eXtreme Data Rate (XDR), respectively. , Octal Data Rate). However, this embodiment is not limited to this.

FIG. 3 is a block diagram for explaining the neural core SoC of FIG. 2 in detail.

2 and 3, the neural core SoC 10 includes at least one neural processor 1000, a shared memory 2000, a direct memory access (DMA) 3000, a non-volatile memory controller 4000, and a volatile memory controller 4000. It may include a memory controller 5000 and a global interconnection 5000.

The neural processor 1000 may be a computational unit that directly performs computational tasks. When there are multiple neural processors 1000, computational tasks may be assigned to each neural processor 1000. Each neural processor 1000 may be connected to each other through a global interconnection 5000.

The shared memory 2000 may be memory shared by several neural processors 1000. The shared memory 2000 can store data of each neural processor 1000. Additionally, the shared memory 2000 can receive data from the off-chip memory 30, temporarily store it, and transmit it to each neural processor 1000. Conversely, the shared memory 2000 may receive data from the neural processor 1000, temporarily store it, and transfer it to the off-chip memory 30 of FIG. 2.

Shared memory 2000 may require relatively fast memory. Accordingly, the shared memory 2000 may include, for example, SRAM. However, this embodiment is not limited to this. That is, the shared memory 2000 may include DRAM.

The shared memory 2000 may be a memory corresponding to the SoC level, that is, level 3 (L3). Accordingly, the shared memory 2000 may be defined as L3 shared memory.

The DMA 3000 can directly control the movement of data without the neural processor 1000 having to control input and output of data. Accordingly, the DMA 3000 can control data movement between memories to minimize the number of interrupts of the neural processor 1000.

The DMA (3000) can control data movement between the shared memory (2000) and the off-chip memory (30). The non-volatile memory controller 4000 and volatile memory controller 5000 can move data through the authority of the DMA (3000).

The non-volatile memory controller 4000 can control read or write operations on the non-volatile memory 31. The non-volatile memory controller 4000 can control the non-volatile memory 31 through the first non-volatile memory interface 40.

The volatile memory controller 5000 can control read or write operations on the volatile memory 32. Additionally, the volatile memory controller 5000 may perform a refresh operation on the volatile memory 32. The volatile memory controller 5000 can control the non-volatile memory 31 through the first volatile memory interface 50.

The global interconnection 5000 may connect at least one neural processor 1000, a shared memory 2000, a DMA 3000, a non-volatile memory controller 4000, and a volatile memory controller 5000. Additionally, the external interface 3 may also be connected to the global interconnection 5000. The global interconnection 5000 provides data between at least one neural processor 1000, shared memory 2000, DMA 3000, non-volatile memory controller 4000, volatile memory controller 5000, and external interface 3. It may be a moving path.

The global interconnection 5000 can transmit signals for transmission and synchronization of control signals as well as data. That is, in the neural processing device according to some embodiments of the present invention, each neural processor 1000 can directly transmit and receive a synchronization signal, rather than having a separate control processor manage the synchronization signal. Accordingly, the latency of the synchronization signal generated by the control processor can be blocked.

That is, when there are multiple neural processors 1000, there may be a dependency of individual tasks in which the task of one neural processor 1000 must be completed before the next neural processor 1000 can start a new task. The end and start of these individual tasks can be confirmed through a synchronization signal, and in the existing technology, the control processor receives the synchronization signal and instructs the start of a new task.

However, as the number of neural processors 1000 increases and the dependency of the task becomes more complex, the number of requests and instructions for this synchronization task increases exponentially. Therefore, latency according to each request and instruction can significantly reduce work efficiency.

Accordingly, in the neural processing device according to some embodiments of the present invention, each neural processor 1000 may directly transmit a synchronization signal to another neural processor 1000 according to the dependency of the task, instead of a control processor. In this case, compared to the method managed by the control processor, multiple neural processors 1000 can perform synchronization tasks in parallel, thereby minimizing latency due to synchronization.

In addition, the control processor must perform task scheduling of the neural processors 1000 according to task dependency, and the overhead of such scheduling may increase significantly as the number of neural processors 1000 increases. Accordingly, in the neural processing device according to some embodiments of the present invention, the scheduling task is performed by the individual neural processor 1000 and there is no scheduling burden, so the performance of the device can be improved.

FIG. 4 is a structural diagram for explaining the global interconnection of FIG. 3 in detail.

Referring to FIG. 4, the global interconnection 5000 may include a data channel 5100, a control channel 5200, and an L3 sync channel 5300.

The data channel 5100 may be a dedicated channel for transmitting data. Through the data channel 5100, at least one neural processor 1000, shared memory 2000, DMA 3000, non-volatile memory controller 4000, volatile memory controller 5000, and external interface 3 exchange data with each other. can be exchanged.

The control channel 5200 may be a dedicated channel that transmits control signals. At least one neural processor 1000, shared memory 2000, DMA 3000, non-volatile memory controller 4000, volatile memory controller 5000, and external interface 3 control each other through the control channel 5200. Signals can be exchanged.

The L3 sync channel 5300 may be a dedicated channel that transmits a synchronization signal. At least one neural processor 1000, shared memory 2000, DMA 3000, non-volatile memory controller 4000, volatile memory controller 5000, and external interface 3 are connected to each other through the L3 sync channel 5300. Synchronization signals can be exchanged.

The L3 sync channel 5300 is set as a dedicated channel within the global interconnection 5000 and can quickly transmit synchronization signals without overlapping with other channels. Accordingly, the neural processing device according to some embodiments of the present invention does not require a new wiring operation and can smoothly perform a synchronization operation using the existing global interconnection 5000.

FIG. 5 is a block diagram for explaining the neural processor of FIG. 3 in detail.

Referring to FIGS. 3 to 5 , the neural processor 1000 may include at least one neural core 100, an L2 shared memory 400, a local interconnection 200, and an L2 sync path 300.

At least one neural core 100 may share and perform the tasks of the neural processor 1000. For example, there may be eight neural cores 100. However, this embodiment is not limited to this. 3 and 5 show that several neural cores 100 are included in the neural processor 1000, but this embodiment is not limited thereto. In other words, the neural processor 1000 can be configured with only one neural core 100.

The L2 shared memory 400 may be a memory shared by each neural core 100 within the neural processor 1000. The L2 shared memory 400 can store data of each neural core 100. Additionally, the L2 shared memory 400 can receive data from the shared memory 2000 of FIG. 4, temporarily store it, and transmit it to each neural core 100. Conversely, the L2 shared memory 400 may receive data from the neural core 100, temporarily store it, and transfer it to the shared memory 2000 of FIG. 3.

The L2 shared memory 400 may be a memory corresponding to the neural processor level, that is, level 2 (L2). L3 shared memory, that is, shared memory 2000, may be shared by the neural processor 1000, and L2 shared memory 400 may be shared by the neural core 100.

The local interconnection 200 may connect at least one neural core 100 and the L2 shared memory 400 to each other. The local interconnection 200 may be a path through which data moves between at least one neural core 100 and the L2 shared memory 400. The local interconnection 200 is connected to the global interconnection 5000 of FIG. 3 and can transmit data.

The L2 sync path 300 may connect at least one neural core 100 and the L2 shared memory 400 to each other. The L2 sync path 300 may be a path along which synchronization signals of at least one neural core 100 and the L2 shared memory 400 travel.

The L2 sync path 300 may be formed physically separately from the local interconnection 200. In the case of the local interconnection 200, unlike the global interconnection 5000, sufficient channels may not be formed internally. In this case, the L2 sync path 300 is formed separately so that transmission of the synchronization signal can be performed quickly and without delay. The L2 sync path 300 can be used for synchronization performed at a level one level lower than the L3 sync channel 5300 of the global interconnection 5000.

FIG. 6 is a diagram illustrating the hierarchical structure of a neural processing device according to some embodiments of the present invention.

Referring to FIG. 6, the neural core SoC 10 may include at least one neural processor 1000. Each neural processor 1000 can transmit data to each other through the global interconnection 5000.

Each neural processor 1000 may include at least one neural core 100. The neural core 100 may be a processing unit optimized for deep learning calculation tasks. The neural core 100 may be a processing unit corresponding to one operation of a deep learning calculation task. In other words, deep learning computational tasks can be expressed as a sequential or parallel combination of multiple operations. The neural core 100 is a processing unit that can each process one operation, and may be the minimum computational unit that can be considered for scheduling from a compiler's perspective.

The neural processing device according to this embodiment can achieve fast and efficient scheduling and performance of calculation tasks by configuring the scale of the minimum calculation unit and hardware processing unit considered from a compiler scheduling perspective to be the same.

In other words, if the processing unit that can be divided into hardware is too large compared to the computational task, inefficiency in the computational task may occur when driving the processing unit. Conversely, it is not appropriate to always schedule a processing unit smaller than the operation, which is the compiler's minimum scheduling unit, because scheduling inefficiencies may occur and hardware design costs may increase.

Therefore, in this embodiment, the scale of the compiler's scheduling unit and the hardware processing unit can be similarly adjusted to simultaneously satisfy fast computational task scheduling and efficient computational task performance without wasting hardware resources.

FIG. 7 is a block diagram for explaining the neural core of FIG. 5 in detail.

Referring to FIG. 7, the neural core 100 includes a Load/Store Unit (LSU) 110, an L0 memory 120, a weight buffer 130, an activation LSU 140, an activation buffer 150, and a processing unit ( 160) may be included.

The LSU 110 may receive at least one of data, control signals, and synchronization signals from the outside through the local interconnection 200 and the L2 sync path 300. The LSU 110 may transmit at least one of the received data, control signal, and synchronization signal to the L0 memory 120. Similarly, the LSU 110 may transmit at least one of data, a control signal, and a synchronization signal to the outside through the local interconnection 200 and the L2 sync path 300.

FIG. 8 is a block diagram for explaining the LSU of FIG. 7 in detail.

Referring to FIG. 8, the LSU 110 includes a local memory load unit 111a, a local memory store unit 111b, a neural core load unit 112a, a neural core store unit 112b, a load buffer (LB), and a store. It may include a buffer (SB), a load engine 113a, a store engine 113b, and a conversion index buffer 114.

The local memory load unit 111a may fetch a load instruction for the L0 memory 120 and issue the load instruction. When the local memory load unit 111a provides an issue load instruction to the load buffer LB, a memory access request can be sequentially transmitted to the load engine 113a according to the order in which the load buffer LB is input.

Additionally, the local memory store unit 111b may fetch a store instruction for the L0 memory 120 and issue the store instruction. When the local memory store unit 111b provides the issue store instruction to the store buffer SB, a memory access request can be sequentially transmitted to the store engine 113b according to the order in which the store buffer SB was input.

The neural core load unit 112a may fetch a load instruction for the neural core 100 and issue the load instruction. When the neural core load unit 112a provides an issue load instruction to the load buffer LB, a memory access request can be sequentially transmitted to the load engine 113a according to the order in which the load buffer LB is input.

Additionally, the neural core store unit 112b may fetch a store instruction for the neural core 100 and issue the store instruction. When the neural core store unit 112b provides the issue store instruction to the store buffer SB, a memory access request can be sequentially transmitted to the store engine 113b according to the order in which the store buffer SB was input.

The load engine 113a may receive a memory access request and load data through the local interconnection 200. At this time, the load engine 113a can quickly find data using the translation table of recently used virtual addresses and physical addresses in the translation index buffer 114. If the virtual address of the load engine 113a is not in the translation index buffer 114, address translation information can be found in another memory.

The store engine 113b may receive a memory access request and load data through the local interconnection 200. At this time, the store engine 113b can quickly find data using the translation table of recently used virtual addresses and physical addresses in the translation index buffer 114. If the virtual address of the store engine 113b is not in the translation index buffer 114, the address translation information can be found in another memory.

The load engine 113a and the store engine 113b may send a synchronization signal to the L2 sync path 300. At this time, the synchronization signal may mean that the task has ended.

Referring again to FIG. 7, the L0 memory 120 is a memory located inside the neural core 100, and allows the neural core 100 to receive all input data required for work from the outside and temporarily store them. Additionally, the L0 memory 120 can temporarily store output data calculated by the neural core 100 in order to transmit it to the outside. The L0 memory 120 may serve as a cache memory of the neural core 100.

The L0 memory 120 may transmit input activation (Act_In) to the activation buffer 150 and receive output activation (Act_Out) by the activation LSU 140. In addition to the activation LSU 140, the L0 memory 120 can directly transmit and receive data with the processing unit 160. That is, the L0 memory 120 can exchange data with each of the PE array 163 and the vector unit 164.

The L0 memory 120 may be a memory corresponding to the neural core level, that is, level 1 (L1). Accordingly, the L0 memory 120 may be defined as the L1 memory. However, unlike the L2 shared memory 400 and the L3 shared memory, that is, the shared memory 2000, the L1 memory is not shared and may be a private memory of the neural core.

The L0 memory 120 can transmit data such as activation or weight through a data path. The L0 memory 120 can send and receive synchronization signals through the L1 Sync Path, which is a separate dedicated path. For example, the L0 memory 120 may exchange a synchronization signal with the LSU 110, the weight buffer 130, the activation LSU 140, and the processing unit 160 through an L1 Sync Path. .

The weight buffer 130 may receive weight from the L0 memory 120. The weight buffer 130 may transmit weight to the processing unit 160. The weight buffer 130 may temporarily store the weight before transmitting the weight.

Input activation (Act_In) and output activation (Act_Out) may refer to the input and output values of the layer of the neural network network. At this time, when the neural network has multiple layers, the output value of the previous layer becomes the input value of the next layer, so the output activation (Act_Out) of the previous layer can be used as the input activation (Act_In) of the next layer.

Weight may refer to a parameter multiplied by the input activation (Act_In) input from each layer. Weight is adjusted and confirmed in the deep learning learning stage, and can be used to derive output activation (Act_Out) through a fixed value in the inference stage.

The activation LSU 140 may transfer input activation (Act_In) from the L0 memory 120 to the activation buffer 150 and transfer the output activation (Act_Out) from the activation buffer 150 to the L0 memory 120. That is, the activation LSU 140 can perform both activation load and store operations.

The activation buffer 150 may provide input activation (Act_In) to the processing unit 160 and receive output activation (Act_Out) from the processing unit 160. The activation buffer 150 can temporarily store input activation (Act_In) and output activation (Act_Out).

The activation buffer 150 can quickly provide activation to the processing unit 160 with a large computational load, especially the PE array 163, and quickly receive activation to increase the computation speed of the neural core 100.

The processing unit 160 may be a module that performs calculations. The processing unit 160 can perform not only one-dimensional operations but also two-dimensional matrix operations, that is, convolution operations. The processing unit 160 may receive the input activation (Act_In), multiply it by the weight, and add it to generate the output activation (Act_Out).

FIG. 9 is a block diagram for explaining the processing unit of FIG. 7 in detail.

Referring to FIGS. 7 and 9 , the processing unit 160 may include a PE array 163, a vector unit 164, a column register 161, and a row register 162.

The PE array 163 can perform multiplication by receiving input activation (Act_In) and weight (Weight). At this time, input activation (Act_In) and weight (Weight) can each be calculated through convolution in matrix form. Through this, the PE array 163 can generate output activation (Act_Out). However, this embodiment is not limited to this. The PE array 163 can generate any number of types of output other than output activation (Act_Out).

The PE array 163 may include at least one processing element 163_1. The processing elements 163_1 are aligned with each other and can perform multiplication of one input activation (Act_In) and one weight (Weight), respectively.

The PE array 163 can generate a subtotal that sums the values for each multiplication. These partial sums can be used as output activation (Act_Out). Since the PE array 163 performs two-dimensional matrix multiplication, it may also be referred to as a two-dimensional matrix compute unit.

The vector unit 164 can perform one-dimensional operations. The vector unit 164 can perform deep learning calculations together with the PE array 163. Through this, the processing unit 160 can be specialized for necessary operations. In other words, the neural core 100 has calculation modules that perform large amounts of two-dimensional matrix multiplication and one-dimensional calculations, so it can efficiently perform deep learning tasks.

The column register 161 may receive the first input (I1). The column register 161 may receive the first input I1, divide it, and provide it to each column of the processing element 163_1.

Low register 162 may receive a second input (I2). The row register 162 may receive the second input I2, divide it, and provide it to each row of the processing element 163_1.

The first input (I1) may be input activation (Act_In) or weight (Weight). The second input (I2) may be a value other than the first input (I1) among input activation (Act_In) or weight (Weight). Alternatively, the first input (I1) and the second input (I2) may be values other than input activation (Act_In) and weight (Weight).

FIG. 10 is a block diagram for explaining the processing element of FIG. 9 in detail.

Referring to FIG. 10, the processing element 163_1 may include a weight register (WR), an input activation register (ACR), a flexible multiplier (FM), and a saturating adder (SA).

The weight register WR can receive and store the weight input to the processing element 163_1. The weight register (WR) can transmit weight to the flexible multiplier (FM).

The input activation register (ACR) can receive and store input activation (Act_In). The input activation register (ACR) can transmit input activation (Act_In) to the flexible multiplier (FM).

The flexible multiplier (FM) can receive weight and input activation (Act_In). The flexible multiplier (FM) can perform multiplication of weight and input activation (Act_In). The flexible multiplier (FM) can receive a mode signal (Mode). At this time, the mode signal (Mode) may be a signal indicating which precision to use among the first precision and the second precision to perform the calculation.

The flexible multiplier (FM) can output the multiplication result as result data. The resulting data may include sine bits (SB) and product bits (PB). At this time, the sign bit (SB) may be a bit indicating the sign of the result data. The product bit (PB) may be a bit indicating the size of the result data. The flexible multiplier (FM) can output result data using the first or second precision.

The Saturating Adder (SA) can receive the resulting data. The saturating adder (SA) can receive the sine bit (SB) and product bit (PB). Saturating Adder (SA) can receive and accumulate result data multiple times. Accordingly, the saturating adder (SA) can generate a partial sum (Psum). These partial sums (Psum) may be output from each processing element 163_1 and finally added. However, this embodiment is not limited to this.

FIG. 11 is a block diagram for explaining the flexible multiplier of FIG. 10 in detail.

Referring to FIG. 11, the flexible multiplier (FM) may include a pass decision unit (PDU), a demultiplexer (Dx), a first multiplier (Mul1), a second multiplier (Mul2), and a multiplexer (Mx). .

The pass decision unit (PDU) can receive weight and input activation (Act_In). Additionally, the pass decision unit (PDU) may receive an operation mode signal (Mode). At this time, the operation mode signal (Mode) may be a signal indicating which precision to perform the multiplication operation among the first precision (Pr1) and the second precision (Pr2). At this time, the first precision (Pr1) may be larger than the second precision (Pr2). According to some embodiments, the first precision (Pr1) may be 2N bits (N is a natural number), and the second precision (Pr2) may be N bits. For example, the first precision (Pr1) may be INT4 (4-bit integer, 4-bit Integer), and the second precision (Pr2) may be INT2 (2-bit integer, 2-bit Integer). However, this is only an example, and the first precision (Pr1) and the second precision (Pr2) may be INT32, INT16, INT16, INT8, or INT8 and INT4, respectively.

The pass decision unit (PDU) may select a multiplier for which the weight and input activation (Act_In) will be calculated according to the operation mode signal (Mode), weight (Weight), and input activation (Act_In). In other words, the pass decision unit (PDU) operates the first multiplier (Mul1) and the second precision (Pr2) of the first precision (Pr1) according to the operation mode signal (Mode), weight (Weight), and input activation (Act_In). ) can generate a pass decision signal (PD) for selecting one of the second multipliers (Mul2). The path decision unit (PDU) may provide the generated path decision signal (PD), weight, and input activation (Act_In) to the demultiplexer (Dx).

According to some embodiments, the pass decision unit (PDU) may include bit division logic (BDL), pass selection logic (PSL), and conversion logic (CVL).

Bit division logic (BDL) can divide weight (Weight) and input activation (Act_In) into precision units corresponding to the operation mode signal (Mode). In other words, the bit division logic (BDL) divides the weight and input activation (Act_In) into one of the first precision (Pr1) and the second precision (Pr2) units according to the operation mode signal (Mode). can do. For example, if the operation mode signal (Mode) means INT4, the bit division logic (BDL) can divide weight (Weight) and input activation (Act_In) into 4-bit units each. The bit division logic (BDL) can provide divided weight (Weight) and input activation (Act_In) to the pass selection logic (PSL).

Pass selection logic (PSL) can receive divided weight (Weight) and input activation (Act_In) from bit division logic (BDL). The pass selection logic (PSL) selects the divided weight (Weight) and input activation (Act_In) and the multiplier on which the divided weight (Weight) and input activation (Act_In) will be calculated according to the operation mode signal (Mode). A pass decision signal (PD) can be generated for.

According to some embodiments, the first multiplier (Mul1) performs a multiplication operation using the input data of the first precision (Pr1), and the second multiplier (Mul2) performs a multiplication operation using the input data of the second precision (Pr2). You can perform multiplication operations using . In other words, the pass selection logic (PSL) considers the divided weight (Weight) and input activation (Act_In) and the operation mode signal (Mode), and determines the precision at which the divided weight (Weight) and input activation (Act_In) will be calculated. can be decided.

Conversion logic (CVL) can convert the divided weight (Weight) and input activation (Act_In) to another precision as needed. According to some embodiments, the conversion logic (CVL) passes the divided weight (Weight) and input activation (Act_In) when the precision associated with the operation mode signal (Mode) and the precision associated with the pass decision signal (PD) are different. It can be converted to precision associated with the decision signal (PD). When the conversion logic (CVL) converts the precision of the weight (Weight) and input activation (Act_In), the conversion logic (CVL) provides the converted weight (Weight) and input activation (Act_In) to the demultiplexer (Dx). can do.

Meanwhile, the conversion logic (CVL) will not convert the precision of the divided weight (Weight) and input activation (Act_In) when the precision associated with the operation mode signal (Mode) and the precision associated with the pass decision signal (PD) are the same. You can.

In summary, the pass decision unit (PDU) receives weight and input activation (Act_In), and converts the received weight and input activation (Act_In) into the first precision (Pr1) according to the operation mode signal (Mode). ) and the second precision (Pr2). The pass decision unit (PDU) uses a first multiplier (Mul1) and a second multiplier according to the weight and input activation (Act_In) divided into one of the first precision (Pr1) and the second precision (Pr2). A path decision signal (PD) for selecting one of (Mul2) can be generated. The pass decision unit (PDU) may provide weight and input activation (Act_In) to the demultiplexer (Dx). Additionally, the path decision unit (PDU) can select a multiplier on which the weight and input activation (Act_In) will be calculated by providing the path decision signal (PD) to the demultiplexer (Dx).

The demultiplexer (Dx) can receive weight and input activation (Act_In) from the pass decision unit (PDU). At this time, the weight and input activation (Act_In) received from the pass decision unit (PDU) may be divided into first precision (Pr1) or second precision (Pr2) units. Additionally, the demultiplexer (Dx) may receive the path decision signal (PD) from the path decision unit (PDU). The demultiplexer (Dx) divides the weight (Weight) and input activation (Act_In) divided into first precision (Pr1) or second precision (Pr2) units according to the pass decision signal (PD) into a plurality of first multipliers (Mul1) ) and a plurality of second multipliers (Mul2).

The first multiplier (Mul1) can perform a multiplication operation with the first precision (Pr1). That is, the first multiplier (Mul1) can perform a multiplication operation using the input data of the first precision (Pr1). For example, the first multiplier (Mul1) may receive the weight (Weight) and the input activation (Act_In) of the first precision (Pr1) and perform a multiplication operation on them. The number of first multipliers (Mul1) may be k (k is a natural number). For example, the flexible multiplier (FM) may include eight first multipliers (Mul1) of INT4, but embodiments are not limited thereto.

The second multiplier (Mul2) can perform a multiplication operation with the second precision (Pr2). That is, the second multiplier (Mul2) can receive a multiplication operation using the input data of the second precision (Pr2). For example, the second multiplier (Mul2) may receive the weight (Weight) and the input activation (Act_In) of the second precision (Pr2) and perform a multiplication operation on them. The number of second multipliers (Mul2) may be 2k. For example, the flexible multiplier (FM) may include 16 second multipliers (Mul2) of INT2, but embodiments are not limited thereto.

The multiplexer (Mx) may receive the result of the multiplication operation from either the first multiplier (Mul1) or the second multiplier (Mul2). In other words, the multiplexer (Mx) can receive the result of the multiplication operation of the weight of the first precision (Pr1) and the input activation (Act_In) of the first precision (Pr1) from the first multiplier (Mul1). Additionally, the multiplexer (Mx) may receive the result of a multiplication operation of the weight of the second precision (Pr2) and the input activation (Act_In) of the second precision (Pr2) from the second multiplier (Mul2).

The multiplexer (Mx) may generate result data by combining the multiplication operation results received from the first multiplier (Mul1) and the second multiplier (Mul2). The resulting data may include sine bits (SB) and product bits (PB). For convenience of explanation, the structure and operation of the flexible multiplier (FM) will be described in more detail by assuming that k is 2. However, when k is 2, it is simply a value selected for convenience of explanation, and the embodiments are not limited thereto.

Figure 12 is a diagram for explaining the structure and operation of a flexible multiplier according to some embodiments of the present invention.

Figure 12 shows a flexible multiplier when k is 2, that is, when the first multiplier (Mul1) of the first precision (Pr1) is 2 and the second multiplier (Mul2) of the second precision (Pr2) is 4. The structure of the flyer (FM) is shown. However, this is only for convenience of explanation, and k may change depending on the hardware design, such as 4 or 8.

Referring to FIGS. 11 and 12, the pass decision unit (PDU) may be provided with weight and input activation (Act_In). Additionally, the path decision unit (PDU) may receive an operation mode signal (Mode) indicating the first precision (Pr1) or the second precision (Pr2). The pass decision unit (PDU) determines the precision of the weight and input activation (Act_In) to be provided to the demultiplexer (Dx) according to the weight, input activation (Act_In), and operation mode signal (Mode), A pass decision signal (PD) can be generated to select a multiplier to calculate weight and input activation (Act_In). For convenience of explanation, the path for performing the multiplication operation using the first multiplier (Mul1) is defined as the first path (Path #1), and the path for performing the multiplication operation using the second multiplier (Mul2) is defined as the first path (Path #1). The path is defined as the second path (Path#2). In other words, the path decision unit (PDU) is one of the first path (Path#1) and the second path (Path#2) depending on the weight (Weight), input activation (Act_In), and operation mode signal (Mode). A path decision signal (PD) for selecting can be generated. The path determination unit (PDU) may provide the weight of the first precision (Pr1) or the second precision (Pr2), input activation (Act_In), and a path decision signal (PD) to the demultiplexer (Dx).

The demultiplexer (Dx) may include a weight demultiplexer (Dx_W) and an input activation demultiplexer (Dx_I). The weight demultiplexer (Dx_W) may receive the weight of the first precision (Pr1) or the second precision (Pr2) and the pass decision signal (PD). The weight demultiplexer (Dx_W) may provide the received weight (Weight) to one of the first multiplier (Mul1) and the second multiplier (Mul2) according to the path decision signal (PD).

Similarly, the input activation demultiplexer (Dx_I) may receive the input activation (Act_In) and the pass decision signal (PD) of the first precision (Pr1) or the second precision (Pr2). The input activation demultiplexer (Dx_I) may provide the received input activation (Act_In) to one of the first multiplier (Mul1) and the second multiplier (Mul2) according to the pass decision signal (PD).

For example, when the pass decision signal (PD) is the first signal, the weight demultiplexer (Dx_W) provides the weight (Weight) of the first precision (Pr1) to the first multiplier (Mul1), and the input activation demultiplexer ( Dx_I) may provide the input activation (Act_In) of the first precision (Pr1) to the first multiplier (Mul1). On the other hand, when the pass decision signal (PD) is the second signal, the weight demultiplexer (Dx_W) provides the weight of the second precision (Pr2) to the second multiplier (Mul2), and the input activation demultiplexer (Dx_I) Can provide the input activation (Act_In) of the second precision (Pr2) to the second multiplier (Mul2).

Refer to FIG. 13 for a more detailed description of the process of generating the pass decision signal PD.

Figure 13 is a diagram for explaining the structure and operation of a path determination unit according to some embodiments of the present invention.

Referring to FIG. 13, bit division logic (BDL) can receive weight and input activation (Act_In). The bit division logic (BDL) may divide the weight and input activation (Act_In) into one of the first precision (Pr1) and the second precision (Pr2) depending on the operation mode signal (Mode). For example, in bit division logic (BDL), when the operation mode signal (Mode) refers to the first precision (Pr1), the weight (Weight) and input activation (Act_In) are each used in the k number of first precisions (Pr1). It can be divided into Weight and Input Activation (Act_In). Similarly, when the operation mode signal (Mode) refers to the second precision (Pr2), the bit division logic (BDL) sets the weight and input activation (Act_In) to each of 2k second precisions (Pr2). It can be divided into Weight and Input Activation (Act_In). For convenience of explanation, the weight divided into the first precision (Pr1) or the second precision (Pr2) is divided into subweights (W0, W1, ...), and the first precision (Pr1) or the second precision (Pr2) is divided into subweights (W0, W1, ...) ) is defined as sub-input activation (IN0, IN1, …). That is, Weight may include k or 2k sub-weights (W0, W1, …), and input activation (Act_In) may include k or 2k sub-input activations (IN0, IN1, …). there is.

For example, assume that weight and input activation (Act_In) are each 8-bit data, the first precision (Pr1) is INT4, the second precision (Pr2) is INT2, and k is 2. When the operation mode signal (Mode) is INT4, the bit division logic (BDL) divides the 8-bit weight into INT4 units and produces the first sub-weight (W0) of INT4 and the second sub-weight (W1) of INT4. ) can be divided into Additionally, the bit division logic (BDL) can divide the 8-bit input activation (Act_In) into the first sub-input activation (IN0) of INT4 and the second sub-input activation (IN1) of INT4.

Meanwhile, when the operation mode signal (Mode) is INT2, the bit division logic (BDL) divides the 8-bit weight into INT2 units, and produces the first sub-weight (W0) of INT2 and the second sub-weight of INT2. (W1), it can be divided into the third sub-weight of INT2 and the fourth sub-weight of INT2. In addition, the bit division logic (BDL) connects the 8-bit input activation (Act_In) to the first sub-input activation (IN0) of INT2, the second sub-input activation (IN1) of INT2, the third sub-input activation of INT2, and the 1st sub-input activation (IN0) of INT2. It can be divided into fourth sub-input activation.

The pass selection logic (PSL) generates a pass decision signal ( PD) can be created.

First, the description will be made assuming that the operation mode signal (Mode) is related to the first precision (Pr1). The pass selection logic (PSL) can generate a pass decision signal (PD) according to the size of a plurality of sub weights (W0, W1, ...) and a plurality of sub input activations (IN0, IN1, ...). there is. For example, in the pass selection logic (PSL), multiple sub weights (W0, W1, ...) and multiple sub input activations (IN0, IN1, ...) exceed the maximum value of the second precision (Pr2). A pass decision signal (PD) can be generated based on whether or not the pass decision signal (PD) is performed.

If the size of at least one of the plurality of sub weights (W0, W1, ...) and the plurality of sub input activations (IN0, IN1, ...) exceeds the maximum value of the second precision (Pr2), pass The selection logic (PSL) may generate a first signal for selecting the first multiplier (Mul1) as the pass decision signal (PD). In other words, when the size of at least one of the plurality of sub weights (W0, W1, ...) and the plurality of sub input activations (IN0, IN1, ...) exceeds the maximum value of the second precision (Pr2), The pass selection logic (PSL) uses a pass decision signal ( PD) can generate the first signal.

On the other hand, if the size of each of the plurality of sub weights (W0, W1, ...) and the plurality of sub input activations (IN0, IN1, ...) does not exceed the maximum value of the second precision (Pr2), pass selection The logic (PSL) may generate a second signal for selecting the second multiplier (Mul2) as the pass decision signal (PD). In other words, if the size of each of the plurality of sub weights (W0, W1, ...) and the plurality of sub input activations (IN0, IN1, ...) does not exceed the maximum value of the second precision (Pr2), pass The selection logic (PSL) uses a pass decision signal (PD) so that a plurality of sub weights (W0, W1, ...) and a plurality of sub input activations (IN0, IN1, ...) are calculated as the second precision (Pr2). ) can generate the second signal.

The conversion logic (CVL) compares the operation mode signal (Mode) and the pass decision signal (PD), and operates multiple sub weights (W0, W1, ...) and multiple sub input activations (IN0, IN1, ...). ) can be converted to precision. Specifically, the conversion logic (CVL) uses a plurality of sub weights (W0, W1, ...) and a plurality of sub weights when the precision associated with the operation mode signal (Mode) and the precision associated with the pass decision signal (PD) are different. The precision of input activation (IN0, IN1, ...) can be converted from the precision associated with the operation mode signal (Mode) to the precision associated with the pass decision signal (PD). For example, when the operation mode signal (Mode) is associated with the first precision (Pr1) and the pass decision signal (PD) is the second signal, the conversion logic (CVL) includes a plurality of sub weights (W0, W1, . ..) and a plurality of sub-input activations (IN0, IN1, ...) can be converted from the first precision (Pr1) to the second precision (Pr2).

When the pass decision signal (PD) generated in the pass selection logic (PSL) is the first signal, the plurality of sub weights (W0, W1, ...) of the first precision (Pr1) and the Multiple sub-input activations (IN0, IN1, ...) can be provided by the demultiplexer (Dx).

Meanwhile, when the pass decision signal (PD) generated in the pass selection logic (PSL) is the second signal, a plurality of sub weights (W0', W1', ...) converted to the second precision (Pr2) and the second precision A plurality of sub-input activations (IN0', IN1', ...) converted to (Pr2) can be provided to the demultiplexer (Dx).

In other words, the demultiplexer (Dx) receives the first signal as the pass decision signal (PD), and receives a plurality of subweights (W0, W1, ...) of the first precision (Pr1) and the first signal (Pr1). Multiple sub-input activations (IN0, IN1, ...) can be provided. Alternatively, the demultiplexer (Dx) receives a second signal as the pass decision signal (PD), and receives a plurality of sub-weights (W0', W1', ...) converted to the second precision (Pr2) and the second precision (Pr2). You can receive multiple sub-input activations (IN0', IN1', ...) converted to .

According to the first signal, the demultiplexer (Dx) performs a plurality of sub weights (W0, W1, ...) of the first precision (Pr1) and a plurality of sub input activations (IN0, IN1, ...) of the first precision (Pr1). ..) can be provided to the first multiplier (Mul1). Alternatively, the demultiplexer (Dx) activates a plurality of sub weights (W0', W1', ...) converted to the second precision (Pr2) and a plurality of sub inputs converted to the second precision (Pr2) according to the second signal. (IN0', IN1', ...) can be provided to the second multiplier (Mul2).

The first multiplier (Mul1) uses a plurality of sub weights (W0, W1, ...) of the first precision (Pr1) and a plurality of sub input activations (IN0, IN1, ...) of the first precision (Pr1). A multiplication operation can be performed and provided to the multiplexer (Mx). Meanwhile, the second multiplier (Mul2) includes a plurality of sub weights (W0', W1', ...) converted to the second precision (Pr2) and a plurality of sub input activations (IN0') converted to the second precision (Pr2). , IN1', ...) can be multiplied and provided to the multiplexer (Mx).

Next, the description will be made assuming that the operation mode signal (Mode) is associated with the second precision (Pr2). The bit division logic (BDL) divides the weight and input activation (Act_In) into units of the second precision (Pr2) and divides them into a plurality of sub-weights (W0, W1, ...) of the second precision (Pr2). and a plurality of sub-input activations (IN0, IN1, ...) of the second precision (Pr2) can be generated.

The pass selection logic (PSL) may generate the second signal as the pass decision signal (PD). A plurality of sub weights (W0, W1, ...) of the second precision (Pr2) and a plurality of sub input activations (IN0, IN1, ...) of the second precision (Pr2) are provided by the demultiplexer (Dx). You can. In other words, the demultiplexer (Dx) receives the second signal as the pass decision signal (PD), and uses the plurality of subweights (W0, W1, ...) of the second precision (Pr2) and the second signal (Pr2). Multiple sub-input activations (IN0, IN1, ...) can be provided.

According to the second signal, the demultiplexer (Dx) performs a plurality of sub weights (W0, W1, ...) of the second precision (Pr2) and a plurality of sub input activations (IN0, IN1, ...) of the second precision (Pr2). ..) can be provided to the second multiplier (Mul2).

The second multiplier (Mul2) uses a plurality of sub weights (W0, W1, ...) of the second precision (Pr2) and a plurality of sub input activations (IN0, IN1, ...) of the second precision (Pr2). A multiplication operation can be performed and provided to the multiplexer (Mx). For an exemplary description of the pass selection logic (PSL), see further reference to FIG. 14.

Figure 14 is a diagram for explaining the configuration of pass selection logic according to some embodiments of the present invention.

For example, the description assumes that the first precision (Pr1) is INT4, the second precision (Pr2) is INT2, the weight is '00100001', and the input activation (Act_In) is '10010001'.

13 and 14, when the operation mode signal (Mode) is INT4, the bit division logic (BDL) refers to the operation mode signal (Mode), divides the weight into INT4 units, and divides the weight into INT4 units. A sub weight (W0) and a second sub weight (W1) can be created. At this time, the first sub-weight (W0) may be '0010' and the second sub-weight (W1) may be '0001'. Additionally, the bit division logic (BDL) may divide the input activation (Act_In) into a first sub-input activation (IN0) and a second sub-input activation (IN1). At this time, the first sub-input activation (IN0) may be '1001', and the second sub-input activation (IN1) may be '0001'.

The pass selection logic (PSL) generates a pass decision signal (PD) according to the sizes of the first sub weight (W0), the second sub weight (W1), the first sub input activation (IN0), and the second sub input activation (IN1). ) can be determined as either the first signal or the second signal. For example, among the first sub weight (W0), second sub weight (W1), first sub input activation (IN0), and second sub input activation (IN1) of INT4, the highest level of the first sub input activation (IN0) Since the bit is 1 (larger than the maximum value of INT2), the pass selection logic (PSL) can output 0 (the first signal) as the pass decision signal (PD).

The first sub weight (W0), second sub weight (W1), first sub input activation (IN0), and second sub input activation (IN1) of INT4 may be provided to the demultiplexer (Dx).

The weight demultiplexer (Dx_W) provides the first sub-weight (W0) and the second sub-weight (W1) of INT4 to the first multiplier (Mul1) according to the pass decision signal (PD), and the input activation demultiplexer (Dx_I) may provide the first sub-input activation (IN0) and the second sub-input activation (IN1) of INT4 to the first multiplier (Mul1) according to the pass decision signal (PD).

For another example, it is assumed that the first precision (Pr1) is INT4, the second precision (Pr2) is INT2, the weight is '00100001', and the input activation (Act_In) is '00010001'. .

13 and 14, when the operation mode signal (Mode) is INT4, the bit division logic (BDL) refers to the operation mode signal (Mode), divides the weight into INT4 units, and divides the weight into INT4 units. A sub weight (W0) and a second sub weight (W1) can be created. At this time, the first sub-weight (W0) may be '0010' and the second sub-weight (W1) may be '0001'. Additionally, the bit division logic (BDL) may divide the input activation (Act_In) into INT4 units to generate the first sub-input activation (IN0) and the second sub-input activation (IN1). At this time, the first sub-input activation (IN0) may be '0001', and the second sub-input activation (IN1) may be '0001'.

The pass selection logic (PSL) generates a pass decision signal (PD) according to the sizes of the first sub weight (W0), the second sub weight (W1), the first sub input activation (IN0), and the second sub input activation (IN1). ) can be determined as either the first signal or the second signal. For example, the upper two bits of the first sub weight (W0), second sub weight (W1), first sub input activation (IN0), and second sub input activation (IN1) of INT4 are all 0 (INT2 (is less than or equal to the maximum value of ), the pass selection logic (PSL) can output 1 (second signal) as the pass decision signal (PD).

Therefore, the conversion logic (CVL) converts the first sub-weight (W0), the second sub-weight (W1), the first sub-input activation (IN0), and the second sub-input activation (IN1), which are INT4, to INT2, and INT2 The converted first sub-weight (W0'), second sub-weight (W1'), first sub-input activation (IN0'), and second sub-input activation (IN1') can be provided to the demultiplexer (Dx). .

The weight demultiplexer (Dx_W) provides the first sub-weight (W0') and the second sub-weight (W1') converted to INT2 to the second multiplier (Mul2) according to the pass decision signal (PD), and input activation The demultiplexer (Dx_I) may provide the first sub-input activation (IN0') and the second sub-input activation (IN1') converted to INT2 to the second multiplier (Mul2) according to the pass decision signal (PD). .

In other words, the pass selection logic (PSL) includes the first sub weight (W0), second sub weight (W1), first sub input activation (IN0), and second sub input activation (IN1) of the first precision (Pr1). If at least one of them is greater than the maximum value of the second precision Pr2, the first signal associated with the first path Path#1 may be output as the path decision signal PD. In addition, the pass selection logic (PSL) operates on the first sub weight (W0), second sub weight (W1), first sub input activation (IN0), and second sub input activation (IN1) of the first precision (Pr1), respectively. If the second precision (Pr2) is less than or equal to the maximum value, a second signal associated with the second path (Path#2) may be output as the path decision signal (PD). However, Figure 14 is only an example of implementing the pass selection logic (PSL) configured for convenience of explanation, and the embodiments are not limited to this configuration.

The first multiplier (Mul1) is a multiplication operator that performs a multiplication operation with the first precision (Pr1), and the second multiplier (Mul2) is a multiplication operator that performs a multiplication operation with the second precision (Pr2). Since the first precision (Pr1) is greater than the second precision (Pr2), the power required when the first multiplier (Mul1) performs the operation is greater than the power required when the second multiplier (Mul2) performs the operation. .

Meanwhile, the flexible multiplier (FM) according to some embodiments uses a plurality of sub weights (W0, W1, ...) and a plurality of sub inputs even when the operation mode signal (Mode) is associated with the first precision (Pr1). Depending on the size of the activation (IN0, IN1, ...), the operation can be performed with the second precision (Pr2). Therefore, the flexible multiplier (FM) according to some embodiments may in some cases use the calculator of the second precision (Pr2) rather than the calculator of the first precision (Pr1), thereby reducing power consumption and reducing costs. There is an advantage to having it.

According to some embodiments, the pass selection logic (PSL) includes the first sub weight (W0), the second sub weight (W1), the first sub input activation (IN0), and the second sub input activation of the first precision (Pr1). If all (IN1) are less than or equal to the maximum value of the second precision (Pr2), this can be calculated as the second precision (Pr2).

However, according to some other embodiments, the pass selection logic (PSL) can generate the pass decision signal (PD) with only a pair of sub weight and sub input activation. In other words, the pass selection logic (PSL) generates a first pass decision signal (PD1) for the first sub weight (W0) and the first sub input activation (IN0), and the second sub weight (W1) and the second A second pass decision signal (PD2) for sub-input activation (IN1) may be generated. At this time, the first path decision signal PD1 and the second path decision signal PD2 may be generated independently. For additional explanation, please refer further to FIGS. 15 and 16.

Figure 15 is a diagram for explaining the structure and operation of a flexible multiplier according to some other embodiments of the present invention. Figure 16 is a diagram for explaining the configuration of pass selection logic according to some other embodiments of the present invention. For convenience of explanation, content that is the same or similar to the content described above is omitted or briefly explained.

FIG. 15 shows, similar to the case of FIG. 12, when k is 2, that is, the first multiplier (Mul1) of the first precision (Pr1) is two, and the second multiplier (Mul2) of the second precision (Pr2) is 2. ) shows the structure of a flexible multiplier (FM) when there are 4.

Referring to FIGS. 15 and 16 , the weight demultiplexer (Dx_W) may include a first weight demultiplexer (Demux_W1) and a second weight demultiplexer (Demux_W2). Additionally, the input activation demultiplexer (Dx_I) may include a first input activation demultiplexer (Demux_I1) and a second input activation demultiplexer (Demux_I2).

The first weight demultiplexer (Demux_W1) may be provided with the first pass decision signal (PD1) and the first sub-weight (W0) of the first precision (Pr1) or the second precision (Pr2). Additionally, the second weight demultiplexer (Demux_W2) may be provided with the second pass decision signal (PD2) and the first precision (Pr1) or the second sub-weight (W1) of the second precision (Pr2).

Similarly, the first input activation demultiplexer (Demux_I1) may be provided with the first pass decision signal (PD1) and the first sub-input activation (IN0) of the first precision (Pr1) or the second precision (Pr2). . Additionally, the second input activation demultiplexer (Demux_I2) may be provided with a second pass decision signal (PD2) and a second sub-input activation (IN1) of the first precision (Pr1) or the second precision (Pr2).

At this time, the first path decision signal PD1 and the second path decision signal PD2 may be generated independently from each other. In other words, the first pass decision signal PD1 is generated based on the operation mode signal (Mode), the first sub weight (W0), and the first sub input activation (IN0), and the second pass decision signal (PD2) is It may be generated based on the operation mode signal (Mode), the second sub weight (W1), and the second sub input activation (IN1).

When the operation mode signal (Mode) is associated with the first precision (Pr1), the pass selection logic (PSL) determines the first pass based on the size of the first sub weight (W0) and the first sub input activation (IN0). A signal (PD1) can be generated. For example, when the size of at least some of the first sub weight (W0) and the first sub input activation (IN0) is greater than the maximum value of the second precision (Pr2), the pass selection logic (PSL) uses the first pass decision signal The first signal can be generated with (PD1).

Meanwhile, when the size of each of the first sub weight (W0) and the first sub input activation (IN0) is less than or equal to the maximum value of the second precision (Pr2), the pass selection logic (PSL) transmits the first pass decision signal (PD1) ) can generate the second signal.

When the operation mode signal (Mode) is associated with the second precision (Pr2), the pass selection logic (PSL) may generate a second signal using the first path decision signal (PD1).

Independently of this, when the operation mode signal (Mode) is associated with the first precision (Pr1), the pass selection logic (PSL) is based on the size of the second sub weight (W1) and the second sub input activation (IN1), A second pass decision signal PD2 may be generated. For example, when the size of at least some of the second sub weight (W1) and the second sub input activation (IN1) is greater than the maximum value of the second precision (Pr2), the pass selection logic (PSL) uses the second pass decision signal The first signal can be generated with (PD2).

Meanwhile, when the size of each of the second sub weight (W1) and the second sub input activation (IN1) is less than or equal to the maximum value of the second precision (Pr2), the pass selection logic (PSL) transmits the second pass decision signal (PD2) ) can generate the second signal.

When the operation mode signal (Mode) is associated with the second precision (Pr2), the pass selection logic (PSL) may generate a second signal using the second path decision signal (PD2). That is, the first path decision signal PD1 and the second path decision signal PD2 may be generated independently.

For example, assume that the first precision (Pr1) is INT4, the second precision (Pr2) is INT2, the weight is '00100101', and the input activation (Act_In) is '00010001'. This explains.

15 and 16, when the operation mode signal (Mode) is INT4, the bit division logic (BDL) refers to the operation mode signal (Mode), divides the weight into INT4 units, and divides the weight into INT4 units. A weight (W0) and a second sub-weight (W1) can be created. At this time, the first sub-weight (W0) may be '0010' and the second sub-weight (W1) may be '0101'. Similarly, the bit division logic (BDL) can divide the input activation (Act_In) into INT4 units to generate the first sub-input activation (IN0) and the second sub-input activation (IN1). At this time, the first sub-input activation (IN0) may be '0001', and the second sub-input activation (IN1) may be '0001'.

The pass selection logic (PSL) may determine the first pass decision signal (PD1) to be one of the first signal and the second signal, depending on the size of the first sub weight (W0) and the first sub input activation (IN0). .

In the pass selection logic (PSL), the upper two bits of the first sub weight (W0) and the first sub input activation (IN0), which are INT4, are all 0, so 1 (second signal) as the first pass decision signal (PD1) ) can be output.

The conversion logic (CVL) combines the first sub-weight (W0, '0010') and first sub-input activation (IN0, '0001') of INT4 with the first sub-weight (W0', '10') of INT2 and the first sub-input activation (IN0, '0001') of INT4. It can be converted to sub-input activation (IN0', '01').

The conversion logic (CVL) provides the first sub-weight (W0') of the converted INT2 to the first weight demultiplexer (Demux_W1), and provides the first sub-input activation (IN0') of the converted INT2 to the first input activation demultiplexer (Demux_W1). It can be provided to Demux_I1).

The first weight demultiplexer (Demux_W1) may provide the first sub-weight (W0') of the converted INT2 to the second multiplier (Mul2) according to the first pass decision signal (PD1). Additionally, the first input activation demultiplexer (Demux_I1) may provide the first sub-input activation (IN0') of the converted INT2 to the second multiplier (Mul2) according to the first pass decision signal (PD1).

Meanwhile, the pass selection logic (PSL) can output 0 (the first signal) as the second pass decision signal (PD2) because the second upper bit of the second sub weight (W1), which is INT4, is 1.

At this time, the second sub-weight (W1, '0101') of INT4 is provided to the second weight demultiplexer (Demux_W2), and the second sub-input activation (IN1, '0001') of INT4 is provided to the second input activation demultiplexer (Demux_I2). can be provided.

The second weight demultiplexer (Demux_W2) provides the second sub-weight (W1) of INT4 to the first multiplier (Mul1) according to the second pass decision signal (PD2), and the second input activation demultiplexer (Demux_I2) According to the 2-pass decision signal PD2, the second sub-input activation (IN1) of INT4 may be provided to the first multiplier (Mul1).

In other words, the pass selection logic (PSL) operates on the first sub weight (W0) and the first sub input activation (IN0) at the second multiplier (Mul2), and the second sub weight (W1) and the second sub input Activation (IN1) may be calculated in the first multiplier (Mul1). However, Figure 16 is only an example of implementing the pass selection logic (PSL) configured for convenience of explanation, and the embodiments are not limited to this configuration.

According to some other embodiments of the present invention, the pass decision signal (PD) is based on each pair of a plurality of sub weights (W0, W1, ...) and a plurality of sub input activations (IN0, IN1, ...) Thus, multiple numbers can be created. That is, even if the weight (Weight) and input activation (Act_In) are the same, some sub-weights (Weight) and sub-input activation (Act_In) are calculated along the first path (Path#1), and some other sub-weights (Weight) and Sub-input activation (Act_In) may be calculated along the second path (Path#2). As described above, since the power consumption of the second multiplier (Mul2) is less than the power consumption of the first multiplier (Mul1), according to some other embodiments of the present invention, some sub weights and sub inputs Activation (Act_In) can be calculated in the second multiplier (Mul2), so power consumption can be further reduced.

Figure 17 is a diagram for explaining the configuration of a flexible multiplier according to another embodiment of the present invention. For convenience of explanation, content that is the same or similar to the content described above will be omitted or briefly explained.

Referring to FIG. 17, the flexible multiplier (FM) may include a control pipeline (CPL). The control pipeline (CPL) can perform the function of synchronizing the input and output of the flexible multiplier (FM). For example, if all operations of the first multiplier (Mul1) or the second multiplier (Mul2) are performed within one cycle, the control pipeline (CPL) may not operate. Meanwhile, when a flip-flop occurs in the operation of the first multiplier (Mul1) or the second multiplier (Mul2) (when the operation is delayed), the operation of the first multiplier (Mul1) or the second multiplier (Mul2) The results can be output in the next cycle. At this time, the control pipeline (CPL) operates the pass decision unit until the operation of the first multiplier (Mul1) or the second multiplier (Mul2) is completed in order to synchronize the input and output of the flexible multiplier (FM). Input to (PDU) can be temporarily blocked.

Figure 18 is a diagram for explaining the configuration of a flexible multiplier according to another embodiment of the present invention. For convenience of explanation, content that is the same or similar to the content described above is omitted or briefly explained.

Referring to FIG. 18, the path decision unit (PDU) may further include error detection logic (EDL). Error detection logic (EDL) can detect whether overflow or underflow occurs in the multiplication result of weight (Weight) and input activation (Act_In). At this time, overflow may be an error that appears when the numerical range is larger than the data precision, and underflow may be an error that appears when the numerical range is smaller than the data precision.

When the operation mode signal (Mode) is associated with the first precision (Pr1) and the pass selection logic (PSL) generates the second signal with the pass decision signal (PD), the conversion logic (CVL) is connected to the first precision (Pr1). )'s weight and input activation (Act_In) can be converted to the weight and input activation (Act_In) of the second precision (Pr2).

At this time, the error detection logic (EDL) can detect whether an overflow or underflow error occurs in the weight (Weight) and input activation (Act_In) converted to the second precision (Pr2). If an overflow or underflow error occurs in the weight (Weight) and input activation (Act_In) converted to the second precision (Pr2), the error detection logic (EDL) sends an error occurrence signal to the pass selection logic (PSL). can be provided. When the path selection logic (PSL) receives an error occurrence signal, it can change the path decision signal (PD) to the first signal.

For example, assume that Weight and input activation (Act_In) are each '0011' of INT4. First, the pass selection logic (PSL) may generate the pass decision signal (PD) as a second signal based on the size of the weight (Weight) and the input activation (Act_In). At this time, the conversion logic (CVL) can convert '0011' of INT4 to '11' of INT2.

At this time, if '11' in INT2 represents the decimal number 9, the product of '11' in INT 2 and '11' is 81 in decimal, so it cannot be expressed in 4-bit INT4, which may cause overflow. In this case, the decimal number 81 can be clearly expressed through the result of the multiplication operation of INT8 by converting '11' back to '0011', which is INT4.

Therefore, the error detection logic (EDL) detects the overflow and provides an error occurrence signal to the pass selection logic (PSL), and the pass selection logic (PSL) generates the pass decision signal (PD) as the first signal again. You can.

Ultimately, when an overflow or underflow error is detected in the error detection logic (EDL), the weight (Weight) and the input activation (Act_In) of the first precision (Pr1) are provided to the demultiplexer (Dx), and the demultiplexer (Dx) According to the pass decision signal PD, the weight of the first precision Pr1 and the input activation Act_In may be provided to the first multiplier Mul1.

According to some embodiments of the present invention, after the weight (Weight) and input activation (Act_In) of the first precision (Pr1) are converted to the second precision (Pr2), the error detection logic (EDL) overflows or underflows. Although it has been described as determining whether an error occurs, the embodiments are not limited thereto. For example, the error detection logic (EDL) uses the weight (Weight) and input activation (Act_In) of the first precision (Pr1) and uses the weight (Weight) and input activation (Act_In) of the second precision (Pr2). It is also possible to detect in advance whether overflow or underflow will occur during conversion. In this case, the range of input data of the error detection logic (EDL) is extended to the first precision (Pr1), but the conversion logic (CVL) controls the weight (Weight) and input activation (Act_In) of the first precision (Pr1). 2 There is an advantage that there is no need to convert to Weight and Input Activation (Act_In) of Precision (Pr2), making it simpler in terms of time and procedures.

Meanwhile, even when the operation mode signal (Mode) is associated with the second precision (Pr2), the error detection logic (EDL) can detect an overflow or underflow error and generate an error occurrence signal. When an error signal is generated, the conversion logic (CVL) may convert the weight and input activation (Act_In) of the second precision (Pr2) to the first precision (Pr1). Additionally, the path selection logic (PSL) may generate a first signal using the path decision signal (PD). The weight and input activation (Act_In) converted to the first precision (Pr1) are provided to the demultiplexer (Dx), and the demultiplexer (Dx) converts them to the first precision (Pr1) according to the pass decision signal (PD). The weight and input activation (Act_In) can be provided to the first multiplier (Mul1).

Therefore, in this embodiment, when overflow or underflow occurs, conversion can be performed to increase the number of bits of data by changing the precision. Through this, you can normally use a low number of bits with high efficiency, but when the operation may become inaccurate, convert to a higher number of bits to improve the precision of the operation while maintaining optimal efficiency.

In particular, in the case of INT2, the range is narrow and quantization is frequent, so such overflow or underflow may occur very frequently. INT2 has high data efficiency due to its small number of bits, so it can be highly useful in cases where hardware resources are limited, such as mobile devices. Therefore, this embodiment can prevent a decrease in accuracy due to overflow or underflow, which frequently occurs in areas where a low-bit number precision such as INT2 is utilized.

FIG. 19 is a block diagram for explaining the L0 memory of FIG. 7 in detail.

Referring to FIG. 19, the L0 memory 120 may include an arbiter 121 and at least one local memory bank 122.

When data is stored in the L0 memory 120, the arbiter 121 may receive the data from the load engine 113a. At this time, data may be allocated to the local memory bank 122 in a round robin manner. Accordingly, data may be stored in any one of at least one local memory bank 122.

Conversely, when data is loaded from the L0 memory 120, the arbiter 121 may receive data from the local memory bank 122 and transfer it to the store engine 113b. The store engine 113b can store data externally through the local interconnection 200.

FIG. 20 is a block diagram for explaining the local memory bank of FIG. 19 in detail.

Referring to FIG. 20, the local memory bank 122 may include a local memory bank controller 122_1 and a local memory bank cell array 122_2.

The local memory bank controller 122_1 can manage read and write operations through the address of data stored in the local memory bank 122. That is, the local memory bank controller 122_1 can manage the overall input and output of data.

The local memory bank cell array 122_2 may have a structure in which cells in which data is directly stored are aligned in rows and columns. The local memory bank cell array 122_2 may be controlled by the local memory bank controller 122_1.

Figure 21 is a block diagram to explain in detail the structure of a neural processing device according to some embodiments of the present invention.

Referring to FIG. 21, the neural core 101, unlike the neural core 100, may have a CGRA structure. The neural core 101 may include an instruction memory 111_1, a CGRA L0 memory 111_2, a PE array 111_3, and a Load/Store Unit (LSU) 111_4.

The instruction memory 111_1 can receive and store instructions. The instruction memory 111_1 may sequentially store instructions internally and provide the stored instructions to the PE array 111_3. At this time, the instruction may direct the operation of the processing element 111_3a included in each PE array 111_3.

The CGRA L0 memory 111_2 is a memory located inside the neural core 101, and allows the neural core 101 to receive all input data required for work from the outside and temporarily store them. Additionally, the CGRA L0 memory 111_2 can temporarily store output data calculated by the neural core 101 in order to transmit it to the outside. The CGRA L0 memory 111_2 may serve as a cache memory for the neural core 101.

The CGRA L0 memory 111_2 can transmit and receive data with the PE array 111_3. The CGRA L0 memory 111_2 may be a memory corresponding to L0 (level 0), which is lower than L1. At this time, the L0 memory may be a private memory of the neural core 101 that is not shared. The CGRA L0 memory (111_2) can transmit data such as activation or weight and programs to the PE array (111_3).

The PE array 111_3 may be a module that performs calculations. The PE array 111_3 can perform not only one-dimensional operations but also two-dimensional or more matrix/tensor operations. The PE array 111_3 may include a plurality of processing elements 111_3a and a specific processing element 111_3b therein.

The processing element 111_3a and the specific processing element 111_3b may be arranged in rows and columns. The processing element 111_3a and the specific processing element 111_3b may be arranged in m columns. Additionally, the processing element 111_3a may be arranged in n rows, and the specific processing element 111_3b may be arranged in l rows. Accordingly, the processing element 111_3a and the specific processing element 111_3b may be arranged in (n+l) rows and m columns.

The LSU 111_4 may receive at least one of data, control signals, and synchronization signals from the outside through the local interconnection 200. The LSU 111_4 may transmit at least one of the received data, control signal, and synchronization signal to the CGRA L0 memory 111_2. Similarly, the LSU 111_4 may transmit at least one of data, a control signal, and a synchronization signal to the outside through the local interconnection 200.

The neural core 101 may have a Coarse Grained Reconfigurable Architecture (CGRA) structure. Accordingly, the neural core 101 configures each processing element 111_3a and the specific processing element 111_3b of the PE array 111_3 at least one of the CGRA L0 memory 111_2, the instruction memory 111_1, and the LSU 111_4. It can be connected to one. That is, the processing element 111_3a and the specific processing element 111_3b do not have to be connected to all of the CGRA L0 memory 111_2, the instruction memory 111_1, and the LSU 111_4, but may be connected to some of them.

Additionally, the processing element 111_3a and the specific processing element 111_3b may be different types of processing elements. Accordingly, among the CGRA L0 memory 111_2, the instruction memory 111_1, and the LSU 111_4, the element connected to the processing element 111_3a and the element connected to the specific processing element 111_3b may be different from each other.

The neural core 101 of the present invention with a CGRA structure is capable of high-level parallel computation and direct data exchange between the processing element 111_3a and the specific processing element 111_3b, so power consumption can be low. Additionally, optimization according to various computational tasks may be possible by including two or more types of processing elements 111_3a.

For example, if the processing element 111_3a is a processing element that performs a two-dimensional operation, the specific processing element 111_3b may be a processing element that performs a one-dimensional operation. However, this embodiment is not limited to this.

Figure 22 is a block diagram for explaining memory reorganization of a neural processing system according to some embodiments of the present invention.

Referring to FIG. 22, the neural core SoC 10 may include first to eighth processing units 160a to 160h and an on-chip memory (OCM). 26 shows eight processing units as an example, but this is only an example and the number of processing units may vary.

The on-chip memory (OCM) may include first to eighth L0 memories 120a to 120h and a shared memory 2000.

The first to eighth L0 memories 120a to 120h may be used as dedicated memories for the first to eighth processing units 160a to 160h, respectively. That is, the first to eighth processing units 160a to 160h and the first to eighth L0 memories 120a to 120h may correspond to each other in a 1:1 manner.

The shared memory 2000 may include first to eighth memory units 2100a to 2100h. The first to eighth memory units 2100a to 2100h may correspond to the first to eighth processing units 160a to 160h and the first to eighth L0 memories 120a to 120h, respectively. That is, the number of memory units may be 8, which is the same as the number of processing units and L0 memories.

Shared memory 2000 may operate in either of two types of on-chip memory formats. That is, the shared memory 2000 may operate in either an L0 memory format or a global memory format. In other words, the shared memory 2000 can implement two logical memories with one hardware.

When the shared memory 2000 is implemented in an L0 memory format, the shared memory 2000 is a dedicated memory for each of the first to eighth processing units 160a to 160h, such as the first to eighth L0 memories 120a to 120h. It can operate with (private memory). The L0 memory can operate at a relatively high clock speed compared to the global memory, and the shared memory 2000 can also use a relatively faster clock when operating in the L0 memory format.

When the shared memory 2000 is implemented in a global memory format, the shared memory 2000 can operate as a common memory used by the first processing unit 100a and the second processing unit 100b. there is. At this time, the shared memory 2000 may be shared not only by the first to eighth processing units 160a to 160h but also to the first to eighth L0 memories 120a to 120h.

Global memory can generally use a lower clock than L0 memory, but is not limited to this. When the shared memory 2000 operates in a global memory format, the first to eighth processing units 160a to 160h may share the shared memory 2000. At this time, the shared memory 2000 is connected to the volatile memory 32 of FIG. 2 through the global interconnection 5000, and may operate as a buffer of the volatile memory 32.

At least part of the shared memory 2000 may operate in an L0 memory format, and the remainder may operate in a global memory format. That is, the entire shared memory 2000 may operate in an L0 memory format, or the entire shared memory 2000 may operate in a global memory format. Alternatively, part of the shared memory 2000 may operate in the L0 memory format, and the remaining part may operate in the global memory format.

Figure 23 is a block diagram showing an example of memory reorganization of a neural processing system according to some embodiments of the present invention.

22 and 23, the first, third, fifth, and seventh processing units (100a, 100c, 100e, and 100g) each have first, third, fifth, and seventh dedicated areas (AE1, AE3). , AE5, and AE7) may include only the first, third, fifth, and seventh L0 memories 120a, 120c, 120e, and 120g, respectively. In addition, the second, fourth, sixth, and eighth dedicated areas (AE2, AE4, AE6, and AE8) of the second, fourth, sixth, and eighth processing units (100b, 100d, 100f, and 100h) are respectively It may include second, fourth, sixth, and eighth L0 memories (120b, 120d, 120f, 120h). Additionally, the second, fourth, sixth, and eighth dedicated areas (AE2, AE4, AE6, and AE8) will include the second, fourth, sixth, and eighth memory units (2100b, 2100d, 2100f, 2100h). You can. The first, third, fifth, and seventh memory units 2100a, 2100c, 2100e, and 2100g of the shared memory 2000 may be used as a common area (AC).

The common area AC may be a memory shared by the first to eighth processing units 160a to 160h. The second dedicated area AE2 may include a second L0 memory 120b and a second memory unit 2100b. The second dedicated area AE2 may be an area in which the hardware-separated second L0 memory 120b and the second memory unit 210b operate in the same manner and logically operate as one L0 memory. The fourth, sixth, and eighth dedicated areas (AE4, AE6, and AE8) may also operate in the same manner as the second dedicated area (AE2).

The shared memory 2000 according to this embodiment can be used by converting the area corresponding to each neural core into logical L0 memory and logical global memory at an optimized ratio. The shared memory 2000 can adjust this ratio at run time.

In other words, each neural core may perform the same tasks, but may also perform different tasks. In this case, the capacity of L0 memory and the capacity of global memory required for the work performed by each neural core are bound to be different each time. Accordingly, if the composition ratio of L0 memory and shared memory is set fixedly, as in the existing on-chip memory, inefficiencies may occur due to the computational tasks assigned to each neural core.

Accordingly, the shared memory 2000 of the neural processing device according to this embodiment can set the optimal ratio of L0 memory and global memory according to the computational task during run time and improve computational efficiency and speed.

Figure 24 is an enlarged block diagram of part A of Figure 22.

22 and 24, the shared memory 2000 includes a first L0 memory controller 122_1a, a second L0 memory controller 122_1b, a fifth L0 memory controller 122_1e, and a sixth L0 memory controller 122_1f. , may include first to eighth memory units 2100a to 2100h and a global controller 2200. Other L0 memory controllers not shown may also be included in this embodiment, but descriptions are omitted for convenience.

The first L0 memory controller 122_1a can control the first L0 memory 120a. Additionally, the first L0 memory controller 122_1a may control the first memory unit 2100a. Specifically, when the first memory unit 2100a is implemented in a logical L0 memory format, control by the first L0 memory controller 122_1a may be performed on the first memory unit 2100a.

The second L0 memory controller 122_1b can control the second L0 memory 120b. Additionally, the second L0 memory controller 122_1b may control the second memory unit 2100b. That is, when the second memory unit 2100b is implemented in a logical L0 memory format, control by the first L0 memory controller 122_1a can be performed on the second memory unit 2100b.

The fifth L0 memory controller 122_1e can control the fifth L0 memory 120e. Additionally, the fifth L0 memory controller 122_1e can control the fifth memory unit 2100e. That is, when the fifth memory unit 2100e is implemented in a logical L0 memory format, control by the fifth L0 memory controller 122_1e can be performed on the fifth memory unit 2100e.

The sixth L0 memory controller 122_1f can control the sixth L0 memory 120f. Additionally, the sixth L0 memory controller 122_1f can control the sixth memory unit 2100f. That is, when the sixth memory unit 2100f is implemented in a logical L0 memory format, control by the sixth L0 memory controller 122_1f can be performed on the sixth memory unit 2100f.

The global controller 2200 can control all of the first to eighth memory units 2100a to 2100h. Specifically, when the first to eighth memory units 2100a to 2100h each logically operate in the global memory format (i.e., do not logically operate in the L0 memory format), the global controller 2200 operates the first memory The units 2100a to 8th memory units 2100h can be controlled.

That is, the first to eighth memory units 2100a to 2100h are controlled by the first to eighth L0 memory controllers 122_1a to 122_1h, respectively, or by the global controller 2200, depending on what type of memory they are logically implemented as. It can be controlled by

When L0 memory controllers including the first, second, fifth, and sixth L0 memory controllers 122_1a, 122_1b, 122_1e, and 122_1f control the first to eighth memory units 2100a to 2100h, respectively, the first Since the first to eighth L0 memory controllers 122_1a to 141h control the first to eighth memory units 2100a to 2100h in the same manner as the first to eighth L0 memories 120a to 120h, the first to eighth processing units It can be controlled with a dedicated memory of (160a~160h). Accordingly, the first to eighth memory units 2100a to 2100h may operate at a clock frequency corresponding to the clock frequency of the first to eighth processing units 160a to 160h.

The L0 memory controllers including the first L0 memory controller 122_1a, the second L0 memory controller 122_1b, the fifth L0 memory controller 122_1e, and the sixth L0 memory controller 122_1f are respectively the LSU 110 of FIG. 7. may include.

When the global controller 2200 controls at least one of the first to eighth memory units 2100a to 2100h, the global controller 2200 controls the first to eighth memory units 2100a to 2100h, respectively. It can be controlled by the global memory of the 8th processing unit (160a ~ 160h). Accordingly, at least one of the first to eighth memory units 2100a to 2100h may operate at a clock frequency that is unrelated to the clock frequency of the first to eighth processing units 160a to 160h. However, this embodiment is not limited to this.

The global controller 2200 may connect the first to eighth memory units 2100a to 2100h with the global interconnection 5000 of FIG. 3 . The first to eighth memory units 2100a to 2100h exchange data with the off-chip memory 30 of FIG. 1 by the global controller 2200, or exchange data with the first to eighth L0 memories 120a to 120h, respectively. can be exchanged.

The first to eighth memory units 2100a to 2100h may each include at least one memory bank. The first memory unit 2100a may include at least one first memory bank 2110a. The first memory bank 2110a may be an area divided by dividing the first memory unit 2100a into a specific size. Each first memory bank 2110a may be a memory element of the same size. However, this embodiment is not limited to this. In Figure 24, four memory banks are shown as being included in one memory unit.

Similarly, the second, fifth, and sixth memory units 2100b, 2100e, and 2100f may each include at least one second, fifth, and sixth memory bank 2110b, 2110e, and 2110f.

Hereinafter, the description will be made based on the first memory bank 2110a and the fifth memory bank 2110e, which may be the same as other memory banks including the second and sixth memory banks 2110b and 2110f.

The first memory bank 2110a may logically operate in an L0 memory format or logically in a global memory format. At this time, the first memory bank 2110a may operate independently from other memory banks in the first memory unit 2100a. However, this embodiment is not limited to this.

When operating independently for each memory bank, the first memory unit 2100a has a first area that operates in the same manner as the first L0 memory 120a and a second area that operates in a different manner from the first L0 memory 120a. Can include 2 areas. At this time, the first area and the second area do not necessarily coexist, and one area may occupy the entire first memory unit 2100a.

Likewise, the second memory unit 2100b may include a third area that operates in the same manner as the second L0 memory 120b and a fourth area that operates in a different manner from the second L0 memory 120b. At this time, the third area and the fourth area do not necessarily coexist, and one area may occupy the entire first memory unit 2100a.

At this time, the ratio of the first area to the second area may be different from the ratio of the third area to the fourth area. However, this embodiment is not limited to this. Accordingly, the ratio of the first area to the second area may be the same as the ratio of the third area to the fourth area. That is, the memory configuration ratio in each memory unit can vary as much as desired.

In general, in the case of existing system-on-chip, the on-chip memory, excluding high-speed L0 memory, is often composed of high-density, low-power SRAM. This is because SRAM has high efficiency in terms of chip area and power usage compared to the required capacity. However, the processing speed of the existing on-chip memory inevitably slowed down significantly in the case of tasks that require more data quickly than the predetermined capacity of the L0 memory, and even when the need for global memory is not large, there is no way to utilize the remaining global memory. There was no use at all, resulting in inefficiency.

In contrast, the shared memory 2000 according to some embodiments of the present invention may be selectively controlled by one of two controllers, depending on the case. At this time, the shared memory 2000 is not controlled as a whole by only one of the two controllers, but can be independently controlled on a memory unit basis or a memory bank basis.

Through this, the shared memory 2000 according to this embodiment can obtain the optimal memory configuration ratio according to the computational task during run time and perform faster and more efficient computational tasks. In the case of processing units specialized in artificial intelligence, the required sizes of L0 memory and global memory may vary for each specific application. Furthermore, even for the same application, when using a deep learning network, the required sizes of L0 memory and global memory for each layer may vary. The shared memory 2000 according to this embodiment can enable fast and efficient deep learning work because the memory composition ratio can be changed during run time despite changes in the calculation steps for each layer.

FIG. 25 is a diagram for explaining the first memory bank of FIG. 24 in detail. Although FIG. 25 illustrates the first memory bank 2110a, other memory banks may also have the same structure as the first memory bank 2110a.

Referring to FIG. 25 , the first memory bank 2110a may include a cell array (Ca), a bank controller (Bc), a first path unit (P1), and a second path unit (P2).

The cell array (Ca) may include a plurality of memory elements (Cells) therein. The cell array Ca may have a plurality of memory elements arranged in a lattice structure. The cell array (Ca) may be, for example, a Static Random Access Memory (SRAM) cell array.

The bank controller (Bc) can control the cell array (Ca). The bank controller Bc may determine whether the cell array Ca will operate in an L0 memory format or a global memory format and control the cell array Ca accordingly.

Specifically, the bank controller (Bc) can determine whether to transmit and receive data in the first path unit (P1) direction or the second path unit (P2) direction during run time. The bank controller (Bc) can determine the direction of data transmission and reception according to the path control signal (Spc).

The path control signal (Spc) can be generated by a pre-designed device driver or compiler. A path control signal (Spc) can be generated according to the characteristics of the computational task. Alternatively, the path control signal (Spc) may be generated by input received from the user. In other words, the user can directly apply the input to the path control signal (Spc) in order to select the most optimal memory configuration ratio.

The bank controller (Bc) can determine a path for transmitting and receiving data stored in the cell array (Ca) through the path control signal (Spc). The data exchange interface may vary depending on how the bank controller (Bc) determines the path through which data is transmitted and received. That is, the bank controller Bc can use the first interface when exchanging data with the first path unit P1, and can use the second interface when exchanging data with the second path unit P2. At this time, the first interface and the second interface may be different from each other.

Additionally, the address system in which data is stored may also vary. That is, when a specific interface is selected, read and write operations can be performed using the corresponding address system.

The bank controller (Bc) can operate at a specific clock frequency. For example, when the cell array Ca is an SRAM cell array, the bank controller Bc can operate at a typical SRAM operating clock frequency.

The first path unit (P1) may be connected to the bank controller (Bc). The first path unit P1 may directly exchange data of the cell array Ca with the first processing unit 100a. At this time, “directly” may mean that they are exchanged without going through the global interconnection (5000). That is, the first processing unit 100a can directly exchange data with the first L0 memory 120a, and the first processing unit 100a can first process the first processing unit 100a when the shared memory 2000 is logically implemented in the L0 memory format. 1 Data can be exchanged through the path unit (P1). The first path unit P1 may include an L0 memory controller including the first L0 memory controller 122_1a and the second L0 memory controller 122_1b of FIG. 14 .

The first path unit (P1) may configure a multi-cycle sync path. That is, the operating clock frequency of the first path unit P1 may be the same as the operating clock frequency of the first processing unit 100a. The first L0 memory 120a can quickly exchange data at the same clock frequency as the operating clock frequency of the first processing unit 100a in order to quickly exchange data at the same speed as the operation of the first processing unit 100a. . The first path unit P1 may also operate at the same clock frequency as the operating clock frequency of the first processing unit 100a.

At this time, the operating clock frequency of the first path unit (P1) may be a multiple of the operating clock frequency of the bank controller (Bc). In this case, a separate CDC (Clock Domain Crossing) operation for clock synchronization between the bank controller (Bc) and the first path unit (P1) is not required, and accordingly, a delay in data transmission may not occur. there is. Accordingly, faster and more efficient data exchange may be possible.

In FIG. 25 , as an example, the operating clock frequency of the first path unit P1 may be 1.5 GHz. This may be twice the frequency of 750 MHz of the bank controller (Bc). However, this embodiment is not limited to this, and it may be possible as long as the first path unit (P1) operates at an integer multiple of the clock frequency of the bank controller (Bc).

The second path unit (P2) may be connected to the bank controller (Bc). The second path unit P2 may exchange data of the cell array Ca through the global interconnection 5000 instead of directly exchanging it with the first processing unit 100a. That is, the first processing unit 100a can exchange data with the cell array Ca through the global interconnection 5000 and the second path unit P2. At this time, the cell array Ca can exchange data not only with the first processing unit 100a but also with other neural cores.

That is, the second path unit P2 may be a data exchange path between the cell array Ca and all neural cores when the first memory bank 2110a is logically implemented in a global memory format. The second path unit (P2) may include the global controller 2200 of FIG. 24.

The second path unit (P2) may configure an Async-Path. The operating clock frequency of the second path unit P2 may be the same as the operating clock frequency of the global interconnection 5000. The second path unit P2 may also operate at the same clock frequency as the operating clock frequency of the global interconnection 5000.

At this time, the operating clock frequency of the second path unit (P2) may not be synchronized with the operating clock frequency of the bank controller (Bc). In this case, a Clock Domain Crossing (CDC) operation may be required to synchronize clocks between the bank controller (Bc) and the second path unit (P2). If the operating clock frequency of the bank controller (Bc) and the operating clock frequency of the second path unit (P2) are not synchronized with each other, the degree of freedom in designing the clock domain may increase. Accordingly, the difficulty of hardware design is lowered and hardware operations can be derived more easily.

The bank controller Bc may use different address systems when exchanging data through the first path unit P1 and when exchanging data through the second path unit P2. That is, the bank controller Bc can use the first address system through the first path unit P1 and the second address system through the second path unit P2. At this time, the first address system and the second address system may be different from each other.

The bank controller (Bc) does not necessarily need to exist for each memory bank. In other words, the bank controller (Bc) is not a part for scheduling but serves to transmit signals, so it is not an essential part for each memory bank with two ports. Therefore, one bank controller (Bc) can control multiple memory banks. Multiple memory banks can operate independently although controlled by the bank controller (Bc). However, this embodiment is not limited to this.

Of course, a bank controller (Bc) may exist for each memory bank. In this case, the bank controller Bc can individually control each memory bank.

24 and 25, when the first memory unit 210a exchanges data through the first path unit P1, it uses the first address system and exchanges data through the second path unit P2. In case of exchange, a second address system can be used. Similarly, when the second memory unit 210b exchanges data through the first path unit P1, a third address system is used, and when data is exchanged through the second path unit P2, the second address system is used. You can use the system. At this time, the first address system and the third address system may be the same. However, this embodiment is not limited to this.

The first address system and the third address system may be used exclusively for the first processing unit 100a and the second processing unit 100b, respectively. The second address system may be commonly applied to the first processing unit 100a and the second processing unit 100b.

In FIG. 25 , as an example, the operating clock frequency of the second path unit P2 may operate at 1 GHz. This may be a frequency that is not synchronized with the 750 MHz operating clock frequency of the bank controller (Bc). That is, the operating clock frequency of the second path unit (P2) can be freely set without being at all dependent on the operating clock frequency of the bank controller (Bc).

Typical global memory uses slow SRAM (e.g., 750 MHz) and faster global interconnection (e.g., 1 GHz), which inevitably causes delays due to CDC operations. In contrast, the shared memory 2000 according to some embodiments of the present invention has room to use the first path unit (P1) in addition to the second path unit (P2), thereby avoiding delays due to CDC operations.

In addition, since a general global memory uses one global interconnection 5000 for multiple neural cores, a decrease in overall processing speed can easily occur when data transfer volume occurs simultaneously. On the other hand, the shared memory 2000 according to some embodiments of the present invention has room to use the first path unit (P1) in addition to the second path unit (P2), thereby appropriately reducing the data processing load on the global controller 2200. A dispersing effect can also be achieved.

Figure 26 is a block diagram to explain the software hierarchy of a neural processing device according to some embodiments of the present invention.

Referring to FIG. 26, the software layer structure of a neural processing device according to some embodiments of the present invention may include a DL framework 10000, a compiler stack 20000, and a backend module 30000.

The DL framework (10000) may refer to a framework for a deep learning model network used by users. For example, a fully trained neural network can be created using programs such as TensorFlow or PyTorch.

The compiler stack 20000 may include an adaptation layer 21000, a compute library 22000, a front-end compiler 23000, a back-end compiler 24000, and a runtime driver 25000.

The adaptation layer (21000) may be a layer in contact with the DL framework (10000). The adaptation layer (21000) can quantize the user's neural network model created in the DL framework (10000) and modify the graph. Additionally, the adaptation layer 21000 can convert the model type into a required type.

The front-end compiler (23000) can convert various neural network models and graphs received from the adaptation layer (21000) into a certain intermediate representation (IR). The converted IR may be a preset expression that is easy to handle later in the backend compiler 24000.

The IR of the front-end compiler 23000 can be optimized in advance at the graph level. Additionally, the front-end compiler 23000 can ultimately generate the IR by converting it into a hardware-optimized layout.

The backend compiler (24000) optimizes the IR converted from the frontend compiler (23000) and converts it into a binary file so that the runtime driver can use it. The back-end compiler (24000) can generate optimized code by dividing the job at a scale that matches the details of the hardware.

The compute library 22000 can store template operations designed in a form suitable for hardware among various operations. The compute library 22000 provides several template operations that require hardware to the backend compiler 24000, allowing optimized code to be generated.

The runtime driver 25000 may continuously perform monitoring while driving to drive the neural network device according to some embodiments of the present invention. Specifically, it may be responsible for executing the interface of the neural network device.

The backend module 30000 may include an application specific integrated circuit (ASIC) 31000, a field programmable gate array (FPGA) 32000, and a C-model (33000). ASIC (31000) may refer to a hardware chip determined according to a predetermined design method. FPGA 32000 may be a programmable hardware chip. C-model (33000) may refer to a model implemented by simulating hardware in software.

The backend module 30000 can perform various tasks and derive results using binary code generated through the compiler stack 20000.

Figure 27 is a conceptual diagram to explain a deep learning operation performed by a neural processing device according to some embodiments of the present invention.

Referring to FIG. 27, the artificial neural network model 40000 is an example of a machine learning model, and in machine learning technology and cognitive science, a statistical learning algorithm implemented based on the structure of a biological neural network or an algorithm thereof. It is a structure that executes .

In the artificial neural network model (40000), as in a biological neural network, nodes, which are artificial neurons that form a network through the combination of synapses, repeatedly adjust the weight of the synapse, creating a gap between the correct output corresponding to a specific input and the inferred output. By learning to reduce the error of , a machine learning model with problem-solving capabilities can be expressed. For example, the artificial neural network model 40000 may include random probability models, neural network models, etc. used in artificial intelligence learning methods such as machine learning and deep learning.

A neural processing device according to some embodiments of the present invention may perform calculations by implementing this type of artificial neural network model (40000). For example, the artificial neural network model 40000 may receive an input image and output information about at least a portion of the object included in the input image.

The artificial neural network model (40000) is implemented as a multilayer perceptron (MLP) consisting of multiple layers of nodes and connections between them. The artificial neural network model 40000 according to this embodiment can be implemented using one of various artificial neural network model structures including MLP. As shown in FIG. 27, the artificial neural network model 40000 includes an input layer 41000 that receives an input signal or data 40100 from the outside, and an output layer that outputs an output signal or data 40200 corresponding to the input data. (44000), located between the input layer 41000 and the output layer 44000, receives n signals from the input layer 41000, extracts the characteristics, and transmits them to the output layer 44000 (where n is a positive integer). It consists of a hidden layer (42000 to 43000). Here, the output layer 44000 receives signals from the hidden layers 42000 to 43000 and outputs them to the outside.

The learning methods of the artificial neural network model (40000) include supervised learning, which learns to optimize problem solving by inputting teacher signals (correct answers), and unsupervised learning, which does not require teacher signals. ) There is a way.

The neural processing device can directly generate learning data for training the artificial neural network model (40000) through simulation. In this way, a plurality of input variables and a plurality of output variables corresponding to the input layer 41000 and the output layer 44000 of the artificial neural network model 40000 are matched, respectively, and the input layer 41000, hidden layers 42000 to 43000, and By adjusting the synapse values between nodes included in the output layer 44000, learning can be done so that the correct output corresponding to a specific input can be extracted. Through this learning process, the characteristics hidden in the input variables of the artificial neural network model (40000) can be identified, and the nodes of the artificial neural network model (40000) can be used to reduce the error between the output variables calculated based on the input variables and the target output. You can adjust the synapse value (or weight) between them.

FIG. 28 is a conceptual diagram illustrating learning and inference operations of a neural network of a neural processing device according to some embodiments of the present invention.

Referring to FIG. 28, in the training process (Training Phase), a number of learning data (TD) may be forwarded to the artificial neural network model (NN) and then forwarded back again. Through this, the weights and biases of each node of the artificial neural network model (NN) are adjusted, and through this, learning can be performed to produce increasingly accurate results. Through this learning process (Training Phase), the artificial neural network model (NN) can be converted into a learned neural network model (NN_T).

In the inference phase, new data (ND) can be input back into the learned neural network model (NN_T). The learned neural network model (NN_T) can take new data (ND) as input and derive result data (RD) through already learned weights and biases. This result data (RD) may be important in terms of which learning materials (TD) were used in the training process (Training Phase) and how much learning materials (TD) were used.

Hereinafter, a calculation method of a neural processing device according to some embodiments of the present invention will be described using FIGS. 11, 18, 29, and 30. Content that is the same or similar to the above content will be omitted or briefly explained.

Figure 29 is a diagram for explaining an operation method of a neural processing device according to some embodiments of the present invention.

Referring to FIGS. 11 and 29, the weight and input activation (Act_In) of the first precision (Pr1) can be received (S100). According to some embodiments, the pass decision unit (PDU) may receive an operation mode signal (Mode) associated with the first precision (Pr1), a weight (Weight), and an input activation (Act_In) of the first precision (Pr1). there is.

It may be determined whether at least part of the weight and input activation (Act_In) is greater than a predetermined value (S101). According to some embodiments, the pass determination unit (PDU) may determine whether the weight and input activation (Act_In) exceed the maximum value of the second precision (Pr2).

If at least part of the weight and input activation (Act_In) is greater than a predetermined value (S101, Y), the weight and input activation (Act_In) can be calculated using the first precision (Pr1) (S102) ). According to some embodiments, when at least a portion of the weight (Weight) and the input activation (Act_In) exceeds the maximum value of the second precision (Pr2), the pass decision unit (PDU) transmits the pass decision signal (PD) to the first pass It can be generated as the first signal to select (Path#1). Subsequently, the pass decision unit (PDU) provides the weight (Weight) and input activation (Act_In) of the first precision (Pr1) and the pass decision signal (PD), which is the first signal, to the demultiplexer (Dx). ) provides the weight (Weight) and input activation (Act_In) of the first precision (Pr1) to the first multiplier (Mul1) according to the pass decision signal (PD), which is the first signal, and provides the weight (Weight) and input activation (Act_In) to the first multiplier (Mul1) Activation (Act_In) can be calculated with the first precision (Pr1).

On the other hand, when at least part of the weight and input activation (Act_In) is not greater than a predetermined value (S101, N), the weight and input activation (Act_In) of the first precision (Pr1) are changed to the second precision. (Pr2) can be converted into weight and input activation (Act_In) (S103), and weight and input activation (Act_In) can be calculated with the second precision (Pr2) (S104). According to some embodiments, when at least a portion of the weight (Weight) and the input activation (Act_In) does not exceed the maximum value of the second precision (Pr2), the pass decision unit (PDU) sends the pass decision signal (PD) to the second It can be generated as a second signal for selecting the path (Path #2). Subsequently, the pass determination unit (PDU) may convert the weight (Weight) and input activation (Act_In) of the first precision (Pr1) into the weight (Weight) and input activation (Act_In) of the second precision (Pr2). Subsequently, the pass decision unit (PDU) provides the weight (Weight) and input activation (Act_In) converted to the second precision (Pr2) and the pass decision signal (PD), which is a second signal, to the demultiplexer (Dx), and the demultiplexer (Dx) provides the weight (Weight) and input activation (Act_In) converted to the second precision (Pr2) to the second multiplier (Mul2) according to the pass decision signal (PD), which is the second signal, to produce the weight ( Weight) and input activation (Act_In) can be calculated with the second precision (Pr2).

Figure 30 is a diagram for explaining a calculation method of a neural processing device according to some other embodiments of the present invention.

Referring to FIGS. 18 and 30, weight and input activation (Act_In) can be received (S200). According to some embodiments, a pass decision unit (PDU) may receive weight and input activation (Act_In). Additionally, the pass decision unit (PDU) may receive an operation mode signal (Mode).

When the operation mode signal (Mode) is associated with the first precision (Pr1) (S201, Y), it can be determined whether at least a portion of the weight and input activation (Act_In) is greater than a predetermined value. (S202). According to some embodiments, the pass determination unit (PDU) may determine whether the weight and input activation (Act_In) exceed the maximum value of the second precision (Pr2).

If at least part of the weight and input activation (Act_In) is greater than a predetermined value (S202, Y), the weight and input activation (Act_In) can be calculated using the first precision (Pr1) (S203) ). According to some embodiments, when at least a portion of the weight (Weight) and the input activation (Act_In) exceeds the maximum value of the second precision (Pr2), the pass decision unit (PDU) transmits the pass decision signal (PD) to the first pass It can be generated as the first signal to select (Path#1). Subsequently, the pass decision unit (PDU) provides the weight (Weight) and input activation (Act_In) of the first precision (Pr1) and the pass decision signal (PD), which is the first signal, to the demultiplexer (Dx). ) provides the weight (Weight) and input activation (Act_In) of the first precision (Pr1) to the first multiplier (Mul1) according to the pass decision signal (PD), which is the first signal, and provides the weight (Weight) and input activation (Act_In) to the first multiplier (Mul1) Activation (Act_In) can be calculated with the first precision (Pr1).

On the other hand, when at least part of the weight and input activation (Act_In) is not greater than a predetermined value (S202, N), the weight and input activation (Act_In) of the first precision (Pr1) are changed to the second precision. When converting the Weight and Input Activation (Act_In) of (Pr2) (S204) and calculating the Weight and Input Activation (Act_In) converted to the second precision (Pr2), an overflow or underflow error occurs. It is possible to determine whether occurs (S205). According to some embodiments, when at least a portion of the weight (Weight) and the input activation (Act_In) does not exceed the maximum value of the second precision (Pr2), the pass decision unit (PDU) sends the pass decision signal (PD) to the second It can be generated as a second signal for selecting the path (Path #2). Subsequently, the pass determination unit (PDU) may convert the weight (Weight) and input activation (Act_In) of the first precision (Pr1) into the weight (Weight) and input activation (Act_In) of the second precision (Pr2). Next, the pass decision unit (PDU) can check whether an overflow or underflow error occurs in the weight (Weight) and input activation (Act_In) converted to the second precision (Pr2).

If an overflow or underflow error is detected in the weight and input activation (Act_In) converted to the second precision (Pr2) (S205, Y), the weight and input activation (Act_In) are converted to the first precision ( Pr1) can be calculated (S203). On the other hand, if no overflow or underflow error is detected in the weight and input activation (Act_In) converted to the second precision (Pr2) (S205, N), the weight and input activation (Act_In) are not detected. 2 It can be calculated with precision (Pr2) (S206).

On the other hand, when the calculation mode signal (Mode) is the second precision (Pr2) (S201, N), an overflow or underflow error occurs when calculating the weight and input activation (Act_In) of the second precision (Pr2). It is possible to determine whether occurs (S205). If overflow or underflow occurs (S205, Y), weight and input activation (Act_In) are calculated using the first precision (Pr1) (S203), and if overflow or underflow does not occur ( S205, N), weight, and input activation (Act_In) can be calculated with the second precision (Pr2) (S206).

The above description is merely an illustrative explanation of the technical idea of the present embodiment, and those skilled in the art will be able to make various modifications and variations without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are not intended to limit the technical idea of the present embodiment, but rather to explain it, and the scope of the technical idea of the present embodiment is not limited by these examples. The scope of protection of this embodiment should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of rights of this embodiment.

Claims (20)

a weight register that receives and stores weight;
Input Activation Register, which stores input activation;
Receive the first sub-weight of the first precision included in the weight, receive the first sub-input activation of the first precision included in the input activation, and receive the first sub-weight and the first sub-input activation Accordingly, a flexible multiplier for generating result data by multiplying the first sub-weight and the first sub-input activation by either the first precision or a second precision different from the first precision; and
Including a saturating adder that generates a partial total using the result data,
Processing elements.
According to claim 1,
The flexible multiplier is,
a pass decision unit generating a pass decision signal based on the first sub-weight and the first sub-input activation;
a first multiplier that performs a multiplication operation with the first precision;
a second multiplier that performs a multiplication operation with the second precision; and
Comprising a demultiplexer that provides the first sub-weight and the first sub-input activation to one of the first multiplier and the second multiplier according to the pass decision signal,
Processing elements.
According to clause 2,
The pass decision unit,
When the size of at least one of the first sub weight and the first sub input activation is greater than a predetermined first size, the pass decision signal is transmitted to the first sub weight and the first sub input activation to the first multi Generated as a first signal to be provided to the flyer,
When the size of each of the first sub weight and the first sub input activation is smaller than or equal to the first size, the pass decision signal is transmitted to the first sub weight and the first sub input activation to the second multiplier. Generated as a second signal to provide,
Processing elements.
According to clause 2,
The pass decision unit,
According to the operation mode signal, the weight is divided into the first precision or the second precision units to generate the first sub-weight, and the input activation is divided into the first precision or the second precision units, bit division logic generating the first sub-input activation;
a pass selection logic that generates the pass decision signal according to the operation mode signal, the first sub-weight, and the first sub-input activation; and
Containing conversion logic for converting the first sub weight and the precision of the first sub input activation,
Processing elements.
According to clause 2,
The first multiplier is k, the second multiplier is 2k, and k is a natural number,
Processing elements.
According to clause 2,
The first precision is 2N bits, the second precision is N bits, and N is a natural number,
Processing elements.
According to clause 6,
The first precision is INT4, and the second precision is INT2,
Processing elements.
According to clause 2,
The weight includes the first sub-weight and the second sub-weight,
The input activation includes the first sub-input activation and the second sub-input activation,
The flexible multiplier is,
Based on the first sub weight and the first sub input activation, generate a first pass decision signal,
Based on the second sub weight and the second sub input activation, generate a second pass decision signal,
The first path decision signal and the second path decision signal are generated independently of each other,
Processing elements.
According to clause 2,
The weight includes the first sub-weight and the second sub-weight,
The input activation includes the first sub-input activation and the second sub-input activation,
The flexible multiplier is,
Generating the pass decision signal based on the first sub-weight, the second sub-weight, the first sub-input activation, and the second sub-input activation,
Processing elements.
According to claim 1,
The flexible multiplier is,
Comprising a control pipeline that synchronizes reception of the first sub-weight and the first sub-input activation and generation of the result data,
Processing elements.
a weight register that receives and stores weight;
Input Activation Register, which stores input activation;
A flexible multi-processor that generates result data by multiplying the weight and the input activation by either a first precision or a second precision smaller than the first precision, depending on the operation mode signal and the size of the weight and the input activation. pliers; and
Including a saturating adder that generates a partial total using the result data,
Processing elements.
According to claim 11,
The flexible multiplier is,
Error detection logic that checks whether overflow or underflow occurs according to a multiplication operation of the weight and the input activation and generates a detection result;
k first multipliers of the first precision;
2k second multipliers of the second precision; and
Comprising a pass selection logic that selects one of the first multiplier and the second multiplier according to the weight and the size of the input activation,
Processing elements.
According to claim 12,
The pass selection logic is,
When the operation mode signal is associated with the first precision, either the first multiplier or the second multiplier based on whether at least one of the weight and the input activation is greater than a maximum value of the second precision. to choose,
Processing elements.
According to claim 13,
The error detection logic generates a first result when overflow or underflow occurs in the multiplication operation of the weight and the input activation, and generates a first result if no overflow or underflow occurs in the multiplication operation of the weight and the input activation. 2 produces results,
When each of the weight and the input activation is less than the maximum value of the second precision, the pass selection logic is:
If the detection result is the first result, select the first multiplier,
If the detection result is the second result, selecting the second multiplier,
Processing elements.
According to claim 12,
The pass selection logic is,
When the operation mode signal is associated with the second precision, selecting one of the first multiplier and the second multiplier according to the detection result,
Processing elements.
According to claim 15,
The error detection logic generates a first result when overflow or underflow occurs in the multiplication operation of the weight and the input activation, and generates a first result if no overflow or underflow occurs in the multiplication operation of the weight and the input activation. 2 produces results,
The pass selection logic selects the first multiplier when the detection result is the first result, and selects the second multiplier when the detection result is the second result.
Processing elements.
Contains at least one neural core,
The neural core is,
A processing unit that performs calculations,
Includes an L0 memory that stores input and output data of the processing unit,
The processing unit includes a PE array including at least one processing element,
The PE array is,
A flexible multiplier that receives a weight and an input activation and, depending on the size of the weight and the input activation, performs a multiplication operation with a first precision or a second precision smaller than the first precision to generate result data;
Including a saturating adder that receives the result data and generates a partial total,
Neural processing device.
According to claim 17,
The flexible multiplier is,
If the size of at least one of the weight and the input activation is greater than the maximum value of the second precision, multiplying the weight and the input activation by the first precision,
If the size of each of the weight and the input activation is less than or equal to the maximum value of the second precision, multiplying the weight and the input activation by the second precision,
Neural processing device.
According to clause 18,
The weight includes first and second sub-weights, and the input activation includes first and second sub-input activation,
The flexible multiplier is,
According to the first sub-weight and the first sub-input activation, multiplying the first sub-weight and the first sub-input activation by the first precision or the second precision,
Depending on the size of the second sub-weight and the second sub-input activation, multiplying the second sub-weight and the second sub-input activation by the first precision or the second precision,
Neural processing device.
According to clause 18,
The weight includes first and second sub-weights, and the input activation includes first and second sub-input activation,
The flexible multiplier is,
Depending on the respective sizes of the first sub-weight, the second sub-weight, the first sub-input activation, and the second sub-input activation, multiplying the weight and the input activation by the first precision or the second precision calculating,
Neural processing device.

Images