TWI616825B - Neural network unit with output buffer feedback and masking capability - Google Patents

Abstract

The output buffer is used to load N characters and is allocated to N / J mutually exclusive output buffer character groups. The N processing units are allocated to N / J mutually exclusive processing unit groups. Each processing unit includes first and second multiplexing registers, an accumulator and an arithmetic unit. Each multiplexing register includes at least J + 1 inputs and an output. One of the first inputs receives an operand from the memory, and the other input receives the text corresponding to the output buffer text group. The accumulator provides its output to the corresponding output buffer text. The arithmetic unit performs operations on the outputs of the first and second multiplexing registers and the output of the accumulator to generate a result and accumulate the result to the accumulator. The output buffer includes a mask input to control which of the N characters will maintain the original value or be updated with the corresponding accumulator output.

Description

God with output buffer feedback and masking function Via network unit

This application claims the following international priority for the United States provisional application. The full text of these priority cases is incorporated in this case for reference.

This application is related to the following U.S. provisional applications that file simultaneously. The full texts of these related applications are incorporated in this case for reference.

In recent years, artificial neural networks (ANN) have regained attention. These studies are often called similar terms such as deep learning, computer learning, and so on. The increase in the computing power of general-purpose processors has also boosted people's interest in artificial neural networks now, decades later. Recent applications of artificial neural networks include language and image recognition. There seems to be an increasing need for increasing the computational efficiency and efficiency of artificial neural networks.

In view of this, the present invention provides a device including an output buffer and an array of N processing units. The output buffer is used to load N characters. The N characters are allocated to N / J mutually exclusive output buffer text groups. The output buffer text group has J characters of N characters, J is greater than 2, N is at least twice that of J. The N processing units in the array are allocated to N / J mutually exclusive processing unit groups. The processing unit group has J processing units among the N processing units, and each processing unit group corresponds to N / J outputs. One of the buffered text groups, each processing unit includes first and second multiplexing registers, one Accumulator and an arithmetic unit. Each multiplexing register includes at least J + 1 inputs, one output and one control input. One of the J + 1 inputs receives an operand from a memory, and the other J inputs of the J + 1 inputs receive the J characters of the corresponding output buffer text group. The control input is used to control the selection of J + 1 inputs to provide to the output. The accumulator has an output to provide one of the N output buffered characters corresponding to the output buffered characters. The arithmetic unit has first, second, and third inputs. The first and second inputs are used to receive the output of the first and second multiplex registers, respectively. The third input is used to receive the output of the accumulator. For the first, second and third inputs, an operation is performed to generate a result accumulated to the accumulator. The output buffer includes a mask input, which is used to control which characters in the N characters will maintain their original values or be updated by the output of the corresponding accumulator.

The invention also provides a processor. The processor includes an execution unit. The execution unit includes an output buffer and an array of N processing units. The output buffer is used to load N characters. The N characters are allocated to N / J mutually exclusive output buffer text groups. The output buffer text group has J characters of N characters, J is greater than 2, N is at least twice that of J. The N processing units in the array are allocated to N / J mutually exclusive processing unit groups. The processing unit group has J processing units among the N processing units, and each processing unit group corresponds to N / J outputs. One of the buffered text groups, each processing unit includes a first and a second multiplexing register, an accumulator and an arithmetic unit. Each multiplexing register includes at least J + 1 inputs, one output and one control input. One of the J + 1 inputs The first input receives an operand from a memory, and J + 1 inputs The other J inputs in the input receive the J characters of the corresponding output buffer text group. The control input is used to control the selection of J + 1 inputs to provide to the output. The accumulator has an output to provide one of the N output buffered characters corresponding to the output buffered characters. The arithmetic unit has first, second and third inputs, the first and second inputs are used to receive the output of the first and second multiplexer registers respectively, and the third input is used to receive the output of the accumulator. The arithmetic unit For the first, second and third inputs, an operation is performed to generate a result accumulated to the accumulator. The output buffer includes a mask input, which is used to control which characters in the N characters will maintain their original values or be updated by the output of the corresponding accumulator.

The invention also provides a computer program product encoded in at least one non-transitory computer usable medium for use by a computer device. This computer program product includes computer-usable code included in the media to describe a device. This computer can use the code including the first code and the second code. The first code is used to describe an output buffer. This output buffer is used to load N characters. The N characters are allocated to N / J mutually exclusive output buffer text groups. The output buffer text group. With J characters of N characters, J is greater than 2, N is at least twice J. The second code is used to describe an array composed of N processing units. The N processing units in this array are allocated to N / J mutually exclusive processing unit groups. The processing unit group has N processing units. Each of the J processing units in the unit corresponds to one of the N / J output buffer text groups. Each processing unit includes first and second multiplexing registers, an accumulator, and an arithmetic unit. unit. Each multiplexing register includes at least J + 1 inputs, one output and one control input. One of J + 1 inputs An input receives an operand from a memory, and the other J inputs of the J + 1 inputs receive the J characters of the corresponding output buffer text group. The control input is used to control the selection of J + 1 inputs to provide to the output. The accumulator has an output to provide one of the N output buffered characters corresponding to the output buffered characters. The arithmetic unit has first, second and third inputs, the first and second inputs are used to receive the output of the first and second multiplexer registers respectively, and the third input is used to receive the output of the accumulator. The arithmetic unit For the first, second and third inputs, an operation is performed to generate a result accumulated to the accumulator. The output buffer includes a mask input, which is used to control which characters in the N characters will maintain their original values or be updated by the output of the corresponding accumulator.

The specific embodiments used in the present invention will be further described by the following embodiments and drawings.

100‧‧‧ processor

101‧‧‧ instruction fetch unit

102‧‧‧Instruction cache

103‧‧‧Architecture Instructions

104‧‧‧Instruction translator

105‧‧‧microinstructions

106‧‧‧ Rename Unit

108‧‧‧ Reserved Station

112‧‧‧Other execution units

114‧‧‧Memory Subsystem

116‧‧‧General purpose register

118‧‧‧Media Register

121‧‧‧ Neural Network Unit

122‧‧‧Data Random Access Memory

124‧‧‧ Weight Random Access Memory

126‧‧‧Neural Processing Unit

127‧‧‧Control and status register

128‧‧‧Sequencer

129‧‧‧program memory

123,125,131‧‧‧Memory address

133,215,133A, 133B, 215A, 215B‧‧‧Result

202‧‧‧ Accumulator

204‧‧‧ Arithmetic Logic Unit

203,209,217,203A, 203B, 209A, 209B, 217A, 217B‧‧‧Output

205,205A, 205B‧‧‧Register

206,207,211,1811,206A, 206B, 207A, 207B, 211A, 211B, 1811A, 1811B, 711‧‧‧Enter

208,705,208A, 208B‧‧‧Multiplex Register

213,713,803‧‧‧Control input

212‧‧‧Start function unit

242‧‧‧Multiplier

244‧‧‧ Adder

246,246A, 246B‧‧‧product

802‧‧‧ Multiplexer

1104‧‧‧column buffer

1112‧‧‧Start function unit

1400‧‧‧MTNN instruction

1500‧‧‧MFNN instruction

1432, 1532‧‧‧ functions

1402,1502‧‧‧‧Code field

1404‧‧‧src1 field

1406‧‧‧src2

1408, 1508‧‧‧‧gpr field

1412, 1512 ‧‧‧ Immediate field

1422, 1522 ‧‧‧ Address

1424, 1426, 1524 ‧‧‧ data blocks

1428, 1528 ‧‧‧selected columns

1434‧‧‧Control Logic

1504‧‧‧dst field

1602‧‧‧Read Port

1604‧‧‧write port

1606‧‧‧Memory Array

1702‧‧‧port

1704‧‧‧Buffer

1898‧‧‧operator selection logic

1896A‧‧‧Wide Multiplexer

1896B‧‧‧Narrow Multiplexer

242A‧‧‧ Wide Multiplier

242B‧‧‧Narrow Multiplier

244A‧‧‧Wide Adder

244B‧‧‧Narrow Adder

204A‧‧‧wide arithmetic logic unit

204B‧‧‧ Narrow Arithmetic Logic Unit

202A‧‧‧Wide Accumulator

202B‧‧‧Narrow Accumulator

212A‧‧‧Wide Start Function Unit

212B‧‧‧Narrow Start Function Unit

2402‧‧‧ Convolution Kernel

2404‧‧‧Data Array

2406A, 2406B‧‧‧ Data Matrix

2602, 2604, 2606, 2608, 2902, 2912, 2914, 2922, 2924, 2926, 2932, 2934, 2942, 2944, 2952, 2954, 2956, 2923, 2962, 2964

3003‧‧‧ Random bit source

3005‧‧‧random bits

3002‧‧‧ Positive Type Converter and Output Binary Decimal point aligner

3004‧‧‧rounder

3006‧‧‧Multiplexer

3008‧‧‧Standard size compressor and saturator

3012‧‧‧Bit Selection and Saturator

3018‧‧‧corrector

3014‧‧‧Countdown Multiplier

3016‧‧‧Right shifter

3028‧‧‧Standard size transfer value

3022‧‧‧Hyperbolic Tangent Module

3024‧‧‧S Type Module

3026‧‧‧Soft add module

3032‧‧‧Multiplexer

3034‧‧‧Symbol Restorer

3036‧‧‧size converter and saturator

3037‧‧‧Multiplexer

3038‧‧‧Output Register

3402‧‧‧Multiplexer

3401‧‧‧Neural Processing Unit pipeline stage

3404‧‧‧ Decoder

3412, 3414, 3418 ‧‧‧ micro-computing

3416‧‧‧Micro Instructions

3422‧‧‧Mode Indicator

3502‧‧‧ Time-frequency generation logic

3504‧‧‧ Time-frequency reduction logic

3514‧‧‧Interface logic

3522‧‧‧Data Random Access Memory Buffer

3524‧‧‧weighted random access memory buffer

3512 ‧ ‧ mitigation indicators

3802‧‧‧Program Counter

3804‧‧‧Loop counter

3806‧‧‧Iteration counter

3912, 3914, 3916 ‧‧‧ fields

4901‧‧‧ Neural Processing Unit Group

4903‧‧‧Mask

4905,4907,5599‧‧‧Enter

The first diagram is a block diagram of a processor including a neural network unit (NNU).

The second diagram is a block diagram of a neural processing unit (NPU), one of the first diagrams.

The third diagram is a block diagram showing the use of N multiplexing registers of the N neural processing units of the neural network unit of the first diagram, for a row of data text obtained from the data random access memory of the first diagram Performs the operation of a rotator or circular shifter of the same N characters.

The fourth diagram is a table showing a program stored in the program memory of the neural network unit of the first diagram and executed by the neural network unit.

The fifth diagram is a timing diagram showing the execution of the program of the fourth diagram by the neural network unit.

The sixth diagram A is a block diagram showing that the neural network unit of the first diagram executes the program of the fourth diagram.

The sixth diagram B is a flowchart showing the processor of the first diagram executing a framework program to use a neural network unit to perform a typical multiply accumulate activation function operation of a neuron associated with a hidden layer of an artificial neural network. It works as if executed by the program in Figure 4.

The seventh diagram is a block diagram showing another embodiment of the neural processing unit of the first diagram.

The eighth diagram is a block diagram showing another embodiment of the neural processing unit of the first diagram.

The ninth figure is a table showing a program stored in the program memory of the neural network unit of the first figure and executed by the neural network unit.

The tenth diagram is a timing chart showing the execution of the program of the ninth diagram by the neural network unit.

The eleventh figure is a block diagram showing an embodiment of the neural network unit of the first figure. In the embodiment of FIG. 11, a neuron system is divided into two parts, that is, an activation function unit part and an arithmetic logic unit part (this part also includes a shift register part), and each activation function unit part is composed of multiple Part of the arithmetic logic unit is shared.

The twelfth figure is a timing chart showing that the neural network unit of the eleventh figure executes the program of the fourth figure.

The thirteenth figure is a timing chart showing the neural network unit of the eleventh figure executing the program of the fourth figure.

The fourteenth figure is a block diagram showing a move to neural network (MTNN) architecture instruction and its operation corresponding to the neural network unit of the first figure.

The fifteenth figure is a block diagram showing a move to neural network (MTNN) architecture instruction and its operation corresponding to the neural network unit of the first figure.

The sixteenth figure is a block diagram showing an embodiment of the data random access memory of the first figure.

The seventeenth figure is a block diagram showing an embodiment of the weighted random access memory and a buffer of the first figure.

The eighteenth figure is a block diagram showing a dynamically configurable neural processing unit, one of the first figures.

The nineteenth figure is a block diagram showing the embodiment according to the eighteenth figure, using the 2N multiplexing registers of the N neural processing units of the neural network unit of the first figure. The random access memory obtains a row of data text and performs the same operation as a rotator.

Figure 20 is a table showing a program stored in the program memory of the neural network unit of the first figure and executed by the neural network unit. The neural network unit has an embodiment as shown in figure 18. Neural Processing Unit shown.

The twenty-first diagram is a timing chart showing a neural network unit executing the program of the twenty-first diagram. The neural network unit has the nerve shown in the eighteenth diagram. The processing unit is executed in a narrow configuration.

The twenty-second figure is a block diagram showing the neural network unit of the first figure. This neural network unit has the neural processing unit shown in the eighteenth figure to execute the program of the twentieth figure.

The twenty-third figure is a block diagram showing another embodiment of the dynamically configurable neural processing unit in the first figure.

The twenty-fourth figure is a block diagram showing an example of a data structure used by the neural network unit of the first figure to perform a convolution operation.

The twenty-fifth figure is a flowchart showing that the processor of the first figure executes a framework program to perform a convolution operation of a convolution kernel using a neural network unit according to the data array of the twenty-fourth figure.

Figure 26A is a program list of a neural network unit program. This neural network unit program uses the convolution kernel of Figure 24 to perform a convolution operation of a data matrix and writes it back to random weights. Access memory.

Figure 26B is a block diagram showing an embodiment of some fields of the control register of the neural network unit in the first figure.

The twenty-seventh figure is a block diagram showing an example of the weighted random access memory filled with input data in the first figure. This input data is performed by the neural network unit in the first figure. ).

The twenty-eighth figure is a program list of a neural network unit program. This neural network unit program performs the common source operation of the input data matrix of the twenty-seventh figure and writes it back to the weight random access memory.

Figure 29A is a block diagram showing an embodiment of the control register of the first figure.

Figure 29B is a block diagram showing another embodiment of the control register of the first figure.

Figure 29C is a block diagram showing one embodiment of storing the reciprocal of Figure 29A in two parts.

The thirtieth figure is a block diagram showing an embodiment of an activation function unit (AFU) of the second figure.

The thirty-first figure is an example showing the operation of the activation function unit of the thirty-first figure.

The thirty-second figure is the second example showing the operation of the activation function unit of the thirty-third figure.

The thirty-third figure is a third example showing the operation of the activation function unit of the thirty-third figure.

The thirty-fourth figure is a block diagram showing some details of the processor and the neural network unit of the first figure.

Figure 35 is a block diagram showing a processor with a variable rate neural network unit.

Figure 36A is a timing diagram showing an example of the operation of a processor with a neural network unit in one of the general modes. This general mode operates at the main time frequency.

Figure 36B is a timing diagram showing an example of the operation of a processor with a neural network unit in a relaxation mode. The frequency of operation of the relaxation mode is lower than the main frequency.

Figure 37 is a flowchart showing the operation of the processor in Figure 35.

The thirty-eighth figure is a block diagram showing the sequence of the neural network units in detail.

Figure 39 is a block diagram showing some fields of the control and status register of the neural network unit.

The fortieth diagram is a block diagram showing an example of an Elman time recurrent neural network (RNN).

The forty-first diagram is a block diagram showing the random access memory and weight random access of the neural network unit when the neural network unit performs the calculation of the Elman time recurrent neural network associated with the forty diagram. An example of data allocation in memory.

Figure 42 is a table showing one of the programs stored in the neural network unit. This program is executed by the neural network unit and uses data and weights according to the configuration of figure 41 to achieve Elman time recurrent neural network

The forty-third diagram is a block diagram showing an example of Jordan time recurrent neural network.

The forty-fourth figure is a block diagram showing that when the neural network unit performs the calculation of the Jordan time recursive neural network associated with the forty-third figure, the data of the neural network unit is randomly stored in memory and weights are stored randomly. Take an example of data configuration in memory.

The forty-fifth figure is a table showing one of the programs stored in the neural network unit. This program is executed by the neural network unit and uses data and weights according to the configuration of the forty-fourth figure to achieve Jordan Time Recurrent Neural Network.

The forty-sixth figure is a block diagram showing an embodiment of a long short term memory (LSTM) cell.

The forty-seventh diagram is a block diagram showing when the neural network unit performs the association In the calculation of the long-term and short-term memory cell in Figure 46, an example of the data allocation in the data of the neural network unit and the weighted random access memory.

Figure 48 is a table showing a program stored in the neural network unit's program memory. This program is executed by the neural network unit and uses data and weights according to the configuration of figure 47 to achieve association. Calculation of long-term and short-term memory cells.

The forty-ninth figure is a block diagram showing an embodiment of a neural network unit. The neural processing unit group in this embodiment has output buffer masking and feedback capabilities.

Figure 50 is a block diagram showing the random access memory of the data of the neural network unit of Figure 49 when the neural network unit performs calculations related to the long-term and short-term memory cell layer of Figure 46. , An example of data allocation in weighted random access memory and output buffer.

Figure 51 is a table showing a program stored in the neural network unit's program memory. This program is executed by the neural network unit of Figure 49 and uses the data according to the configuration of Figure 50. Weights to achieve calculations related to long- and short-term memory cells.

The fifty-second figure is a block diagram showing an embodiment of a neural network unit. The neural processing unit group in this embodiment has output buffer masking and feedback capabilities, and shares an activation function unit.

Figure 53 is a block diagram showing the random access memory of the data of the neural network unit of Figure 49 when the neural network unit performs calculations related to the long-term and short-term memory cell layer of Figure 46. Another embodiment of the data allocation in the memory, the weighted random access memory and the output buffer.

Figure 54 is a table showing a program stored in the neural network unit's program memory. This program is executed by the neural network unit in Figure 49 and uses the data according to the configuration of Figure 53. And weights to achieve calculations related to long- and short-term memory cells.

The fifty-fifth figure is a block diagram showing a part of a neural processing unit according to another embodiment of the present invention.

The fifty-sixth diagram is a block diagram showing the calculation of the neural network unit when the neural network unit performs calculations of the Jordan time recurrent neural network associated with the forty-third diagram and uses the embodiment of the fifty-fifth diagram An example of data allocation in data random access memory and weighted random access memory.

Figure 57 is a table showing one of the programs stored in the neural network unit. This program is executed by the neural network unit and uses data and weights according to the configuration of Figure 56 to achieve Jordan. Time recurrent neural network.

Processor with architecture neural network unit

The first diagram is a block diagram of a processor 100 including a neural network unit (NNU) 121. As shown in the figure, the processor 100 includes an instruction fetch unit 101, an instruction cache 102, an instruction translator 104, a rename unit 106, multiple reservation stations 108, multiple media registers 118, multiple The general purpose register 116 includes a plurality of execution units 112 and a memory subsystem 114 outside the aforementioned neural network unit 121.

The processor 100 is an electronic device as an integrated circuit Central Processing Unit. The processor 100 receives the input digital data, processes the data according to the instructions fetched from the memory, and generates the processing result of the operation instructed by the instructions as its output. The processor 100 may be used in a desktop computer, a mobile device, or a tablet computer, and used in applications such as computing, word processing, multimedia display, and web browsing. The processor 100 may be provided in an embedded system to control various devices including devices, mobile phones, smart phones, vehicles, and industrial controllers. The central processing unit is an electronic circuit (ie, hardware) that executes instructions of a computer program (or computer application or application) by performing operations on data, including arithmetic, logic, and input / output. Integrated circuits are a group of electronic circuits made from a small semiconductor material, usually silicon. Integrated circuits are also commonly used to represent wafers, microchips, or dies.

The instruction fetching unit 101 controls the operation of fetching the architectural instructions 103 to the instruction cache 102 from the system memory (not shown). The instruction fetch unit 101 provides a fetch address to the instruction cache 102 to specify the memory address of the cache line of the architecture instruction byte of the processor 100 to fetch to the cache memory 102. The selection of the fetch address is based on the current value of the instruction index (not shown) of the processor 100 or the program counter. Generally, the program counter is incremented according to the instruction size until a control instruction such as a branch, call, or return appears in the instruction stream, or an exception condition such as an interrupt, a trap, an exception, or an error occurs. Update the program counter with non-sequential addresses such as branch target address, return address, or exception vector. In a word, the program counter is updated according to the execution instructions of the execution units 112/121. The program counter can also be updated when an exception condition is detected, such as when the instruction translator 104 encounters an undefined Instruction 103, meaning the instruction set architecture of the processor 100.

The instruction cache 102 stores architecture instructions 103 obtained from a system memory coupled to the processor 100. These architectural instructions 103 include a move to neural network (MTNN) instruction and a move from neural network (MFNN) instruction, as described later. In one embodiment, the architecture instruction 103 is an instruction of the x86 instruction set architecture, and MTNN instruction and MFNN instruction are added. In this disclosure, an x86 instruction set architecture processor is understood to be a processor that produces the same result as the Intel® 80386® processor at the instruction set architecture layer when executing the same mechanical language instructions. However, other instruction set architectures, such as Advanced Reduced Instruction Set Machine Architecture (ARM), Sun's Scalable Processor Architecture (SPARC), or Enhanced Reduced Instruction Set Performance Computing Performance Optimization Architecture (PowerPC), It can also be used in other embodiments of the present invention. The instruction cache 102 provides the architectural instruction 103 to the instruction translator 104 to translate the architectural instruction 103 into a micro instruction 105.

The microinstruction 105 is provided to the renaming unit 106 and is finally executed by the execution unit 112/121. These micro-instructions 105 implement architectural instructions. In a preferred embodiment, the instruction translator 104 includes a first part for translating frequently executed and / or relatively less complex architectural instructions 103 into microinstructions 105. The instruction translator 104 also includes a second part having a microcode unit (not shown). The microcode unit has a microcode memory loaded with microcode instructions to execute complex and / or rarely used instructions in the architecture instruction set. The microcode unit also includes a microsequencer to provide a non-architecture microprogram counter (micro-PC) to the microcode memory. In a preferred embodiment, these micro-instructions are translated into micro-instructions 105 via a micro-translator (not shown). Selector by micro Whether the code unit currently has control right, the microinstruction 105 from the first part or the second part is selected and provided to the rename unit 106.

The renaming unit 106 renames the architecture register specified by the architecture instruction 103 to the physical register of the processor 100. According to a preferred embodiment, the processor 100 includes a rearrangement buffer (not shown). The renaming unit 106 assigns the reordering buffer items to each microinstruction 105 in a program order. In this way, the processor 100 can remove the micro instruction 105 and its corresponding architecture instruction 103 according to the program sequence. In one embodiment, the media register 118 has a width of 256 bits, and the general purpose register 116 has a width of 64 bits. In one embodiment, the media register 118 is an x86 media register, such as an advanced vector extension (AVX) register.

In one embodiment, each item of the rearrangement buffer has a storage space to store the result of the microinstruction 105. In addition, the processor 100 includes an architecture register file. The architecture register file has a physical register corresponding to each architecture register, such as the media register 118, the general purpose register 116, and other architecture registers. Device. (For a preferred embodiment, for example, the media register 118 and the general-purpose register 116 are different in size, and separate register files can be used to correspond to these two registers.) For micro Each source operand of an architecture register is specified in the instruction 105. The rename unit will use the reordering buffer directory of the latest microinstruction in the old microinstruction 105 written in the architecture register and fill in the microinstruction 105. The source operand field. When the execution unit 112/121 completes the execution of the microinstruction 105, the execution unit 112/121 writes its result into the reorder buffer item of the microinstruction 105. When the microinstruction 105 is removed, the removal unit (not shown) will write the result from the rearranged buffer field of this microinstruction Register of the physical register file. This physical register file is associated with the register for the architectural purpose specified by the removal instruction 105.

In another embodiment, the processor 100 includes a physical register file, which has more physical registers than the number of architecture registers. However, the processor 100 does not include an architecture register file. , And the result storage space is not included in the rearrangement buffer item. (For a preferred embodiment, because the size of the media register 118 and the general-purpose register 116 are different, a separate register file can be used to correspond to these two registers.) This processor 100 does It includes an index table with corresponding indexes of each architecture register. For each operand in the microinstruction 105 where the architecture register is specified, the rename unit will use an indicator pointing to a free register in the physical register file to fill in the destination operand field in the microinstruction 105. If there is no free register in the physical register file, the renaming unit 106 will temporarily suspend the pipeline. For each source operand in the microinstruction 105 where the architecture register is specified, the rename unit will use a pointer to the physical register file and assign it to the latest microinstruction in the old microinstruction 105 written to the architecture register. The index of the register is filled in the source operand field in the microinstruction 105. When the execution unit 112/121 finishes executing the micro instruction 105, the execution unit 112/121 writes the result to the register of the destination operand field of the micro instruction 105 in the physical register file. When the microinstruction 105 is removed, the removal unit will copy the value of the destination operand field of the microinstruction 105 to the indicator associated with the index table of the architectural purpose register specified by the removal microinstruction 105.

The reservation station 108 loads the micro-instructions 105 until the micro-instructions are issued to the execution units 112/121 for preparation for execution. When one When all the source operands of a microinstruction 105 are available and the execution units 112/121 are also available for execution, the preparation for issuing the microinstruction 105 is completed. The execution unit 112/121 receives the register source operand from the rearrangement buffer or the structure register file described in the first embodiment, or the physical register file described in the second embodiment. In addition, the execution unit 112/121 can directly receive the register source operand through the result transmission bus (not shown). In addition, the execution unit 112/121 may receive the immediate operand specified by the micro instruction 105 from the reservation station 108. The MTNN and MFNN architecture instructions 103 include an immediate operand to specify the function to be performed by the neural network unit 121. This function is provided by one or more micro-instructions 105 generated by the translation of the MTNN and MFNN architecture instructions 103, as described later. .

The execution unit 112 includes one or more load / store units (not shown). The memory subsystem 114 loads data and stores the data in the memory subsystem 114. In a preferred embodiment, the memory subsystem 114 includes a memory management unit (not shown). The memory management unit may include, for example, a plurality of lookaside buffers, a table movement ( table walk) unit, a level one data cache (and instruction cache 102), a level two unified cache, and a bus interface unit as an interface between the processor 100 and the system memory. In the embodiment, the processor 100 in the first figure is represented by one of a plurality of processing cores of a multi-core processor, and the multi-core processor shares a last-level cache memory. The execution unit 112 may include multiple integer units, multiple media units, multiple floating point units, and a branch unit.

The neural network unit 121 includes a weighted random access memory (RAM) 124, a data random access memory 122, and N nerve units. A processing unit (NPU) 126, a program memory 129, a sequencer 128, and a plurality of control and status registers 127. These neural processing units 126 are conceptually functioning as neurons in a neural network. The weight random access memory 124, the data random access memory 122, and the program memory 129 can be written to and read from the MTNN and MFNN architecture instructions 103, respectively. The weight random access memory 124 is arranged in W rows, each row having N weight characters, and the data random access memory 122 is arranged in D rows, each row having N data characters. Each data text and each weight text are a plurality of bits. In a preferred embodiment, they can be 8 bits, 9 bits, 12 bits, or 16 bits. Each data text is used as the output value (sometimes expressed by the activation value) of a neuron in the previous layer in the network, and each weight text is used as the weight of a link in the network that is associated with a neuron entering the current layer of the network . Although in many applications of the neural network unit 121, the text or operand loaded in the weight random access memory 124 is actually the weight associated with the connection into a neuron, it should be noted that in the neural network In some applications of the unit 121, the words loaded in the weight random access memory 124 are not weighted, but because these words are stored in the weight random access memory 124, they are still expressed in terms of "weight words". For example, in some applications of the neural network unit 121, for example, examples of convolution operations of the twenty-fourth to twenty-sixth graphs A or examples of common source operations of the twenty-seventh to twenty-eighth graphs The weight random access memory 124 will load objects other than weights, such as elements of a data matrix (such as image pixel data). Similarly, although in many applications of the neural network unit 121, the text or operand loaded in the data random access memory 122 is essentially the output value or activation value of the neuron, However, it should be noted that in some applications of the neural network unit 121, the text loaded in the data random access memory 122 is not the same, but because these texts are stored in the data random access memory 122, they are still Expressed in terms of "text text". For example, in some applications of the neural network unit 121, such as the example of the convolution operation of the twenty-fourth to twenty-sixth A graphs, the data random access memory 122 will load non-neuronal outputs, such as Elements of a convolution kernel.

In one embodiment, the neural processing unit 126 and the sequencer 128 include combination logic, sequence logic, state machine, or a combination thereof. An architecture instruction (such as MFNN instruction 1500) loads the content of the state register 127 into one of the general purpose registers 116 to confirm the state of the neural network unit 121. For example, the neural network unit 121 has been completed from the program memory 129 A command or a program operates, or the neural network unit 121 is free to receive a new command or start a new neural network unit program.

The number of neural processing units 126 can be increased according to demand, and the width and depth of the random storage memory 124 and the data random access memory 122 can be adjusted and expanded accordingly. For a preferred embodiment, the weight random access memory 124 is larger than the data random access memory 122, because there are many links in a typical neural network layer, so a larger storage space is needed The weight of each neuron. This document discloses many embodiments regarding the size of data and weight text, the size of weight random access memory 124 and data random access memory 122, and the number of different neural processing units 126. In one embodiment, the neural network unit 121 has a size of 64 KB (8192 bits x 64 columns). Data random access memory 122, a weighted random access memory 124 with a size of 2MB (8192 bits x 2048 rows), and 512 neural processing units 126. The neural network unit 121 is manufactured by Taiwan Semiconductor Manufacturing Corporation (TSMC) in a 16 nanometer manufacturing process, and its area is approximately 3.3 mm square.

The sequencer 128 fetches instructions from the program memory 129 and executes them. The operations performed by the sequencer 128 include generating an address and a control signal to the data random access memory 122, a weight random access memory 124, and a neural processing unit 126. The sequencer 128 generates a memory address 123 and a read command and provides it to the data random access memory 122, so as to select one of the N data characters in the D rows and provide it to the N neural processing units 126. The sequencer 128 also generates a memory address 125 and a read command and provides them to the weighted random access memory 124, so as to select one of the W weighted N weighted text and provide it to the N neural processing units 126. The sequence of the addresses 123, 125 generated by the sequencer 128 and provided to the neural processing unit 126 determines the "connection" between the neurons. The sequencer 128 also generates a memory address 123 and a write command to the data random access memory 122, so that one of the N data characters in D rows is selected for writing by the N neural processing units 126. Into. The sequencer 128 also generates a memory address 125 and a write command to provide the weighted random access memory 124, so that one of the N weighted text in W columns is selected for writing by the N neural processing units 126. Into. The sequencer 128 also generates a memory address 131 to the program memory 129 to select a neural network unit instruction provided to the sequencer 128. This section will be described in the subsequent sections. The memory address 131 corresponds to a program counter (not shown). The sequencer 128 is usually The program counter is incremented according to the position sequence of the program memory 129, unless the sequencer 128 encounters a control instruction, such as a loop instruction (see Figure 26A), in which case the sequence is The controller 128 updates the program counter to the target address of the control instruction. The sequencer 128 also generates control signals to the neural processing unit 126, instructing the neural processing unit 126 to perform various operations or functions, such as initialization, arithmetic / logical operations, rotation / shift operations, activation functions, and write back Operations, related examples will be explained in more detail in the subsequent chapters (refer to the micro-operation 3418 shown in Figure 34).

The N neural processing units 126 generate N result texts 133, and these result texts 133 can be written back to a row of the weight random access memory 124 or the data random access memory 122. In a preferred embodiment, the weight random access memory 124 and the data random access memory 122 are directly coupled to the N neural processing units 126. Further, the weight random access memory 124 and the data random access memory 122 are transferred to these neural processing units 126 and are not shared with other execution units 112 in the processor 100. These neural processing units 126 can continuously A row is obtained and completed from one or both of the weighted random access memory 124 and the data random access memory 122 in each time-frequency cycle. For a preferred embodiment, it can be processed in a pipeline manner. In one embodiment, each of the data random access memory 122 and the weighted random access memory 124 can provide 8192 bits to the neural processing unit 126 in each time-frequency period. These 8192 bits can be treated as 512 16-bytes or 1024 8-bytes, as described later.

The size of the data set processed by the neural network unit 121 is It is not limited to the size of the weight random access memory 124 and the data random access memory 122, but is only limited by the size of the system memory. This is because data and weight can be accessed in the system memory and weight random access. The memory 124 and the data random access memory 122 are moved by using MTNN and MFNN instructions (for example, through the media register 118). In one embodiment, the data random access memory 122 is given a dual port, so that when the data random access memory 122 reads data text or writes data text to the data random access memory 122, Write data text to data random access memory 122. In addition, the large memory hierarchy of the memory subsystem 114 including the cache memory can provide a very large data bandwidth for data transmission between the system memory and the neural network unit 121. In addition, in a preferred embodiment, the memory subsystem 114 includes a hardware data pre-fetcher, which tracks memory access patterns, such as neural data and weights loaded by the system memory, and caches the data. The hierarchical structure performs data pre-fetching to facilitate high-bandwidth and low-latency transmission during transmission to the weighted random access memory 124 and data random access memory 122.

Although in the embodiment herein, one of the operands provided by the weight memory to each neural processing unit 126 is labeled as a weight, this term is commonly used in neural networks, but it should be understood that these operands can also be other and Calculate related types of data, and their speed can be increased by these devices.

The second diagram is a block diagram of the neural processing unit 126, one of the first diagrams. As shown in the figure, the operation of the neural processing unit 126 can perform many functions or operations. In particular, this neural processing unit 126 can operate as a neuron or node in an artificial neural network to perform typical multiply-accumulate-plus functions or operations. That is, in general, the neural network unit 126 (neuron) is used to: (1) receive an input value from each neuron with a connection to it, this connection usually but not necessarily from an artificial neural network (2) Multiply each output value by a corresponding weight value associated with one of its links to produce a product; (3) Add all products to produce a total; (4) Perform a start on this total Function to generate neuron output. However, unlike traditional methods that need to perform all multiplication operations associated with all connected inputs and sum their products, each neuron of the present invention can perform weighted multiplication operations associated with one of the connected inputs within a given time-frequency period. And the product is added (accumulated) to the accumulated value of the product of the connected inputs performed in the time-frequency period before the time point. Assume that there are a total of M links connected to this neuron. After the M products are summed up (approximately M time-frequency cycles are required), this neuron will execute a start function on this accumulated number to generate an output or result. The advantage of this method is that it can reduce the number of multipliers required, and only a small, simpler and faster adder circuit (such as an adder using two inputs) is needed in the neuron, without the need for Use an adder required to add up the product of all linked inputs or even a subset of them. This method is also conducive to using a large number (N) of neurons (neural processing unit 126) in the neural network unit 121. In this way, after about M time-frequency cycles, the neural network unit 121 can generate this large number (N) Neuron output. Finally, for a large number of different connection inputs, the neural network unit 121 composed of these neurons can be effectively implemented as an artificial neural network layer. In other words, if It is because the number of M in different layers increases or decreases, the number of time-frequency cycles required to generate the memory cell output will increase or decrease accordingly, and resources (such as multipliers and accumulators) will be fully utilized. In contrast, for smaller M values in the traditional design, some multipliers and adders cannot be used. Therefore, according to the number of connected outputs of the neural network unit, the embodiment described herein has both advantages of flexibility and efficiency, and can provide extremely high performance.

The neural processing unit 126 includes a register 205, a dual-input multiplexing register 208, an arithmetic logic unit (ALU) 204, an accumulator 202, and an activation function unit (AFU) 212. The register 205 receives a weighted text 206 from the weighted random access memory 124 and provides its output 203 in a subsequent time-frequency period. The multiplexer register 208 selects one of the two inputs 207, 211 to store in its register and provides it to its output 209 in a subsequent time-frequency period. Input 207 receives the data text from the data random access memory 122. The other input 211 receives the output 209 from the neighboring neural processing unit 126. The neural processing unit 126 shown in the second figure is labeled as the neural processing unit J among the N neural processing units shown in the first figure. That is, the neural processing unit J is a representative example of the N neural processing units 126. For a preferred embodiment, the input 211 of the multiplexing register 208 of the J example of the neural processing unit 126 receives the output 209 of the multiplexing register 208 of the J-1 example of the neural processing unit 126, and the neural The output 209 of the multiplexing register 208 of the processing unit J is provided to the input 211 of the multiplexing register 208 of the J + 1 example of the neural processing unit 126. In this way, the multiplexing registers 208 of the N neural processing units 126 can work together, such as the spinner or cyclic shifter of the same N characters, which will be described in more detail in the subsequent third figure. Multiplex register 208 uses A control input 213 controls which of the two inputs is selected by the multiplexer register 208 to be stored in its register and subsequently provided to the output 209.

The arithmetic logic unit 204 has three inputs. One of the inputs receives the weight text 203 from the register 205. The other input receives the output 209 of the multiplexer register 208. The other input receives the output 217 of the accumulator 202. The arithmetic logic unit 204 performs arithmetic and / or logic operations on its input to generate a result to provide its output. For a preferred embodiment, the arithmetic and / or logical operations performed by the arithmetic logic unit 204 are specified by instructions stored in the program memory 129. For example, the multiply-accumulate instruction in the fourth figure specifies a multiply-accumulate operation, that is, the result 215 will be the product of the accumulator 202 value 217 and the weight text 203 and the multiplexer register 208 output 209 data text total. However, other operations can also be specified. These operations include but are not limited to: result 215 is the value passed by the multiplexer register 209; result 215 is the value passed by the weight text 203; result 215 is the zero value; result 215 is the accumulator 202 The sum of the value 217 and the weight 203; the result 215 is the sum of the value 217 of the accumulator 202 and the multiplexer output 209; the result 215 is the maximum of the value 217 of the accumulator 202 and the weight 203; the result 215 is the accumulator 202 is the maximum value of 217 and multiplexer register 209.

The arithmetic logic unit 204 provides its output 215 to the accumulator 202 for storage. The arithmetic logic unit 204 includes a multiplier 242 to perform a multiplication operation on the weight text 203 and the data text output from the multiplexer register 208 to generate a product 246. In one embodiment, the multiplier 242 multiplies two 16-bit operands to produce a 32-bit result. The arithmetic logic unit 204 also includes an adder 244 at the output 217 of the accumulator 202 The product 246 is added to generate a total, which is the result 215 of the accumulation operation stored in the accumulator 202. In one embodiment, the adder 244 is a 41-bit value 217 of the accumulator 202 and a 32-bit result of the multiplier 242 to generate a 41-bit result. In this way, the neural processing unit 126 can use the rotator characteristics of the multiplexer register 208 during a plurality of time-frequency periods to complete the product sum operation of the neurons required by the neural network. The arithmetic logic unit 204 may also include other circuit elements to perform other arithmetic / logical operations as described above. In one embodiment, the second adder subtracts the weight text 203 from the data text output 209 of the multiplexer register 208 to generate a difference, and the adder 244 adds the difference to the output 217 of the accumulator 202. Value to produce a result 215, which is the accumulated result in the accumulator 202. In this way, during a plurality of time-frequency periods, the neural processing unit 126 can achieve the sum of the difference values. In a preferred embodiment, although the weight text 203 and the data text 209 have the same size (in bits), they may also have different positions of the decimal point, as described later. In a preferred embodiment, the multiplier 242 and the adder 244 are integer multipliers and adders. Compared with an arithmetic logic unit using floating-point arithmetic, the arithmetic logic unit 204 has low complexity, small size, and fast speed. And the advantages of low energy consumption. However, in other embodiments of the present invention, the arithmetic logic unit 204 may also perform floating-point operations.

Although only one multiplier 242 and an adder 244 are shown in the arithmetic logic unit 204 of the second figure, according to a preferred embodiment, the arithmetic logic unit 204 further includes other components to perform the aforementioned different operations. For example, the arithmetic logic unit 204 may include a comparator (not shown) to compare the accumulator 202 with a data / weight text. And a multiplexer (not shown) selects the larger (maximum) value from the two values specified by the comparator and stores it in the accumulator 202. In another example, the arithmetic logic unit 204 includes selection logic (not shown), using a data / weight text to skip the multiplier 242, so that the adder 224 adds the data / weight text to the value 217 of the accumulator 202 It is stored in the accumulator 202 to generate a total. These additional operations will be explained in more detail in subsequent chapters such as Figures 18 to 29A, and these operations are also helpful for the implementation of convolution operations and common source operations.

The start function unit 212 receives the output 217 of the accumulator 202. The startup function unit 212 executes a startup function on the output of the accumulator 202 to generate a result 133 of the first figure. Generally speaking, the activation function in the neurons of the intermediary layer of the artificial neural network can be used to standardize the total after multiplying and accumulating, especially in a non-linear manner. In order to "normalize" the cumulative total, the activation function of the current neuron will generate a result value within the range of values expected to be received by other neurons connected to the current neuron. (The standardized result is sometimes called "startup". In this article, startup is the output of the current node, and the receiving node will multiply this output by a weight associated with the connection between the output node and the receiving node to produce a product. , And this product is added to the product of other inputs associated with this receiving node.) For example, if the receiving / connected neuron expects to receive an input value between 0 and 1, the output neuron The cumulative total beyond the range of 0 and 1 may need to be squeezed and / or adjusted non-linearly (eg, shifted up to convert negative values to positive values) to fall within this expected range. Therefore, the operation performed by the activation function unit 212 on the value 217 of the accumulator 202 will bring the result 133 to a known range. The results 133 of the N neural execution units 126 can all be written back to the data random access memory 122 or the weight random access memory 124 simultaneously. In a preferred embodiment, the startup function unit 212 is used to execute multiple startup functions. For example, an input from the control register 127 selects one of these startup functions to be executed on the output 217 of the accumulator 202. . These activation functions may include, but are not limited to, step functions, correction functions, sigmoid functions, hyperbolic tangent functions, and soft addition functions (also known as smooth correction functions). The analytic formula of the soft addition function is f (x) = ln (1 + e ^x ), which is the natural logarithm of the sum of 1 and e ^x , where "e" is Euler's number, and x is this Function input 217. In a preferred embodiment, the start-up function may also include a pass-through function, which directly passes the value 217 or a part of the accumulator 202, as described later. In one embodiment, the circuit of the activation function unit 212 executes the activation function in a single time-frequency period. In an embodiment, the activation function unit 212 includes a plurality of forms, which receive an accumulated value and output a value. For some activation functions, such as an S-type function, a double-taken tangent function, and a soft addition function, the value will be approximately The value provided by the real startup function.

For a preferred embodiment, the width (in bits) of the accumulator 202 is greater than the width of the output 133 of the activation function 212. For example, in an embodiment, the width of the accumulator is 41 bits, to avoid the situation of accumulating up to a maximum of 512 32-bit products (this part in the subsequent chapters corresponds to the position shown in Figure 30). There will be a more detailed explanation) loss of accuracy, and the width of the result 133 is 16 bits. In an embodiment, in the subsequent time-frequency cycle, the activation function unit 212 passes the other unprocessed parts of the accumulator 202 output 217, and writes these parts back to the data random access memory 122 or weight random storage. Take the memory 124. This part will be explained in more detail in the subsequent chapter corresponding to the eighth figure. In this way, the value of the unprocessed accumulator 202 can be loaded back to the media register 118 through the MFNN instruction, so that the instructions executed by the other execution units 112 of the processor 100 can perform the complex startup that the function unit 212 cannot perform. A function, such as the common softmax function, is also called a normalized exponential function. In an embodiment, the instruction set architecture of the processor 100 includes executing an instruction of this exponential function, which is usually expressed as e ^x or exp (x). This instruction can be used by other execution units 112 of the processor 100 to enhance the soft start Function execution speed.

In one embodiment, the neural processing unit 126 is a pipeline design. For example, the neural processing unit 126 may include a register of the arithmetic logic unit 204, such as a register located between the multiplier and the adder and / or other circuits of the arithmetic logic unit 204. The neural processing unit 126 may further include A register that loads the output of the start function function 212. Other embodiments of the neural processing unit 126 will be described in subsequent chapters.

The third diagram is a block diagram showing the use of N multiplexing registers 208 of N neural processing units 126 of the neural network unit 121 of the first diagram to obtain data from the random access memory 122 of the data of the first diagram One row of data characters 207 performs the operation of a rotator or a circular shifter of the same N characters. In the embodiment of the third figure, N is 512. Therefore, the neural network unit 121 has 512 multiplexing registers 208, labeled 0 to 511, corresponding to 512 neural processing units 126, respectively. Each multiplexer register 208 will receive data random access records Corresponding data text 207 on one of the D columns of the memory 122. In other words, multiplexer register 0 will receive data text 0 from 122 rows of data random access memory, multiplexer register 1 will receive data text 1 from 122 rows of data random access memory, and multiplex temporarily The device 2 will receive the data text 2 from the 122 rows of the data random access memory, and so on, and the multiplexer register 511 will receive the data text 511 from the 122 rows of the data random access memory. In addition, multiplexer register 1 will receive output 209 of multiplexer register 0 as another input 211, and multiplexer register 2 will receive output 209 of multiplexer register 1 as another input 211. Register 3 will receive output 209 of multiplex register 2 as another input 211, and so on, multiplex register 511 will receive output 209 of multiplex register 510 as another input 211, and more The working register 0 will receive the output 209 of the multiplexing register 511 as the other input 211. Each multiplexer register 208 receives a control input 213 to control whether it selects data text 207 or cyclic input 211. In one mode of this operation, the control input 213 controls each multiplexing register 208 to select the data text 207 to be stored in the register in a first time-frequency period and provided to the arithmetic logic unit 204 in the subsequent steps. In the subsequent time-frequency cycles (such as the aforementioned M-1 time-frequency cycles), the control input 213 controls each multiplexing register 208 to select the cyclic input 211 to store in the register and provide it to the subsequent steps. Arithmetic logic unit 204.

Although in the embodiment described in the third diagram (and subsequent seventh and nineteenth diagrams), multiple neural processing units 126 can be used to rotate the values of the multiplex registers 208/705 to the right, that is, by the nerves. The processing unit J moves toward the neural processing unit J + 1, but the present invention is not limited to this. In other embodiments (for example, corresponding to the twenty-fourth to twenty-sixth) The embodiment of the figure), the plurality of neural processing units 126 can be used to rotate the value of the multiplex register 208/705 to the left, that is, move from the neural processing unit J toward the neural processing unit J-1. In addition, in other embodiments of the present invention, the neural processing units 126 can selectively rotate the values of the multiplexer registers 208/705 to the left or right. For example, this selection can be instructed by the neural network unit. Specified.

The fourth diagram is a table showing a program stored in the program memory 129 of the neural network unit 121 of the first diagram and executed by the neural network unit 121. As mentioned earlier, this example program performs calculations related to one layer of an artificial neural network. The table in Figure 4 shows four columns and three rows. Each row corresponds to an address marked in the first row in the program memory 129. The second line specifies the corresponding instruction, and the third line indicates the number of time-frequency cycles associated with this instruction. In a preferred embodiment, the foregoing number of time-frequency cycles refers to a valid number of time-frequency cycles per instruction time-frequency cycle value in the embodiment executed by the pipeline, rather than an instruction delay. As shown in the figure, because the neural network unit 121 has the nature of pipeline execution, each instruction has an associated time-frequency cycle. The exception is the instruction at address 2, which actually repeats itself 511 times. Therefore, 511 time-frequency cycles are required, as described later.

All neural processing units 126 process each instruction in the program in parallel. In other words, all N neural processing units 126 will execute the first row of instructions in the same time-frequency cycle, and all N neural processing units 126 will execute the second row of instructions in the same time-frequency cycle. analogy. However, the present invention is not limited to this. In other embodiments of the subsequent chapters, some instructions are based on partially parallel partial sequences. Mode execution, for example, as described in the embodiment of FIG. 11, in the embodiment in which multiple neural processing units 126 share a startup function unit, the startup function and the output instructions at addresses 3 and 4 are This way is performed. The example in the fourth figure assumes that one layer has 512 neurons (neural processing unit 126), and each neuron has 512 connection inputs from the 512 neurons of the previous layer, for a total of 256K connections. Each neuron receives a 16-bit data value from each link input, and multiplies this 16-bit data value by an appropriate 16-bit weight value.

The first row at address 0 (can also be assigned to other addresses) will specify an instruction to initialize the neural processing unit. This initialization instruction clears the value of accumulator 202 to zero. In an embodiment, the initialization instruction may also load a row of the data random access memory 122 or the weight random access memory 124 into the accumulator 202, and the corresponding text specified by the instruction. This initialization instruction will also load the configuration value into the control register 127. This part will be explained in more detail in the subsequent figures 29A and 29B. For example, the width of the data text 207 and the weight text 209 can be loaded for use by the arithmetic logic unit 204 to confirm the size of the operation performed by the circuit. This width also affects the result 215 stored in the accumulator 202. In one embodiment, the neural processing unit 126 includes a circuit to fill the output 215 before the output 215 of the arithmetic logic unit 204 is stored in the accumulator 202, and the initialization instruction will load a configuration value into the circuit, and the configuration value will Affects the aforementioned fill operation. In an embodiment, it may also be specified in an arithmetic logic unit function instruction (such as a multiply accumulate instruction at address 1) or an output instruction (such as a write start function unit output instruction at address 4) so that the accumulator 202 is cleared to zero.

The second row at address 1 specifies a multiply-accumulate instruction to instruct the 512 neural processing units 126 to load a corresponding data text from one row of the data random access memory 122 and from the weight random access memory 124. A column is loaded with a corresponding weight text, and a first multiplication and accumulation operation is performed on the data text input 207 and the weight text input 206, that is, the zero value of the initialized accumulator 202 is added. Further, this instruction instructs the sequencer 128 to generate a value at the control input 213 to select the data text input 207. In the example of the fourth figure, the designated row of the data random access memory 122 is row 17 and the designated row of the weight random access memory 124 is row 0, so the sequencer is instructed to output the value 17 as a data random The memory address 123 is accessed, and the value 0 is output as a weight of the random access memory address 125. Therefore, 512 data words from row 17 of data random access memory 122 are provided as corresponding data inputs 207 of 512 neural processing units 126, and 512 weights from row 0 of weight random access memory 124 The text provides the corresponding weight input 206 as 512 neural processing units 126.

The third column at address 2 specifies a multiply accumulate rotation instruction, which has a count of 511 to instruct the 512 neural processing units 126 to perform 511 multiply accumulate operations. This instruction instructs the 512 neural processing units 126 to input the data text 209 of the arithmetic logic unit 204 in each operation of the 511 multiply-accumulate operations as the rotation value 211 from the neighboring neural processing unit 126. That is, this instruction instructs the sequencer 128 to generate a value at the control input 213 to select the rotation value 211. In addition, this instruction will instruct the 512 neural processing units 126 to assign a corresponding weight to each of the 511 multiply-accumulate operations. The reloading loads the "next" row of the weighted random access memory 124. That is, this instruction will instruct the sequencer 128 to increase the weight random access memory address 125 by one from the previous time-frequency period. In this example, the first time-frequency period of the instruction is column 1, and the next time The frequency period is column 2, the next time-frequency period is column 3, and so on, and the 511th time-frequency period is column 511. In each of these 511 multiply-accumulate operations, the product of the rotation input 211 and the weighted text input 206 is added to a value before the accumulator 202. The 512 neural processing units 126 will perform the 511 multiply-accumulate operations in 511 time-frequency cycles. Each neural processing unit 126 will perform different operations on the different data characters from column 17 of data random access memory 122. That is, Adjacent neural processing units 126 performed data texts in the previous time-frequency cycle, and performed a multiply-accumulate operation on texts with different weights associated with the data texts, which are conceptually different inputs for different connections of neurons. This example assumes that each neural processing unit 126 (neuron) has 512 link inputs, and therefore involves the processing of 512 data words and 512 weight words. After the multiply accumulate and rotate instruction of column 2 is repeated for the last iteration, the accumulator 202 stores the sum of the products of the 512 linked inputs. In one embodiment, the instruction set of the neural processing unit 126 includes an "execute" instruction to instruct the arithmetic logic unit 204 to perform one of the arithmetic logic unit operations specified by the instruction of the initialization neural processing unit, such as the arithmetic of figure 29A The logic unit function 2926 specifies a separate instruction for each different type of arithmetic logic operation (such as the aforementioned multiply-accumulate, accumulator, and maximum values of weights, etc.).

The fourth column at address 3 specifies a starting function make. This startup function instruction instructs the startup function unit 212 to execute the specified startup function on the value of the accumulator 202 to produce a result 133. An example of the startup function will be described in more detail in the subsequent sections.

The fifth column at address 4 specifies a write enable function unit output instruction to instruct the 512 neural processing units 216 to output their enable function unit 212 as a result 133 and write it back to one of the data random access memory 122 rows. , In this example, column 16. That is, this instruction instructs the sequencer 128 to output the value 16 as the data random access memory address 123 and a write command (corresponding to the read command specified by the multiply-accumulate instruction at address 1). In a preferred embodiment, because of the characteristics of pipeline execution, the write start function unit output instruction can be executed simultaneously with other instructions, so the write start function unit output instruction can actually be executed in a single time-frequency cycle.

In a preferred embodiment, each neural processing unit 126 serves as a pipeline, and the pipeline has various functional elements, such as the multiplexer register 208 (and the multiplexer register 705 in the seventh figure), arithmetic The logic unit 204, the accumulator 202, the start function unit 212, the multiplexer 802 (see FIG. 8), the column buffer 1104, and the start function unit 1112 (see FIG. 11), etc., some of the elements themselves are Can be executed in pipeline. In addition to the data text 207 and the weight text 206, this pipeline also receives instructions from the program memory 129. These instructions flow along the pipeline and control multiple functional units. In another embodiment, the program does not include a start function instruction. Instead, the initialization neural processing unit instruction specifies the start function to be executed on the accumulator 202 value 217, indicating that one of the specified start function values is stored in a configuration. Register for pipeline startup function The unit 212 generates the last value 217 of the accumulator 202, that is, after the multiply accumulate and rotate instruction of address 2 is repeated for the last execution, it is used. In a preferred embodiment, in order to save energy consumption, the startup function unit 212 portion of the pipeline will be in a non-starting state before the output instruction of the write startup function unit is reached. When the instruction arrives, the startup function unit 212 will start and The accumulator 202 designated by the initialization instruction outputs 217 to execute the start function.

The fifth diagram is a timing chart showing that the neural network unit 121 executes the program of the fourth diagram. Each column of this timing diagram corresponds to the continuous time-frequency period indicated by the first row. The other lines correspond to different neural processing units 126 of the 512 neural processing units 126 and indicate their operations. The figure only shows the operations of the neural processing unit 0,1,511 to simplify the description.

At time-frequency period 0, each of the 512 neural processing units 126 executes the initialization instruction of the fourth figure, and in the fifth figure, a zero value is assigned to the accumulator 202.

In the time-frequency cycle 1, each of the 512 neural processing units 126 executes a multiply-accumulate instruction at address 1 in the fourth figure. As shown in the figure, the neural processing unit 0 adds the value of accumulator 202 (that is, zero) to the text 0 of row 17 of data random access memory 122 and the text 0 of row 0 of weight random access memory 124 Product; the neural processing unit 1 adds the value of accumulator 202 (ie, zero) to the product of text 1 in row 17 of data random access memory 122 and text 1 in row 0 of weight random access memory 124; By analogy, the neural processing unit 511 adds the value of the accumulator 202 (ie, zero) to the data random access memory 122 The product of the character 511 in column 17 and the character 511 in column 0 of the weight random access memory 124.

In time-frequency cycle 2, each of the 512 neural processing units 126 performs the first iteration of the multiply-accumulate rotation instruction at address 2 in the fourth figure. As shown in the figure, the neural processing unit 0 adds the value of the accumulator 202 to the rotated data text 211 received by the multiplexer register 208 output 209 of the neural processing unit 511 (that is, the data received by the data random access memory 122). Data text 511) multiplied by text 0 of column 1 of weighted random access memory 124; neural processing unit 1 adds the value of accumulator 202 to the rotation received by output 209 of multiplex register 208 of neural processing unit 0 The product of data text 211 (data text 0 received by data random access memory 122) and text 1 of row 1 of weight random access memory 124; and so on, the neural processing unit 511 sets the value of accumulator 202 Add the rotation data text 211 (i.e., the data text 510 received by the data random access memory 122) received by the multiplexing register 208 output 209 of the neural processing unit 510 and the weight random access memory 124 in line 1. Product of text 511.

At time-frequency period 3, each of the 512 neural processing units 126 performs a second iteration of the multiply-accumulate rotation instruction at address 2 in the fourth figure. As shown in the figure, the neural processing unit 0 adds the value of the accumulator 202 to the rotated data text 211 received by the multiplexer register 208 output 209 of the neural processing unit 511 (that is, the data received by the data random access memory 122). (Data text 510) multiplied by text 0 of weight 2 of column 2 of random access memory 124; neural processing unit 1 adds the value of accumulator 202 to the multiplexing register of neural processing unit 0 Product of 208 output 209 received rotated data text 211 (ie, data text 511 received by data random access memory 122) and weight 1 of text 2 in random access memory 124; and so on, neural processing unit 511 The value of the accumulator 202 is added to the rotated data text 211 (i.e. the data text 509 received by the data random access memory 122) and the weighted random access memory received by the multiplexer register 208 output 209 of the neural processing unit 510. Product of characters 511 in column 2 of body 124. As shown in the fifth figure, the 509 time-frequency cycles will continue in this manner until the time-frequency cycle 512.

In the time-frequency cycle 512, each of the 512 neural processing units 126 performs the 511th iteration of the multiply-accumulate rotation instruction at address 2 in the fourth figure. As shown in the figure, the neural processing unit 0 adds the value of the accumulator 202 to the rotated data text 211 received by the multiplexer register 208 output 209 of the neural processing unit 511 (that is, the data received by the data random access memory 122). Data text 1) Multiplied by text 0 of weighted random access memory 124 column 511; neural processing unit 1 adds the value of accumulator 202 to the rotation received by output 209 of multiplex register 208 of neural processing unit 0 The product of data text 211 (data text 2 received by data random access memory 122) and text 1 of row 511 of weight random access memory 124; and so on, the neural processing unit 511 sets the value of accumulator 202 Add the rotated data text 211 (ie, the data text 0 received by the data random access memory 122) received by the multiplexer register 208 output 209 of the neural processing unit 510 and the weighted random access memory 124 row 511 Product of text 511. In one embodiment, multiple time-frequency cycles are required from the data random access memory 122 and the weight random storage. The fetch memory 124 reads the data text and weight text to execute the multiply-accumulate instruction of address 1 in the fourth figure; however, the data random access memory 122, the weight random access memory 124, and the neural processing unit 126 are pipelines. Configured so that after the first multiply-accumulate operation starts (as shown in time-frequency period 1 of the fifth figure), subsequent multiply-accumulation operations (as shown in time-frequency period 2-512 of the fifth figure) will start at Executed in subsequent time-frequency cycles. According to a preferred embodiment, according to the architecture instructions, such as MTNN or MFNN instructions (which will be described in the following fourteenth and fifteenth drawings), the data random access memory 122 and / or weight random access memory These neural processing units 126 are temporarily put on hold by the access operation of the body 124 or the micro instructions translated from the architectural instructions.

In the time-frequency period 513, the activation function unit 212 of each of the 512 neural processing units 126 executes the activation function at address 3 in the fourth figure. Finally, in the time-frequency cycle 514, each of the 512 neural processing units 126 will execute the fourth by writing the result 133 back to the corresponding text in column 16 of the data random access memory 122. In the figure, the write start function unit of address 4 outputs an instruction, that is, the result 133 of the neural processing unit 0 will be written into the data 0 of the random access memory 122, and the result 133 of the neural processing unit 1 will be written. The text 1 of the data random access memory 122 is entered, and so on, the result 133 of the neural processing unit 511 is written into the text 511 of the data random access memory 122. The corresponding block diagram corresponding to the operation of the aforementioned fifth figure is shown in the sixth A figure.

The sixth diagram A is a block diagram showing that the neural network unit 121 of the first diagram executes the program of the fourth diagram. This neural network unit 121 includes 512 neural processing units 126, a data random access memory 122 that receives an address input 123, and a weight random access memory 124 that receives an address input 125. When the time-frequency cycle is 0, the 512 neural processing units 126 execute initialization instructions. This operation is not shown in the figure. As shown in the figure, at time-frequency period 1, 512 16-bit data characters of row 17 are read from the data random access memory 122 and provided to the 512 neural processing units 126. During the time-frequency period from 1 to 512, the 512 16-bit weight characters of columns 0 to 511 will be read from the weight random access memory 122 and provided to the 512 neural processing units 126. At time-frequency period 1, the 512 neural processing units 126 perform their corresponding multiply-accumulate operations on the loaded data text and weight text. This operation is not shown in the figure. During the time-frequency cycle from 2 to 512, the multiplex register 208 of 512 neural processing units 126 will operate as a rotator with 512 16-bit characters, and will previously use data random access memory The data text loaded in column 17 of 122 is rotated to the adjacent neural processing unit 126, and these neural processing units 126 execute the corresponding data text after rotation and the corresponding weight text loaded by the weight random access memory 124. Multiply and accumulate operations. At the time-frequency period 513, the 512 startup function units 212 execute the startup instruction. This operation is not shown in the figure. At the time-frequency period 514, the 512 neural processing units 126 write their corresponding 512 16-bit results 133 back to the 16th row of the data random access memory 122.

As shown in the figure, the number of time-frequency cycles required to generate the result text (neuronal output) and write back the data to the random access memory 122 or the weighted random access memory 124 is approximately as received by the current layer of the neural network. The square root of the number of data inputs (links). For example, if the current layer has 512 neurons and each neuron has 512 connections from the previous layer, the total number of these connections is 256K, and the number of time-frequency cycles required to generate the current layer results will be slightly greater than 512. Therefore, the neural network unit 121 can provide extremely high performance in neural network computing.

The sixth diagram B is a flowchart showing that the processor 100 of the first diagram executes a structural program to perform a typical multiplication and accumulation start function operation of a neuron associated with a hidden layer of an artificial neural network using the neural network unit 121 The operation is the same as that performed by the program of the fourth figure. The example in Figure 6B assumes that there are four hidden layers (variable NUM_LAYERS marked in the initialization step 602), each hidden layer has 512 neurons, and each neuron is connected to all 512 neurons in the previous layer (through the fourth Figure program). However, it should be understood that the selection of the number of these layers and the number of neurons is to illustrate the invention of this case. The neural network unit 121 can apply similar calculations to the embodiment of different numbers of hidden layers, each layer has a different number of neurons. Embodiments, or embodiments in which neurons are not all connected. In one embodiment, a weight value for a non-existent neuron or a non-existent neuron connection in this layer is set to zero. In a preferred embodiment, the architecture program writes the first set of weights into the weight random access memory 124 and activates the neural network unit 121. When the neural network unit 121 is performing calculations associated with the first layer This architecture program will write the second set of weights into the weight random access memory 124. In this way, once the neural network unit 121 completes the calculation of the first hidden layer, the neural network unit 121 can start the calculation of the second layer. In this way, the framework program will go back and forth between the two regions of the weight random access memory 124 Domain to ensure that the neural network unit 121 can be fully utilized. This process starts at step 602.

In step 602, as described in the relevant section of FIG. 6A, the processor 100 executing the architecture program writes the input value into the current neuron hidden layer of the data random access memory 122, that is, writes the data random access 17 of memory 122. These values may also be located in the row 17 of the data random access memory 122 as the operation result 133 (such as convolution, common source or input layer) of the neural network unit 121 for the previous layer. Second, the framework program initializes the variable N to the value 1. The variable N represents the current layer in the hidden layer to be processed by the neural network unit 121. In addition, the framework program initializes the variable NUM_LAYERS to the value 4 because there are four hidden layers in this example. The flow then proceeds to step 604.

In step 604, the processor 100 writes the weight text of the layer 1 into the weight random access memory 124, for example, the columns 0 to 511 shown in FIG. 6A. The flow then proceeds to step 606.

In step 606, the processor 100 uses the MTNN instruction 1400 that specifies a function 1432 to write the program memory 129, and writes a multiply-accumulate activation function program (as shown in the fourth figure) into the neural network unit 121 program memory. 129. The processor 100 then uses a MTNN instruction 1400 to start the neural network unit program, which instruction specifies a function 1432 to start executing the program. The flow then proceeds to step 608.

In decision step 608, the architecture program determines whether the value of the variable N is less than NUM_LAYERS. If yes, the flow proceeds to step 612; otherwise, it proceeds to step 614.

In step 612, the processor 100 weights the layer N + 1 The text write weight is random access memory 124, for example, rows 512 to 1023. Therefore, the architecture program can write the weight text of the next layer into the weight random access memory 124 when the neural network unit 121 performs the calculation of the hidden layer of the current layer, thereby completing the calculation of the current layer, that is, writing After the data random access memory 122, the neural network unit 121 can immediately perform the hidden layer calculation of the next layer. Next, proceed to step 614.

In step 614, the processor 100 confirms whether the executing neural network unit program (in the case of layer 1, the execution is started in step 606, and in the case of layers 2 to 4, it is executed in step 618) whether the execution has been completed. . In a preferred embodiment, the processor 100 reads the state register 127 of the neural network unit 121 by executing an MFNN instruction 1500 to confirm whether the execution has been completed. In another embodiment, the neural network unit 121 generates an interrupt, indicating that the multiplication and accumulation start function layer program has been completed. The flow then proceeds to decision step 616.

In decision step 616, the architecture program determines whether the value of the variable N is less than NUM_LAYERS. If yes, the flow proceeds to step 618; otherwise, it proceeds to step 622.

In step 618, the processor 100 updates the multiply accumulate activation function program to enable execution of the hidden layer calculation of the layer N + 1. Further, the processor 100 updates the data random access memory 122 row value of the multiply accumulate instruction at address 1 in the fourth figure to the row in which the calculation result of the previous layer in the data random access memory 122 is written ( For example, update to column 16) and update the output column (for example, update to column 15). The processor 100 then starts updating the neural network unit program. In another embodiment, the program of the fourth figure specifies the same row of the output instruction at address 4 as the multiplication and accumulation of address 1. The row specified by the instruction (that is, the row read by the data random access memory 122). In this embodiment, the current row of input data text will be overwritten (because this row of data text has been read into the multiplexer register 208 and rotated between these neural processing units 126 through the N text rotator, as long as this row Data text need not be used for other purposes, such processing is allowed). In this case, the neural network unit program does not need to be updated in step 618, but only needs to be restarted. The flow then proceeds to step 622.

In step 622, the processor 100 reads the result of the neural network unit program of the layer N from the data random access memory 122. However, if these results are only used in the next layer, the architecture program does not need to read these results from the data random access memory 122, but can retain them in the data random access memory 122 for the next hidden layer calculation. Use. The flow then proceeds to step 624.

In decision step 624, the architecture program determines whether the value of the variable N is less than NUM_LAYERS. If yes, the flow proceeds to step 626; otherwise, the flow is terminated.

In step 626, the framework program increases the value of N by one. The flow then returns to decision step 608.

As shown in the example of Figure 6B, roughly every 512 time-frequency cycles, these neural processing units 126 perform a read and a write on the data random access memory 122 (through the neural network of Figure 4) Effect of the path unit program). In addition, these neural processing units 126 read the weight random access memory 124 for each time-frequency cycle to read a list of weight text. Therefore, weighted random access The entire bandwidth of the memory 124 will be consumed because the neural network unit 121 performs hidden layer operations in a hybrid manner. In addition, it is assumed that in one embodiment, there is a write and read buffer, such as the buffer 1704 of FIG. 17, while the neural processing unit 126 reads, the processor 100 performs the weight random access memory 124. Write, so that the buffer 1704 performs a write to the weight random access memory 124 approximately every 16 time-frequency cycles to write the weight text. Therefore, in the embodiment in which the weight random access memory 124 is a single port (as described in the corresponding section of FIG. 17), these neural processing units 126 temporarily suspend the weighting for approximately every 16 time-frequency cycles. The read by the random access memory 124 enables the buffer 1704 to write to the weighted random access memory 124. However, in the dual-port weight random access memory 124 embodiment, these neural processing units 126 need not be left unused.

The seventh diagram is a block diagram showing another embodiment of the neural processing unit 126 of the first diagram. The neural processing unit 126 in the seventh figure is similar to the neural processing unit 126 in the second figure. However, the neural processing unit 126 in the seventh figure additionally has a dual-input multiplexer register 705. The multiplexer register 705 selects one of the inputs 206 or 711 to be stored in its register, and provides it to its output 203 in the subsequent time-frequency cycle. Input 206 receives weighted text from weighted random access memory 124. The other input 711 receives the output 203 of the second multiplexer register 705 of the neighboring neural processing unit 126. In a preferred embodiment, the input 711 of the neural processing unit J receives the output 203 of the multiplexer register 705 arranged in the neural processing unit 126 of J-1, and the output 203 of the neural processing unit J provides Input to multiplexer register 705 of neural processing unit 126 arranged in J + 1 711. In this way, the multiplexer registers 705 of the N neural processing units 126 can work together. For example, the rotation of the same N characters is similar to the way shown in the third figure above, but it is used for weighted characters. Non-data text. The multiplexer register 705 uses a control input 213 to control which of the two inputs is selected by the multiplexer register 705 to be stored in its register and subsequently provided to the output 203.

The use of the multiplexer register 208 and / or the multiplexer register 705 (and the multiplexer register in other embodiments as shown in Figures 18 and 23) actually constitutes a large rotator By rotating the data / weights from one row of the data random access memory 122 and / or the weight random access memory 124, the neural network unit 121 does not need to randomly store data in the data random access memory 122 and / or the weights. A very large multiplexer is used between the memory 124 to provide the required data / weight text to the appropriate neural network unit.

Write back the accumulator value in addition to the start function result

For some applications, the processor 100 receives (eg, the media register 118 via the MFNN instruction in Figure 15) the unprocessed accumulator 202 value 217 to provide for execution to other execution units 112 The instructions for performing calculations have their usefulness. For example, in one embodiment, the startup function unit 212 is not configured for the execution of the soft maximum startup function to reduce the complexity of the startup function unit 212. Therefore, the neural network unit 121 outputs the unprocessed accumulator 202 value 217 or one of its subsets to the data random access memory 122 or the weight random access memory 124, and the architecture program can be changed in the subsequent steps by The data random access memory 122 or the weight random access memory 124 reads and calculates this unprocessed value. However, the application of the unprocessed accumulator 202 value 217 is not limited to performing soft maximum operations, other applications are also covered by the present invention.

The eighth diagram is a schematic block diagram showing another embodiment of the neural processing unit 126 of the first diagram. The neural processing unit 126 of the eighth figure is similar to the neural processing unit 126 of the second figure. However, the neural processing unit 126 of the eighth figure includes a multiplexer 802 in the activation function unit 212, and the activation function unit 212 has a control input 803. The width (in bits) of the accumulator 202 is greater than the width of the data text. The multiplexer 802 has a plurality of inputs to receive the data text width portion of the accumulator 202 output 217. In an embodiment, the width of the accumulator 202 is 41 bits, and the neural processing unit 216 can output a 16-bit result text 133. Thus, for example, the multiplexer 802 (or the thirty-third figure) The multiplexer 3032 and / or the multiplexer 3037) has three inputs respectively receiving bits [15: 0], bits [31:16], and bits [47:32] of the output 217 of the accumulator 202. For a preferred embodiment, output bits (such as bits [47:41]) not provided by the accumulator 202 are forced to be set to zero-valued bits.

The sequencer 128 generates a value at the control input 803, and controls the multiplexer 802 to select one of the text (such as 16 bits) of the accumulator 202 to write an accumulator instruction corresponding to one, for example, the ninth figure below Write accumulator instructions at addresses 3 to 5. For a preferred embodiment, the multiplexer 802 has one or more inputs to receive the output of a startup function circuit (such as components 3022, 3024, 3026, 3018, 3014, and 3016 in the thirty figure), and The width of the output produced by these startup function circuits Degree is equal to one profile text. The sequencer 128 will generate a value at the control input 803 to control the multiplexer 802 to choose one of these startup function circuit outputs, instead of choosing one of the words in the accumulator 202, in response to the position in the fourth figure The start function unit of address 4 outputs the instruction.

The ninth figure is a table showing a program stored in the program memory 129 of the neural network unit 121 of the first figure and executed by the neural network unit 121. The example program in Figure 9 is similar to the program in Figure 4. In particular, the instructions at addresses 0 to 2 are exactly the same. However, the instructions at addresses 3 and 4 in the fourth figure are replaced by the write accumulator instruction in the ninth figure. This instruction instructs 512 neural processing units 126 to write their accumulator 202 output 217 as the result 133 and write it back. The three rows of data random access memory 122, in this example, rows 16 to 18. In other words, the write accumulator instruction instructs the sequencer 128 to output a data random access memory address 123 with a value of 16 in the first time-frequency period and a write command, and output a data command in the second time-frequency period. The data random access memory address 123 with a value of 17 and a write command are outputted in the third time-frequency period with a data random access memory address 123 with a value of 18 and a write command. In a preferred embodiment, the execution time of the write accumulator instruction can overlap with other instructions, so that the write accumulator instruction can actually be executed in these three time-frequency cycles, each of which A cycle is written into one row of the data random access memory 122. In one embodiment, the user specifies the value of the start function 2934 and the output command 2956 column of the control register 127 (Figure 29A) to write the required part of the accumulator 202 into the data for random access. Memory 122 or weight random access memory 124. In addition, the write accumulator instruction can optionally write back to the accumulator A subset of 202 instead of writing back the entire contents of accumulator 202. In one embodiment, the standard type accumulator 202 can be written back. This section will be explained in more detail in the subsequent chapters corresponding to the twenty-ninth to thirty-first figures.

The tenth diagram is a timing chart showing the execution of the routine of the ninth diagram by the neural network unit 121. The timing diagram of the tenth diagram is similar to the timing diagram of the fifth diagram, in which the time-frequency periods 0 to 512 are the same. However, in the time-frequency period 513-515, the start function unit 212 of each of the 512 neural processing units 126 will execute one of the write accumulator instructions at addresses 3 to 5 in the ninth figure . In particular, in the time-frequency period 513, each of the 512 neural processing units 126 will output bit 217 [15: 0] of the accumulator 202 as its result 133 and write it back to the data random access memory 122 Corresponding text in column 16; in the time-frequency cycle 514, each of the 512 neural processing units 126 will output bit 217 [31:16] of accumulator 202 as the result 133 and write it back to the data Corresponding text in column 17 of random access memory 122; and in the time-frequency cycle 515, each of the 512 neural processing units 126 will output 217 bits of accumulator 202 [40:32] As a result 133, the corresponding text in column 18 of the data random access memory 122 is written back. In a preferred embodiment, the bits [47:41] are forced to be set to zero.

Shared startup function unit

The eleventh figure is a block diagram showing an embodiment of the neural network unit 121 in the first figure. In the embodiment of FIG. 11, a neuron system is divided into two parts, that is, the activation function unit part and the arithmetic The logic unit section (this section does not include the shift register section), and each start function unit section is shared by multiple arithmetic logic unit sections. In the eleventh figure, the arithmetic logic unit part refers to the neural processing unit 126, and the shared activation function unit part refers to the activation function unit 1112. Compared with the embodiment shown in the second figure, each neuron includes its own activation function unit 212. Accordingly, in an example of the embodiment in the eleventh figure, the neural processing unit 126 (the arithmetic logic unit part) may include the accumulator 202, the arithmetic logic unit 204, the multiplexer register 208 and the register in the second figure. 205, but does not include the start function unit 212. In the embodiment of the eleventh figure, the neural network unit 121 includes 512 neural processing units 126, but the present invention is not limited thereto. In the example of FIG. 11, the 512 neural processing units 126 are divided into 64 groups, which are labeled as groups 0 to 63 in FIG. 11, and each group has eight neural processing units 126.

The neural network unit 121 includes a column buffer 1104 and a plurality of shared activation function units 1112. These activation function units 1112 are coupled between the neural processing unit 126 and the column buffer 1104. The width (in bits) of the row buffer 1104 is the same as one row of the data random access memory 122 or the weight random access memory 124, for example, 512 characters. Each group of neural processing units 126 has an activation function unit 1112, that is, each activation function unit 1112 corresponds to a group of neural processing units 126; thus, in the embodiment of FIG. 11, there are 64 The activation function unit 1112 corresponds to a group of 64 neural processing units 126. The eight neural processing units 126 of the same group share the activation function unit 1112 corresponding to this group. The invention can also be applied to units with different numbers of activation functions and to different numbers of groups in each group. Examples of neural processing units. For example, the present invention can also be applied to an embodiment in which two, four, or sixteen neural processing units 126 share the same activation function unit 1112 in each group.

The shared activation function unit 1112 helps to reduce the size of the neural network unit 121. Size reduction will sacrifice performance. That is, depending on the sharing rate, additional time-frequency cycles will be required to generate the result 133 of the entire neural processing unit 126 array. For example, as shown in the twelfth figure below, the sharing rate at 8: 1 In this case, seven additional time-frequency cycles are required. However, in general, compared with the number of time-frequency cycles required to generate the cumulative total (for example, for each neuron with one layer of 512 connections, 512 time-frequency cycles are required), the aforementioned additional time The number of frequency cycles (for example, 7) is relatively small. Therefore, the effect of the shared activation function unit on the performance is very small (for example, an increase of about one percent of the calculation time), which may be a cost effective for reducing the size of the neural network unit 121.

In one embodiment, each neural processing unit 126 includes an activation function unit 212 to execute a relatively simple activation function. These simple activation function units 212 have a small size and can be included in each of the neural processing units 126. Conversely, the shared complex startup function unit 1112 executes a relatively complex startup function, and its size will be significantly larger than the simple startup function unit 212. In this embodiment, only when the complex startup function is specified and needs to be executed by the shared complex startup function unit 1112, an additional time-frequency cycle is required. In the case where the designated startup function can be executed by the simple startup function unit 212, This extra time-frequency period is not needed.

The twelfth and thirteenth diagrams are timing charts showing that the neural network unit 121 of the eleventh diagram executes the program of the fourth diagram. The timing diagram of the twelfth figure is similar to the timing diagram of the fifth figure, and the time-frequency periods 0 to 512 are the same. However, the operation in the time-frequency period 513 is not the same, because the neural processing unit 126 of the eleventh figure will share the activation function unit 1112; that is, the neural processing units 126 of the same group will share the functions associated with this group. The function unit 1112 is activated, and the eleventh figure shows the shared architecture.

Each column of the timing chart of the thirteenth figure corresponds to a continuous time-frequency period marked on the first row. The other lines correspond to different start function units 1112 of the 64 start function units 1112 and indicate their operations. The figure only shows the operations of the neural processing units 0,1,63 to simplify the description. The time-frequency cycle of the thirteenth figure corresponds to the time-frequency cycle of the twelfth figure, but the operation of the neural processing unit 126 sharing the activation function unit 1112 is shown in different ways. As shown in the thirteenth figure, in the time-frequency period of 0 to 512, the 64 startup function units 1112 are in a non-starting state, and the neural processing unit 126 executes an instruction to initialize the neural processing unit, a multiply accumulate instruction, and a multiply accumulate rotation instruction. .

As shown in the twelfth and thirteenth figures, in the time-frequency period 513, the activation function unit 0 (the activation function unit 1112 associated with the group 0) starts performing the specified activation on the value 217 of the accumulator 202 of the neural processing unit 0 Function, the neural processing unit 0 is the first neural processing unit 216 in the group 0, and the output of the activation function unit 1112 will be stored in the character 0 of the column register 1104. Also in the time-frequency period 513, each start-up function unit 1112 will start to correspond to the first in the corresponding neural processing unit 216 group. The accumulator 202 value 217 of a neural processing unit 126 executes the specified activation function. Therefore, as shown in the thirteenth figure, in the time-frequency period 513, the activation function unit 0 starts to execute the specified activation function on the accumulator 202 of the neural processing unit 0 to generate the text 0 that will be stored in the column register 1104 As a result, the activation function unit 1 starts executing the specified activation function on the accumulator 202 of the neural processing unit 8 to generate a result of the text 8 which will be stored in the register 1104; and so on, the activation function unit 63 starts The accumulator 202 of the neural processing unit 504 executes the designated activation function to generate a result of the text 504 to be stored in the register 1104.

In the time-frequency period 514, the activation function unit 0 (the activation function unit 1112 associated with group 0) starts to execute the specified activation function on the accumulator 202 value 217 of the neural processing unit 1. The neural processing unit 1 is in group 0. The second neural processing unit 216, and the output of the activation function unit 1112 will be stored in the text 1 of the column register 1104. Also in the time-frequency period 514, each activation function unit 1112 starts to execute the specified activation function on the value 217 of the accumulator 202 of the second neural processing unit 126 in the corresponding neural processing unit 216 group. Therefore, as shown in the thirteenth figure, in the time-frequency period 514, the activation function unit 0 starts to execute the specified activation function on the accumulator 202 of the neural processing unit 1 to generate the text 1 that will be stored in the column register 1104. As a result, the activation function unit 1 starts executing the specified activation function on the accumulator 202 of the neural processing unit 9 to produce a result of the text 9 which will be stored in the column register 1104; and so on, the activation function unit 63 starts to The accumulator 202 of the neural processing unit 505 executes the designated activation function to generate a result of the text 505 which will be stored in the column register 1104. This process will continue until the time-frequency cycle 520, the activation function unit 0 (the activation function unit 1112 associated with group 0) starts to execute the specified activation function on the value 217 of the accumulator 202 of the neural processing unit 7, the neural processing unit 7 is the eighth in group 0 ( The last one is the neural processing unit 216, and the output of the activation function unit 1112 will be stored in the character 7 of the column register 1104. Also in the time-frequency period 520, each activation function unit 1112 starts to execute the specified activation function on the value 217 of the accumulator 202 of the eighth neural processing unit 126 in the corresponding neural processing unit 216 group. Therefore, as shown in the thirteenth figure, in the time-frequency period 520, the activation function unit 0 starts to execute the specified activation function on the accumulator 202 of the neural processing unit 7 to generate a character 7 that will be stored in the column register 1104. As a result, the activation function unit 1 starts executing the specified activation function on the accumulator 202 of the neural processing unit 15 to produce a result of the character 15 which will be stored in the register 1104; and so on, the activation function unit 63 starts The accumulator 202 of the neural processing unit 511 executes the designated activation function to generate a result of the text 511 which will be stored in the column register 1104.

In the time-frequency cycle 521, once all 512 results of the 512 neural processing units 126 have been generated and written to the column register 1104, the column register 1104 will begin to write its contents into the data random access memory. 122 or weight random access memory 124. In this way, the activation function unit 1112 of each group of neural processing units 126 executes a part of the activation function instruction at address 3 in the fourth figure.

The embodiment in which the activation function unit 1112 is shared in the group of the arithmetic logic unit 204 as shown in FIG. 11 is particularly helpful for the use with the integer arithmetic logic unit 204. This part in the subsequent chapters corresponds to Relevant explanations will be shown at the 29th to 33rd drawings.

MTNN and MFNN architecture instructions

The fourteenth figure is a block diagram showing the operation of a move to neural network (MTNN) architecture instruction 1400 and its portion corresponding to the neural network unit 121 of the first figure. The MTNN instruction 1400 includes an execution code field 1402, an src1 field 1404, a src2 field, a gpr field 1408, and an immediate field 1412. The MTNN instruction is an architecture instruction, that is, the instruction is included in the instruction set architecture of the processor 100. For a preferred embodiment, the instruction set architecture uses a preset value in the execution code field 1402 to distinguish the MTNN instruction 1400 from other instructions in the instruction set architecture. The execution code 1402 of the MTNN instruction 1400 may include a prefix that is common in the x86 architecture and the like, or may not be included.

The immediate field 1412 provides a value to specify a function 1432 to the control logic 1434 of the neural network unit 121. For a preferred embodiment, this function 1432 is an immediate operand of one of the microinstructions 105 in the first figure. These functions 1432 that can be executed by the neural network unit 121 include write data random access memory 122, write weight random access memory 124, write program memory 129, write control register 127, and start execution The program in the program memory 129, the execution of the program in the program memory 129 is suspended, the notification request (such as interruption) after the execution of the program in the program memory 129 is completed, and the neural network unit 121 is reset, but is not limited thereto. According to a preferred embodiment, the neural network unit instruction set includes an instruction, and the result of the instruction indicates that the neural network unit program has been completed. In addition, the neural network unit instruction set includes an explicit product Generate interrupt instruction. In a preferred embodiment, the operation of resetting the neural network unit 121 includes removing the data from the neural network unit 121 in addition to the data random access memory 122, weight random access memory 124, and program memory 129. The data will remain intact and effectively reset to a reset state (for example, empty the internal state machine and set it to idle state). In addition, internal registers, such as the accumulator 202, will not be affected by the reset function and must be explicitly cleared, for example, using the instruction of initializing the neural processing unit at address 0 in the fourth figure. In an embodiment, the function 1432 may include a direct execution function, and a first source register thereof includes a micro operation (for example, refer to the micro operation 3418 in FIG. 34). This direct execution function instructs the neural network unit 121 to directly execute the specified micro-operation. In this way, the architecture program can directly control the neural network unit 121 to perform operations, instead of writing instructions to the program memory 129 and subsequently instructing the neural network unit 121 to execute the instructions located in the program memory 129 or via MTNN instructions 1400 (or MFNN instruction 1500 in the fifteenth figure). FIG. 14 shows an example of the function of the write data random access memory 122.

This gpr field specifies one of the general purpose registers in the general purpose register file 116. In one embodiment, each general purpose register is 64 bits. The universal register file 116 provides the value of the selected universal register to the neural network unit 121, as shown in the figure, and the neural network unit 121 uses the value as the address 1422. This address 1422 selects a row of memory specified in function 1432. For data random access memory 122 or weight random access memory 124, an additional data block is selected at this address 1422, and its size is the position of the media register in this selected row. Twice (such as 512 bits). For a preferred embodiment, this location is located on a 512-bit boundary. In an embodiment, the multiplexer selects the address 1422 (or the address 1422 in the case of the MFNN instruction 1400 described below) or the address 123/125/131 from the sequencer 128 to provide data. Random access memory 124 / weight random access memory 124 / program memory 129. In one embodiment, the data random access memory 122 has dual ports, so that the neural processing unit 126 can use the media register 118 to read / write the data random access memory 122 and read / write at the same time. Enter this data into the random access memory 122. In an embodiment, for similar purposes, the weight random access memory 124 also has dual ports.

The src1 field 1404 and the src2 field 1406 in the figure both specify a media register of the media register file 118. In one embodiment, each media register 118 is 256 bits. The media register file 118 provides the connected data (e.g., 512 bits) from the selected media register to the data random access memory 122 (or weight random access memory 124 or program memory). 129) The selected row 1428 designated by write address 1422 and the position designated by address 1422 in the selected row 1428 are shown in the figure. Through the execution of a series of MTNN instructions 1400 (and the MFNN instruction 1500 described below), the architecture program executed on the processor 100 can fill 122 rows of data random access memory and 124 rows of weight random access memory and A program is written into the program memory 129, for example, the program described in this article (such as the programs shown in Figures 4 and 9) enables the neural network unit 121 to perform operations on data and weights at a very fast speed to complete this manual operation. Neural network. In one embodiment, the architecture program directly controls the neural network unit 121 instead of writing the program into the program memory. 忆 体 129.

In one embodiment, the MTNN instruction 1400 specifies an initial source register and the number of source registers, that is, Q, rather than specifying two source registers (such as those specified in fields 1404 and 1406). This form of MTNN instruction 1400 instructs the processor 100 to write the media register 118 designated as the starting source register and the next Q-1 successive media registers 118 to the neural network unit 121. That is, the designated data random access memory 122 or weight random access memory 124 is written. For a preferred embodiment, the instruction translator 104 translates the MTNN instruction 1400 into the micro-instructions required to write all Q designated media registers 118. For example, in one embodiment, when the MTNN instruction 1400 designates the register MR4 as the starting source register and Q is 8, the instruction translator 104 translates the MTNN instruction 1400 into four micro instructions, where The first microinstruction is written into the registers MR4 and MR5, the second microinstruction is written into the registers MR6 and MR7, the third microinstruction is written into the registers MR8 and MR9, and the fourth microinstruction is written into the registers MR8 and MR9. The instructions are written into the registers MR10 and MR11. In another embodiment, the data path from the media register 118 to the neural network unit 121 is 1024 bits instead of 512 bits. In this case, the instruction translator 104 translates the MTNN instruction 1400 into two Micro instructions. The first micro instruction is written into the registers MR4 to MR7, and the second micro instruction is written into the registers MR8 to MR11. The present invention can also be applied to the embodiment where the MFNN instruction 1500 specifies an initial destination register and the number of destination registers, so that each MFNN instruction 1500 can access the data random access memory 122 or weight random access memory. One row of 124 reads data larger than single media register 118 Piece.

The fifteenth figure is a block diagram showing the operation of a move to neural network (MTNN) architecture instruction 1500 and its portion corresponding to the neural network unit 121 of the first figure. The MFNN instruction 1500 includes an execution code field 1502, a dst field 1504, a gpr field 1508, and an immediate field 1512. The MFNN instruction is an architecture instruction, that is, the instruction is included in the instruction set architecture of the processor 100. For a preferred embodiment, the instruction set architecture uses a preset value in the execution code field 1502 to distinguish the MFNN instruction 1500 from other instructions in the instruction set architecture. The execution code 1502 of the MFNN instruction 1500 may include a prefix commonly used in the x86 architecture and the like, or may not be included.

The immediate field 1512 provides a value to specify a function 1532 to the control logic 1434 of the neural network unit 121. For a preferred embodiment, this function 1532 is an immediate operand of one of the microinstructions 105 in the first figure. The functions 1532 that these neural network units 121 can execute include read data random access memory 122, read weight random access memory 124, read program memory 129, and read status register 127, but not Limited to this. The example in FIG. 15 shows a function 1532 for reading data from the random access memory 122.

This gpr field 1508 specifies one of the general purpose registers in the general purpose register file 116. This universal register file 116 provides the value of the selected universal register to the neural network unit 121, as shown in the figure, and the neural network unit 121 uses this value as the address 1522 and resembles tenth. The four-picture address 1422 performs calculations to select a row of memory specified in function 1532. Random Access Memory for Data 122 For weight random access memory 124, an additional data block is selected at this address 1522, and its size is the position of the media register (such as 256 bits) in the selected row. For a preferred embodiment, this location is on a 256-bit boundary.

The dst field 1504 specifies a media register in a media register file 118. As shown in the figure, the media register file 118 receives data (such as 256 bits) from the data random access memory 122 (or weight random access memory 124 or program memory 129) to the selected media Register, this data is read from the selected row 1528 specified by the address 1522 in the data receiving and the location specified by the address 1522 in the selected row 1528.

Port allocation of internal random access memory in neural network unit

The sixteenth diagram is a block diagram showing an embodiment of the data random access memory 122 of the first diagram. The data random access memory 122 includes a memory array 1606, a read port 1602, and a write port 1604. The memory array 1606 is loaded with data characters. For a preferred embodiment, these data are arranged in an array of N characters in D rows as described above. In one embodiment, the memory array 1606 includes an array of 64 horizontally arranged static random access memory cells, where each memory cell has a width of 128 bits and a height of 64 bits. A 64 KB data random access memory 122 is provided, which has a width of 8192 bits and has 64 rows. The die area used by this data random access memory 122 is approximately 0.2 mm 2. However, the present invention is not limited to this.

In a preferred embodiment, the write port 1602 is multiplexed to the neural processing unit 126 and the media register 118. Further, these media registers 118 may be coupled to the read port through a result bus, and the result bus is also used to provide data to the reorder buffer and / or the result transfer bus to provide to other execution units 112 . The neural processing unit 126 and the media register 118 share the read port 1602 to read the data random access memory 122. In addition, in a preferred embodiment, the write port 1604 is also multiplexed to the neural processing unit 126 and the media register 118. The neural processing units 126 and the media register 118 share the write port 1604 to write the data to the random access memory 122. In this way, the media register 118 can write to the data random access memory 122 while the neural processing unit 126 reads the data random access memory 122, and the neural processing unit 126 can also temporarily store the data in the media. While the device 118 is reading the data random access memory 122, it is writing into the data random access memory 122. This way of doing things can improve performance. For example, these neural processing units 126 can read the data random access memory 122 (for example, continuously perform calculations), and at the same time, the media register 118 can write more data text into the data random access memory 122 . In another example, the neural processing units 126 can write the calculation results into the data random access memory 122, and at the same time, the media register 118 can read the calculation results from the data random access memory 122. In one embodiment, the neural processing unit 126 can write a row of calculation results into the data random access memory 122, and also read a row of data characters from the data random access memory 122. In one embodiment, the memory array 1606 is configured as a memory area. Bank. When the neural processing unit 126 accesses the data random access memory 122, all the memory blocks will be activated to access a complete row of the memory array 1606; however, the media register 118 accesses the data randomly When accessing the memory 122, only the designated memory block will be activated. In one embodiment, the width of each memory block is 128 bits, and the width of the media register 118 is 256 bits. Thus, for example, each time the media register 118 is accessed, Need to start two memory blocks. In one embodiment, one of the ports 1602/1604 is a read / write port. In one embodiment, these ports 1602/1604 are read / write ports.

The advantage of having these neural processing units 126 with the ability of a rotator as described herein is that compared to ensuring that the neural processing unit 126 can be fully utilized, the architecture program (through the media register 118) can continuously provide data to The data random access memory 122 and the memory array required to retrieve the results from the data random access memory 122 while the neural processing unit 126 performs calculations, this capability helps reduce the amount of data in the random access memory 122. The number of rows of the memory array 1606 can be reduced.

Internal random access memory buffer

The seventeenth figure is a block diagram showing an embodiment of the weight random access memory 124 and a buffer 1704 of the first figure. The weighted random access memory 124 includes a memory array 1706 and a port 1702. This memory array 1706 is loaded with weighted texts. For a preferred embodiment, these weighted texts are arranged in an array of N texts in W rows as described above. In one embodiment, the memory array 1706 includes An array of 128 horizontally arranged static random access memory cells, where each memory cell has a width of 64 bits and a height of 2048 bits, so as to provide a 2MB weight random access memory 124, Its width is 8192 bits and it has 2048 rows. The grain area used by this weighted random access memory 124 is approximately 2.4 mm 2. However, the present invention is not limited to this.

In a preferred embodiment, the port 1702 is multiplexed to the neural processing unit 126 and the buffer 1704. The neural processing unit 126 and the buffer 1704 read and write the weight random access memory 124 through the port 1702. The buffer 1704 is coupled to the media register 118 of the first figure. In this way, the media register 118 can read and write the weight random access memory 124 through the buffer 1704. The advantage of this method is that when the neural processing unit 126 is reading or writing the weight random access memory 124, the media register 118 can also write to or read from the buffer 118 (but if it is The neural processing unit 126 is being executed, and in a better case, these neural processing units 126 are shelved to avoid accessing the weight random access memory 124 when the buffer 1704 accesses the weight random access memory 124). This method can improve performance, especially because the reading and writing of the weighted random access memory 124 by the media register 118 is relatively significantly smaller than the reading and writing of the weighted random access memory 124 by the neural processing unit 126. . For example, in one embodiment, the neural processing unit 126 reads / writes 8192 bits (one column) at a time. However, the width of the media register 118 is only 256 bits, and each MTNN instruction 1400 only Two media registers 118 are written, that is, 512 bits. Therefore, sixteen MTNN instructions 1400 are executed in the framework program to fill In the case of the full buffer 1704, the conflict time between the neural processing unit 126 and the architecture program of the access weight random access memory 124 will be less than about 6% of the total time. In another embodiment, the instruction translator 104 translates one MTNN instruction 1400 into two microinstructions 105, and each microinstruction writes a single data register 118 into the buffer 1704. Thus, the neural processing unit 126 The frequency of conflicts with the framework program when accessing the random access memory 124 is further reduced.

In the embodiment including the buffer 1704, writing the weighted random access memory 124 using a framework program requires multiple MTNN instructions 1400. One or more MTNN instructions 1400 specify a function 1432 to write the data block specified in the buffer 1704, and then an MTNN instruction 1400 specifies a function 1432 to instruct the neural network unit 121 to write the contents of the buffer 1704 to the weight random access One of the memory 124 is selected. The size of a single data block is twice the number of bits in the media register 118, and these data blocks are naturally aligned in the buffer 1704. In one embodiment, the MTNN instruction 1400 of each designated function 1432 to write a designated block of data to the buffer 1704 includes a bit mask having a bit corresponding to each data block of the buffer 1704. The data from the two designated source registers 118 are written into the data block of the buffer 1704, and the corresponding bits in the bit mask are the respective data blocks that are set. This embodiment facilitates the case where duplicate data values exist in one row of the weighted random access memory 124. For example, in order to reset buffer 1704 (and the next row of weighted random access memory 124) to zero, the programmer can load a zero value into the source register and set all bits of the bit mask. . In addition, the bit mask allows the programmer to write to buffer 1704 only. The selected data block, while keeping the other data blocks in their previous data state.

In the embodiment including the buffer 1704, reading the weighted random access memory 124 using a framework program requires multiple MFNN instructions 1500. The initial MFNN instruction 1500 specifies a function 1532 to load a designated column of the weighted random access unit 124 into the buffer 1704, and then one or more MFNN instructions 1500 specify a function 1532 to read a specified data block of the buffer 1704 to Purpose register. The size of a single data block is the number of bits in the media register 118, and these data blocks are naturally aligned in the buffer 1704. The technical features of the present invention can also be applied to other embodiments. For example, the weight random access memory 124 has multiple buffers 1704. By increasing the accessible number of the architecture program when the neural processing unit 126 is executed, the neural processing unit can be further reduced. The conflict between 126 and the structure program due to the access weight random storage memory 124 is increased. In the time-frequency period of the neural processing unit 126 without accessing the weight random access memory 124, it is stored by the buffer 1704 instead. Take the possibility.

The sixteenth figure illustrates a dual-port data random access memory 122, but the present invention is not limited thereto. The technical features of the present invention can also be applied to other embodiments in which the weight random access memory 124 is also a dual-port design. In addition, FIG. 17 illustrates a buffer used with the weighted random access memory 124, but the present invention is not limited thereto. The technical features of the present invention can also be applied to the embodiment in which the data random access memory 122 has a corresponding buffer similar to the buffer 1704.

Dynamically configurable neural processing unit

The eighteenth figure is a block diagram showing a dynamically configurable neural processing unit 126, one of the first figures. Figure 18: Neural Processing The unit 126 is similar to the neural processing unit 126 in the second figure. However, the neural processing unit 126 of Fig. 18 can be dynamically configured to operate in one of two different configurations. In the first configuration, the operation of the neural processing unit 126 of the eighteenth figure is similar to that of the neural processing unit 126 of the second figure. That is, in the first configuration, which is marked here as a "wide" configuration or a "single" configuration, the arithmetic logic unit 204 of the neural processing unit 126 performs a single wide data text and a single wide weight text (E.g., 16 bits) performs an operation to produce a single wide result. In contrast, in the second configuration, which is marked as a “narrow” configuration or a “double” configuration, the neural processing unit 126 performs two narrow data texts and two narrow weight texts (for example, eight (Bits) performing operations each yield two narrow results. In one embodiment, the configuration (wide or narrow) of the neural processing unit 126 is achieved by an instruction to initialize the neural processing unit (eg, an instruction located at address 0 in the twentieth graph). In addition, this configuration can also be achieved by an MTNN instruction with a configuration (wide or narrow) specified by the function 1432 to set the settings of the neural processing unit. In a preferred embodiment, the program memory 129 instructions or MTNN instructions that determine the configuration (wide or narrow) will fill the configuration register. For example, the output of the configuration register is provided to the arithmetic logic unit 204, the start function unit 212, and the logic for generating the multiplexer register control signal 213. Basically, the elements of the neural processing unit 126 of the eighteenth figure perform similar functions as the elements of the same number in the second figure, and reference can be obtained from it to understand the embodiment of the eighteenth figure. The following is a description of the embodiment of the eighteenth figure including the differences from the second figure.

The neural processing unit 126 of the eighteenth figure includes two temporary Registers 205A and 205B, two three-input multiplexing registers 208A and 208B, one arithmetic logic unit 204, two accumulators 202A and 202B, and two start function units 212A and 212B. The registers 205A / 205B each have half the width (eg, 8 bits) of the register 205 of the second figure. The registers 205A / 205B respectively receive a corresponding narrow weight text 206A / B206 (for example, 8 bits) from the weight random access memory 124 and provide the output 203A / 203B to the arithmetic logic in a subsequent time-frequency cycle. The operand selection logic of unit 204 is 1898. When the neural processing unit 126 is in the wide configuration, the registers 205A / 205B will work together to receive a wide weight text 206A / 206B (for example, 16 bits) from the weight random access memory 124, similar to the second The register 205 in the embodiment of the figure; when the neural processing unit 126 is in a narrow configuration, the registers 205A / 205B will actually operate independently, each receiving a narrow weight text from the weight random access memory 124 206A / 206B (for example, 8 bits). In this way, the neural processing unit 126 is actually equivalent to two narrow neural processing units operating independently. However, regardless of the configuration of the neural processing unit 126, the same output bits of the weight random access memory 124 are coupled and provided to the registers 205A / 205B. For example, register 205A of neural processing unit 0 receives byte 0, register 205B of neural processing unit 0 receives byte 1, register 205A of neural processing unit 1 receives byte 2, neural Register 205B of processing unit 1 receives byte 3, and so on, and register 205B of neural processing unit 511 receives byte 1023.

The multiplex register 208A / 208B has half the width (eg, 8 bits) of the register 208 of the second figure, respectively. Multiplex Register 208A One of the inputs 207A, 211A, and 1811A will be stored in its register and will be provided by the output 209A in the subsequent time-frequency period. The multiplex register 208B will be selected in the inputs 207B, 211B, and 1811B and stored in its register. The device is provided to the operand selection logic 1898 by the output 209B in the subsequent time-frequency period. Input 207A receives a narrow data character (for example, 8 bits) from the data random access memory 122, and input 207B receives a narrow data character from the data random access memory 122. When the neural processing unit 126 is in a wide configuration, the multiplexer registers 208A / 208B actually work together to receive one wide data character 207A / 207B (e.g., 16 bits) from the data random access memory 122. ), Similar to the multiplex register 208 in the embodiment of the second figure; when the neural processing unit 126 is in a narrow configuration, the multiplex register 208A / 208B will actually operate independently, each receiving random data from One of the narrow data characters 207A / 207B (for example, 8 bits) of the memory 122 is accessed. In this way, the neural processing unit 126 is actually equivalent to two narrow neural processing units operating independently. However, regardless of the configuration of the neural processing unit 126, the same output bits of the data random access memory 122 will be coupled and provided to the multiplexer registers 208A / 208B. For example, multiplexer register 208A of neural processing unit 0 receives byte 0, multiplexer register 208B of neural processing unit 0 receives multiplexer register 208A of byte 1, neural processing unit 1 receives The multiplexer register 208B of byte 2 and neural processing unit 1 receives byte group 3, and so on, and the multiplexer register 208B of neural processing unit 511 receives byte group 1023.

Input 211A receives the output 209A of the multiplex register 208A of the neighboring neural processing unit 126, and input 211B receives the neighboring nerve The output 209B of the multiplexing register 208B of the processing unit 126. Input 1811A receives the output 209B of the multiplexer register 208B adjacent to the neural processing unit 126, and input 1811B receives the output 209A of the multiplexer register 208A adjacent to the neural processing unit 126. The neural processing unit 126 shown in FIG. 18 belongs to one of the N neural processing units 126 shown in the first figure and is labeled as a neural processing unit J. That is, the neural processing unit J is a representative example of these N neural processing units. For a preferred embodiment, the input 211A of the multiplexing register 208A of the neural processing unit J will receive the output 209A of the multiplexing register 208A of the neural processing unit 126 of Example J-1, and the number of the neural processing unit J is as large as 211A. The input 1811A of the industrial register 208A will receive the output 209B of the multiplex register 208B of the neural processing unit 126 of Example J-1, and the output 209A of the multiple register 208A of the neural processing unit J will be provided to the example J + 1 at the same time. The multiplexer register 208A of the neural processing unit 126 inputs 211A and the multiplexer register 208B of the neural processing unit 126 inputs 211B of the example J; the input 211B of the multiplexer register 208B of the neural processing unit J receives the example J The multiplexer register 208B of the neural processing unit 126 of -1 outputs 209B, and the input 1811B of the multiplexer register 208B of the neural processing unit J will receive the multiplexer register 208A of the neural processing unit 126 of example J to output 209A Moreover, the output 209B of the multiplexing register 208B of the neural processing unit J will be provided to the input 1811A of the multiplexing register 208A of the neural processing unit 126 of example J + 1 and the neural processing unit 126 of example J + 1. The multiplexer register 208B inputs 211B.

The control input 213 controls each of the multiple registers 208A / 208B. Select one of the three inputs to store in its corresponding register, and provide the corresponding output in the subsequent steps. 209A / 209B. When the neural processing unit 126 is instructed to load a row from the data random access memory 122 (for example, the multiply-accumulate instruction of address 1 in the twentieth figure, as described later), whether the neural processing unit 126 is in a wide configuration Or a narrow configuration, the control input 213 controls each of the multiplexer registers 208A / 208B, and selects a corresponding one from the corresponding narrow text in the selected row of the data random access memory 122. Narrow data text 207A / 207B (such as 8 bits).

When the neural processing unit 126 receives an instruction that it is necessary to rotate the previously received data column value (for example, the multiply accumulate rotation instruction of address 2 in the twentieth figure, as described later), if the neural processing unit 126 is in a narrow configuration, control Input 213 will control each multiplexer register 208A / 208B to select the corresponding input 1811A / 1811B. In this case, the multiplexing registers 208A / 208B will actually operate independently so that the neural processing unit 126 will actually act like two independent narrow neural processing units. In this way, the multiplex registers 208A and 208B of the N neural processing units 126 work together as the same 2N narrow-text rotator, which is explained in more detail in the subsequent part corresponding to the nineteenth figure.

When the neural processing unit 126 receives an instruction that it is necessary to rotate the value of the previously received data column, if the neural processing unit 126 is in a wide configuration, the control input 213 controls each of the multiplexing registers in the multiplexing registers 208A / 208B. Select the corresponding input 211A / 211B. In this case, the multiplexer registers 208A / 208B work together as if the neural processing unit 126 is a single wide neural processing unit 126. As such, the multiplexing of N neural processing units 126 Registers 208A and 208B work together as a spinner for the same N wide characters, similar to the way described in the third figure.

Arithmetic logic unit 204 includes operand selection logic 1898, a wide multiplier 242A, a narrow multiplier 242B, a wide dual-input multiplexer 1896A, a narrow dual-input multiplexer 1896B, a wide adder 244A, and a narrow addition器 244B. In fact, the arithmetic logic unit 204 can be understood to include operand selection logic, a wide arithmetic logic unit 204A (including the aforementioned wide multiplier 242A, the aforementioned wide multiplexer 1896A, and the aforementioned wide adder 244A) and a narrow arithmetic logic unit 204B (including the aforementioned narrow multiplier 242B, the aforementioned narrow multiplexer 1896B, and the aforementioned narrow adder 244B). In a preferred embodiment, the wide multiplier 242A can multiply two wide characters, similar to the multiplier 242 of the second figure, such as a 16-bit multiplier by 16-bit multiplier. The narrow multiplier 242B can multiply two narrow characters, such as an 8-bit by 8-bit multiplier to produce a 16-bit result. When the neural processing unit 126 is in a narrow configuration, with the assistance of the operator selection logic 1898, the wide multiplier 242A can be fully used as a narrow multiplier to multiply two narrow texts, so the neural processing unit 126 will be as Two effectively functioning narrow neural processing units. For a preferred embodiment, the wide adder 244A adds the output of the wide multiplexer 1896A and the output 217A of the wide accumulator 202A to generate a total 215A for the wide accumulator 202A. Its operation is similar to that of the first Adder 244 of the second figure. The narrow adder 244B adds the output of the narrow multiplexer 1896B and the output of the narrow accumulator 202B to 217B to generate a total 215B for the narrow accumulator 202B. In one embodiment, the narrow accumulator 202B has a width of 28 bits to avoid performing up to 1024 16 bits Accumulation of meta products will lose accuracy. When the neural processing unit 126 is in a wide configuration, the narrow multiplier 244B, the narrow accumulator 202B, and the narrow start function unit 212B are preferably in a non-start state to reduce energy consumption.

Operand selection logic 1898 selects operands from 209A, 209B, 203A, and 203B to provide other elements of the arithmetic logic unit 204, as described later. For a preferred embodiment, the operand selection logic 1898 also has other functions, such as performing sign extension of signed numeric data text and weight text. For example, if the neural processing unit 126 is in a narrow configuration, the operator selection logic 1898 will extend the symbols of the narrow data text and weight text to the width of the wide text, and then provide it to the wide multiplier 242A. Similarly, if the arithmetic logic unit 204 accepts an instruction to pass a narrow data / weight text (using the wide multiplexer 1896A to skip the wide multiplier 242A), the operator selection logic 1898 will extend the sign of the narrow data text and weight text to The width of the wide text is then supplied to the wide adder 244A. For a preferred embodiment, the logic for performing the symbol extension function also exists in the arithmetic logic operation 204 of the neural processing unit 126 in the second figure.

The wide multiplexer 1896A receives the output of the wide multiplier 242A and an operand from the operand selection logic 1898, and selects one of these inputs to provide to the wide adder 244A. The narrow multiplexer 1896B receives the narrow multiplier 242B. The output and one of the operands from the operand selection logic 1898 selects one of these inputs and supplies it to the narrow adder 244B.

The operand selection logic 1898 is based on the configuration of the neural processing unit 126 and the arithmetic and / The OR operation provides an operand, and the arithmetic / logical operation is determined according to a function specified by an instruction executed by the neural processing unit 126. For example, if the instruction instructs the arithmetic logic unit 204 to perform a multiply-accumulate operation and the neural processing unit 126 is in a wide configuration, the operand selection logic 1898 provides a wide text formed by concatenating the outputs 209A and 209B to the wide multiplier 242A. One input, and a wide text formed by concatenating the outputs 203A and 203B to the other input, and the narrow multiplier 242B is not activated. In this way, the operation of the neural processing unit 126 will be similar to a single one similar to the second The wide neural processing unit 126 of the neural processing unit 126 in the figure. However, if the instruction instructs the arithmetic logic unit to perform a multiply-accumulate operation and the neural processing unit 126 is in a narrow configuration, the operand selection logic 1898 provides an extended or expanded version of the narrow data text 209A to one of the wide multipliers 242A Input, and the extended version of the narrow weight text 203A is provided to another input; in addition, the operator selection logic 1898 provides the narrow data text 209B to one of the narrow multipliers 242B input, and the narrow weight text 203B to another One input. In order to achieve the operation of extending or expanding the narrow text as described above, if the narrow text has a sign, the operand selection logic 1898 will extend the narrow text; if the narrow text has no sign, the operand selection logic 1898 Will add the top zero bit to the narrow text.

In another example, if the neural processing unit 126 is in a wide configuration and the instruction instructs the arithmetic logic unit 204 to perform an accumulation operation on a weight literal, the wide multiplier 242A will be skipped, and the operand selection logic 1898 will output 203A In series with 203B, a wide multiplexer 1896A is provided to provide a wide adder 244A. However, if it is neural processing Unit 126 is in a narrow configuration and the instruction instructs the arithmetic logic unit 204 to perform an accumulation operation on the weighted text. The wide multiplier 242A will be skipped, and the operand selection logic 1898 will provide an extended version of the output 203A up to wide The multiplexer 1896A is provided to the wide adder 244A; in addition, the narrow multiplier 242B is skipped, and the operand selection logic 1898 provides the output 203B of the extended version to the narrow multiplexer 1896B for the narrow adder 244B.

In another example, if the neural processing unit 126 is in a wide configuration and the instruction instructs the arithmetic logic unit 204 to perform an accumulation operation on a data literal, the wide multiplier 242A will be skipped, and the operand selection logic 1898 will output 209A In series with 209B, a wide multiplexer 1896A is provided to a wide adder 244A. However, if the neural processing unit 126 is in a narrow configuration and the instruction instructs the arithmetic logic unit 204 to perform an accumulation operation on the data text, the wide multiplier 242A will be skipped, and the operand selection logic 1898 will output an extended version 209A is provided to the wide multiplexer 1896A to be provided to the wide adder 244A; in addition, the narrow multiplier 242B will be skipped, and the operand selection logic 1898 will provide the output of the extended version 209B to the narrow multiplexer 1896B to provide Narrow adder 244B. Accumulation of weights / data text is helpful for averaging. The averaging can be used as a pooling layer in some artificial neural network applications such as image processing.

In a preferred embodiment, the neural processing unit 126 further includes a second wide multiplexer (not shown) for skipping the wide adder 244A, so as to facilitate the configuration of a wide data / weight text in a wide configuration. Or one of the narrow data / weight text in the narrow configuration after loading wide accumulator 202A, and a second narrow multiplexer (not shown), which is used to skip the narrow adder 244B to facilitate loading a narrow data / weight text in the narrow configuration into the narrow accumulator 202B. According to a preferred embodiment, the arithmetic logic unit 204 further includes a wide / narrow comparator / multiplexer combination (not shown). The comparator / multiplexer combination receives a corresponding accumulator value 217A. / 217B and the corresponding multiplexer 1896A / 1896B output, so as to select the maximum value between the accumulator value 217A / 217B and a data / weight text 209A / 209B / 203A / 203B, a common source of some artificial neural network applications The (pooling) layer uses this operation. This part will be explained in more detail in the subsequent chapters, for example, corresponding to the 27th and 28th figures. In addition, the operand selection logic 1898 is used to provide an operand with a value of zero (for addition operations to add zeros or to clear the accumulator), and an operand for a value of one (for multiplications by multiplication by one).

The narrow start function unit 212B receives the output 217B of the narrow accumulator 202B and executes a start function on it to produce a narrow result 133B. The wide start function unit 212A receives the output 217A of the wide accumulator 202A and executes a start function on it to generate a Wide result 133A. When the neural processing unit 126 is in a narrow configuration, the wide start function unit 212A will understand the output 217A of the accumulator 202A according to this configuration and execute a start function on it to generate a narrow result, such as 8 bits. More detailed descriptions are provided in the 29th to 30th drawings.

As mentioned before, a single neural processing unit 126 can actually operate as two narrow neural processing units when in a narrow configuration, so for smaller text, compared to the wide configuration, it can roughly provide Up to twice the processing power. For example, suppose God The network layer has 1024 neurons, and each neuron receives 1024 narrow inputs (and has narrow weighted text) from the previous layer, which results in one million connections. For the neural network unit 121 with 512 neural processing units 126, in a narrow configuration (equivalent to 1024 narrow neural processing units), although it processes narrow text instead of wide text, it can process links The number can reach four times that of the wide configuration (a million connections with 256K connections), and the time required is about half (about 1026 time-frequency cycles versus 514 time-frequency cycles).

In an embodiment, the dynamically configured neural processing unit 126 of FIG. 18 includes a three-input multiplex register similar to the multiplex registers 208A and 208B to replace the registers 205A and 205B to form a rotator. Processing the weight character string received by the weight random access memory 124. This operation part is similar to the method described in the embodiment of FIG. 7 but applied to the dynamic configuration described in FIG. 18.

The nineteenth figure is a block diagram showing the embodiment according to the eighteenth figure, using the 2N multiplexing registers 208A / 208B of the N neural processing units 126 of the neural network unit 121 of the first figure. The data random access memory 122 of the first figure obtains a row of data characters 207 and performs the same operation as the same rotator. In the embodiment of the nineteenth figure, N is 512, and the neural processing unit 121 has 1024 multiplexing registers 208A / 208B, which are labeled as 0 to 511, respectively corresponding to 512 neural processing units 126 and actually 1024. Narrow nerve processing unit. The two narrow neural processing units in the neural processing unit 126 are labeled A and B, respectively. In each multiplexing register 208, the corresponding narrow neural processing units are also labeled. Further, the multiplexing of the neural processing unit 126 labeled 0 The register 208A is labeled as 0-A, the multiplex register 208B of the neural processing unit 126 labeled as 0, and the register 208A of the nerve processing unit 126 is labeled 0-B, and the multiplex register 208A of the neural processing unit 126 is labeled 1. Is 1-A, the multiplex register 208B of the neural processing unit 126 labeled 1 is labeled 1-B, the multiplex register 208A of the neural processing unit 126 labeled 511 is labeled 511-A, and The multiplex register 208B of the neural processing unit 126 labeled 511 is labeled 511-B, and its value also corresponds to the narrow neural processing unit described in the following twenty-first figure.

Each multiplexer register 208A receives its corresponding narrow data text 207A in one of the D rows of data random access memory 122, and each multiplexer register 208B stores data in the data random access memory One of the D columns of 122 receives its corresponding narrow data text 207B. That is, the multiplexer register 0-A receives the narrow data text 0 of 122 rows of the random access memory, and the multiplex register 0-B receives the narrow data text 1 of 122 rows of the random access memory. Multiplex register 1-A receives narrow data text 2 in 122 rows of random access memory, multiplex register 1-B receives narrow data text 3 in 122 rows of random access memory, and so on, Multiplex register 511-A receives narrow data characters 1022 of 122 rows of random access memory, and multiplex register 511-B receives narrow data characters 1023 of 122 rows of random access memory. In addition, multiplexer 1-A receives the output 209A of multiplexer 0-A as its input 211A, and multiplexer 1-B receives the output 209B of multiplexer 0-B as its input 211B, and so on, multiplexer register 511-A receives the output of multiplexer register 510-A 209A as its input 211A, multiplexer register 511-B receives the output of multiplexer register 510-B 209B as its input 211B, and multiplexer register 0-A receives the output 209A of multiplexer register 511-A as its input 211A, multiplexer register 0-B receives the output 209B of multiplexer register 511-B as its input Enter 211B. Each multiplexer register 208A / 208B will receive control input 213 to control its selection of data text 207A / 207B or input 211A / 211B after rotation or input 1811A / 1811B after rotation. Finally, multiplexer 1-A receives the output 209B of multiplexer 0-B as its input 1811A, and multiplexer 1-B receives the output 209A of multiplexer 1-A as its input 1811B, and so on, multiplexer register 511-A receives the output of multiplexer register 510-B 209B as its input 1811A, multiplexer register 511-B receives the output of multiplexer register 511-A 209A is used as its input 1811B, and multiplex register 0-A receives the output of multiplex register 511-B 209B is used as its input 1811A, multiplex register 0-B receives the multiplex register 0-A Output 209A as its input 1811B. Each multiplexer register 208A / 208B will receive control input 213 to control its selection of data text 207A / 207B or input 211A / 211B after rotation or input 1811A / 1811B after rotation. In an operation mode, in the first time-frequency period, the control input 213 controls each multiplexer register 208A / 208B to select the data text 207A / 207B and store it in the register for subsequent supply to the arithmetic logic unit 204; For subsequent time-frequency cycles (such as the aforementioned M-1 time-frequency cycle), the control input 213 will control each multiplexer register 208A / 208B after selecting rotation and input 1811A / 1811B to store in the register for subsequent supply to the arithmetic logic unit 204, this part will be explained in more detail in subsequent chapters.

Figure 20 is a table showing a program memory 129 stored in the neural network unit 121 of the first figure The program executed by the unit 121, and the neural network unit 121 has a neural processing unit 126 as shown in the embodiment in FIG. The example program in Figure 20 is similar to the program in Figure 4. The differences are explained below. The initialization neural processing unit at address 0 specifies that the neural processing unit 126 will enter a narrow configuration. In addition, as shown in the figure, the multiply-accumulate rotation instruction at address 2 specifies a count value of 1023 and requires 1023 time-frequency cycles. This is because the example in Figure 20 assumes that there are actually 1024 narrow (e.g. 8-bit) neurons (i.e., neural processing units) in a layer, and each narrow neuron has 1024 neurons from the previous layer Yuan's link input, so there are 1024K links in total. Each neuron receives an 8-bit data value from each link input and multiplies this 8-bit data value by an appropriate 8-bit weight value.

The twenty-first diagram is a timing chart showing a neural network unit 121 executing the program of the twentieth diagram. The neural network unit 121 has the neural processing unit 126 shown in the eighteenth diagram executed in a narrow configuration. The timing chart of the twenty-first figure is similar to the timing chart of the fifth figure. The differences are explained below.

In the timing diagram of the twenty-first figure, these neural processing units 126 will be in a narrow configuration, because the initialization neural processing unit instruction at address 0 initializes them to a narrow configuration. Therefore, these 512 neural processing units 126 actually work like 1024 narrow neural processing units (or neurons). These 1024 narrow neural processing units are connected with neural processing units 0-A and neural processing units in the column. 0-B (two narrow neural processing units of the neural processing unit 126 labeled 0), neural processing unit 1-A and neural processing unit 1-B (the neural processing unit 126 labeled 1) Two narrow neural processing units), and so on until the neural processing unit 511-A and the neural processing unit 511-B (the two narrow neural processing units of the neural processing unit 126 labeled 511) are specified. In order to simplify the description, only the operations of the narrow neural processing units 0-A, 0-B, and 511-B are shown in the figure. Because the count value specified by the multiply accumulate rotation instruction at address 2 is 1023, which requires 1023 time-frequency cycles to operate, the number of columns in the timing chart of Figure 21 includes up to 1026 time-frequency cycles.

At time-frequency period 0, each of the 1024 neural processing units will execute the initialization instruction of the fourth figure, that is, the operation of assigning a zero value to the accumulator 202 shown in the fifth figure.

At time-frequency period 1, each of the 1024 narrow neural processing units executes the multiply-accumulate instruction at address 1 in the twentieth figure. As shown in the figure, the narrow neural processing unit 0-A adds the value of accumulator 202A (ie, zero) to the column 17 of the data random access unit 122 and the narrow text 0 and the weight 0 of the random access unit 124 to the narrow text 0. Product: Narrow nerve processing unit 0-B multiplies the value of accumulator 202B (ie, zero) plus data random access unit 122 column 17 narrow text 1 and weight random access unit 124 column 0 narrow text 1; By analogy, the narrow neural processing unit 511-B multiplies the value of accumulator 202B (ie, zero) by the product of column 17 narrow text 1023 of data random access unit 122 and column 0 narrow text 1023 of weight random access unit 124.

In time-frequency cycle 2, each of the 1024 narrow neural processing units executes the first iteration of the multiply-accumulate rotation instruction at address 2 in the twentieth figure. As shown in the figure, the narrow nerve processing unit 0-A adds the value 217A of the accumulator 202A to the narrow nerve processing unit 511-B. The multiplexer register 208B outputs the rotated narrow data text 1811A (i.e., the narrow data text 1023 received by the data random access memory 122) received by the multiplexer 208B and the weight random access unit 124 of the narrow text 0 of 0 Product; the narrow neural processing unit 0-B adds the accumulator 202B value 217B to the multiplexed register 208A of the narrow neural processing unit 0-A and outputs the rotated narrow data text 1811B received by the 209A (that is, the data is stored randomly) Take the product of the narrow data character 0 received by the memory 122 and the narrow character 1 of column 1 of the weight random access unit 124; and so on, until the narrow neural processing unit 511-B adds the value 217B of the accumulator 202B by the narrow The multiplexing register 208A of the neural processing unit 511-A outputs the rotated narrow data text 1811B (that is, the narrow data text 1022 received by the data random access memory 122) received by the 209A and the weight random access unit 124 Product of column 1 narrow text 1023.

At time-frequency period 3, each of the 1024 narrow neural processing units will execute the second iteration of the multiply-accumulate rotation instruction at address 2 in the twentieth figure. As shown in the figure, the narrow neural processing unit 0-A adds the value 217A of the accumulator 202A plus the rotated narrow data text 1811A received by the multiplexer register 208B output 209B of the narrow neural processing unit 511-B (that is, The product of the narrow data text 1022) received by the data random access memory 122 and the narrow text 0 of column 2 of the weight random access unit 124; the narrow neural processing unit 0-B adds the value 217B of the accumulator 202B by the narrow nerve The multiplexing register 208A of the processing unit 0-A outputs the narrow data text 1811B (that is, the narrow data text 1023 received by the data random access memory 122) received by the 209A and the weight random access unit 124. Column 2 is the product of narrow text 1; and so on, until the narrow neural processing order Element 511-B adds the accumulator 202B value 217B to the multiplexed register 208A output 209A of the narrow neural processing unit 511-A and receives the rotated narrow data text 1811B (that is, the data random access memory 122). The product of the received narrow data text 1021) and the narrow text 1023 of column 2 of the weight random access unit 124. As shown in the twenty-first figure, this operation will continue in the subsequent 1021 time-frequency cycles until the time-frequency cycle 1024 described below.

In a time-frequency cycle of 1024, each of the 1024 narrow neural processing units will execute the 1023th iteration of the multiply-accumulate rotation instruction at address 2 in the twentieth figure. As shown in the figure, the narrow neural processing unit 0-A adds the value 217A of the accumulator 202A plus the rotated narrow data text 1811A received by the multiplexer register 208B of the narrow neural processing unit 511-B to output 209B (that is, Product of narrow data text received by data random access memory 122 1) and weighted random access unit 124 1023 narrow text 0; product of narrow neural processing unit 0-B accumulator 202B value 217B plus narrow nerve The multiplexing register 208A of the processing unit 0-A outputs the narrow data text 1811B (that is, the narrow data text 2 received by the data random access memory 122) received by the 209A and the weight random access unit 124. Column 1023 the product of the narrow text 1; and so on, until the narrow neural processing unit 511-B adds the accumulator 202B value 217B plus the rotation received by the multiplexer register 208A of the narrow neural processing unit 511-A to output 209A The product of the narrow data text 1811B (that is, the narrow data text 0 received by the data random access memory 122) and the narrow text 1023 of the row 1023 of the weight random access unit 124.

In the time-frequency cycle 1025, the activation function unit 212A / 212B of each of the 1024 narrow neural processing units will execute the twentieth The start function instruction at address 3 in the figure. Finally, in the time-frequency cycle 1026, each of the 1024 narrow neural processing units will write its narrow result 133A / 133B back to the corresponding narrow text in column 16 of the data random access memory 122 to execute the second Figure 10 shows the write start function unit instruction at address 4. That is, the narrow result 133A of the neural processing unit 0-A will be written into the narrow text 0 of the data random access memory 122, and the narrow result 133B of the neural processing unit 0-B will be written into the data random access memory 122 Narrow text 1, and so on, until the narrow result 133B of the neural processing unit 511-B is written into the narrow text 1023 of the data random access memory 122. The twenty-second figure is a block diagram showing the aforementioned operations corresponding to the twenty-first figure.

The twenty-second figure is a block diagram showing the neural network unit 121 of the first figure. The neural network unit 121 has a neural processing unit 126 as shown in the eighteenth figure to execute the program of the twentieth figure. The neural network unit 121 includes 512 neural processing units 126, that is, 1024 narrow neural processing units, a data random access memory 122, and a weight random access memory 124. The data random access memory 122 receives its bits. The address input 123 and the weight random access memory 124 receive its address input 125. Although it is not shown in the figure, at a time-frequency period of 0, the 1024 narrow neural processing units will execute the initialization instructions of the twentieth figure. As shown in the figure, in time-frequency period 1, 1024 8-bit data characters of row 17 are read from the data random access memory 122 and provided to the 1024 narrow neural processing units. In the time-frequency period of 1 to 1024, 1024 8-bit weight characters of columns 0 to 1023 are read from the weight random access memory 124 and provided to the 1024 narrow neural processing units. Although not shown in the figure However, at time-frequency period 1, the 1024 narrow neural processing units will perform corresponding multiply-accumulate operations on the loaded data text and weight text. In the time-frequency period of 2 to 1024, the multiplex register 208A / 208B of the 1024 narrow neural processing units operates like a 1024 8-bit text rotator, which will randomly load previously loaded data into random access memory. The data text in column 17 of 122 is rotated to the adjacent narrow neural processing units, and these narrow neural processing units perform multiplication on the corresponding rotated data text and the corresponding narrow weight text loaded by the weight random access memory 124. Accumulation. Although not shown in the figure, the 1024 narrow start function units 212A / 212B will execute the start instruction in the time-frequency cycle 1025. In the time-frequency cycle 1026, the 1024 narrow neural processing units write their corresponding 1024 8-bit results 133A / 133B back to the 16th row of the data random access memory 122.

It can be found that, compared to the embodiment of the second figure, the embodiment of the eighteenth figure gives the programmer flexibility to use wide data and weighted text (such as 16 bits) and narrow data and weighted text (such as 8-bit) to perform calculations in response to the need for accuracy in a particular application. From an aspect, for the application of narrow data, the embodiment of Fig. 18 can provide twice the performance compared to the embodiment of Fig. 2. However, additional narrow components (such as multiplexed temporary storage) must be added. 208B, register 205B, narrow arithmetic logic unit 204B, narrow accumulator 202B, and narrow start function unit 212B). At the cost of these additional narrow elements, the area of the neural processing unit 126 is increased by about 50%.

Trimodal nerve processing unit

The twenty-third figure is a schematic block diagram showing another embodiment of the dynamically configurable neural processing unit 126, one of the first figures. The neural processing unit 126 of the twenty-third figure can be used not only in a wide configuration and a narrow configuration, but also in a third configuration, which is referred to herein as a "funnel" configuration. The neural processing unit 126 of the twenty-third figure is similar to the neural processing unit 126 of the eighteenth figure. However, the wide adder 244A in the eighteenth figure is replaced by a three-input wide adder 2344A in the neural processing unit 126 in the twenty-third figure. The three-input wide adder 2344A receives a third adder 2399. It is an extended version of the output of the narrow multiplexer 1896B. The program executed by the neural network unit having the neural processing unit of the twenty-third diagram is similar to that of the twentieth diagram. However, the initialization neural processing unit instruction at address 0 initializes these neural processing units 126 to a funnel configuration instead of a narrow configuration. In addition, the count value of the multiply accumulate rotation instruction at address 2 is 511 instead of 1023.

In the funnel configuration, the operation of the neural processing unit 126 is similar to that in the narrow configuration. When the multiply-accumulate instruction at address 1 in the twentieth figure is executed, the neural processing unit 126 will receive two narrow data characters 207A / 207B and Two narrow weight texts 206A / 206B; wide multiplier 242A will multiply data text 209A and weight text 203A to produce a product 246A selected by wide multiplexer 1896A; narrow multiplier 242B will phase data text 209B and weight text 203B Multiply by 246B to produce the selection of the narrow multiplexer 1896B. However, the wide adder 2344A adds the product 246A (selected by the wide multiplexer 1896A) and the product 246B / 2399 (selected by the wide multiplexer 1896B) to the wide accumulator 202A output 217A, and the narrow adder 244B and The narrow accumulator 202B is not activated. Also, in the funnel When configured to execute a multiply accumulate rotation instruction such as address 2 in the twentieth figure, the control input 213 causes the multiplexer register 208A / 208B to rotate two narrow characters (such as 16 bits). The registers 208A / 208B will select their corresponding inputs 211A / 211B, as if in wide configuration. However, the wide multiplier 242A will multiply the data text 209A and the weight text 203A to produce the product 246A selected by the wide multiplexer 1896A; the narrow multiplier 242B will multiply the data text 209B and the weight text 203B to produce a narrow multiplexer. The product 246B selected by 1896B; and the wide adder 2344A adds the product 246A (selected by the wide multiplexer 1896A) and the product 246B / 2399 (selected by the wide multiplexer 1896B) to the wide accumulator 202A output 217A. The narrow adder 244B and the narrow accumulator 202B are not activated as described above. Finally, when the start function instruction at address 3 is executed in the funnel configuration, the wide start function unit 212A executes the start function on the total number of results 215A to produce a narrow result 133A, and the narrow start function unit 212B is Does not start. In this way, only the narrow neural processing unit labeled A will produce a narrow result 133A, and the narrow neural processing unit labeled B will produce a narrow result 133A. Therefore, the column of the write-back result (such as column 16 indicated by the instruction of address 4 in the twentieth figure) will contain holes, because only the narrow result 133A is valid, and the narrow result 133B is invalid. Therefore, conceptually, in each time-frequency period, each neuron (the neural processing unit of the twenty-third figure) will perform two input of connected data, that is, multiply two narrow data characters by their corresponding weights and By adding these two products, in comparison, the embodiments of the second graph and the eighteenth graph perform only one connection data input in each time-frequency period.

It can be found in the embodiment of the twenty-third figure that And write back the data random access memory 122 or weight random access memory 124 with the result text (neuronal output) half the square root of the number of data inputs (links) received, and the result write-back row has holes That is, the result of every other narrow text is invalid. More precisely, the result of the narrow neural processing unit labeled B is meaningless. Therefore, the embodiment of the twenty-third figure is particularly efficient for a neural network with two consecutive layers. For example, the number of neurons in the first layer is twice that of the second layer (for example, the first layer has 1024 The neurons are fully connected to the 512 neurons in the second layer). In addition, other execution units 122 (for example, media units, such as x86 advanced vector expansion unit) may perform a pack operation on a scattered result column (that is, have a hole) to make it compact (that is, have no hole) when necessary. . Subsequently, when the neural processing unit 121 performs other calculations related to the other rows of the data random access memory 122 and / or the weight random access memory 124, the processed data row may be used for calculation.

Hybrid Neural Network Unit Operations: Convolution and Common Source Computing Capabilities

The advantage of the neural network unit 121 according to the embodiment of the present invention is that the neural network unit 121 can operate in a manner similar to that of a coprocessor executing its own internal program at the same time and executes all operations in a processing unit similar to a processor. Issued architectural instructions (or micro-instructions translated from architectural instructions). The architectural instructions are contained in an architectural program executed by a processor having a neural network unit 121. In this way, the neural network unit 121 can operate in a mixed manner, and the high utilization rate of the neural processing unit 121 can be maintained. For example, the twenty-fourth to twenty-six illustrations show The neural network unit 121 performs the operation of the convolution operation. Among them, the neural network unit is fully utilized. The twenty-seventh through twenty-eighth figures show the operation of the neural network unit 121 performing the common source operation. Applications such as convolutional layers, common source layers, and other digital data calculations, such as image processing (such as edge detection, sharpening, blurring, recognition / classification), require these operations. However, the hybrid operation of the neural processing unit 121 is not limited to performing convolution or common source operations. This hybrid feature can also be used to perform other operations, such as the traditional neural network multiply-accumulate operations and activation functions described in Figures 4-13. Operation. That is, the processor 100 (more precisely, the reserved station 108) will issue MTNN instructions 1400 and MFNN instructions 1500 to the neural network unit 121. In response to the issued instructions, the neural network unit 121 will write data into the memory The body 122/124/129 reads the result from the memory 122/124 written by the neural network unit 121. At the same time, in order to execute the program written by the processor 100 (via the MTNN1400 instruction) into the program memory 129 The neural network unit 121 will read and write the memory 122/124/129.

The twenty-fourth figure is a block diagram showing an example of a data structure used by the neural network unit 121 of the first figure to perform a convolution operation. The block diagram includes a convolution kernel 2402, a data array 2404, and the data random access memory 122 and the weight random access memory 124 of the first figure. In a preferred embodiment, the data array 2404 (for example, corresponding to an image pixel) is loaded in a system memory (not shown) connected to the processor 100 and is loaded into the nerve by the processor 100 by executing the MTNN instruction 1400 The weight of the network unit 121 is a random access memory 124. The convolution operation is to convolve a first array and a second array, and this second array is the convolution kernel described herein. As described herein, the convolution kernel A matrix of coefficients. These coefficients can also be called weights, parameters, elements or values. According to a preferred embodiment, the convolution kernel 2042 is static data of a framework program executed by the processor 100.

The data array 2404 is a two-dimensional array of data values, and the size of each data value (such as an image pixel value) is the size of the text in the data random access memory 122 or the weight random access memory 124 (for example, 16 Bit or 8 bit). In this example, the data value is a 16-bit text, and the neural network unit 121 is configured with 512 wide-configured neural processing units 126. In addition, in this embodiment, the neural processing unit 126 includes a multiplexing register to receive the weight text 206 from the weight random access memory 124, such as the multiplexing register 705 of the seventh figure, so as to randomize the weights. The access memory 124 receives a row of data values and performs a collective rotator operation, which will be described in more detail in subsequent sections. In this example, the data array 2404 is a pixel array of 2560 rows x 1600 columns. As shown in the figure, when the architecture program performs a convolution calculation on the data array 2404 and the convolution kernel 2402, the data array 2402 is divided into 20 data blocks, and each data block is a 512x400 data array 2406.

In this example, the convolution kernel 2402 is a 3x3 array of coefficients, weights, parameters, or elements. The first column of these coefficients is labeled C0,0; C0,1; and C0,2; the second column of these coefficients is labeled C1,0; C1,1; and C1,2; the third column of these coefficients is labeled C2,0; C2,1; and C2,2. For example, a convolution kernel with the following coefficients can be used to perform edge detection: 0,1,0,1, -4,1,0,1,0. In another embodiment, a convolution kernel with the following coefficients can be used to perform a Gaussian fuzzy operation: 1,2,1,2,4,2,1,2,1. In this example, it is often tiring A division is performed on the added value, where the divisor is the sum of the absolute values of the elements of the convolution kernel 2042, which is 16 in this example. In another example, the divisor may be the number of elements of the convolution kernel 2042. In yet another example, the divisor may be a value used to compress the convolution operation to a target value range. The divisor is determined by the element value of the convolution kernel 2042, the target range, and the input value array that performs the convolution operation. Decided by the scope.

Please refer to the twenty-fourth figure and the twenty-fifth figure in which the details are detailed. The architecture program writes the coefficients of the convolution kernel 2042 into the data random access memory 122. In a preferred embodiment, all text on each row of the nine consecutive rows of data random access memory 122 (the number of elements in the convolution kernel 2402) will use different elements of the convolution kernel 2402 to The columns are written in their main order. That is, as shown in the figure, each character in the same column is written with a first coefficient C0,0; the next column is written with a second coefficient C0,1; the next column is written with a third coefficient C0,2 is written; the next column is written with a fourth coefficient C1,0; and so on until each character in the ninth column is written with a ninth coefficient C2,2. In order to perform a convolution operation on the data matrix 2406 of the data block segmented by the data array 2404, the neural processing unit 126 repeatedly reads the nine rows of the coefficients of the convolution kernel 2042 loaded in the random access memory 122 according to the sequence. This part In the subsequent chapters, especially the part corresponding to Figure 26A, there will be more detailed explanation.

Please refer to the twenty-fourth figure and the twenty-fifth figure in which the details are detailed. The architecture program writes the values of the data matrix 2406 into the weight random access memory 124. When the neural network unit program performs a convolution operation, it writes the resulting array back to the weight random access memory 124. Just compare In the preferred embodiment, the architecture program writes a first data matrix 2406 into the weight random access memory 124 and starts the neural network unit 121. When the neural network unit 121 is When the kernel 2402 performs a convolution operation, the architecture program writes a second data matrix 2406 into the weight random access memory 124. In this way, after the neural network unit 121 completes the convolution operation of the first data matrix 2406, it can The convolution operation of the second data matrix 2406 is started. This part is explained in more detail in the subsequent part corresponding to the twenty-fifth figure. In this way, the framework program will go back and forth between the two regions of the weight random access memory 124 to ensure that the neural network unit 121 is fully used. Therefore, the example in the twenty-fourth figure shows a first data matrix 2406A and a second data matrix 2406B. The first data matrix 2406A corresponds to the first data block in the row 0 to 399 in the occupied random access memory 124. The second data matrix 2406B corresponds to the second data blocks of columns 500 to 899 in the occupied random access memory 124. In addition, as shown in the figure, the neural network unit 121 writes the result of the convolution operation back to columns 900-1299 and 1300-1699 of the weight random access memory 124, and then the architecture program will randomly access the memory from the weights. The body 124 reads these results. The data values of the data matrix 2406 loaded in the weight random access memory 124 are marked as "Dx, y", where "x" is the number of rows of the weight random access memory 124, and "y" is the weight random access memory Text, or number of lines. For example, the data text 511 in column 399 is labeled D399,511 in the twenty-fourth figure, and this data text is received by the multiplexer register 705 of the neural processing unit 511.

The twenty-fifth figure is a flowchart showing that the processor 100 of the first figure executes a framework program to use the neural network unit 121 to The data array 2404 of the fourteenth figure performs the convolution operation of the convolution kernel 2042. This process starts at step 2502.

In step 2502, the processor 100, that is, the processor 100 executing the architecture program, writes the convolution kernel 2402 of FIG. 24 into the data random access memory in the manner described in the display of FIG. 24. 122. In addition, the framework program initializes a variable N to a value of 1. The variable N indicates a data block being processed by the neural network unit 121 in the data array 2404. In addition, the framework program initializes a variable NUM_CHUNKS to the value 20. The flow then proceeds to step 2504.

In step 2504, as shown in FIG. 24, the processor 100 writes the data matrix 2406 of the data block 1 into the weight random access memory 124 (such as the data matrix 2406A of the data block 1). The flow then proceeds to step 2506.

In step 2506, the processor 100 uses a MTNN instruction 1400 that specifies a function 1432 to write the program memory 129, and writes a convolution program to the program memory 129 of the neural network unit 121. The processor 100 then uses a MTNN instruction 1400 that specifies a function 1432 to start executing the program to start the neural network unit convolution program. An example of a neural network unit convolution program is explained in more detail at the position corresponding to the twenty-sixth A figure. The flow then proceeds to step 2508.

In decision step 2508, the architecture program determines whether the value of the variable N is less than NUM_CHUNKS. If yes, the flow advances to step 2512; otherwise, it advances to step 2514.

At step 2512, as shown in Figure 24, the processor 100 writes the data matrix 2406 of data block N + 1 into the weight random access memory 124 (such as the data matrix 2406B of data block 2). Therefore, when the neural network unit 121 is performing a convolution operation on the current data block, the architecture program can write the data matrix 2406 of the next data block into the weight random access memory 124. Thus, after completing the current data block, After the convolution operation, that is, after the weight random access memory 124 is written, the neural network unit 121 can immediately start performing a convolution operation on the next data block.

At step 2514, the processor 100 confirms whether the executing neural network unit program (for data block 1 but from step 2506 and for data block 2-20 from step 2518) has completed execution. In a preferred embodiment, the processor 100 reads the state register 127 of the neural network unit 121 by executing an MFNN instruction 1500 to confirm whether the execution has been completed. In another embodiment, the neural network unit 121 generates an interrupt, indicating that the convolution routine has been completed. The flow then proceeds to decision step 2516.

In decision step 2516, the architecture program determines whether the value of the variable N is less than NUM_CHUNKS. If yes, the flow proceeds to step 2518; otherwise, it proceeds to step 2522.

In step 2518, the processor 100 updates the convolution program to execute on the data block N + 1. More precisely, the processor 100 updates the column value of the initialization neural processing unit instruction corresponding to the address 0 in the weight random access memory 124 to the first column of the data matrix 2406 (for example, to the data matrix 2406A). Column 0 or column 500 of data matrix 2406B), and the output column is updated (for example, to column 900 or 1300). The processor 100 will then execute the updated neural network unit volume. Product formula. The flow then proceeds to step 2522.

In step 2522, the processor 100 reads the execution result of the neural network unit convolution program of the data block N from the weighted random access memory 124. The flow then proceeds to decision step 2524.

In decision step 2524, the architecture program determines whether the value of the variable N is less than NUM_CHUNKS. If yes, the flow advances to step 2526; otherwise, it ends.

In step 2526, the framework program increases the value of N by one. The flow then returns to decision step 2508.

Figure 26A is a program list of a neural network unit program. This neural network unit program uses the convolution kernel 2402 of figure 24 to perform a convolution operation of a data matrix 2406 and writes it back. Weight random access memory 124. This program loops a certain number of instruction loops consisting of the instructions at addresses 1 to 9. The initialization neural processing unit instruction at address 0 specifies the number of times each neural processing unit 126 executes this instruction. The example in Figure 26A has a loop count value of 400, corresponding to the twenty-fourth. The number of columns in the data matrix 2406 of the figure, and the loop instruction (located at address 10) located at the loop terminal will decrement the current loop count value. If the result is non-zero, it will return to the command loop. Top (ie the instruction to return to address 1). The instruction to initialize the neural processing unit also clears the accumulator 202 to zero. For a preferred embodiment, the loop instruction at address 10 also clears the accumulator 202 to zero. In addition, the multiply-accumulate instruction at address 1 can also clear the accumulator 202 to zero.

For each execution of the in-program command loop, this The 512 neural processing units 126 will simultaneously perform the convolution operations of 512 3x3 convolution kernels and 512 corresponding 3x3 submatrices of the data matrix 2406. The convolution operation is the sum of the nine products calculated from the elements of the convolution kernel 2042 and the corresponding elements in the corresponding sub-matrix. In the embodiment of Figure 26A, the origin (central element) of each of the 512 corresponding 3x3 sub-matrixes is the data text Dx + 1, y + 1 in Figure 24, where y (Row number) is the number of the neural processing unit 126, and x (row number) is the row number read by the current weight random access memory 124 by the multiply-accumulate instruction of address 1 in the program of Figure 26A (This column number will also be initialized by the initialization neural processing unit instruction at address 0. It will also be incremented when the multiply accumulate instruction at addresses 3 and 5 is executed, and it will also be updated by the decrement instruction at address 9.) Thus, in each loop of the program, the 512 neural processing units 126 calculate 512 convolution operations and write the results of the 512 convolution operations back to the instruction line of the weight random access memory 124. In this article, edge handling is omitted to simplify the description, but it should be noted that using the collective rotation features of these neural processing units 126 will cause the data matrix 2406 (for the image processor, the data matrix of the image). Wrapping occurs in two rows of the multi-line data from the vertical edge on one side to the other vertical edge (for example, from the left edge to the right edge and vice versa). The instruction loop will now be described.

Address 1 is a multiply accumulate instruction. This instruction will specify row 0 of data random access memory 122 and secretly use the current weight of random access memory 124. This row is preferably loaded in sequencer 128 (and Initialized to zero by the instruction at address 0 to execute the first instruction Operation of loop pass). That is, the instruction at address 1 will cause each neural processing unit 126 to read its corresponding text from column 0 of the data random storage memory 122 and read its corresponding from the current weight random access memory 124 column. Text, and perform a multiply-accumulate operation on the two texts. So, for example, the neural processing unit 5 multiplies C0,0 by Dx, 5 (where "x" is the current weight random access memory 124 columns), adds the result to the accumulator 202 value 217, and adds the total Write back to accumulator 202.

Address 2 is a multiply accumulate instruction. This instruction specifies that the row of data random access memory 122 is incremented (ie, increased to 1), and then the row is read from the address of the data random access memory 122 after incrementing. This instruction also specifies that the value in the multiplex register 705 of each neural processing unit 126 is rotated to the neighboring neural processing unit 126. In this example, it is the random access memory from the weight corresponding to the instruction at address 1. The row of data matrix 2406 values read by volume 124. In the embodiments of the twenty-fourth to twenty-sixth figures, these neural processing units 126 are used to rotate the values of the multiplex register 705 to the left, that is, from the neural processing unit J to the neural processing unit J- 1, instead of rotating from the neural processing unit J to the neural processing unit J + 1 as described in the third, seventh, and nineteenth figures. It is worth noting that in the embodiment where the neural processing unit 126 rotates to the right, the architecture program will write the convolution kernel 2042 coefficient values into the data random access memory 122 (for example, rotate around its center row) in different orders to achieve The purpose of similar convolution results. In addition, the architecture program can perform additional pre-processing of the convolution kernel (such as transposition) when needed. In addition, the instruction specifies a count value of two. Therefore, the instruction at address 2 will cause each neural processing unit 126 to read its corresponding text from row 1 of the data random access memory 122 and rotate it. The post characters are received in the multiplex register 705, and a multiply-accumulate operation is performed on the two characters. Because the count value is 2, this instruction also causes each neural processing unit 126 to repeat the aforementioned operation. In other words, the sequencer 128 increments the data random access memory 122 row address 123 (ie, increases to 2), and each neural processing unit 126 reads from the data random access memory row 122 The corresponding characters and the rotated characters are received into the multiplexing register 705, and a multiply-accumulate operation is performed on the two characters. So, for example, assuming that the current weight random access memory 124 is column 27, after executing the instruction of address 2, the neural processing unit 5 will multiply the product of C0,1 and D27,6 by C0,2 and D27, The multiplication of 7 is added to its accumulator 202. In this way, after completing the instructions of address 1 and address 2, the product of C0,0 and D27,5, the product of C0,1 and D27,6 and C0,2 and D27,7 will be added to accumulator 202, add All other accumulated values from the previously passed instruction loop.

The operations performed by the instructions at addresses 3 and 4 are similar to the instructions at addresses 1 and 2, using the effect of increasing the index of the 124 rows of random access memory. These instructions will Operations are performed, and these instructions perform operations on the next three rows of data random access memory 122, that is, rows 3 to 5. That is, taking the neural processing unit 5 as an example, after completing the instructions of addresses 1 to 4, the product of C0,0 and D27,5, the product of C0,1 and D27,6, and the product of C0,2 and D27,7 The product, the product of C1,0 and D28,5, the product of C1,1 and D28,6, and the product of C1,2 and D28,7 are accumulated to accumulator 202, and all other loops from the previously passed instruction loop are added. Cumulative value.

The instructions at addresses 5 and 6 perform operations similar to the instructions at addresses 3 and 4. These instructions will randomly access the memory 124. The next row, and the subsequent three rows of the data random access memory 122, rows 6 to 8, perform operations. That is, taking the neural processing unit 5 as an example, after completing the instructions of addresses 1 to 6, the product of C0,0 and D27,5, the product of C0,1 and D27,6, and the product of C0,2 and D27,7 Product, product of C1,0 and D28,5, product of C1,1 and D28,6, product of C1,2 and D28,7, product of C2,0 and D29,5, product of C2,1 and D29,6, And the product of C2,2 and D29,7 will be accumulated to accumulator 202, adding all other accumulated values from the previous command loop. That is, after completing the instructions at addresses 1 to 6, it is assumed that at the beginning of the instruction loop, the weight random access memory 124 is listed as 27. Taking the neural processing unit 5 as an example, the convolution kernel 2042 will be used for the following 3x3 The submatrix performs the convolution operation: D27,5 D27,6 D27,7 D28,5 D28,6 D28,7 D29,5 D29,6 D29,7 In general, after completing the instructions of addresses 1 to 6, these 512 Each of the neural processing units 126 has used the convolution kernel 2042 to perform convolution operations on the following 3x3 submatrices: Dr, n Dr, n + 1 Dr, n + 2 Dr + 1, n Dr + 1, n + 1 Dr + 1 , n + 2 Dr + 2, n Dr + 2, n + 1 Dr + 2, n + 2 where r is the value of the column address of the random access memory 124 at the beginning of the instruction loop, and n is the neural processing Number of unit 126.

The instruction at address 7 passes the value 217 of the accumulator 202 through the activation function unit 121. This pass-through function passes a text whose size (in bits) is equivalent to data random access memory Text read by 122 and weighted random access memory 124 (in this example, 16 bits). In a preferred embodiment, the user can specify an output format, such as how many bits in the output bits are fractional bits. This section will be described in more detail in subsequent sections. In addition, this designation can specify a division start function instead of specifying a pass-through start function. This division start function divides the value 217 of the accumulator 202 by a divisor, as shown in the figure corresponding to the figures 29A and 30 For example, one of the "dividers" 3014/3016 of the thirty figure is used. For example, for a convolution kernel 2042 with coefficients, such as the aforementioned Gaussian fuzzy kernel with a coefficient of one-sixteenth, the instruction at address 7 will specify a division start function (for example, divide by 16), and Unspecified a transfer function. In addition, the structure program can perform the operation of dividing the convolution kernel 2042 coefficient by 16 before writing the convolution kernel coefficient into the data random access memory 122, and adjust the decimal point of the convolution kernel 2042 value accordingly. For example, use the data decimal point 2922 of the 29th figure as shown below.

The instruction at address 8 will write the output of the activation function unit 212 into the row specified by the current value of the output register in the weight random access memory 124. This current value is initialized by the instruction at address 0, and the value is incremented by the increment indicator within the instruction every time a loop is passed.

As shown in the examples of the twenty-fourth to twenty-sixth figures with a 3x3 convolution kernel 2402, the neural processing unit 126 reads the weight random access memory 124 to read the data matrix 2406 approximately every three time-frequency cycles. One column, and the convolution kernel result matrix is written into the weight random access memory 124 approximately every twelve time-frequency periods. In addition, it is assumed that an embodiment There is a write and read buffer as one of the buffers 1704 in the seventeenth figure. While the neural processing unit 126 performs reading and writing, the processor 100 can read the weight random access memory 124 Fetch and write. The buffer 1704 performs read and write operations on the weighted random access memory approximately every sixteen time-frequency cycles to read the data matrix and write the convolution kernel result matrix, respectively. Therefore, about half of the bandwidth of the weighted random access memory 124 is consumed by the convolution kernel operation performed by the neural network unit 121 in a hybrid manner. This example contains a 3x3 convolution kernel 2042, but the invention is not limited to this. Convolution kernels of other sizes, such as 2x2, 4x4, 5x5, 6x6, 7x7, 8x8, etc., can also be applied to different neural networks. Unit program. In the case of using a larger convolution kernel, because the rotated version of the multiply accumulate instruction (such as the instructions at address 2, 4, and 6 of Figure 26A, larger convolution kernels will need to use these instructions). With a larger count value, the proportion of time for the neural processing unit 126 to read the weighted random access memory 124 will decrease, so the bandwidth usage ratio of the weighted random access memory 124 will also decrease.

In addition, the structural program can make the neural network unit program overwrite the rows that are no longer needed in the input data matrix 2406 instead of writing the convolution operation results back to different rows of the weight random access memory 124 (such as row 900 -1299 and 1300-1699). For example, for a 3x3 convolution kernel, the architecture program can write the data matrix 2406 into the weighted random access memory 124 rows 2-401 instead of the rows 0-399, and the neural processing unit program The result of the convolution operation is written from column 0 of the weight random access memory 124, and the number of columns is incremented after each instruction loop is passed. In this way, the neural network unit program will only Use the columns for overwriting. For example, after the first pass through the instruction loop (or more precisely, after the instruction at address 1 is executed, its load weight random access memory 124 row 0), the data of row 0 can be overwritten Write, however, the data in columns 1-3 need to be left for the second pass through the instruction loop operation and cannot be overwritten; similarly, after the second pass through the instruction loop, the data in column 1 can be overwritten However, the data in columns 2-4 need to be left for the third pass through the instruction loop operation and cannot be overwritten; and so on. In this embodiment, the height (for example, 800 rows) of each data matrix 2406 (data block) can be increased, so fewer data blocks can be used.

In addition, the structure program can make the neural network unit program write the results of the convolution operation back to the data random access memory 122 rows above the convolution kernel 2402 (for example, above the row 8), instead of writing the convolution operation results back. The weight random access memory 124, when the neural network unit 121 writes the result, the framework program can read the result from the data random access memory 122 (for example, using the data random access memory 122 in the twenty-sixth figure). Address 2606 was recently written). This configuration is suitable for the embodiment with a port weight random access memory 124 and a dual port data random access memory.

According to the calculations of the neural network unit 121 in the embodiments of the twenty-fourth to twenty-sixth figures A, it can be found that each execution of the program of the twenty-sixth figure A requires about 5000 time-frequency cycles. Thus, the second The convolution operation of the entire 2560x1600 data array 2404 in the fourteenth figure requires about 100,000 time-frequency cycles, which is significantly less than the number of time-frequency cycles required to perform the same task in a traditional manner.

Figure 26B shows the neural network sheet of the first picture Block diagram of one embodiment of some fields of the control register 127 of the element 121. The status register 127 includes a field 2602 indicating the address of the row in the weight random access memory 124 that was recently written by the neural processing unit 126; a field 2606 indicating the latest address in the data random access memory 122. The address of the column written by the neural processing unit 126; a field 2604 indicating the address of the row in the weight random access memory 124 that was recently read by the neural processing unit 126; and a field 2608 indicating that the data is random The address in the memory 122 that was recently read by the neural processing unit 126. In this way, the architecture program running on the processor 100 can confirm the processing progress of the neural network unit 121, and read and / or write data from the data random access memory 122 and / or the weight random access memory 124. Time. Utilize this ability, plus the above option to overwrite the input data matrix (or write the result to the data random access memory 122 as described above), as shown in the following example, the data array 2404 of Figure 24 It can be implemented as 5 512x1600 data blocks instead of 20 512x400 data blocks. The processor 100 writes the first 512x1600 data block starting from column 2 of the weight random access memory 124 and starts the neural network unit program (this program has a cycle count of 1600 and randomly weights The output column of the access memory 124 is initialized to 0). When the neural network unit 121 executes the neural network unit program, the processor 100 monitors the output position / address of the weighted random access memory 124 to read the weighted random access memory by (1) (using the MFNN instruction 1500) The volume 124 has a column of valid convolution operation results written by the neural network unit 121 (starting from column 0); and (2) overwriting the second 512x1600 data matrix 2406 (starting from column 2) to the Read the results of valid convolution operations, In this way, when the neural network unit 121 completes the neural network unit program for the first 512x1600 data block, the processor 100 can immediately update the neural network unit program and start the neural network unit program again to run on the second 512x1600 when necessary. Data block. This procedure will be repeated three times to execute the remaining three 512x1600 data blocks so that the neural network unit 121 can be fully used.

In one embodiment, the starting function unit 212 has the ability to effectively perform an effective division operation on the value 217 of the accumulator 202, which is particularly corresponding to the figures in the subsequent chapters, 29A, 29B, and 30. There will be more detailed instructions. For example, a start function neural network unit instruction that divides the value of the accumulator 202 by a division of 16 can be used for a Gaussian fuzzy matrix described below.

The convolution kernel 2402 used in the example in Figure 24 is a small static convolution kernel applied to the entire data matrix 2404. However, the present invention is not limited to this. The convolution kernel can also be a large matrix. Having specific weights correspond to different data values of the data array 2404, such as convolution kernels commonly used in convolutional neural networks. When the neural network unit 121 is used in this way, the architecture program will interchange the positions of the data matrix and the convolution kernel, that is, place the data matrix in the data random access memory 122 and place the convolution kernel in random weights. The memory 124 is accessed, and the number of rows required to execute the neural network unit program is relatively small.

The twenty-seventh figure is a block diagram showing an example of the weighted random access memory 124 filled with input data in the first figure, and the input data is performed by the neural network unit 121 in the first figure (pooling operation). The common source operation is performed by a common source layer of an artificial neural network. By obtaining a sub-region or sub-matrix of the input matrix and calculating the maximum or average value of the sub-matrix as a result matrix, that is, the common source matrix, the input data matrix is reduced ( (Such as an image or a convolution image). In the examples of Figures 27 and 28, the common source operation calculates the maximum value of each sub-matrix. Common source computing is particularly useful for artificial neural networks such as performing object classification or detection. In general, the common source operation can actually reduce the factor of the input matrix to the number of elements of the detected sub-matrix, in particular, reduce the dimensions of the input matrix to the number of elements in the corresponding dimension of the sub-matrix. In the example in the twenty-seventh figure, the input data is a 512x1600 matrix of wide text (such as 16 bits), which is stored in rows 0 to 1599 of the weight random access memory 124. In the twenty-seventh figure, these characters are indicated by their column positions. For example, the characters at column 0 and row 0 are labeled D0,0; the characters at column 0 and row 1 are labeled D0,1; The text in column 0, row 2 is labeled D0,2; and so on, the text in column 0, row 511 is labeled D0,511. Similarly, the text on column 1 row 0 is labeled D1,0; the text on column 1 row 1 is labeled D1,1; the text on column 1 row 2 is labeled D1,2; and so on, the column The text on line 511 is marked as D1,511; and so on, the text on line 1599 is marked as D1599,0 on line 0; the text on line 1599 is marked on D1599,1 is located on line 1599 on line 2 The text is labeled D1599,2; and so on, the text at column 1599, line 511 is labeled D1599,511.

The twenty-eighth figure is a program list of a neural network unit program. The neural network unit program executes the output of the twenty-seventh figure. The common source operation of the data matrix is entered and written back to the weight random access memory 124. In the example in Figure 28, the common source operation calculates the maximum value of each 4x4 sub-matrix in the input data matrix. This program will execute the instruction loop consisting of instructions 1 to 10 multiple times. The initialized neural processing unit instruction at address 0 specifies the number of times each instruction is executed by the neural processing unit 126. In the example in Figure 28, the loop count value is 400, and at the end of the loop (in place) The loop instruction at address 11) will decrement the current loop count value, and if the result is a non-zero value, return it to the top of the instruction loop (ie, return to the instruction at address 1). The input data matrix in the weight random access memory 124 will be substantially regarded by the neural network unit program as 400 mutually exclusive groups consisting of four adjacent rows, that is, rows 0-3, 4-7, and 4 8-11, and so on, until column 1596-1599. Each group consisting of four adjacent columns includes 128 4x4 sub-matrixes. These sub-matrixes are 4x4 sub-matrixes formed by the elements at the intersection of the four columns of the group and four adjacent rows. Adjacent lines are lines 0-3, 4-7, lines 8-11, and so on until lines 508-511. Of the 512 neural processing units 126, each four is a group of calculations. The fourth neural processing unit 126 (a total of 128) performs a common source operation on a corresponding 4x4 submatrix, while the other three neural processing units 126 is not used. More precisely, the neural processing unit 0, 4, 8, and so on until the neural processing unit 508 will perform a common source operation on its corresponding 4x4 submatrix, and the leftmost row number of this 4x4 submatrix corresponds to The number of the neural processing unit, and the lower row corresponds to the value of the current weight random access memory 124. This value will be initialized to zero by the initialization instruction at address 0 and will increase by 4 after each instruction cycle. Some will be explained in more detail in subsequent chapters Bright. The repeated operations of the 400 instruction loops correspond to the number of 4x4 sub-matrix groups in the input data matrix of the twenty-seventh figure (that is, the 1,600 columns of the input data matrix divided by 4). Initializing the neural processing unit instructions also clears the accumulator 202 to zero. In a preferred embodiment, the loop instruction at address 11 also clears the accumulator 202 to zero. In addition, the maxwacc instruction at address 1 specifies to clear accumulator 202 to zero.

Each time the instruction loop of the program is executed, the 128 used neural processing units 126 perform 128 common source operations on 128 individual 4x4 sub-matrices in the current four-row group of the input data matrix. Further, the common source operation will confirm the maximum element among the 16 elements of the 4x4 sub-matrix. In the embodiment of the twenty-eighth figure, for each of the 128 neural processing units 126 used, the left element below the 4x4 sub-matrix is the element Dx in the twenty-seventh figure , y, where x is the number of rows of the current weight random access memory 124 at the beginning of the instruction loop, and this row of data is read by the maxwacc instruction at address 1 in the program in Figure 28 (the number of rows is also It will be initialized by the initialization neural processing unit instruction at address 0 and incremented each time the maxwacc instruction at addresses 3, 5 and 7 is executed). Therefore, for each loop of this program, the 128 used neural processing units 126 write back the weighted random access memory 124 of the maximum element of the corresponding 128 4x4 sub-matrix of the current column group. Of the specified column. The following is a description of this instruction loop.

The maxwacc instruction at address 1 implicitly uses the current weighted random access memory 124 rows. This row is preferably loaded in the sequencer 128 (and initialized to zero by the instruction at address 0 to execute One pass passes through the instruction loop operation). The instruction at address 1 will cause each neural processing unit 126 to read its corresponding text from the current row of the weight random access memory 124, compare this text with the value 217 of the accumulator 202, and add the two values to The largest is stored in the accumulator 202. Therefore, for example, the neural processing unit 8 confirms the maximum value in the accumulator 202 value 217 and the data text Dx, 8 (where "x" is the current weight random access memory 124 rows) and writes it back to the accumulator 202.

Address 2 is a maxwacc instruction. This instruction specifies that the value in the multiplexer register 705 of each neural processing unit 126 is rotated to the nearest to the neural processing unit 126. This is the address corresponding to the instruction of address 1. The weight random access memory 124 reads a row of input data array values. In the embodiments of Figures 27 to 28, the neural processing unit 126 is used to rotate the value of the multiplexer 705 to the left, that is, from the neural processing unit J to the neural processing unit J-1, as described above. Corresponding to the chapters in figures 24 to 26. In addition, this instruction specifies a count value of 3. In this way, the instruction at address 2 will cause each neural processing unit 126 to receive the rotated text to the multiplexer register 705 and confirm the maximum value between the rotated text and the accumulator 202 value, and then repeat this operation twice more. . In other words, each neural processing unit 126 performs three operations of receiving the rotated text to the multiplexer register 705 and confirming the maximum value between the rotated text and the value of the accumulator 202. So, for example, suppose that at the start of this instruction loop, the current weight random access memory 124 is 36, and the neural processing unit 8 is taken as an example. After executing the instructions of addresses 1 and 2, the neural processing unit 8 will It will store the accumulator 202 and its four weighted random access memories 124 at the start of the loop in its accumulator 202. The text D36,8, The maximum of D36,9, D36,10 and D36,11.

The operations performed by the maxwacc instructions at addresses 3 and 4 are similar to the instructions at address 1, using the power of the 124-row incrementing index of the weight random access memory. The instructions at addresses 3 and 4 will perform random access to the weight. Executed under 124. In other words, assuming that the current weight random access memory 124 column is 36 at the beginning of the instruction loop, the neural processing unit 8 is taken as an example. After completing the instructions at addresses 1 to 4, the neural processing unit 8 will accumulate the instructions. The accumulator 202 and the eight weighted random access memories 124 at the beginning of the loop are stored in the memory 202. The maximum value of D37,11.

The operations performed by the maxwacc instructions at addresses 5 to 8 are similar to the instructions at addresses 1 to 4. The instructions at addresses 5 to 8 are executed on the two rows below the weight random access memory 124. That is, assuming that the current weight random access memory 124 column is 36 at the beginning of the instruction loop, the neural processing unit 8 is taken as an example. After completing the instructions at the addresses 1 to 8, the neural processing unit 8 will accumulate the instructions. The accumulator 202 and sixteen weighted random access memories 124 at the beginning of the loop are stored in the memory 202. The text D36, 8, D36, 9, D36, 10, D36, 11, D37, 8, D37, 9, D37, 10 , D37,11, D38,8, D38,9, D38,10, D38,11, D39,8, D39,9, D39,10 and D39,11. That is, assuming that the current weight random access memory 124 column at the beginning of the instruction loop is 36, taking the neural processing unit 8 as an example, after completing the instructions at addresses 1 to 8, the neural processing unit 8 will complete the confirmation of the following Maximum of 4x4 sub-matrix: D36,8 D36,9 D36,10 D36,11 D37,8 D37,9 D37,10 D37,11 D38,8 D38,9 D38,10 D38,11 D39,8 D39,9 D39,10 D39,11 Basically, after completing the instructions of addresses 1 to 8, the 128 of the 128 neural processing units used Each neural processing unit 126 will confirm the maximum of the following 4x4 sub-matrix: Dr, n Dr, n + 1 Dr, n + 2 Dr, n + 3 Dr + 1, n Dr + 1, n + 1 Dr + 1, n + 2 Dr + 1, n + 3 Dr + 2, n Dr + 2, n + 1 Dr + 2, n + 2 Dr + 2, n + 3 Dr + 3, n Dr + 3, n + 1 Dr + 3, n + 2 Dr + 3, n + 3 where r is the column address value of the current weight random access memory 124 at the beginning of the instruction loop, and n is the number of the neural processing unit 126.

The instruction at address 9 passes the value 217 of the accumulator 202 through the enable function unit 212. This transfer function will pass a text whose size (in bits) is equivalent to the text read by the weight random access memory 124 (in this example, 16 bits). In a preferred embodiment, the user can specify an output format, such as how many bits in the output bits are fractional bits. This section will be described in more detail in subsequent sections.

The instruction at address 10 will write the value 217 of the accumulator 202 into the weighted random access memory 124 in the row specified by the current value of the output register. This current value will be initialized by the instruction at address 0, and Use the increment indicator in the instruction to increment this value after each pass through the loop. Further, the instruction at the address 10 writes a wide text (such as 16 bits) of the accumulator 202 into the weight random access memory 124. In a preferred embodiment, this instruction will output the 16 bits according to the output binary. The decimal point is 2916 for writing. This part will be explained in more detail in the following figures corresponding to the 29th A and 29B figures.

As mentioned above, the iterative instruction loop write weight random access memory 124 row contains holes with invalid values. That is, the wide characters 1 to 3, 5 to 7, 9 to 11, and so on of the result 133 are invalid or unused until the wide characters 509 to 511. In one embodiment, the startup function unit 212 includes a multiplexer that enables the results to be merged into the adjacent text of the column buffer, such as the column buffer 1104 of FIG. 11 to write back the output weight random access memory. 124 columns. In a preferred embodiment, the start function instruction specifies the number of characters in each hole, and the number of characters in the hole controls the multiplexer merge result. In one embodiment, the number of holes can be specified as a value from 2 to 6 to combine the outputs of the 3x3, 4x4, 5x5, 6x6, or 7x7 sub-matrix of the common source. In addition, the architecture program running on the processor 100 reads the generated sparse (ie, has a hole) result row from the weighted random access memory 124 and uses other execution units 112, such as the media unit using the architecture merge instruction, such as The x86 single instruction multiple data flow extension (SSE) instruction performs a merge function. In a manner similar to the foregoing, and utilizing the hybrid nature of the neural network unit 121, the architecture program executing on the processor 100 can read the status register 127 to monitor the most recently written row of the random access memory 124 ( (For example, column 26602 in Figure 26B) to read one of the sparse result rows, merge them and write them back to the same row of the weight random access memory 124, so that the preparation is completed and it can be used as an input data. The matrix is provided for use by the next layer of the neural network, such as a convolutional layer or a traditional neural network layer (ie, a multiply-accumulate layer). In addition, the embodiments described herein perform co-sourcing with a 4x4 sub-matrix Operation, but the present invention is not limited to this. The neural network unit program of the twenty-eighth figure can be adjusted, and common source operations are performed with sub-matrices of other sizes, such as 3x3, 5x5, 6x6, or 7x7.

As can be found from the foregoing, the number of result rows written into the weighted random access memory 124 is a quarter of the number of rows of the input data matrix. Finally, the data random access memory 122 is not used in this example. However, the data random access memory 122 may be used instead of the weight random access memory 124 to perform common source operations.

In the embodiments of the twenty-seventh and twenty-eighth figures, the common source operation calculates the maximum value of the sub-region. However, the program in the twenty-eighth figure can be adjusted to calculate the average value of the sub-region. It is beneficial to replace the maxwacc instruction with the sumwacc instruction (summing the weight text with the accumulator 202 value 217) and starting the address 9 The function instruction is modified to divide the accumulation result by the number of elements in each subregion (preferably through the inverse multiplication operation described below), which is sixteen in this example.

It can be found from the calculation of the twenty-seventh and twenty-eighth graphs by the neural network unit 121 that each execution of the program of the twenty-eighth graph requires about 6000 time-frequency cycles to perform the calculations shown in the twenty-seventh graph. The entire 512x1600 data matrix performs a common source operation. The number of time-frequency cycles used in this operation is significantly less than the number of time-frequency cycles required to perform similar tasks in a traditional manner.

In addition, the framework program can make the neural network unit program write the results of the common source operation back to the 122 rows of data random access memory, instead of writing the results back to the weight random access memory 124. When the neural network unit 121 writes the results When writing data to random access memory 122 (e.g. using twentieth The data random access memory 122 in Figure 6B was recently written to the address of row 2606), and the architecture program will read the results from the data random access memory 122. This configuration is applicable to the embodiment having the port weight random access memory 124 and the dual port data random access memory 122.

Fixed-point arithmetic operations, with users providing binary decimal points, full-precision fixed-point accumulation, user-specified reciprocal values, random rounding of accumulator values, and optional start / output functions

Generally speaking, a hardware unit that performs arithmetic operations in a digital computing device is an integer or a floating-point number according to its object of performing arithmetic operations, and can generally be divided into "integer" units and "floating point" units. Floating point numbers have a magnitude (or mantissa) and an exponent, and usually have a sign. The index is an indicator of the position of the radix point (usually a binary decimal point) relative to the value. In contrast, integers have no exponent, but only have a value, and usually have a sign. The floating-point unit allows programmers to obtain the number they want to use for their work from a very large range of different values, while the hardware is responsible for adjusting the index value of this number when needed without the programmer's handling. For example, suppose two floating-point numbers 0.111 x 10 ^{29 are} multiplied by 0.81 x 10 ³¹ . (Although floating-point units usually work with 2-based floating-point numbers, decimal decimals or 10-based floating-point numbers are used in this example.) Floating-point units are automatically responsible for multiplying the mantissa and exponential And then normalize the result to the value .8911 x 10 ⁵⁹ . In another example, suppose the same two floating-point numbers are added. Floating-point units are automatically responsible for aligning the mantissa's decimal points before adding to produce a total of .81111 x 10 ³¹ .

However, it is well known that such a complicated operation will result in an increase in the size of the floating-point unit, an increase in power consumption, an increase in the number of time-frequency cycles required per instruction, and / or an increase in cycle time. For this reason, many devices (such as embedded processors, microcontrollers, and relatively low cost and / or low power microprocessors) do not have floating point units. As can be seen from the foregoing example, the complex structure of a floating-point unit contains logic for performing exponential calculations related to floating-point addition and multiplication / divide The adder subtracts the operand exponent to confirm that the decimal point of the floating-point addition is aligned with the offset of the offset). It includes an offsetr for achieving the decimal point alignment of the mantissa in floating-point addition. Contains an offsetter that normalizes floating-point results. In addition, the flow usually requires the logic of performing rounding operations on floating-point results, performing conversions between integer and floating-point formats, and conversions between different floating-point formats (such as augmented precision, double precision, single precision, and half precision). Logic for leading, zero and leading one detectors, and logic for handling special floating-point numbers, such as outliers, non-numeric values, and infinite values.

In addition, the verification of the accuracy of floating-point units will greatly increase the complexity of the design due to the increase in the numerical space that needs to be verified, which will extend the product development cycle and time to market. In addition, as mentioned above, floating-point arithmetic operations need to store and use the mantissa and exponent fields of each floating-point number used for calculation, respectively, and will increase the required storage space and / or in a given storage space to Reduced accuracy when storing integers. Many of these disadvantages can be avoided by performing arithmetic operations on integer units.

Programmers often need to write procedures for handling decimals Formula, the decimal is the value of the incomplete number. This program may need to be executed on a processor without a floating-point unit, or the processor may have a floating-point unit, but it is faster to execute integer instructions from the processor's integer unit. In order to take advantage of the performance advantages of integer processors, programmers use conventional fixed-point arithmetic operations on fixed-point numbers. Such a program would include instructions that execute on integer units to process integers or integer data. Software knows that data is decimal. This software does not include instructions to perform operations on integer data and processing this data is actually a decimal problem, such as aligning the offset. Basically, fixed-point software can manually perform functions that some or all floating-point units can perform.

In this article, a "fixed-point" number (or value or operand or input or output) is a number whose storage bits are understood to include bits to represent a fractional part of this fixed-point number. This bit is called here Is "decimal places". The storage point of a fixed-point number is contained in a memory or a register, such as an 8-bit or 16-bit text in the memory or the register. In addition, all the storage bits of a fixed-point number are used to express a value, and in some cases, one of the bits is used to express a symbol. However, there is no fixed-point number to store the bit. index. In addition, the fixed-point number or binary decimal point position is specified in a storage space different from the fixed-point storage bit, and the number or decimal of the decimal point is indicated in a shared or universal manner. The position of the decimal point is shared with a fixed-point number set containing the fixed-point number, such as a set of input operands, accumulated values, or output results of an array of processing units.

In the embodiment described here, the arithmetic logic unit is an integer Number unit, however, the start function unit contains hardware assistance or acceleration for floating point arithmetic. This can make the part of the arithmetic logic unit smaller and faster, which facilitates the use of more arithmetic logic units on a given chip space. This also means that more neurons can be set in the unit chip space, which is particularly beneficial to the neural network unit.

In addition, compared to each floating-point number requiring exponential storage bits, the fixed-point numbers in the embodiments described herein use one index to express the number of storage bits that belong to the decimal place in the entire set of numbers, however, This indicator is located in a single, shared storage space and broadly indicates all numbers of the entire set, such as the input set of a series of operations, the set of cumulative numbers of a series of operations, and the set of outputs, among which the number of decimal places. In a preferred embodiment, the user of the neural network unit can specify the number of decimal storage bits for this number set. Therefore, it can be understood that although in many cases (such as general mathematics), the term "integer" refers to a signed complete number, that is, a number without a decimal part, but in the context of this article, " The term "integer" can mean a number with a decimal part. In addition, in the context of this article, the term "integer" is used to distinguish from floating-point numbers. For floating-point numbers, some bits in their respective storage space are used to express the exponent of floating-point numbers. Similarly, integer arithmetic operations, such as integer multiplication or addition or comparison operations performed by integer units, assume that there is no exponent in the operands. Therefore, integer components of integer units, such as integer multipliers, integer adders, integer comparators, There is no need to include logic to handle the exponents, for example, it is not necessary to move the mantissa to align the decimal points for addition or comparison operations, and it is not necessary to add the exponents for multiplication operations.

In addition, the embodiments described herein include a large hardware integer accumulator to accumulate a large series of integer operations (such as 1000 multiply-accumulate operations) without losing accuracy. This can prevent the neural network unit from processing floating-point numbers, while maintaining the accumulative number with full precision without saturating it or producing inaccurate results due to overflow. Once the result of this series of integer operations is added to the full-accuracy accumulator, the fixed-point hardware assist will perform the necessary scaling and saturation operations to use the user-specified number of decimal places for the accumulated value and the output value required. The number of decimal places, this full-precision accumulative value is converted into an output value, this part will be explained in more detail in subsequent chapters.

When the accumulated value needs to be compressed from the full-precision form for input to one of the startup functions or used for transfer, in a preferred embodiment, the startup function unit can selectively perform a random rounding operation on the accumulated value. This section will be explained in more detail in subsequent chapters. Finally, depending on the different needs of a given layer of one of the neural networks, the neural processing unit can selectively accept instructions to use different activation functions and / or output many different forms of accumulated values.

Figure 29A is a block diagram showing an embodiment of the control register 127 of the first figure. The control register 127 may include a plurality of control registers 127. As shown in the figure, the control register 127 includes the following fields: configuration 2902, signed data 2912, signed weight 2914, data binary decimal point 2922, weighted binary decimal point 2924, arithmetic logic unit function 2926, Round control 2932, start function 2934, countdown 2942, offset 2944, output random access memory 2952, output binary decimal point 2954, and output command 2956. control The value of the register 127 can be written using the MTNN instruction 1400 and the NNU program instructions, such as the start instruction.

The configuration 2902 value specifies whether the neural network unit 121 belongs to a narrow configuration, a wide configuration, or a funnel configuration, as described above. The configuration 2902 also sets the size of the input text received by the data random access memory 122 and the weight random access memory 124. In the narrow and funnel configurations, the size of the input text is narrow (for example, 8-bit or 9-bit), but in the wide configuration, the size of the input text is wide (for example, 12-bit or 16-bit) ). In addition, the configuration 2902 also sets the size of the output result 133 the same as the size of the input text.

When the signed data value 2912 is true, it means that the data text received by the data random access memory 122 is a signed value. If it is false, it means that these data texts are unsigned values. When the signed weight value 2914 is true, it means that the weight text received by the weight random access memory 122 is a signed value. If it is false, it means that these weight text are unsigned values.

The data binary decimal point 2922 value indicates the binary decimal point position of the data text received by the data random access memory 122. For a preferred embodiment, for the position of the binary decimal point, the value of the data binary decimal point 2922 represents the number of bit positions calculated from the right side of the binary decimal point. In other words, the data decimal point 2922 represents the number of decimal places in the least significant digits of the data text, that is, the number of digits to the right of the binary decimal point. Similarly, the weight binary decimal point 2924 value indicates the position of the binary decimal point of the weight text received by the weight random access memory 124. For a preferred embodiment, When the arithmetic logic unit function 2926 is a multiplication and accumulation or output accumulation, the neural processing unit 126 determines the number of digits to the right of the binary decimal point of the value loaded in the accumulator 202 as the data decimal point 2922 and the weight decimal point. The sum of points 2924. Therefore, for example, if the value of the data binary decimal point 2922 is 5 and the weighted binary decimal point 2924 is 3, the value in the accumulator 202 will have 8 bits to the right of the binary decimal point. When the arithmetic logic unit function 2926 is a total / maximum accumulator and data / weight text or passed data / weight text, the neural processing unit 126 will place the number of digits to the right of the decimal point of the binary value of the accumulator 202 Decided as data / weight binary decimal point 2922/2924 respectively. In another embodiment, a single accumulator binary decimal point 2923 is specified instead of specifying an individual data decimal point 2922 and a weighted decimal point 2924. This section will be explained in more detail in the subsequent section corresponding to Figure 29B.

The arithmetic logic unit function 2926 specifies a function to be executed by the arithmetic logic unit 204 of the neural processing unit 126. As mentioned above, the arithmetic logic unit function 2926 may include the following operations but is not limited to: multiplying the data text 209 by the weight text 203 and adding this product to the accumulator 202; adding the accumulator 202 and the weight text 203; adding up Adder 202 and data text 209; maximum value of accumulator 202 and data text 209; maximum value of accumulator 202 and weight text 209; output accumulator 202; pass data text 209; pass weight text 209; output zero value. In one embodiment, the arithmetic logic unit function 2926 is specified by a neural network unit initialization instruction and used by the arithmetic logic unit 204 to respond to an execution instruction (not shown). In one embodiment, the arithmetic logic sheet The metafunction 2926 is specified by individual neural network unit instructions, such as the aforementioned multiply accumulate and maxwacc instructions.

The rounding control 2932 specifies (Figure 30) the form of the rounding operation used by the rounder 3004. In an embodiment, the rounding modes that can be specified include, but are not limited to: no rounding, rounding to the nearest value, and random rounding. For a preferred embodiment, the processor 100 includes a random bit source 3003 (refer to Figure 30) to generate random bits 3005. These random bits 3005 are sampled to perform random rounding to reduce The possibility of rounding bias. In one embodiment, when the rounded bit 3005 is one and the sticky bit is zero, if the sampled random bit 3005 is true, the neural processing unit 126 will round up. If it is the sampled random bit 3005 is false, the neural processing unit 126 will not round up. In one embodiment, the random bit source 3003 is sampled based on the random electronic characteristics of the processor 100 to generate random bits 3005. These random electronic characteristics are thermal noise of semiconductor diodes or resistors. Not limited to this.

The activation function 2934 specifies a function for the value 217 of the accumulator 202 to generate the output 133 of the neural processing unit 126. As described herein, the start function 2934 includes, but is not limited to: S-type function; hyperbolic tangent function; soft addition function; correction function; division by a specified power of two; multiplication by a user-specified inverse value to achieve, etc. Divide by effect; pass the entire accumulator; and pass the accumulator in standard size, which will be explained in more detail in the following sections. In one embodiment, the activation function is specified by a neural network unit activation function instruction. In addition, the startup function can also be specified by the initialization instruction and used in response to an output instruction. For example, the start function unit at address 4 in the fourth figure outputs an instruction. In this embodiment, the start function instruction at address 3 in the fourth figure is included in the output instruction.

The reciprocal 2942 value specifies a value that is multiplied by the value 217 of the accumulator 202 to achieve a division of the value 217 of the accumulator 202. In other words, the user-specified inverse 2942 value will be the inverse of the divisor actually desired to be performed. This facilitates the use of convolution or common source operations as described herein. In a preferred embodiment, the user designates the penultimate 2942 value as two parts, which will be described in more detail at a subsequent point corresponding to the twenty-ninth figure C. In one embodiment, the control register 127 includes a field (not shown) so that the user can specify one of a plurality of built-in division values to perform division. The size of these built-in division values is equivalent to a common convolution. The size of the core, such as 9, 25, 36, or 49. In this embodiment, the activation function unit 212 stores the inverses of these built-in divisors for multiplication with the value 217 of the accumulator 202.

The offset 2944 specifies the number of bits to shift the value 217 of the accumulator 202 to the right by one of the shifters of the activation function unit 212 to achieve the operation of dividing it by a power of two. This is conducive to the operation with a convolution kernel with a power of two.

The output random access memory 2952 value will designate one of the data random access memory 122 and the weight random access memory 124 to receive the output result 133.

The output binary point 2954 value indicates the position of the binary point of the output result 133 bis. For a preferred embodiment, for the position of the decimal point of the output 133 bis, the output binary is small. A 2954 count indicates the number of bit positions calculated from the right. In other words, the output decimal point 2954 represents the number of decimal places in the least significant bit of the output result 133, that is, the number of bits to the right of the binary decimal point. The startup function unit 212 will be based on the output binary decimal point 2954 (in most cases, it will also be based on the data binary decimal point 2922, the weighted binary decimal point 2924, the startup function 2934 and / or the value of the configuration 2902) Perform rounding, compression, saturation, and size conversion operations.

The output command 2956 outputs the result 133 from a number of control-oriented. In one embodiment, the activation function unit 121 uses the concept of a standard size, which is twice the width (in bits) specified by the configuration 2902. So, for example, if the configuration 2902 sets the size of the input text received by the data random access memory 122 and the weight random access memory 124 to 8 bits, the standard size will be 16 bits; in another example If the configuration 2902 sets the size of the input text received by the data random access memory 122 and the weight random access memory 124 to 16 bits, the standard size will be 32 bits. As described in this article, the size of the accumulator 202 is large (for example, the narrow accumulator 202B is 28 bits, and the wide accumulator 202A is 41 bits) to maintain intermediate calculations, such as 1024 and 512 nerves. Network unit multiply accumulate instructions with full precision. In this way, the value 217 of the accumulator 202 will be greater than (in bits) the standard size, and for most of the values of the start function 2934 (except for passing the entire accumulator), the start function unit 212 (for example, the following corresponds to the one in Figure 30) The standard size compressor 3008) described in the paragraph will compress the value 217 of the accumulator 202 to the size of the standard size. Output the first preview of command 2956 The setting value instructs the startup function unit 212 to execute the specified startup function 2934 to generate an internal result and output the internal result as an output result 133. The size of the internal result is equal to the size of the original input text, that is, half of the standard size. The second preset value of the output command 2956 will instruct the startup function unit 212 to execute the specified startup function 2934 to generate an internal result and output the lower half of the internal result as the output result 133. The size of the internal result is equal to the original input text. Twice the size, which is the standard size; and the third preset value of the output command 2956 instructs the activation function unit 212 to output the upper half of the internal result of the standard size as the output result 133. The fourth preset value of output command 2956 instructs startup function unit 212 to output the unprocessed least significant text of accumulator 202 as output result 133; and the fifth preset value of output command 2956 instructs startup function unit 212 to The unprocessed intermediate valid text of the accumulator 202 is output as the output result 133; the sixth preset value of the output command 2956 will instruct the activation function unit 212 to process the unprocessed highest valid text of the accumulator 202 (the width of which is determined by the configuration (Specified by 2902) is output as the output result 133, which is explained in more detail in the previous section corresponding to the eighth to tenth figures. As described above, outputting the internal result of the size of the accumulator 202 or the standard size is helpful to enable other execution units 112 of the processor 100 to execute startup functions, such as soft maximum startup functions.

The fields described in Figure 29A (and Figures 29B and 29C) are located inside the control register 127. However, the present invention is not limited to this. One or more of the fields It may also be located in other parts of the neural network unit 121. In a preferred embodiment, many of the fields may be contained within a neural network unit instruction and provided by the sequencer 128. It is decoded to generate a micro-instruction 3416 (refer to FIG. 34) to control the arithmetic logic unit 204 and / or start the function unit 212. In addition, these fields can also be included in the micro-operation 3414 (see FIG. 34) stored in the media register 118 to control the arithmetic logic unit 204 and / or activate the function unit 212. This embodiment can reduce the use of the initialization neural network unit instruction, and in other embodiments, the initialization neural network unit instruction can be removed.

As mentioned above, the neural network unit instruction can specify a memory operand (such as text from data random access memory 122 and / or weight random access memory 123) or a rotated operand (such as Registers 208/705) perform arithmetic logic instruction operations. In an embodiment, the neural network unit instruction may further designate an operand as a register output of an activation function (such as the output of the register 3038 in FIG. 30). In addition, as described above, the neural network unit instruction may be designated to increment the current row address of one of the data random access memory 122 or the weight random access memory 124. In one embodiment, the neural network unit instruction may specify an immediate signed integer difference value to be added to the current column to achieve the purpose of increasing or decreasing by a value other than one.

Figure 29B is a block diagram showing another embodiment of the control register 127 of the first figure. The control register 127 of Fig. 29B is similar to the control register 127 of Fig. 29A, but the control register 127 of Fig. 29B includes an accumulator binary decimal point 2923 . The accumulator binary decimal point 2923 indicates the position of the decimal point of the accumulator 202 binary. For a preferred embodiment, the value of the accumulator binary decimal point 2923 indicates that the position of the binary decimal point from the right Number of meta positions. In other words, the accumulator binary decimal point 2923 represents the number of decimal places in the least significant bit of the accumulator 202, that is, the bit located to the right of the binary decimal point. In this embodiment, the binary decimal point 2923 of the accumulator is clearly indicated, instead of being secretly confirmed as in the embodiment of Figure 29A.

Figure 29C is a block diagram showing an example of storing the penultimate 2942 of Figure 29A in two parts. The first part 2962 is an offset value indicating that the user wants to multiply the number 2962 of the true inverse value of the value 217 of the accumulator 202 by the leading zero before being suppressed. The number of leading zeros is the number of zeros arranged consecutively to the right of the decimal point. The second part 2694 is the leading zero suppression inverse value, that is, the true inverse value after removing all leading zeros. In one embodiment, the suppressed leading zero number 2962 is stored in 4 bits, and the leading zero suppression reciprocal value 2964 is stored in an 8-bit unsigned value.

For example, suppose a user wants to multiply the value 217 of the accumulator 202 by the inverse value of the value 49. The reciprocal value of the value 49 is displayed in two dimensions and the 13 decimal places are set to 0.0000010100111, of which there are five leading zeros. In this way, the user will fill the number of suppressed leading zeros 2962 with a value of 5 and the number of leading zeros with a suppression of 2964 into a value of 10100111. After the reciprocal multiplier "divider A" 3014 (refer to Figure 30) multiplies the value 217 of the accumulator 202 with the leading zero suppression reciprocal value 2964, the resulting product is shifted to the right by the suppressed leading zero number 2962. Such an embodiment helps to use a relatively small number of bits to express the inverse 2942 value to achieve the requirement of high accuracy.

Figure 30 shows the starting function unit of Figure 2. A block diagram of an embodiment of 212. The start function unit 212 includes the control logic 127 of the first figure, a positive type converter (PFC) and an output binary decimal point aligner (OBPA) 3002 to receive the accumulator 202 value 217, and a rounder 3004 to receive Index of accumulator 202 value 217 and number of bits shifted by output binary point aligner 3002, a random bit source 3003 as previously described to generate random bits 3005, and a first multiplexer 3006 to receive positive types The output of the converter and output binary decimal aligner 3002 and the output of the rounder 3004, a standard size compressor (CCS) and saturator 3008 to receive the output of the first multiplexer 3006, a bit selector And saturator 3012 to receive the output of the standard size compressor and saturator 3008, a corrector 3018 to receive the output of the standard size compressor and saturator 3008, and a reciprocal multiplier 3014 to receive the standard size compressor and saturator 3008 Output, a right shifter 3016 to receive the output of the standard size compressor and saturator 3008, a double tangent (tanh) module 3022 to receive the output of the bit selector and saturator 3012 An S-type module 3024 receives the output of the bit selector and saturator 3012, a soft addition module 3026 receives the output of the bit selector and saturator 3012, and a second multiplexer 3032 receives the double tangent The output of module 3022, S-type module 3024, soft addition module 3026, corrector 3018, reciprocal multiplier 3014 and right shifter 3016, and standard size output 3028 transmitted by standard size compressor and saturator 3008, A symbol restorer 3034 receives the output of the second multiplexer 3032, a size converter and saturator 3036 receives the output of the symbol restorer 3034, and a third multiplexer 3037 receives the output of size converter and saturator 3036. Output and accumulator output 217, and an output register 3038 receives the output of the multiplexer 3037, and the output is the result 133 in the first figure.

The positive type converter and output binary point aligner 3002 receives the accumulator 202 value 217. For a preferred embodiment, as mentioned above, the accumulator 202 value 217 is a full precision value. That is, the accumulator 202 has a sufficient number of storage bits to load an accumulative number. The accumulative number is the total number generated by the integer adder 244 adding a series of products generated by the integer multiplier 242, and this operation None of the individual products of the multiplier 242 or the individual totals of the adder are discarded to maintain accuracy. For a preferred embodiment, the accumulator 202 has at least a sufficient number of bits to load the maximum number of multiply-accumulate additions that can be generated by a neural network unit 121 that is programmed. For example, please refer to the program in the fourth figure. In a wide configuration, the maximum number of multiply-accumulate sums that can be generated by the neural network unit 121 by programmatic execution is 512, and the accumulative number is 202 bits wide. In another example, please refer to the program in the twentieth chart. In a narrow configuration, the maximum number of multiply-accumulate additions that can be generated by the neural network unit 121 by programmatic execution is 1024, and the accumulative number 202-bit width is 28. Basically, the full precision accumulator 202 has at least Q bits, where Q is the sum of M and log ₂ P, where M is the bit width of the integer product of multiplier 242 (for example, for narrow multiplier 242 16 bits, 32 bits for wide multiplier 242), and P is the maximum allowable number of products that accumulator 202 can accumulate. In a preferred embodiment, the maximum number of multiplications is specified according to the program specifications of the programmer of the neural network unit 121. In one embodiment, it is assumed that a previous multiply accumulate instruction is used to load data / weight random access memory 122/124 from the data / weight text 206/207 rows (such as the instruction at address 1 in the fourth figure). In the above, the sequencer 128 will execute a multiply accumulate neural network unit instruction (such as the instruction at address 2 in the fourth figure). The maximum count is, for example, 511.

The design of the arithmetic logic unit 204 of the neural processing unit 126 can be simplified by using an accumulator 202 having a sufficient bit width and capable of performing an accumulating operation on one of the maximum number of accumulative full-precision values that can be accumulated. In particular, this process can alleviate the need to use logic to perform saturation operations on the totals produced by the integer adder 244, because the integer adder 244 can cause a small accumulator to overflow, and it needs to keep track of the decimal places of the accumulator Click on the position to see if an overflow occurs to see if a saturation operation needs to be performed. For example, for a design with a non-full-precision accumulator but with saturation logic to handle the overflow of the non-full-precision accumulator, assume the following.

(1) The range of data text values is between 0 and 1 and all storage bits are used to store decimal places. The weight text value ranges from -8 to +8 and all storage bits except three are used to store decimal places. The accumulative value of the input as a hyperbolic tangent activation function is between -8 and 8, and all storage bits except three are used to store decimal places.

(2) The bit width of the accumulator is not full precision (such as only the bit width of the product).

(3) Assuming that the accumulator is full precision, the final accumulative value will also be between -8 and 8 (such as +4.2); however, the product before "point A" in this sequence will produce a positive value more frequently , And the product after point A will produce negative values more frequently. In this case, incorrect results (such as results other than +4.2) may be obtained. This is because at some points in front of point A, when the accumulator needs to reach a value that exceeds its saturation maximum +8, such as +8.2, an extra 0.2 will be lost. The accumulator will even maintain the remaining multiply-accumulate results at a saturated value, and lose more positive values. Therefore, the final value of the accumulator may be smaller than the value calculated using the accumulator with full precision bit width (ie, less than +4.2).

The positive type converter 3004 converts the accumulator 202 to a positive type when the value 217 of the accumulator 202 is negative, and generates an extra bit to indicate the positive or negative value of the original value. This bit is passed down to the start function unit 212 along with this value. Pipeline. Converting a negative number to a positive type can simplify the operation of the subsequent activation function unit 121. For example, after this processing, only positive values will be input to the hyperbolic tangent module 3022 and the S-type module 3024, so the design of these modules can be simplified. In addition, the rounder 3004 and the saturator 3008 can be simplified.

The output binary point aligner 3002 shifts or scales this positive type value to the right to align it with the output binary point 2954 specified in the control register 127. For a preferred embodiment, the output decimal point aligner 3002 calculates the number of decimal places of the value 217 of the accumulator 202 (for example, specified by the accumulator binary decimal point 2923 or the data binary decimal point 2922). And the weighted binary decimal point (total of 2924) minus the output decimal place number (eg, specified by the output binary decimal point 2954) as the offset. So, for example, if the accumulator 202 has a binary decimal point 2923 of 8 (that is, the above embodiment) and the output binary decimal point 2954 is 3, the binary decimal point aligner is output. 3002 will shift this positive type value 5 bits to the right to produce the result provided to multiplexer 3006 and rounder 3004.

The rounder 3004 performs a rounding operation on the accumulator 202 value 217. For a preferred embodiment, the rounder 3004 generates a rounded version of one of the positive type values generated by the positive type converter and the output binary point aligner 3002, and provides this rounded version Up to multiplexer 3006. The rounder 3004 performs a rounding operation according to the aforementioned rounding control 2932. As described herein, the aforementioned rounding control includes random rounding using a random bit 3005. The multiplexer 3006 will control 2932 according to the rounding (as described in this article, which may include random rounding), and select one of its multiple inputs, that is, from the positive type converter and the output decimal point aligner 3002 The positive type value is also a rounded version from the rounder 3004, and the selected value is provided to the standard size compressor and saturator 3008. In a preferred embodiment, if the rounding control specifies no rounding, the multiplexer 3006 will select the output of the positive type converter and the output binary decimal aligner 3002, otherwise the rounder will be selected. 3004 output. In other embodiments, an additional rounding operation may be performed by the startup function unit 212. For example, in one embodiment, when the bit selector 3012 compresses the output (as described below) of the standard size compressor and the saturator 3008, the bit selector 3012 will base on the missing low order bits Performs rounding. In another example, the product of the inverse multiplier 3014 (described below) is rounded. In another example, the size converter 3036 needs to convert an appropriate output size (as described later). This conversion may involve dropping some low order bits used to determine rounding, and then perform a rounding operation.

The standard size compressor 3008 compresses the output value of the multiplexer 3006 to a standard size. Therefore, for example, if the neural processing unit 126 is in a narrow configuration or a funnel configuration 2902, the standard size compressor 3008 can compress the output value of the 28-bit multiplexer 3006 to 16 bits; and if the neural processing unit 126 is In a wide configuration 2902, the standard size compressor 3008 can compress the output value of a 41-bit multiplexer 3006 to 32-bit. However, before compression to the standard size, if the value before compression is greater than the maximum value that can be expressed by the standard pattern, the saturator 3008 will fill this value before compression to the maximum value that can be expressed by the standard pattern. For example, if any bit in the pre-compression value that is to the left of the most significant pre-compression value bit is the value 1, the saturator 3008 will fill up to the maximum value (for example, if it fills all 1).

In a preferred embodiment, the hyperbolic tangent module 3022, the S-shaped module 3024, and the soft-plus module 3026 all include look-up tables, such as a programmable logic array (PLA), read-only memory (ROM) , Combined logic gates, and so on. In an embodiment, in order to simplify and reduce the size of these modules 3022/3024/3026, the input value provided to these modules has a type of 3.4, that is, three integer bits and four decimal places, that is, The input value has four bits to the right of the decimal point and three bits to the left of the decimal point. These values are chosen because the output value will approach the minimum / maximum value asymptotically at the extremes of the input value range (-8, +8) of type 3.4. However, the present invention is not limited to this, and the present invention can also be applied to other embodiments in which a binary decimal point is placed at a different position, such as a 4.3 type or a 2.5 type. Bit selector 3012 selects between the bits of the standard size compressor and saturator 3008 output. Bits that satisfy the 3.4 type specification involve compression processing, that is, some bits are lost, because the standard type has a larger number of bits. However, before selecting / compressing the output values of the standard size compressor and saturator 3008, if the value before compression is greater than the maximum value that can be expressed by 3.4 type, the saturator 3012 will fill the value before compression to the value that can be expressed by 3.4 type. The maximum value. For example, if any bit on the left side of the most significant 3.4 type bit in the pre-compression value is the value 1, the saturator 3012 will be filled to the maximum value (for example, filled to all 1).

The hyperbolic tangent module 3022, the S-type module 3024 and the soft-add module 3026 perform a corresponding activation function (as described above) on the 3.4-type values output by the standard size compressor and the saturator 3008 to produce a result. In a preferred embodiment, the hyperbolic tangent module 3022 and the S-shaped module 3024 produce a 0.7-bit 7-bit result, that is, zero integer bits and seven decimal places, that is, input The value has seven digits to the right of the decimal point. In a preferred embodiment, the soft add module 3026 produces a 7-bit result of the 3.4 type, that is, the type is the same as the input type of the module 3026. For a preferred embodiment, the output of the hyperbolic tangent module 3022, the S-shaped module 3024 and the soft plus module 3026 will be extended to the standard type (for example, leading zeros are added if necessary) and aligned so that The binary decimal point is specified by the output binary decimal point 2954 value.

The corrector 3018 generates a corrected version of one of the output values of the standard size compressor and the saturator 3008. That is to say, if the output values of the standard size compressor and saturator 3008 (the symbols are shifted down the pipeline as described above) are negative, the corrector 3018 will output a zero value; otherwise, the corrector 3018 will output its input value . For a preferred embodiment, The output of the corrector 3018 is a standard type and has a decimal point specified by the output binary point 2954 value.

The inverse multiplier 3014 multiplies the output of the standard size compressor and saturator 3008 by the user-specified inverse value specified in the inverse value 2942 to produce a standard size product. This product is actually the standard size compressor and saturation. The output value of the generator 3008 is a quotient calculated by taking the inverse of the inverse value 2942 as the divisor. In a preferred embodiment, the output of the reciprocal multiplier 3014 is a standard type and has a decimal point specified by the output binary point 2954 value.

The right shifter 3016 shifts the output of the standard size compressor and the saturator 3008 by the user-specified number of bits specified at the offset value 2944 to generate a standard size quotient. According to a preferred embodiment, the output of the right shifter 3016 is a standard type and has a decimal point specified by the output binary point 2954 value.

The multiplexer 3032 selects the appropriate input specified by the value of the start function 2934 and provides the selection to the symbol restorer 3034. If the original accumulator 202 value 217 is negative, the symbol restorer 3034 outputs the multiplexer 3032 A positive type value is converted to a negative type, such as a two's complement type.

The size converter 3036 converts the output of the symbol restorer 3034 to an appropriate size according to the value of the output command 2956 as shown in FIG. 29A. For a preferred embodiment, the output of the sign restorer 3034 has a decimal point specified by the output binary point 2954 value. For a preferred embodiment, for the first preset value of the output command, the size converter 3036 will discard the symbol restorer. 3034 outputs the upper half element. In addition, if the output of the symbol restorer 3034 is positive and exceeds the maximum value of the text size specified by the configuration 2902, or the output is negative and less than the minimum value of the text size, the saturator 3036 will output it Fill up to the maximum / minimum value of this text size. For the second and third preset values, the size converter 3036 passes the output of the symbol restorer 3034.

The multiplexer 3037 selects one of the data converter and saturator 3036 output and the accumulator 202 output 217 according to the output command 2956 to provide to the output register 3038. Further, for the first and second preset values of the output command 2956, the multiplexer 3037 selects the lower text of the output of the size converter and the saturator 3036 (the size is specified by the configuration 2902). For the third preset value, the multiplexer 3037 selects the text above the output of the size converter and the saturator 3036. For the fourth preset value, the multiplexer 3037 selects the text below the value 217 of the unprocessed accumulator 202; for the fifth preset value, the multiplexer 3037 selects the middle of the value 217 of the unprocessed accumulator 202 Text; and for the sixth preset value, the multiplexer 3037 selects the text above the value 217 of the unprocessed accumulator 202. As mentioned above, in a preferred embodiment, the activation function unit 212 adds a zero value upper azimuth element to the text above the value 217 of the unprocessed accumulator 202.

The thirty-first figure is an example of the operation of the activation function unit 212 of the thirty-first figure. As shown in the figure, the configuration 2902 of the neural processing unit 126 is set to a narrow configuration. In addition, the signed data 2912 and the signed weight 2914 are true. In addition, the value of data binary decimal point 2922 indicates that for the data random access memory 122 text, the binary There are 7 bits to the right of the decimal point position. An example value of the first data text received by the neural processing unit 126 is 0.1001110. In addition, the weight binary decimal point 2924 value indicates that for the weight random access memory 124 text, there are 3 bits to the right of the binary decimal point position, and an example value of the first weight text received by the neural processing unit 126 The line appears as 00001.010.

The 16-bit product of the first data and the weight text (this product will be added to the initial zero value of the accumulator 202) is presented as 000000.1100001100. Because the data decimal point 2912 is 7 and the weighted decimal point 2914 is 3, for the implied accumulator 202 binary decimal point, there will be 10 digits to the right. In the case of a narrow configuration, as shown in this embodiment, the accumulator 202 has a width of 28 bits. For example, after completing all arithmetic logic operations (such as all 1024 multiply-accumulate operations in the twentieth graph), the value 217 of the accumulator 202 will be 000000000000000001.1101010100.

The output 2954 decimal point value indicates that there are 7 digits to the right of the output binary decimal point. Therefore, after passing the output binary decimal aligner 3002 and the standard size compressor 3008, the value 217 of the accumulator 202 is scaled, rounded, and compressed to the standard type value, that is, 000000001.1101011. In this example, the output binary decimal point address represents 7 decimal places, and accumulator 202 binary decimal point position represents 10 decimal places. Therefore, the output binary aligner 3002 calculates the difference 3 and scales the accumulator 202 value 217 by 3 bits to the right. In the thirty-first figure, it is shown that the value 217 of the accumulator 202 will lose the 3 least significant bits (100 binary digits). In addition, In this example, the rounding control 2932 value means that random rounding is used, and in this example, the sampling random bit 3005 is assumed to be true. In this way, as mentioned above, the least significant bit will be rounded up. This is because of the rounding bit of the value 217 of the accumulator 202 (of the 3 bits that were shifted out due to the scaling operation of the value 217 of the accumulator 202) The most significant bit is one, and the sticky bit (the three least significant bits of the 3 bits that are shifted out due to the scaling operation of the accumulator 202 value 217) is zero.

In this example, the activation function 2934 indicates that an S-type function is used. In this way, the bit selector 3012 selects the bits of the standard type value so that the input of the S-type module 3024 has three integer bits and four decimal places, as described above, that is, the value 001.1101 shown. The output value of the S-type module 3024 will be put into the standard type, that is, the value 000000000.1101110 shown.

The output command 2956 of this example specifies the first preset value, that is, the text size indicated by the output configuration 2902, in this case, the narrow text (8 bits). In this way, the size converter 3036 converts the standard S-type output value into an 8-bit quantity, which has an implicit binary decimal point, that is, 7 bits to the right of the binary decimal point, and produces an output The value 01101110, as shown in the figure.

The thirty-second figure is a second example showing the operation of the activation function unit 212 of the thirty figure. The example in the thirty-second figure describes the operation of the activation function unit 212 when the activation function 2934 indicates that the value 217 of the accumulator 202 is passed in a standard size. As shown in the figure, this configuration 2902 is set as a narrow configuration of the neural processing unit 216.

In this example, the width of the accumulator 202 is 28 bits, and there are 10 bits to the right of the position of the decimal point of the accumulator 202 (this is because in one embodiment the data decimal point 2912 and the weighted binary bit The decimal point 2914 adds up to 10, or in another embodiment the accumulator binary decimal point 2923 is explicitly specified to have the value 10). For example, after performing all the arithmetic logic operations, the value 217 of the accumulator 202 shown in the thirty-second figure is 000001100000011011.1101111010.

In this example, the output of the decimal point 2954 value means that for the output, there are 4 digits to the right of the decimal point. Therefore, after passing the output binary aligner 3002 and the standard size compressor 3008, the value 217 of the accumulator 202 will saturate and compress to the standard type value 111111111111.1111 shown. This value is received by the multiplexer 3032. Pass the value 3028 as a standard size.

Two output commands 2956 are shown in this example. The first output command 2956 specifies the second preset value, that is, the text below the standard type size is output. Because the size indicated by the configuration 2902 is narrow text (8 bits), the standard size will be 16 bits, and the size converter 3036 will select the 8 bits below the standard size transfer value 3028 to generate the figure The 8-bit value is 11111111. The second output command 2956 specifies the third preset value, that is, the text above the standard size is output. In this way, the size converter 3036 selects 8 bits above the standard size transfer value 3028 to generate the 8-bit value 11111111 as shown in the figure.

The thirty-third figure is a third example showing the operation of the activation function unit 212 of the thirty-third figure. The example in Figure 33 reveals that when the activation function 2934 indicates that the entire unprocessed accumulator 202 is to be passed When the value is 217, the operation of the function unit 212 is started. As shown in the figure, this configuration 2902 is set to a wide configuration of the neural processing unit 126 (for example, a 16-bit input text).

In this example, the width of the accumulator 202 is 41 bits, and there are 8 bits to the right of the decimal point position of the accumulator 202 (this is because in one embodiment the data decimal point 2912 and the weighted binary bit The decimal point 2914 adds up to 8, or in another embodiment the accumulator binary decimal point 2923 is explicitly specified to have the value 8). For example, after performing all arithmetic logic operations, the value 217 of the accumulator 202 shown in Figure 33 is 001000000000000000001100000011011.11011110.

Three output commands 2956 are shown in this example. The first output command specifies the fourth preset value, that is, the text below the value of the unprocessed accumulator 202 is output; the second output command specifies the fifth preset value, which is the middle of the value of the unprocessed accumulator 202 Text; and the third output command specifies a sixth preset value, that is, the text above the value of the unprocessed accumulator 202 is output. Because the size indicated by configuration 2902 is wide text (16 bits), as shown in Figure 33, in response to the first output command 2956, the multiplexer 3037 will select the 16-bit value 0001101111011110; in response to the second output command 2956 The multiplexer 3037 will select the 16-bit value 0000000000011000; in response to the third output command 2956, the multiplexer 3037 will select the 16-bit value 0000000001000000.

As mentioned above, the neural network unit 121 can be executed on integer data instead of floating-point data. In this way, it helps to simplify each of the neural processing units 126, or at least the arithmetic logic unit 204 thereof. For example In other words, the arithmetic logic unit 204 does not need to include an adder for adding the exponents of the multipliers in the floating-point operation for the multiplier 242. Similarly, the arithmetic logic unit 204 does not need to include a shifter for aligning the decimal point of the addend in the floating-point operation for the adder 234. Those skilled in the art can understand that floating-point units are often very complex; therefore, the examples described in this article are only simplified for the arithmetic logic unit 204. The use of hardware fixed-point assistance allows users to specify related two The integer embodiment of the decimal point can also be used to simplify other parts. Compared with the floating-point embodiment, using an integer unit as the arithmetic logic unit 204 can generate a smaller (and faster) neural processing unit 126, which is advantageous for integrating a large array of neural processing units 126 into a neural network. Unit 121. The part of the activation function unit 212 can process the scaling and saturation operations of the value 217 of the accumulator 202 based on the user specified, the number of decimal places required by the accumulated number and the number of decimal places required by the output value, and the better is based on the use Designator. Any additional complexity and accompanying increase in size, and energy and / or time consumption in the fixed-point hardware assistance of the activation function unit 212 can be shared by sharing the activation function unit 212 among the arithmetic logic unit 204. This is because, as shown in the embodiment in FIG. 11, the embodiment adopting the sharing method can reduce the number of activation function units 1112.

The embodiments described herein can enjoy many advantages of using integer arithmetic units to reduce hardware complexity (compared to using floating-point arithmetic units), and can also be used for arithmetic operations on decimals, that is, with binary decimal digital. The advantage of floating-point arithmetic is that it can provide The individual values given to the data by the arithmetic operation fall within a very wide range of values (actually only limited by the size of the exponential range, so it will be a very large range). That is, each floating-point number has its potentially unique exponential value. However, the embodiments described herein understand and take advantage of the fact that in some applications the input data is highly parallel and falls within a relatively narrow range so that all parallel data have the same "index" characteristics. As such, these embodiments allow the user to assign binary decimal point positions to all input values and / or accumulated values at once. Similarly, by understanding and utilizing the characteristics of parallel outputs with similar ranges, these embodiments allow the user to assign the binary decimal point position to all output values at once. Artificial neural networks are one example of such applications, but embodiments of the present invention can also be applied to perform calculations for other applications. By assigning the decimal point position to multiple inputs at one time rather than to individual input numbers, the embodiment of the present invention can use the memory space more efficiently (if less memory is required) than using floating-point arithmetic Volume) and / or increase accuracy with a similar amount of memory, because the bits of the exponent used for floating-point arithmetic can be used to increase numerical accuracy.

In addition, the embodiments of the present invention understand that accuracy may be lost when performing accumulation on a large series of integer operations (such as overflow or loss of less significant decimal places), so a solution is provided, mainly by using a large enough Accumulator to avoid loss of accuracy.

Direct execution of micro-operations in neural network units

The thirty-fourth figure is a block diagram showing some details of the processor 100 and the neural network unit 121 in the first figure. Neural network The unit 121 includes a pipeline stage 3401 of the neural processing unit 126. Each pipeline stage 3401 is distinguished by a stage register and includes combinational logic to achieve the operations of the neural processing unit 126 in this article, such as a Bollinger logic gate, a multiplexer, an adder, a multiplier, a comparator, and so on. The pipeline stage 3401 receives a micro-operation 3418 from the multiplexer 3402. The micro-operation 3418 flows down to the pipeline stage 3401 and controls its combination logic. Micro operation 3418 is a set of bits. In a preferred embodiment, the micro-operation 3418 includes data random access memory 122, memory address 123, weight random access memory 124, memory address 125, and program memory 129. Bit 131, multiplex register 208/705 control signal 213/713, and many fields of control register 217 (such as the control registers of Figures 29A to 29C). In one embodiment, the micro-operation 3418 includes approximately 120 bits. The multiplexer 3402 receives micro-operations from three different sources and selects one of them as the micro-operation 3418 provided to the pipeline stage 3401.

One micro-computing source of the multiplexer 3402 is the sequencer 128 of the first figure. The sequencer 128 decodes the neural network unit instructions received by the program memory 129 and generates a micro-operation 3416 to provide the first input to the multiplexer 3402.

The second micro-computing source of the multiplexer 3402 is a decoder 3404 that receives micro-instructions 105 from the reservation station 108 in the first figure and receives operands from the general-purpose register 116 and the media register 118. In a preferred embodiment, as mentioned above, the microinstruction 105 is generated by the instruction translator 104 in response to the translation of the MTNN instruction 1400 and the MFNN instruction 1500. Microinstruction 105 may include an immediate field to specify a specific function (specified by an MTNN instruction 1400 or an MFNN instruction 1500), such as program memory Start and stop execution of the program in the body 129, perform a micro-operation directly from the media register 118, or read / write memory of one of the neural network units as described above. The decoder 3404 decodes the micro instruction 105 and generates a micro operation 3412 to provide a second input to the multiplexer. For a preferred embodiment, for some functions 1432/1532 of the MTNN instruction 1400 / MFNN instruction 1500, the decoder 3404 does not need to generate a micro-operation 3412 and send it down to the pipeline 3401, for example, write control temporary storage Device 127, start program in program memory 129, pause program in program memory 129, wait for program in program memory 129 to finish executing, read from state register 127, and reset neural network unit 121 .

The third micro-computing source of the multiplexer 3402 is the media register 118 itself. For a preferred embodiment, as described above corresponding to the fourteenth figure, the MTNN instruction 1400 may specify a function to instruct the neural network unit 121 to directly execute a first step provided by the media register 118 to the multiplexer 3402 Three-input micro operation 3414. Directly performing the micro-operations 3414 provided by the architecture media register 118 is beneficial for testing the neural network unit 121, such as a built-in self-test (BIST), or a debugging action.

For a preferred embodiment, the decoder 3404 generates a mode indicator 3422 to control the selection of the multiplexer 3402. When MTNN instruction 1400 specifies a function to start executing a program from program memory 129, the decoder 3404 will generate a mode indicator 3422 value to cause the multiplexer 3402 to select the micro-operation 3416 from the sequencer 128 until an error occurs or until decoding The processor 3404 encounters an MTNN instruction 1400 specifying a function to stop executing a program from the program memory 129. When MTNN instruction 1400 specifies a function to instruct neural network unit 121 to directly execute The register 118 provides a micro operation 3414. The decoder 3404 will generate a mode indicator 3422 value to enable the multiplexer 3402 to select the micro operation 3414 from the specified media register 118. Otherwise, the decoder 3404 will generate a mode indicator 3422 value to cause the multiplexer 3402 to select the micro-operation 3412 from the decoder 3404.

Variable rate neural network unit

In many cases, after executing the program, the neural network unit 121 enters an idle state and waits for the processor 100 to process something that needs to be processed before executing the next program. For example, if it is in a situation similar to that described in Figures 3 to 6A, the neural network unit 121 accumulates a multiplying activation function program (also known as a feed forward neural network layer program (feed forward neural network layer program)) is performed two or more times in a row. Compared with the time taken by the neural network unit 121 to execute the program, the processor 100 obviously takes a longer time to write the 512KB weight value into the weight random access memory 124 for the next neural network unit program. . In other words, the neural network unit 121 executes the program in a short time, and then enters the standby state until the processor 100 writes the next weight value into the weight random access memory 124 for the next program execution. In this case, please refer to Figure 36A, which will be described in detail later. In this case, the neural network unit 121 can run at a lower time frequency to extend the execution time of the program, so that the energy consumption required to execute the program is spread over a longer time range, so that the neural network unit 121 and even For the entire processor 100, Witter is at a lower temperature. This situation is called a relaxation mode, and you can refer to Figure 36B, which will be described in detail later.

The thirty-fifth figure is a block diagram showing a variable Processor 100 with rate neural network unit 121. The processor 100 is similar to the processor 100 in the first figure, and the components with the same reference numerals in the figure are similar. The processor 100 in the thirty-fifth figure has time-frequency generating logic 3502, which is a functional unit coupled to the processor 100. These functional units are the instruction fetch unit 101, the instruction cache 102, the instruction translator 104, and the rename unit 106. Reservation station 108, neural network unit 121, other execution units 112, memory subsystem 114, general purpose register 116 and media register 118. The time-frequency generating logic 3502 includes a time-frequency generator, such as a phase-locked loop (PLL), to generate a time-frequency signal having a main time-frequency or time-frequency. For example, the main frequency can be 1GHz, 1.5GHz, 2GHz, and so on. The time frequency represents the number of cycles per second, such as the number of oscillations of the time frequency signal between high and low states. Preferably, the time-frequency signal has a duty cycle, that is, one half of the period is high and the other half is low. In addition, the time-frequency signal may also have an unbalanced period, which is time-frequency A signal is in a high state longer than it is in a low state, and vice versa. Preferably, the phase-locked loop is used to generate a main time-frequency signal with multiple time-frequency. Preferably, the processor 100 includes a power management module that automatically adjusts the main time frequency based on a variety of factors, including dynamic detection of operating temperature, utilization of the processor 100, and system software (such as an operating system). , Basic Input Output System (BIOS) command to indicate required performance and / or energy saving indicators. In one embodiment, the power management module includes microcode of the processor 100.

The time-frequency generation logic 3502 includes a time-frequency dispersion network, or a time-frequency tree. The time-frequency tree will distribute the main time-frequency signals to the functional units of the processor 100. As shown in Figure 35, this distribution The action is to transmit the time-frequency signal 3506-1 to the instruction fetch unit 101, the time-frequency signal 3506-2 to the instruction cache 102, the time-frequency signal 3506-10 to the instruction translator 104, and the time-frequency signal 3506- 9 to the renaming unit 106, the time-frequency signal 3506-8 to the reservation station 108, the time-frequency signal 3506-7 to the neural network unit 121, and the time-frequency signal 3506-4 to the other execution units 112, The time-frequency signal 3506-3 is transmitted to the memory subsystem 114, the time-frequency signal 3506-5 is transmitted to the general purpose register 116, and the time-frequency signal 3506-6 is transmitted to the media register 118. These signals are collectively called Is time-frequency signal 3506. The time-frequency tree has nodes or lines to transmit the main time-frequency signal 3506 to its corresponding functional unit. In addition, preferably, the time-frequency generation logic 3502 may include a time-frequency buffer, when a cleaner time-frequency signal is needed and / or a voltage level of the main time-frequency signal needs to be raised, especially for a distant node, The time-frequency buffer can regenerate the main time-frequency signal. In addition, each functional unit has its own child time-frequency tree, and when necessary, reproduces and / or boosts the voltage level of the corresponding main time-frequency signal 3506 received.

The neural network unit 121 includes time-frequency reduction logic 3504. The time-frequency reduction logic 3504 receives a relaxation index 3512 and a main time-frequency signal 3506-7 to generate a second time-frequency signal. The second time-frequency signal has a time-frequency. At this time, if the frequency is not the same as the main time frequency, it is in a relaxation mode. A value is reduced from the main time frequency to reduce heat generation. This value is programmed to the relaxation index 3512. The time-frequency reduction logic 3504 is similar to the time-frequency generation logic 3502. It has a time-frequency distribution network, or a time-frequency tree, to distribute the second time-frequency signal to the various functional blocks of the neural network unit 121. This distribution action is to divide the time Frequency signal 3508-1 to God Through the processing unit array 126, the time-frequency signal 3508-2 is transmitted to the sequencer 128 to transmit the time-frequency signal 3508-3 to the interface logic 3514. These signals are collectively referred to as the second time-frequency signal 3508. Preferably, the neural processing units 126 include a plurality of pipeline stages 3401. As shown in FIG. 34, the pipeline stage 3401 includes pipeline stage registers to receive the second time-frequency signal 3508 from the time-frequency reduction logic 3504. -1.

The neural network unit 121 also has interface logic 3514 to receive the main time-frequency signal 3506-7 and the second time-frequency signal 3508-3. The interface logic 3514 is coupled between the lower part of the front end of the processor 100 (such as the reservation station 108, the media register 118 and the general purpose register 116) and the various functional blocks of the neural network unit 121. These functional blocks are reduced frequently. Logic 3504, data random access memory 122, weight random access memory 124, program memory 129, and sequencer 128. The interface logic 3514 includes a data random access memory buffer 3522, a weighted random access memory buffer 3524, a decoder 3404 of the thirty-fourth figure, and a relaxation index 3512. The relaxation indicator 3512 is loaded with a value that specifies how slowly the neural processing unit array 126 will execute the neural network unit program instructions. Preferably, the relaxation index 3512 specifies a division value N, and the time-frequency reduction logic 3504 divides the main time-frequency signal 3506-7 by the division value to generate a second time-frequency signal 3508. Thus, at the time of the second time-frequency signal, The frequency will be 1 / N. Preferably, the value of N can be programmed into any one of a plurality of different preset values, which can cause the time-frequency reduction logic 3504 to correspondingly generate a plurality of second time-frequency signals 3508 with different time frequencies. These The time frequency is smaller than the main time frequency.

In one embodiment, the time-frequency reduction logic 3504 includes a The time-frequency divider circuit is used to divide the main time-frequency signal 3506-7 by the value of the relaxation index 3512. In one embodiment, the time-frequency reduction logic 3504 includes a time-frequency gate (such as an AND gate). The time-frequency gate can gate the main time-frequency signal 3506-7 through a start signal. The start signal is every N of the main time-frequency signal. True values are generated only once in each cycle. Taking a circuit including a counter to generate a start signal as an example, this counter can count up to N. When the accompanying logic circuit detects that the output of the counter matches N, the logic circuit will generate a true value pulse on the second time-frequency signal 3508 and reset the counter. Preferably, the value of the mitigation index 3512 can be stylized by a framework instruction, such as the MTNN instruction 1400 of the fourteenth figure. Preferably, before the architecture program instructs the neural network unit 121 to start executing the neural network unit program, the architecture program running on the processor 100 programs the relaxation value to the relaxation index 3512, which corresponds to the thirtieth in the following. There will be a more detailed explanation at the seventh figure.

The weighted random access memory buffer 3524 is coupled between the weighted random access memory 124 and the media register 118 as a buffer for data transmission therebetween. Preferably, the weighted random access memory buffer 3524 is one or more embodiments similar to the buffer 1704 of FIG. Preferably, the part of the weighted random access memory buffer 3524 receiving data from the media register 118 uses the main time-frequency signal 3506-7 with the main time-frequency as the time-frequency, and the weight random-access memory buffer 3524 from The part of the data received by the weighted random access memory 124 uses the second time-frequency signal 3508-3 with the second time frequency as the time frequency. The second time frequency can be adjusted from the main time frequency according to the value programmed in the relaxation index 3512. Decrease or no, that is, according to whether the neural network unit 121 is executed in the relaxation or normal mode. Downgrade or No. In an embodiment, the weight random access memory 124 is a port. As described in the seventeenth figure, the weight random access memory 124 can be buffered by the media register 118 through the weight random access memory 3524. And it is buffered 1104 by the neural processing unit 126 or the list of the eleventh figure, and is accessed in an arbitrated fashion. In another embodiment, the weight random access memory 124 is a dual port. As described in the sixteenth figure above, each port can be buffered by the media register 118 through the weight random access memory buffer 3524 and by the neural processing unit 126. OR column buffer 1104 is accessed in parallel.

Similar to the weighted random access memory buffer 3524, the data random access memory buffer 3522 is coupled between the data random access memory 122 and the media register 118 as a buffer for data transmission therebetween. Preferably, the data random access memory buffer 3522 is one or more embodiments similar to the buffer 1704 of FIG. Preferably, the part of the data random access memory buffer 3522 that receives data from the media register 118 uses the main time-frequency signal 3506-7 with the main time frequency as the time frequency, and the data random access memory buffer 3522 from The part of data received by the data random access memory 122 uses the second time-frequency signal 3508-3 with the second time frequency as the time frequency. The second time frequency can be adjusted from the main time frequency according to the value programmed in the relaxation index 3512. Decrease or not, that is, decrease or not according to the execution of the neural network unit 121 in the relaxation or normal mode. In one embodiment, the data random access memory 122 is a port. As described in the seventeenth figure, the data random access memory 122 can be buffered by the media register 118 through the data random access memory 3522. And buffered by the nerve processing unit 126 or the eleventh figure, and stored in arbitration take. In another embodiment, the data random access memory 122 is a dual port. As described in the sixteenth figure, each port can be buffered by the media register 118 through the data random access memory buffer 3522 and by the neural processing unit 126. OR column buffer 1104 is accessed in parallel.

Preferably, regardless of whether the data random access memory 122 and / or the weight random access memory 124 are port or dual port, the interface logic 3514 will include a data random access memory buffer 3522 and a weight random access memory buffer. 3524 synchronizes the main time-frequency domain with the second time-frequency domain. Preferably, the data random access memory 122, the weight random access memory 124 and the program memory 129 all have a static random access memory (SRAM), which contains individual read enable signals and write enable Enable signal and memory select enable signal.

As described above, the neural network unit 121 is an execution unit of the processor 100. The execution unit is a micro instruction translated from the execution of the architectural instruction in the processor or a functional unit executed by the execution of the architectural instruction itself. The execution unit receives operands from a general purpose register of the processor, such as from the general purpose register 116 and the media register 118. After the execution unit executes the micro instruction or the architecture instruction, a result is generated, and the result is written into the general purpose register. The MTNN instruction 1400 and MFNN instruction 1500 described in the fourteenth and fifteenth figures are examples of the architecture instruction 103. Micro instructions are used to implement architectural instructions. More precisely, the collective execution of one or more micro-instructions translated by the architectural instruction by the execution unit is to perform the operation specified by the architectural instruction on the input specified by the architectural instruction to produce the result defined by the architectural instruction.

Figure 36A is a timing diagram showing the processor 100 having an operation example in which the neural network unit 121 operates in one of the general modes, and this general mode operates at the main time frequency. In the timing diagram, the progress of time is from left to right. The processor 100 executes a framework program at a main time frequency. More precisely, the front end of the processor 100 (for example, the instruction fetch unit 101, the instruction cache 102, the instruction translator 104, the renaming unit 106, and the reservation station 108) is fetched at the main time frequency, decodes and issues architectural instructions to the nerve Network unit 121 and other execution units 112.

Initially, the framework program executes a framework instruction (such as MTNN instruction 1400). The processor front-end 100 issues this framework instruction to the neural network unit 121 to instruct the neural network unit 121 to start executing the neural network in its program memory 129. Unit program. Previously, the architecture program would execute a framework instruction to write a value specifying the main time frequency into the relaxation indicator 3512, even if the neural network unit was in normal mode. More precisely, the value programmed to the relaxation indicator 3512 causes the time-frequency reduction logic 3504 to generate the second time-frequency signal 3508 at the main time-frequency of the main time-frequency signal 3506. Preferably, in this example, the time-frequency buffer of the time-frequency reduction logic 3504 simply raises the voltage level of the main time-frequency signal 3506. In addition, before, the architecture program executes the architecture instruction to write data to the random access memory 122, weights the random access memory 124, and writes the neural network unit program to the program memory 129. In response to the neural network unit program MTNN instruction 1400, the neural network unit 121 will start to execute the neural network unit program at the main time frequency, because the relaxation index 3512 is programmed with the main time frequency value. After the execution of the neural network unit 121, the architecture program will continue to execute the architecture instructions at the main frequency, including the main The MTNN instruction 1400 is used to write and / or read data random access memory 122 and weight random access memory 124 to complete the next instance of the neural network unit program, or invocation. Or run to prepare.

In the example of Figure 36A, compared with the time it takes for the architecture program to complete the writing / reading of the data random access memory 122 and the weight random access memory 124, the neural network unit 121 can Significantly less time (such as a quarter of the time) to complete the execution of the neural network unit program. For example, in the case of operating at the main time frequency, the neural network unit 121 takes about 1,000 time-frequency cycles to execute the neural network unit program, but the architecture program takes about 4000 time-frequency cycles. In this way, the neural network unit 121 will be in a standby state for the rest of the time. In this example, this is a relatively long time, such as about 3000 major time frequency cycles. As shown in the example of Figure 36A, depending on the size and configuration of the neural network, the aforementioned mode will be executed again, and may continue to be executed many times. Because the neural network unit 121 is a relatively large and transistor-intensive functional unit in the processor 100, the operation of the neural network unit 121 will generate a large amount of thermal energy, especially when it operates at the main frequency.

Figure 36B is a timing diagram showing that the processor 100 has an operation example in which the neural network unit 121 operates in one of the relaxation modes. The frequency of the relaxation mode is lower than the main time frequency. The timing diagram of Figure 36B is similar to Figure 36A. In Figure 36A, the processor 100 executes an architecture program at the main time frequency. This example assumes that the architecture program in Figure 36B is the same as the neural network unit program. The structure program and neural network unit program of Figure 36A. However, before starting the neural network unit program, the framework program will execute a MTNN instruction 1400 to program the relaxation indicator 3512 with a value that will cause the time-frequency reduction logic 3504 to generate a second at a second time frequency that is less than the main time frequency. Time-frequency signal 3508. In other words, the structure program will put the neural network unit 121 in the relaxation mode of the thirty-sixth figure B, instead of the general mode of the thirty-sixth figure A. In this way, the neural processing unit 126 executes the neural network unit program at the second time frequency. In the relaxation mode, the second time frequency is less than the main time frequency. In this example, it is assumed that the relaxation index 3512 is stylized with a value that specifies the second time frequency as a quarter of the main time frequency. In this way, the time taken by the neural network unit 121 to execute the neural network unit program in the mitigation mode will be four times the time it spends in the normal mode, as shown in Figures 36A and 36B. Comparing these two figures, it can be seen that the length of time that the neural network unit 121 is in the standby state is significantly shortened. In this way, the duration of the energy consumed by the neural network unit 121 in executing the program of the neural network unit 121 in FIG. 36B is approximately four times that of the execution of the program in the normal mode by the neural network unit 121 in FIG. 36A. Therefore, the thermal energy generated by the neural network unit 121 executed by the neural network unit 121 in the thirty-sixth figure in a unit time will be about a quarter of that in the thirty-sixth figure A, and has the advantages described herein.

The thirty-seventh figure is a flowchart showing the operation of the processor 100 in the thirty-fifth figure. The operation described in this flowchart is similar to the operation corresponding to the thirty-fifth, thirty-sixth A, and thirty-sixth B drawings. This process starts at step 3702.

In step 3702, the processor 100 executes the MTNN instruction. The 1400 is instructed to write the weight into the weight random access memory 124 and write the data into the data random access memory 122. The flow then proceeds to step 3704.

In step 3704, the processor 100 executes the MTNN instruction 1400 to program the relaxation index 3512 with a value that specifies a time frequency lower than the main time frequency, even if the neural network unit 121 is in the relaxation mode. The flow then proceeds to step 3706.

In step 3706, the processor 100 executes the MTNN instruction 1400 to instruct the neural network unit 121 to start executing the neural network unit program, which is similar to the manner shown in FIG. 36B. The flow then proceeds to step 3708.

In step 3708, the neural network unit 121 starts executing the neural network unit program. At the same time, the processor 100 executes the MTNN instruction 1400 and writes new weights into the weight random access memory 124 (may also write new data into the data random access memory 122), and / or executes the MFNN instruction 1500 The results are read from the data random access memory 122 (the results may also be read from the weight random access memory 124). The flow then proceeds to step 3712.

In step 3712, the processor 100 executes the MFNN instruction 1500 (for example, the read status register 127) to detect that the neural network unit 121 has finished program execution. Assuming that the framework program selects a good value of the mitigating index 3512, the time taken by the neural network unit 121 to execute the neural network unit program will be the same as that of the processor 100 executing part of the framework program to access the weight random access memory 124 and / Or the time taken by the data random access memory 122 is shown in FIG. 36B. The next process Proceed to step 3714.

In step 3714, the processor 100 executes the MTNN instruction 1400 and uses a value to program the relaxation index 3512. This value specifies the main time frequency, even if the neural network unit 121 is in the general mode. Then proceed to step 3716.

In step 3716, the processor 100 executes the MTNN instruction 1400 to instruct the neural network unit 121 to start executing the neural network unit program, which is similar to the manner shown in FIG. 36A. The flow then proceeds to step 3718.

In step 3718, the neural network unit 121 starts executing the neural network unit program in a general mode. This process ends at step 3718.

As mentioned above, compared with executing the neural network unit program in the general mode (that is, the main frequency of the processor is executed), the execution in the relaxation mode can disperse the execution time and avoid high temperature. Further, when the neural network unit executes a program in a relaxation mode, the neural network unit generates thermal energy at a lower time frequency, and this thermal energy can smoothly pass through the neural network unit (such as a semiconductor device, a metal layer and a lower layer). The substrate) is discharged from the surrounding packages and cooling mechanisms (such as heat sinks, fans). Therefore, the devices (such as transistors, capacitors, and wires) in the neural network unit are more likely to operate at lower temperatures. Overall, operating in mitigation mode also helps reduce device temperature in other parts of the processor die. The lower operating temperature, especially for the junction temperature of these devices, can reduce the generation of leakage current. In addition, because the amount of current flowing in per unit time is reduced, inductor noise and IR voltage drop Noise will also be reduced. In addition, the temperature decrease has a positive impact on the negative bias temperature instability (NBTI) and positive bias instability (PBSI) of the metal-oxide-semiconductor field-effect transistor (MOSFET) in the processor, which can improve reliability and / Or device and processor life. The decrease in temperature can reduce the effects of Joule heat and electromigration in the metal layer of the processor.

Communication mechanism between architecture program and non-architecture program of neural network unit shared resources

As mentioned above, in the examples of the twenty-fourth to twenty-eight and thirty-five to thirty-seven figures, the resources of the data random access memory 122 and the weight random access memory 124 are shared. The neural processing unit 126 and the front end of the processor 100 share a data random access memory 122 and a weight random access memory 124. More precisely, the neural processing unit 126 and the front end of the processor 100, such as the media register 118, will both read and write the data random access memory 122 and the weight random access memory 124. In other words, the architecture program running on the processor 100 and the neural network unit program running on the neural network unit 121 share data random access memory 122 and weight random access memory 124, and in some cases As mentioned before, the flow between the framework program and the neural network unit program needs to be controlled. The resources of the program memory 129 are also shared to a certain extent, because the framework program will write them and the sequencer 128 will read them. The embodiment described herein provides a high-performance solution to control the flow of accessing shared resources between the architecture program and the neural network unit program.

In the embodiment described herein, the neural network unit program is also referred to as non-architecture program, and the neural network unit instruction is also referred to as non-architecture program. Instructions, and the neural network unit instruction set (also referred to as the neural processing unit instruction set as described above) is also referred to as the non-architecture instruction set. Non-architecture instruction sets are different from architectural instruction sets. In the embodiment where the processor 100 includes the instruction translator 104 to translate the architectural instructions into micro instructions, the non-architecture instruction set is also different from the micro instruction set.

The thirty-eighth figure is a block diagram showing the sequencer 128 of the neural network unit 121 in detail. The sequencer 128 provides a memory address to the program memory 129 to select non-architecture instructions provided to the sequencer 128, as described above. As shown in Figure 38, the memory address is loaded into a program counter 3802 of the sequencer 128. The sequencer 128 generally increments in sequence from the address of the program memory 129, unless the sequencer 128 encounters a non-architecture instruction, such as a loop or branch instruction, in which case the sequencer 128 will The program counter 3802 is updated to the target address of the control instruction, that is, the address of the non-framework instruction located at the target of the control instruction. Therefore, the address 131 loaded in the program counter 3802 specifies the address in the program memory 129 of the non-framework instruction of the non-framework program currently being fetched for execution by the neural processing unit 126. The value of the program counter 3802 can be obtained by the framework program through the neural network unit program counter field 3912 of the state register 127, as described in the following figure 39. In this way, based on the progress of the non-architecture program, the architecture program can determine the position where the data random storage memory 122 and / or the weight random access memory 124 read / write data.

The sequencer 128 also includes a loop counter 3804. This loop counter 3804 will operate with a non-framework loop instruction, such as the loop to address 1 instruction at address 10 and the twenty-eighth in the 26th A diagram. Figure The loop from the middle address 11 to the 1 instruction. In the examples of the twenty-sixth A and the twenty-eighth figures, the loop counter 3804 contains the value specified by the non-architecture initialization instruction at address 0, for example, the value 400 is loaded. Each time the sequencer 128 encounters a loop instruction and jumps to the target instruction (such as the multiply accumulate instruction at address 1 in Figure 26A or the maxwacc instruction at address 1 in Figure 28), The sequencer 128 decrements the loop counter 3804. Once the loop counter 3804 is reduced to zero, the sequencer 128 moves to the next non-architecture instruction that is ordered. In another embodiment, when a loop instruction is first encountered, a loop count value specified in the loop instruction is loaded into the loop counter to eliminate the need to initialize the loop counter 3804 with a non-architecture initialization instruction. Therefore, the value of the loop counter 3804 indicates the number of times that the loop group of the non-schema program has to be executed. The value of the loop counter 3804 can be obtained by the framework program through the loop count field 3914 of the state register 127, as shown in the subsequent thirty-ninth figure. In this way, based on the progress of the non-architecture program, the architecture program can determine the position where the data random storage memory 122 and / or the weight random access memory 124 read / write data. In one embodiment, the sequencer includes three additional loop counters to match the nested loops in the non-architecture program. The values of the three loop counters can also be read through the state register 127. There is a bit in the loop instruction to indicate which of the four loop counters is provided for the current loop instruction.

The sequencer 128 also includes a number of iterations counter 3806. The iteration counter 3806 is matched with non-architectural instructions, such as the multiply-accumulate instruction of address 2 in the fourth, ninth, twenty, and twenty-sixth A and the maxwacc instruction of address 2 in the twenty-eighth illustration. Since then Will be called an "execute" instruction. In the foregoing example, each execution instruction specifies an execution count 511, 511, 1023, 2 and 3. When the sequencer 128 encounters an execution instruction specifying a non-zero iteration count, the sequencer 128 loads the iteration number counter 3806 with the specified value. In addition, the sequencer 128 generates an appropriate micro-operation 3418 to control the logic execution in the neural processing unit 126 pipeline stage 3401 in the thirty-fourth figure and decrements the iteration counter 3806. If the number of iterations counter 3806 is greater than zero, the sequencer 128 will again generate an appropriate micro operation 3418 to control the logic in the neural processing unit 126 and decrement the number of iterations counter 3806. The sequencer 128 continues to operate in this manner until the value of the number of iterations counter 3806 returns to zero. Therefore, the value of the iteration number counter 3806 is the number of operations to be executed specified in the non-architecture execution instruction (such as multiply accumulating the accumulated value and a data / weight text, take the maximum value, add the total operation, etc.). The value of the iteration count counter 3806 can be obtained by the framework program through the iteration count field 3916 of the state register 127, as described in the following figure 39. In this way, based on the progress of the non-architecture program, the architecture program can determine the position where the data random storage memory 122 and / or the weight random access memory 124 read / write data.

The thirty-ninth figure is a block diagram showing the fields of the control and status register 127 of the neural network unit 121. These fields include the address 2602 of the weighted random access memory row recently written by the neural processing unit 126 executing the non-structural program, and the position of the weighted random access memory row recently read by the neural processing unit 126 executing the non-structural program. Address 2604, the neural processing unit 126 executes a non-architecture program to write the recently written data to the random access memory row address 2606, and the neural processing unit 126 The address 2608 of the random access memory row recently read by the non-architecture program is executed, as shown in the aforementioned twenty-sixth B diagram. In addition, these fields also include a neural network unit program counter 3912 field, a loop counter 3914 field, and an iteration number counter 3916 field. As described above, the architecture program can read the data in the state register 127 to the media register 118 and / or the general register 116, for example, read the counter 3912 including the neural network unit program through the MFNN instruction 1500. The values of the counter 3914 and the iteration number counter 3916. The value of the program counter field 3912 reflects the value of the program counter 3802 in the thirty-eighth figure. The value of the lap counter field 3914 reflects the value of the lap counter 3804. The value of the iteration number counter field 3916 reflects the value of the iteration number counter 3806. In one embodiment, the sequencer 128 updates the program counter field 3912, the loop counter field 3914, and the iteration number counter field each time the program counter 3802, the loop counter 3804, or the iteration number counter 3806 is adjusted. The value of bit 3916, so when the framework program reads the value of these fields will be the current value. In another embodiment, when the neural network unit 121 executes an architectural instruction to read the state register 127, the neural network unit 121 only obtains the values of the program counter 3802, the loop counter 3804 and the iteration number counter 3806 and It provides back-to-architecture instructions (eg, to media register 118 or general purpose register 116).

From this, it can be found that the value of the field in the state register 127 of the thirty-ninth figure can be understood as the information of the execution progress of the non-architecture instruction executed by the neural network unit. Some specific aspects of the execution progress of non-structural programs, such as the value of the program counter 3802, The value of the circle counter 3804, the number of iteration counters 3806, the weight of the most recent read / write random access memory 124, the field 125 of the address 2602/2604, and the most recently read / written data random access memory 122 The field 2606/2608 at address 123 has been described in the previous chapter. The architecture program running on the processor 100 can read the progress value of the non-architecture program in the thirty-ninth figure from the state register 127 and use this information to make decisions, for example, through architecture instructions such as comparison and branch instructions. For example, the architecture program may decide to read / write data / weight rows for the data random access memory 122 and / or the weight random access memory 124 to control the data random access memory 122 or the weight The inflow and outflow of the data in the random access memory 124 is especially for the large data sets and / or the overlapping execution of different non-architecture instructions. These examples of making decisions using the framework program can refer to the description in the previous and subsequent chapters of this article.

For example, as described in Figure 26A above, the structural program sets the non-structural program to write the result of the convolution operation back to the row above the convolution kernel 2402 in the data random access memory 122 (as above row 8) When the neural network unit 121 writes the result using the address of the recently written data random access memory 122 row 2606, the architecture program reads the result from the data random access memory 122.

In another example, as described in Figure 26B above, the framework program uses the information from the status register 127 field of Figure 38 to confirm that the non-schema program will use the data array 2404 of Figure 24 Divide into 5 512 x 1600 data blocks to perform the progress of the convolution operation. The framework program writes the first 512 x 1600 data block of this 2560 x 1600 data array to the weight random access memory 124 and starts the non-schema program, which returns The cycle count is 1600 and the output of the weighted random access memory 124 initialization is 0. When the neural network unit 121 executes the non-architecture program, the architecture program reads the status register 127 to confirm the weight of the recent write row 2602 of the random access memory 124, so that the architecture program can read the non-architecture program. The effective convolution operation result, and overwrite the effective convolution operation result with the next 512 x 1600 data block after reading, so, complete the non-architecture program for the first 512 x 1600 data block in the neural network unit 121 After execution, the processor 100 can immediately update the non-architecture program when necessary and start the non-architecture program again to execute the next 512 x 1600 data block.

In another example, suppose the architecture program causes the neural network unit 121 to execute a series of typical neural network multiply-accumulate activation functions, wherein the weights are stored in the weight random access memory 124 and the results are written back to the data random Access memory 122. In this case, after the architecture program reads a row of the weight random access memory 124, it will not read it again. In this way, after the current weight has been read / used by the non-architecture program, the architecture program can be used to start overwriting the new weight with the weight on the random access memory 124 to provide the next example of the non-architecture program ( Such as the next neural network layer). In this case, the architecture program reads the state register 127 to obtain the address of the most recent read row 2604 of the weighted random access memory to determine the write of the new weighted reassembly in the weighted random access memory 124. position.

In another example, suppose the architecture program knows that the non-architecture program includes an execution instruction with a large iteration count, such as the non-architecture multiply accumulate instruction at address 2 in the twentieth figure. In this situation, The framework program needs to know the number of iterations to 3916 before it can know how many time-frequency cycles are needed to complete this non-structural instruction to determine which of the two or more actions the framework program will take next. For example, if it takes a long time to complete execution, the framework program will give up control to another framework program, such as the operating system. Similarly, it is assumed that the architecture program knows that the non-architecture program includes a loop group with a considerable loop count, such as the non-architecture program of FIG. In this case, the architecture program needs to know the loop count of 3914 to know how many time-frequency cycles are needed to complete this non-architecture instruction to determine which of two or more actions to take next.

In another example, suppose that the structural program causes the neural network unit 121 to perform a common source operation similar to that described in Figures 27 and 28, where the data to be common source is stored in the weight random access memory 124 The result is written back to the weight random access memory 124. However, unlike the examples in figures 27 and 28, it is assumed that the results of this example will be written back to the top 400 rows of the weighted random access memory 124, such as rows 1600 to 1999. In this case, after the non-architecture program finishes reading the four rows of weighted random access memory 124 data it wants to share in common, the non-architecture program will not read it again. Therefore, once the current four rows of data have been read / used by the non-schema program, you can use the framework program to start the new data (such as the weight of the next example of the non-schema program, for example, perform a typical The multiplication and accumulation start function calculation is a non-structural program) overwriting the data of the weight random access memory 124. In this case, the architecture program reads the status register 127 to obtain the weight of the address of the most recently read row 2604 of the random access memory to determine the new weight rewrite The weighted random access memory 124 is located.

Time recurrent neural network acceleration

Traditional feed-forward neural networks do not have memory to store the network's previous inputs. Feedforward neural networks are often used to perform tasks in which multiple inputs to the network are independent of each other over time, and so are multiple outputs. In contrast, time-recurrent neural networks often help perform tasks where the order of input to the neural network over time is important in the task. (The sequence here is often referred to as a time step.) Therefore, a time-recurrent neural network includes a conceptual memory or internal state to load information generated by the network in response to calculations performed by previous inputs in the sequence The output of the time recurrent neural network is related to this internal state and the input of the next time step. The following tasks, such as speech recognition, language models, text generation, language translation, image description generation, and some forms of handwriting recognition, are good examples of how time-recurrent neural networks can perform.

Three examples of conventional time recurrent neural networks are Elman time recurrent neural network, Jordan time recurrent neural network and long short-term memory (LSTM) neural network. The Elman time recurrent neural network contains content nodes to memorize the hidden layer state of the time recurrent neural network in the current time step. This state will be used as the input to the hidden layer in the next time step. Jordan time recurrent neural network is similar to Elman time recurrent neural network, except that the content nodes in it will remember the state of the output layer of the time recurrent neural network instead of the state of the hidden layer. The long-term and short-term memory neural network includes a long-term and short-term memory layer composed of long-term and short-term memory cells. Each long-term and short-term memory cell has one current state and one current time step. Output, and a new or new time step and a new state and a new output. The long-term and short-term memory cells include an input gate, an output gate, and a forget gate. The forget gate can cause a neuron to lose its memorized state. These three types of temporal recurrent neural networks are described in more detail in subsequent chapters.

As described in this article, for a time-recurrent neural network, such as Elman or Jordan time-recurrent neural network, each time the neural network unit executes, it uses a time step to obtain a set of input layer node values, and performs the necessary calculations so that It is propagated through a time-recursive neural network to generate output layer node values and hidden layer and content layer node values. Therefore, the input layer node values will be related to the time steps of calculating hidden, output and content layer node values; and hidden, output and content layer node values will be related to the time steps that generate these node values. The input layer node values are sample values of the system simulated by the time-recurrent neural network, such as images, voice samples, and snapshots of business market data. For long-term and short-term memory neural networks, each execution of the neural network unit uses a time step to obtain a set of memory cell input values and perform the necessary calculations to generate memory cell output values (as well as memory cell states and input gates, Forget gate and output gate values), which can also be understood as the transmission of memory cell input values through long and short-term memory cells. Therefore, the input value of the memory cell will be related to the time step of calculating the state of the memory cell and the values of the input gate, the forget gate and the output gate; and the memory state and the input gate, the value of the forget gate and the output gate will be related to the time of generating these node values. step.

The content node value, also known as the state node, is the state value of the neural network. This state value is based on the value associated with the previous time step. Input layer node values, not just the input layer node values associated with the current time step. The calculation performed by the neural network unit for the time step (for example, the hidden layer node value calculation of the Elman or Jordan time recurrent neural network) is a function of the content layer node value generated by the previous time step. Therefore, the network state value (content node value) at the beginning of the time step will affect the output layer node value generated during the time step. In addition, the network status value at the end of the time step will be affected by the input node value of this time step and the network status value at the beginning of the time step. Similarly, for long-term and short-term memory cells, the memory cell state value is related to the memory cell input value of the previous time step, rather than only the memory cell input value of the current time step. Because the calculation performed by the neural network unit on the time step (such as the calculation of the next memory cell state) is a function of the memory cell state value generated by the previous time step, the network state value (memory cell state value) at the beginning of the time step Affects the output value of the memory cell in this time step, and the network state value at the end of this time step will be affected by the memory cell input value and the previous network state value at this time step.

Figure 40 is a block diagram showing an example of an Elman time recurrent neural network. The Elman time recurrent neural network of the fortieth figure includes input layer nodes, or neurons, labeled D0, D1 to Dn, collectively referred to as multiple input layer nodes D and individually collectively referred to as input layer nodes D; hidden layer nodes / Neurons, labeled Z0, Z1 to Zn, collectively referred to as multiple hidden layer nodes Z and individually collectively referred to as hidden layer nodes Z; output layer nodes / neurons, labeled Y0, Y1 to Yn, collectively referred to as multiple outputs Layer node Y and are collectively referred to as output layer node Y; and content layer nodes / neural Yuan, labeled as C0, C1 to Cn, collectively referred to as multiple content layer nodes C and individually referred to as content layer nodes C. In the example of the Elman time recurrent neural network in the fortieth figure, each hidden layer node Z has an input connected to each input layer node D's output, and has an input connected to each content layer node C's output; each output The layer node Y has an input connected to an output of each hidden layer node Z; and each content layer node C has an input connected to an output of a corresponding hidden layer node Z.

In many ways, Elman time recurrent neural networks operate similarly to traditional feedforward artificial neural networks. That is, for a given node, each input link of this node will have an associated weight; the value received by a node at an input link will be multiplied by the associated weight to produce a product; this node will Adding the products associated with all input links to generate a total (the total may also include an offset term); generally, an activation function is performed on the total to generate the node's output value, which Sometimes called the start value of this node. For traditional feedforward networks, data always flows from the input layer to the output layer. That is, the input layer provides a value to the hidden layer (usually there will be multiple hidden layers), and the hidden layer will generate its output value to provide to the output layer, and the output layer will generate an output that can be accessed.

However, unlike the traditional feedforward network, the Elman time recurrent neural network also includes some feedback links, that is, the link from the hidden layer node Z to the content layer node C in the fortieth figure. The Elman time recurrent neural network works as follows. When the input layer node D provides an input value to the hidden layer node Z in a new time step, the content node C provides a value to the hidden layer node Z. This value is the hidden layer node Z. In response to previous losses In, which is the output value of the current time step. In this sense, the content node C of the Elman time recurrent neural network is a memory based on the input value of the previous time step. The forty-first and forty-second diagrams will describe an operation example of the neural network unit 121 that performs the calculation of the Elman time recurrent neural network associated with the forty diagrams.

To illustrate the present invention, the Elman time recurrent neural network is a time recurrent neural network including at least one input node layer, a hidden node layer, an output node layer and a content node layer. For a given time step, the content node layer stores the results generated by the hidden node layer in the previous time step and fed back to the content node layer. The result of the feedback to the content layer may be the execution result of the startup function or the result of the hidden node layer performing the accumulation operation without executing the startup function.

The forty-first diagram is a block diagram showing the random access memory 122 and weights of the data of the neural network unit 121 when the neural network unit 121 performs the calculation of the Elman time recurrent neural network associated with the forty diagram. An example of data configuration in the random access memory 124 is. In the example of Figure 41, it is assumed that the Elman time recurrent neural network of Figure 40 has 512 input nodes D, 512 hidden nodes Z, 512 content nodes C, and 512 output nodes Y. In addition, it is also assumed that this Elman time recurrent neural network is fully connected, that is, all 512 input nodes D are connected to each hidden node Z as input, all 512 content nodes C are connected to each hidden node Z as input, and all 512 The hidden node Z is connected to each output node Y as an input. In addition, the neural network unit 121 is configured as 512 neural processing units 126 or neurons, such as adopting a wide configuration. Finally, this example assumes that content node C is related to hidden The weights of the nodes Z are all equal to 1, so there is no need to store these weights as one.

As shown in the figure, the 512 rows (columns 0 to 511) below the weight random access memory 124 are loaded with weight values associated with the connection between the input node D and the hidden node Z. More precisely, as shown in the figure, column 0 is the weight associated with the input link from the input node D0 to the hidden node Z, that is, the text 0 will load the link associated with the input node D0 and the hidden node Z0. Text 1 will load the weight associated with the connection between input node D0 and hidden node Z1, text 2 will load the weight associated with the connection between input node D0 and hidden node Z2, and so on, text 511 will load association The weight of the link between the input node D0 and the hidden node Z511; column 1 contains the weight associated with the input link from the input node D1 to the hidden node Z, that is, the text 0 will load the input node D1 and the hidden node Z0 The weight of the link between texts, text 1 will load the weight associated with the link between input node D1 and hidden node Z1, the text 2 will load the weight of the link associated with input node D1 and hidden node Z2, and so on, text 511 Will load the weights associated with the link between input node D1 and hidden node Z511; up to column 511, column 511 will load the weights associated with the input link from input node D511 to hidden node Z, also That is, text 0 will be loaded with the weight associated with the connection between the input node D511 and the hidden node Z0, text 1 will be loaded with the weight associated with the connection between the input node D511 and the hidden node Z1, and text 2 will be loaded with the input node D511 and the hidden node The weight of the link between the hidden nodes Z2, and so on, the text 511 will load the weight associated with the link between the input node D511 and the hidden node Z511. This configuration and usage is similar to the previous correspondence The embodiments described in Figures 4 to 6A.

As shown in the figure, the subsequent 512 columns (columns 512 to 1023) of the weight random storage memory 124 load the weights associated with the connection between the hidden node Z and the output node Y in a similar manner.

The data random access memory 122 is loaded with Elman time recurrent neural network node values for a series of time steps. Further, the data random access memory 122 loads the node values of a given time step in groups of three rows. As shown in the figure, a data random access memory 122 with 64 rows is taken as an example. The data random access memory 122 can load node values for 20 different time steps. In the example of the forty-first figure, columns 0 to 2 are loaded with node values for time step 0, columns 3 to 5 are loaded with node values for time step 1, and so on, and columns 57 to 59 are loaded for time steps 19 node value used. The first column in each group is the value of the input node D at this time step. The second column in each group contains the value of the hidden node Z at this time step. The third column in each group contains the value of output node Y at this time step. As shown in the figure, each row of the data random access memory 122 is loaded with the node value of its corresponding neuron or neural processing unit 126. That is, row 0 loads the node values associated with nodes D0, Z0, and Y0, and its calculation is performed by the neural processing unit 0; row 1 loads the node values associated with nodes D1, Z1, and Y1. The calculation is Executed by the neural processing unit 1; and so on, row 511 loads the node values associated with the nodes D511, Z511 and Y511, and the calculation is performed by the neural processing unit 511. This part corresponds to the forty-second figure in the following. There will be more detailed instructions.

As indicated in Figure 41, for a given time step In short, the value of the hidden node Z in the second column of the three rows of memory in each group will be the value of the content node C in the next time step. That is, the value of node Z calculated and written by the neural processing unit 126 in a time step will become the value of node C used by this neural processing unit 126 to calculate the value of node Z in the next time step ( With the value of input node D for this next time step). The initial value of content node C (the value of node C used to calculate the value of node Z in column 1 at time step 0) is assumed to be zero. This will be explained in more detail in the subsequent sections of the non-structural program corresponding to Figure 42.

Preferably, the value of the input node D (the values of columns 0, 3 in the example in FIG. 41, and so on to the value of column 57) is written by the architecture program running on the processor 100 through the MTNN instruction 1400 / The data random access memory 122 is filled in, and is read / used by a non-architecture program running on the neural network unit 121, such as the non-architecture program of FIG. 42. Conversely, the values of the hidden / output nodes Z / Y (columns 1 and 2, 4 and 5 in the example in Figure 41, and so on to the values of columns 58 and 59) are executed by the neural network The non-architecture program of the unit 121 writes / fills in the data random access memory 122, and is read / used by the architecture program running on the processor 100 through the MFNN instruction 1500. The example in Figure 41 assumes that the framework program will perform the following steps: (1) For 20 different time steps, fill the value of the input node D into the data random access memory 122 (rows 0, 3, according to (Similarly to column 57); (2) start the non-architecture program in Figure 42; (3) detect whether the non-architecture program has been executed; (4) read out the output node Y from the data random access memory 122 Values (columns 2, 5, and so on to column 59); and (5) repeat step (1) Go to (4) several times until the task is completed, such as the calculation required to identify the words of the mobile phone user.

In another implementation method, the architecture program will perform the following steps: (1) for a single time step, fill the data random access memory 122 (such as row 0) with the value of the input node D; (2) start the non-architecture Program (a modified version of one of the non-architecture programs shown in Figure 42 without looping, and only accesses a single set of three rows of random storage memory 122); (3) detects whether the non-architecture program has been executed (4) read out the value of the output node Y from the data random access memory 122 (such as column 2); and (5) repeat steps (1) to (4) several times until the task is completed. Which of these two methods is optimal depends on the sampling method of the input value of the time recurrent neural network. For example, if this task allows the input to be sampled at multiple time steps (for example, about 20 time steps) and perform calculations, the first method is ideal because this method may bring more computing resources efficiency and / Or better performance, but if this task only allows sampling in a single time step, you need to use the second method.

The third embodiment is similar to the foregoing second method, but different from the second method using a single set of three rows of data random access memory 122, the non-structural program of this method uses multiple sets of three rows of memory, that is, in the Each time step uses a different set of three columns of memory, this section is similar to the first method. In this third embodiment, preferably, the architecture program includes a step before step (2). In this step, the architecture program will update the non-architecture program before it is started, for example, in the address 1 instruction. The 122 rows of data random access memory are updated to point to the next set of three rows of memory.

Figure 42 is a table showing a program stored in the program memory 129 of the neural network unit 121. This program is executed by the neural network unit 121 and uses data and weights according to the configuration of figure 41 To achieve Elman time recurrent neural network. The instructions in the non-architecture program of the forty-second figure (and forty-fifth, forty-eight, fifty-one, fifty-four, and fifty-seven) are as described above (for example, MULT-ACCUM). (LOOP), initialization (INITIALIZE) instructions, the following paragraphs assume that these instructions are consistent with the previous description, unless there is a different description.

The example program in Figure 42 contains 13 non-framework instructions, located at addresses 0 to 12, respectively. The instruction at address 0 (INITIALIZE NPU, LOOPCNT = 20) clears the accumulator 202 and initializes the loop counter 3804 to the value 20 to execute the loop group 20 times (the instructions at addresses 4 to 11). Preferably, this initialization instruction will also make the neural network unit 121 in a wide configuration. In this way, the neural network unit 121 will be configured as 512 neural processing units 126. As described in the subsequent chapters, during the execution of instructions at addresses 1 to 3 and addresses 7 to 11, these 512 neural processing units 126 operate as 512 corresponding hidden layer nodes Z, and at address 4 During the execution of instructions from 6 to 6, these 512 neural processing units 126 operate as 512 corresponding output layer nodes Y.

The instructions at addresses 1 to 3 do not belong to the loop group of the program and will only be executed once. These instructions calculate the initial value of node Z in the hidden layer and write it into the data random access memory 122 in column 1 for the first execution of the instructions at addresses 4 to 6, to calculate the first time step (time step 0) output node Y. In addition, these instructions are Calculate and write the value of the hidden layer node Z in row 1 of the random access memory 122 into the value of the content layer node C for the first execution of the instructions at addresses 7 and 8 to calculate the hidden layer node The value of Z is used for the second time step (time step 1).

During the execution of the instructions at addresses 1 and 2, each of the 512 neural processing units 126 will perform 512 multiplication operations, and will place 512 input nodes located at 122 rows 0 of the data random access memory. The value of D is multiplied by the weight corresponding to the row of the neural processing unit 126 in the columns 0 to 511 of the weighted random access memory 124 to generate 512 multiply accumulations and add the accumulator 202 to the corresponding neural processing unit 126. During the execution of the instruction at the address 3, the values of the 512 accumulators 202 of the 512 neural processing units will be transferred and written into the data random access memory 122 row 1. In other words, the output instruction at address 3 writes the value of the accumulator 202 of each of the 512 neural processing units 512 into the data random access memory 122 column 1. This value is the initial hidden layer Z value. Subsequently, this instruction clears the accumulator 202.

The operations performed by the instructions at addresses 1 to 2 of the non-architecture program in Figure 42 are similar to the operations performed by the instructions at addresses 1 to 2 of the non-architecture instruction in Figure 4. Further, the instruction at address 1 (MULT_ACCUM DR ROW 0) will instruct each of the 512 neural processing units 126 to read the corresponding text in column 0 of the data random access memory 122 The work register 208 reads the corresponding text in column 0 of the weight random access memory 124 into its multiplex register 705, multiplies the data text and the weight text to generate a product, and adds the product to the accumulator 202. Address 2 instruction (MULT-ACCUM ROTATE, WR ROW + 1, COUNT = 511) instructs each of the 512 neural processing units 126 to transfer the text from the adjacent neural processing unit 126 into its multiplex register 208 (using the neural network unit 121). 512 multiplexer registers 208 collectively operated by 512 text spinners, these registers are address register 1 instructions to read data from the random access memory 122 row register), The corresponding text in the next row of the weight random access memory 124 is read into its multiplexing register 705, the data text is multiplied with the weight text to generate a product, and the product is added to the accumulator 202, and the foregoing operation is performed 511 times.

In addition, a single non-architecture output instruction (OUTPUT PASSTHRU, DR OUT ROW 1, CLR ACC) at address 3 in the forty-second figure will write the operation of the start function instruction and the write output at addresses 3 and 4 in the fourth figure Instruction merging (although the program in Figure 42 passes the value of accumulator 202, the program in Figure 4 performs an activation function on the value of accumulator 202). That is to say, in the program of the forty-second figure, the start function executed on the value of the accumulator 202, if any, is specified in the output instruction (also specified in the output instructions at addresses 6 and 11), and It is not specified in the program of the fourth figure in a different non-framework startup function instruction. Another embodiment of the non-architecture program in the fourth figure (and twentieth, twenty-sixth A, and twenty-eight) is to start the operation of the function instruction and write the output instruction (such as address 3 in the fourth figure). And 4) combined into a single non-architecture output instruction as shown in FIG. 42 also belongs to the scope of the present invention. The example in Figure 42 assumes that the nodes of the hidden layer (Z) will not execute the start function on the accumulator value. However, the embodiments in which the hidden layer (Z) executes the activation function on the accumulator value are also within the scope of the present invention. These embodiments You can use instructions at addresses 3 and 11 to perform operations such as S-type, hyperbolic tangent, correction function, etc.

In contrast to the instructions at addresses 1 to 3, which are executed only once, the instructions at addresses 4 to 11 are located within the program loop and are executed a number of times, which is specified by the loop count (for example, 20). The first nineteen executions of the instructions at addresses 7 to 11 calculate the value of the hidden layer node Z and write it to the data random access memory 122 for the second to twenty executions of the instructions at addresses 4 to 6 To calculate the output layer node Y for the remaining time steps (time steps 1 to 19). (The last / twentieth execution of the instructions at addresses 7 to 11 is to calculate the value of the hidden layer node Z and write it to the data random access memory row 122, 61. However, these values are not used.)

In the first execution of the instructions at addresses 4 and 5 (MULT-ACCUM DR ROW + 1, WR ROW 512 and MULT-ACCUM ROTATE, WR ROW + 1, COUNT = 511) (corresponding to time step 0), this Each of the 512 neural processing units 126 performs 512 multiplication operations to randomize the data to the values of 512 hidden nodes Z in row 1 of the memory 122 (these values are from addresses 1 to 3) Instructions are generated and written in a single execution) Multiplied by the weights of rows 512 to 1023 of the random access memory 124 corresponding to the trip of this neural processing unit 126 to generate 512 multiplying accumulations and adding to the corresponding neural processing unit 126 The accumulator 202. In the first execution of the instruction at address 6 (OUTPUT ACTIVATION FUNCTION, DR OUT ROW + 1, CLR ACC), a start function (such as S-type, hyperbolic tangent, correction function) will be executed for the 512 accumulated values. To calculate the value of node Y in the output layer, the execution result will be written into row 2 of the data random access memory 122.

In the second execution of the instructions at addresses 4 and 5 (corresponding to time step 1), each of the 512 neural processing units 126 will perform 512 multiplication operations to randomly access the data in memory. The values of 512 hidden nodes Z in column 4 of 122 (these values are generated and written by the first execution of the instructions at addresses 7 to 11) are multiplied by the weight of columns 512 to 1023 of random access memory 124 Corresponds to the weight of the trip of the neural processing unit 126 to generate 512 multiply accumulated and add to the accumulator 202 of the corresponding neural processing unit 126, and in the second execution of the instruction at address 6, the 512 accumulated values will be Execute a start function to calculate the value of output layer node Y. The result is written into column 5 of data random access memory 122; in the third execution of the instructions at addresses 4 and 5 (corresponding to time step 2) Each of the 512 neural processing units 126 will perform 512 multiplication operations to randomize the data to the values of 512 hidden nodes Z in row 7 of memory 122 (these values are from address 7 to 11 is generated and written by the second execution of the instruction) The weights corresponding to the row of the neural processing unit 126 in columns 512 to 1023 of the weight random access memory 124 above are generated to generate 512 multiply-accumulate and add to the accumulator 202 of the corresponding neural processing unit 126, and the instruction at address 6 In the third execution, an activation function will be executed for the 512 accumulated values to calculate the value of the output layer node Y. This result is written into the data random access memory row 122; and so on, at the address During the twentieth execution of the 4 and 5 instructions (corresponding to time step 19), each of the 512 neural processing units 126 will perform 512 multiplication operations to randomize the data in the memory 122. The values of the 512 hidden nodes Z in column 58 (these values are generated and written by the nineteenth execution of the instructions at addresses 7 to 11 (In) Multiply the weights of rows 512 to 1023 of the random access memory 124 corresponding to the trip of this neural processing unit 126 to generate 512 multiply accumulations and add them to the accumulator 202 of the corresponding neural processing unit 126, and at the address In the twentieth execution of the 6 instruction, an activation function is executed for the 512 accumulated values to calculate the value of the output layer node Y. The execution result is written into the row 59 of the data random access memory 122.

In the first execution of the instructions at addresses 7 and 8, each of the 512 neural processing units 126 accumulates the values of the 512 content nodes C in the random access memory 122 in column 1 to it. Accumulator 202. These values are generated by a single execution of the instructions at addresses 1 to 3. Further, the instruction at address 7 (ADD_D_ACC DR ROW + 0) will instruct each of the 512 neural processing units 126 to randomly access data in the current row of the memory 122 (in the first execution process) The corresponding text in column 0) is read into its multiplexer register 208, and this text is added to the accumulator 202. The instruction at address 8 (ADD_D_ACC ROTATE, COUNT = 511) instructs each of the 512 neural processing units 126 to transfer the text from the adjacent neural processing unit 126 into its multiplexing register 208 (using the The 512 multiplex register 208 of the neural network unit 121 is a 512-character rotator composed of collective operations. These multiplex registers are instructions at address 7 to read data into the random access memory 122 row. Register), add this text to the accumulator 202, and perform the aforementioned operation 511 times.

In the second execution of the instructions at addresses 7 and 8, each of the 512 neural processing units 126 will randomly fetch the data from the number 512 of content node C in memory row 4 The values are accumulated to its accumulator 202, which are generated and written by the first execution of the instructions at addresses 9 to 11; during the third execution of the instructions at addresses 7 and 8, the 512 neural processes Each of the neural processing units 126 in the unit 126 will accumulate the values of the 512 content nodes C in the random access memory 122 to the accumulator 202. These values are the second from the instructions at addresses 9 to 11 Generated and written by multiple executions; and so on, during the twentieth execution of the instructions at addresses 7 and 8, each of the 512 neural processing units 126 will randomly fetch data into memory The values of 512 content nodes C of column 122 and 58 are added to its accumulator 202, and these values are generated and written by the nineteenth execution of the instructions at addresses 9 to 11.

As mentioned above, the example of the forty-second figure assumes that the weight of the link associated with the content node C to the hidden layer node Z has a value of one. However, in another embodiment, these links in the Elman time recurrent neural network have non-zero weight values. These weights are placed in the weight random access memory 124 before the program in Figure 42 is executed. (For example, rows 1024 to 1535), the program command at address 7 is MULT-ACCUM DR ROW + 0, WR ROW 1024, and the program command at address 8 is MULT-ACCUM ROTATE, WR ROW + 1, COUNT = 511. Preferably, the instruction of address 8 does not access the weight random access memory 124, but the instruction of rotating the address 7 reads the value of the multiplexer register 705 from the weight random access memory 124. Without accessing the weight random access memory 124 during the time-frequency period of the 511 execution address 8 instructions, more bandwidth can be reserved for use by the framework program access weight random access memory 124.

In the first execution of the instructions at addresses 9 and 10 (MULT-ACCUM DR ROW + 2, WR ROW 0 and MULT-ACCUM ROTATE, WR ROW + 1, COUNT = 511) (corresponding to time step 1), this Each of the 512 neural processing units 126 performs 512 multiplication operations to multiply the value of the 512 input node D of column 3 of data random access memory 122 by the column of weight random access memory 124 The weights of 0 to 511 corresponding to the trip of this neural processing unit 126 to generate 512 products, together with the accumulation operation performed by the instructions at addresses 7 and 8 on the 512 content node C values, are accumulated in the corresponding neural processing unit 126. Accumulator 202 calculates the value of node Z in the hidden layer. In the first execution of the instruction at address 11 (OUTPUT PASSTHRU, DR OUT ROW + 2, CLR ACC), the 512 accumulators of the 512 neural processing units 126 The value of 202 is passed and written into the data random access memory 122, column 4, and the accumulator 202 will be cleared; in the second execution of the instructions at addresses 9 and 10 (corresponding to time step 2), this 512 Each of the neural processing units 126 performs 512 multiplication operations, The value of the 512 input nodes D in row 6 of the random access memory 122 is multiplied by the weight corresponding to the row of the neural processing unit 126 in the rows 0 to 511 of the random access memory 124 to generate 512 products. Together with the instructions of addresses 7 and 8 for the accumulation of 512 content node C values, they are accumulated in the accumulator 202 of the corresponding neural processing unit 126 to calculate the value of the hidden layer node Z. In the second execution, the values of the 512 accumulators 202 of the 512 neural processing units 126 are transferred and written into the data random access memory 122 column 7, and the accumulator 202 is cleared; and so on, in place Addresses 9 and 10 In the nineteenth execution (corresponding to time step 19), each of the 512 neural processing units 126 will perform 512 multiplication operations to input data to the 512 inputs of the 57th row of the memory 122 The value of node D is multiplied by the weight of rows 0 to 511 of the random access memory 124 corresponding to the row of this neural processing unit 126 to produce 512 products, together with instructions at addresses 7 and 8 for 512 content nodes C The accumulation operation performed by the values is accumulated in the accumulator 202 of the corresponding neural processing unit 126 to calculate the value of the node Z in the hidden layer. In the nineteenth execution of the instruction at address 11, the 512 neural processing units 126 The values of the 512 accumulators 202 are passed and written into the data random access memory 122 column 58, and the accumulators 202 are cleared. As mentioned above, the hidden layer node Z value generated and written in the twentieth execution of the instructions at addresses 9 and 10 will not be used.

The instruction at address 12 (LOOP 4) will decrement the loop counter 3804 and return to the instruction at address 4 if the new loop counter 3804 value is greater than zero.

The forty-third diagram is a block diagram showing an example of Jordan time recurrent neural network. The Jordan Time Recurrent Neural Network in Figure 43 is similar to the Elman Time Recurrent Neural Network in Figure 40. It has an input layer node / neuron D, a hidden layer node / neuron Z, and an output layer node / neuron Y. , With content layer node / neuron C. However, in the Jordan time recursive neural network of Figure 43, the content layer node C uses the output feedback from its corresponding output layer node Y as its input link, instead of the Elman time recursion as shown in Figure 40. In the neural network, the output from the hidden layer node Z is used as its input link.

To illustrate the present invention, the Jordan time-recurrent neural network is a time-recurrent neural network including at least one input node layer, a hidden node layer, an output node layer and a content node layer. At the beginning of a given time step, the content node layer stores the results produced by the output node layer in the previous time step and fed back to the content node layer. The result fed back to the content layer may be the result of the activation function or the output node layer performs an accumulation operation without executing the activation function.

The forty-fourth diagram is a block diagram showing that when the neural network unit 121 performs the calculation of the Jordan time recurrent neural network associated with the forty-third diagram, the data of the neural network unit 121 and the random access memory 122 and An example of data allocation in the weighted random access memory 124 is. In the example of Figure 44, it is assumed that the Jordan time recurrent neural network of Figure 43 has 512 input nodes D, 512 hidden nodes Z, 512 content nodes C, and 512 output nodes Y. In addition, it is also assumed that this Jordan time recurrent neural network is fully connected, that is, all 512 input nodes D are connected to each hidden node Z as input, all 512 content nodes C are connected to each hidden node Z as input, and all 512 The hidden node Z is connected to each output node Y as an input. The example of the Jordan time recursive neural network shown in Figure 44 applies an activation function to the value of the accumulator 202 to generate the value of the output node Y. However, this example assumes that the accumulation before the activation function is applied. The value of the router 202 is passed to the content layer node C, instead of the actual output layer node Y value. In addition, the neural network unit 121 is provided with 512 neural processing units 126, or neurons, for example, adopting a wide configuration. Finally, this example assumes that the weights associated with the links from content node C to hidden node Z have a value of 1; There is no need to store these weight values as one.

As in the example of the forty-first figure, as shown in the figure, the 512 rows (rows 0 to 511) below the weight random access memory 124 will be loaded with the weight values associated with the connection between the input node D and the hidden node Z , And the subsequent 512 rows (columns 512 to 1023) of the weight random access memory 124 will be loaded with the weight values associated with the connection between the hidden node Z and the output node Y.

Data random access memory 122 is loaded with Jordan time recurrent neural network node values for a series of time steps similar to the example in Figure 41; however, the example in Figure 44 uses a set of four The memory loads listed provide the node values for a given time step. As shown in the figure, in an embodiment with 64 rows of data random access memory 122, data random access memory 122 can be loaded with the node values required for 15 different time steps. In the example of Figure 44, columns 0 to 3 are loaded with node values for time step 0, columns 4 to 7 are loaded with node values for time step 1, and so on, and 60 to 63 are loaded. Node value for time step 15. The first row of the four-row set of memories is the value of the input node D loaded with this time step. The second row of the four-row set of memories is the value of the hidden node Z that is loaded at this time step. The third column of the four-column set of memory is the value of the content node C that contains this time step. The fourth column of the four-column set of memory is the value of the output node Y loaded with this time step. As shown in the figure, each row of the data random access memory 122 is loaded with the node value of its corresponding neuron or neural processing unit 126. That is, row 0 loads the node values associated with nodes D0, Z0, C0, and Y0, and the calculation is performed by the neural processing unit 0; row 1 It loads the node values associated with nodes D1, Z1, C1, and Y1, and its calculation is performed by the neural processing unit 1. By analogy, line 511 loads the node values associated with nodes D511, Z511, C511, and Y511. Is executed by the neural processing unit 511. This part will be explained in more detail in the subsequent section corresponding to the 44th figure.

The value of the content node C at a given time step in Figure 44 is generated during this time step and used as the input for the next time step. That is, the value of node C calculated and written by the neural processing unit 126 in this time step will become the value of node C used by this neural processing unit 126 to calculate the value of node Z in the next time step ( With the value of input node D for this next time step). The initial value of the content node C (that is, the value of the node C used for calculating the value of the node 1 in the time step 0) is assumed to be zero. This part will be explained in more detail in the subsequent chapter of the non-structural program corresponding to Figure 45.

As described in the forty-first figure above, preferably, the value of the input node D (the values of columns 0 and 4 in the example of the forty-fourth figure, and so on to the value of column 60) is executed by the processor 100. The architecture program is written / filled into the data random access memory 122 through the MTNN instruction 1400, and is read / used by a non-architecture program running on the neural network unit 121, such as the non-architecture program of FIG. 45. Conversely, the values of hidden node Z / content node C / output node Y (in the example in Figure 44 are columns 1/2/3, 5/6/7, and so on to column 61/62/63 The value) is written / filled into the data random access memory 122 by a non-architecture program executed on the neural network unit 121, and is executed by the processor 100 The program is read / used by MFNN instruction 1500. The example in Figure 44 assumes that the framework program will perform the following steps: (1) For 15 different time steps, fill the value of the input node D into the data random access memory 122 (rows 0, 4, according to And so on to column 60); (2) start the non-architecture program in Figure 45; (3) detect whether the non-architecture program has been executed; (4) read out the output node Y from the data random access memory 122 Values (columns 3, 7, and so on to column 63); and (5) repeat steps (1) to (4) several times until the task is completed, such as the calculation required to identify the words of the mobile phone user.

In another implementation method, the architecture program will perform the following steps: (1) for a single time step, fill the data random access memory 122 (such as row 0) with the value of the input node D; (2) start the non-architecture Program (a modified version of one of the non-architecture programs in Figure 45 without looping, and only accesses a single set of four rows of random storage memory 122); (3) detects whether the non-architecture program has been executed (4) read the value of the output node Y from the data random access memory 122 (such as column 3); and (5) repeat steps (1) to (4) several times until the task is completed. Which of these two methods is optimal depends on the sampling method of the input value of the time recurrent neural network. For example, if this task allows the input to be sampled over multiple time steps (for example, about 15 time steps) and perform calculations, the first method is ideal because this method can bring more computing resources efficiency and / Or better performance, but if this task only allows sampling in a single time step, you need to use the second method.

The third embodiment is similar to the foregoing second method, but different from the second method using a single set of four data random access records The memory is 122 rows. The non-architecture program in this method uses multiple sets of four rows of memory, that is, different sets of four rows of memory are used at each time step. This part is similar to the first method. In this third embodiment, preferably, the architecture program includes a step before step (2). In this step, the architecture program will update the non-architecture program before it is started, such as the instruction of address 1. The 122 rows of data in the RAM are updated to point to the next set of four rows of memory.

The forty-fifth diagram is a table showing a program stored in the program memory 129 of the neural network unit 121. This program is executed by the neural network unit 121 and uses data and weights according to the configuration of the forty-fourth diagram. To achieve Jordan time recurrent neural network. The non-structural program in Figure 45 is similar to the non-structural program in Figure 42. For the differences between the two, refer to the descriptions in the relevant sections of this article.

The example program in Figure 45 includes 14 non-framework instructions, which are located at addresses 0 to 13. The instruction at address 0 is an initialization instruction that clears the accumulator 202 and initializes the loop counter 3804 to a value of 15 to execute the loop group 15 times (instructions at addresses 4 to 12). Preferably, the initialization instruction does not place the neural network unit 121 in a wide configuration and configures 512 neural processing units 126. As described in this article, during the execution of instructions at addresses 1 to 3 and addresses 8 to 12, the 512 neural processing units 126 correspond to and operate as 512 hidden layer nodes Z, and at addresses 4, 5 During the execution of the instruction of 7, these 512 neural processing units 126 correspond to and operate as 512 output layer nodes Y.

The instructions at addresses 1 to 5 and 7 are the same as the instructions at addresses 1 to 6 in Figure 42 and have the same functions. Addresses 1 to 3 Calculate the initial value of the hidden layer node Z and write it to the data in random access memory 122. Row 1 is used for the first execution of the instructions at addresses 4, 5, and 7 to calculate the first time step (time step 0) output node Y.

During the first execution of the output instruction at address 6, the 512 accumulator 202 values generated by the accumulation of the instructions at addresses 4 and 5 (these values will be used by the output instruction at address 7 to calculate And write the value of output layer node Y) will be passed and written into the data random access memory 122 column 2, these values are the content layer node C value generated in the first time step (time step 0) and are Used in the second time step (time step 1); during the second execution of the output instruction at address 6, the values of the 512 accumulator 202 generated by the accumulation of the instructions at addresses 4 and 5 (next, These values will be used by the output instruction at address 7 to calculate and write the value of node Y in the output layer) will be passed and written into the data random access memory row 122, these values are the second time step (time The content layer node C value generated in step 1) is used in the third time step (time step 2); and so on, during the fifteenth execution of the output instruction at address 6, the 512 Accumulator 202 values generated by instruction accumulation at addresses 4 and 5 (these are the following The value will be used by the output instruction at address 7 to calculate and write the value of node Y in the output layer.) It will be passed and written to the data in random access memory 122 column 58. These values are the fifteenth time step (time The content layer node C value generated in step 14) (and read by the instruction at address 8, but will not be used).

The instructions at addresses 8 to 12 are roughly the same and have the same functions as the instructions at addresses 7 to 11 in Figure 42. Difference. This difference is that the instruction at address 8 (ADD_D_ACC DR ROW + 1) in the forty-fifth figure will increase the number of rows in the data random access memory 122 by one, while the instruction at address seven in the forty-second figure (ADD_D_ACC DR ROW + 0) will increase the number of rows of data random access memory 122 to zero. This difference is due to the difference in the data configuration in the data random access memory 122. In particular, the four-column configuration in Figure 44 includes an independent row for the content layer node C value (such as row 2 , 6, 10, etc.), and the three-column configuration in the forty-first figure does not have this independent column, but the value of the content layer node C and the value of the hidden layer node Z share the same column (such as column 1, 4, 7 etc.). Fifteen executions of the instructions at addresses 8 to 12 calculate the value of the hidden layer node Z and write it to the data random access memory 122 (write to columns 5, 9, 13, and so on until column 57) for The second to sixteen executions of the instructions at addresses 4, 5, and 7 are used to calculate the output layer node Y for the second to fifteenth time steps (time steps 1 to 14). (The last / fifteenth execution of the instructions at addresses 8 to 12 is to calculate the value of the hidden layer node Z and write it to the data random access memory row 122, but these values are not used.)

The loop instruction at address 13 decrements the loop counter 3804 and returns to the instruction at address 4 if the new loop counter 3804 value is greater than zero.

In another embodiment, the design of the Jordan time recurrent neural network uses the content node C to load the activation function value of the output node Y, and the activation function value is the cumulative value after the activation function is executed. In this embodiment, because the value of the output node Y is the same as the value of the content node C, the non-framework instruction at address 6 is not included in the non-framework program. thus The number of rows used in the data random access memory 122 can be reduced. More precisely, none of the columns (for example, columns 2, 6, 59) of the values of each load content node C in the forty-fourth figure exist in this embodiment. In addition, each time step of this embodiment only requires three rows of data random access memory 122, and 20 time steps are used instead of 15, the address of the non-architecture program instruction in the forty-fifth figure Appropriate adjustments will also be made.

Long short-term memory cells

The use of long short-term memory cells for temporal recurrent neural networks is a concept well known in the art. For example, Long Short-Term Memory, Sepp Hochreiter and Jürgen Schmidhuber, Neural Computation, November 15, 1997, Vol. 9, No. 8, Pages 1735-1780; Learning to Forget: Continental Prediction with LSTM, Felix A. Gers , Jürgen Schmidhuber, and Fred Cummins, Neural Computation, October 2000, Vol. 12, No. 10, Pages 2451-2471; these documents can be obtained from the MIT Press Journals. Long and short-term memory cells can be constructed in many different types. The long-term and short-term memory cell 4600 of the forty-sixth figure described below is based on the URL http://deeplearning.net/tutorial/lstm.html and is titled as the LSTM Networks for Sentiment Analysis. The short and long-term memory cells described in the tutorial are models. A copy of this tutorial was downloaded on October 19, 2015 (hereinafter referred to as the "long-and-short-term memory tutorial") and provided in the US application information disclosure report of this case. This long short-term memory cell 4600 can be used to generally describe the ability of the embodiment of the neural network unit 121 described herein to effectively perform calculations related to long-term and short-term memory. It is worth noting that the embodiments of these neural network units 121, including the embodiment described in FIG. 49, can effectively perform other long-term and short-term events other than the long-term and short-term memory cells described in FIG. 46. Calculation of memory cells.

Preferably, the neural network unit 121 may be configured to perform calculations on a time-recurrent neural network having a long-term short-term memory cell layer connected to other layers. For example, in this long-short-term memory tutorial, the network includes a mean-source layer to receive the output of the long-term and short-term memory cells from the long-term and short-term memory layer (H), and a logistic regression layer to receive the output of the mean-source source .

The forty-sixth figure is a block diagram showing one embodiment of a long-term short-term memory cell 4600.

As shown in the figure, the long-term and short-term memory cell 4600 includes a memory cell input (X), a memory cell output (H), an input gate (I), an output gate (O), and a forget gate (F). A memory cell state (C) and a candidate memory cell state (C '). The input gate (I) can gate the signal transmission from the memory cell input (X) to the memory cell state (C), and the output gate (O) can gate the signal transmission from the memory cell state (C) to the memory cell output (H). . This memory cell state (C) is fed back as a candidate memory cell state (C ') at a time step. The forget gate (F) can gate this candidate memory cell state (C '), and this candidate memory cell state will be fed back and become the memory cell state (C) at the next time step.

The embodiment of Figure 46 uses the following equations to calculate the aforementioned various values: (1) I = SIGMOID (Wi * X + Ui * H + Bi)

(2) F = SIGMOID (Wf * X + Uf * H + Bf)

(3) C ’= TANH (Wc * X + Uc * H + Bc)

(4) C = I * C ’+ F * C

(5) O = SIGMOID (Wo * X + Uo * H + Bo)

(6) H = O * TANH (C)

Wi and Ui are weight values associated with the input gate (I), and Bi is an offset value associated with the input gate (I). Wf and Uf are weight values associated with forget gate (F), and Bf is an offset value associated with forget gate (F). Wo and Uo are weight values associated with the output gate (O), and Bo is an offset value associated with the output gate (O). As before, equations (1), (2) and (5) calculate the input gate (I), the forget gate (F) and the output gate (O), respectively. Equation (3) calculates the candidate memory cell state (C '), and equation (4) calculates the candidate memory cell state (C') with the current memory cell state (C) as the input. The current memory cell state (C) is Memory cell status at the current time step (C). Equation (6) calculates the memory cell output (H). However, the present invention is not limited to this. Embodiments of calculating the input gate, forget gate, output gate, candidate memory cell state, memory cell state, and long-term and short-term memory cell output using other methods are also covered by the present invention.

To illustrate the present invention, the long-term and short-term memory cells include a memory cell input, a memory cell output, a memory cell state, a candidate memory cell state, an input gate, an output gate, and a forget gate. For each time step, the state of the input gate, output gate, forget gate and candidate memory cell is a function of the memory cell input of the current time step and the memory cell output and related weights of the previous time step. Memory cells at this time step The state is a function of the memory cell state, candidate memory cell state, input gate and output gate of the previous time step. In this sense, the state of the memory cell is fed back to calculate the state of the memory cell for the next time step. The memory cell output at this time step is a function of the memory cell state and output gate calculated at this time step. A long-term short-term memory neural network is a neural network with a long-term short-term memory cell.

The forty-seventh diagram is a block diagram showing the data of the neural network unit 121 when the neural network unit 121 performs the calculation of the long-term and short-term memory cells of the long-term and short-term memory neural network associated with the forty-sixth diagram. An example of the data arrangement in the random access memory 122 and the weighted random access memory 124. In the example in Figure 47, the neural network unit 121 is configured as 512 neural processing units 126 or neurons, such as a wide configuration. However, there are only 128 neural processing units 126 (such as neural processing units 0 to 127). The value generated by) will be used because the long-term and short-term memory layer in this example has only 128 long-term and short-term memory cells 4600.

As shown in the figure, the weight random access memory 124 loads the weight values, offset values, and median values of the corresponding neural processing units 0 to 127 of the neural network unit 121. Rows 0 to 127 of the weight random access memory 124 load the weight values, offset values, and intermediate values of the corresponding neural processing units 0 to 127 of the neural network unit 121. Each of the columns 0 to 14 is loaded with 128 values corresponding to the aforementioned equations (1) to (6) to be provided to the neural processing unit 0 to 127. These values are: Wi, Ui, Bi, Wf, Uf, Bf, Wc, Uc, Bc, C ', TANH (C), C, Wo, Uo, Bo. Preferably, the weight and offset values-Wi, Ui, Bi, Wf, Uf, Bf, Wc, Uc, Bc, Wo, Uo, Bo (located in columns 0 to 8 and columns 12 to 14)-are performed by On processor 100 The architecture program is written / filled into the weight random access memory 124 through the MTNN instruction 1400, and is read / used by a non-architecture program running on the neural network unit 121, as shown in the non-architecture program in FIG. Preferably, the median values-C ', TANH (C), C (located in columns 9 to 11)-are written / filled into the weight random access memory 124 by a non-structural program executed on the neural network unit 121 and Read / use as described below.

As shown in the figure, the data random access memory 122 is loaded with input (X), output (H), input gate (I), forget gate (F) and output gate (O) values for a series of time steps. Further, the memory is loaded with a set of five values of X, H, I, F, and O for a given time step. Take a data random access memory 122 with 64 rows as an example. As shown in the figure, the data random access memory 122 can be loaded with memory cell values for 12 different time steps. In the example in Figure 47, columns 0 to 4 are the values of memory cells for time step 0, columns 5 to 9 are the values of memory cells for time step 1, and so on, 55 to 59. Load the memory cell value for time step 11. The first column of the five-column set of memory contains the X value for this time step. The second column of the five-column set contains the H value for this time step. The third column of the five-column set contains the I value for this time step. The fourth column of the five-column set contains the F value for this time step. The fifth column in the five-column set contains the value of O for this time step. As shown in the figure, each row in the data random access memory 122 is loaded with a value for use by a corresponding neuron or neural processing unit 126. That is, row 0 is loaded with the value associated with long-term and short-term memory cell 0, and its calculation is performed by the neural processing unit 0; row 1 is loaded with the The value associated with long-term and short-term memory cells 1 is calculated by the neural processing unit 1. By analogy, line 127 is loaded with the value associated with long-term and short-term memory cells 127, and its calculation is performed by the neural processing unit 127. Implementation, as detailed in the subsequent forty-eighth figure.

Preferably, the X value (located in rows 0, 5, 9, and so on to row 55) is written / filled into the data random access memory 122 by the architecture program running on the processor 100 through the MTNN instruction 1400, and The non-framework program executed by the neural network unit 121 reads / uses the non-framework program shown in FIG. 48. Preferably, the I value, the F value and the O value (located in columns 2/3/4, 7/8/9, 12/13/14, and so on to columns 57/58/59) are performed by neural processing The non-schema program of the unit 121 writes / fills in the data random access memory 122, as described later. Preferably, the H value (located in columns 1, 6, 10, and so on to column 56) is written / filled into the data random access memory 122 and read by a non-architecture program executed on the neural processing unit 121. / Use, and read by the framework program running on the processor 100 through the MFNN instruction 1500.

The example in Figure 47 assumes that the framework program will perform the following steps: (1) For 12 different time steps, fill the value of the input X into the data random access memory 122 (rows 0, 5, and so on) (Analog to column 55); (2) Start the non-architecture program of Figure 48; (3) detect whether the non-architecture program has been executed; (4) read the value of output H from the data random access memory 122 ( (Columns 1, 6, and so on to column 59); and (5) Repeat steps (1) to (4) several times until the task is completed, such as the calculation required to recognize the words of the mobile phone user.

In another implementation, the framework program runs Next steps: (1) For a single time step, fill in the data random access memory 122 (such as row 0) with the value of X; (2) start the non-structural program (one of the 48 non-structural programs in Figure 48) The revised version does not require loops, and only accesses a single set of five rows of data random storage memory 122); (3) detects whether the non-structural program is completed; (4) random access memory 122 from data Read out the value of output H (such as column 1); and (5) repeat steps (1) to (4) several times until the task is completed. Which of these two methods is optimal may depend on the sampling method of the input X value of the long-term and short-term memory layers. For example, if this task allows the input to be sampled at multiple time steps (for example, about 12 time steps) and perform calculations, the first method is ideal because this method may bring more computing resources efficiency and / Or better performance, but if this task only allows sampling in a single time step, you need to use the second method.

The third embodiment is similar to the foregoing second method, but different from the second method using a single set of five rows of data random access memory 122, the non-structural program of this method uses multiple sets of five rows of memory, that is, in Each time step uses a different set of five columns of memory, this part is similar to the first way. In this third embodiment, preferably, the architecture program includes a step before step (2). In this step, the architecture program will update the non-architecture program before it is started, for example, in the address 0 instruction. The 122 rows of data random access memory are updated to point to the next set of five rows of memory.

The forty-eighth figure is a table showing a program stored in the program memory 129 of the neural network unit 121. This program is executed by the neural network unit 121 and uses data according to the configuration of the forty-seventh figure. Weights to achieve calculations related to long- and short-term memory cells. The example program in Figure 48 includes 24 non-framework instructions at addresses 0 to 23. The instruction at address 0 (INITIALIZE NPU, CLR ACC, LOOPCNT = 12, DR IN ROW = -1, DR OUT ROW = 2) will clear the accumulator 202 and initialize the loop counter 3804 to the value 12 to execute 12 cycles. Circle group (addresses 1 to 22). This initialization command initializes the row to be read of the data random access memory 122 to a value of -1, and after the first execution of the instruction at address 1, this value will increase to zero. This initialization command also initializes the data random access memory 122 to be written into the row (for example, the register 2606 of the twenty-sixth and thirty-ninth figures) as the row two. Preferably, the initialization command does not place the neural network unit 121 in a wide configuration. Thus, the neural network unit 121 is configured with 512 neural processing units 126. As described in the subsequent chapters, during the execution of the instructions at addresses 0 to 23, 128 of the 512 neural processing units 126 correspond to and operate as 128 long-term and short-term memory cells 4600.

In the first execution of the instructions at addresses 1 to 4, each of the 128 neural processing units 126 (ie, neural processing units 0 to 127) will target the first long-term short-term memory cell 4600. Time step (time step 0) calculates the value of the input gate (I) and writes the value of I into the corresponding text in column 2 of the data random access memory 122; in the second execution of the instructions at addresses 1 to 4, Each of the 128 neural processing units 126 calculates the I value for the second time step (time step 1) corresponding to the long-term short-term memory cell 4600 and writes the I value into the data random access memory 122. Corresponding text of column 7; By analogy, in the twelfth execution of the instructions at addresses 1 to 4, each of the 128 neural processing units 126 will address the twelfth time step of the corresponding long-term and short-term memory cell 4600 ( Step 11) Calculate the I value and write the I value into the corresponding text in column 57 of the data random access memory 122, as shown in FIG. 47.

Further, the multiply-accumulate instruction at address 1 reads the next row behind the current row of the random access memory 122 (row 0 in the first execution, row 5 in the second execution, and so on, etc.) In the twelfth execution, it is column 55). This column contains the memory cell input (X) value associated with the current time step. This instruction will read the weight 0 in the random access memory 124 which contains the Wi value. And the aforementioned read values are multiplied to generate a first multiplying accumulation to be added to the accumulator 202 which has just been cleared by the initialization instruction at address 0 or the instruction at address 22. Subsequently, the multiply-accumulate instruction at address 2 reads the next row of random access memory 122 rows (row 1 in the first execution, row 6 in the second execution, and so on, in the twelfth execution This is column 56). This column contains the memory cell output (H) value associated with the current time step. This instruction will read column 1 containing the Ui value in the weight random access memory 124 and compare the aforementioned value with Multiplying produces a second multiplying accumulation and adds it to the accumulator 202. The H value associated with the current time step is read by the instruction at address 2 (and the instructions at addresses 6, 10, and 18) by the data random access memory 122, generated at the previous time step, and by the address 22 The output instruction writes to the data random access memory 122; however, in the first execution, the instruction at address 2 will write an initial value to the data random access memory row 1 as the H value. Preferably, the structured program will write the initial H value into the data random access record before starting the non-structured program shown in Figure 48. Column 1 of memory 122 (for example, using MTNN instruction 1400); however, the present invention is not limited to this. Non-architecture programs include initialization instructions to write the initial H value to the data. Examples also belong to the scope of the present invention. In one embodiment, the initial H value is zero. Next, the instruction (ADD_W_ACC WR ROW 2) at the address 3 to add the weight text to the accumulator reads the column 2 containing the Bi value in the weight random access memory 124 and adds it to the accumulator 202. Finally, the output instruction at address 4 (OUTPUT SIGMOID, DR OUT ROW + 0, CLR ACC) executes an S-shaped start function on the value of accumulator 202 and writes the execution result to the current output row of the data random access memory 122 (Column 2 in the first execution, column 7 in the second execution, and so on, and column 57 in the twelfth execution) and the accumulator 202 is cleared.

In the first execution of the instructions at addresses 5 to 8, each of the 128 neural processing units 126 calculates the first time step (time step 0) of the corresponding long-term short-term memory cell 4600. Forget the gate (F) value and write the F value into the corresponding text in column 3 of the data random access memory 122; in the second execution of the instructions at addresses 5 to 8, the 128 neural processing units 126 Each neural processing unit 126 calculates its forgetting gate (F) value for the second time step (time step 1) corresponding to the long-term short-term memory cell 4600 and writes the F value into the 8th row of the data random access memory 122. Corresponding text; and so on, in the twelfth execution of the instructions at addresses 5 to 8, each of the 128 neural processing units 126 will target the tenth corresponding to the short-term memory cell 4600. The second time step (time step 11) calculates its forget gate (F) value and writes the F value into the data random access memory 122. The corresponding text of column 58 is shown in Figure 47. The instructions for addresses 5 to 8 calculate the F value similarly to the instructions for addresses 1 to 4, except that the instructions for addresses 5 to 7 will access random access memory 124 from row 3, row 4, and row respectively. 5 Read Wf, Uf and Bf values to perform multiplication and / or addition.

In the twelve executions of the instructions at addresses 9 to 12, each of the 128 neural processing units 126 will calculate its candidate memory cell state for the corresponding time step corresponding to the long-term short-term memory cell 4600 ( C ') value and write the C' value into the corresponding text in column 9 of the weight random access memory 124. The instructions for addresses 9 to 12 calculate the C 'value similarly to the instructions for addresses 1 to 4 above, but the instructions for addresses 9 to 11 will access random access memory 124 from row 6, row 7, and Column 8 reads Wc, Uc and Bc values to perform multiplication and / or addition operations. In addition, the output instruction at address 12 will execute the hyperbolic tangent start function instead of (such as the output instruction at address 4) the S-type start function.

Further, the multiply-accumulate instruction at address 9 reads the current row of data random access memory 122 (row 0 in the first execution, row 5 in the second execution, and so on. The twelfth execution is column 55). This current column contains the memory cell input (X) value associated with the current time step. This instruction also reads column 6 containing the Wc value in the random access memory 124. And multiplying the aforementioned values to generate a first multiplying accumulation and adding to the accumulator 202 just cleared by the instruction at address 8. Next, the multiply-accumulate instruction at address 10 reads the next row of the data random access memory 122 (row 1 in the first execution, row 6 in the second execution, and so on. Twelve executions is column 56), this column It contains the memory cell output (H) value associated with the current time step. This instruction will read the weight 7 of the random access memory 124 containing the Uc value, and multiply the aforementioned values to generate a second multiplied accumulation and add to Accumulator 202. Next, the instruction of adding the weight text to the accumulator at address 11 reads the column 8 containing the Bc value in the weight random access memory 124 and adds it to the accumulator 202. Finally, the output instruction at address 12 (OUTPUT TANH, WR OUT ROW 9, CLR ACC) performs a hyperbolic tangent start function on the value of accumulator 202 and writes the execution result to column 9 of weight random access memory 124. And the accumulator 202 is cleared.

In the twelve executions of the instructions at addresses 13 to 16, each of the 128 neural processing units 126 will calculate a new memory cell state for the corresponding time step corresponding to the long-term short-term memory cell 4600 ( C) value and write the new C value into the corresponding text in column 11 of the weighted random access memory 122. Each neural processing unit 126 also calculates tanh (C) and writes it into the weighted random access memory. Corresponding text in column 10 of 124. Further, the multiply-accumulate instruction at address 13 reads the next row behind the current row of the random access memory 122 (row 2 in the first execution and row 7 in the second execution, and so on) By analogy, in the twelfth execution, it is column 57). This column contains the input gate (I) value associated with the current time step. This instruction reads the weight of the random access memory 124 and contains the state of the candidate memory cell (C ') Value column 9 (just written by the instruction at address 12), and multiplying the aforementioned values to generate a first multiplication accumulation is added to the accumulator 202 just cleared by the instruction at address 12. Next, the multiply-accumulate instruction at address 14 reads the row below the data random access memory 122 (row 3 in the first execution, and in the second execution It is column 8, and so on. In the twelfth execution, it is column 58. This column contains the value of the forget gate (F) associated with the current time step. This instruction reads the weight contained in the random access memory 124. Column 11 of the current memory cell state (C) value (written by the most recent execution of the instruction at address 15) calculated in the previous time step, and multiplying the aforementioned values to generate a second product is added to the accumulator 202. Next, the output instruction at address 15 (OUTPUT PASSTHRU, WR OUT ROW 11) will pass the value of this accumulator 202 and write it into the column 11 of the weight random access memory 124. It should be understood that the C value read by the instruction at address 14 from column 11 of the data random access memory 122 is the C value generated and written by the instructions at addresses 13 to 15 in the most recent execution. The output instruction at address 15 does not clear the accumulator 202, so its value can be used by the instruction at address 16. Finally, the output instruction at address 16 (OUTPUT TANH, WR OUT ROW 10, CLR ACC) executes a hyperbolic tangent start function on the value of accumulator 202 and writes its execution result to column 10 of weight random access memory 124 Used by instruction at address 21 to calculate memory cell output (H) value. The instruction at address 16 clears accumulator 202.

In the first execution of the instructions at addresses 17 to 20, each of the 128 neural processing units 126 calculates the first time step (time step 0) of the corresponding long-term short-term memory cell 4600. Output the gate (O) value and write the O value into the corresponding text in column 4 of the data random access memory 122; in the second execution of the instructions at addresses 17 to 20, the 128 neural processing units 126 Each neural processing unit 126 calculates its output gate (O) value for the second time step (time step 1) of the corresponding long-term and short-term memory cell 4600 and sets the O value Write the corresponding text in column 9 of data random access memory 122; and so on, in the twelfth execution of the instructions at addresses 17 to 20, each of the 128 neural processing units 126 126 calculates the output gate (O) value for the twelfth time step (time step 11) of the corresponding long-term and short-term memory cell 4600 and writes the O value into the corresponding text of column 58 of the random access memory 122, As shown in Figure 47. The instructions for addresses 17 to 20 calculate the O value similar to the instructions for addresses 1 to 4 above. However, the instructions for addresses 17 to 19 will access random access memory 124 from row 12, row 13, and row respectively. 14 Read Wo, Uo and Bo values to perform multiplication and / or addition.

In the first execution of the instructions at addresses 21 to 22, each of the 128 neural processing units 126 calculates the first time step (time step 0) of the corresponding long-term short-term memory cell 4600. The memory cell outputs the (H) value and writes the H value into the corresponding text in column 6 of the data random access memory 122; in the second execution of the instructions at addresses 21 to 22, the 128 neural processing units 126 Each of the neural processing units 126 calculates the memory cell output (H) value for the second time step (time step 1) corresponding to the long-term short-term memory cell 4600 and writes the H value into the data random access memory 122 column. The corresponding text of 11; and so on, in the twelfth execution of the instructions at addresses 21 to 22, each of the 128 neural processing units 126 will target the corresponding long-term and short-term memory cells 4600. The twelfth time step (time step 11) calculates the memory cell output (H) value and writes the H value into the corresponding text of column 60 of the data random access memory 122, as shown in FIG. 47.

Further, the multiply-accumulate instruction at address 21 reads the third row behind the current row of the data random access memory 122 (row 4 in the first execution and row 9 in the second execution. By analogy, in the twelfth execution, it is column 59). This column contains the output gate (O) value associated with the current time step. This instruction reads the weight of the random access memory 124 which contains the value of tanh (C). Column 10 (written by the instruction at address 16), and multiplying the aforementioned values to generate a multiplying accumulation is added to the accumulator 202 just cleared by the instruction at address 20. Subsequently, the output instruction at address 22 will pass the value of accumulator 202 and write it to the data in random access memory 122. The second output row 11 (in the first execution, it will be row 6, in the second execution The execution is column 11 and so on, and the twelfth execution is column 61), and the accumulator 202 is cleared. It should be understood that the value of H in row 122 of the random access memory is written by the instruction at address 22 (in the first execution, it is row 6, in the second execution, it is row 11, and so on. The twelfth execution is column 61) which is the H value consumed / read in the subsequent execution of the instructions at addresses 2, 6, 10, and 18. However, the value of H written in column 61 during the twelfth execution will not be consumed / read by the execution of the instructions at addresses 2, 6, 10, and 18; for a preferred embodiment, this value will be It is consumed / read by the framework program.

The instruction at address 23 (LOOP 1) decrements the loop counter 3804 and returns to the instruction at address 1 if the new loop counter 3804 value is greater than zero.

The forty-ninth figure is a block diagram showing an embodiment of a neural network unit 121. The neural processing unit group in this embodiment has output buffer masking and feedback capabilities. The forty-ninth figure shows a single one by four A neural processing unit group 4901 composed of three neural processing units 126. Although the forty-ninth figure only shows a single neural processing unit group 4901, it should be understood that each neural processing unit 126 in the neural network unit 121 is included in a neural processing unit group 4901. Therefore, a total of There will be N / J neural processing unit groups 4901, where N is the number of neural processing units 126 (for example, 512 for a wide configuration and 1024 for a narrow configuration) and J is a single group The number of neural processing units 126 in 4901 (for example, four for the embodiment of FIG. 49). In the forty-ninth figure, the four neural processing units 126 in the neural processing unit group 4901 are referred to as neural processing unit 0, neural processing unit 1, neural processing unit 2, and neural processing unit 3.

Each neural processing unit in the embodiment of the forty-ninth figure is similar to the neural processing unit 126 of the aforementioned seventh figure, and the components with the same reference numerals in the figure are also similar. However, the multiplex register 208 is adjusted to include four additional inputs 4905, the multiplex register 705 is adjusted to include four additional inputs 4907, and the selection input 213 is adjusted to be able to recover from the original input The selection input 211 and 207 and the additional input 4905 are provided to the output 209, and the selection input 713 is adjusted to provide the selection 203 from the original inputs 711 and 206 and the additional input 4907.

As shown in the figure, the column buffer 1104 in the eleventh figure is the output buffer 1104 in the forty-ninth figure. Further, the characters 0, 1, 2 and 3 of the output buffer 1104 shown in the figure receive the corresponding outputs of the four activation function units 212 associated with the neural processing units 0, 1, 2 and 3. The output buffer 1104 in this section contains N text correspondences In a neural processing unit group 4901, these texts are referred to as an output buffered text group. In the embodiment of the forty-ninth figure, N is four. The four characters of the output buffer 1104 are fed back to the multiplexer registers 208 and 705 and received as four additional inputs 4905 by the multiplex register 208 and as four additional inputs 4907 by the multiplex register 705. Received. The output buffer text group feeds back to the corresponding action of its corresponding neural processing unit group 4901, so that the arithmetic instructions of the non-architecture program can read the text from the output buffer 1104 associated with the neural processing unit group 4901 (that is, the output buffer text group Group) to select one or two characters as its input. For an example, refer to the non-framework program in the following fifty-first figure, as shown in the instructions at addresses 4, 8, 11, 12, and 15. That is, the text in the output buffer 1104 specified in the non-architecture instruction will confirm the value generated by selecting the input 213/713. This capability actually enables the output buffer 1104 to be used as a scratch pad memory, which enables non-architectural programs to reduce the amount of data written to the random access memory 122 and / or the weighted random access memory 124 and subsequent slaves. The number of readings, such as reducing the number of intervening generations and uses during the process. Preferably, the output buffer 1104, or column buffer 1104, includes a one-dimensional register array for storing 1024 narrow characters or 512 wide characters. Preferably, the reading to the output buffer 1104 can be performed in a single time-frequency cycle, and the writing to the output buffer 1104 can also be performed in a single time-frequency cycle. Unlike data random access memory 122 and weighted random access memory 124, which can be accessed by architecture programs and non-architecture programs, the output buffer 1104 cannot be accessed by architecture programs, but can only be stored by non-architecture programs. take.

The output buffer 1104 is adjusted to receive a mask input 4903. Preferably, the mask input 4903 includes four characters corresponding to four characters of the output buffer 1104, and the four characters are associated with the four neural processing units 126 of the neural processing unit group 4901. Preferably, if the mask input 4903 corresponding to the text of the output buffer 1104 is true, the text of the output buffer 1104 will maintain its current value; otherwise, the text of the output buffer 1104 will be The output of the activation function unit 212 is updated. That is, if the mask input 4903 corresponding to the text of the output buffer 1104 is false, the output of the activation function unit 212 will be written into the text of the output buffer 1104. In this way, the output instruction of the non-structural program can selectively write the output of the activation function unit 212 into some characters of the output buffer 1104 and keep the current values of other characters of the output buffer 1104 unchanged. For an example, please refer to The following instructions of the non-framework program in the fifty-first figure are the instructions of addresses 6, 10, 13 and 14 in the figure. That is, the text specified in the output buffer 1104 in the non-architecture program is generated from the value of the mask input 4903.

To simplify the description, the input 1811 of the multiplexer register 208/705 is not shown in the forty-ninth figure (as shown in the eighteenth, nineteenth, and twenty-third). However, embodiments that support the feedback / masking of the dynamically configurable neural processing unit 126 and the output buffer 1104 are also within the scope of the present invention. Preferably, in these embodiments, the output buffer text group can be dynamically configured correspondingly.

It should be understood that although the number of neural processing units 126 in the neural processing unit group 4901 in this embodiment is four, however, The present invention is not limited thereto. Embodiments in which the number of neural processing units 126 in the group is larger or smaller are all within the scope of the present invention. In addition, for an embodiment with a shared activation function unit 1112, as shown in FIG. 52, the number of neural processing units 126 in a neural processing unit group 4901 and the nerves in a activation function unit 212 group The number of processing units 126 has a synergistic effect. The shielding and feedback capabilities of the output buffer 1104 in the neural processing unit group are particularly helpful to improve the computational efficiency associated with long-term and short-term memory cells 4600, as described in the subsequent figures 50 and 51.

Figure 50 is a block diagram showing the neural network of Figure 49 when the neural network unit 121 performs a one-level calculation associated with 128 long and short-term memory cells 4600 in Figure 46. The data random access memory 122, the weight random access memory 124, and the data buffer in the output buffer 1104 of the unit 121 are an example of data configuration. In the example in FIG. 50, the neural network unit 121 is configured as 512 neural processing units 126 or neurons, for example, a wide configuration is adopted. As in the examples of the forty-seventh and forty-eight diagrams, the long-short-term memory layer in the examples of the fifty-first and fifty-one diagrams has only 128 long-short-term memory cells 4600. However, in the example in Figure 50, the values generated by all 512 neural processing units 126 (such as neural processing units 0 to 127) will be used. When the non-architecture program of the fifty-first figure is executed, each neural processing unit group 4901 collectively operates as a long-term and short-term memory cell 4600.

As shown in the figure, the data random access memory 122 is loaded with memory cell input (X) and output (H) values for a series of time steps. Further, for a given time step, there will be a pair of two rows of memory loaded with X and H values, respectively. Randomly store a data with 64 rows Take the memory 122 as an example. As shown in the figure, the value of the memory cell loaded in the data random access memory 122 can be used in 31 different time steps. In the example in Figure 50, rows 2 and 3 are used for time step 0, rows 4 and 5 are used for time step 1, and so on, and rows 62 and 63 are used for time step 30. Value. The first of the two rows of memory contains the X value of this time step, and the second of the two rows contains the H value of this time step. As shown in the figure, each group of four rows in the data random access memory 122 corresponds to the memory system of the neural processing unit group 4901, which is loaded with values for its corresponding long-term and short-term memory cells 4600. That is, rows 0 to 3 are loaded with the values associated with long-term and short-term memory cells 0, and the calculation is performed by the neural processing unit 0-3, that is, the neural processing unit group 0; rows 4 to 7 are loaded with the long The calculation of the short-term memory cell 1 is performed by the neural processing unit 4-7, that is, the neural processing unit group 1. By analogy, lines 508 to 511 are the values associated with the long-term and short-term memory cell 127. It is executed by the neural processing units 508-511, that is, the neural processing unit group 127, as shown in the following fifty-first figure. As shown in the figure, column 1 is not used, and column 0 is loaded with the initial memory cell output (H) value. For a preferred embodiment, a zero value can be filled in by a structural program. However, the present invention is not limited to this. Therefore, it is also within the scope of the present invention to fill the initial memory cell output (H) value of column 0 with non-architectural program instructions.

Preferably, the X value (located in rows 2, 4, 6 and so on to row 62) is written / filled into the data random access memory 122 by the architecture program running on the processor 100 through the MTNN instruction 1400, and Non-architecture program running on the neural network unit 121 for reading / using, for example The non-structural program shown in Figure 50. Preferably, the H value (located in columns 3, 5, 7 and so on to column 63) is written / filled into the data random access memory 122 and read by a non-architecture program executed on the neural network unit 121. / Use, as described below. Preferably, the H value is read by a framework program running on the processor 100 through the MFNN instruction 1500. It should be noted that the non-architecture program of the fifty-first figure assumes that each group of four rows of memory corresponding to the neural processing unit group 4901 (such as rows 0-3, rows 4-7, rows 5-8, and so on) By analogy to rows 508-511), the four X values in a given column are filled with the same value (for example, by a framework program). Similarly, the non-architecture program of the fifty-first figure calculates and writes the same value to the four H values in a given column in the four rows of memory corresponding to the neural processing unit group 4901.

As shown in the figure, the weight random access memory 124 is a weight, offset and memory cell state (C) value required by the neural processing unit loaded with the neural network unit 121. In each of the four rows of memory corresponding to the neural processing unit group 121 (eg, rows 0-3, rows 4-7, rows 5-8, and so on to rows 508-511): (1) line number divided by Rows with the remainder of 4 equal to 3 will be loaded with the values of Wc, Uc, Bc, and C in columns 0, 1, 2, and 6, respectively. (2) The row with the remainder divided by 4 equal to 2 Columns 3, 4 and 5 carry the values of Wo, Uo, and Bo, respectively; (3) The row where the remainder of the row number divided by 4 equals 1 will load the values of Wf, Uf, and Bf in columns 3, 4, and 5, respectively; And (4) The line whose number is divided by 4 and whose remainder is equal to 0 will have the values of Wi, Ui, and Bi in columns 3, 4, and 5, respectively. Preferably, these weights and offset values-Wi, Ui, Bi, Wf, Uf, Bf, Wc, Uc, Bc, Wo, Uo, Bo (in columns 0 to 5)-are executed by the processor 100 Framework program through The MTNN instruction 1400 writes / fills in the weight random access memory 124 and is read / used by a non-architecture program executed on the neural network unit 121, as shown in the non-architecture program of Fig. 51. Preferably, the intermediate C value is written / filled into the random access memory 124 and read / used by a non-framework program executed on the neural network unit 121, as described later.

The example in Figure 50 assumes that the framework program will perform the following steps: (1) For 31 different time steps, fill the value of the input X into the data random access memory 122 (rows 2, 4, and so on) (Column 62); (2) start the non-schema program of the fifty-first figure; (3) detect whether the non-schema program has been executed; (4) read the value of output H from the data random access memory 122 (column 3) , 5, and so on to column 63); and (5) Repeat steps (1) to (4) several times until the task is completed, such as the calculation required to identify the speech of the mobile phone user.

In another implementation method, the architecture program will perform the following steps: (1) for a single time step, fill the data random access memory 122 (such as column 2) with the value of X; (2) start the non-schema program (Figure 51 is a revised version of one of the non-architecture programs, which does not require loops, and only accesses a single pair of two rows of the random storage memory 122); (3) detects whether the non-architecture program has completed execution; (4) Read the value of the output H from the data random access memory 122 (such as column 3); and (5) repeat steps (1) to (4) several times until the task is completed. Which of these two methods is optimal may depend on the sampling method of the input X value of the long-term and short-term memory layers. For example, if this task allows the input to be sampled at multiple time steps (for example, about 31 time steps) and perform calculations, the first method is ideal because this method may bring more computing resources efficiency and / Or better Performance, however, if this task only allows sampling to be performed in a single time step, the second method is needed.

The third embodiment is similar to the foregoing second method, but different from the second method using a single pair of two rows of data random access memory 122, the non-structural program of this method uses multiple pairs of memory rows, that is, in each Time steps use different pairs of memory columns, this section is similar to the first way. Preferably, the architecture program of this third embodiment includes a step before step (2). In this step, the architecture program will update the non-architecture program before it is started, for example, the instruction in the address 1 is updated. The 122 rows of data random access memory are updated to point to the next two rows of memory.

As shown in the figure, for the neural processing units 0 to 511 of the neural network unit 121, the output buffer 1104 is loaded with the memory cell output (H ), Candidate memory cell state (C '), input gate (I), forget gate (F), output gate (O), intermediate value of memory cell state (C) and tanh (C), each output buffers text group In the group (for example, the output buffer 1104 corresponds to the group of four characters of the neural processing unit group 4901, such as the characters 0-3, 4-7, 5-8 and so on to 508-511), the text number is divided by Characters with a remainder of 4 being 3 are represented as OUTBUF [3], characters with a remainder of 4 divided by 4 are represented by OUTBUF [2], characters with a remainder of 4 divided by 4 are represented as OUTBUF [ 1], and the character whose remainder is 0 when the character number is divided by 4 is represented as OUTBUF [0].

As shown in the figure, after the instruction of address 2 in the non-architecture program of the fifty-first figure is executed, for each neural processing unit group 4901, all four characters of the output buffer 1104 are written correspondingly. The initial memory cell output (H) value of the long-term and short-term memory cell 4600. After the instruction at address 6 is executed, for each neural processing unit group 4901, the word OUTBUF [3] in the output buffer 1104 is written into the candidate memory cell state (C ') value corresponding to the long-term and short-term memory cell 4600. , And the other three characters of the output buffer 1104 will maintain their previous values. After the instruction at address 10 is executed, for each neural processing unit group 4901, the OUTBUF [0] characters of the output buffer 1104 will be written into the input gate (I) value corresponding to the long-term short-term memory cell 4600, OUTBUF [ 1] The text will write the value of the forget gate (F) corresponding to the long short-term memory cell 4600, the OUTBUF [2] text will write the value of the output gate (O) corresponding to the long short-term memory cell 4600, and the OUTBUF [3] text Is to maintain its previous value. After the instruction at address 13 is executed, for each neural processing unit group 4901, the word OUTBUF [3] in the output buffer 1104 will be written into the new memory cell state (C) value corresponding to the short-term memory cell 4600. (For the output buffer 1104, the value of C including slot 3 is written in column 6 of the weight random access memory 124, as described in the following fifty-first figure), and the output buffer 1104 The other three texts maintain their previous values. After the instruction at address 14 is executed, for each neural processing unit group 4901, the word OUTBUF [3] in the output buffer 1104 is written into the tanh (C) value corresponding to the long-term short-term memory cell 4600, and the output buffer The other three characters of the device 1104 maintain their previous values. After the instruction at address 16 is executed, for each neural processing unit group 4901, all four characters of the output buffer 1104 are written into the new memory cell output (H) value corresponding to the long-term and short-term memory cell 4600. The execution flow of the aforementioned addresses 6 to 16 (that is, the execution of address 2 is excluded, because address 2 does not belong to one of the program loops. Part) will be repeated 30 times, as a program loop from address 17 back to address 3.

Figure 51 is a table showing a program stored in the program memory 129 of the neural network unit 121. This program is executed by the neural network unit 121 of the 49th figure and is configured according to the 50th figure Use data and weights to achieve calculations related to long- and short-term memory cells. The example program in Figure 51 contains 18 non-framework instructions at addresses 0 to 17, respectively. The instruction at address 0 is an initialization instruction that clears the accumulator 202 and initializes the loop counter 3804 to a value of 31 to execute the loop group 31 times (instructions at addresses 1 to 17). This initialization command also initializes the to-be-written row of the data random access memory 122 (for example, the register 2606 of the twenty-sixth / thirty-nine figure) to the value 1, and the first in the address 16 instruction After the first execution, this value will increase to 3. Preferably, the initialization command does not place the neural network unit 121 in a wide configuration. Thus, the neural network unit 121 is configured with 512 neural processing units 126. As described in the subsequent chapters, during the execution of instructions at addresses 0 to 17, the 128 neural processing unit group 4901 composed of the 512 neural processing units 126 operates as 128 corresponding long-term and short-term memory cells 4600.

The instructions at addresses 1 and 2 do not belong to the loop group of the program and will only be executed once. These instructions generate the initial memory cell output (H) value (for example, 0) and write it to all text in the output buffer 1104. The instruction at address 1 reads the initial H value from row 0 of the data random access memory 122 and places it in the accumulator 202 cleared by the instruction at address 0. The instruction at address 2 (OUTPUT PASSTHRU, NOP, CLR ACC) will pass the value of accumulator 202 to the output buffer 1104, as shown in Figure 50. Bit The "NOP" mark in the output instruction at address 2 (and other output instructions in Figure 51) indicates that the output value will only be written to the output buffer 1104, and will not be written to memory, that is, will not be The data random access memory 122 or the weight random access memory 124 is written. The instruction at address 2 will clear the accumulator 202.

The instructions at addresses 3 to 17 are located in the loop group, and the number of executions is the value of the loop count (such as 31).

Each execution of the instructions at addresses 3 to 6 will calculate the tanh (C ') value of the current time step and write it into the text OUTBUF [3]. This text will be used by the instruction at address 11. More precisely, the multiply-accumulate instruction at address 3 reads the memory associated with this time step from the current read row of data random access memory 122 (such as row 2, 4, 6, and so on to row 62). The cell inputs the (X) value, reads the value of Wc from column 0 of the weighted random access memory 124, and multiplies the aforementioned values to generate a product that is added to the accumulator 202 cleared by the instruction at address 2.

The multiply accumulate instruction at address 4 (MULT-ACCUM OUTBUF [0], WR ROW 1) will read the H value from the text OUTBUF [0] (that is, all four neural processing units 126 of the neural processing unit group 4901), from Column 1 of the weighted random access memory 124 reads the value of Uc, and multiplies the aforementioned values to generate a second product and adds it to the accumulator 202.

The add weight instruction to the accumulator instruction (ADD_W_ACC WR ROW 2) at address 5 reads the value of Bc from column 2 of the weight random storage memory 124 and adds it to the accumulator 202.

The output instruction at address 6 (OUTPUT TANH, NOP, MASK [0: 2], CLR ACC) performs a hyperbolic operation on the value of accumulator 202 The tangent starts the function and only writes the execution result to the text OUTBUF [3] (that is, only the neural processing unit 126 whose number is divided by 4 in the neural processing unit group 4901 will write this result), and accumulates器 202 will be cleared. In other words, the output instruction at address 6 will obscure the text OUTBUF [0], OUTBUF [1], and OUTBUF [2] (as indicated by the instruction term MASK [0: 2]) and maintain its current value, such as the 50th As shown. In addition, the output instruction at address 6 is not written into the memory (as indicated by the instruction term NOP).

Each execution of the instructions at addresses 7 to 10 will calculate the input gate (I) value, forget gate (F) value and output gate (O) value at the current time step and write them into the text OUTBUF [0], OUTBUF respectively. [1], and OUTBUF [2], these values will be used by instructions at addresses 11, 12, and 15. More precisely, the multiply accumulate instruction at address 7 reads the memory associated with this time step from the current read row of data random access memory 122 (such as row 2, 4, 6 and so on to row 62). The cell input (X) value reads Wi, Wf, and Wo values from column 3 of the weighted random access memory 124, and multiplies the foregoing values to generate a product that is added to the accumulator 202 cleared by the instruction at address 6. More precisely, in the neural processing unit group 4901, the neural processing unit 126 with the remainder of the number divided by 4 is 0 and calculates the product of X and Wi, and the neural processing unit 126 with the remainder of the number divided by 4 is 1 calculates X and The product of Wf, and the neural processing unit 126 with a remainder of 2 divided by 4 calculates the product of X and Wo.

The multiply accumulate instruction at address 8 reads the H value from the text OUTBUF [0] (that is, all four neural processing units 126 of the neural processing unit group 4901), and randomly accesses the memory 124 from the weights. Column 4 reads the values of Ui, Uf and Uo, and multiplies the aforementioned values to generate a second product and adds it to the accumulator 202. More precisely, in the neural processing unit group 4901, the neural processing unit 126 with the remainder of the number divided by 4 is 0 to calculate the product of H and Ui, and the neural processing unit 126 with the remainder of the number divided by 4 is 1 calculates H and The product of Uf, and the neural processing unit 126 with a remainder of 2 divided by 4 calculates the product of H and Uo.

The instruction of adding weight text to the accumulator at address 9 (ADD_W_ACC WR ROW 2) will read the values of Bi, Bf and Bo from column 5 of the weight random storage memory 124 and add them to the accumulator 202. More precisely, in the neural processing unit group 4901, the neural processing unit 126 with the remainder of the number divided by 4 is 0 to perform an addition calculation of the Bi value, and the neural processing unit 126 with the remainder of the number divided by 4 is 1 performs Bf The numerical value is added, and the neural processing unit 126 whose number is divided by 4 and whose remainder is 2 performs the addition of the Bo value.

The output instruction at address 10 (OUTPUT SIGMOID, NOP, MASK [3], CLR ACC) will execute an S-shaped start function on the value of accumulator 202 and write the calculated I, F and O values to the text OUTBUF [0 ], OUTBUF [1] and OUTBUF [2]. This instruction does not clear the accumulator 202 without writing to the memory. In other words, the output instruction at address 10 will obscure the text OUTBUF [3] (as indicated by the instruction term MASK [3]) and maintain the current value of the text (that is, C '), as shown in Figure 50.

Each execution of the instructions at addresses 11 to 13 calculates a new memory cell state (C) value generated by the current time step and writes it into the weight random access memory 124 column 6 for the next time step (That is, it is used when the instruction at address 12 is executed in the next loop). More precisely, this value is written in column 6. The remainder of the label divided by 4 in the four lines of text corresponding to the neural processing unit group 4901 is 3. Text. In addition, each execution of the instruction at address 14 will write the value of tanh (C) to OUTBUF [3] for the instruction at address 15.

More precisely, the multiply-accumulate instruction at address 11 (MULT-ACCUM OUTBUF [0], OUTBUF [3]) reads the value of the input gate (I) from the text OUTBUF [0], and reads it from the text OUTBUF [3] Candidate memory cell state (C ') values are multiplied to generate a first product and added to the accumulator 202 cleared by the instruction at address 10. More specifically, each of the four neural processing units 126 in the four neural processing units 126 of the neural processing unit group 4901 calculates the first product of the I value and the C 'value.

The multiply-accumulate instruction at address 12 (MULT-ACCUM OUTBUF [1], WR ROW 6) instructs the neural processing unit 126 to read the value of the forget gate (F) from the text OUTBUF [1], and randomly access the memory 124 from the weight. Column 6 reads its corresponding text, and multiplies it to generate a second product and adds the first product in the accumulator 202 to the instruction at address 11. More precisely, for the neural processing unit 126 with the remainder of the label divided by 4 in the neural processing unit group 4901, the text read from column 6 is the current memory cell state (C) value calculated in the previous time step. The sum of the first product and the second product is the new memory cell state (C). However, for the other three neural processing units 126 of the neural processing unit group 4901, the text read from column 6 is a value that is ignored, because the cumulative value generated by these values will not be used, and That will not be The instructions at addresses 13 and 14 are placed in the output buffer 1104 and will be cleared by the instruction at address 14. In other words, only the new memory cell state (C) value generated by the neural processing unit 126 with the remainder of the number 3 divided by 4 in the neural processing unit group 4901 will be used, that is, the instructions at addresses 13 and 14 use. For the second to thirty-first executions of the instruction at address 12, the value of C read from row 6 of the weighted random access memory 124 is written by the instruction at address 13 during the previous execution of the loop group Value. However, for the first execution of the instruction at address 12, the value of C in column 6 was written by the framework program before starting the non-schema program in Figure 51 or after an adjusted version of one of the non-schema programs. Into the initial value.

The output instruction at address 13 (OUTPUT PASSTHRU, WR ROW 6, MASK [0: 2]) will only pass the value of accumulator 202, that is, the calculated C value, to the text OUTBUF [3] (that is, only neural processing In the unit group 4901, the neural processing unit 126 whose number is divided by 4 and whose remainder is 3 will write the calculated C value to the output buffer 1104), and the weight 6 of the random access memory 124 is the updated one The output buffer 1104 is written as shown in the fiftieth figure. In other words, the output instruction at address 13 will obscure the characters OUTBUF [0], OUTBUF [1] and OUTBUF [2] and maintain their current values (ie, I, F and O values). As mentioned above, only the value of C in column 4 corresponding to the number 4 in the four lines of the text of the neural processing unit group 4901 and the remainder of 3 will be used, that is, used by the instruction at address 12; therefore, non-architecture The program will ignore the values in row 6-2, row 4-6 in row 6 of weight random access memory 124, and so on to the values in row 508-510, as shown in Figure 50 (that is, I, F, and O value).

The output instruction at address 14 (OUTPUT TANH, NOP, MASK [0: 2], CLR ACC) performs a hyperbolic tangent start function on the value of accumulator 202, and writes the calculated tanh (C) value into the text OUTBUF [3], this instruction does not clear the accumulator 202 without writing to the memory. The output instruction at address 14 is the same as the output instruction at address 13. It will obscure the characters OUTBUF [0], OUTBUF [1] and OUTBUF [2] and maintain their original values, as shown in Figure 50.

Each execution of the instructions at addresses 15 to 16 will calculate the memory cell output (H) value generated by the current time step and write it into the second row behind the current output row of the data random access memory 122. The value will be Read by the framework program and used for the next time step (that is, used by the instructions at addresses 3 and 7 in the next loop execution). More precisely, the multiply-accumulate instruction at address 15 reads the output gate (O) value from the text OUTBUF [2], reads the tanh (C) value from the text OUTBUF [3], and multiplies it to produce a The product is added to the accumulator 202 cleared by the instruction at address 14. More precisely, each of the four neural processing units 126 of the four neural processing units 126 of the neural processing unit group 4901 calculates the product of the value O and tanh (C).

The output instruction at address 16 will pass the value of accumulator 202 and write the calculated H value in column 3 in the first execution, and write the calculated H value in column 5 in the second execution, and so on. In the thirty-first execution, the calculated H values are written into column 63, as shown in the fifty figure, and these values will be used by the instructions at addresses 4 and 8. In addition, as shown in Figure 50, these calculated H values will be placed in the output buffer 1104 for subsequent use of the instructions at addresses 4 and 8. Address 16 output Instruction and clears accumulator 202. In one embodiment, the design of the long-term and short-term memory cell 4600 is such that the output instruction at address 16 (and / or the output instruction at address 22 in the forty-eighth figure) has an activation function, such as an S-type or hyperbolic tangent Function instead of passing the accumulator 202 value.

The loop instruction at address 17 decrements the loop counter 3804 and returns to the instruction at address 3 if the new loop counter 3804 value is greater than zero.

It can be found that because of the feedback and shielding capabilities of the output buffer 1104 in the embodiment of the neural network unit 121 in Figure 49, the number of instructions in the loop group of the non-architecture program in Figure 51 is compared. The non-structural instructions in Figure 48 were reduced by approximately 34%. In addition, because of the feedback and shielding capabilities of the output buffer 1104 in the embodiment of the neural network unit 121 in FIG. 49, the memory allocation in the data random access memory 122 of the non-architecture program in FIG. 51 The number of matching time steps is roughly three times that of the forty-eighth picture. The aforementioned improvements are helpful for certain framework program applications that use the neural network unit 121 to perform long-term and short-term memory cell calculations, especially for applications where the number of long-term and short-term memory cells in the long-term and short-term memory cells is less than or equal to 128.

The embodiments of the forty-seventh to fifty-one diagrams assume that the weights and offset values in each time step remain unchanged. However, the present invention is not limited to this, and other embodiments in which the weights and offset values change with time steps are also within the scope of the present invention. The weight random access memory 124 is not as shown in Figures 47 to 50. Fill in a single set of weights and offsets, but fill in different sets of weights and offsets at each time step. The weights of the non-architecture programs in Figures 48 to 51 are stored randomly in memory. The 124 address will be adjusted accordingly.

Basically, in the foregoing embodiments of the forty-seventh to fifty-one figures, the weights, offsets, and intermediate values (such as C, C 'values) are stored in the weight random access memory 124, and the input and output values (Such as X and H values) are stored in the data random access memory 122. This feature is beneficial to the embodiment in which the data random access memory 122 is dual-port and the weight random access memory 124 is a port. This is because the non-architecture program and the architecture program to the data random access memory 122 will have more changes. More traffic. However, because the weight random access memory 124 is relatively large, in another embodiment of the present invention, it is a memory that stores non-architecture and structure program write values interchangeably (that is, the data random access memory 122 and the weight random access memory). Access memory 124). That is, the values of W, U, B, C ', tanh (C) and C are stored in the data random access memory 122 and the values of X, H, I, F and O are stored in the weight random access memory. Body 124 (the modified embodiment of the forty-seventh figure); and W, U, B, and C values are stored in the data random access memory 122 and X and H values are stored in the weight random access memory 124 (the fifty-second embodiment after adjustment). Because the weighted random access memory 124 is larger, these embodiments can process more time steps in a batch. For applications that use the neural network unit 121 to execute calculation-based programs, this feature is beneficial to some applications that can benefit from more time steps and can be designed for the port (such as weight random access memory). 124) Provide sufficient bandwidth.

The fifty-second figure is a block diagram showing an embodiment of a neural network unit 121. The neural processing unit group in this embodiment has output buffer masking and feedback capabilities, and shares an activation function unit. 1112. The neural network unit 121 of the fifty-second figure is similar to the neural network unit 121 of the forty-seventh figure, and the components with the same reference numerals in the figure are similar. However, the four activation function units 212 in FIG. 49 are replaced by a single shared activation function unit 1112 in this embodiment. This single activation function unit will receive four outputs from the four accumulators 202. 217 and generate four outputs to the text OUTBUF [0], OUTBUF [1], OUTBUF [2] and OUTBUF [3]. The operation mode of the neural network unit 212 in the fifty-second diagram is similar to the embodiment described in the forty-ninth to fifty-one diagrams above, and the operation of the shared activation function unit 1112 is similar to the eleventh to the eleventh to The embodiment described in the thirteenth figure.

The fifty-third diagram is a block diagram showing the neural network of the forty-ninth diagram when the neural network unit 121 performs a calculation related to a hierarchy of 128 long-term and short-term memory cells 4600 in the forty-sixth diagram. The data random access memory 122, the weight random access memory 124, and the data buffer in the output buffer 1104 of the unit 121 are another embodiment of data allocation. The example of Figure 53 is similar to the example of Figure 50. However, in the 53rd figure, the values of Wi, Wf and Wo are in column 0 (instead of the 50th figure in column 3); Ui, Uf and Uo values are in the column 1 (instead of 5th Figure 10 is in column 4); Bi, Bf and Bo values are in column 2 (as opposed to column 50 in Figure 50); value C is in column 3 (not column 50 as in Figure 50) ). In addition, the content of the output buffer 1104 of Figure 53 is similar to that of Figure 50, but because of the difference between the non-structural programs of Figure 54 and Figure 51, the content of the third column (that is, I, F, O, and C 'values) appear in output buffer 1104 after the instruction at address 7 is executed (Rather than the fifteenth figure is the instruction at address 10); the contents of the fourth column (that is, I, F, O, and C values) appear in the output buffer 1104 after the instruction at address 10 is executed (instead of (Figure 50 is the instruction at address 13); the contents of the fifth column (ie, I, F, O, and tanh (C) values) appear in the output buffer 1104 after the instruction at address 11 is executed (instead of If the figure 50 is the instruction at address 14); and the contents of the sixth column (that is, the H value) appear in the output buffer 1104 after the instruction at address 13 is executed (as opposed to the address at 50) 16 instructions), as described below.

The fifty-fourth figure is a table showing a program stored in the program memory 129 of the neural network unit 121. This program is executed by the neural network unit 121 of the forty-ninth figure and according to the fifty-third figure. Configure usage data and weights to achieve calculations related to long and short-term memory cells. The example program of figure 54 is similar to the program of figure 51. More precisely, the instructions in addresses 54 to 54 are the same as those in address 51 to 51; the instructions at addresses 7 and 8 in the fifty-fourth illustration are the same as those in the fifty-first illustration. The instructions of 10 and 11; and the instructions of addresses 10 to 14 in the fifty-fourth figure are the same as the instructions of addresses 13 to 17 in the fifty-first figure.

However, the instruction at address 6 in the fifty-fourth figure will not clear the accumulator 202 (in contrast, the instruction at address six in the fifty-first figure will clear the accumulator 202). In addition, the instructions at addresses 7 to 9 in the fifty-first figure do not appear in the non-structural program in the fifty-fourth figure. Finally, with regard to the instruction at address 9 in the fifty-fourth figure and the instruction at address 12 in the fifty-first figure, except for the instruction at address ninety-four in the fifty-fourth figure, the read weight random access memory Column 3 of 124 and the instruction at address 12 in the fifty-first figure are read weights of random access memory except column 6, and the other parts are the same.

Because of the difference between the non-structural program in Figure 54 and the non-structural program in Figure 51, the number of rows of weighted random access memory 124 used in the configuration of Figure 53 is reduced by three, and the program returns The number of instructions in the circle will also be reduced by three. The size of the loop group in the non-structural program of the fifty-fourth figure is substantially only half the size of the loop group in the non-structural program of the 48th figure, and is roughly only the size of the non-structured program in the fifty-first chart. 80% of the size of the loop group.

The fifty-fifth figure is a block diagram showing a part of a neural processing unit 126 according to another embodiment of the present invention. More precisely, for a single neural processing unit 126 of the plurality of neural processing units 126 in the forty-ninth figure, the figure shows the multiplexing register 208 and its associated inputs 207, 211, and 4905, and multiple The work register 705 is associated with inputs 206, 711, and 4907. In addition to the input of the forty-ninth figure, the multiplexer register 208 and the multiplexer register 705 of the neural processing unit 126 each receive a group number (index_within_group) input 5599. In-group number input 5599 indicates the number of a particular neural processing unit 126 within its neural processing unit group 4901. Therefore, for example, taking an embodiment in which each neural processing unit group 4901 has four neural processing units 126 as an example, within each neural processing unit group 4901, one of the neural processing units 126 is numbered in its group and entered 5599 receives the value zero, one of the neural processing units 126 receives the value one in its group number input 5599, one of the neural processing units 126 receives the value two in its group number input 5599, and one of the neural processing units 126 receives the value three in its group number entry 5599. In other words, the number in the group received by the neural processing unit 126 is 5599. The number of the processing unit 126 in the neural network unit 121 divided by J, where J is the number of the neural processing units 126 in the neural processing unit group 4901. Therefore, for example, the neural processing unit 73 enters the number 5599 in its group to receive the value one, the neural processing unit 353 enters the number 5599 in its group to receive the value three, and the neural processing unit 6 enters the number in its group to enter 5599 receives the value two.

In addition, when the control input 213 specifies a preset value, which is represented as "SELF" here, the multiplexer register 208 will select the output buffer 1104 corresponding to the number 5599 in the group to output 4905. Therefore, when a non-architecture instruction is designated to receive data from the output buffer 1104 (indicated as OUTBUF [SELF] in the instructions at addresses 2 and 7 in Figure 57), the value of each neural processing unit 126 The multiplexer register 208 receives its corresponding text from the output buffer 1104. Therefore, for example, when the neural network unit 121 executes the non-architecture instructions at addresses 2 and 7 in the fifty-seventh figure, the multiplexer register 208 of the neural processing unit 73 will select the second among the four inputs 4905. (Number 1) input to receive the text 73 from the output buffer 1104, the multiplexing register 208 of the neural processing unit 353 will select the fourth (number 3) input from the four inputs 4905 to receive from the output buffer The text 353 of 1104, and the multiplexing register 208 of the neural processing unit 6 will select the third (number 2) input among the four inputs 4905 to receive the text 6 from the output buffer 1104. Although it is not used in the non-architecture program in the fifty-seventh figure, the non-architecture instructions can also use the SELF value (OUTBUF [SELF]) to specify the reception of data from the output buffer 1104 and the control input 713 to specify a preset value The multiplexer register 705 of each neural processing unit 126 receives its phase from the output buffer 1104. Corresponding text.

The fifty-sixth diagram is a block diagram showing the neural network unit 121 when the neural network unit performs calculations of the Jordan time recursive neural network associated with the forty-third diagram and uses the embodiment of the fifty-fifth diagram. An example of the data arrangement in the data random access memory 122 and the weight random access memory 124 is. The weight allocation in the weight random access memory 124 in the figure is the same as the example in Figure 44. The configuration of the values in the data random access memory 122 in the figure is similar to the example in Figure 44 except that in this example, each time step has a corresponding pair of two rows of memory to load the input layer node D. Value and output layer node Y value instead of using a set of four rows of memory as in the example in Figure 44. That is, in this example, the values of the hidden layer Z and the content layer C are not written into the data random access memory 122. Instead, the output buffer 1104 is used as a type of draft memory that is one of the hidden layer Z value and the content layer C value, as described in the non-architecture program in FIG. 57. The aforementioned feedback characteristics of the OUTBUF [SELF] output buffer 1104 can make non-architecture programs operate faster (this is to perform two writes and two reads to the data random access memory 122 to The output buffer 1104 performs two writes and two reads instead) and reduces the space of the data random access memory 122 used in each time step, so that the data random access memory 122 of this embodiment The loaded data can be used for about twice the time steps of the embodiment of Figures 44 and 45, as shown in the figure, which is 32 time steps.

Figure 57 is a table showing a program stored in the program memory 129 of the neural network unit 121. The network unit 121 executes and uses data and weights according to the configuration of the fifty-sixth figure to achieve a Jordan time recurrent neural network. The non-structural program in Figure 57 is similar to the non-structural program in Figure 45. The differences are as follows.

The example program in Figure 57 has twelve non-framework instructions at addresses 0 to 11, respectively. The initialization instruction at address 0 clears the accumulator 202 and initializes the value of the loop counter 3804 to 32, so that the loop group (instructions at addresses 2 to 11) is executed 32 times. The output instruction at address 1 places the zero value of accumulator 202 (cleared by the instruction at address 0) into output buffer 1104. It can be observed that during the execution of the instructions at addresses 2 to 6, the 512 neural processing units 126 correspond to and operate as 512 hidden layer nodes Z, and the execution of the instructions at addresses 7 to 10 In the process, it corresponds to and operates as 512 output layer nodes Y. In other words, the 32 executions of the instructions at addresses 2 to 6 will calculate the hidden layer node Z values of 32 corresponding time steps and put them into the output buffer 1104 for the correspondence of the instructions at addresses 7 to 9. 32 executions are used to calculate the output layer node Y of the 32 corresponding time steps and write them into the data random access memory 122, and provide the corresponding 32 executions of the address 10 instruction to use this The content layer node C of 32 corresponding time steps is put into the output buffer 1104. (The content layer node C placed in the 32nd time step in the output buffer 1104 will not be used.)

In the first execution of instructions 2 and 3 (ADD_D_ACC OUTBUF [SELF] and ADD_D_ACC ROTATE, COUNT = 511), each of the 512 neural processing units 126 will output 512 of the output buffer 1104. Content node C value It accumulates to its accumulator 202. These content node C values are generated and written by the instruction execution at addresses 0 to 1. In the second execution of the instructions at addresses 2 and 3, each of the 512 neural processing units 126 will accumulate the 512 content node C values of the output buffer 1104 to its accumulator 202. These The content node C value is generated and written by the execution of instructions at addresses 7 to 8 and 10. More precisely, the instruction at address 2 will instruct the multiplexer register 208 of each neural processing unit 126 to select its corresponding output buffer 1104 text, as described above, and add it to the accumulator 202; address 3 The instruction will instruct the neural processing unit 126 to rotate the content node C value in a 512-character rotator. The 512-character rotator is collectively operated by the multiplexed register 208 of the 512 neural processing units. So that each neural processing unit 126 can accumulate the 512 content node C values to its accumulator 202. The instruction at address 3 does not clear the accumulator 202, so the instructions at addresses 4 and 5 can add the D value of the input layer node (multiplied by its corresponding weight) to the value accumulated by the instructions at addresses 2 and 3. Content layer node C value.

In the execution of instructions at addresses 4 and 5 (MULT-ACCUM DR ROW + 2, WR ROW 0 and MULT-ACCUM ROTATE, WR ROW + 1, COUNT = 511), one of the 512 neural processing units 126 Each neural processing unit 126 performs 512 multiplication operations to associate the row of the data random access memory 122 with the current time step (for example, for time step 0, it is column 0, and for time step 1, it is Column 2, and so on. For time step 31, 512 input node D values are multiplied by weights in columns 0 to 511 of the random access memory 124 corresponding to this neural processing unit 126. Weight of the line 512 products are generated, and the cumulative results of the execution of the instructions C and 2 on the 512 content node C values are accumulated to the accumulator 202 of the corresponding neural processing unit 126 to calculate the hidden node Z layer value.

In each execution of the instruction at address 6 (OUTPUT PASSTHRU, NOP, CLR ACC), the values of the 512 accumulators 202 of the 512 neural processing units 126 are passed and written to the corresponding text in the output buffer 1104, and The accumulator 202 is cleared.

During the execution of instructions at addresses 7 and 8 (MULT-ACCUM OUTBUF [SELF], WR ROW 512 and MULT-ACCUM ROTATE, WR ROW + 1, COUNT = 511), each of these 512 neural processing units 126 The neural processing unit 126 will perform 512 multiplication operations, and the 512 hidden node Z values in the output buffer 1104 (produced and written by the corresponding execution of the instructions at addresses 2 to 6) will be multiplied by the random storage. The weights in the columns 512 to 1023 of the memory 124 corresponding to the trip of the neural processing unit 126 are taken to generate 512 multiply-accumulate and add to the accumulator 202 of the corresponding neural processing unit 126.

In each execution of the instruction at address 9 (OUTPUT ACTIVATION FUNCTION, DR OUT ROW + 2), a start function (such as hyperbolic tangent function, S-shaped function, correction function) will be executed for the 512 accumulated values to calculate Output node Y value, this output node Y value will be written into the column of the data random access memory 122 corresponding to the current time step (for example, for time step 0, it is column 1 and for time step 1, it is Is column 3, and so on, for time step 31 it is column 63). The instruction at address 9 does not clear accumulator 202.

The instruction at address 10 (OUTPUT PASSTHRU, (NOP, CLR ACC) in each execution, the 512 values accumulated by the instructions at addresses 7 and 8 will be put into the output buffer 1104 for the next execution under the instructions at addresses 2 and 3, and the accumulator 202 Will be cleared.

The loop instruction at address 11 will decrement the value of the loop counter 3804. If the value of the new loop counter 3804 is still greater than zero, it will instruct the instruction to return to address 2.

As described in the section corresponding to figure 44, in the example of executing a Jordan time recursive neural network using a non-architecture program of figure 57, although an activation function is applied to the value of accumulator 202 to generate an output Layer node Y value, however, this example assumes that the accumulator 202 value is passed to the content layer node C before the start function is applied, rather than the actual output layer node Y value. However, for the Jordan time recursive neural network that applies the activation function to the value of the accumulator 202 to generate the content layer node C, the instruction at address 10 will be removed from the non-architecture program in Figure 57. In the embodiment described herein, the Elman or Jordan time-recurrent neural network has a single hidden node layer (such as the fortieth and forty-two graphs), but it should be understood that these processors 100 and neural networks The embodiment of unit 121 may use a method similar to that described herein to efficiently perform computations associated with a temporal recurrent neural network with multiple hidden layers.

As described in the previous section corresponding to the second figure, each neural processing unit 126 operates as a neuron in an artificial neural network, and all neural processing units 126 in the neural network unit 121 are processed in a large-scale parallel. This method effectively calculates the neuron output at one level of this network. The parallel processing method of this neural network unit, especially In particular, a rotator composed of a multiplex register of a neural processing unit is not intuitively conceivable in the traditional way of calculating the output of a neuron layer. Furthermore, traditional methods usually involve calculations that are associated with a single neuron or a very small subset of neurons (for example, using parallel arithmetic units to perform multiplication and addition calculations), and then continue to perform associations below the same level The calculation of a neuron, and so on, continues in a sequential manner until the calculation of all neurons in this level is completed. In contrast, in the present invention, in each time-frequency period, all the neural processing units 126 (neurons) of the neural network unit 121 execute a small set (e.g. A multiplication and accumulation). After about M time-frequency periods have ended-M is the number of nodes connected in the current level-the neural network unit 121 will calculate the output of all neurons. In many artificial neural network configurations, because there are a large number of neural processing units 126, the neural network unit 121 can calculate its neuron output value for all neurons in the entire hierarchy at the end of M time-frequency cycles. As described in this article, this calculation is efficient for all types of artificial neural network calculations. These artificial neural networks include, but are not limited to, feedforward and time-recurrent neural networks such as Elman, Jordan, and long-term and short-term memory networks. . Finally, although the neural network unit 121 is configured as 512 neural processing units 126 (for example, in a wide text configuration) to perform time-recurrent neural network calculations in the embodiment herein, the present invention is not limited to this. The neural network unit 121 is configured as an embodiment of 1024 neural processing units 126 (for example, in a narrow text configuration) to perform the calculation of the time recursive neural network unit, and as described above, the number of neural processing units 126 other than 512 and 1024 Neural Network Unit 121 also belongs to the scope of the present invention.

However, the above are only the preferred embodiments of the present invention. When the scope of implementation of the present invention cannot be limited by this, that is, the simple equivalent changes and modifications made according to the scope of the patent application and the description of the invention, All are still within the scope of the invention patent. For example, software can perform the functions, manufacturing, shaping, simulation, description, and / or testing of the devices and methods described in the present invention. This can be achieved by general programming languages (such as C, C ++), hardware description language (HDL) including Verilog HDL, VHDL, etc., or other existing programs. This software can be installed on any known computer-usable medium, such as magnetic tape, semiconductor, magnetic disk, optical disk (such as CD-ROM, DVD-ROM, etc.), network connection, wireless or other communication media. Embodiments of the devices and methods described herein may be included in a semiconductor intellectual property core, such as a micro-processing core (such as an implementation in a hardware description language) and converted into hardware through the fabrication of integrated circuits. In addition, the devices and methods described herein may include a combination of hardware and software. Therefore, any embodiments described herein are not intended to limit the scope of the invention. In addition, the present invention can be applied to a microprocessor device of a general-purpose computer. Finally, those skilled in the art can use the concepts and embodiments disclosed in the present invention as a basis to design and adjust different structures to achieve the same purpose without departing from the scope of the present invention.

100‧‧‧ processor

101‧‧‧ instruction fetch unit

102‧‧‧Instruction cache

103‧‧‧Architecture Instructions

104‧‧‧Instruction translator

105‧‧‧microinstructions

106‧‧‧ Rename Unit

108‧‧‧ Reserved Station

112‧‧‧Other execution units

114‧‧‧Memory Subsystem

116‧‧‧General purpose register

118‧‧‧Media Register

121‧‧‧ Neural Network Unit

122‧‧‧Data Random Access Memory

124‧‧‧ Weight Random Access Memory

126‧‧‧Neural Processing Unit

127‧‧‧Control and status register

128‧‧‧Sequencer

129‧‧‧program memory

123,125,131‧‧‧Memory address

133‧‧‧ results

Claims (23)

A device includes: an output buffer for loading N characters, the N characters are allocated to N / J mutually exclusive output buffer text groups, and the output buffer text group has one of the N characters J characters, J is greater than 2, N is at least twice J; an array of N processing units, the N processing units are assigned to N / J mutually exclusive processing unit groups, the processing unit group The group has J processing units among the N processing units, and each processing unit group corresponds to one of the N / J output buffer text groups, and each of the processing units includes: first and second multiplexing Registers, each of the multiplexing registers includes: at least J + 1 inputs, one of the J + 1 inputs receives a operand from a memory, and the other of the J + 1 inputs J inputs receive the J characters of the corresponding output buffer text group; an output; and a control input to control the selection of the J + 1 inputs to provide to the output; an accumulator having a Output to provide output buffer text corresponding to one of the N output buffer texts; and The operation unit has first, second, and third inputs. The first and second inputs are respectively used to receive the outputs of the first and second multiplexing registers. The third input is used to receive the accumulator. Output of the arithmetic unit for the first, second and third inputs Perform an operation to generate a result to accumulate to the accumulator; wherein the output buffer includes a mask input to control which characters in the N characters will maintain their original values or output from the corresponding accumulator Update. For example, the device of the first patent application range, wherein the mask input specifies J values, and each of the J values controls the corresponding text of one of the J characters of each output buffer text group Keep its current value or update it with the value of its corresponding accumulator. For example, the device of the scope of patent application, wherein the mask input is generated in response to the device executing an instruction. For example, the device of the scope of application for patent No. 3 further includes: a program memory for loading a plurality of program instructions, the program instructions including the instructions executed by generating the mask input; wherein the device is a processor An execution unit, the processor includes the device, the processor has an architecture instruction set, and the instructions loaded in the program memory are non-architecture instructions different from the architecture instructions in the architecture instruction set of the processor . For example, the device in the fifth item of the patent application scope further includes: a plurality of activation function units, selectively performing an activation function on the output of the accumulator to generate a result and providing the corresponding output in the N output buffer characters. Buffer text. For example, the device of the scope of patent application of the invention further includes: an iteration count, which is generated in response to an instruction executed by the device. The iteration count is controlled by the arithmetic unit for the first, second and third. Enter the number of times the operation is performed to produce a result that is accumulated in the accumulator. For example, the device in the scope of patent application of the invention further includes: a first memory for providing a first set of N characters to provide the first input to the corresponding first multiplexing register; and The second memory is used to provide a second set of N characters to provide the first input to the corresponding second multiplexer register. For example, the device in the seventh scope of the patent application, wherein the N characters of the output buffer can be written into the first or second memory. For example, the device of the scope of patent application, wherein the control input of the first and second multiplexer registers is generated in response to an instruction executed by the device. For example, the device in the ninth scope of the patent application, wherein the control input is provided to each of the N processing units in response to the execution of an instruction. For example, the device of claim 1 in the patent scope, wherein the first multiplexer register includes an offset input for receiving the output of the first multiplexer register of the adjacent processing unit, when The control input of the first multiplexer register specifies the offset input, and the first multiplexer registers of the N processing units collectively operate as an N-character rotator. For example, the device of claim 11 of the patent scope, wherein the second multiplexer register includes an offset input for receiving the output of the second multiplexer register of the adjacent processing unit, when The control input of the second multiplexer register specifies the offset input, and the N processing orders The second multiplex registers are operated collectively as a spinner of N characters. A processor includes: an execution unit including: an output buffer for loading N characters, the N characters are allocated to N / J mutually exclusive output buffer text groups, and the output buffer text groups With J characters in the N characters, J is greater than 2, N is at least twice J; an array of N processing units, the N processing units are allocated to N / J mutually exclusive processing units Group, the processing unit group has J processing units of the N processing units, each processing unit group corresponds to one of the N / J output buffering text groups, and each processing unit includes: First and second multiplexing registers, each of which includes: at least J + 1 inputs, one of the J + 1 inputs having a first input receiving an operand from a memory, the J The other J inputs of the +1 input receive the J characters of the corresponding output buffer text group; an output; and a control input to control the selection of the J + 1 inputs to provide to the output ; An accumulator with an output corresponding to one of the N output buffer characters A text buffer; and an arithmetic unit having a first, second and third input, the first and second inputs for receiving respectively the first and second multiplexing The output of the register. The third input is used to receive the output of the accumulator. The arithmetic unit performs an operation on the first, second, and third inputs to generate a result to accumulate to the accumulator. Among them, the The output buffer includes a mask input to control which characters in the N characters will maintain their original values or be updated with the output of their corresponding accumulator. For example, the processor of claim 13 in the patent scope, wherein the mask input specifies J values, and each of the J values controls the corresponding one of the J characters of each output buffer text group. The text maintains its current value or is updated with the value of its corresponding accumulator. For example, the processor of claim 13, wherein the mask input is generated according to an instruction executed by the execution unit. For example, the processor of claim 15 in which the execution unit further includes a program memory for loading a plurality of program instructions, and the program instructions include the instruction executed by generating the mask input; wherein, the The processor has an architecture instruction set, and the instructions loaded in the program memory are non-architecture instructions different from the architecture instructions in the architecture instruction set of the processor. For example, the processor according to item 13 of the patent application, wherein the execution unit further includes a plurality of activation function units, and selectively executes an activation function on the output of the accumulator to generate a result and provide the result to the N output buffer words. The corresponding output buffer text. For example, the processor of the scope of application for item 13, wherein, The execution unit further includes a count of the number of iterations, which is generated in response to an instruction executed by the execution unit. The number of iterations controls the arithmetic unit to perform the operation on the first, second, and third inputs to generate a result accumulated to the Accumulator times. For example, the processor of claim 13 in which the execution unit further includes: a first memory for providing a first set of N characters to the corresponding first multiplexing register. A first input; and a second memory for providing a second set of N characters to provide the first input to the corresponding second multiplexer register. For example, the processor according to item 13 of the patent application, wherein the control input of the first and second multiplexer registers is generated according to an instruction executed by an execution unit. For example, the processor of claim 13 in which the first multiplexer register includes an offset input for receiving the output of the first multiplexer register of an adjacent processing unit, When the control input of the first multiplexer register specifies the offset input, the first multiplexer registers of the N processing units collectively operate as a rotator of N characters. For example, the processor of claim 21, wherein the second multiplexer register includes an offset input for receiving the output of the second multiplexer register of the adjacent processing unit, When the control input of the second multiplexer register specifies the offset input, the second multiplexer registers of the N processing units operate collectively as a N-character rotator. A computer program product encoded in at least one non-transitory computer-useable medium for use by a computer device, including: a computer-useable code included in the medium, used to describe a device, the computer-useable code Including: a first code for describing an output buffer, the output buffer is used to load N characters, the N characters are allocated to N / J mutually exclusive output buffer text groups, and the output buffer The text group has J characters of the N characters, J is greater than 2, N is at least twice of J; the second code is used to describe an array of N processing units, and the N processing units are Assigned to N / J mutually exclusive processing unit groups, the processing unit group has J processing units of the N processing units, and each processing unit group corresponds to N / J output buffer text groups In one of them, each of the processing units includes: first and second multiplexing registers, each of which includes: at least J + 1 inputs, and one of the J + 1 inputs Receives an operand from a memory, the other J of the J + 1 inputs are lost To receive the J characters of the corresponding output buffer text group; an output; and a control input to control the selection of the J + 1 inputs to provide to the output; an accumulator having an output to Output buffer text corresponding to one of the N output buffer texts; and An arithmetic unit having first, second, and third inputs, the first and second inputs are used to receive outputs of the first and second multiplexer registers, respectively, and the third input is used to receive the accumulation The arithmetic unit performs an operation on the first, second and third inputs to generate a result and accumulate the result to the accumulator; wherein the output buffer includes a mask input to control the N characters Which characters in the text will maintain their original values or be updated with the output of their corresponding accumulator.